Gregory's Server
(I've already written at length about how this phenomenon is fucking up medical records and consequently medical science. Series starts here https://siderea.dreamwidth.org/1540620.html , particularly see part 2 "Power's Deformation of Data".)
Bugs and other faults in such software are like mold growing in crevasses that sunlight can't reach, and which stagnant water never wholly evaporates.
So we can either assume that in these systems in which it's hard-to-impossible to detect faults, there are none to detect, or we have to assume there's probably undetected faults. Probably many, because they will accumulate.
Into these irremediable reservoirs of pestilential technological suckitude, AI is going to be - and presumably already has - been injected.
So that will be just ducky.
And that's why I say I suspect there's already systems screwing things up for everyone already.
7/?
PS, I would be remiss if I didn't call out the Horizon Scandal, currently unfolding in the UK, as an example of how power relations X software breed bugs that then become social oppression and justice.
For those of you who hadn't heard, it's recently come to light that the UK postal system [pro|per]secuted some 700 of their own postmasters for theft and fraud which it has turned out to be entirely because of bugs in the accounting software the post office rolled out.
"More than 700 former Post Office staff were wrongly prosecuted for theft and false accounting in what has been described as "the most widespread miscarriage of justice in UK history"." (SkyNews)
"Some ended up in jail, others became bankrupt trying to repay money they did not owe, and a few even took their own lives before their names could be cleared." (SkyNews)
8/8
PS, I would be remiss if I didn't call out the Horizon Scandal, currently unfolding in the UK, as an example of how power relations X software breed bugs that then become social oppression and justice.
For those of you who hadn't heard, it's recently come to light that the UK postal system [pro|per]secuted some 700 of their own postmasters for theft and fraud which it has turned out to be entirely because of bugs in the accounting software the post office rolled out.