>If a human made a mistake he or she is held accountable individually and punished.

Really? I am not so sure about that. Who'd get punished for misidentifying a face in book, say, of faces?

Fine, can be held accountable. Seriously, are you intentionally trying to miss the point in order to nitpick on this minor detail?

Engineers can be held accountable for a failure in a system they said would work, and it happens when something goes wrong with a bridge or building. I do not agree with the point. It is like saying, "A condominium in Florida cannot be held accountable." :-)

You are entirely missing the point.

We're talking about systems, algorithms, black box models that we don't understand or have the ability to reason about, that are being used to make decisions for us.

Those systems are already proven to have embedded biases.

But now instead of a human we can question, we only have inscrutable machines that no one, not even the people who built them, understand.

If condominiums in Florida were making the decision about who could get a mortgage and live in them, then your analogy would make sense.

But they don't.

There are, on the other hand, computer systems that absolutely do, and it's very possible (I would predict probable) that they will end up effectively re-creating redlining.

Furthermore, going back to your broken analogy, unlike the case with building codes and engineering professional organizations, there are no laws or regulations that could be used to hold accountable those who build or use those systems.

You can inspect the design of a condo but ML models can't be examined, their training data is the only thing you can examine. It's like being able to inspect the raw materials but nothing else for the condo.

If a cop keeps misidentifying black people's faces, they can be removed from their job or disciplined by the department.

+1!

And further to your point, if a DA's office keeps arresting people incorrectly, they won't keep that job.

The fact that we as a society are not okay with just arresting the wrong person is how we started talking about 'stop & frisk' and its shortcomings in the public domain. It's how we had BLM summer and many other protests after the killing/arrest/etc. of innocent people.

and that's how we should respond when the wrong person is arrested

who can I call when "the AI got it wrong"? It's worse than google customer service.

The reality here is none of that will happen in all but the most egregious cases.

If either of those scenarios came to pass the police unions stand ready to defend them. At most the cop will get some paid leave and a settlement payout (as well as the victim at great expense to the town/city) and maybe cop will be transferred to a new department while the whole complaint is put under a NDA.

> Even if not on paper, humans now have an excuse to lay responsibility on the software.

This, right here, is the real key and I hope people really fully appreciate this.

We can now have machines that can be biased for us, and when that bias becomes apparent, people can shrug their shoulders and say "what can you do, it's the machine's fault".

That is a very dangerous road, and we're already well on our way down that path.

It honestly seems to me that the core issue here is actually the same one found in digital voting technology - where is the vetting? There must be vetting at multiple levels of the process, or these use cases are negligent at best and malicious at worst.

"Our highly advanced computer technology has analyzed the voting results, and thrown out all of the results it has determined to be fraudulent. If your vote was thrown out due to the algorithm's fraud determination, you will be arrested for attempting to subvert democracy. What's that? You want to double check the computer? That's not necessary, because it's a computer after all, it doesn't make mistakes."

"Our highly advanced computer technology said with 100% certainty that you are the person responsible for this murder. You say you were on the other side of town at that time, and you claim to have witnesses. This is impossible, because our computer algorithm doesn't make mistakes, it's a computer after all. You are hereby sentenced to the death penalty, and the computer has recommended that you receive no appeal, since it has determined you to be guilty beyond any doubt."

> There must be vetting at multiple levels of the process, or these use cases are negligent at best and malicious at worst.

You've got it exactly.

The problem is modern AI systems cannot be vetted because neither their creators nor their users actually understand how they work (yes, the mechanics are understood, but the emergent properties are not).

I believe we need regulations that prevent the use of unauditable technologies in a bunch of key industries, including banking, law enforcement, healthcare, etc. Basically anywhere where a person's rights may be violated by these systems.

We actually fought this same battle not too long ago with photo radar systems due to issues with their accuracy, and the result was mandated transparency (at least in some jurisdictions). These new technologies just supercharge the problem.

> If a human made a mistake he or she is held accountable individually and punished.

Is she?

Yes, in a properly run police department, rare as they may be.

Maybe try reading up on the reasoning behind the whole "blameless retrospective" thing that's popular to advocate for when software development discussions pop up around here?

Blameless retrospectives only make sense when everyone is actually incentivized towards quality of output. That is historically true of neither (American) police organizations nor the vendors who sell to them.