Is Checking AI’s Work More Important Than Understanding It?

Picture an AI assistant that pays your company’s invoices. It reads the bills, decides which ones are real, and sends the money. Fast and cheap. But what happens when it falls for a fake invoice?

Normally you’d treat this like hiring a new employee: give rewards, keep an eye on them, punish mistakes. Law professor Noam Kolt argues that doesn’t work well with AI. The AI knows things you don’t. It makes its own choices. You can’t easily see why it decided something. And it works far too fast for any manager to keep up.

Check the action, not the thinking

A June 2026 paper by Jakob Salfeld-Nebgen borrows a trick from how we treat doctors. Nobody checks what a doctor was thinking. But to prescribe a controlled drug, she needs a confirmed patient record, a check that the medicines don’t clash, and a valid licence, each confirmed by someone else.

Now apply that to the invoice AI. Let it think however it likes. But before any money leaves the account, independent parties have to confirm the supplier is real, the invoice matches an order, and the amount was approved. No confirmations, no payment.

It works. In one public test, people tricked an AI agent into doing things it shouldn’t three times out of four. With strict checks before each action, none of 879 attempts got through.

Business is already doing it

Payments lead the way, because mistakes cost real money. Google’s payment system for AI shopping assistants, backed by more than 60 companies, keeps a signed record of exactly what the customer agreed to buy. Mastercard is testing a similar system with PayPal, so that if something goes wrong, the normal card-dispute process still works.

Governments are slower. Singapore asks companies to get a human’s OK before an AI does something that can’t be undone, like a payment, but it’s only guidance. Critics say the EU’s AI law assumes you can simply stop an AI and reverse what it did. Often you can’t. And people asked to approve every step tend to click “yes” without looking.

Three problems remain

Someone has to check the checkers. If the confirmations come from a party with its own interests, we’ve seen how that ends: think of credit rating agencies before 2008.

Checks cost time and money, so they only make sense for big decisions. Someone has to decide which ones.

And a clever attacker can split a harmful job into small steps that each look harmless.

So the biggest question isn’t technical. Who decides which AI actions are risky enough to need checks? Right now, it’s often the company that built the tool. Would you let a carmaker decide whether its own car needs a safety test?

Sources: Kolt, Notre Dame Law Review (2026); Salfeld-Nebgen, arXiv (June 2026); Uchibeke, arXiv (March 2026); Google AP2 announcement (2025); Singapore IMDA (Jan 2026); Jones, Dragan & Steinhardt, ICML 2025.