Not Everything Needs a Brain

In 2016, a startup called Juicero raised over a hundred million dollars to build a Wi-Fi connected juicer. Four hundred dollars, a companion app, one job: squeeze proprietary packets of pre-juiced fruit into a cup. Then two Bloomberg reporters tried squeezing the same packets by hand. It worked. Slightly faster than the machine built specifically to do it.

That's the whole story: a well-funded company solved a problem nobody had, with technology nobody needed, because the technology was the point and the juice was the excuse. It became Silicon Valley's favourite punchline for a year, then faded, the way punchlines do.

We're doing it again. Same shape, bigger stage, a far more capable technology standing in for the Wi-Fi chip. This time it's not a kitchen gadget. It's how entire companies are being run.

The decision, not the tool

Late 2025: Amazon's ecommerce leadership mandates Kiro, its own AI coding assistant, as company standard, with an 80 percent weekly usage target tracked on dashboards. About 1,500 engineers push back internally, most preferring Claude Code. Leadership holds the line.

What follows reads like a string of near misses that stop missing. December: an AI agent deletes and rebuilds an AWS environment on its own initiative, 13-hour outage. February: engineers let another tool resolve an incident unsupervised, badly. March 2: bad delivery estimates cost 120,000 orders and 1.6 million site errors. March 5: a six-hour outage drops North American order volume 99 percent. Six point three million orders, gone, in a day.

The executive who signed the original mandate is now the one adding human guardrails back in. That's the honest part of the story: not that it happened, but that fixing it meant admitting the guardrails were removed on purpose.

You'll recognise a smaller version if you follow dating apps instead of AWS status pages. Grindr's CEO pushed an AI-native mandate through his engineering org over internal objection, then admitted, almost in passing, that the coding agents had learned their bad habits from Grindr's own buggy codebase. Around the same time, Grindr rolled out its AI companion opt-out by default, right after telling users privacy was central to the product. Nobody was asked either time. The mandate is what broke things, not the tool. A mandate is a way of skipping the check on whether a decision was good before scaling it past the point where anyone can quietly say no.

Amplifying what's already there

SAP's Joule assistant now sits inside more than thirty SAP products, doing real work in finance, HR, and supply chain. Nearly every analyst who's looked closely lands on the same line, worded differently each time: Joule is only as good as the data underneath it. Bad master data, unreliable output. Messy systems, answers nobody trusts.

That's not new information, and that's the interesting part. Steven Freemantle has been optimising SAP supply chains since 2003, and working inside SAP implementations since 1994. Three decades of watching this exact failure mode play out, long before agentic AI arrived. His description of how firefighting takes hold in SAP shops is uncomfortably familiar: poor data integrity becomes daily workarounds, blame cascades from warehouse to procurement to the MRP team until everyone just blames SAP, and the fault sits upstream the whole time, in data nobody cleaned up. His conclusion: reactive, unstable behaviour cannot be automated into stability. The system doesn't correct bad inputs, it amplifies them into more confident-looking bad outputs.

Swap "MRP logic" for "AI agent" and you've got the exact caveat every Joule analyst is writing now, years later, about a much smarter system running on the same unfixed foundation.

To SAP's credit, there's a real answer here, not just a caveat. Their AI ethics policy requires human sign-off for anything high-risk: hiring, performance reviews, supplier risk, financial approvals. Joule can draft and recommend, but a person has to say yes. That's care built into the workflow, not a values statement nobody reads.

What's missing is the customer-facing half of that care. Everyone agrees clean data is the prerequisite. Few describe SAP actively walking customers there before switching Joule on. If the tool amplifies whatever's underneath it, the responsible move is helping people fix the underneath first, not handing them a megaphone and hoping the sound was good to begin with.

What I've learned trying to do this properly

I've been building with AI for a while, and I've written before about what that actually looks like when you're paying attention. The tool isn't the variable that decides whether the output is any good. You are. I push back on drafts I don't like, ask for a different approach when the first one is technically fine but wrong in spirit, and I still read every line before it ships, because I'm the one standing behind it, not the model.

That's not a productivity story. Some days it's slower, on purpose. But the code still has to work for the person using it, and they don't care how fast it was written. They care whether it works, whether their data is safe, and whether anyone was paying attention while it was built.

What care actually looks like

Not "don't use AI." That ship sailed, and it should have. Used well, it's a genuinely good editor, a tireless first pass, and increasingly good at catching the scam pattern or logic bug a tired human misses at four in the afternoon. Amazon and Grindr aren't arguments against the tool. They're arguments against skipping the judgement.

So: review the code like a junior developer's pull request, not a package that arrived complete. Ask where AI is solving something real, not just where it's the thing everyone's doing this quarter. Listen to the people using what you build before the usage dashboard. Amazon had a dashboard too, and it said exactly the wrong thing right up until it very publicly didn't. And where AI genuinely helps, point it at care rather than speed: catching the bug before it ships, flagging the fraud pattern before it costs someone money, reviewing the edge case a human would have rushed past.

Juicero didn't fail because internet-connected juicers are a bad idea. It failed because nobody stopped to ask what the juice actually needed. That's still the whole question, for every version of this we keep building. Not can we. Whether you understand what you're making well enough to want it to be good, on behalf of someone who's going to trust you with it.