When AI Goes Off The Rails

Alex Shannon: There are two ways this story could end. In the first version, companies learn to deploy AI thoughtfully, with proper controls and human oversight, creating genuine value while protecting jobs that matter. In the second version, we get decision-makers drunk on AI hype, burning through hundreds of millions of dollars while laying off a quarter of their workforce for digital agents they don’t understand.

Sam Hinton: And early reports suggest we just got a very expensive preview of which future we’re heading toward. Half a billion dollars expensive.

Alex Shannon: The question isn’t whether AI will transform business - it’s whether businesses will survive their own AI transformation. And based on what we’re seeing today, that’s not a given.

Sam Hinton: Yeah, because when you’re spending Claude money faster than some countries’ GDP, you’ve crossed a line from innovation into something much more dangerous.

Alex Shannon: You’re listening to Build By AI, I’m Alex Shannon, and what you just heard? That’s the sound of 2026’s biggest AI reality check happening in real time.

Sam Hinton: And I’m Sam Hinton. Today we’ve got everything from a mystery company’s half-billion-dollar oops moment to actual life-saving AI deployments that are working exactly as intended. Plus Google’s latest demos, military AI training, and why one startup just raised $650 million to completely change direction.

Alex Shannon: It’s honestly the perfect snapshot of where we are right now - AI doing incredible things in some places while creating absolute chaos in others. Let’s dive in.

Sam Hinton: Starting with that mystery company, because honestly, how do you even explain this to your board?

Mystery company accidentally blew $500 million on Claude AI in a single month — failed to put usage limit on licenses for employees

Alex Shannon: Alright, so according to early reports from Tom’s Hardware, an unidentified company - and we’re talking completely anonymous here - spent $500 million on Claude AI usage in a single month. Half a billion dollars. The reason? They failed to put usage limits on employee licenses, so people just… went wild with it.

Sam Hinton: Dude, this is like giving every employee a company credit card with no limit and being shocked when someone books a private jet to lunch. Except worse, because at least with the jet you get frequent flyer miles.

Alex Shannon: But wait, how does this even happen? Like, are we talking about a Fortune 500 company that just didn’t read the fine print on their Anthropic contract?

Sam Hinton: That’s exactly what I think happened. Look, Claude is incredibly powerful, but it’s also incredibly expensive when you’re doing heavy usage. If you’ve got thousands of employees all using it for everything - drafting emails, writing code, analyzing documents, having philosophical debates with an AI - those API calls add up fast. Like, really fast.

Alex Shannon: OK but I have to push back here. How do you not notice $500 million leaving your bank account? This isn’t a rounding error - this is like, multiple quarters of revenue for most companies.

Sam Hinton: Right, and that’s what makes this so terrifying. It suggests this company either has absolutely terrible financial controls, or they’re so big that half a billion dollars can slip through the cracks. Either scenario is pretty alarming when you think about AI governance.

Alex Shannon: And think about the psychology here. If your employees are burning through that much AI usage, what were they using Claude for? Were they just… talking to it all day? Having it write their entire jobs for them?

Sam Hinton: That’s the million-dollar question. Or in this case, the five-hundred-million-dollar question. Because if people were using Claude to literally do their entire workday, that tells us something pretty significant about either how easy their jobs were to automate, or how much they didn’t understand the cost structure.

Alex Shannon: And this ties into something bigger we’re seeing. Box founder Aaron Levie is calling this kind of thing ‘AI psychosis’ - decision-makers using AI to replace jobs without understanding those roles. ClickUp just laid off 22% of their workforce for AI agents.

Sam Hinton: Exactly, and here’s the thing - if you can accidentally spend $500 million on AI, you probably shouldn’t be making decisions about which humans to replace with AI. This is the corporate equivalent of texting while driving a bulldozer.

Alex Shannon: But let’s think about this from a business strategy perspective. What kind of company even has access to unlimited Claude usage? Are we talking about a major tech company, a financial services firm, maybe a consulting giant?

Sam Hinton: My guess? It’s either a massive consulting firm where everyone thought they could use Claude to write client proposals faster, or it’s a tech company where engineers were using it for everything from debugging to documentation. The scary part is, without usage limits, there’s no feedback mechanism to tell people they’re being expensive.

Alex Shannon: And the ripple effects here are huge. If one company can accidentally spend half a billion on AI, how many smaller companies are accidentally spending millions? This might just be the tip of the iceberg.

Sam Hinton: Absolutely. And here’s what’s really concerning - this probably happened because someone at the C-level said ‘give everyone access to the best AI tools’ without understanding what ‘unlimited access to Claude’ actually costs. It’s like ordering ‘the best wine they have’ without checking the price.

Alex Shannon: So what’s the takeaway here for companies looking at AI deployment? Because clearly the ‘move fast and break things’ approach has some serious downsides when things cost half a billion dollars.

Sam Hinton: Start small, set hard limits, and for the love of all that’s holy, monitor your usage. This isn’t just about money - it’s about proving you can handle AI responsibly before you scale it up. Because if you can’t manage Claude costs, you definitely can’t manage Claude replacing your workforce.

Alex Shannon: And maybe, just maybe, understand what your employees are actually doing with the AI tools you give them. Because $500 million buys a lot of human employees who actually understand their own cost structure.

After Nvidia’s $20B not-acqui-hire, AI chip startup Groq reportedly raising $650M

Alex Shannon: Speaking of massive amounts of money, early reports suggest chipmaker Groq is seeking to raise $650 million in internal funding. But here’s the interesting part - they’re pivoting their entire business strategy. Instead of focusing on hardware, they’re shifting to AI inference optimization, which is basically refining how AI models respond to user prompts.

Sam Hinton: This is huge, and I think people are missing why. Groq built these incredible inference chips - like, stupid fast for running AI models - but now they’re saying the real value isn’t in the chips themselves, it’s in optimizing how AI thinks and responds.

Alex Shannon: But wait, isn’t that a pretty dramatic pivot? They’ve spent years building hardware, and now they’re essentially saying ‘actually, software is where the magic happens’?

Sam Hinton: Yeah, but think about it - Nvidia just did that $20 billion not-quite-acquisition we mentioned, and the whole chip space is getting incredibly competitive. Meanwhile, inference optimization is this massive unsolved problem. Every company using AI is struggling with getting consistent, reliable responses from their models.

Alex Shannon: So they’re basically betting that knowing how to make AI models think faster and better is more valuable than making the chips that run them?

Sam Hinton: Exactly. And honestly, they might be right. Hardware becomes commoditized eventually, but if you can consistently make AI models give better responses faster, that’s something every AI company needs. It’s like going from making race car engines to teaching drivers how to win races.

Alex Shannon: But here’s what I’m curious about - if Groq’s hardware is already optimized for fast inference, don’t they have an advantage in understanding how to optimize the software side too? Like, they’ve seen both sides of the equation.

Sam Hinton: That’s exactly why this pivot makes sense. They understand the hardware limitations better than anyone, so they know exactly where the software optimizations can have the biggest impact. They’re not just guessing at what will make models faster - they’ve been measuring it.

Alex Shannon: The timing is interesting too. This comes right as we’re seeing companies struggle with AI deployment - like our mystery company burning through Claude budget. Better inference optimization could actually help with cost control.

Sam Hinton: Right! If Groq can make AI models more efficient and predictable, that directly addresses the chaos we just talked about. Companies need AI that doesn’t just work well, but works predictably and affordably. That’s worth $650 million if they can pull it off.

Alex Shannon: And think about the market timing here. Every company is trying to deploy AI right now, but most of them are struggling with consistency and cost. If Groq can solve both problems simultaneously, they’re not just selling a product - they’re selling peace of mind.

Sam Hinton: Exactly. Plus, inference optimization is something that gets more valuable as AI models get more complex. Like, if GPT-5 or Claude-4 are even more powerful but also more expensive to run, Groq’s optimization could be the difference between profitable AI deployment and burning money.

Alex Shannon: I’m also wondering if this signals something broader about the AI infrastructure space. Maybe we’re moving past the ‘throw more compute at the problem’ phase into the ‘make compute smarter’ phase.

Sam Hinton: That’s a really good point. Raw computational power was the bottleneck for a while, but now that we have these incredibly powerful models, the bottleneck is figuring out how to use them efficiently. Groq might be betting that efficiency is the new frontier.

Alex Shannon: Keep an eye on this because it might signal a broader shift in the AI space - from ‘let’s build bigger, faster chips’ to ‘let’s make AI actually usable for normal businesses.’

Sam Hinton: And honestly, if they can help prevent more half-billion-dollar AI accidents, that alone might justify the investment. Sometimes the best innovation is just making existing technology less dangerous to deploy.

Boston Children’s uses AI to unlock new diagnoses

Alex Shannon: Now, while companies are burning money and pivoting strategies, let’s talk about AI actually working exactly as intended. Boston Children’s Hospital has implemented OpenAI technology and successfully diagnosed over 40 rare disease cases. We’re talking about real kids getting real diagnoses for conditions that might have gone undetected.

Sam Hinton: This is what I love about AI in healthcare - it’s not about replacing doctors, it’s about making doctors superhuman. Rare disease diagnosis is this incredible puzzle where you need to connect symptoms that might seem unrelated, and AI is brilliant at pattern recognition across massive datasets.

Alex Shannon: But how does this actually work in practice? Are doctors just feeding symptoms into ChatGPT and hoping for the best?

Sam Hinton: No, it’s way more sophisticated than that. The AI is probably trained on medical literature, case studies, and diagnostic criteria for thousands of rare diseases. When a doctor inputs a complex case, it can suggest conditions that a human might not immediately think of, especially for diseases that show up once every few years.

Alex Shannon: And here’s what’s fascinating - the contrast with our first story is pretty stark. Here’s a hospital using AI thoughtfully to save lives, while some mystery company is accidentally spending half a billion dollars. What’s the difference?

Sam Hinton: Purpose and process. Boston Children’s clearly defined what they wanted AI to do - help with diagnosis - and they built systems around that goal. They probably have doctors who understand both the technology and the medical context making decisions. It’s the opposite of just throwing AI at everything and hoping it works.

Alex Shannon: But let’s think about this from the hospital’s perspective. Implementing AI in healthcare is incredibly risky from a liability standpoint. If the AI suggests the wrong diagnosis and someone makes a treatment decision based on that, who’s responsible?

Sam Hinton: That’s exactly why this implementation is so smart. The AI isn’t making diagnoses - it’s giving doctors additional perspectives to consider. The human doctor is still making the final call, but now they have a tool that can say ‘hey, have you considered this rare condition that matches these symptoms?’

Alex Shannon: And it’s improving both patient care and operational efficiency according to the reports. So we know AI can be deployed successfully when done right. But what makes a hospital better at this than a tech company?

Sam Hinton: Hospitals are already used to high-stakes decision making with multiple checks and balances. They have protocols for everything, they understand risk management, and they know that failure can literally be life or death. Tech companies often optimize for speed and iteration - hospitals optimize for safety and accuracy.

Alex Shannon: That’s such a good point. And when you’re dealing with rare diseases, accuracy is everything. These are conditions where a misdiagnosis might mean years of unnecessary treatment or missed opportunities for actual treatment.

Sam Hinton: Exactly. And here’s what’s beautiful about this - every rare disease diagnosis potentially helps the AI get better at the next case. These aren’t just wins for individual patients, they’re building a knowledge base that could help kids worldwide.

Alex Shannon: So we’re talking about a virtuous cycle here. Better diagnoses lead to better training data, which leads to better AI performance, which leads to even better diagnoses. That’s the kind of AI deployment that actually makes sense long-term.

Sam Hinton: And think about the cost-benefit analysis here. Instead of spending $500 million accidentally, Boston Children’s is probably spending a fraction of that to literally save lives and improve patient outcomes. That’s ROI that actually matters.

Alex Shannon: This should be the template for AI deployment everywhere - clear objectives, expert oversight, measurable outcomes that actually matter. Not just ‘AI will solve everything’ but ‘AI will help us solve this specific problem better.’

Sam Hinton: And notice how they’re not replacing doctors - they’re making doctors more effective. That’s the sweet spot for AI deployment. You’re not eliminating human expertise; you’re amplifying it with tools that can process information at scale.

AI warfare is here, and CBS News got a look at the U.S. military training to use it on the battlefield

Alex Shannon: Shifting to something that’s both fascinating and concerning - early reports indicate CBS News got access to U.S. military training for AI deployment on the battlefield. We’re not talking about science fiction here; this is apparently happening right now.

Sam Hinton: This was inevitable, right? If AI can help doctors diagnose diseases faster, it was always going to end up helping military personnel make tactical decisions faster. The question isn’t whether this would happen, it’s how responsibly it’s being deployed.

Alex Shannon: But there’s a huge difference between AI suggesting a rare disease diagnosis and AI making decisions in combat situations. The stakes are completely different, and so are the consequences of getting it wrong.

Sam Hinton: Absolutely, and that’s probably why CBS got access - the military wants public oversight and discussion about this. They’re not hiding it. AI in warfare could be anything from better logistics and supply chain management to real-time threat assessment and target identification.

Alex Shannon: I have to say, part of me is worried about the same ‘AI psychosis’ we talked about earlier showing up in military contexts. Decision-makers who don’t understand the technology making choices about life and death systems.

Sam Hinton: That’s a fair concern, but military procurement is usually way more rigorous than corporate AI adoption. They test everything extensively, they have clear chains of responsibility, and they understand that failure isn’t just expensive - it’s deadly. That might actually make them better at AI governance than most companies.

Alex Shannon: But think about the speed of warfare versus the speed of AI development. Military systems are typically built to last decades, but AI models become obsolete in months. How do you reconcile those timelines?

Sam Hinton: That’s a really good point. The military might need to completely rethink how they approach technology procurement. Instead of buying a system that works for 20 years, they might need to buy a platform that can continuously integrate new AI capabilities as they develop.

Alex Shannon: On the other hand, if our mystery company can accidentally spend $500 million on Claude, what happens when AI military systems have similar oversight failures?

Sam Hinton: Right, which is why the training aspect is so important. If they’re investing heavily in teaching people how to use these systems properly, that suggests they understand the risks. The biggest danger would be rushing AI into combat without proper human understanding and oversight.

Alex Shannon: And there’s the ethical dimension too. Even if the AI works perfectly, there are questions about autonomous weapons systems and where you draw the line on human control over life-and-death decisions.

Sam Hinton: Exactly. But notice that they’re emphasizing training - training implies human operators who understand and control the systems. That’s very different from fully autonomous weapons. It sounds more like they’re building AI-assisted decision making rather than AI-replacement decision making.

Alex Shannon: Which actually brings us back to the Boston Children’s model. The AI provides information and suggestions, but humans make the critical decisions. That seems like a much safer approach than fully autonomous systems.

Sam Hinton: Right, and if they can maintain that human-in-the-loop approach, AI could actually make warfare more precise and reduce civilian casualties. Better intelligence, faster threat assessment, more accurate targeting - all of that could save lives on both sides.

Alex Shannon: Keep watching this space because it’s going to force conversations about AI governance that we probably should have been having already. Military applications have a way of clarifying the stakes for everyone else.

Sam Hinton: And honestly, if the military can develop good AI governance frameworks under life-or-death pressure, maybe the rest of us can figure out how to avoid accidentally spending half a billion dollars on chatbots.

9 demos of Gemini Omni and Gemini 3.5 in action

Alex Shannon: Alright, let’s hit some rapid-fire updates. Google showcased nine demos of their new Gemini Omni and Gemini 3.5 models that were announced at Google I/O 2026.

Sam Hinton: Classic Google - announce at I/O, then flood us with demos to prove it actually works. But honestly, nine demos suggests they’re really confident in these models. You don’t put out that much content unless you think you’ve got something special.

Alex Shannon: And the timing is interesting with Groq pivoting to inference optimization. Everyone’s trying to solve the same problem - making AI responses better and more reliable.

Sam Hinton: Yeah, and Gemini Omni sounds like it’s trying to be the everything AI - text, voice, video, the works. If they can nail consistent performance across all those modalities, that’s a pretty big deal for developers.

Alex Shannon: But nine demos also means nine opportunities to spot inconsistencies. I’m curious whether they’re showing cherry-picked examples or if this represents typical performance.

Sam Hinton: That’s always the question with AI demos, right? Though after seeing companies accidentally spend $500 million on Claude, maybe the bar for ‘good enough to deploy’ is higher than we thought.

Alex Shannon: And Google’s positioning this as multimodal AI that can handle everything seamlessly. If that’s true, it could simplify a lot of the complexity we see in AI deployment.

Sam Hinton: Which brings us back to cost control and governance. One model that does everything well might be easier to budget for than multiple specialized models that each have different pricing structures.

Governor Newsom Signs Executive Order to Confront Economic Impacts of AI

Alex Shannon: Governor Newsom signed an executive order addressing AI’s economic impacts. If confirmed, this could be significant policy movement at the state level.

Sam Hinton: California basically setting the tone for AI regulation again. With companies like ClickUp cutting 22% of their workforce for AI agents, someone needs to be thinking about the economic disruption. Newsom’s probably seeing the same ‘AI psychosis’ trends we talked about.

Alex Shannon: And California has both the biggest AI companies and some of the biggest potential job impacts. Makes sense they’d want to get ahead of this.

Sam Hinton: Right, because if the mystery company burning $500 million is in California, the state definitely wants to understand what’s happening in their own backyard before it spreads.

Alex Shannon: I’m curious what specific economic impacts they’re targeting. Are we talking about job displacement, wage effects, or broader economic disruption from AI deployment?

Sam Hinton: Probably all of the above. When companies can accidentally spend half a billion on AI or lay off 22% of their workforce for AI agents, you need policy frameworks that can handle multiple types of economic chaos.

Alex Shannon: And California’s executive orders tend to become models for other states. This could be the beginning of a broader policy response to AI’s economic disruption.

Sam Hinton: Exactly. Someone has to figure out how to manage AI adoption responsibly, and it might as well be the state where most of these AI companies are headquartered. They have the most skin in the game.

AI-led solutions of Erdős problems spark debate over the future of mathematics

Alex Shannon: Here’s something wild - early reports suggest AI has solved some Erdős mathematical problems, and apparently it’s sparking major debate about the future of mathematics as a discipline.

Sam Hinton: Oh man, this is like AI in medicine but for pure mathematics. Erdős problems are legendarily difficult - we’re talking about puzzles that have stumped brilliant mathematicians for decades. If AI can crack these, it’s not just solving problems, it’s potentially changing how mathematical discovery works.

Alex Shannon: But I imagine mathematicians have feelings about this? Like, if AI can solve the hard problems, what’s left for human mathematicians?

Sam Hinton: Exactly the debate they’re having! Though honestly, it might be like AI in medicine - not replacing mathematicians but making them capable of tackling even bigger, more complex problems. The real question is whether AI solutions can teach us new approaches to mathematical thinking.

Alex Shannon: And there’s probably a difference between AI finding solutions and AI helping mathematicians understand why those solutions work. The proof might be more important than the answer.

Sam Hinton: That’s a great point. Mathematics isn’t just about getting the right answer - it’s about understanding the underlying principles. If AI can solve problems but can’t explain the mathematical intuition behind the solution, that’s still a huge gap.

Alex Shannon: Plus, Erdős problems are famously elegant and often connect different areas of mathematics in surprising ways. I wonder if AI solutions maintain that elegance or if they’re just brute-force computational approaches.

Sam Hinton: And that gets to the heart of the debate, right? Mathematics as a field values insight and elegance, not just correctness. If AI solutions lack those qualities, they might solve problems without advancing mathematical understanding.

What happens when companies become too AI-pilled?

Alex Shannon: And finally, we’ve got more context on that ‘AI psychosis’ trend. Reports suggest tech layoffs in 2026 are nearly matching 2025 totals, with companies replacing workers with AI agents they don’t fully understand.

Sam Hinton: This ties everything together perfectly. Aaron Levie from Box nailed it - decision-makers are using AI for job replacement without understanding those jobs. It’s the same mindset that leads to accidentally spending $500 million on Claude. Rush first, think later.

Alex Shannon: And ClickUp cutting 22% of their workforce for AI agents is a perfect example. That’s not optimization, that’s disruption without a plan.

Sam Hinton: Right, and while Boston Children’s Hospital is saving lives with thoughtful AI deployment, other companies are creating chaos with thoughtless AI deployment. Same technology, completely different outcomes based on how humans choose to use it.

Alex Shannon: What’s scary is that 2026 layoffs are nearly matching 2025 totals. That suggests this isn’t a temporary adjustment period - it’s becoming a pattern of how companies think about AI adoption.

Sam Hinton: And the ‘AI-pilled’ terminology is perfect because it captures the obsessive, almost addictive quality of how some companies are approaching this. They’re not using AI strategically; they’re using it compulsively.

Alex Shannon: The contrast with military AI training is interesting too. The military is investing in training people to use AI properly, while companies are just replacing people with AI entirely.

Sam Hinton: Exactly. One approach builds capability and maintains human oversight. The other approach eliminates oversight and hopes the AI can handle whatever complexity the humans were managing. Guess which one leads to half-billion-dollar accidents?

BIGGER PICTURE

Alex Shannon: If you zoom out and look at everything we covered today, there’s a clear pattern emerging. AI is simultaneously our most powerful tool and our most dangerous temptation.

Sam Hinton: Yeah, and the deciding factor isn’t the AI itself - it’s human judgment. Boston Children’s Hospital saves lives because doctors understand both the technology and medicine. Groq pivots successfully because they understand both hardware and software. Meanwhile, mystery companies burn through hundreds of millions because they understand neither their costs nor their employees.

Alex Shannon: And that’s what makes Governor Newsom’s executive order and the military training so important. We need frameworks for responsible AI deployment before the technology gets so powerful that our mistakes become irreversible.

Sam Hinton: But here’s what’s really interesting - the successful AI deployments all have something in common. They’re augmenting human expertise, not replacing it. Boston Children’s enhances doctors’ diagnostic capabilities. Military training enhances soldiers’ decision-making. Even Groq’s pivot is about enhancing AI models’ performance.

Alex Shannon: While the failures are all about replacement without understanding. Replacing human oversight with unlimited AI spending. Replacing workers with AI agents. Replacing mathematical intuition with computational solutions. There’s a pattern there.

Sam Hinton: Exactly. Because if 2026 is teaching us anything, it’s that AI doesn’t fail - humans fail at AI. The question for 2027 is whether we’ll learn from Boston Children’s success story or keep repeating the mystery company’s very expensive mistakes.

Alex Shannon: And the stakes are getting higher. We’re not just talking about business efficiency anymore. We’re talking about healthcare outcomes, military applications, mathematical discovery, and entire economic systems. AI is becoming too important to deploy carelessly.

Sam Hinton: Which brings us back to Aaron Levie’s point about ‘AI psychosis.’ When decision-makers get obsessed with AI as a solution, they stop thinking clearly about what problems they’re actually trying to solve. That’s how you get half-billion-dollar accidents.

Alex Shannon: But it’s also how you get breakthrough medical diagnoses and potentially revolutionary mathematical discoveries. The same technology that can bankrupt a company in a month can also save lives and advance human knowledge. The difference is entirely in how we choose to use it.

Sam Hinton: And that’s why policy interventions like Newsom’s executive order matter. We need guardrails that encourage the Boston Children’s approach while preventing the mystery company approach. The technology is too powerful for pure market forces to handle responsibly.

Alex Shannon: Keep watching for companies that can clearly articulate why they’re using AI and what success looks like. Those are probably the ones that won’t accidentally spend your quarterly revenue on chatbot conversations.

Sam Hinton: And look for the human expertise that’s being enhanced, not eliminated. Because the most successful AI deployments seem to be the ones where humans and AI work together, each doing what they’re best at.

OUTRO

Sam Hinton: Alright, that’s Build By AI for today. If your company is deploying AI, maybe set some usage limits first. Just a thought.

Alex Shannon: And if you found today’s stories helpful, hit subscribe so you don’t miss tomorrow’s episode. We’ll be back with more AI news and hopefully fewer half-billion-dollar oops moments.

Sam Hinton: Though honestly, those make for pretty entertaining stories too. See you tomorrow!