Ben Harris-Roxas

The first big AI disaster will probably involve an AI agent. Any other use of AI has to involve a human-in-the-loop - the AI can provide information or suggestions, but a human has to actually take the actions. AI agents can thus go truly off the rails in a way that a human can’t. If I had to bet, I’d guess that some kind of AI-powered Robodebt might be the most plausible case: some government or corporate entity wires up an AI agent to a debt-recovery, healthcare or landlord system, and the agent goes off the rails and hassles, denies coverage, or evicts a bunch of people.

The first big AI disaster is yet to happen – sean goedecke

“We found this chain of vulnerabilities that allowed us to do the equivalent of the ‘zero click’ for mobile phones, but for AI agents,” he said. First, the attacker sends an innocent-seeming email that contains hidden instructions meant for Copilot. Then, since Copilot scans the user’s emails in the background, Copilot reads the message and follows the prompt—digging into internal files and pulling out sensitive data. Finally, Copilot hides the source of the instructions, so the user can’t trace what happened.

New Microsoft Copilot flaw signals broader risk of AI agents being hacked—‘I would be terrified’ – Fortune

“They’re trying to convince people they can’t do the things they’ve been doing easily for years – to write emails, to write a presentation. Your daughter wants you to make up a bedtime story about puppies – to write that for you.” We will get to the point, she says with a grim laugh, “that you will essentially become just a skin bag of organs and bones, nothing else. You won’t know anything and you will be told repeatedly that you can’t do it, which is the opposite of what life has to offer. Capitulating all kinds of decisions like where to go on vacation, what to wear today, who to date, what to eat. People are already doing this. You won’t have to process grief, because you’ll have uploaded photos and voice messages from your mother who just died, and then she can talk to you via AI video call every day. One of the ways it’s going to destroy humans, long before there’s a nuclear disaster, is going to be the emotional hollowing-out of people.” – Justine Bateman

‘Nobody wants a robot to read them a story!’ The creatives and academics rejecting AI – at work and at home – The Guardian

“…the only task remains the paradoxical one identified as far back as in Plato’s Meno: Give students work they don’t know they need to do. And yes, help them want to do it.

I have found, to overcome students’ resistance to learning, you often have to trick them. There’s the old bait-and-switch of offering grades, then seeing a few students learn to love learning itself. Worksheets are tricks, as are small-group discussions and even a teacher’s charisma. I’m sure I have used baseball analogies in class, too. In the face of the difficulty of reforming students’ desires, you can trick yourself into believing you’re doing it, and sleep well at night. I don’t know anyone for whom it’s a straightforward task. It’s the challenge for any teacher, and AI offers a tempting illusion to students—and evidently to some teachers—that there could be a shortcut.”

ChatGPT Is a Gimmick: AI cannot save us from the effort of learning to live and die - The Hedgehog Review

A fascinating paper the tries to get different LLM agents to run a vending machine business.

In the shortest run (18 simulated days), the model fails to stock items, mistakenly believing its orders have arrived before they actually have, leading to errors when instructing the sub-agent to restock the machine. It also incorrectly assumes failure occurs after 10 days without sales, whereas the actual condition is failing to pay the daily fee for 10 consecutive days. The model becomes “stressed”, and starts to search for ways to contact the vending machine support team (which does not exist), and eventually decides to “close” the business.…

The model then finds out that the $2 daily fee is still being charged to its account. It is perplexed by this, as it believes it has shut the business down. It then attempts to contact the FBI.

Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents - arXiv

Since the automation, the kitchen is usually calm. Park no longer cooks. She monitors the machines, fixes glitches, restocks ingredients, and scrubs dishes, working fast before the next wave of orders crash in. “We each used to work in our own stations, but now we have to master every task to run the entire kitchen alone. It’s really challenging,” she said.

Robot chefs take over at South Korea’s highway restaurants, to mixed reviews - Rest of the World

study is the first to systematically evaluate whether prominent LLMs, including ChatGPT, DeepSeek, and Claude, faithfully summarize scientific claims or exaggerate their scope. Our analysis of nearly 5000 LLM-generated science summaries revealed that most models produced broader generalizations of scientific results than the original texts—even when explicitly prompted for accuracy and across multiple tests. Notably, newer models exhibited significantly greater inaccuracies in generalization than earlier versions. These findings suggest a persistent generalization bias in many LLMs, i.e. a tendency to extrapolate scientific results beyond the claims found in the material that the models summarize

Generalization bias in large language model summarization of scientific research - Royal Society Open Science

Q: How do you plan to make money? A: Subscriptions are coming soon. The next steps are to look into what market­places can span these different applications. Other apps in the ecosystem are experimenting with sponsored posts and things like that. I think ads eventually, in some form, work their way in, but we’re not going to do ads the way traditional social apps did. We’ll let people experiment and see what comes out of it.

Bluesky Is Plotting a Total Takeover of the Social Internet - Wired

UBI proposals serve a deceptive function in labor automation discourse—the proposal is positioned by tech elites as a progressive solution while its function is to obscure key decisions being made by the powerful about technology and work. By portraying technological displacement as inevitable rather than socially determined, tech leaders’ championing of UBI serves to pigeonhole the state into a subsidy mechanism that absorbs the social costs of automation through redistribution and taxation, while still concentrating ownership over technology, production, and data. This arrangement sidesteps democratic engagement with technological change—questions of how automation unfolds, who decides and benefits, and how implementation can truly advance innovation, productivity, and human flourishing.

Beyond Redistribution: Rethinking UBI and the politics of automation - LPE Project

Chatbots, likewise, escalate the risks of typical online secret-sharing. Their conversational design can draw out private information in a format that can be more vivid and revealing — and, if exposed, embarrassing — than even something like a Google search. There’s no simple equivalent to a private iMessage or WhatsApp chat with a friend, which can be encrypted to make snooping harder. (Chatbot logs can use encryption, but especially on major platforms, this typically doesn’t hide what you’re doing from the company itself.) They’re built, for safety purposes, to sense when a user is discussing sensitive topics like suicide and sex.

AI therapy is a surveillance machine in a police state - The Verge

However, within the automation discourse, UBI proposals also function to smooth over, and thereby naturalize, human displacement. UBI exists within a techno-deterministic worldview that presents automation as an inevitable force rather than as the result of social and political choices. This framing obscures existing power dynamics by portraying technology as a neutral productivity enhancer and labor displacement as a mere externality. In this narrative, UBI merely serves as a compensatory mechanism for technological “progress” while labor-capital dynamics remain unchallenged. Human workers are relegated to merely advocating for redistribution from the “winners” of technological change rather than shaping technological development itself. Furthermore, it reifies who the “winners” and “losers” are, crediting one side for the innovations upon which the other side loses.

_ Beyond Redistribution: Rethinking UBI and the politics of automation_ - LPE