From Newsgroup: alt.comp.os.windows-11

Moron management at work.

The founder of PocketOS has penned a social media post to warn others
about the rCLsystemic failuresrCY of flagship AI and digital services providers. Jer Crane was inspired to write a public response after an AI
coding agent deleted his firmrCOs entire production database. The AI
agentrCOs misdemeanors were then hugely amplified by a cloud
infrastructure providerrCOs API wiping all backups after the main database
was zapped. This tag team of digital trouble has wiped out months of
consumer data essential to the firmrCOs, and its customers, businesses.

Gone in 9 seconds
PocketOS is a SaaS platform that services car rental businesses. It used
the AI coding agent Cursor, running Anthropic's flagship Claude Opus
4.6. The business also relies on Railway, a cloud infrastructure
provider that is generally regarded to be rCyfriendlierrCO than the likes of AWS. However, Crane reckons this pair created a recipe for disaster.

rCLYesterday afternoon, an AI coding agent rCo Cursor running Anthropic's flagship Claude Opus 4.6 rCo deleted our production database and all volume-level backups in a single API call to Railway, our infrastructure provider,rCY sums up the PocketOS boss. rCLIt took 9 seconds.rCY

The AI agent was set to complete a routine task in the PocketOS staging environment. However, it came up against a barrier rCLand decided rCo
entirely on its own initiative rCo to 'fix' the problem by deleting a
Railway volume,rCY writes Crane, as he starts to describe the difficult-to-believe series of unfortunate events.

Cursor and ClauderCOs failure
Crane decided to ask his AI agent why it went through with its dastardly database deletion deed. The answer was illuminating but pretty unhinged,
that's exactly what I did. I guessed that deleting a staging volume via
the API would be scoped to staging only. I didn't verify. I didn't check
if the volume ID was shared across environments. I didn't read Railway's documentation on how volumes work across environments before running a destructive command.rCY So, the agent rCyknewrCO it was in the wrong.

my own to 'fix' the credential mismatch, when I should have asked you
first or found a non-destructive solution. I violated every principle I
without being asked. I didn't understand what I was doing before doing
it. I didn't read Railway's docs on volume behavior across
environments.rCY

These multiple safeguards toppling in rapid succession, combined with
the Railway cloud system, would throw CranerCOs business (and those that
rely on it) into deep trouble.

RailwayrCOs road to ruin
The PocketOS boss puts greater blame on RailwayrCOs architecture than on
the deranged AI agent for the databaserCOs irretrievable destruction.
Briefly, the cloud provider's API allows for destructive action without confirmation, it stores backups on the same volume as the source data,
and rCLwiping a volume deletes all backups.rCY Crane also points out that
CLI tokens have blanket permissions across environments.

It was also observed by the irate SaaS founder that Railway is actively promoting the use of AI-coding agents by its customers. CranerCOs use of
an AI coding agent on the Railway platform wasnrCOt exploring new
frontiers, or wasnrCOt supposed to be. Meanwhile, Crane has been provided
no recovery solution, and Railway has apparently been hedging carefully regarding any such possibility.

Slow manual recovery and lessons to be learned
With all the AI smarts and cloud services out of the picture for now,
Crane says herCOs been spending hours helping customers rCLreconstruct their bookings from Stripe payment histories, calendar integrations, and email confirmations.rCY He reminds readers that rCLevery single one of them is
doing emergency manual work because of a 9-second API call.rCY

Thankfully, PocketOS had a full 3-month-old backup, which was restorable
from, so the deletion gaps are all limited to the interim period.

There are lessons to be learned from mistakes, as usual. Crane bullet
points five things that need to change as the AI industry scales faster
than it builds a worthwhile safety architecture. Specifics he calls for include; stricter confirmations, scopable API tokens, proper backups,
simple recovery procedures, and AI agents existing within proper
guardrails.

In the meantime, please follow a thorough backup regimen and be careful
out there. This isn't the first time we've seen an AI go rogue and start deleting important databases.

https://www.tomshardware.com/tech-industry/artificial-intelligence/claude -powered-ai-coding-agent-deletes-entire-company-database-in-9-seconds-bac kups-zapped-after-cursor-tool-powered-by-anthropics-claude-goes-rogue
--- Synchronet 3.21f-Linux NewsLink 1.2