Post
It turns out GenAI code changes are causing serious incidents and outages at Amazon with "high blast radius" https://arstechnica.com/ai/2026/03/after-outages-amazon-to-make-senior-engineers-sign-off-on-ai-assisted-changes/
Junior / middle engineers no longer allowed to push GenAI code to production without senior engineer review
(HT @KimPerales )
EDIT: Better link above than before. Old one is here:
https://www.ft.com/content/7cab4ec7-4712-4137-b602-119a44f771de
@cwebber @KimPerales Something something chickens roosting
@cwebber @KimPerales so the senior engineers become reverse centaurs aka accountabiluty sinks
This fiasco is the plumbing equivalent of hooking the water main up to the sewage line. It's going to take a while to flush out the system.
@cwebber @KimPerales aaaahahahahaha… 🤣🤣🤣🤣🤣
This is a logical consequence of the "all experts are full of shit" zeitgeist/narrative that certain groups keep pushing. Starting to feel really sympathetic towards climate scientists and virologists.
@cwebber @KimPerales At least Management were smart enough to put a Senior Engineer in the loop so they have someone to blame/fire when it happens again.
@cwebber @KimPerales who ever thought this was a good idea shouldnt be in charge.... "no longer allowed to push GenAI code to production without senior engineer review"
Who pushes to prod without review to begin with 🤨
@vico c'est le futur !
@cwebber @KimPerales Literally no-one predicted this. 🙄
@cwebber
i think in general, too many devs dont understand or dont practice the difference between a code review and clicking the „approve“ button in a Pull Request… why scroll down and look at all that code if the button is on the top of the page already?! ;)
with AI and the mass and speed of code creation it‘s tempting to think you have to review it in 2min if it was written in just 5… but often it will be more than 20, or even 2h or 2d depending on the issue at hand
@cwebber whyyyyyy was anyone being allowed to push to prod without review?! GenAI or not that feels like a bad idea. Like omg
@adastra1930 @cwebber this sentiment exactly. I imagine the no QA situation (maybe cuts) could be involved. Seems more common now to just let people do a quick self- review and blame it on them in the short term.
I don't even know what to say except that the greatest myth of all is that rich people are rich bc they deserve it.
@cwebber @KimPerales Same senior engineers they laid off or pushed out a few months ago, right?
@cwebber @KimPerales nobody could have predicted that
Narrators Voice: everybody did, in fact, predict that
@cwebber @KimPerales juniors are juniors, AI won’t change that :>
@cwebber @KimPerales AI usage requirements + mandatory senior reviews of AI code = one hell of a bottleneck.
Oh, and I would hate to be a senior programmer whose job suddenly become reviewing iffy code that nobody can explain, because they didn't write it.
@cwebber loginwall 😐 but it is wild that they let anyone push code to prod without review
Wow! It's not like that wasn't predictable in any way.
What could possibly go wrong? 🙄
@cwebber@social.coop this is what happens when business units throw away decades of established software development best practices in the pursuit of speed.
I don't know who to blame here, management, or the dev and ops people who went along with it.
AWS suffered a 13-hour interruption to a cost calculator used by customers in mid-December after engineers allowed the group’s Kiro AI coding tool to make certain changes, and the AI tool opted to “delete and recreate the environment”, the FT previously reported.
Just wonderful.
@cwebber
Why were they, I do not know.
@KimPerales
@cwebber @KimPerales "without senior engineer review"
okay so this hits on an issue i've been thinking a *lot* about
i won't use LLM code generators. i believe i can find a job where i don't have to use LLM code generators. but i would probably still be forced to *work with people who do*. and if someone on your team vibecodes, you vibecode. the work of a vibecoder is code-review and checking. but everyone checks everyone's code. so they're pushing their work onto me. i'm vibecoding for them.
@cwebber @KimPerales amazon just formalized this. my criticism. "they tweeted it out". if what the junior vibecoder does is review the generated code, but amazon must mandated the senior engineer does the actual review, then why do you have junior engineers at all? the senior engineer in this case now does the junior engineer's entire job as amazon defines it.
Imagine being a senior engineer and half your day is just spent reviewing slop code, we may really be in hell already
@Purple
It's me, I am that senior engineer! I often see the most mishaps occur with junior devs that haven't learned enough to understand the codebase pushing code that they don't understand. And then when asked for justification for why they approached something a certain way, I get mixed responses and reverts.
Juniors also suffer here because they don't learn the code if they're just believing what AI says about the code. genAI is often woefully incorrect about what certain code is doing, and with how big enterprise codebases often are, there's no way it can paint the picture as well as a dev that's been working in that code (without any AI) for years. The juniors either fail to thrive as a result, or take a lot longer to come around to the realization that genAI is actually slowing them down.
@cwebber @KimPerales@toad.social
@cwebber @KimPerales interesting senior devs weren't reviewing them
@Ox1de @KimPerales A bunch of people are saying this, but it's common for mid-level engineers to do code reviews also normally
@cwebber @KimPerales yeah, not were I worked, we developed applications for electrical, safe-shutdown and fire protection engineers for the nuclear industry and were regular audited by the NRC.
I get it's a different situation, or is it? Doesn't amazon have government contracts that deal with safety and civilian lives?
@cwebber @KimPerales None saw this coming 🙄
@cwebber @KimPerales I'll bet $100 the incidents would have happened even with sign off. No one wants to review slop.
@KimPerales@toad.social @deech @cwebber not a matter of “if” it would happen in my mind, but when.
Human review is a limited control gap in the system of pushing changes to a production system and has limited utility in preventing errors.
Most empirical studies I’ve read seem to agree that human code review has a marginal effect on error rates that disappears of the reviewer had read a couple hundred SLOC in the last hour.
CEOs aren’t engineers and most software developers aren’t either, IMO. Liability is very screwy in our industry.
Making developers sign off on changes they can’t even hope to vet is a disaster waiting to happen.
@deech @cwebber @KimPerales I never know what the actual technical name for it is but I swear there's an attention blindness or fatigue that comes with this sort of thing - like FAA employees deal with. You can't be vigilant constantly or actually critical when 95% of it is fine
Here's a different article about the same topic https://www.tomshardware.com/tech-industry/artificial-intelligence/amazon-calls-engineers-to-address-issues-caused-by-use-of-ai-tools-report-claims-company-says-recent-incidents-had-high-blast-radius-and-were-allegedly-related-to-gen-ai-assisted-changes
@cwebber
Who could have predicted this?
@cwebber re. Tom's Hardware article: Why must every story about how genAI sucks end with "While GenAI is definitely useful in some circumstances..." or something similar. Stories about how a Coal plant poisoned a river don't end in "While it's true that coal plants generate useful electricity..." as though that is something that needs reinforcing.
Why am I not surprised that Amazon is just now creating a policy of senior code review?
Just have senior engineers write all the code. Probably easier than reviewing AI code.
If that gets to be too much work, just declare that everyone is now a senior engineer.
@cwebber is there a link thats readable without paying for a subscription?
This is a forward thinking server running the Bonfire social media platform.
LGBTQA+ and BPOC friendly.
