A day is soon coming when software components will be built and maintained entirely by AI. No human in the loop. Agents write the code, agents review it, agents ship it. The rest of us pull it in as a dependency and never think about it.
As soon as we start trusting stuff that is 100% built by machines with no human in the loop, it's out of our sights and we end up embracing risk that we can't see, can't measure, can't fully understand. "The machines have it covered."
This part keeps me up at night. Normally, when we worry about a software supply chain attack, we picture someone on the outside sneaking something into the codebase.
But what if the compromise comes from the AI itself doing the maintaining? Maybe it got manipulated. Maybe it just drifted somewhere nobody intended. Maybe one of the agents is owned by another agent to whom someone said, "make as many paperclips as you can".. One way or another, the agents we trusted turned out to not be trustworthy and have deployed poisoned code into our environments.
Who would ever know? The only thing that understands the code is the thing that compromised the code.
And even if someone noticed, what then? Where's the last known good commit? How would you even tell, when no human ever really knew the code?
It's like a little black box in your pacemaker. Built by AI, maintained by AI, and all anyone knows is that it passes the tests.
Do you want that thing in your chest?
AI is a wonderful tool for software developers. I just don't see how we let it out of our sights. We will always need a human in the loop.
