This post is part of a series.

First, let’s talk about my day job. I work for a cybersecurity company, specifically on the part of our product that runs on our customers’ computers, scanning for suspicious files.

Some of you just got a chill down your spine. A cybersecurity company is using GenAI. And you’re right – that is scary. And it’s not just us – “everybody is doing it”, and our industry is no more proof against the AI marketing juggernaut than any other is.

But I’ve always been a team player, and even though I suspected this was going to be a disaster, I’ve been using the latest Claude Code in my daily work for the last six months. It’s been a really mixed bag, in just about exactly the way I expected. An LLM, as you might know, doesn’t actually know anything. It just predicts which symbols are most-likely to follow other symbols. Or maybe that’s not an important distinction – someone remind me to write up a blog post about the Chinese Room thought experiment sometime soon.

What’s the worst possible way to use AI in development?

At this time, I would like to direct your attention to this article, by Cory Doctorow. He’s got a new book out, called The Reverse Centaur’s Guide to Life After AI, which introduced me to the wonderful idea of the “reverse centaur” as applied to automation theory. It’s a very useful way to think about how management wants to apply AI to programming work. And it also reminded me of this fantastic comic:

Go read the whole article, but here’s the relevant bit:

Start with what a reverse centaur is. In automation theory, a “centaur” is a person who is assisted by a machine. You’re a human head being carried around on a tireless robot body. Driving a car makes you a centaur, and so does using autocomplete.

And obviously, a reverse centaur is a machine head on a human body, a person who is serving as a squishy meat appendage for an uncaring machine.

This is, fundamentally, the great promise of AI-enhanced development. The AI will write all the code, and it’ll fall to the humans to debug it, and to be responsible for any failures, since the AI can’t be held responsible.

That just sucks.

Management is taking away the creative, fun, mentally-engaging part of the job, and doubling-up on the most-difficult, painstaking part of the process. If you spend any time around the office of your local AI fanatic, you’ll hear things like this:

“It’s okay to have the AI write contracts. They’re mostly boilerplate, and we’ll have a human lawyer review them before sending them out, anyway.”

or

“The AI will handle driving the car, but we’ll require a human to be in the driver’s seat, paying attention, and will hand over control to them on a moment’s notice when the AI isn’t sure what to do”.

or

“AI won’t make hiring decisions. It’ll just apply a set of heuristics to candidates, and the human HR professionals will be responsible for checking that it isn’t doing racism-as-a-service“

This weirdly holds true across every industry where the suits are excited to bring AI into their industry. Because AI is bad at detail-oriented, legally fraught parts of the work, humans have to do that. And because you can’t hold AI responsible for any mistakes it makes, that also falls on the humans who accept the AI’s output.

Claude isn’t good at doing novel things

Because Claude is trained by looking at examples, it does very poorly at several things that are really important to doing my job, specifically. In that huge corpus of training data, there is very little Swift code, and it’s written for every possible version of Swift, a language that maybe only really earns its name on the basis of how quickly the language definition changes.

Almost all of the Swift code on the Internet is written for iOS, not MacOS. And after that, the amount of low-level system code for MacOS (that you can find) is a tiny subset of that. Unsurprisingly, Claude is clumsy with this shit, and I keep having to correct–or more often–totally rewrite everything it produces.

A recent example was that I had some code, which I inherited from someone who didn’t really know what they were doing, which combined some very low-level synchronization primitives with the higher-level GrandCentralDispatch library. The details aren’t really important here, except that this code worked fine most of the time, but would get stuck or crash, under poorly-understood circumstances.

I told Claude to rewrite it, using only the most-recent concurrency support in Swift. And sure enough, after being told to “fix that compiler error”, “and that one over there”, a few times – Claude came up with something that built successfully. Did it work? It did not.

I spent several days trying to fix it, and eventually ended up writing a bunch of it myself, anyway. And then I found a serious problem in what was left of what Claude wrote, and rewrote that too.

What was the point of all my mental effort then?

It felt like I was making great progress, right up until it turned out that I actually wasn’t. Because, here’s the hell of it: much like other generated text, the code that Claude produces looks reasonable, at first glance.

It is a machine designed to make plausible-sounding text. It can write plausible-sounding programs, too. And if you’ve got an expert watching it really carefully, they can hopefully figure out whether or not it actually works properly before the code goes into the system.

If I’m being totally honest, I didn’t even find one of the worst problems in Claude’s code by inspection. I had to write a test for it that really stressed its safety before it failed. So, exactly like the code we started from, it appeared to work properly most of the time, but then failed when put under heavy load.

Just like the studies predicted

In the end, for this one smallish task, I was somewhat slower at coming to a working solution than I would have been without using Claude. If I’d gotten the Claude code to work, I’d be a lot less-confident in the quality of the output.

Am I ten times more-productive with Claude? Only if you count lines of code written per hour. Claude is faster at typing than I am. But we have known, for at least 40 years, that that is a terrible way to measure productivity for a programmer. It is easy to measure, though, which explains why management teams focus on that.

“But surely this isn’t universal?” you might be thinking – We’ve all read and heard so much online about how people with no coding experience are writing apps from scratch in no time. There must be something to that, right? It can’t all be lies, can it?

What Claude is “good at”

Well, I’ve got some good news and bad news for you. Those stories aren’t wrong, but they’re glossing over some pretty important caveats. See the next post in this series for details.