What happens when the human in the loop isn’t a subject-matter expert? Well, that’s what I found out in the next experiment I tried with Claude. There’s a video game I want to write, an epic story about identity, the future of our species, and what it actually means to be human. It’s a project I’ve dabbled with for several years, on and off. I have written the final dialog between the main character and the antagonist, and I’ve drawn sketches of what it’s going to look like. I started on a version in a text-based interactive fiction environment, but what this story actually wants is a fully 3D walkable environment, and for that, I need graphical power.

WebGL is a 3D graphics library for the Web that I’ve always wanted to learn more about. And I have some 3D graphics background from the ancient days, so how hard could it be?

Learning by making a game

Many times in the past, I’ve used writing a video game as a way to learn a new graphics library, or a new programming language. I like games, and building something interactive is more intrinsically motivating to me than solving a bunch of pseudo-accounting puzzles when it comes to learning.

So I made a plan: I’ll use Claude to make a game. Not the game, because I’m not crazy. I’ll start with something relatively-simple, and keep adding stuff to it, to see how Claude handles it. To get the full “Vibe Coding” experience, I’ll have Claude write all of the code, and I’ll interact with it only through prompting it to change things.

Here’s the concept:

You’re a scavenger, exploring the wrecks of spooky abandoned space stations. You need to collect the resources on each level to get to the next.

It’s an infinite maze-running challenge, with spooky sounds, items to collect on each level. There’s a mini-map of where you are, and a best time scoreboard so you can compete with your friends. No enemy AI, no really fancy graphics.

Piece of cake, right? Let me tell you, it’s been painful.

You can play it here. (It takes a while to load – sorry about that)

At first, it all went pretty well. Claude “borrowed” an implementation of a dig-and-tunnel dungeon generator, set up the basic geometry, and provided keyboard navigation and mouselook. This is going great, I thought…

Cracks start to appear

“Claude, create pointed star polyhedra at 10 open spaces in the maze, each a different color”.

Yeah, no – that simply didn’t work. Claude went off, grabbed a definition of a 3D star from…somewhere, and managed to totally mess it up. You could see parts of the geometry, but some faces were transparent when they should have been solid, and vice-versa. Something that I know, but which Claude doesn’t, is that hidden-surface culling typically wants vertices and their normal vectors to be defined in a particular order, following a winding rule. If you just specify vertices willy-nilly, and don’t specify your rule, you’re going to be in trouble.

I tried for a while to explain this, and Claude made very plausible responses indicating that he understood me, but of course, he doesn’t understand anything, so was entirely unable to apply the feedback in a systematic way to get this very simple shape properly rendered. Eventually, I gave up, and just said:

“Claude, use a Dodecahedron for the stars”

And just like that, Claude came up with a definition for the geometry, placed it in the scene, and it all worked. Except for one small thing…

“Claude, I asked for a Dodecahedron. That object is an Icosahedron. Can you use the right shape?”

…

“No, that’s still an icosahedron”

…

“Do you know the difference between an icosahedron and a dodecahedron?”

The stars are still icosahedrons. Whatever, it’s fine. I’m moving on to the next thing. We need some lights in this here scene…

Oh, the pain…

Lighting is, arguably, the hard part in computer graphics. The geometry, the tessellations, the texture mapping, most kinds of shader calculations – that’s all “just math”. It’s vector math, and you need to just know things (see discussion above about winding rules and normals), but it’s all relatively straightforward.

Lighting is the black magic of graphics. It’s really easy to light a scene, if you don’t care about performance or realism. We have known the “right way” to light computer graphics since the 1970s at least. It’s called ray tracing, and it’s still too computationally intensive to use in interactive graphics in a naive way, 50 years later. So we fake it. We use a bunch of shortcuts to make it look like objects are casting shadows, without actually calculating realistic light propagation. We add invisible lights in places where getting the actual effect we want to model would be too much work.

And good grief, is this the kind of thing where just feeling your way around and asking Claude why nothing works is not an effective technique. He can spit out all the proper terminology, and if I ask a question like, “why are all of the walls facing in the +X direction transparent?”, he can actually figure that out. Because “why are my polygons invisible?” is probably the most-common newbie question in all of 3D graphics.

But lighting? It’s all vibes, in some sense. You can’t actually put accurate, physics-based light rendering into a WebGL application without a lot of work, and a lot of expertise. I eventually resorted to making a test arena, where I could place lights and objects, and give Claude extremely specific feedback like “the ceiling lights at (some grid location) render properly on +X and +Z facing walls, unless there is also an adjacent -Z or -X wall, in which case all of the walls get a harsh circular highlight”.

And I eventually got it to work, well enough. But good god, was it frustrating. And in the end, do I feel like I learned much about WebGL? Not really.

Verdict: I didn’t learn anything, and I can’t use any of this

I could probably read through this code to figure out how some of it works, but it’s not particularly well-organized, and since I didn’t design the structure myself, it isn’t immediately obvious how it all goes together. Some parts are re-usable, at best.

Because I wasn’t reading the code and figuring out what was wrong as I went along, I never learned any of the lessons I was hoping to get from this exercise. I still don’t have a good handle on how to design the code libraries I’d want to use for the larger project, and I’d likely be starting over from scratch, so what would I do then? Prompt Claude to create a new project, and go through all of this again?

I’m really disappointed in how little actual useful experience I got out of this. Arguing with Claude about things not working, or not working like I wanted them to, wasted hours of time.

It’d be interesting to try this again, using a game engine which has more data in the training set, maybe Unity or Godot. Maybe that would be “magical” in the way this simply was not. And I could have played to Claude’s strengths, rather than its weaknesses. Maybe just download a bunch of free (or cheap) 3D models and textures, and say “make me a Flappy Bird clone, except with dolphins in a post-global-warming ocean environment”, and see what happens.

[Future Mark here – I later asked Claude to make a console mode rogue-like dungeon exploration game in Go, and that went much better. It took just a couple of hours to make something playable, and I didn’t want to pour soup into my laptop even once. Much like human artists, arbitrary constraints can sometimes improve results]

But it’s not just about the problems you ask the AI to solve, it’s also about how you work with the AI. You can’t just abdicate responsibility for design and structure to the machine, because, again – it doesn’t know anything. So you have to ask for something that makes sense, and you have to know enough about the problem space, if not the specific technology, to know when the AI is building plausible, but useless text.

Would I use Claude to write a Python script to crawl through a bunch of photos, identify all of the identical copies of the same file, and move the duplicates to the trash? Yes, I would. I did, and it worked fine. I can read Python much faster than I can write it, and this sort of thing is one-off and I’ll probably never have to revisit, expand, or change it later. For more-complex needs, especially for day-job work, I’m going to need to keep Claude on a much tighter leash.

So, how do you work successfully with AI?

That’s in the next chapter. See you in a couple of days.