Working With AI: A concrete example

> I suspect that this is a fundamental limitation of LLMs I suspect there's also a strong sociological bias at play: LLMs are being made by people who are familiar with coding but aren't software engineers. So they design their RL policies around the idea that the LLM must learn how to code, not that they must learn to design a maintenable piece of software.

I feel as though that world model strongly correlates with memory - the experience of having jumped to a conclusion early and full-steaming ahead, only to be bitten by constraints and problems later down the track. Part of that is critical thinking and projecting forward / simulating potential issues, and part of that is that memory which in humans we probably would see as "wisdom". I don't know if that's a fundamental limitation of LLMs, or, rather, that this can be solved moving forward with better memory systems, harnesses, and context windows.

In my experience harness can do wonders to improve this. Instead of asking it to generically to analyze and do X, you can use brainstorming skills like those from superpowers [1]. This makes it approach the problem better and keeps you in the loop. Another step is then to have it review its plans by another LLM acting doing adversarial review. I have a claude skill [2] that calls codex to do it, and they chat among each other. It's a tremendous boost in design quality. [1] https://github.com/obra/Superpowers [2] https://gist.github.com/enricopolanski/6c5038a8e20cc4098cd99...

Have to disagree with this as it's excellent at helping you wide and broad before converging. I suggest trying OpenSpec and use /ospx:explore to state your problem and go from there.

One partial mitigation is to ask it to use plan mode -- and then very carefully review the plan before allowing it to execute.

Exactly, LLM is good at "code inpainting" : define clear structures and goals, and it will fill the boilerplate. But it doesn't work for reasoning and abstraction, so it fails to synthesise and propose novel views. But that's integral to the way it's designed and has been trained, to do a kind of "averaging" which limits it's capacity to explore novel designs

It's just because not enough people had this very specific problem before. This article will be part of the next model training set, and probably it will be able to solve it despite not understanding anything about world or not studying or thinking.

I very much love your work Carson, it has always been and remain a fresh breath of air. The example is mundane but to the point; and I very much enjoyed this article. It's a concrete example which is rare to read when it comes to using LLMs. To the risk of being told that we "hold it wrong", it resonates with my experience of using LLMs.

Always exciting to see a former professor on the front page and always an enjoyable read Mr. Gross!

This is kind of what coding with LLMs feels like. Gradually increase guard rails "outside of it's context (automated)" to get the results you want out of it. Static typing, quick compilation, not having nulls, and lints are a great start (I would also argue for managed side effects and functional, but to each their own). It gets pretty far to the solution on it's own and quickly, but then you spend time adjacent to the problem, building out it's cage while iterating through the remainder of the solution.

As humans we have a concept of viscosity. That resistance, like being in quicksand or a swamp, is how you “easily” identify a code smell, something that needs to be refactored, etc. Part of it is human laziness, part of it some concept of elegance, an itch of being not quite tidy as it can be, etc. LLM, being a tiresome little helper, will gladly output hundreds of lines, hacks, and what have you. I don’t think any amount of tests, prompts, harnesses and other “my shaman is a better shaman” will help it to acquire this trait. Some other AI architecture someday maybe — just not today. And that’s why it is good at what it is and really bad at stuff like code “design” (unless it is a well-known solution being baked in the training set)

Opus 4.whatever (it was last week) via a command line interface in the IntelliJ Claude plugin. The series of prompts weren't particularly interesting or innovative on my part: a paste in of the user report then a few back and forths on fixing it, me reviewing the changes and coming up with the final answer.

It's more like asking what editor and keyboard layout they use. Highly relevant to the user but you should simply assume someone describing work is using a setup for it they find productive. If you decide to dismiss their output it wouldn't be over these details.

:) i let a younger person on the core team create the new website for something different it is using astro, we are scaling down the use of tailwind (I wanted to give it a try, but didn't really click with it.) I don't mind someone doing something kind of fun with the website and trying something new out, I know some people don't like it but some people do. All good.

yeuch … should’ve used https://harcstack.org, like the new https://raku.foundation site

> I saw what I forgot immediately; but soon after, with engagement, I saw how quickly I was able to remember. We actually have pretty good models for how long it takes to forget things. It's the same basic math that powers Anki. To oversimplify, if you force yourself to remember something right before you would have otherwise forgetten it, you will remember it roughly 2.5 times as long before forgetting it again. (This changes at both the shortest time intervals and the longer ones, so treat it as a rough rule of thumb, not an exact formula.) But this provides a handy bound! If you've been doing something professionally for 20 years, you should expect to remember it for another 50. At which point you're likely well into old-age, and memory performance may decrease for other reasons. Where AI kills you is actually at the other end: initial learning. You are much less likely to need to recall something after 1 day, 2.5 days, 6.25 days, etc. And thanks to the lack of the "testing effect", memory formation will be much weaker. In other words, I would naively expect AI to make long-used skills a bit rusty, but to drastically impede formation of new skills and knowledge.

do you propose its maybe closer to the idea that you can regain strength faster after having lost it (in the context of bodybuilding and extended time off)? Gaining something from scratch requires much effort and experimentation, regaining it less so?

In all my side projects, instead of thinking about architecture or design decisions, I just ask it what I want the end effect to be. "I want this button to do a thing". You're saying this is good for my brain?

Yeah, no. The AI was unable to come up with a good solution whereas the human was. Point human.