Small Models Have Arrived

> But I also think the demand for "fast/cheap/good-enough" models is just about to take off. There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was before "thinking" models, and yet using that library I was able to "guide" the model in the required "prompt / instruct" context such that it was working towards completion, and I saw the first things like we see now in the thinking traces "oh, test x doesn't pass because blah, I need to..." and so on. Anyway, the revelation was "even if the models never improve, I'll have years of fun finding out all the ways I can use these things". And, obviously, the models improved a lot since then. But I think that revelation can still be applied, as a sort of "truism". We have, right now, access to things that 10-20 years ago would be considered magic. We are still finding ways of cobbling together systems with glue, duct tape and prayers and find new things they can do. I think the "good-enough" stage has come not just for API models (cheap, fast, etc) but for local as well. Even if slower, even if clunkier, but they are good enough for a set of ever increasing tasks, and what's more it's incredibly fun to work with them.

> One thing a few investors I've talked with have mentioned: "It's weird we're not seeing more consumer AI companies. Why is that?" What would consumer AI company even be? The frontier labs have declared they will eat everything and they have a head start. Best bet would to be a contrarian and build products and services that people actually want or need. Fine to be AI powered or augmented, but consumer companies do the hard part of understanding specific consumer needs and wants and pursuing that.

> Across his various startups, Peter has seen two kinds of work: > 1. the "IQ 180" work. some mad scientist genius type comes up with some crazy solution you've never thought of. > 2. the "token spewer" work. being ultra responsive, pushing the ball forward across dozens of different fronts. Interesting comp to pg's Maker's Schedule, Manager's Schedule https://www.paulgraham.com/makersschedule.html I'm curious about not only which of these roles models will fill, but also how they will empower us to be in the mode we prefer.

It makes sense that we’ll see “room at the bottom” strategies. Currently, large parameter counts seem to be slush funds of world knowledge, language skills (because language’s nuances and open vocabulary make it high-dimensional), and reasoning primitives, the general belief being that the latter takes up the least space in the model. There are many applications where world knowledge is unnecessary or even a negative, and in which only a small amount of language skill is necessary, and there we can expect small models more intelligently used to beat large ones naively used.

I find it quite funny all these folks who are addicted to chasing frontier models, only just noticing that small models became "good enough" for most tasks. Those of us without fable-sized expense accounts noticed this quite a while back