Untitled

One thing that is exciting about Jev-style decision models is that it makes it cheap for things where they can do zero-shot of things where it's not worth your time building a curated dataset. Bonus: after using them for a while, you get the beginnings of a training set if you change your mind. You can do that with LLMs too - it's just a cost and latency question. E.g. I use an LLM to label my e-mail. I do it because it's low risk enough that I don't care about it being incredibly precise - it does not automatically act on anything, it just labels - and it means I don't have to curate a dataset of what e.g. should get the "receipts" label. An LLM zero-shots it well enough and cheaply enough to remove my incentive to train a model for my individual e-mail. If I wanted to scale it up, Jev or similar would probably be better. A dedicated classifier would almost certainly provide even better precision if I invested the time to curate a dataset. But the threshold where that is worth it for me given the precision I get from a cheap LLM is pretty high, and it's also less flexible. E.g. one of the nice things about using a generic model for this is that whenever I add a new tag, it starts using it immediately and odds are good that at most I need to add a line to a file with one-line descriptions of what the folder name means to get decent results. So there's a space for both LLMs/Jev-likes and custom classifiers. Use Jev-like models (or LLMs) when the effort to curate a dataset to train a dedicated classifier is too high, and switch to a dedicated classifier when/if you need to scale something to a point where the economics justifies the effort or when you already have the dataset anyway.

Jev's launch was the best AI era marketing campaign I've seen so far. It was executed and timed so perfectly that Jev became synonymous with decision models overnight. I think it reveals how many people were only familiar with LLMs as the generic AI tool for everything. It only took one ambitious marketing campaign for someone to come along and present a simpler technology as a new breakthrough to audiences that hadn't considered anything else. The largest LLMs are really amazing for how much they can accomplish on unseen tasks. That is undeniably remarkable. The mistake that gets made everywhere is thinking that they're the best tool for every task, because LLMs are the only AI tool in most developer's AI toolbelt right now.

Agreed. I did do image processing on skin and have to say it is just very very difficult to get 100% correct. In my opinion, even if some systems do outperform some doctors, it can only work as diagnostic support. Anomalies are very hard to detect on certain skin tones and creating data sets from image material is quite a lot of work, even if you have access to large clinics specialised on dermatology with diagnosed and undiagnosed material. Not seeing LLM to perform very well here and hallucination would be a major issue (not at all a solved problem, even frontier models do hallucinate/match the most probable path. Perhaps a problem that cannot be solved without arbitrary rules.) And I guess some SVM still outperform CNN, just that CNN provide a pretty good learning environment that can improve upon ingesting more data. Although there is a risk associated with that as well. Still, if we had well documented data material, many algorithms and AIs would perform well. The work is getting the data in a dependable state.

There are a lot of people working in software that have limited knowledge of information theory or machine learning. Unfortunately LLMs are magic to them. When you lack a first principles understanding of how something really works, you will bolt on hacks and duct tape on the wrong approach.

I think there are a lot of practical applications where something that works few-shot and doesn't require training is quite appealing if the cost of inference is low. That's the sweet spot for Jev and similar models. If you have time and an eval set to train it, clearly a "real" special-purpose classifier should have much better performance (and for most applications cheaper cost/faster inference), but Jev and friends are way better than using a full-fat transformer LLM for a classification task. As an example, I've been using winnow running locally as a decision model in openchamber to select between llms for doing various tasks. I don't have an eval set of different coding and other tasks that I could use to train a proper classifier. This I just download off huggingface and it works out of the box and has cut my token spend to a fraction of what it was. So it's good enough to be beneficial.