Code as Art I covered this before in Art or tool? If you think of software as artistic output then generated software isn't real because it doesn't have the creative ineffability that's a sign of true art. There's no argument against this one except that the people who want software typically aren't paying for artworks,...
I see agents as more like a cnc and its effect on manufacturing. You can still make things by hand but for the most part you can achieve the highest quality and scale with automation. Obviously as we become cnc programmers and operators we lose the hand-filing skills we once used, just as our forefathers forgot how to use flint-knapped hand axes.
I think a big part of the angst is that we’ve had to look around and engage with what the things we’re building are actually for, instead of just resting in our craft. There’s lots of good work to be done for things that are good. The question is how do we (or our agent wielding successors) only work on good stuff and not bad stuff.
Not passing comment on the analogy (unless…?) but the categories of machining work it makes sense to be CNC-only are not that broad. It’s just that there’s a lot of times you’d want 50-10,000 identical widgets. There’s always going to be a reason to pick up a file – and often that’s the best way to get the highest quality work. :)
I’d say the closer analogy to CNC is the compiler. The input to CNC is still a precise transformation of the intent of the designer. It does the work faster but it’s the same work. On the other hand, sometimes a human needs to precisely define the lower level, doing some manual filing because the cutting tool won’t fit in the space, or writing some assembly because the compiler doesn’t know how to do indirect threaded code.
As a craftsperson or client you may look at the result and miss the imperfections (the “hand”, it’s called) of the old way: the curvature is implausibly perfect, or the assembly is too clever to have been written by a human. But you can always see the human intent everywhere.
LLMs, on the other hand, take imprecise descriptions and turn them into precise output by making stuff up instead of waiting for the human to fill the gap. It’s the “imprecise description” part that’s really new. I don’t know of a very good analogy for that in any pre-existing tool.
Somewhere there is an optimizing compiler, where you give it intended semantics and it “makes stuff up” for the codegen that satisfies those semantics, hopefully in a way more optimal than the original description translated verbatim. Then somewhere far else on that spectrum is an LLM which does the same thing, but with general purpose tasks, not just codegen for a specified semantic operation set (can do it deterministically as well).
There’s always been “imprecise translators”. How people use them effectively is by building up and intuition or mental model about how that translator processes their input to help generate the desired output, even if they can’t trace by hand what exact decisions it’s making.
What would be the new thing in this case (genuine question)? Would it be the “codegen targets” of the translator, it’s “acceptable frontend IRs”, some other capability entirely?
I agree it’s a spectrum, but at some point just saying “it’s a spectrum” erases the interesting distinctions. The boundary of impreciseness that can’t be crossed by the “compiler” used to be limited to a fairly narrow range. Think of the possible abstracts of a paper in a PLT journal.
But with an LLM you can literally say nothing but “make a website for scheduling a kids’ soccer league” and it will do it. That is, even if I had a mental model, I didn’t tell the “compiler” what it was. Not only that, while it’s working the “compiler” can ask me what my mental model is, rather than just telling me the model I gave it is incomplete or inconsistent and stopping.
And where it really gets into a new category for me is that the output of the compiler isn’t even defined. You could just as well say “make a presentation for a soccer league” or “get the accounting in order for my soccer league” or “automatically answer questions about our soccer league”. The analogy just seems to completely fall apart at that point.
“Mental model” is an expectation of how the tool works / will translate or react to some given input. It’s not something you tell the tool - it’s how you reason about what to tell it. We do this for human personalities too.
In this case, the mental model I have for SOTA models when it comes to “make a website” is that it’ll write some ugly js & HTML but it can get it done. But the mental model for a older smaller model is that ill need to give it a lot of guidance to achieve that.
For the case of “LLM can ask clarifying questions”, this is a harness diff. When you type statically detected UB or compile errors in the compiler example, it technically could interactively ask you want you mean for each instance. The rust compiler does half of this as well. It’s just usually inefficient when “IDE is a chat interface” isn’t true.
The output of LLMs is defined as much as optimizing compilers is: you give some code and it has infinite ways to achieve some set of behavior. Maybe I’m misunderstanding what you mean by defined?
Sorry, I misread your earlier comment. In my reply when I said “mental model” I meant your mental model of the thing you’re trying to build, not of the tool.
The output of LLMs is defined as much as optimizing compilers is
It definitely is not. Compilers perform a very well defined transformation from input to behavior (not 100% defined, but as close as practical politics can get it). A compiler that “optimizes” by violating that definition is a broken compiler. (The 1% that’s undefined causes a lot of discussion, which takes place while everyone relies absolutely on the other 99% being correct.)
Whereas in most cases you can’t even tell if the output of an LLM did achieve desired behavior, because the input isn’t precise enough to determine the desired behavior in the first place. LLMs not backstopped by deterministic tools (compilers, test harnesses, proof engines, even just plain reliable arithmetic) in a loop have a hard time even making artifacts that are functional.
Humans have real issues writing large chunks of C++ without UBs, though. Looking at what each GCC bump uncovers across a large distro, I do not really believe compilers being used as well-defined transformations within their well-defined zone is a reality. And all fast-moving compilers have a long list of actually-correct code they miscompile, it is just about exotic enough combinations that people do not prioritise cleaning this all up.
It’s the same tolerance to for-profit companies releasing paid-for network-touching software full of UBs that now enables careless use of LLM-generated code is well. I sometimes wonder if the correct approach would be to pretend to believe, and to say that now that LLMs can fix all the bugs any defects are intentionally left in thus normal consumer product liability rules can finally be applied to proprietary software by for-profit companies.
Speaking of notionally-functional, I am kind of surprised of what latest-gen local-ish LLMs (aggressively quantised 120B or less aggressively quantised smaller ones) sometimes can one-shot. And if you do not give me a compiler/interpreter as a deterministic backstop, neither am I any good at one-shotting longer pieces of code than that.
LLMs not backstopped by deterministic tools […] in a loop have a hard time even making artifacts that are functional.
This isnt the case for most frontier models today; They can do the job of an optimizing compiler (generate working & efficient) asm from just source without calling out to verification tools. Even with pseudocode unseen before by guestimating semantics.
I agree though that it’s not the case for smaller models. Mixing both under the “LLM” umbrella, while technically correct, loses a lot of nuance for claims.
the input isn’t precise enough to determine the desired behavior in the first place.
The evaluator/prompter can. It’s how miscompilations and misoptimizations are found in existing compilers.
Compilers perform a very well defined transformation from input to behavior.
The behavior is defined. The transformation is not.
If you have a set of bitwise and ternary expressions with bit operations, theres hundreds of ways to get that behavior; Could it vpternlog with a vblendmp at the end for multiple instances? Could it cmp+branch for each one? Could it eager-eval then cmov? These decisions and their various heuristics change across compiler versions.
Only a few years ago, you had to use inline asm to get clang to emit lock bts on x86. Now, it understands the semantics of atomic_fetch_or_*, the args going into it, and how its result is operated on, such that you can write the semantically equiv without hacks to get lock bts codegen. But if it can’t figure some aspects out, it still falls back to a lock cmpxchg loop. The transformations (or “mental model”) is something intuitively built up as you use the specific tool, not defined somewhere. This applies to LLMs as well.
I see agents as more like a cnc and its effect on manufacturing. You can still make things by hand but for the most part you can achieve the highest quality and scale with automation. Obviously as we become cnc programmers and operators we lose the hand-filing skills we once used, just as our forefathers forgot how to use flint-knapped hand axes.
I think a big part of the angst is that we’ve had to look around and engage with what the things we’re building are actually for, instead of just resting in our craft. There’s lots of good work to be done for things that are good. The question is how do we (or our agent wielding successors) only work on good stuff and not bad stuff.
score 2 · canonical
Not passing comment on the analogy (unless…?) but the categories of machining work it makes sense to be CNC-only are not that broad. It’s just that there’s a lot of times you’d want 50-10,000 identical widgets. There’s always going to be a reason to pick up a file – and often that’s the best way to get the highest quality work. :)
score 1 · canonical
I’d say the closer analogy to CNC is the compiler. The input to CNC is still a precise transformation of the intent of the designer. It does the work faster but it’s the same work. On the other hand, sometimes a human needs to precisely define the lower level, doing some manual filing because the cutting tool won’t fit in the space, or writing some assembly because the compiler doesn’t know how to do indirect threaded code.
As a craftsperson or client you may look at the result and miss the imperfections (the “hand”, it’s called) of the old way: the curvature is implausibly perfect, or the assembly is too clever to have been written by a human. But you can always see the human intent everywhere.
LLMs, on the other hand, take imprecise descriptions and turn them into precise output by making stuff up instead of waiting for the human to fill the gap. It’s the “imprecise description” part that’s really new. I don’t know of a very good analogy for that in any pre-existing tool.
score 1 · canonical
I see the “imprecise description” as a spectrum;
Somewhere there is an optimizing compiler, where you give it intended semantics and it “makes stuff up” for the codegen that satisfies those semantics, hopefully in a way more optimal than the original description translated verbatim. Then somewhere far else on that spectrum is an LLM which does the same thing, but with general purpose tasks, not just codegen for a specified semantic operation set (can do it deterministically as well).
There’s always been “imprecise translators”. How people use them effectively is by building up and intuition or mental model about how that translator processes their input to help generate the desired output, even if they can’t trace by hand what exact decisions it’s making.
What would be the new thing in this case (genuine question)? Would it be the “codegen targets” of the translator, it’s “acceptable frontend IRs”, some other capability entirely?
score 1 · canonical
I agree it’s a spectrum, but at some point just saying “it’s a spectrum” erases the interesting distinctions. The boundary of impreciseness that can’t be crossed by the “compiler” used to be limited to a fairly narrow range. Think of the possible abstracts of a paper in a PLT journal.
But with an LLM you can literally say nothing but “make a website for scheduling a kids’ soccer league” and it will do it. That is, even if I had a mental model, I didn’t tell the “compiler” what it was. Not only that, while it’s working the “compiler” can ask me what my mental model is, rather than just telling me the model I gave it is incomplete or inconsistent and stopping.
And where it really gets into a new category for me is that the output of the compiler isn’t even defined. You could just as well say “make a presentation for a soccer league” or “get the accounting in order for my soccer league” or “automatically answer questions about our soccer league”. The analogy just seems to completely fall apart at that point.
score 1 · canonical
“Mental model” is an expectation of how the tool works / will translate or react to some given input. It’s not something you tell the tool - it’s how you reason about what to tell it. We do this for human personalities too.
In this case, the mental model I have for SOTA models when it comes to “make a website” is that it’ll write some ugly js & HTML but it can get it done. But the mental model for a older smaller model is that ill need to give it a lot of guidance to achieve that.
For the case of “LLM can ask clarifying questions”, this is a harness diff. When you type statically detected UB or compile errors in the compiler example, it technically could interactively ask you want you mean for each instance. The rust compiler does half of this as well. It’s just usually inefficient when “IDE is a chat interface” isn’t true.
The output of LLMs is defined as much as optimizing compilers is: you give some code and it has infinite ways to achieve some set of behavior. Maybe I’m misunderstanding what you mean by defined?
score 1 · canonical
Sorry, I misread your earlier comment. In my reply when I said “mental model” I meant your mental model of the thing you’re trying to build, not of the tool.
It definitely is not. Compilers perform a very well defined transformation from input to behavior (not 100% defined, but as close as practical politics can get it). A compiler that “optimizes” by violating that definition is a broken compiler. (The 1% that’s undefined causes a lot of discussion, which takes place while everyone relies absolutely on the other 99% being correct.)
Whereas in most cases you can’t even tell if the output of an LLM did achieve desired behavior, because the input isn’t precise enough to determine the desired behavior in the first place. LLMs not backstopped by deterministic tools (compilers, test harnesses, proof engines, even just plain reliable arithmetic) in a loop have a hard time even making artifacts that are functional.
score 1 · canonical
Humans have real issues writing large chunks of C++ without UBs, though. Looking at what each GCC bump uncovers across a large distro, I do not really believe compilers being used as well-defined transformations within their well-defined zone is a reality. And all fast-moving compilers have a long list of actually-correct code they miscompile, it is just about exotic enough combinations that people do not prioritise cleaning this all up.
It’s the same tolerance to for-profit companies releasing paid-for network-touching software full of UBs that now enables careless use of LLM-generated code is well. I sometimes wonder if the correct approach would be to pretend to believe, and to say that now that LLMs can fix all the bugs any defects are intentionally left in thus normal consumer product liability rules can finally be applied to proprietary software by for-profit companies.
Speaking of notionally-functional, I am kind of surprised of what latest-gen local-ish LLMs (aggressively quantised 120B or less aggressively quantised smaller ones) sometimes can one-shot. And if you do not give me a compiler/interpreter as a deterministic backstop, neither am I any good at one-shotting longer pieces of code than that.
score 1 · canonical
This isnt the case for most frontier models today; They can do the job of an optimizing compiler (generate working & efficient) asm from just source without calling out to verification tools. Even with pseudocode unseen before by guestimating semantics.
I agree though that it’s not the case for smaller models. Mixing both under the “LLM” umbrella, while technically correct, loses a lot of nuance for claims.
The evaluator/prompter can. It’s how miscompilations and misoptimizations are found in existing compilers.
The behavior is defined. The transformation is not.
If you have a set of bitwise and ternary expressions with bit operations, theres hundreds of ways to get that behavior; Could it
vpternlogwith avblendmpat the end for multiple instances? Could it cmp+branch for each one? Could it eager-eval thencmov? These decisions and their various heuristics change across compiler versions.Only a few years ago, you had to use inline asm to get clang to emit
lock btson x86. Now, it understands the semantics ofatomic_fetch_or_*, the args going into it, and how its result is operated on, such that you can write the semantically equiv without hacks to getlock btscodegen. But if it can’t figure some aspects out, it still falls back to alock cmpxchgloop. The transformations (or “mental model”) is something intuitively built up as you use the specific tool, not defined somewhere. This applies to LLMs as well.score 1 · canonical