Humans are bad at writing secure code, and GNOME developers are no exception. GNOME is primarily written using unsafe programming languages where simple mistakes in our code lead to devastating consequences for our users, and we make these mistakes all the time. No matter how much we try, GNOME developers will fail write secure code...
There is zero hope of maintaining quality software in 2026 without AI vulnerability scanning. Any claims to the contrary are unserious and delusional.
You’re kidding me, right? What about, I dunno, not using those unsafe languages? What about just using a vector sometimes instead of utilizing pointers 24/7?
This is just a bad faith argument to lionize LLM usage.
Even if you don’t like Rust, I guarantee you that you could write a language that actually suits your needs and compile it to LLVM. It’s much easier than banging your head against the wall, i.e. keep writing C/C++/Vala and hope it doesn’t blow up.
My silly pet project (the pi searcher) is in Rust, and I still got some useful feedback about a potential XSS and a potential high resource use DoS path on my code from an LLM. Nothing terrible or RCE-ish but it was still useful.
The power dynamics are not the same. You are a hobbyist and it’s difficult for you to get feedback.
They, on the other hand, are the GNOME project and have much stronger backing. They don’t get to say ‘well, clanker gave the green light, must be quality software’.
The author doesn’t understand or think about what quality software is or what it takes to write some.
Perhaps more accurate to note it’s a hobby project, or one where I’m acting as a hobbyist. (I’m a computer science professor.) It’s correct that I don’t bother soliciting feedback on this one as it’s just a personal project that amuses me, and that lack of code review does create more opportunities for bugs to linger. But that said, many , many bugs escape many rounds of code review in the real world too. It seems silly to not take advantage of LLM review if it’s providing a good ratio of accurate feedback to false positives.
I meant no disrespect by calling you a hobbyist, sorry about that. My point is, we are people, and they’re the GNOME project, and applying the same standards to us and to them would be unfair.
We should be able to gain power that’s otherwise inaccessible to us by LLM reviews, and they shouldn’t need it, at least, they shouldn’t need it as much as we do.
And they certainly shouldn’t throw their hands up and say “if you disagree with our perspective on LLM use, you’re insane or lying”.
The author doesn’t understand or think about what quality software is or what it takes to write some.
The author: “… a member of the GNOME Release Team, the Fedora Workstation Working Group, and the desktop team at Red Hat. Before joining Red Hat, I worked on the WebKit team at Igalia, maintaining Epiphany, WebKitGTK, and glib-networking.”
Seems more likely they understand what quality software is, you just disagree with them.
(I’m happy to see more nuanced LLM takes emerging, at least.)
After an appeal to authority like this, I’ll surely retract all my opinions and abandon all my positions.
Obviously.
Also, if ‘nuance’ means “let’s all joyously accept our LLM overlords, because frankly, we can’t write any quality software without them”, go ahead, exclude me from all ‘nuance’, I don’t need it.
People love to cite “appeal to authority” whenever an expert disagrees with them, as if it magically negates their expertise. But unless you can prove someone is NOT an expert despite having credentials/title/position/etc, their “authority” is a relevant shorthand for expertise. From a classical logic perspective, this is not true, but classical logic is simplistic. From a Bayesian perspective, authority should usefully update one’s priors.
Regardless, you provided even less justification for your assessment of the author, so it’s not like you have a logically better argument. There’s not even an argument there, just an assertion.
I know we’re being snarky, but I’ll respond to the nuance bit as if it were real. Nuance doesn’t mean we can’t write quality software without LLMs, but if we’re up-to-date on current LLM capabilities, we have to admit they can do a lot of what they say on the tin. In some areas, like debugging and security, they’re already better or faster than most human devs. Any argument that starts off assuming LLMs don’t work in any scenario, is either naive or out-of-date.
A nuanced argument involves looking at them and deciding how to handle the trade-offs instead. How do we enable junior devs to grow? How to we triage large numbers of issues/PRs? How do we compensate for LLMs’ (currently) weak high-level architectural design? How do I balance learning with throughput, so my skills don’t atrophy? Which skills are still relevant? Etc.
I guess this is where I should drop the snark and tell you that my attitude towards this discussion isn’t purely grounded in technical terms. In an idealized environment where LLMs have no adverse impact on the world they reside in, a developer can write code by hand, iterate until they can’t see an issue, and only then ask an LLM for a review. This evades a lot of the “technical” problems with LLM usage at a slight cost of development time. More importantly, it works.
I just think it’s especially ironic to make the claim “there is no way to write quality software without LLM use”.
I never for a second considered GNOME, Epiphany, GLib, WebKit, Fedora in general to be “quality software”. Quality software acknowledges that being readable is more important than having been written. Quality software is Lua, it’s TCC, it’s NetBSD, it’s Guix. And none of them required an LLM touch.
From what I’ve seen, Red Hat has never cared if people that are not their engineers can read their source code. (In fact, you may remember that they actively tried to close the sources of their operating systems.) Red Hat has no stronger incentive to create quality software than Microsoft does.
So no, I could not disagree more with the author about what “quality software” is. And hearing the author say “no quality software without stochastic parrot” is funny, because I think the author wouldn’t know quality software if it kicked down their door and announced it was suing Red Hat for impersonation.
I have no technical basis of rejecting LLMs altogether. The right open weights model may or may not be able to outperform your average developer, but there is no question if it would help when it’s used alongside a human brain (rather than as a substitute for one).
I have a cultural basis of rejecting them and objecting to them. LLMs are marketed as a way to replace developers, not make their lives easier. LLMs are not our friend, they exist so that we get paid less. In my opinion, no discussion that overlooks this or discounts this can be “nuanced”. And in my opinion, this one didn’t even get close to nuance.
Most importantly, there is absolutely no nuance in stating “you either agree with me, or you’re unserious or delusional”. Who does the author think they are? Jesus Christ? Anakin Skywalker? “Either you’re with me, or you’re my enemy”? Where’s the nuance in that?
We’ve spent more than a decade seeing anti-Rust people say “Rust doesn’t prevent all security issues”, with everyone patiently explained “of course not, but Rust is still worth it.”
So it’s wild to see a post that actually seems to imply Rust is an adequate solution to all security problems—no need to use automated vulnerability scans once you’ve used Rust, and no need to try and improve the security of the millions (billions?) of existing lines of C and C++ code in the wild.
I’m not saying Rust is an adequate solution to all problems. I’m just saying it’s dishonest to say ‘LLMs are a prerequisite of quality software’ by pointing at memory safety issues of the languages they chose for themselves.
You can write quality software without LLMs. We’ve done it for decades, and we can still do it. Rust deals with the intractable memory safety, and care deals with the logic bugs.
You can write quality software without LLMs. We’ve done it for decades, and we can still do it.
An interesting thing I’ve been doing lately is point an LLM to code I wrote years ago that I put a lot of effort into, and give me feedback about how good it is and what’s broken. Generally the feedback is it’s quite good (things like differential randomized testing work very well!), but they consistently find issues I hadn’t thought about. I’ve never received a completely clean bill of health.
That LLMs have been finding bugs is more so an effect of the tech industry not taking security seriously than them being amazing at it. It also doesn’t hurt that GenAI companies are pumping millions into making their models look as amazing as possible. If you hire a team of pentesters for a few million bucks, I’m sure you’ll get way more (both quality and quantity) than whatever the LLMs regurgitated.
If planet-burning slop machines are what it takes for the tech industry to start taking security seriously… heh. Though, that being said, throwing slop-machines at it doesn’t mean they’re taking it seriously, so I don’t even think it had that effect. Either way, all we can do is object to this extremely unethical technology.
If you hire a team of pentesters for a few million bucks, I’m sure you’ll get way more (both quality and quantity) than whatever the LLMs regurgitated.
How about hiring pentesters for the ~$100 it takes to run a comprehensive security scan with a frontier model?
Ah, yes, ignoring all of the negative externalities (like pollution), massive capital investment in data centers, LLM companies not turning a profit, etc. we can pretend the slop machines are really cheap!
If a drug dealer offers me the first hit for free, that’s free drugs, that’s fantastic, I love free drugs.
When you compare emissions from a computer to emissions from a human, this is a false comparison.
The human already exists and will be producing more or less the same emissions no matter what work you assign to them. However many pentesters you hire, if they are working on pentesting instead of e.g. web development, there’s negligible difference in emissions. Even if they are unemployed and sitting around at home, they will produce similar emissions, perhaps slightly less if they lose their work commute and/or videoconferencing.
The only way you get rid of the human emissions is if you kill the human. The only way to drastically reduce a human’s emissions is to drastically change their society, e.g. by destroying all fossil fuel generators, or by deporting them to an undeveloped country with much lower emissions (changing which society they exist in). Do either poorly and it’s just a fancy way to kill the human.
But you could turn off the LLM data centers and shut down their gas turbines, without killing sentient beings, and their emissions would stop. Humans survived without LLMs only a few years ago, while fossil fuel infrastructure has been baked into our society for over a century.
It feels like you didn’t really read or understand what I said. $100 in tokens is not the true cost. Let me rephrase:
Negative externalities by definition are not included in the price, and pollution is only one of many negative externalities from LLMs. I’ll mention scrapers pushing websites offline or behind paywalls, and RAM prices making personal computing less accessible, but I could spend days listing harms not captured in the price, and then you wouldn’t read my post because it would be too long.
Capital costs may be better considered of as embodied emissions in this context. The emissions from running an LLM doesn’t reflect the emissions that were created while building the building, manufacturing the computers inside it, or training the model. It’s not correct to only count the incremental emissions from additional use of the LLM.
Losing money means that LLM companies are somehow failing to make back their capital costs. It may be a play to dominate the market and therefore be in a position to raise prices later, it may be incompetence, but whatever the cause, it’s financially unsustainable. It seems the prices of tokens don’t even reflect the true cost to the company, let alone the true costs to society.
So, salaries for human pentesters cannot be compared to tokens for slop machines.
I also don’t think you can get the same results as $1 million on pentester salaries with $100 in tokens, but whatever.
I also don’t think you can get the same results as $1 million on pentester salaries with $100 in tokens, but whatever.
You will certainly get an equivalent security analysis for orders of magnitude less than $1m in tokens though.
Anyway, your alternative scenarios for humans (pentest or loaf at home) leave out the obvious: move humans away from activity where computers do it better, and have them do something more productive, like high-level architecture and design. It’s the same logic as switching to power drills and cranes, instead of making artisanal woodworking.
Your focus on the costs without mentioning the benefits, assumes LLMs are useless (given you call them “slop machines”). But since they have workable use cases (and in the case of security, they clearly do), then it becomes like any other tech, where we weigh tradeoffs.
I personally think the environmental harms will be short-term, and mitigated over time in a variety of ways. More efficient models, cheap software rewrites into more efficient langs, increased model efficiency from more software, model-driven research into environmental tech, etc.
(Now, whether humanity would actually build the current amount of stuff more efficiently, or just build more is a separate question. But that’s much older than LLMs, and is more your-underlying-problem-is-actually-capitalism again, which cannot be solved by addressing the symptoms instead of the root cause. But that’s a whole 'nother conversation.)
There is some very strange cognitive dissonance in your comment. On the one hand, you say that efficiency will solve the climate impact of LLMs. On the other hand, you recognize that efficiency won’t solve the climate impact, in the same comment! You handwave the problem with efficiency away by blaming it on capitalism, but who cares why efficiency doesn’t work if you’ve just invalidated your own solution?
For those not familiar, we’re discussing the Jevons Paradox, which says as we make a technology more efficient, we use it more, thereby increasing use of resources. I recently saw a fantastic framing of the Jevons Paradox:
Making use of a resource more profitable increases the return on investment of exploiting that resource.
So increased systems efficiency increases, not decreases, resource use.
Unless your approach is to have a fixed expectation for capability and just optimize for lower and lower resource use to achieve that capability. Which is not how capitalism works, at all.
I agree that this problem is almost impossible to solve under capitalism, capitalism needs to go, but let’s be fair, it’s still a very difficult problem for any alternative economic system. I’d say we should reject any alternative economic system if it doesn’t have some answer for this formulation of the Jevons Paradox; it seems like a good test for whether it can handle existential threats like the climate crisis.
Ahh, the “dissonance” is you’re thinking of the Jevons Paradox as being about increased energy use, when it’s more specifically about the increased usage of goods and services, not necessarily energy.
You’re on the right track, but e.g., if LLM use increased 10x, but we’re able to make them 20x more efficient, the energy use will drop, despite Jevons. Alternatively, Jevons could drive 20x more carbon-neutral solar/wind development making the increased energy use of LLMs (and everything else) a moot point. This is what I think (and hope) will happen, even under capitalism.
It’s unclear to me that the Jevons Paradox would be as much of a problem under other economic systems. To use its original observation about coal: yes, ordinary citizens would continue to want coal and electricity, but much of the usage was by capitalist-driven industry. Companies had to use more coal to ensure they weren’t out-competed by those who did.
A more democratic economic system could weigh the trade-offs, and voters could decide how quickly and how much to adopt a new tech.
Isn’t it pretty clear though that for something like a security scan or similar that even the externalities included price far cheaper than the alternative? Whether that’s a good thing or not can still be an open question but I’d be interested in hearing the argument that LLMs are only cost effective for finding vulnerabilities because tokens are subsidized/should cost more for some other reason
I’m not disputing that the prices as is may not reflect the true cost of the tokens, or anything like that. What I’m trying to get at is that I think it’s clear that LLMs are extremely good at this sort of thing, so even if it were true that the price of the tokens were 100x off, for this particular use case it still might be viable given the counterfactual is something like “spend a lot of money paying pentesters”.
I agree with you that 1M on pentester salaries is likely to produce a better result than 100 dollars to whatever your favorite LLM provider is, but what I’d be interested to know is what you think the breakpoint there is and how far off current token prices are from their true cost
What about, I dunno, not using those unsafe languages?
This is presumably a problem for the large quantity of existing software written in unsafe languages that is not going to be rewritten overnight (as well as things that need to toe the boundary frequently like JITs).
Unsafe languages need to deal with all the same high-level bugs (like access control or SQL injection) in addition to memory safety bugs. C is not trading memory unsafety for some other safety, it’s strictly more unsafe.
Rust does a lot to prevent logic bugs, e.g. error handling can be enforced by the compiler, while C can’t even decide whether returning 0 means OK or false.
Memory unsafety vulnerabilities are exceptional in their unbound scope and severity.
Without arbitrary code execution, the danger is roughly limited to what the code is supposed to do, and plenty of programs have lots of boring code that doesn’t touch anything sensitive. Even if you’re implementing access control, the worst case without RCE is failing open on that endpoint, rather than enabling installation of a rootkit or such.
A memory-safe JSON parser can at worst give you garbage data or heat your CPU or eat your RAM, but it won’t run arbitrary attacker-supplied code. If you’re implementing a widget that queries the current weather, the the worst case is not getting the weather info. In C, the worst case is getting the whole system pwned through a weather info widget. It’s a massive difference in the attack surface.
It’s really really hard to make an off-by-one error in Rust that turns it into a JIT or scripting language.
When a language has an API for executing arbitrary code, misuse of that API leads to arbitrary code execution, duh. That still fits under “the danger is roughly limited to what the code is supposed to do”. But you’re missing the point that in C you don’t need an API for RCE get RCE. In C you have a much larger problem of memory unsafety and UB turning potentially any API into an arbitrary code execution gadget.
I don’t think anyone is arguing that C is as safe as Rust? Certainly my point is that Rust can still have bugs, and those bugs can be security critical.
There is no programming language that can’t have bugs, so such argument doesn’t state anything useful. It’s an empty truism.
OTOH the whataboutist/defeatist responses create a false equivalence. Even though both languages may have bugs: (P(bug | C) > 0) == (P(bug | Rust) > 0) stating it that way obscures the major difference in likelihood of the bugs: P(bug | C) >> P(bug | Rust).
You’re kidding me, right? What about, I dunno, not using those unsafe languages? What about just using a vector sometimes instead of utilizing pointers 24/7?
This is just a bad faith argument to lionize LLM usage.
Even if you don’t like Rust, I guarantee you that you could write a language that actually suits your needs and compile it to LLVM. It’s much easier than banging your head against the wall, i.e. keep writing C/C++/Vala and hope it doesn’t blow up.
score 1 · canonical
deleted by creator
My silly pet project (the pi searcher) is in Rust, and I still got some useful feedback about a potential XSS and a potential high resource use DoS path on my code from an LLM. Nothing terrible or RCE-ish but it was still useful.
score 2 · canonical
The power dynamics are not the same. You are a hobbyist and it’s difficult for you to get feedback.
They, on the other hand, are the GNOME project and have much stronger backing. They don’t get to say ‘well, clanker gave the green light, must be quality software’.
The author doesn’t understand or think about what quality software is or what it takes to write some.
score 1 · canonical
Perhaps more accurate to note it’s a hobby project, or one where I’m acting as a hobbyist. (I’m a computer science professor.) It’s correct that I don’t bother soliciting feedback on this one as it’s just a personal project that amuses me, and that lack of code review does create more opportunities for bugs to linger. But that said, many , many bugs escape many rounds of code review in the real world too. It seems silly to not take advantage of LLM review if it’s providing a good ratio of accurate feedback to false positives.
score 1 · canonical
I meant no disrespect by calling you a hobbyist, sorry about that. My point is, we are people, and they’re the GNOME project, and applying the same standards to us and to them would be unfair.
We should be able to gain power that’s otherwise inaccessible to us by LLM reviews, and they shouldn’t need it, at least, they shouldn’t need it as much as we do.
And they certainly shouldn’t throw their hands up and say “if you disagree with our perspective on LLM use, you’re insane or lying”.
score 1 · canonical
The author: “… a member of the GNOME Release Team, the Fedora Workstation Working Group, and the desktop team at Red Hat. Before joining Red Hat, I worked on the WebKit team at Igalia, maintaining Epiphany, WebKitGTK, and glib-networking.”
Seems more likely they understand what quality software is, you just disagree with them.
(I’m happy to see more nuanced LLM takes emerging, at least.)
score 1 · canonical
After an appeal to authority like this, I’ll surely retract all my opinions and abandon all my positions.
Obviously.
Also, if ‘nuance’ means “let’s all joyously accept our LLM overlords, because frankly, we can’t write any quality software without them”, go ahead, exclude me from all ‘nuance’, I don’t need it.
score 1 · canonical
People love to cite “appeal to authority” whenever an expert disagrees with them, as if it magically negates their expertise. But unless you can prove someone is NOT an expert despite having credentials/title/position/etc, their “authority” is a relevant shorthand for expertise. From a classical logic perspective, this is not true, but classical logic is simplistic. From a Bayesian perspective, authority should usefully update one’s priors.
Regardless, you provided even less justification for your assessment of the author, so it’s not like you have a logically better argument. There’s not even an argument there, just an assertion.
I know we’re being snarky, but I’ll respond to the nuance bit as if it were real. Nuance doesn’t mean we can’t write quality software without LLMs, but if we’re up-to-date on current LLM capabilities, we have to admit they can do a lot of what they say on the tin. In some areas, like debugging and security, they’re already better or faster than most human devs. Any argument that starts off assuming LLMs don’t work in any scenario, is either naive or out-of-date.
A nuanced argument involves looking at them and deciding how to handle the trade-offs instead. How do we enable junior devs to grow? How to we triage large numbers of issues/PRs? How do we compensate for LLMs’ (currently) weak high-level architectural design? How do I balance learning with throughput, so my skills don’t atrophy? Which skills are still relevant? Etc.
score 1 · canonical
I guess this is where I should drop the snark and tell you that my attitude towards this discussion isn’t purely grounded in technical terms. In an idealized environment where LLMs have no adverse impact on the world they reside in, a developer can write code by hand, iterate until they can’t see an issue, and only then ask an LLM for a review. This evades a lot of the “technical” problems with LLM usage at a slight cost of development time. More importantly, it works.
I just think it’s especially ironic to make the claim “there is no way to write quality software without LLM use”.
I never for a second considered GNOME, Epiphany, GLib, WebKit, Fedora in general to be “quality software”. Quality software acknowledges that being readable is more important than having been written. Quality software is Lua, it’s TCC, it’s NetBSD, it’s Guix. And none of them required an LLM touch.
From what I’ve seen, Red Hat has never cared if people that are not their engineers can read their source code. (In fact, you may remember that they actively tried to close the sources of their operating systems.) Red Hat has no stronger incentive to create quality software than Microsoft does.
So no, I could not disagree more with the author about what “quality software” is. And hearing the author say “no quality software without stochastic parrot” is funny, because I think the author wouldn’t know quality software if it kicked down their door and announced it was suing Red Hat for impersonation.
I have no technical basis of rejecting LLMs altogether. The right open weights model may or may not be able to outperform your average developer, but there is no question if it would help when it’s used alongside a human brain (rather than as a substitute for one).
I have a cultural basis of rejecting them and objecting to them. LLMs are marketed as a way to replace developers, not make their lives easier. LLMs are not our friend, they exist so that we get paid less. In my opinion, no discussion that overlooks this or discounts this can be “nuanced”. And in my opinion, this one didn’t even get close to nuance.
Most importantly, there is absolutely no nuance in stating “you either agree with me, or you’re unserious or delusional”. Who does the author think they are? Jesus Christ? Anakin Skywalker? “Either you’re with me, or you’re my enemy”? Where’s the nuance in that?
score 2 · canonical
We’ve spent more than a decade seeing anti-Rust people say “Rust doesn’t prevent all security issues”, with everyone patiently explained “of course not, but Rust is still worth it.”
So it’s wild to see a post that actually seems to imply Rust is an adequate solution to all security problems—no need to use automated vulnerability scans once you’ve used Rust, and no need to try and improve the security of the millions (billions?) of existing lines of C and C++ code in the wild.
score 2 · canonical
I’m not saying Rust is an adequate solution to all problems. I’m just saying it’s dishonest to say ‘LLMs are a prerequisite of quality software’ by pointing at memory safety issues of the languages they chose for themselves.
You can write quality software without LLMs. We’ve done it for decades, and we can still do it. Rust deals with the intractable memory safety, and care deals with the logic bugs.
score 1 · canonical
An interesting thing I’ve been doing lately is point an LLM to code I wrote years ago that I put a lot of effort into, and give me feedback about how good it is and what’s broken. Generally the feedback is it’s quite good (things like differential randomized testing work very well!), but they consistently find issues I hadn’t thought about. I’ve never received a completely clean bill of health.
score 1 · canonical
That LLMs have been finding bugs is more so an effect of the tech industry not taking security seriously than them being amazing at it. It also doesn’t hurt that GenAI companies are pumping millions into making their models look as amazing as possible. If you hire a team of pentesters for a few million bucks, I’m sure you’ll get way more (both quality and quantity) than whatever the LLMs regurgitated.
If planet-burning slop machines are what it takes for the tech industry to start taking security seriously… heh. Though, that being said, throwing slop-machines at it doesn’t mean they’re taking it seriously, so I don’t even think it had that effect. Either way, all we can do is object to this extremely unethical technology.
score 1 · canonical
How about hiring pentesters for the ~$100 it takes to run a comprehensive security scan with a frontier model?
score 1 · canonical
Ah, yes, ignoring all of the negative externalities (like pollution), massive capital investment in data centers, LLM companies not turning a profit, etc. we can pretend the slop machines are really cheap!
If a drug dealer offers me the first hit for free, that’s free drugs, that’s fantastic, I love free drugs.
score 2 · canonical
I’m pretty sure spending a million dollars on pentesters produces a whole lot more pollution than $100 on tokens.
score 1 · canonical
When you compare emissions from a computer to emissions from a human, this is a false comparison.
The human already exists and will be producing more or less the same emissions no matter what work you assign to them. However many pentesters you hire, if they are working on pentesting instead of e.g. web development, there’s negligible difference in emissions. Even if they are unemployed and sitting around at home, they will produce similar emissions, perhaps slightly less if they lose their work commute and/or videoconferencing.
The only way you get rid of the human emissions is if you kill the human. The only way to drastically reduce a human’s emissions is to drastically change their society, e.g. by destroying all fossil fuel generators, or by deporting them to an undeveloped country with much lower emissions (changing which society they exist in). Do either poorly and it’s just a fancy way to kill the human.
But you could turn off the LLM data centers and shut down their gas turbines, without killing sentient beings, and their emissions would stop. Humans survived without LLMs only a few years ago, while fossil fuel infrastructure has been baked into our society for over a century.
It feels like you didn’t really read or understand what I said. $100 in tokens is not the true cost. Let me rephrase:
So, salaries for human pentesters cannot be compared to tokens for slop machines.
I also don’t think you can get the same results as $1 million on pentester salaries with $100 in tokens, but whatever.
score 1 · canonical
You will certainly get an equivalent security analysis for orders of magnitude less than $1m in tokens though.
Anyway, your alternative scenarios for humans (pentest or loaf at home) leave out the obvious: move humans away from activity where computers do it better, and have them do something more productive, like high-level architecture and design. It’s the same logic as switching to power drills and cranes, instead of making artisanal woodworking.
Your focus on the costs without mentioning the benefits, assumes LLMs are useless (given you call them “slop machines”). But since they have workable use cases (and in the case of security, they clearly do), then it becomes like any other tech, where we weigh tradeoffs.
I personally think the environmental harms will be short-term, and mitigated over time in a variety of ways. More efficient models, cheap software rewrites into more efficient langs, increased model efficiency from more software, model-driven research into environmental tech, etc.
(Now, whether humanity would actually build the current amount of stuff more efficiently, or just build more is a separate question. But that’s much older than LLMs, and is more your-underlying-problem-is-actually-capitalism again, which cannot be solved by addressing the symptoms instead of the root cause. But that’s a whole 'nother conversation.)
score 1 · canonical
There is some very strange cognitive dissonance in your comment. On the one hand, you say that efficiency will solve the climate impact of LLMs. On the other hand, you recognize that efficiency won’t solve the climate impact, in the same comment! You handwave the problem with efficiency away by blaming it on capitalism, but who cares why efficiency doesn’t work if you’ve just invalidated your own solution?
For those not familiar, we’re discussing the Jevons Paradox, which says as we make a technology more efficient, we use it more, thereby increasing use of resources. I recently saw a fantastic framing of the Jevons Paradox:
I agree that this problem is almost impossible to solve under capitalism, capitalism needs to go, but let’s be fair, it’s still a very difficult problem for any alternative economic system. I’d say we should reject any alternative economic system if it doesn’t have some answer for this formulation of the Jevons Paradox; it seems like a good test for whether it can handle existential threats like the climate crisis.
score 3 · canonical
Ahh, the “dissonance” is you’re thinking of the Jevons Paradox as being about increased energy use, when it’s more specifically about the increased usage of goods and services, not necessarily energy.
You’re on the right track, but e.g., if LLM use increased 10x, but we’re able to make them 20x more efficient, the energy use will drop, despite Jevons. Alternatively, Jevons could drive 20x more carbon-neutral solar/wind development making the increased energy use of LLMs (and everything else) a moot point. This is what I think (and hope) will happen, even under capitalism.
It’s unclear to me that the Jevons Paradox would be as much of a problem under other economic systems. To use its original observation about coal: yes, ordinary citizens would continue to want coal and electricity, but much of the usage was by capitalist-driven industry. Companies had to use more coal to ensure they weren’t out-competed by those who did.
A more democratic economic system could weigh the trade-offs, and voters could decide how quickly and how much to adopt a new tech.
score 1 · canonical
Isn’t it pretty clear though that for something like a security scan or similar that even the externalities included price far cheaper than the alternative? Whether that’s a good thing or not can still be an open question but I’d be interested in hearing the argument that LLMs are only cost effective for finding vulnerabilities because tokens are subsidized/should cost more for some other reason
score 1 · canonical
The Enron-like accounting of LLM companies alone should be enough to make these prices unclear. https://news.bloombergtax.com/financial-accounting/big-tech-ai-spree-revives-accounting-devices-that-toppled-enron
One negative externality is the risk of global financial collapse. So much money is tied up in LLMs and they’re so fishy.
score 1 · canonical
I’m not disputing that the prices as is may not reflect the true cost of the tokens, or anything like that. What I’m trying to get at is that I think it’s clear that LLMs are extremely good at this sort of thing, so even if it were true that the price of the tokens were 100x off, for this particular use case it still might be viable given the counterfactual is something like “spend a lot of money paying pentesters”.
I agree with you that 1M on pentester salaries is likely to produce a better result than 100 dollars to whatever your favorite LLM provider is, but what I’d be interested to know is what you think the breakpoint there is and how far off current token prices are from their true cost
score 2 · canonical
This is presumably a problem for the large quantity of existing software written in unsafe languages that is not going to be rewritten overnight (as well as things that need to toe the boundary frequently like JITs).
score 1 · canonical
Ok but what if it’s possible to still write bugs in rust?
score 2 · canonical
Most bugs don’t really matter; what matters are vulnerabilities. And Rust is really good at preventing memory corruption bugs.
score 1 · canonical
There are so many vulnerabilities that can be present without memory corruption. Not checking for an access control is perfectly memory safe
score 2 · canonical
All 10 of the OWASP Top 10! https://top10.owasp.org/2025/
score 1 · canonical
Unsafe languages need to deal with all the same high-level bugs (like access control or SQL injection) in addition to memory safety bugs. C is not trading memory unsafety for some other safety, it’s strictly more unsafe. Rust does a lot to prevent logic bugs, e.g. error handling can be enforced by the compiler, while C can’t even decide whether returning 0 means OK or false.
Memory unsafety vulnerabilities are exceptional in their unbound scope and severity.
Without arbitrary code execution, the danger is roughly limited to what the code is supposed to do, and plenty of programs have lots of boring code that doesn’t touch anything sensitive. Even if you’re implementing access control, the worst case without RCE is failing open on that endpoint, rather than enabling installation of a rootkit or such.
A memory-safe JSON parser can at worst give you garbage data or heat your CPU or eat your RAM, but it won’t run arbitrary attacker-supplied code. If you’re implementing a widget that queries the current weather, the the worst case is not getting the weather info. In C, the worst case is getting the whole system pwned through a weather info widget. It’s a massive difference in the attack surface.
score 2 · canonical
Which is why log4shell was impossible to exploit in the memory safe languages Java and shell.
These sorts of things are called weird machines because they’re really weird.
score 1 · canonical
It’s really really hard to make an off-by-one error in Rust that turns it into a JIT or scripting language.
When a language has an API for executing arbitrary code, misuse of that API leads to arbitrary code execution, duh. That still fits under “the danger is roughly limited to what the code is supposed to do”. But you’re missing the point that in C you don’t need an API for RCE get RCE. In C you have a much larger problem of memory unsafety and UB turning potentially any API into an arbitrary code execution gadget.
score 1 · canonical
I don’t think anyone is arguing that C is as safe as Rust? Certainly my point is that Rust can still have bugs, and those bugs can be security critical.
score 1 · canonical
There is no programming language that can’t have bugs, so such argument doesn’t state anything useful. It’s an empty truism.
OTOH the whataboutist/defeatist responses create a false equivalence. Even though both languages may have bugs:
(P(bug | C) > 0) == (P(bug | Rust) > 0)stating it that way obscures the major difference in likelihood of the bugs:P(bug | C) >> P(bug | Rust).score 1 · canonical