• I hate that the author is correct: AI-generated bug reports have that weird jargon about “gated” array access, every version or test is “pinned”, every test case is a “fixture”. Between the omnijargon and the meandering sentences, I’m always about half certain that any given bug report is nonsense. But this article does explain the tension between real, albeit exotic issues and figuring out inhuman verbiage.


    score 1 · canonical

  • There is zero hope of maintaining quality software in 2026 without AI vulnerability scanning. Any claims to the contrary are unserious and delusional.

    You’re kidding me, right? What about, I dunno, not using those unsafe languages? What about just using a vector sometimes instead of utilizing pointers 24/7?

    This is just a bad faith argument to lionize LLM usage.

    Even if you don’t like Rust, I guarantee you that you could write a language that actually suits your needs and compile it to LLVM. It’s much easier than banging your head against the wall, i.e. keep writing C/C++/Vala and hope it doesn’t blow up.


    score 1 · canonical

    • lobsters__dga@crustaceanson.theshelf.topM
      link
      fedilink
      arrow-up
      1
      ·
      19 days ago

      My silly pet project (the pi searcher) is in Rust, and I still got some useful feedback about a potential XSS and a potential high resource use DoS path on my code from an LLM. Nothing terrible or RCE-ish but it was still useful.


      score 2 · canonical

      • The power dynamics are not the same. You are a hobbyist and it’s difficult for you to get feedback.

        They, on the other hand, are the GNOME project and have much stronger backing. They don’t get to say ‘well, clanker gave the green light, must be quality software’.

        The author doesn’t understand or think about what quality software is or what it takes to write some.


        score 1 · canonical

        • lobsters__dga@crustaceanson.theshelf.topM
          link
          fedilink
          arrow-up
          1
          ·
          19 days ago

          Perhaps more accurate to note it’s a hobby project, or one where I’m acting as a hobbyist. (I’m a computer science professor.) It’s correct that I don’t bother soliciting feedback on this one as it’s just a personal project that amuses me, and that lack of code review does create more opportunities for bugs to linger. But that said, many , many bugs escape many rounds of code review in the real world too. It seems silly to not take advantage of LLM review if it’s providing a good ratio of accurate feedback to false positives.


          score 1 · canonical

          • I meant no disrespect by calling you a hobbyist, sorry about that. My point is, we are people, and they’re the GNOME project, and applying the same standards to us and to them would be unfair.

            We should be able to gain power that’s otherwise inaccessible to us by LLM reviews, and they shouldn’t need it, at least, they shouldn’t need it as much as we do.

            And they certainly shouldn’t throw their hands up and say “if you disagree with our perspective on LLM use, you’re insane or lying”.


            score 1 · canonical

        • The author doesn’t understand or think about what quality software is or what it takes to write some.

          The author: “… a member of the GNOME Release Team, the Fedora Workstation Working Group, and the desktop team at Red Hat. Before joining Red Hat, I worked on the WebKit team at Igalia, maintaining Epiphany, WebKitGTK, and glib-networking.”

          Seems more likely they understand what quality software is, you just disagree with them.

          (I’m happy to see more nuanced LLM takes emerging, at least.)


          score 1 · canonical

          • After an appeal to authority like this, I’ll surely retract all my opinions and abandon all my positions.

            Obviously.

            Also, if ‘nuance’ means “let’s all joyously accept our LLM overlords, because frankly, we can’t write any quality software without them”, go ahead, exclude me from all ‘nuance’, I don’t need it.


            score 1 · canonical

            • People love to cite “appeal to authority” whenever an expert disagrees with them, as if it magically negates their expertise. But unless you can prove someone is NOT an expert despite having credentials/title/position/etc, their “authority” is a relevant shorthand for expertise. From a classical logic perspective, this is not true, but classical logic is simplistic. From a Bayesian perspective, authority should usefully update one’s priors.

              Regardless, you provided even less justification for your assessment of the author, so it’s not like you have a logically better argument. There’s not even an argument there, just an assertion.


              I know we’re being snarky, but I’ll respond to the nuance bit as if it were real. Nuance doesn’t mean we can’t write quality software without LLMs, but if we’re up-to-date on current LLM capabilities, we have to admit they can do a lot of what they say on the tin. In some areas, like debugging and security, they’re already better or faster than most human devs. Any argument that starts off assuming LLMs don’t work in any scenario, is either naive or out-of-date.

              A nuanced argument involves looking at them and deciding how to handle the trade-offs instead. How do we enable junior devs to grow? How to we triage large numbers of issues/PRs? How do we compensate for LLMs’ (currently) weak high-level architectural design? How do I balance learning with throughput, so my skills don’t atrophy? Which skills are still relevant? Etc.


              score 1 · canonical

              • I guess this is where I should drop the snark and tell you that my attitude towards this discussion isn’t purely grounded in technical terms. In an idealized environment where LLMs have no adverse impact on the world they reside in, a developer can write code by hand, iterate until they can’t see an issue, and only then ask an LLM for a review. This evades a lot of the “technical” problems with LLM usage at a slight cost of development time. More importantly, it works.

                I just think it’s especially ironic to make the claim “there is no way to write quality software without LLM use”.

                I never for a second considered GNOME, Epiphany, GLib, WebKit, Fedora in general to be “quality software”. Quality software acknowledges that being readable is more important than having been written. Quality software is Lua, it’s TCC, it’s NetBSD, it’s Guix. And none of them required an LLM touch.

                From what I’ve seen, Red Hat has never cared if people that are not their engineers can read their source code. (In fact, you may remember that they actively tried to close the sources of their operating systems.) Red Hat has no stronger incentive to create quality software than Microsoft does.

                So no, I could not disagree more with the author about what “quality software” is. And hearing the author say “no quality software without stochastic parrot” is funny, because I think the author wouldn’t know quality software if it kicked down their door and announced it was suing Red Hat for impersonation.

                I have no technical basis of rejecting LLMs altogether. The right open weights model may or may not be able to outperform your average developer, but there is no question if it would help when it’s used alongside a human brain (rather than as a substitute for one).

                I have a cultural basis of rejecting them and objecting to them. LLMs are marketed as a way to replace developers, not make their lives easier. LLMs are not our friend, they exist so that we get paid less. In my opinion, no discussion that overlooks this or discounts this can be “nuanced”. And in my opinion, this one didn’t even get close to nuance.

                Most importantly, there is absolutely no nuance in stating “you either agree with me, or you’re unserious or delusional”. Who does the author think they are? Jesus Christ? Anakin Skywalker? “Either you’re with me, or you’re my enemy”? Where’s the nuance in that?


                score 2 · canonical

    • That LLMs have been finding bugs is more so an effect of the tech industry not taking security seriously than them being amazing at it. It also doesn’t hurt that GenAI companies are pumping millions into making their models look as amazing as possible. If you hire a team of pentesters for a few million bucks, I’m sure you’ll get way more (both quality and quantity) than whatever the LLMs regurgitated.

      If planet-burning slop machines are what it takes for the tech industry to start taking security seriously… heh. Though, that being said, throwing slop-machines at it doesn’t mean they’re taking it seriously, so I don’t even think it had that effect. Either way, all we can do is object to this extremely unethical technology.


      score 1 · canonical

      • If you hire a team of pentesters for a few million bucks, I’m sure you’ll get way more (both quality and quantity) than whatever the LLMs regurgitated.

        How about hiring pentesters for the ~$100 it takes to run a comprehensive security scan with a frontier model?


        score 1 · canonical

        • Ah, yes, ignoring all of the negative externalities (like pollution), massive capital investment in data centers, LLM companies not turning a profit, etc. we can pretend the slop machines are really cheap!

          If a drug dealer offers me the first hit for free, that’s free drugs, that’s fantastic, I love free drugs.


          score 2 · canonical

          • lobsters__dv@crustaceanson.theshelf.topM
            link
            fedilink
            arrow-up
            1
            ·
            19 days ago

            Isn’t it pretty clear though that for something like a security scan or similar that even the externalities included price far cheaper than the alternative? Whether that’s a good thing or not can still be an open question but I’d be interested in hearing the argument that LLMs are only cost effective for finding vulnerabilities because tokens are subsidized/should cost more for some other reason


            score 1 · canonical

              • lobsters__dv@crustaceanson.theshelf.topM
                link
                fedilink
                arrow-up
                1
                ·
                19 days ago

                I’m not disputing that the prices as is may not reflect the true cost of the tokens, or anything like that. What I’m trying to get at is that I think it’s clear that LLMs are extremely good at this sort of thing, so even if it were true that the price of the tokens were 100x off, for this particular use case it still might be viable given the counterfactual is something like “spend a lot of money paying pentesters”.

                I agree with you that 1M on pentester salaries is likely to produce a better result than 100 dollars to whatever your favorite LLM provider is, but what I’d be interested to know is what you think the breakpoint there is and how far off current token prices are from their true cost


                score 2 · canonical

            • lobsters__skyfaller@crustaceanson.theshelf.top
              link
              fedilink
              arrow-up
              1
              ·
              edit-2
              19 days ago

              When you compare emissions from a computer to emissions from a human, this is a false comparison.

              The human already exists and will be producing more or less the same emissions no matter what work you assign to them. However many pentesters you hire, if they are working on pentesting instead of e.g. web development, there’s negligible difference in emissions. Even if they are unemployed and sitting around at home, they will produce similar emissions, perhaps slightly less if they lose their work commute and/or videoconferencing.

              The only way you get rid of the human emissions is if you kill the human. The only way to drastically reduce a human’s emissions is to drastically change their society, e.g. by destroying all fossil fuel generators, or by deporting them to an undeveloped country with much lower emissions (changing which society they exist in). Do either poorly and it’s just a fancy way to kill the human.

              But you could turn off the LLM data centers and shut down their gas turbines, without killing sentient beings, and their emissions would stop. Humans survived without LLMs only a few years ago, while fossil fuel infrastructure has been baked into our society for over a century.

              It feels like you didn’t really read or understand what I said. $100 in tokens is not the true cost. Let me rephrase:

              • Negative externalities by definition are not included in the price, and pollution is only one of many negative externalities from LLMs. I’ll mention scrapers pushing websites offline or behind paywalls, and RAM prices making personal computing less accessible, but I could spend days listing harms not captured in the price, and then you wouldn’t read my post because it would be too long.
              • Capital costs may be better considered of as embodied emissions in this context. The emissions from running an LLM doesn’t reflect the emissions that were created while building the building, manufacturing the computers inside it, or training the model. It’s not correct to only count the incremental emissions from additional use of the LLM.
              • Losing money means that LLM companies are somehow failing to make back their capital costs. It may be a play to dominate the market and therefore be in a position to raise prices later, it may be incompetence, but whatever the cause, it’s financially unsustainable. It seems the prices of tokens don’t even reflect the true cost to the company, let alone the true costs to society.

              So, salaries for human pentesters cannot be compared to tokens for slop machines.

              I also don’t think you can get the same results as $1 million on pentester salaries with $100 in tokens, but whatever.


              score 1 · canonical

              • I also don’t think you can get the same results as $1 million on pentester salaries with $100 in tokens, but whatever.

                You will certainly get an equivalent security analysis for orders of magnitude less than $1m in tokens though.


                Anyway, your alternative scenarios for humans (pentest or loaf at home) leave out the obvious: move humans away from activity where computers do it better, and have them do something more productive, like high-level architecture and design. It’s the same logic as switching to power drills and cranes, instead of making artisanal woodworking.

                Your focus on the costs without mentioning the benefits, assumes LLMs are useless (given you call them “slop machines”). But since they have workable use cases (and in the case of security, they clearly do), then it becomes like any other tech, where we weigh tradeoffs.

                I personally think the environmental harms will be short-term, and mitigated over time in a variety of ways. More efficient models, cheap software rewrites into more efficient langs, increased model efficiency from more software, model-driven research into environmental tech, etc.

                (Now, whether humanity would actually build the current amount of stuff more efficiently, or just build more is a separate question. But that’s much older than LLMs, and is more your-underlying-problem-is-actually-capitalism again, which cannot be solved by addressing the symptoms instead of the root cause. But that’s a whole 'nother conversation.)


                score 1 · canonical

                • lobsters__skyfaller@crustaceanson.theshelf.top
                  link
                  fedilink
                  arrow-up
                  1
                  ·
                  edit-2
                  18 days ago

                  There is some very strange cognitive dissonance in your comment. On the one hand, you say that efficiency will solve the climate impact of LLMs. On the other hand, you recognize that efficiency won’t solve the climate impact, in the same comment! You handwave the problem with efficiency away by blaming it on capitalism, but who cares why efficiency doesn’t work if you’ve just invalidated your own solution?

                  For those not familiar, we’re discussing the Jevons Paradox, which says as we make a technology more efficient, we use it more, thereby increasing use of resources. I recently saw a fantastic framing of the Jevons Paradox:

                  Making use of a resource more profitable increases the return on investment of exploiting that resource.

                  So increased systems efficiency increases, not decreases, resource use.

                  Unless your approach is to have a fixed expectation for capability and just optimize for lower and lower resource use to achieve that capability. Which is not how capitalism works, at all.

                  I agree that this problem is almost impossible to solve under capitalism, capitalism needs to go, but let’s be fair, it’s still a very difficult problem for any alternative economic system. I’d say we should reject any alternative economic system if it doesn’t have some answer for this formulation of the Jevons Paradox; it seems like a good test for whether it can handle existential threats like the climate crisis.


                  score 3 · canonical

    • lobsters__hyperpape@crustaceanson.theshelf.topM
      link
      fedilink
      arrow-up
      1
      ·
      edit-2
      19 days ago

      We’ve spent more than a decade seeing anti-Rust people say “Rust doesn’t prevent all security issues”, with everyone patiently explained “of course not, but Rust is still worth it.”

      So it’s wild to see a post that actually seems to imply Rust is an adequate solution to all security problems—no need to use automated vulnerability scans once you’ve used Rust, and no need to try and improve the security of the millions (billions?) of existing lines of C and C++ code in the wild.


      score 2 · canonical

      • I’m not saying Rust is an adequate solution to all problems. I’m just saying it’s dishonest to say ‘LLMs are a prerequisite of quality software’ by pointing at memory safety issues of the languages they chose for themselves.

        You can write quality software without LLMs. We’ve done it for decades, and we can still do it. Rust deals with the intractable memory safety, and care deals with the logic bugs.


        score 1 · canonical

        • You can write quality software without LLMs. We’ve done it for decades, and we can still do it.

          An interesting thing I’ve been doing lately is point an LLM to code I wrote years ago that I put a lot of effort into, and give me feedback about how good it is and what’s broken. Generally the feedback is it’s quite good (things like differential randomized testing work very well!), but they consistently find issues I hadn’t thought about. I’ve never received a completely clean bill of health.


          score 1 · canonical

    • What about, I dunno, not using those unsafe languages?

      This is presumably a problem for the large quantity of existing software written in unsafe languages that is not going to be rewritten overnight (as well as things that need to toe the boundary frequently like JITs).


      score 1 · canonical

          • lobsters__kornel@crustaceanson.theshelf.topM
            link
            fedilink
            arrow-up
            1
            ·
            edit-2
            19 days ago
            1. Unsafe languages need to deal with all the same high-level bugs (like access control or SQL injection) in addition to memory safety bugs. C is not trading memory unsafety for some other safety, it’s strictly more unsafe. Rust does a lot to prevent logic bugs, e.g. error handling can be enforced by the compiler, while C can’t even decide whether returning 0 means OK or false.

            2. Memory unsafety vulnerabilities are exceptional in their unbound scope and severity.

            Without arbitrary code execution, the danger is roughly limited to what the code is supposed to do, and plenty of programs have lots of boring code that doesn’t touch anything sensitive. Even if you’re implementing access control, the worst case without RCE is failing open on that endpoint, rather than enabling installation of a rootkit or such.

            A memory-safe JSON parser can at worst give you garbage data or heat your CPU or eat your RAM, but it won’t run arbitrary attacker-supplied code. If you’re implementing a widget that queries the current weather, the the worst case is not getting the weather info. In C, the worst case is getting the whole system pwned through a weather info widget. It’s a massive difference in the attack surface.


            score 2 · canonical

            • A memory-safe JSON parser can at worst give you garbage data or heat your CPU or eat your RAM, but it won’t run arbitrary attacker-supplied code

              Which is why log4shell was impossible to exploit in the memory safe languages Java and shell.

              These sorts of things are called weird machines because they’re really weird.


              score 1 · canonical

              • lobsters__kornel@crustaceanson.theshelf.topM
                link
                fedilink
                arrow-up
                1
                ·
                edit-2
                18 days ago

                It’s really really hard to make an off-by-one error in Rust that turns it into a JIT or scripting language.

                When a language has an API for executing arbitrary code, misuse of that API leads to arbitrary code execution, duh. That still fits under “the danger is roughly limited to what the code is supposed to do”. But you’re missing the point that in C you don’t need an API for RCE get RCE. In C you have a much larger problem of memory unsafety and UB turning potentially any API into an arbitrary code execution gadget.


                score 1 · canonical

  • I’m trying to contribute to clippy now and the situation looks bleak for maintainers.

    Previously the signal was: could make a PR. Things like docs and tests were “wow this person is good” and a description that linked to the issue it closes and actually explained that it did was a 10/10.

    Mantainers have always relied on signals to determine where to invest their limited time. When they do invest it, it was clear their empathy is met and reciprocated. Now those signals are gone and there’s not a good way to distinguish between “I spent days and hours meticulously promoting fans getting reviews and iterating combined with my own human iteration” and “told Claude to get me internet points to boost my resume in a Ralph loop”. There’s not even a guarantee the person on the other side is in fact a person.

    The rust/clippy policy is very thoughtful and clearly spells out how to use LLMs with empathy for the eventual maintainer/reviewer. I ran afoul of it by some degrees and even though I’m in the “meticulous code” camp of LLM users cannot genuinely say code produced by the machine isn’t in the final product. I can though, and will. So closed it and will rewrite it.

    I think LLM scanning is good. But LLM issues and PRs are not. We need a human who has the skills and empathy for reviewers to make it easy enough for them. The challenge has never been the work, the challenge has always been doing the work in a way that is reviewable, maintainable, and sustainable. We need to find community standards and guidelines to reflect that reality.


    score 1 · canonical

    • I’m kind of disappointed that LLM review is so effective. No specification, no simulation testing, no structured method.

      Why is it disappointing to you? By all accounts, the fact that something works better than one expected, with less prerequisites than one expected, should be a welcome news?


      score 2 · canonical

      • I think it’s the bitterness of the bitter lesson. It’s an area where LLMs have already surpassed programmers.

        I suppose the interesting question is whether LLMs can find the same bugs as eg generative test suites or not, or if the result sets are at least partially disjoint.


        score 1 · canonical

        • I suppose the interesting question is whether LLMs can find the same bugs as eg generative test suites or not, or if the result sets are at least partially disjoint.

          Hm, what exactly is a “generative test suite”? Is this another way to spell “fuzzing”, or something else entirely? Google says this refers to something with LLMs (again), which I’m wary of believing as you seem to contrast the two.


          score 1 · canonical

  • The very definition of “Software quality” has shifted due to the adversarial effects of LLMs being widespread, to a degree that we now must further lionize LLMs to mitigate this shift. It feels sad that we have allowed this to happen.


    score 2 · canonical

    • The problems with the quality of our software were there all along. We merely have discovered a cost-effective way to find them (which is a good thing!).

      It feels sad that we have allowed this to happen.

      Yes, agreed. It is profoundly sad that we allowed the industry (and ourselves) to play ostrich and neglect software quality for such a long time. Thankfully, these times are over.


      score 0 · canonical

      • lobsters__ellio_6fd8@crustaceanson.theshelf.topM
        link
        fedilink
        arrow-up
        1
        ·
        edit-2
        19 days ago

        I don’t think you do agree with me.

        The problems with the quality of our software were there all along.

        You could make the same argument concerning any system designed to live in an equilibrium with the environment it was created for. I wouldn’t say cities built by the coast are poor quality, or had problems that were there all along, because they do not have flood defences, even if it is now becoming a major problem due to a shifting climate.

        I don’t mean to say that you are wrong for being glad we are finding vulnerabilities, perhaps it leads to a better place. I hope however you can understand that I feel sad about it, as I feel that the benefits we see from these things are outweighed by the negative externalities of their existence and widespread adoption, in a similar way to how the delicious wine we can now grow in my country is not worth the externalities of the climate crisis.


        score 2 · canonical

      • Thankfully, these times are over.

        Are they? For me a natural response to being forced to be honest about the broken state of most systems software would be massive rewrite efforts with better tools (memory-safe language, capability-based security, judicious use of formal methods, etc.). But what I see happening in practice is more “exhaust ourselves fixing all the LLM-found issues and then when the current stream dries out we can claim victory, without otherwise changing our programming approach”. And duh, of course, this is more incremental and the upcost front is smaller, and the result is worse, so of course everyone does it.


        score 1 · canonical

        • lobsters__intelfx@crustaceanson.theshelf.topM
          link
          fedilink
          arrow-up
          1
          ·
          edit-2
          19 days ago

          And duh, of course, this is more incremental and the upcost front is smaller, and the result is worse, so of course everyone does it.

          It is entirely well known that in every human field of endeavor there is a tradeoff between costs and benefits, and not every thing that needs doing calls for maximum effort. Unless and until someone successfully implements a post-scarcity economy, it is going to stay that way.

          Thus, something that raises the zero-effort floor (even if it doesn’t change the ceiling or the amount of effort required to attain the ceiling) is still valuable—maybe even more valuable than advancing the ivory-tower ceiling.


          score 1 · canonical

  • lobsters__motet_a@crustaceanson.theshelf.topM
    link
    fedilink
    arrow-up
    1
    ·
    edit-2
    19 days ago

    Although Rust mostly eliminates memory safety risk, any use of Cargo to download dependencies dramatically increases supply chain security risk. The risk of bundling a trojanized dependency arguably — I would even say probably — outweighs the benefit of eliminating memory safety flaws. This problem is inherent to any programming language package manager. Currently the best solution is to not use programming language package managers, but GNOME’s Rust code depends heavily on Cargo. Accordingly, I recommend against using Rust for writing GNOME software.

    I totally agree regarding the risk of using Cargo dependencies, but at the end, the reasoning is kinda flawed IMO. Why would GNOME’s official libraries for Rust be compromised? Why GNOME software would use Cargo to fetch something else?

    I agree that there is a lot of suspicious libraries on https://crates.io/. But that does not mean it’s entirely insecure. It’s a bit like saying “you shouldn’t use internet at all because there are some trojans in the wild”…


    score 4 · canonical

  • Projects that continue to prohibit AI-generated vulnerability reports are no longer suitable dependencies for GNOME, and should be developed someplace other than GNOME GitLab.

    So I have to spend all my time dealing with highly variable quality bug reports from the bullshit machine or I get kicked off the worst development platform in the world? What a tough choice.


    score 1 · canonical

    • The truth is, LLMs make mediocre developers feel powerful. In light of that fact, this is perfectly consistent behavior. If you prohibit LLM-generated ‘reports’, you’re not just making a technical choice, you’re threatening their sense of identity.

      It’s always a red flag when someone insists on calling it AI, artificial intelligence. It shows their position on what intelligence is.


      score 2 · canonical

    • That’s the part I’m most bothered about. I agree with the “Vulnerability reporting is a public service, not an obligation” part, but it’s not a binary choice, it’s not the reporter either reports with slop or doesn’t report at all.

      A better solution is that the reporter only reports the existence of the vulnerability, and the prompt they used, if they don’t feel like writing a report themselves. Then the maintainer can either check themselves, or point their agent to it if they want to. Since basically everyone has access to a frontier agent, why submit the build artifact?

      I think this works better than slop reports, because AI is becoming infectious. The reports are so impenetrable to humans, you end up having to use an agent yourself. But it doesn’t have to be this way. And if the reporter can bother to distill the report to a human-made version, all the better for everyone involved!


      score 2 · canonical

  • Rust will indeed eliminate most memory safety issues (except in unsafe blocks), and you can reasonably expect a Rust project to have an order of magnitude fewer vulnerabilities than a comparable project written in C or C++ or Vala.

    Currently the best solution is to not use programming language package managers, but GNOME’s Rust code depends heavily on Cargo. Accordingly, I recommend against using Rust for writing GNOME software.

    In my opinion the response should be to pin versions of the dependencies or fork and maintain them in the GNOME project, rather than avoiding a programming language the author claims reduces vulnerabilities by an order of magnitude.


    score 1 · canonical