Humans are bad at writing secure code, and GNOME developers are no exception. GNOME is primarily written using unsafe programming languages where simple mistakes in our code lead to devastating consequences for our users, and we make these mistakes all the time. No matter how much we try, GNOME developers will fail write secure code...
I’m kind of disappointed that LLM review is so effective. No specification, no simulation testing, no structured method.
Why is it disappointing to you? By all accounts, the fact that something works better than one expected, with less prerequisites than one expected, should be a welcome news?
I think it’s the bitterness of the bitter lesson. It’s an area where LLMs have already surpassed programmers.
I suppose the interesting question is whether LLMs can find the same bugs as eg generative test suites or not, or if the result sets are at least partially disjoint.
I suppose the interesting question is whether LLMs can find the same bugs as eg generative test suites or not, or if the result sets are at least partially disjoint.
Hm, what exactly is a “generative test suite”? Is this another way to spell “fuzzing”, or something else entirely? Google says this refers to something with LLMs (again), which I’m wary of believing as you seem to contrast the two.
I’m kind of disappointed that LLM review is so effective. No specification, no simulation testing, no structured method.
My guess is that it reveals that LLMs can really pay attention?
score 1 · canonical
Why is it disappointing to you? By all accounts, the fact that something works better than one expected, with less prerequisites than one expected, should be a welcome news?
score 2 · canonical
I think it’s the bitterness of the bitter lesson. It’s an area where LLMs have already surpassed programmers.
I suppose the interesting question is whether LLMs can find the same bugs as eg generative test suites or not, or if the result sets are at least partially disjoint.
score 1 · canonical
Hm, what exactly is a “generative test suite”? Is this another way to spell “fuzzing”, or something else entirely? Google says this refers to something with LLMs (again), which I’m wary of believing as you seem to contrast the two.
score 1 · canonical
Well it depends on your definition of fuzzing :) but sure, fuzzing, deterministic simulation testing, and the like.
score 1 · canonical