implementations like musl’s are hacks where it’s trivial to find stuff like shell bypasses
Yes I am going to start using the term “parser differential” for this. It is often used in the context of say web firewalls trying to parse HTML incorrectly and block stuff, but I see it extremely often in relation to shell – simply because shell is hard to parse, and people write a second, crappy, and incomplete shell parser.
And even bash can’t parse shell correctly in many cases, after 30 years of trying :-) That is one reason I started https://oils.pub/
I don’t have the bugs handy, but simply parsing nested () {} $() ${} in shell is extremely difficult, and the unbalanced case 0 in *) is one reason, but not the only one.
Oh and now I remember the reason I was looking into parser differentials. I noticed that there were two cases of longstanding design bugs in the Go stdlib that can be thought of as parser differentials:
(2) The other parser differential is in contextual auto-escaping in Go’s template packages
https://nvd.nist.gov/vuln/detail/cve-2023-24538 - Templates do not properly consider backticks (`) as Javascript string delimiters, and do not escape them as expected. Backticks are used, since ES6, for JS template literals
Basically Go has a crazy HTML/CSS/JS incomplete parser / state machine, in order to do automatic escaping. But the difference between that parser and the real one can lead to missed escaping of untrusted variables, and thus a vulnerability.
If shell and HTML and other “crappy languages” weren’t already entrenched enough, then LLMs really put the nail in the coffin. We will be using shell and HTML “forever”.
But maybe we can do better with LLMs … i.e. nobody wants to do all this work to migrate toward stricter languages (e.g. OSH vs bash, which is easy in 99.9% of cases), but we really should.
I’d define parser differential as a security issue – and in that case, the bug is that wordexp() exists, and it has a flag that implies parsing shell, which it does incorrectly. (The problem is not even well-posed, because Parsing bash is undecidable, and bash is /bin/sh on Red Hat machines).
So even if everyone switched to OSH, wordexp() would STILL be bad and unsafe. That’s just fundamental.
On the other hand, It’s true there are differences between bash and OSH. Some of the differences I see as a feature, rather than a bug.
At first, OSH was indeed stricter than it should have been by default (more differences with bash). Over time, more of the differences have moved under shopt --set strict:all.
So OSH is extremely compatible with bash, but you can opt into a better behavior.
One thing we could do to reduce “parser differences” is subsume the functionality of shfmt (parser in Go) and ShellCheck (parser in Haskell). However I am only going to experiment along those lines, not promise it will be done, because it’s a huge amount of work. :-)
Probably the experiment will need to involve ways to make the problem more tractable – formatting bash is harder than formatting Python or JavaScript, simply because it has so much more syntax.
Yes I am going to start using the term “parser differential” for this. It is often used in the context of say web firewalls trying to parse HTML incorrectly and block stuff, but I see it extremely often in relation to shell – simply because shell is hard to parse, and people write a second, crappy, and incomplete shell parser.
https://nhimg.org/glossary/parser-differential/
And even bash can’t parse shell correctly in many cases, after 30 years of trying :-) That is one reason I started https://oils.pub/
I don’t have the bugs handy, but simply parsing nested
() {} $() ${}in shell is extremely difficult, and the unbalancedcase 0 in *)is one reason, but not the only one.score 1 · canonical
Oh and now I remember the reason I was looking into parser differentials. I noticed that there were two cases of longstanding design bugs in the Go stdlib that can be thought of as parser differentials:
(1) One them was a motivation for JSON v2 - https://go.dev/blog/jsonv2-exp (Sept 2025)
e.g. two parsers handling
{"admin": false, "admin": true}differentlyAn example (not involving Go) is https://nvd.nist.gov/vuln/detail/cve-2017-12635 - Due to differences in the Erlang-based JSON parser and JavaScript-based JSON parser
(2) The other parser differential is in contextual auto-escaping in Go’s template packages
https://nvd.nist.gov/vuln/detail/cve-2023-24538 - Templates do not properly consider backticks (`) as Javascript string delimiters, and do not escape them as expected. Backticks are used, since ES6, for JS template literals
Basically Go has a crazy HTML/CSS/JS incomplete parser / state machine, in order to do automatic escaping. But the difference between that parser and the real one can lead to missed escaping of untrusted variables, and thus a vulnerability.
If shell and HTML and other “crappy languages” weren’t already entrenched enough, then LLMs really put the nail in the coffin. We will be using shell and HTML “forever”.
But maybe we can do better with LLMs … i.e. nobody wants to do all this work to migrate toward stricter languages (e.g. OSH vs bash, which is easy in 99.9% of cases), but we really should.
score 1 · canonical
With bash being as widespread as it is, by doing it “correctly” in oils, aren’t you creating just another parser differential?
score 1 · canonical
I’d define parser differential as a security issue – and in that case, the bug is that
wordexp()exists, and it has a flag that implies parsing shell, which it does incorrectly. (The problem is not even well-posed, because Parsing bash is undecidable, and bash is/bin/shon Red Hat machines).So even if everyone switched to OSH,
wordexp()would STILL be bad and unsafe. That’s just fundamental.On the other hand, It’s true there are differences between bash and OSH. Some of the differences I see as a feature, rather than a bug.
The most obvious one is that OSH lacks this security issue that bash, ksh, and zsh have: https://lobste.rs/s/mla0ns/til_some_surprising_code_execution (now I see this post is also from @yossarian !)
At first, OSH was indeed stricter than it should have been by default (more differences with bash). Over time, more of the differences have moved under
shopt --set strict:all.So OSH is extremely compatible with bash, but you can opt into a better behavior.
One thing we could do to reduce “parser differences” is subsume the functionality of
shfmt(parser in Go) and ShellCheck (parser in Haskell). However I am only going to experiment along those lines, not promise it will be done, because it’s a huge amount of work. :-)Probably the experiment will need to involve ways to make the problem more tractable – formatting bash is harder than formatting Python or JavaScript, simply because it has so much more syntax.
score 1 · canonical