lobsters score: 2 · tags: show, unix
create programs that are highly portable, able to run on any contemporary Unix system with no runtime dependencies other than the basic utilities specified by POSIX.1-2024. (Of course, your programs may have their own runtime dependencies; that’s fine. But everything in Brut adheres to the POSIX constraint.) That means you can run your program on a wide range of hosts, from a Linux virtual machine, to a BSD server, to a Mac laptop, to a tiny single-board computer
I don’t believe in “just use POSIX shell” … Here is a recent article (not from me) that makes some good points:
I actually forgot about this example, and it’s pretty bad:
$ sh -c "echo 'a\nb'" a b$ bash -c "echo 'a\nb'" # same program has a different meaning!! a\nb # POSIX doesn't help you !!!i.e. busybox ash and dash are the default
/bin/shon Alpine and Debian respectively, and they differ a lot from bash, which is the/bin/shon other distros (I think Red Hat based systems, although I am not sure. Maybe Arch)Also I am not sure but on OS X did
/bin/shused to be bash 3 (which has many differences from bash 4 !!), and then it changed to zsh? Or do they have a different stable /bin/sh ? I suspect the former
If you really want to make shell portable in shell, you will get into crazy hacks like this: https://github.com/modernish/modernish
Modernish is a library for shell script programming which provides features like safer variable and command expansion, new language constructs for loop iteration, and much more.
i.e. It’s not even that you can’t use
echo, but you have to turn shell into a different programming language. Look at modernish code – it’s a different language, that happens to run on many shells.
As far as Oils, I would say OSH makes a large subset of bash a reasonable portable target – it is the most bash-compatible shell in the world: https://pages.oils.pub/spec-compat/2025-11-02/renamed-tmp/spec/compat/TOP.html
And YSH is a different language, which is better done in native code, at the interpreter level, than in shell (as modernish does)
score 9 · canonical
POSIX shell aka /bin/sh is bash on OS X, while it’s dash on Debian, busybox ash on Alpine, etc.
The OP makes this claim, which I’m saying is not really true in a practical sense:
That means you can run your program on a wide range of hosts, from a Linux virtual machine, to a BSD server, to a Mac laptop, to a tiny single-board computer
As an analogy, imagine if there was no such thing as JavaScript. Imagine you wrote v8 code (Chrome), or you wrote SpiderMonkey code (Mozilla), or you wrote Opera browser code.
That more accurately describes the situation with Unix shell. It is not as standardized as JavaScript, basically because there is not that much industry motivation do make it so.
And not just shell – it’s all the utilities that shell calls, which you need to get things done.
score 1 · canonical
And not just shell – it’s all the utilities that shell calls, which you need to get things done.
I think this is a very important point. Most shell interpreters are dependent on an externalised modularity—even
zshprovides very minimal builtin functionality (andzsh’s modules don’t add that much more); as a result, we are dependent on external binaries. I once wrote a simple web framework in purezshas a gimmick, and it was a gimmick (unlike in Python, where this would work have been considered very pedestrian,) simply because shell scripting is so limited for many common tasks.As a result, almost any interesting shell script (whose motivation reaches even a little bit beyond the managing processes) is going to be reliant on the surrounding environment. While POSIX does specify available “utilities”, the list isn’t very long, and there are a lot of common modalities (e.g.,
chown --reference) that are not mandated.So, sure, a very simple shell script may run anywhere, but it’s unlikely that it will be able to do anything useful, unless you bootstrap an environment and standard library around it, at which point you might as well deploy a Janet statically-linked executable.
score 2 · canonical
I understand the pain points when it comes to trying to adhere to POSIX. I don’t do it myself. I think shell differentiation is good. I think the benefits of non-standard features outweigh portability.
I think there might be something in your example that is clear to you but not to me. The last line claims that POSIX doesn’t save you but if you opt into POSIX it does, even in bash.
What I was envisioning, bash invoked as
/bin/shbehaves like the other shells you mentioned:/bin/sh --version GNU bash, version 3.2.57(1)-release (arm64-apple-darwin26) Copyright (C) 2007 Free Software Foundation, Inc. /bin/sh -c "echo 'a\nb'" a bIt’s somehow detatched from the actual POSIX opt-in behavior though because:
/bin/bash --posix -c "echo 'a\nb'" a b
score 1 · canonical
That is not what I get on my Debian machine! Further adding to the confusion :-)
$ /bin/bash --posix -c "echo 'a\nb'" a\nb$ /bin/bash --version GNU bash, version 5.2.15(1)-release (x86_64-pc-linux-gnu)$ /bin/sh -c "echo 'a\nb'" a bThat doesn’t surprise me, because POSIX does not specify which one is right. bash is POSIX-compatible and dash (
/bin/shon Debian) is POSIX-compatible, in this respect, and they are incompatible/contradictory behaviors.I think you are on OS X because your
/bin/shis the last GPLv2 version of bash, 3.2.57, which is ancient.So apparently bash 3 and bash 5 have incompatible/contradictory behaviors, and both are POSIX-compatible. POSIX doesn’t help you!
Just to prove the point more, here’s a Red Hat machine:
$ /bin/sh -c "echo 'a\nb'" a\nb$ /bin/sh --version GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu)Red Hat uses bash as
/bin/sh, and it behaves differently than Debian which uses dash as/bin/sh.i.e.
/bin/shbeing “POSIX” is a weak guarantee, not something most people want to rely upon.
score 2 · canonical
Apparently it’s not a bash version difference. It is an Apple patch in order to make make macOS into a proper unix: https://www.quora.com/What-goes-into-making-an-OS-to-be-Unix-compliant-certified/answer/Terry-Lambert
score 1 · canonical
Your link doesn’t support that. I’m saying that if you have the script
#!/bin/sh echo 'a\nb'Then you will get different behavior on Red Hat and Debian, because on Red Hat /bin/sh is bash, and on Debian it’s dash.
Are you saying that one of Red Hat and Debian is not POSIX, or not “proper Unix”?
If so, you are mistaken. POSIX allows both behaviors, which someone pointed out to me here:
https://news.ycombinator.com/item?id=48711403 - comments on “POSIX is not a Shell”, which I linked in my first comment
Looking at https://pubs.opengroup.org/onlinepubs/9699919799/utilities/echo.html#tag_20_37_05
POSIX says:
It is not possible to use echo portably across all POSIX systems unless both -n (as the first argument) and escape sequences are omitted.
So POSIX is saying that the problem is with the shell script
echo 'a\nb', not with dash or bash.The author of the shell script has to change their code to make it POSIX-compatible. It’s not dash or bash that need to change.
And Apple does not need to patch bash, for this reason.
score 1 · canonical
Are you saying that one of Red Hat and Debian is not POSIX, or not “proper Unix”?
Almost. I’m saying that the default bash behavior that Red Hat ships is:
- Irrelevant to POSIX (which I understand is part of your point).
- Is not UNIX, at least by the definition of OG
Then you will get different behavior on Red Hat and Debian, because on Red Hat /bin/sh is bash, and on Debian it’s dash.
Yes. I agree with that. Someone also pointed out in this thread that
echois not what POSIX-adherants use, for the same reason as noted in that HN thread. POSIX doesn’t help you because it doesn’t claim to help you.My comment re: the UNIX certification was only in relation to my experience on Apple. You’re right that they didn’t need to patch it to satisfy POSIX. They needed to do it to satisfy OG. It’s not that bash changed between v3 and v4 or v5. It’s that bash v3 on macOS does not work the same as bash v3 on linux.
podman run --rm -it --entrypoint /usr/local/bin/bash docker.io/bash:3 --posix -c "echo 'a\nb'" a\nb podman run --rm -it --entrypoint /usr/local/bin/bash docker.io/bash:4 --posix -c "echo 'a\nb'" a\nb podman run --rm -it --entrypoint /usr/local/bin/bash docker.io/bash:5 --posix -c "echo 'a\nb'" a\nbIt looks like their patch wasn’t so much a source patch as likely defining:
config.h.in 68:/* Define DEFAULT_ECHO_TO_XPG if you want the echo builtin to interpret 69- the backslash-escape characters by default, like the XPG Single Unix 70- Specification V2 for echo. 71- This requires that V9_ECHO be defined. */ 72:#undef DEFAULT_ECHO_TO_XPG
score 1 · canonical
So I have to use a shitty programming environment, have a mess of files on my hand that I somehow need to get into place, have to work around all the pecularities of basic posix utilities on different platforms, and in exchange I get slow execution speed with tons of processes spawned all over the place.
I am not sold on this one…
score 1 · canonical
So I have to use a shitty programming environment, write tons of boilerplate code, have a mess of files on my hand that I somehow need to get into place, have to work around all the pecularities of “basic posix utilities” on different platforms, and in exchange I get slow execution speed with tons of processes spawned all over the place.
I am not sold on this one…
score 1 · canonical
I think the end goal of this tool is not desirable (to me, at least). The point of grouping together commands like
git addandgit commitinto a single binary is not just to allow you to typegit addinstead ofgit-add. The point is that the two subcommands share a ton of logic (parsing and operating on the git file structure). So, if you wanted to compilegit-addandgit-commitas separate binaries, you would be duplicating the same code for each subcommand. By my count,githas 178 subcommands and thegitbinary is 4.7MB. So assuming that most of that binary size would be shared by different subcommands, that would be over 800MB of duplicated binaries and a rather annoying build process.
score 3 · canonical
The point is that the two subcommands share a ton of logic (parsing and operating on the git file structure). So, if you wanted to compile git-add and git-commit as separate binaries, you would be duplicating the same code for each subcommand.
Couldn’t the shared logic be a shared library?
score 1 · canonical
Yes, in principle, but that means you have to ship a shared library file in addition to your binaries and those binaries need to be configured to know where to look for the shared library. Definitely doable (easier in some languages than others, like Go, where it’s not normal), but it adds another layer of complexity. It also commits you to designing your code as a shared library/client structure, which may not be a natural structure in your program’s shared code.
Some projects like Busybox have gone to the opposite extreme of bundling hundreds of different unix programs into a single 1.3MB static executable and dispatching to the appropriate tool’s code at runtime based on which symlink invoked it.
score 1 · canonical
Thank you for sharing. I’m surprised by how little imagination we have with the possibilities of better shell scripting.
However, I am trying to figure out the motivation for this tool.
I suspect it is a consequence of shell scripting never having developed internal mechanisms for modularisation—e.g., whereas in Python, we may use
condaorpipto install packages, no such standard package manager exists for shell (instead, we have our system package manager); whereas in Python, we may write, distribute, andimportlibraries to incrementalise the complexity of our work, in shell we write separate executables; whereas in Python we may subordinate our functionality to a larger system like a web framework, there really isn’t any equivalent in shell, other than subordination to process execution mechanisms provided by the kernel or by systemd or its equivalent.Since I strongly suspect this is the case, I am skeptical that we gain much through the unification of effort around a single shell scripting framework, which is what I think this project means in describing itself as a “router.” If effort could be so readily unified, then why not unify it around a programming language which provides these missing modularisation, structuring, and organisational tooling? Surely, we already have ways to distribute software to end users without needing a package manager or compiler available—e.g., there are statically linked binaries as well as a variety of dependency isolation and packing tricks like Python
venv. If I am going to unify my effort around something that I want to deploy, which I want to work in some uniform fashion across systems, I might as well deploy a Janet or Zig executable; distributing a shell script (or a directory of shell scripts) doesn’t really confer much advantage over this, and it comes at the cost of having to write “POSIX shell.”(I am similarly skeptical that writing “POSIX shell” is something that anyone actually does—as the other comment illustrates, it’s questionable whether this thing actually exists, and, even if it did, there are very references and tools that can affirm that I have succeeded in writing such a thing. More likely, I’ll write a very constrained subset of mostly inoffensive Bash/Dash/Busybox Ash-like that bootstraps me into something more standardised that does most of the work… and wait for user complaints if this fails, at which time I may very well decline to support outliers.)
Instead, it seems like the usefulness of shell is precisely in its “messiness.” I currently have probably more than fifty thousand lines of hand-written shell across probably more than two hundred scripts that I use in my work. Some of these help with daily tasks; some for projects I will never revisit. It has taken me many years to develop an effective “style” for writing (
zsh) shell scripts, since shell is, in generally, one of the most “under-opinionated” programming languages (i.e., it provides insufficient systematic guidance to users, so while it is mostly syntactically quite simple to write shell scripts, users are on their own to discover latent or emergent patterns and structures. As a result, irrespective of whether I can defend myzshstyle, it probably comes across as extremely idiosyncratic…) I have some shell scripts that I think are written extremely well—to the same quality I could achieve in programming languages like Zig, Python, C++, Janet, &c.—and I have some shell scripts that are not. I might defend the latter by saying that those scripts did not demand the same exactitude or that they were written before I discovered systematic approaches, but it is precisely this unevenness that I think is core to shell scripting’s success. It’s “messy,” it’s “uneven,” it’s just kind of all over the place, and yet it’s probably one of the most successful programming languages ever developed, if we measure by the totality of useful efforts accomplished in its manifestations. Terrible yet terrific.I have no motivation to touch (much less rewrite) a working shell script, except to add new functionality. (Despite taking great pride in the quality of my work, conformance with an unmeasured or unmotivated standard of “elegance” has never been sufficient motivation for me to ever to anything.)
(It is notable that despite the large quantity of shell scripts that I have had to write, I have tried many times to develop common libraries and frameworks and have failed in every attempt. I have recently come to the conclusion that this organisational strategy is antithetical to effective shell scripting, because the resulting compositions are far too brittle, and the goal of shell scripting should be decomposable or disaggregated automation.)
As a result, I suspect any organisational effort that mandates, for example, precise directory structures for shell scripts are likely to be successful outside of specific operational niches (e.g., Pacman
PKGBUILD,mkinitcpio,arch-boxes, &c. all have some framework-like organisation around what are effectively shell scripts, and these seem to be mostly successful efforts.) Instead, if the genesis of this framework is to address the shortcomings of shell—namely that it lacks the internal (organisational) mechanisms that we have come to expect from high-level programming languages—then it occurs to me that that is what should be addressedThere is surprising richness in this area. For example, Linux namespaces are extremely useful but do not (logically or mechanically) compose. As a result, there is space for a lot of creativity and ingenuity to devising a compositional mechanism for namespacing for shell scripts, and the availability of such a mechanism would dramatically improve the correctness of a lot of work while also simplifying the work itself.
Similarly, there seems to have been little progress in extending the piping metaphor. Of the author’s other projects, I think
dyadis actually more interesting and potentially more useful thanbrut, but I think it’s still thinking too small.Inspired by recent work in Zig, I have come to the conclusion that effective and useful tools inevitably find themselves composed in deeper and deeper ways. If I write a very useful shell script, it will get called from another shell script which is called from a
subprocess.runin a Python script, which is triggered through a method on a class in a Python library, which is part of a Python CLI, which is called through a REST API, which is triggered by a shell script… I suspect the widespread use of LLMs will significantly amplify this compositional depth: their answer to nearly any question is “write more code.” In the case of shell scripts, I think our new compositional strategies need to address the deficiencies that arise from this extreme composition. For example, inbrut, it seems we have shell scripts calling shell scripts calling shell scripts to accomplish some task. (After all, this is what we might expect from a “router.”) I don’t think this is an effective direction for shell scripting for a variety of reasons (e.g., performance,) which is why I don’t think a routing framework is likely to be particularly useful as an organisational strategy.Ultimately, though, managing compositional complexity, designing for decomposition and disaggregation, finding new compositional strategies, and developing new compositional mechanisms is very important work.
Thank you to the author for sharing this work, and providing an opportunity to discuss these topics.
score 1 · canonical
I very much like the idea of namespacing commands, and understand where the author came from and why Brut exists.
However this makes me learn a new “language”, and way to write programs, which I don’t quite like.
I wish POSIX would make it as simple to call " subcommands" as it is on plan9 : separate programs under a directory.
git/clone $REPO ip/add $IP/$MASK dev $DEVIt “looks” like you could simply allow relative path to commands from anywhere in
$PATHrather than just from., and everyone will be happy. I don’t even think such behavior would “break” POSIX as I doubt the spec specifies how executables must be called.Edit: Damn you, POSIX !
If the pathname does not begin with a <slash>, the predecessor of the first filename of the pathname shall be taken to be either the current working directory of the process or for certain interfaces the directory identified by a file descriptor passed to the interface (such pathnames are referred to as “relative pathnames”).
– POSIX.1-2024 - General concepts, 4.16 Pathname Resolution
score 1 · canonical
git/clone $REPO
This is possible with
zshPATH_DIRS/-Q, and I would guess other shells have similar functionality.
score 1 · canonical
This is neat, but the fact that it’s not POSIX compliant means that nobody will ever develop a software that works this way, because it would rely on a non-standard shell behavior.
score 1 · canonical
it’s not POSIX compliant means that nobody will ever develop a software that works this way
As my longer comment and other comments here have stated: it’s unlikely that anyone actually cares much about POSIX compliance (and considerations of POSIX compliance affords one very little in practice.) In practice, I think POSIX usually means something between “the
bashI have installed on my LTS distribution,” “ship Busybox,” and “bootstrap to Python (or similar.)”I am not particularly fond of
bashmyself: it’s missing some obvious functionality that is necessary for correctness (e.g., we have the equivalent ofzsh${(q-)…}but no${(Q)…}; though, sadly, evenzshhas some correctness issues with round-tripping of arbitrary bytes…) As a result, I write almost exclusivelyzshscripts these days, which is fine, because all of my deployment environments havezsh. In fact, for some VM administration tasks for Arch Linux hosts, I usebashon a fresh instance to bootstrap intozsh(i.e.,pacman -Syy zsh) and then do all the work inzsh.However, I do not rely on
PATH_DIRSnot because it is not POSIX-compliant but because it introduces compositional problems, which is the case for many otherzshextensions of shell interpreter behaviour. For example,zsh${(@)path}does not have the in-band encoding ambiguity of${PATH}whereby there is no way to express a search path location with a colon in it. Therefore,path+=( abc:def )will behave as expected, but, obviously, this cannot be relied on outside of the scope of the currently running interpreter instance (or opaque optimisations where zsh avoids unnecessary forks.) I only know aboutPATH_DIRSbecause it is an important consideration when trying to determine how, for any given configuration ofzsh, how we may name a shell function so that it cannot conflict with a binary that may inevitably need to be called, which is a question we encounter along the way to discovering how to introduce usable namespacing into shell scripts.Ultimately, I don’t think
PATH_DIRSis actually that interesting. It is a very small personal convenience that is interesting almost exclusively in interactive modes. If this functionality were extended such that it were available in all shells, I still don’t think it would be that interesting. There are already many other ways to address this organisational problem.
score 1 · canonical
it’s unlikely that anyone actually cares much about POSIX compliance
I do. I run shell scripts on many systems daily, with many different shells (mksh, oksh, ksh, dash, ash, busybox, bash), and by sticking to POSIX shell scripting, I avoid the troubles to rewrite my scripts or even think about the environment I writing the script for. Even features like arrays are a big no for me, because it would break on most of my systems (mostly BSD).
Ultimately, I don’t think PATH_DIRS is actually that interesting. It is a very small personal convenience that is interesting almost exclusively in interactive modes.
The ability to “namespace” commands goes beyond interactive usage. It’s a totally different way to write and bundle software. This would allow for great standardization, and composability. For example, creating new git commamds or aliases would be as easy as adding files under the
$HOME/.local/bin/gitdirectory.
score 1 · canonical