| View previous topic :: View next topic |
| Author |
Message |
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
 Posted: Fri Sep 26, 2014 3:37 pm Post subject: Bash bashing (Rant) |
 |
|
This is not a rant about the bashshock bug(s) - such mistakes can happen - but about the behaviour of bash over the previous years which eventually culminated in making such a bug possible due to bad concepts; in fact, this rant is bugging me since years, and now I just have to release it. 
Actually, I see a lot of parallels between the behaviour of bash and systemd:
Both have been pushed to being a "standard" (although admittedly with a different level of aggression), and once this had happened, they both started to misuse their power to break existing and working stuff.
To avoid a misunderstanding: I am fine with bash extensions like arrays or <(...): They do no collide with existing syntax and really bring features which cannot be obtained easily by other means. (However, they are all stolen from zsh which had implemented all of these much earlier, and most in a better way.)
The extensions (( ... )) or [[ ... ]] are already not so nice, since their functionality is already available in POSIX - just with a different syntax. This is bad, not because it breaks orthogonality (which IMHO would be a questionable aim), but because it encourages people to write stuff in an imcompatible way when they could use the compatible (POSIX) way without any drawbacks.
The regular expression test is an exception of this remark: Since standard shell does not provide regular expressions, it is IMHO fine to use [[ ... ]] for this if it is really needed, but it should then be used only for this to keep compatibility as far as possible.
I know that the above incompatibilities have historical reasons, and so one cannot hold it against bash that this syntax is supported. However, one can hold it against bash that the incompatible syntax is still not heavily discouraged in the manpage.
Some years ago, one could see what comes from this nonsense when all of a sudden bash decided to treat [[ ... == .... ]] differently, depending on whether the right-hand side is quoted: A lot of previous bash scripts just broke. This dramatic change did not even get a big announcement. (It was announced as a "bugfix"...)
The &> is similar to (( ... )):
I can understand that bash wants to keep it for historical reasons, but why explicitly encourage users in the manpage to use this syntax which only saves a few keystrokes and which in a real POSIX shell breaks silently with unexpected behaviour? Moreover, why does bash not treat it as any POSIX shell if it is started in "POSIX" compatbility mode?
The latest and biggest problem I am aware of is this function exporting which also happened without a big announcement: bash had announced since several years that some day they will introduce a mechanism to export functions to subprocesses. Of course, everybody was hoping that they will hold it back until they really find a working solution.
A working solution might have been to do the export over some pipe/socket mechanism. Or, if it really should be done over the envionment - which IMHO is a bad idea to begin with - to use one dedicated variable containing all exported functions (or at least their names if on some systems the variable length should be an issue). And, of course, I would have expected that this mechanism is then switched off by default and must be activated with some new command or by setting a new shell option, at least in non-interactive shells: There are very good reasons why "alias" does not work in non-interactive shells by default. For the much more dangerous thing of functions, bash suddenly forgets this earlier learnt lesson.
And not only that this dangerous mechanism which is completely against POSIX is switched on, by default (even if started in POSIX mode as /bin/sh), it even breaks a lot of working scripts in an unexpected way:
Environment variables can no longer be used to transport information to subprocesses, since certain values now change everything. Commands can change all of a sudden, etc.
Here is a very primitive example:
| Code: | | myscript="$1" find . -exec 'myscript "$myscript"' sh '{}' '+' |
If "myscript" is a reliable script which decently checks the validity of its argument, everything looks completely safe. And actually it is - in POSIX and older bash versions. Not so in bash after the "great" function export feature. In fact, by just setting $1 appropriately, this code snippet can execute practically everything.
OK, this is because "by accident" this code snipped relied on the previous truth that variable and command names belong to separate namespaces - a natural rule which bash decided to break for no valid reason.
So let's assume you go through all existing scripts and rename variables so that they never coincide with any command. So you can be sure that you never fall on this new bash trap:
| Code: | | myscript_="$1" find . -exec 'myscript "$myscript_"' sh "{}" + |
Now this should work reliable, shouldn't it?
In bash, it does not: Just give $1 the "wrong" magic value, and myscript will falsely be called with an empty argument.
Also e.g. the environment value | Code: | | export echo='() { rm -rf ~/*; }' |
will give you a sad experience with a lot of scripts, even without using any vulnerability.
There is only one answer to this: Replace /bin/sh with a symlink to a sane shell (e.g. dash), and do not use bash anymore.
Fortunately, many distributions already support it, and also gentoo does (although there are still some broken openrc scripts around, even in the gentoo tree, which still use bashisms).
Unfortunately, due to a very questionable (IMHO) gentoo decision, ebuilds require bash. But at least on my systems, this is more or less the only place where this broken shell is still in use...
(Of course, there are still some questionable uses of bash by some upstream projects...)
BTW: bash is also the slowest shell around; especially when subshells are spawned, the speed is a nightmare.
Edit: It seems that some of the issues are going to be fixed in future releases of bash, but not properly. For instance to get the bad experience, you will have to replace "echo" by BASH_FUNCTION_echo": At least the namespace collission is partially omitted, but that this is on by default is still a severe bug in the concept IMHO. |
|
| Back to top |
|
 |
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Fri Sep 26, 2014 4:48 pm Post subject: Re: Bash bashing (Rant) |
|
|
I'm just going to answer a couple of preliminary points, then I'll come back to this topic when I've got a bit more time.
| mv wrote: | | Actually, I see a lot of parallels between the behaviour of bash and systemd: |
This is utter nonsense.
| Quote: | | Both have been pushed to being a "standard" (although admittedly with a different level of aggression), and once this had happened, they both started to misuse their power to break existing and working stuff. |
Systemd has always misused its position, as has every other "app" released by that idiot.
Chet is nothing like Poettering. You're just so wrong about this, it's staggering.
| Quote: | | To avoid a misunderstanding: I am fine with bash extensions like arrays or <(...): They do no collide with existing syntax and really bring features which cannot be obtained easily by other means. (However, they are all stolen from zsh which had implemented all of these much earlier, and most in a better way.) |
Actually most of the bash extensions to sh are stolen from ksh.
| Quote: | | The extensions (( ... )) or [[ ... ]] are already not so nice, since their functionality is already available in POSIX - just with a different syntax. This is bad, not because it breaks orthogonality (which IMHO would be a questionable aim), but because it encourages people to write stuff in an imcompatible way when they could use the compatible (POSIX) way without any drawbacks. |
Again, ksh so you should be ranting about how awful ksh93 and mksh are, but clearly you've never even explored them.
You're wrong about [[ .. ]]; it's very much better than [ .. ], which has drawbacks you appear not to be aware of.
| Quote: | | The regular expression test is an exception of this remark: Since standard shell does not provide regular expressions, it is IMHO fine to use [[ ... ]] for this if it is really needed, but it should then be used only for this to keep compatibility as far as possible. |
Lul; you should know what you're talking about before you start making sweeping recommendations.
I agree the function in envvar thing is a terrible idea. Never liked it when I read about the possibility in the POSIX sh documentation (which I've linked you to several times in the past), though I'd never seen the syntax of an implementation allowing it. It necessitates some slightly tricky setup in configure scripts, but then we'd do the same setup irrespective of what bash does, since the same possibility exists everywhere (that functions are in effect before our script is started.) |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Fri Sep 26, 2014 6:02 pm Post subject: Re: Bash bashing (Rant) |
|
|
Please do not misundarstand my rant as an attack against some people. It is not meant so.
| steveL wrote: | | Chet is nothing like Poettering. |
I did not compare people. I did not claim that the bash maintainers wanted to press it through as a "standard".
I just realize that now that it has become quasi-standard, it is not to the best, due to misuse of power: Bad features are added and/or recommended.
| Quote: | | Actually most of the bash extensions to sh are stolen from ksh |
This is hard to decide, and I won't take this red herring: zsh had arrays, and some years later, bash had them, too, with essentially the same syntax. Wether it was originally ksh which both stole it from: maybe. It really plays no role. In free software, it is not a crime to steal a good idea.
| Quote: | | Again, ksh so you should be ranting about how awful ksh93 and mksh are, but clearly you've never even explored them. |
These shells and their recommendations never caused trouble on my systems. Bash did. The reason is simple: bash is the "standard", and people follow recommendations in its manpage. I know that this is unfair, but the "standard" simply has an increased responsibility.
| Quote: | | You're wrong about [[ .. ]]; it's very much better than [ .. ] |
No. It is nothing but syntactic sugar to save you from some quoting or splitting tests. There is a reason that it was considered for POSIX but decided to drop due to the lack of any advantage.
| Quote: | | you should know what you're talking about before you start making sweeping recommendations. |
Yep, that's why I very carefuly checked the POSIX standard concerning "test", and as long as you use "test" only for a single test and split combined tests, there is nothing which can go wrong with it. Of course, you should not use the non-POSIX and non-recommended -a and -o which can cause horrible breakage. |
|
| Back to top |
|
|
Dr.Willy
Guru
Joined: 15 Jul 2007
Posts: 547
Location: NRW, Germany
|
| Posted: Sat Sep 27, 2014 2:32 pm Post subject: |
|
|
To be honest I've never been quite happy with any of the bourne-shell derivates.
As far as I can see there are two use cases that a language needs to be good at to be a good shell:
1) interactive use from the command-line
2) small glue-code scripts
Now bourne's original syntax has many shortcomings. So zsh, ksh and bash all make different language extensions to it to fix these shortcomings.
However, this doesn't really solve anything: When you want to distribute a script you can't really expect anyone to install your favored shell just to run your script. Thus you probably want to stick to POSIX-sh. So while you have these extensions theoretically available, you don't really get to use them.
On the other hand for interactive use, there is no point in being compatible to POSIX-sh in the first place. Nobody but you is going to see, let alone run these commands. Who cares which, if any, standard they follow? So if you are to develop a new shell, why would you copy over every single bit of a syntax that was created in the 70's?
Just use a small shell like dash for /bin/sh and see that your shell shines with good interactive use. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Sat Sep 27, 2014 4:24 pm Post subject: |
|
|
Dr.Willy, I completely agree with you. (Although I would distinguish between "bourne's original shell" and POSIX; the former is so buggy that practically it cannot be used).
The problem is that practice just shows that one gets many files which falsely rely on bash: There are many examples of ./configure or also Makeilfes which rely on bash features, or many other things (one particular example I remember well is fvwm-crystal conatining many bashisms for a long time, until it was fixed). So /bin/sh being a symlink to non-bash is risky and probably always will be. This is actually the reason why I was so angry about bash: I had submitted so many fixes to so many projects - and yet .... just a few days ago, I emerged libcgroup and found it completely breaking with openrc+dash, since its init-file is full of bashisms.
I am really fed up with fixing this kind of stuff for other people, so I hope that some of these people read my rant and maybe change their mind (although admittedly chances are not very big for this). |
|
| Back to top |
|
|
Dr.Willy
Guru
Joined: 15 Jul 2007
Posts: 547
Location: NRW, Germany
|
| Posted: Sat Sep 27, 2014 6:21 pm Post subject: |
|
|
| mv wrote: | | Dr.Willy, I completely agree with you. (Although I would distinguish between "bourne's original shell" and POSIX; the former is so buggy that practically it cannot be used). |
Oh you mean bourne's original interpreter implementation. Well yes, that is thankfully long dead.
What I meant is that bourne's original language/syntax will continue to exists in POSIX, no matter how horrible it is (`backticks` anyone?)
| mv wrote: | | The problem is that practice just shows that one gets many files which falsely rely on bash: There are many examples of ./configure or also Makeilfes which rely on bash features, or many other things (one particular example I remember well is fvwm-crystal conatining many bashisms for a long time, until it was fixed). So /bin/sh being a symlink to non-bash is risky and probably always will be. |
Maybe. After all, people eventually figured out that optimizing websites for IE6 was a terrible idea, so I'd say there is hope. |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Sat Sep 27, 2014 7:30 pm Post subject: Re: Bash bashing (Rant) |
|
|
| mv wrote: | | steveL wrote: | | Chet is nothing like Poettering. |
I did not compare people. I did not claim that the bash maintainers wanted to press it through as a "standard". |
But he is the (only) bash maintainer.
| Quote: | | I just realize that now that it has become quasi-standard, it is not to the best, due to misuse of power: Bad features are added and/or recommended. |
In fact it's not Chet who's pushed it like this, but the dumbass Linux distros who wrote such crappy shell-scripts for so long (usually feeling proud of having read TLDP/ABS, which we tell you never to read in #bash unless you want to learn how to write buggy scripts) and then turned around and let themselves be manipulated into using systemd instead of confronting the actual problem: they don't have competent scripters.
| Quote: | | Quote: | | Actually most of the bash extensions to sh are stolen from ksh |
This is hard to decide, and I won't take this red herring: zsh had arrays, and some years later, bash had them, too, with essentially the same syntax. Wether it was originally ksh which both stole it from: maybe. It really plays no role. In free software, it is not a crime to steal a good idea. |
I was just answering your point; it's your herring, I don't like the smell either.. ;p
In any event you disparage several features which actually originate in ksh, and along the way namecheck zsh, so it's natural for someone who knows the background to pick you up on it; but the main point is that actually those features are useful.
| Quote: | | Quote: | | Again, ksh so you should be ranting about how awful ksh93 and mksh are, but clearly you've never even explored them. |
These shells and their recommendations never caused trouble on my systems. Bash did. The reason is simple: bash is the "standard", and people follow recommendations in its manpage. I know that this is unfair, but the "standard" simply has an increased responsibility.
|
No, it's incumbent on the user and admin community around that freely-developed package to bear that responsibility; that distros didn't do their job is not the fault of the voluntary developer, whom I have a lot of respect for, only increased by his response to shellshock.
Some nub was on the news the other night, making a fool of himself by stating that that's the problem with free software; it's not "professionally" produced, so there aren't the same quality-checks. lol. Anyone who's used proprietary software knows that bugs are left baked in for all time; whereas Ramey put out patched releases for every version going back 15 years.
| Quote: | | Quote: | | You're wrong about [[ .. ]]; it's very much better than [ .. ] |
No. It is nothing but syntactic sugar to save you from some quoting or splitting tests. |
Which actually makes execution quicker, as does ((..)), since tokenisation is sufficient; we don't have to account for evaluated flags and field-splitting making 10 args out of one parameter expansion (which is usually a buggy script, lacking quoting in [, in actual usage.)
| Quote: | | There is a reason that it was considered for POSIX but decided to drop due to the lack of any advantage. |
In fact it's being standardised now (and will be in the next issue). Historically there were disagreements over the semantics of things like -nt, for example, iirc, and the fact that it's a different lex/parse state, but then shell has lots of those (overlapping) anyway. Awk only has one (telling the lexer that a slash starts a regex.) |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Sat Sep 27, 2014 7:43 pm Post subject: |
|
|
| Dr.Willy wrote: | | What I meant is that bourne's original language/syntax will continue to exists in POSIX, no matter how horrible it is (`backticks` anyone?) |
POSIX standardised $(..) at least by 1996. Anyone still using backticks has likely been brainwashed by autohell.
I agree that sh scripts should conform to POSIX sh, fwtw; although I'm fine about using local, too, so long as you bear in mind the portability caveats. And indeed tweaking behaviour for *ksh and zsh (which can be a pita, but that's life.) |
|
| Back to top |
|
|
Dr.Willy
Guru
Joined: 15 Jul 2007
Posts: 547
Location: NRW, Germany
|
| Posted: Sat Sep 27, 2014 8:17 pm Post subject: |
|
|
| steveL wrote: | | Dr.Willy wrote: | | What I meant is that bourne's original language/syntax will continue to exists in POSIX, no matter how horrible it is (`backticks` anyone?) |
POSIX standardised $(..) at least by 1996. Anyone still using backticks has likely been brainwashed by autohell. |
That is true, but even so the features still exist. So even things that were a bad idea in 1977 are still being implemented in 2014. |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Sun Sep 28, 2014 1:51 am Post subject: |
|
|
| Dr.Willy wrote: | | steveL wrote: | | POSIX standardised $(..) at least by 1996. Anyone still using backticks has likely been brainwashed by autohell. |
That is true, but even so the features still exist. So even things that were a bad idea in 1977 are still being implemented in 2014. |
Yeah there's a thing called backward-compatibility; doesn't mean you should use backticks yourself, any more than you should use signal() nowadays. |
|
| Back to top |
|
|
Ottre
Tux's lil' helper
Joined: 23 Dec 2012
Posts: 129
|
| Posted: Sun Sep 28, 2014 1:58 am Post subject: Re: Bash bashing (Rant) |
|
|
| mv wrote: |
There is only one answer to this: Replace /bin/sh with a symlink to a sane shell (e.g. dash), and do not use bash anymore.
|
Keep in mind that dash isn't GPL-licensed software. Since the shell is a central part of the Operating System, your primary consideration should be license, not features or speed. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Sun Sep 28, 2014 12:45 pm Post subject: Re: Bash bashing (Rant) |
|
|
| Ottre wrote: | | Keep in mind that dash isn't GPL-licensed software. |
It is BSD licensed which gives you even more freedom (by not forcing you to missionizing the license). At least, this is how I understand it (being a non-lawyer but trying to read the intention and reasons why some projects moved from GPL to BSD). |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Sun Sep 28, 2014 12:58 pm Post subject: Re: Bash bashing (Rant) |
|
|
| steveL wrote: | | In fact it's not Chet who's pushed it like this, but the dumbass Linux distros |
That's why I emphasize once more that my rant was not directed at certain people: I did not mention any people, because I did not mean particular people. I mean the situation which I consider being bad, and I would be glad if my rant can help to convince at least a few people to program in a more standard manner (or at keast clearly declaring their code as bash and taking means that it is executed by bash).
| Quote: | | that distros didn't do their job is not the fault of the voluntary developer |
Again, I am not looking for some "guilty" person but would prefer if the situation could just be fixed. However, bashing only the distributions as you suggest now is not correct either: the manpage is not the fault of any distribution, and e.g. the recommendation to use &> in it is clearly a recommendation against the standard. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Sun Sep 28, 2014 1:06 pm Post subject: |
|
|
| Quote: | | Which actually makes execution quicker, as does ((..)), since tokenisation is sufficient; |
Speed is a different argument than usefulness. However, the difference of speed due to this feature is negligible, probably below what you can even reasonably benchmark.
To be honest, I also disliked it when I heard that [[ ]] was rejected by POSIX, because it is more elegant than test, of course, but I can understand their argument that it is not something which is really needed. |
|
| Back to top |
|
|
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
| Posted: Sun Sep 28, 2014 4:24 pm Post subject: Re: Bash bashing (Rant) |
|
|
| Ottre wrote: | | mv wrote: |
There is only one answer to this: Replace /bin/sh with a symlink to a sane shell (e.g. dash), and do not use bash anymore.
|
Keep in mind that dash isn't GPL-licensed software. Since the shell is a central part of the Operating System, your primary consideration should be license, not features or speed. |
Go use HURD then, see what worshipping a legal text gets you. Most of us are here for an OS that works. |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Sun Sep 28, 2014 4:32 pm Post subject: |
|
|
| mv wrote: | | Quote: | | Which actually makes execution quicker, as does ((..)), since tokenisation is sufficient; |
Speed is a different argument than usefulness. However, the difference of speed due to this feature is negligible, probably below what you can even reasonably benchmark. |
Untrue: I tested all these things after I'd got past learning the basics of BASH. I see it as part of learning a language properly: first you learn the basics, understand the underlying metaphor, and then you learn the idioms. Then you explore.
For instance, one thing I found which no-one who taught me knew at the time, was that ((..)) is "slippery": you can assign variables formulae such as b='c + d' and they will be evaluated, as many times as necessary, to get the result, only throwing an error if there is an invalid number somewhere in the mix (or a divide-by-zero). And ofc empty vars are treated as zero, which is why you see an explanation to that effect in libIgli/update ("this is how we do C-style flags".)
I found this out because I took the time to explore the semantic of the construct, just as with [[
| Quote: | | To be honest, I also disliked it when I heard that [[ ]] was rejected by POSIX, because it is more elegant than test, of course, but I can understand their argument that it is not something which is really needed. |
And yet here it is, on the horizon for standardisation as pointed out.
You cannot criticise a language if you don't grok it. That's actually a rule in #friendly-coders; by all means vent but only if your diatribe is based on at least 3 months of actually working with something (more than just tiddling around on the weekend) and you provide insight with your venom.
Sorry but not reading anything like that here. I agree nubs standardising bash without ever being able to write it properly wasn't such a great thing, fwtw. I still wouldn't want to implement ebuild, eclasses et al in shell.
Further I think if people had gone for zsh, in some hypothetical scenario, nowadays you'd be ranting about all the crappy zsh-isms rampant in linux distros, and how the glory days of Bourne SH have been denied (but look at that eval..;)
The standardisation on one shell was never the problem; the complete inability to write shell was, and still is; coupled with the ignorant snobbery of non-CS graduates coming out of "IT" departments (or more honestly "Business and Marketing" depts) all over the world. You still see it in Gentoo, as well as other distros; "developers" who make a point of being incompetent at shell, and snortingly laugh at their incompetence instead of growing up and learning the basics. All they're doing is proving the adage; those who don't understand Unix, are bound to reinvent it: badly. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Mon Sep 29, 2014 9:46 am Post subject: |
|
|
| steveL wrote: | | Further I think if people had gone for zsh, in some hypothetical scenario, nowadays you'd be ranting about all the crappy zsh-isms rampant in linux distros |
That's rather likely, since the main problem is incmpatibility. I am fine if somebody is writing bash code in a #!/bin/bash script, but the problems do arise if it is used in a Makefile, etc. And actually, I do not understand the reasons why anybody would write #!/bin/bash: If one has a task which is so complex that one needs the additional features, I cannot understand why one does not use instead a more powerful language like zsh/perl/whatever.
Of course, there are exceptions, e.g. if one substantially needs <(...) but otherwise the script is rather trivial. But even in such a case, I would write all of the rest in POSIX to simplify copying parts into/from other scripts if needed.
That's also the reason why I have written only very few zsh scripts, although zsh contains a strict superset of bash features: If I need such features, I immediately use a language like perl which is even more powerful. I use zsh scripts essentially only for programmable completetion - there is no interactive shell yet which is closely comparable with its features.
Well, also to this rule, there was one exception, recently: I used zsh for some scripts in which I make assignments to USERNAME to drop permssions; it's a pity that there is nothing comparable in POSIX/bash since all workarounds with su/sudo for dropping permissions have serious drawbacks (e.g. no inheritance of current variables).
| Quote: | | and how the glory days of Bourne SH have been denied (but look at that eval..;) |
Are you seriously telling this to me who just a few posts ago has written that the Bourne SH is so buggy that it is practically unusable?
I am talking about the standard on which one should rely. And whether one likes it or not, the standard is called POSIX. |
|
| Back to top |
|
|
mirabilos
n00b
Joined: 01 Oct 2010
Posts: 3
Location: probably looking for a geocache somewhere, or a terracache, or a dashpoint…
|
| Posted: Thu Oct 09, 2014 8:37 pm Post subject: Re: Bash bashing (Rant) |
|
|
Hi there! mksh developer here, and a regular in an IRC channel with igli, who wanted me to comment here.
| mv wrote: | | The extensions (( ... )) or [[ ... ]] are already not so nice, since their functionality is already available in POSIX - just with a different syntax. |
I disagree here, because of several things:
• arithmetics are not in POSIX – $((…)) is since 2008, but that’s very recent, and still incomplete
• [[…]] comparisons are more secure, and the globbing feature (which ksh have always done) is really useful, especially as it works more naturally than a case block
• when a shell has [[…]] you have a certain minimum standard you *know* is there; otherwise you *have* to resort to things like “if test x"$foo" = x"bar"” despite POSIX not requiring those ‘x’ any more, because there’s just too many things predating POSIX, or plain buggy, around (additionally, POSIX changes over the years, so you cannot depend on the latest POSIX anyway)
| mv wrote: | | This is bad, not because it breaks orthogonality (which IMHO would be a questionable aim), but because it encourages people to write stuff in an imcompatible way when they could use the compatible (POSIX) way without any drawbacks. |
Agreed – except for the “without any drawbacks” part – except people should only do that when targetting bash, ksh, mksh, etc. explicitly.
Maybe I should really make a posh-like fork of mksh, which does not expose all these additions, for use as /bin/sh, while still keeping mksh’s decent codebase and all that. Hrm. Time…
| mv wrote: | | Some years ago, one could see what comes from this nonsense when all of a sudden bash decided to treat [[ ... == .... ]] differently, depending on whether the right-hand side is quoted: A lot of previous bash scripts just broke. This dramatic change did not even get a big announcement. (It was announced as a "bugfix"...) |
Probably – because Korn Shell (where the [[…]] syntax originated) have always done that (I just verified this on a ksh88 on Solaris 8 on stinky.trash.net where I have a shell on).
By the way: neither test/[…] nor [[…]] normally take a doubled equals sign (though ksh93 introduced it, mksh now accepts it, and I got told in #ksh that ksh93 even prefers it, it’s still bad mksh style *and* makes people try to use it with test/[…] as well).
| mv wrote: | | The &> is similar to (( ... )) |
Absolutely not! It’s a disgusting bashism that breaks POSIX syntax and leads to lots of problems. It’s just so widespread now that mksh, intending to make switching from GNU bash somewhat easy, has to implement it and disables it in POSIX mode. (Well documented in the manpage, of course.)
| mv wrote: | | I can understand that bash wants to keep it for historical reasons, but why explicitly encourage users in the manpage to use this syntax which only saves a few keystrokes and which in a real POSIX shell breaks silently with unexpected behaviour? Moreover, why does bash not treat it as any POSIX shell if it is started in "POSIX" compatbility mode? |
Agreed. The mksh manpage and us people in #ksh and #!/bin/mksh on IRC discourage it actively.
| mv wrote: | | The latest and biggest problem I am aware of is this function exporting |
Yeah. I never got that. Especially, I never got why GNU bash uses the *value* of the variable to determine the type. The original POSIX spec used env 'x()={…}', but the bash devs from back then said that breaks too many traditional/ancient shells and other utilities. That should have been a warning sign.
| mv wrote: | Also e.g. the environment value | Code: | | export echo='() { rm -rf ~/*; }' |
will give you a sad experience with a lot of scripts, even without using any vulnerability. |
Exactly. But setting both “builtin” and “unset” is more fun, because you can’t even get out of it.
| mv wrote: | | BTW: bash is also the slowest shell around; especially when subshells are spawned, the speed is a nightmare. |
Right. Some of my users regularily repeat speedups that are, on average, a factor of 3, by replacing GNU bash with mksh. Even more to gain by choosing a sane libc and/or linking it statically. That being said, AT&T ksh93 is said to use some sort of bytecode compile, and is therefore faster on repeated runs within one process (loops), and possibly can freeze the bytecode (like .java → .class) and run that.
| steveL wrote: | | For instance, one thing I found which no-one who taught me knew at the time, was that ((..)) is "slippery": you can assign variables formulae such as b='c + d' and they will be evaluated, as many times as necessary, to get the result, only throwing an error if there is an invalid number somewhere in the mix (or a divide-by-zero). |
Indeed. But you have to be careful with that, yet it’s a damn useful feature.
| steveL wrote: | | And ofc empty vars are treated as zero, which is why you see an explanation to that effect in libIgli/update ("this is how we do C-style flags".) |
Actually, we do C-style flags by initialising variables explicitly to 0, setting them to 1, and using “(( foo ))” and “(( !foo ))” to check. Initalising them to empty breaks semantics. Not initialising them leaves you wide open to attacks from the environment. (It took some time for me to persuade Stéphane Chazelas of what I wrote in this paragraph and the one before, too.)
| mv wrote: | | Quote: | | and how the glory days of Bourne SH have been denied (but look at that eval..;) |
Are you seriously telling this to me who just a few posts ago has written that the Bourne SH is so buggy that it is practically unusable?
I am talking about the standard on which one should rely. And whether one likes it or not, the standard is called POSIX. |
Bourne shell: “echo `x` ^ tee y”
POSIX shell: “echo $(x) | tee y”
These are *very* distinct. Please do not confuse them. (Heirloom sh comes close.)
But yeah, eval… ksh93 introduced “nameref” which makes all previously-valid uses of “eval” in my scripts superseded with a better alternative; I added it to mksh, too.
Just one more reason why I believe people should *not* code for sh, but for one shell (ideally mksh):
• keep the shell portable, not the scripts (only really works with mksh because it takes great pains to behave the same on all platforms, all CPUs, etc.)
• less dependency on external tools (less awk (sorry igli), cut, sed, tr, etc.) also means more portability and less fork’n’exec
• advanced shell functions make shell programming (not just scripting) somewhat easier and safer (but, like with Perl or Python, you still have got to learn the intrinsics of the language before)
_________________
--
MirBSD Founder • mksh Founder
FreeWRT Core Developer
Debian Developer, systemd opponent
… and s̲o̲ much more, just ask ☺ |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Fri Oct 10, 2014 12:21 am Post subject: Re: Bash bashing (Rant) |
|
|
| mirabilos wrote: | | mv wrote: | | The extensions (( ... )) or [[ ... ]] are already not so nice, since their functionality is already available in POSIX - just with a different syntax. |
I disagree here, because of several things:
• arithmetics are not in POSIX – $((…)) is since 2008, but that’s very recent, and still incomplete |
There is nothing you can do with (( ... )) which you can't do with $(( ... )); you just perhaps need a few more characters. The same holds for [[ ... ]] vs. [ ].
| Quote: | | • [[…]] comparisons are more secure |
This is like saying "if A; then B; fi" is more secure than "! A || B" or vice versa: It is just wrong. If you make no mistakes, everything is secure. That there are common mistakes which poor programmers make just means that the documentation has to be better and has nothing to do with "secure".
| Quote: | | and the globbing feature (which ksh have always done) is really useful, especially as it works more naturally than a case block |
case blocks can do the same thing, and they are a documented standard.
| Quote: | | • when a shell has [[…]] you have a certain minimum standard you *know* is there |
When a shell has [[ ... ]] it also supports at least POSIX [ ... ]
| Quote: | | otherwise you *have* to resort to things like “if test x"$foo" = x"bar"” |
Your argument is faulty: You claim that if you do not use non-POSIX features, you have to work around all bugs of all shells which ever existed.
POSIX was created so that you can (and should) rely on certain things to behave. So the right thing is to do exactly that: Use the POSIX features and rely that they work as documented. Shells which do not satisfy the standard should be fixed sooner or later.
| Quote: | | (additionally, POSIX changes over the years, so you cannot depend on the latest POSIX anyway) |
Of course, it all depends on what platforms you do want to support. Anyway, you are probably better off with using current POSIX features than using non-POSIX features right from the start.
| Quote: | | By the way: neither test/[…] nor [[…]] normally take a doubled equals sign |
Yes, I forgot to mention this in my first post: This is one of the frequent syntax mistakes done in init or make scripts, perhaps even more frequent than [[ ... ]] and &>
At least, the bash manpage recommends to use = in test for POSIX compatibility.
| Quote: | | mv wrote: | | The &> is similar to (( ... )) |
Absolutely not! It’s a disgusting bashism |
I had always heard that bash had taken it from another shell (wasn't it ksh?)? This other shell supported originally only &> and not >&, and this is the reason why bash recommends the former (although with POSIX it is the worse recommendation).
| Quote: | | Actually, we do C-style flags by initialising variables explicitly to 0, setting them to 1, and using “(( foo ))” and “(( !foo ))” to check. |
I do not like this because of unnecessary (( ... )) incompatibility. It is more natural, and perhaps even quicker in execution, to define variables as "false" or "true" (or even ":" since this is guaranteed to be a built-in) and to use the variables as commands.
| Quote: | | Just one more reason why I believe people should *not* code for sh, but for one shell (ideally mksh): |
I can understand your favour of mksh but I still think that if you have something non-trivial to do, you should use a real high-level language like perl or python.
| Quote: | | • keep the shell portable, not the scripts |
Not having portable scripts means you cannot use them (or snippet's thereof) for init-scripts, configure, make, ... Thus, it is better to have POSIX scripts in any case.
| Quote: | | • less dependency on external tools (less awk (sorry igli), cut, sed, tr, etc.) also means more portability and less fork’n’exec |
You can mainly avoid these already with POSIX. For instance, eix-functions.sh contains a "Replace" (local and global) using only the POSIX ${ % } ${ ## } etc., push.sh can treat arrays of strings in a safe manner by using similar tricks. This is not nice code, but it is possible. If you need something more complex you are probably already using the wrong language: switch to perl or python.
| Quote: | | • advanced shell functions make shell programming (not just scripting) somewhat easier and safer |
If you are using a few-lines script, because there is exactly a function for what you need, this is fine. Otherwise, you are probably better off with perl or python which have huge libraries and probably can do what you need in a much simpler way.
In fact, what Diego has written in his blog concerning shellshock expresses also my opinion rather well: It is a mistake that gentoo encouraged too much using shell/bash as a programming language. |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Fri Oct 10, 2014 2:22 am Post subject: Re: Bash bashing (Rant) |
|
|
| mirabilos wrote: | | • arithmetics are not in POSIX – $((…)) is since 2008, but that’s very recent, and still incomplete |
As I pointed out to mira on IRC, this is incorrect; $((..)) has been in POSIX since at least 1996, as has $(..).
| Quote: | | steveL wrote: | | And ofc empty vars are treated as zero, which is why you see an explanation to that effect in libIgli/update ("this is how we do C-style flags".) |
Actually, we do C-style flags by initialising variables explicitly to 0, setting them to 1, and using “(( foo ))” and “(( !foo ))” to check. Initalising them to empty breaks semantics. Not initialising them leaves you wide open to attacks from the environment. (It took some time for me to persuade Stéphane Chazelas of what I wrote in this paragraph and the one before, too.) |
Lol.
What about the fact that it's arithmetic context?
Anyhow, that's from 2007 or 2008, and it's only for flags which don't mean very much. iirc practically every single one of them is set by the script, and mentioned in the config docs (update -h config) if the user can override it from /etc/update.
Further that script is explicitly designed to be run as root; it will bail out otherwise. So afaic you should know what you're doing if you're running it, since it means you're a Gentoo user, operating as root.
Hmm I see your point: in arithmetic context things are expanded, so implicit $(..) will execute. Bugger. I'll verify they're all set by the script; pretty sure they are, but I'll verify.
| Quote: | | • less dependency on external tools (less awk (sorry igli), cut, sed, tr, etc.) also means more portability and less fork’n’exec |
Lol; doesn't bother me. I've told mv the same thing when he's been banging on about how bad shell is. You can write crap in any language, especially when you don't know it idiomatically. Even more so when you don't grok the underlying metaphor.
perl is hardly forgiving, very large and a total pita to bootstrap. I wouldn't care, but for the fact that it's used by autohell, to compensate for m4, instead of just using clean sh for the lower layer. |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Fri Oct 10, 2014 2:29 am Post subject: Re: Bash bashing (Rant) |
|
|
| mv wrote: | | In fact, what Diego has written in his blog concerning shellshock expresses also my opinion rather well: It is a mistake that gentoo encouraged too much using shell/bash as a programming language. |
Oh God, more crap from people who don't grok sh. (apart from the comment by Donn.)
Please: all of you who think shell is bad: DON'T USE IT. YOU'RE CRAP AT IT, ANYWAY.
edit: Well, it is a rant.. ;) |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Fri Oct 10, 2014 9:29 am Post subject: Re: Bash bashing (Rant) |
|
|
| steveL wrote: | | perl is hardly forgiving |
As is shell. My experience is that with "use strong" and "use warnings", I get a lot of typos (like in variable/function names) which I might have missed in a shell. Of course, if you do not understand the language well, you can get unexpected results in both languages. Concerning safety from typical mistakes, a language with static typechecking would be preferable over shell and perl. Unfortunately, there is no typical interpreter language with such a property; something which I really miss.
The library is large, perl itself is not: The binary has roughly only about double the size of bash (I included in this size of course perl's core library which is necessary to run perl); with this binary you can program already practically everything which you can code in shell, with a comparable simple syntax.
It would be unfair to include in the size comparison all the remaining functions added into the core distribution of perl. Of course, one could spend weeks to discuss whether it is a good idea to add all these libraries to the core distribution, but only because perl does and shells do not (because shells force you to use external programs instead of providing the possibility of a powerful library) it is not appropriate to use that optional code for comparison.
| Quote: | | and a total pita to bootstrap. |
I installed perl only manually at the times of perl-3 and perl-4, and once a "minimal" perl-5 (without any libraries, that's why I know that tihs works very well) in my user space (on a NeXt system without any perl, for which I had no root permissions). However, I would expect that the make system of perl organizes all of the necessary bootstrapping. For cross-compilation, it might be more complicated, but I guess this should only be necessary for extremely restricted embedded systems.
When I consider the systems on which perl claims to run, I can hardly imagine that there is any shell which is really more compatible.
Can anybody name a system where one of the common shells but not perl can be installed? (If possible with a reason why perl does not run there.) |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Sat Oct 11, 2014 4:01 am Post subject: |
|
|
@mv I've noticed a few times in conversation with you, that you tend to miss the larger point I'm making and instead focus on the part that interests you, which is fair enough, but quite annoying sometimes. Mostly because I'd actually like to have that larger conversation with you, but you refuse to engage. (an example would be discussing the missing deps from BSD, rather than how we workaround the fact that they're missing.)
It's not a criticism; more an expression of slight frustration (quite amusing to my higher mind.. ;)
Is it that you see the wider points as needless distraction?
That's the impression I get, but it's not for me to attribute motivation to you: I'd rather just ask you. I see it as part of the process of conversation.
Anyhow, from my pov, larger points aren't a distraction: they usually explain why there's a shed-load of work in the wrong place. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Sat Oct 11, 2014 5:20 am Post subject: |
|
|
@steveL: Maybe you have to state clearer what you want to talk about. I fail to see that computer ingeneering provides any bigger picture at all; mathematics does, but this is not appropriate for this forum.
| steveL wrote: | | an example would be discussing the missing deps from BSD |
I have no idea what you are talking about. So far, I never used BSD and hardly know what they are doing, and in this thread we did not talk about BSD at all (except for the BSD license). |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Sat Oct 11, 2014 8:33 am Post subject: |
|
|
| mv wrote: | | @steveL: Maybe you have to state clearer what you want to talk about. I fail to see that computer engineering provides any bigger picture at all; mathematics does, but this is not appropriate for this forum. |
Lul; I know what you mean, perhaps.
| Quote: | | steveL wrote: | | an example would be discussing the missing deps from BSD |
I have no idea what you are talking about. |
The discussion we were having which moved from the mailing-list. (Which I think you should pursue on the implementation-level, fwiw.) |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Gentoo Chat
