Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
systemd ???
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3  
Reply to topic    Gentoo Forums Forum Index Gentoo Chat
View previous topic :: View next topic  
Author Message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Fri Apr 11, 2014 11:23 am    Post subject: Reply with quote

mv wrote:
For a decently complex script with some successively calliing functions/scripts you are easily executing hundreds of subshells. If one subshell takes 10ms just to start, you are already losing seconds only due to the subshell calls - the actual time to actually execute the script and do something useful is not even counted here.

Sounds like a really badly written script to me. I write complex bash, and shell too, and I've been amazed at the performance you can get out of bash. I begrudgingly gave up trying to get it to fall over on a 32-bit system. Functions do not require a subshell, that's the point of them.

Sure it's a slower than sh; but when you absolutely need arrays, and ebuilds are written in bash, it makes no sense not to use the standard GNU shell for scripting. I know I know, zsh is the "best". It's just not guaranteed to be on an end-user system, whereas bash is. Note that this is not about "everyone uses bash", but about ebuilds using bash, which means Gentoo users always have it on their machine (unless custom, and not a full running Gentoo install, in which case not my concern.)

It's funny though, on the one hand I get people telling me "bash is too big, use a smaller shell" (and I do, for other things) and on the other I get "use python, since emerge is in python", which is even more bloat afaic. As is perl, while we're on the subject. If you ever did a stage 1 install, you know what I mean.
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6227
Location: Room 101

PostPosted: Fri Apr 11, 2014 1:23 pm    Post subject: Reply with quote

steveL wrote:
Sure it's a slower than sh; but when you absolutely need arrays, and ebuilds are written in bash, it makes no sense not to use the standard GNU shell for scripting.

steve ... I think you mean "standard posix shell", but having said "standard GNU shell" you reveal part of the problem ... the expectation that /bin/sh will be /bin/bash and the knock on effect of that expectation. Mostly this goes unnoticed because /bin/sh *is* /bin/bash and it accepts any bashism thrown at it (even if provided '--posix' or called as /bin/sh). There is nothing inherently wrong with bash (sorry, did I just say that? hehe), but its not "standard", except that, as everyone knows, standards are written with one hand while the other hand is crossing its fingers :)

steveL wrote:
I know I know, zsh is the "best". It's just not guaranteed to be on an end-user system, whereas bash is.

Yeah ... but this is a chicken and egg, no? Its not guaranteed but bash is ... why might that be? I've yet to see a script that expects that /bin/sh is /bin/zsh or a script that uses zshisms with the expectation that this fact will go unnoticed. If this was the case then we might expect a mighty muddle ... unless of course we call it "the GNU standard" and cross our fingers.

steveL wrote:
Note that this is not about "everyone uses bash", but about ebuilds using bash, which means Gentoo users always have it on their machine (unless custom, and not a full running Gentoo install, in which case not my concern.)

Well, at some point there was a *decision* to "use bash", portage didn't come into existence by happenstance. That *decision* was pre-conditioned by certain factors, namely the fact that "everyone uses bash". It *could* have been otherwise, but the scales were weighted in favour of bash.

None of the above is an argument for zsh (or some other shell) replacing bash ... it just a matter of illustrating why things are they way they are.

best ... khay
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Fri Apr 11, 2014 7:15 pm    Post subject: Reply with quote

steveL wrote:
Sounds like a really badly written script to me.

Maybe, but it is typical: Just look at the eclasses of gentoo. Many of them call sed or grep or use a pipe (any of them spawns first a subshell and then another process, pipes even two or three), when one could do with inline-string-handling.
Quote:
Functions do not require a subshell, that's the point of them.

They can be written such that they do not require a subshell, but it is common practice to handle return values over stdout which means a subshell (also when you have local variables or use "cd" and do not want to use "pushd" bashism, it is cleaner to use a subshell for a function, anyway). Then you easily have some string functions calling each other and returning over stdout, and you have easily spawned 10-20 subshells or so just in preparing some of the variables which you later want to work with...
Do not misunderstand me: I am not saying that it is impossible or hard to avoid these subshells. It is just something that you see frequently when you look at random scripts (often just copied and not written by specialists). Most of these could be improved concerning efficiency...
Quote:
use python [...] which is even more bloat afaic. As is perl

If it were not for the startup times or perhaps certain embedded systems, I would use perl exclusively for my scripts: I have not yet seen a system on which not at least some version of perl is running, and once you use it, you do not have random limitations concerning line-lengths or other inconvenient restrictoins; no need to worry about quoting or expansion, the same syntax of regular expressions throughout (and in one of the most powerful variants, and even in a way of efficiency which you do not get in any other language I know), and if you suddenly realize that arrays or hashes would come in handy, you can just use it without any problems and again in a very efficient way: No fighting with various awk or sed variants and their tiny differences: It just gives you a "union" of all features of sh, awk, sed, sort and some other text tools, all surprisingly close to their original syntax (I can give you examples if you want). And if there is a need, you just add "use strict;" to turn your script into a "serious" programs with serious checking (not as serious as C++, but still quite well, running often even more efficient due to some features of the language).

Sorry for being OT, but I feel perl is often very underestimated, and considering it as bloat is a wrong point of view IMHO.
Back to top
View user's profile Send private message
Anon-E-moose
Watchman
Watchman


Joined: 23 May 2008
Posts: 6095
Location: Dallas area

PostPosted: Fri Apr 11, 2014 7:23 pm    Post subject: Reply with quote

mv wrote:
but I feel perl is often very underestimated, and considering it as bloat is a wrong point of view IMHO.


I've written a lot of programs in perl (business not personal, though I do that too), and I agree as a general rule.
If one has to write shell scripts, and call grep, sed, etc then perl is quite easily much more efficient.
Oh one can write cr*p in perl, but I can say the same about any language.

I understand that people don't like it, because like the original C language, there are many ways to do things.
If one only wants one way to do things, then go with pascal, ada, python, etc.

Anyway....
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Fri Apr 11, 2014 8:57 pm    Post subject: Reply with quote

khayyam wrote:
steveL wrote:
Sure it's a slower than sh; but when you absolutely need arrays, and ebuilds are written in bash, it makes no sense not to use the standard GNU shell for scripting.

steve ... I think you mean "standard posix shell", but having said "standard GNU shell" you reveal part of the problem

No I meant the standard GNU shell, not /bin/sh. And I meant in the context of writing complex scripts for Gentoo installs. One of the first things you learn in #bash is not to run your scripts with a #!/bin/sh shebang. So with respect, I've snipped and will discount the points about bash-as-sh.
Quote:
Well, at some point there was a *decision* to "use bash", portage didn't come into existence by happenstance. That *decision* was pre-conditioned by certain factors, namely the fact that "everyone uses bash". It *could* have been otherwise, but the scales were weighted in favour of bash.

Sure, but again you come to the point that a Gentoo install is an application of GNU/Linux. The GNU shell is bash, and has extensions usually derived from ksh, for the specific reason that you can use things like arrays and script more capably. Of course that doesn't help if you don't know how to shell-script in the first place, or you don't learn from decent sources like #bash and wooledge.org, and have instead learnt from tldp ABS, or other distro developers who once learnt from it. Down that road lies the thinking which led to systemd: our shell-scripts are terrible, let's blame shell, since we've all spent the last few years looking down our noses at it.
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Fri Apr 11, 2014 9:16 pm    Post subject: Reply with quote

mv wrote:
Do not misunderstand me: I am not saying that it is impossible or hard to avoid these subshells. It is just something that you see frequently when you look at random scripts (often just copied and not written by specialists). Most of these could be improved concerning efficiency...

Yes, but that's exactly my point. You don't start blaming C when someone messes up in it. Why blame shell? I for one have always known to avoid subshells like the plague, and I only fork to externals when it makes sense. That's why update is such a big script. Admittedly part of that was my perverse intent to see how far I could go before bash fell over (it was after all started as a learning exercise) but it never did.
Quote:
If it were not for the startup times or perhaps certain embedded systems, I would use perl exclusively for my scripts..
Sorry for being OT, but I feel perl is often very underestimated, and considering it as bloat is a wrong point of view IMHO.

Yeah and I'm sorry for poking you: I know you love perl. To my mind it should have been a compiled language from the beginning. That it wasn't shows people didn't really see what Kernighan was getting at in the awkbook, K&R and UPE, in my opinion, but that's just based on my reading of as much of his work as I could track down. It screams to me of "here's all the pieces, build the next language," and instead people took his tutorial scripting code and ran with that part instead.

As I said though, "If you ever did a stage 1 install, you know what I mean." perl is a b1tch to bootstrap. I'm well aware that bash requires it built first, just like all other GNU software, but that's because of the whole mess that is autoconf, of which its own author said about a year later "using m4 was probably a bad idea" (or words to that effect.) Again, I simply cannot understand why people didn't listen.

Given that you have a pre-distributed 40,000+ line-long configure script, you can bootstrap.. heh.

Anyhow, as moose said, you can write terrible perl too. Would you then blame perl for that, or the coder who pretended to know what he was doing, instead of finding out how to do it properly?

Same thing with shell. Only shell has to accept dumb input, as its purpose is interactive terminal use. That's why it has such a complex grammar, requiring many more lexer states than is common. So on the one hand it accepts all kinds of input, and on the other it does its best to do what you asked. Of course it's easier to write badly. It still does not stand that somehow the language is to blame when someone messes up in it, nor does it somehow mean you don't need professionalism.

You got me with C++ though. That is one awful language. Good one ;-)
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6227
Location: Room 101

PostPosted: Fri Apr 11, 2014 11:51 pm    Post subject: Reply with quote

steveL wrote:
khayyam wrote:
steveL wrote:
Sure it's a slower than sh; but when you absolutely need arrays, and ebuilds are written in bash, it makes no sense not to use the standard GNU shell for scripting.

steve ... I think you mean "standard posix shell", but having said "standard GNU shell" you reveal part of the problem

No I meant the standard GNU shell, not /bin/sh.

steve ... sorry, somehow I missed the double negative in the above and read it as a "to use".

steveL wrote:
One of the first things you learn in #bash is not to run your scripts with a #!/bin/sh shebang. So with respect, I've snipped and will discount the points about bash-as-sh.

But that is part of the "standard", the "standard posix shell" is also /bin/bash ... this is the case with practically every "GNU/Linux". Its not hard to figure out why this has created a situation in which the two are confused, and it doesn't help that the bash man page claims that it "can be configured to be POSIX-conformant by default" via --posix and invocation as sh. In my link above I showed that this isn't the case, bashisms are accepted even if --posix is used. So, with similar respect, discount that if you will, but that is breaking a "standard" for the purpose of establishing the "GNU/Linux standard".

steveL wrote:
khayyam wrote:
Well, at some point there was a *decision* to "use bash", portage didn't come into existence by happenstance. That *decision* was pre-conditioned by certain factors, namely the fact that "everyone uses bash". It *could* have been otherwise, but the scales were weighted in favour of bash.

Sure, but again you come to the point that a Gentoo install is an application of GNU/Linux. The GNU shell is bash [...]

I was responding to:

steveL wrote:
Note that this is not about "everyone uses bash", but about ebuilds using bash, which means Gentoo users always have it on their machine [...]

... so, my point: "it could have been otherwise", and if you say "its GNU/Linux therefore we use bash as its GNU", I'll repeat myself, "it could have been otherwise". Now, as far as GNU is concerned, it seems as if its also a recursive syllogism.

steveL wrote:
Of course that doesn't help if you don't know how to shell-script in the first place, or you don't learn from decent sources like #bash and wooledge.org, and have instead learnt from tldp ABS, or other distro developers who once learnt from it. Down that road lies the thinking which led to systemd: our shell-scripts are terrible, let's blame shell, since we've all spent the last few years looking down our noses at it.

This seems to be an entirely different question altogether ...

best ... khay
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Sat Apr 12, 2014 5:47 am    Post subject: Reply with quote

steveL wrote:
perl. To my mind it should have been a compiled language from the beginning.

It is as close to compiled as a language can be in which you can overload anything you want at runtime. The latter is really useful sometimes. For instance, I once had a complex script which eventually was applied to more data than I had RAM. No task to get a "database" for the most RAM-consuming parts without having to reconcept the whole script. Or I had a numerical script which I wanted to verify with more accuracy...
Quote:
You got me with C++ though. That is one awful language.

When it was developed, it was certainly the best general-purpose language, and I still see nothing which comes marginally close: It takes practically everything from C but making it possible to program more compatible through a more standardized library, allowing all the goodies of operator/function overloading, providing means of working with abstract data types through templates (if you have ever tried to write a general-purpose splay-tree implementation with C or to use some sort algorithm from some library you learn these advantages: Either you can only work with void* pointers and function callbacks, or you write templates as C macros, both of which are just horrible hacks), and last but not least, has static datatyping thus allowing for rather safe "testing" at compilation time if you program cleanly (instead of having the tendency to give you bad results at exceptional cases at runtime - we just got the ssl heartbeat example).
I admit that many graphical toolkit libraries misuse classes in a too convoluted manner which in turn can have unexpected side effects due to the terrible ad-hoc multiple inheritance rules, but one should not blame a few bad libraries for the language itself. Maybe it is in principle not possible to provide better library interfaces for GUIs: From what I have seen from Java, you can do things still much worse.
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Sun Apr 13, 2014 8:44 am    Post subject: Reply with quote

khayyam wrote:
steveL wrote:
One of the first things you learn in #bash is not to run your scripts with a #!/bin/sh shebang. So with respect, I've snipped and will discount the points about bash-as-sh.

But that is part of the "standard", the "standard posix shell" is also /bin/bash ... this is the case with practically every "GNU/Linux". Its not hard to figure out why this has created a situation in which the two are confused, and it doesn't help that the bash man page claims that it "can be configured to be POSIX-conformant by default" via --posix and invocation as sh. In my link above I showed that this isn't the case, bashisms are accepted even if --posix is used. So, with similar respect, discount that if you will, but that is breaking a "standard" for the purpose of establishing the "GNU/Linux standard".

I'm just talking from the perspective of getting things done. Any serious bash scripter learns very early on not to use bash as sh, it simply cripples it. I agree with your wider point, it's awful to pretend to be sh, but then accepts bashishms, since it leads to bad scripts.
But it's not what really matters to me, even if I agree it should have been done differently: getting a result is what matters.
Quote:
... so, my point: "it could have been otherwise", and if you say "its GNU/Linux therefore we use bash as its GNU", I'll repeat myself, "it could have been otherwise". Now, as far as GNU is concerned, it seems as if its also a recursive syllogism.

Yes but then GNU's Not Unix is already recursive ;-) My point is that GNU was a very successful attempt at replacing the whole of the userland with Free alternatives, from the ground up. Part of that was having a next-generation shell along the lines of ksh. OFC "it could have been otherwise" but I don't see the relevance in terms of actually getting things done in the here and now. We are where we are, and ebuilds use BASH. If I want to wrap package management, the best choice for me is bash, since by definition it has to be available otherwise portage cannot build anything, and nor can any other package manager.

If you want to talk about alternative formats, by all means, although I'd point out it's been done before, and I don't actually see any utility in avoiding bash. Not when you have to bootstrap perl in order to run autotools in any case, so you already have a world of bring-up to do, if someone else isn't providing you with a lovely stage3. And not when you're about to run a massive build process. It really is not a bottleneck, as my own experience has taught me. You want a "bloated" shell available at that point so you can set things up properly in a relatively maintainable way. If you got rid of the crappy Gentoo house "style" and dumped some of the more baroque excursions in hypothesis of the last few years, ebuilds are a really nice format. They can be written very cleanly, purely declaratively in most cases.

GNU doesn't stop you using alternatives, by any means. Nor do I use bash for everything: I run my makefiles under busybox, and will test with dash too. Based on this thread, I'll likely try it out as sh symlink as well.
Quote:
steveL wrote:
Of course that doesn't help if you don't know how to shell-script in the first place, or you don't learn from decent sources like #bash and wooledge.org, and have instead learnt from tldp ABS, or other distro developers who once learnt from it. Down that road lies the thinking which led to systemd: our shell-scripts are terrible, let's blame shell, since we've all spent the last few years looking down our noses at it.

This seems to be an entirely different question altogether ...

Well we are in a discussion about systemd and people are discussing the performance of shell, and look try this shell, try that shell, lending credence to the "motivation", when really shell is not the problem. Inept scripting is, and it's better that those "developers" stop using shell, as it only gives the rest of us a bad name. OFC it would have been better if they'd realised the problem a decade ago and brought in some scripters, instead of buying a bridge, but that would have required admitting ignorance, and inability in "only shell."
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 7470

PostPosted: Sun Apr 13, 2014 10:22 am    Post subject: Reply with quote

I only think shell was a problem for systemd, not for performance issue, but because it gave anyone ability to build your own solution.

If i made a really performance/code... ugly bash script that do the work i need, at least i endup with something that works, and do what i want.
Removing shell usage is preventing anyone from easy customize and adapt his own system to his need, binding users again to systemd.

After that, it's better to tell anyone you can do dumb things with shell and so shell sucks.
I don't think any language prevent anyone from doing dumb things with them. And it could be worst to make a leaking memory prog with non scripting language. Even a 10 lines of code that leak memory and running on a server months would be worst compare to any dumbest made shell script.
And everyone could then compare performance of the server running without memory vs any dumb shells script that would do zillions call.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Sun Apr 13, 2014 2:29 pm    Post subject: Reply with quote

krinn wrote:
I only think shell was a problem for systemd, not for performance issue, but because it gave anyone ability to build your own solution.

Although the discussion about shell speeds happened in a thread with "systemd" in the title, I was not meaning to speak in favor of or against systemd in this connection (it was more concerning bash than systemd).
However, saying that systemd's idea to avoid shell scripts was only invented to take abilities from the user is supposing a bit too much. The actual "taking of abilities" of sytemd is not that it makes it less convient ot call shell scripts - you only have to give the path to such a script which is not really an issue - but that many things like clock handling or the logging daemon are hardcoded into systemd; it is not a question of calling a shell but a question of honouring anything user-provided at all...
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo Chat All times are GMT
Goto page Previous  1, 2, 3
Page 3 of 3

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum