Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Basic about "shell" and shell script
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
midnite
Guru
Guru


Joined: 09 Apr 2006
Posts: 471
Location: Hong Kong

PostPosted: Thu Apr 14, 2022 3:35 pm    Post subject: Basic about "shell" and shell script Reply with quote

As far as I understand (please correct me if wrong), different shells are different programs. They can be find in packages.

In most systems (including mine), /bin/sh -> bash. AFAIU sh and bash (as well as zsh) are different shells. I have chosen to use zsh as my daily user (not root). zsh is the shell used in the Gentoo version of SysRescueCD. It is more convenient than bash IMO. I installed app-shells/zsh from portage. How can I install /bin/sh? As it is not available in packages. /bin/sh is just a symlink pointing to /bin/bash. So /bin/sh is actually not present in my system.

AFAIU different shells are different [Wikipedia: Comparison of command shells]. And /bin/sh is kind of the ancestor of most shells. Most later shells are developed base on /bin/sh.

To write shell scripts as most portable as possible, we should use syntax of the /bin/sh. Even if I use #!/bin/sh but my system is symlinking it to /bin/bash, I cannot run (try and debug) my scripts in /bin/sh, is it?

As many people use the terms Bourne shell (the basic shell) and POSIX shell interchangeably, they are actually two entries (two rows in the table) in Wikipedia. Are they the same or actually different? Which one has the maximum compatibility?

https://www.geeksforgeeks.org/difference-between-sh-and-bash/
Quote:
sh is also called Bourne Shell. sh is a command programming language described by POSIX standard.

_________________
- midnite.
Back to top
View user's profile Send private message
Goverp
Advocate
Advocate


Joined: 07 Mar 2007
Posts: 2295

PostPosted: Thu Apr 14, 2022 5:34 pm    Post subject: Reply with quote

On Gentoo (and I think most Linuxes) /bin/sh is a symbolic link to whichever shell you choose. On Gentoo you can set the link with "eselect sh".
That's sort of the "system" shell - anything shell script that starts "#!/bin/sh" will use it. If you want to code a script for a particular shell, you'd write "#!/bin/rsh" as the first line.
I'm too young to know the full history of Unix shells. If you want to code for "most" shells you should stick to POSIX compliant shells. Bash is extended way past the POSIX definitions; it has a setting somewhere to make it properly compliant. A good POSIX compliant shell for Gentoo is dash. It has the advantage of being very fast and reliable. A number of Linux distributions use dash as their "system" shell, and if you install dash and then "eselect sh set dash" it will be the system shell for your Gentoo system.

What you get at the command line is usually bash. It's specified in the /etc/passwd entry for each user, and the default entry is /bin/bash. It's more friendly than dash - its history commands are better, and dash doesn't do tab-completion.
_________________
Greybeard
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 10078
Location: almost Mile High in the USA

PostPosted: Thu Apr 14, 2022 6:00 pm    Post subject: Reply with quote

The bourne shell describes a shell language. POSIX tried to standardize this language.

/bin/sh just needs to be a program that can execute bourne shell scripts, ideally understands the POSIX standard bourne shell. It does not matter what it is, just needs to execute it. Any shell that understands bourne shell will do as long as the consumers (all the scripts with #!/bin/sh as its first line) does not use extended feaures -- and this isn't always true.)

Bash does fine, along with ksh, ash, dash, etc., etc. as long as it's not csh, tcsh, or zsh. Technically csh also has a POSIX standard which tcsh tries to adhere to, but it sure won't run bourne shell scripts.

As for bourne shell compatibility, you just have to know your specific target. For the most part I don't see many people using strict bourne shell anymore as it's slow due to a lot of optimizations added in later shells that were not design issues in the original bourne shell.

Note that for so called "embedded" designs that end up with 1GB of memory, just use bash and all its bashisms. Unless your "embedded unix" is less than say 64MB or so, don't bother trying to not use bash.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
midnite
Guru
Guru


Joined: 09 Apr 2006
Posts: 471
Location: Hong Kong

PostPosted: Thu Apr 14, 2022 7:18 pm    Post subject: Reply with quote

Thank you Greybeard and eccerr0r.

So I cannot literally emerge (install) /bin/sh?

I found a good online tool which checks my scripts against the POSIX standard.
https://www.shellcheck.net/

But ShellCheck is masked in Gentoo packages.
https://packages.gentoo.org/packages/dev-util/shellcheck
_________________
- midnite.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6781

PostPosted: Thu Apr 14, 2022 8:40 pm    Post subject: Reply with quote

eccerr0r wrote:
/bin/sh just needs to be a program that can execute bourne shell scripts, ideally understands the POSIX standard bourne shell.

On general unix systems (e.g. on most sun machines), /bin/sh is not POSIX compliant. Neither is /usr/bin/env sh. In general unix systems, you can expect only a Bourne shell which is full of bugs and miles away from being POSIX. There is a POSIX way to get the path to a (rather) POSIX compatible shell, but I forgot how this was. Something with getconf, probably.

On Linux, /bin/sh is usually either bash or dash, both being somewhat POSIX compatible (at least, by far more compatible than the classical Bourne shell ever was).

AFAIK, there exists only one free fully POSIX compatible shell: Bosh from Jörg Schilling. You can find it e.g. in app-shells/schily-tools in the mv overlay.
Quote:
Bash does fine, along with ksh, ash, dash, etc., etc. as long as it's not csh, tcsh, or zsh.

zsh should be completely fine as /bin/sh: It is rather similar to ksh but has much greater interactive features.
Quote:
Unless your "embedded unix" is less than say 64MB or so, don't bother trying to not use bash.

Using dash instead of bash can rather speed up the boot process (if you do not use systemd). But some init-scripts might still have bashisms (which are usually not a problem if you use bash or zsh [and probably also ksh - I never tried]) as /bin/sh.


Last edited by mv on Thu Apr 14, 2022 8:43 pm; edited 1 time in total
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 10078
Location: almost Mile High in the USA

PostPosted: Thu Apr 14, 2022 8:43 pm    Post subject: Reply with quote

Again there is no base bourne shell, nobody would use it.

- hell no, there is definitely no tab completion.
- It doesn't even have command line editing. Typo? Type it again from the beginning unless it was at the end of the line. If you already pressed enter...
- I believe the "! style history" was specified in the original bourne shell (which still exists in bash) but don't bother trying using your arrow keys to recall
- It has no job control. Can't suspend jobs. Forgot to make it asynchronous? Either abort it or wait till it's done before you can get your prompt back.
- more things would fork (as it was defined) so you pay your fork() penalty much more often.

There are some shells like ash and dash that sort of tries to at least add readline(or at least a homemade subset) and job control features in and then some, to make it somewhat "functional" instead of "braindead".

Anyway, shellcheck is simply keyworded ~ unstable, not masked - you can just accept that, and it should merge fine in Gentoo just fine.

---

And hmm... looks like zsh will support csh-like syntax and bourne-like syntax. What is it trying to do...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?


Last edited by eccerr0r on Thu Apr 14, 2022 8:50 pm; edited 1 time in total
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6781

PostPosted: Thu Apr 14, 2022 8:50 pm    Post subject: Reply with quote

eccerr0r wrote:
Again there is no base bourne shell, nobody would use it.

- hell no, there is definitely no tab completion.
- It doesn't even have command line editing.
...

You mentioned only the interactive features. What is worse is that things like ${var#...} and ${var%...} are missing which are used quite often in scripts. I think very early versions did not even allow you to define functions. That's why autoconf generates so horrible configure scripts: If you need to call a function several times you have to define it in m4 to copy the function code several times into the configure script.
Back to top
View user's profile Send private message
midnite
Guru
Guru


Joined: 09 Apr 2006
Posts: 471
Location: Hong Kong

PostPosted: Thu Apr 14, 2022 9:36 pm    Post subject: Reply with quote

eccerr0r wrote:
- hell no, there is definitely no tab completion.
- It doesn't even have command line editing. Typo? Type it again from the beginning unless it was at the end of the line. If you already pressed enter...
Sounds like the adb shell in my old Android.

I don't actually want to use bourne shell. If it is the ancestor of most shells, if my scripts run correctly in bourne shell, then they are compatible enough. I am trying to find a baseline standard, the largest common factor.

Quote:
What is worse is that things like ${var#...} and ${var%...} are missing which are used quite often in scripts.

Parameter Expansion is POSIX standard, I think.

Hmm....I think I was misled by the Wikipedia table. There is no such thing as "POSIX shell". POSIX is a standard (a family of standards). People build shell programs, some try to follow the POSIX standard, while some opt for more features and ignoring the POSIX standard.
_________________
- midnite.
Back to top
View user's profile Send private message
figueroa
Advocate
Advocate


Joined: 14 Aug 2005
Posts: 3026
Location: Edge of marsh USA

PostPosted: Fri Apr 15, 2022 3:20 am    Post subject: Reply with quote

My opinion, it would be a favor to you and your system to re-emerge bash. It is widely assumed across the Linuxverse that bash exists. On most Debian variants, /bin/sh is a symlink to dash, though bash is usually the main shell. I think dash is a bit of a red herring. Having bash on board would give you a less breakable and more normal system.
_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi -wayland
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6781

PostPosted: Fri Apr 15, 2022 6:05 am    Post subject: Reply with quote

midnite wrote:
Parameter Expansion is POSIX standard, I think.

Yes, it is POSIX standard. But as I have written, the original Bourne shell is miles away from the POSIX standard.

As mentioned on another place, also dash (even less bash) are not POSIX. For instance, they lack a lot of i18n support required by POSIX. That's why dash is so fast: Supporting locales as required by POSIX would require much more sophisticated parsing (note that there are locales like Big5 and many more which are much less handy than utf8).
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 4275
Location: Rasi, Finland

PostPosted: Fri Apr 15, 2022 12:48 pm    Post subject: Reply with quote

If I write a shell script to other than bash, I'll make it run on busybox sh. I also tend to test them on mksh (ksh -like shell).
Busybox sh has just enough features but is also enough limited, so that all scripts I've written work on most shells (those which feel like POSIX, so excludes fish for example) then.

If you require arrays, then busybox sh (and POSIX too) wouldn't cut it, unless you use push from fellow member mv. ;)
But then again, there's awk in almost every *NIX system. I write inline awk code for, more compllex things, inside my shell scripts and make them into shell functions.

There's also another topic about which package owns /bin/sh.
_________________
..: Zucca :..

My gentoo installs:
init=/sbin/openrc-init
-systemd -logind -elogind seatd

Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
xanderal
Tux's lil' helper
Tux's lil' helper


Joined: 06 Mar 2019
Posts: 133
Location: Germany

PostPosted: Tue Apr 19, 2022 3:08 pm    Post subject: Reply with quote

Ok, I'm running both Linux and BSD systems, so portability is very important for me and something along those lines seems to be your use case as well, so here are my two cents:

When scripting on e.g. OpenBSD stuff is really easy because /bin/sh is actually a proper POSIX-compliant executable, so that's close to foolproof to work with (sadly I'm not aware of any Linux distro that makes it that easy for us).
But in general I use #!/bin/sh and test it with shellcheck(1). That has support for sh/bash/dash/ksh but is not perfect. Nevertheless it should flag most blunders.
I don't know how far you want to take your portability. If it's only about your stuff running in different shells on a Linux system with GNU core utils, then this should suffice.

But if you're looking into making it run with busybox or other core utils (like e.g. the different BSD ones), then you might have to consider some more stuff:
Not every flag you like is supported, so read manpages (well written ones include a STANDARDS section that among other things tells you which flags are extensions) and other proper reference material (stack overflow is not a reference).

Just an example: Say you want to replace some string, what do you use? 'sed -i'? Nope, you don't, at least not always. :wink: It works with GNU and OpenBSD, but e.g. on FreeBSD this might not work.
Or to quote from the OpenBSD manpage on sed(1):
https://man.openbsd.org/sed wrote:

The sed utility is compliant with the IEEE Std 1003.1-2008 (“POSIX.1”) specification.

The flags [-aEiru] are extensions to that specification.

So in summary: If you want your
midnite wrote:

shell scripts as most portable as possible

you might have to dig further than just what shebang you use. But using #!/bin/sh and making sure shellcheck is happy should get you at least 90% of the way there.

ETA:
Zucca wrote:

But then again, there's awk in almost every *NIX system. I write inline awk code for, more compllex things, inside my shell scripts and make them into shell functions.

Same thing here: Sure there might be an awk on the target system but which one? GNU awk? The original "one true awk" by Aho, Kernighan and Weinberger? Something else entirely?
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 4275
Location: Rasi, Finland

PostPosted: Tue Apr 19, 2022 4:32 pm    Post subject: Reply with quote

xanderal wrote:
Sure there might be an awk on the target system but which one? GNU awk? The original "one true awk" by Aho, Kernighan and Weinberger? Something else entirely?
I use Busybox there too. It really is the easiest way to "guarantee" a some level of compability, not bullet proof.
_________________
..: Zucca :..

My gentoo installs:
init=/sbin/openrc-init
-systemd -logind -elogind seatd

Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
xanderal
Tux's lil' helper
Tux's lil' helper


Joined: 06 Mar 2019
Posts: 133
Location: Germany

PostPosted: Wed Apr 20, 2022 12:24 am    Post subject: Reply with quote

Zucca wrote:
I use Busybox there too. It really is the easiest way to "guarantee" a some level of compability, not bullet proof.
But that doesn't get you far when you don't know the system your tool is running on. Some of my stuff is done not just for myself but for others as well, so I don't know whether the scripts will be run on some Linux, BSD, MacOS, whatever...
I would agree though that (provided the feature set of Busybox suffices) this could be a useful starting point for midnite - very much depends on how portable this stuff needs to be/how much control they have over the OS it's supposed to be running on.

If I might ask a tiny question for my own benefit (but hopefully OP's as well), @Zucca:
Have you in real world application ever run into a problem where Busybox was too restricted in the implemented features? Or have I misunderstood you earlier and the limitations are more of a hypothetical worry one can mainly forget about?
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 4275
Location: Rasi, Finland

PostPosted: Wed Apr 20, 2022 7:59 am    Post subject: Reply with quote

@xanderal, so far I only miss arrays in busybox shell, but there are ways to overcome that limitation if one really needs arrays.
busybox awk works just fine. I just need to avoid some functions like gensub. I've written my custom initramfs scripts to be run with with busybox.
I also looked at toybox to replace busybox, but it doesn't implement sh and awk yet (or they reside in 'pending' directory).
One quite frustrating, but manageable, incompability between GNU, busybox and toybox was mktemp. With careful choosing of arguments it "beacame" portable too. Then there's date. It's a same story, but you can also use awk for most date realted things.

So it's not like busybox is ever too limited, but the portability is the tricky part in my opinion. Since shell scripting isn't just the syntax, but the command line tools too.
_________________
..: Zucca :..

My gentoo installs:
init=/sbin/openrc-init
-systemd -logind -elogind seatd

Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9648
Location: beyond the rim

PostPosted: Wed Apr 20, 2022 8:25 am    Post subject: Reply with quote

midnite wrote:
So I cannot literally emerge (install) /bin/sh?

Nope, as that "project" kinda died decades ago. Even if you get the source, you'll have a hard time to get it running on a modern OS. See https://www.in-ulm.de/~mascheck/bourne/ for some background information.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6781

PostPosted: Thu Apr 21, 2022 6:57 pm    Post subject: Reply with quote

Genone wrote:
Nope, as that "project" kinda died decades ago. Even if you get the source, you'll have a hard time to get it running on a modern OS

bosh is based on the original source code. Schily has also maintained obosh which has no features added compared to the SVR4 variant. You can install obosh from schily-tools from the mv overlay.
Back to top
View user's profile Send private message
xanderal
Tux's lil' helper
Tux's lil' helper


Joined: 06 Mar 2019
Posts: 133
Location: Germany

PostPosted: Sat Apr 23, 2022 11:31 pm    Post subject: Reply with quote

Zucca wrote:
@xanderal, so far I only miss arrays in busybox shell, but there are ways to overcome that limitation if one really needs arrays.
busybox awk works just fine. I just need to avoid some functions like gensub. I've written my custom initramfs scripts to be run with with busybox.
I also looked at toybox to replace busybox, but it doesn't implement sh and awk yet (or they reside in 'pending' directory).
One quite frustrating, but manageable, incompability between GNU, busybox and toybox was mktemp. With careful choosing of arguments it "beacame" portable too. Then there's date. It's a same story, but you can also use awk for most date realted things.

So it's not like busybox is ever too limited, but the portability is the tricky part in my opinion. Since shell scripting isn't just the syntax, but the command line tools too.
Thanks for that, @Zucca! Trying busybox was in the back of my mind for a while now but I never got around to it. Good to know what to watch out for :)
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 4275
Location: Rasi, Finland

PostPosted: Tue Apr 26, 2022 7:17 pm    Post subject: Reply with quote

Zucca wrote:
Then there's date. It's a same story, but you can also use awk for most date realted things.
I'll add a clarification here. There are no date functions in nawk/bwk/one true awk. However the busybox awk seems to have them... along with gawk.

Off-topic:
    If you need to process large datasets where you need to convert dates, an awk implementation which has these date functions is really fast compared to executing date for each conversion. But you don't want to implement your own date conversion functions in raw awk either. Timezones (changing ones), leap years and leap seconds will guarantee you a place in some kind of asylum if you try. ;)

_________________
..: Zucca :..

My gentoo installs:
init=/sbin/openrc-init
-systemd -logind -elogind seatd

Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
palak231
n00b
n00b


Joined: 04 Jun 2022
Posts: 1

PostPosted: Sat Aug 06, 2022 7:25 am    Post subject: Basic about shell and shell script Reply with quote

Hi this is Palak Sharma.
Well while writing any shell ( https://www.techgeekbuzz.com/blog/shell-commands-in-linux/ ) programs we use /bin/sh and /bin/bash. And there is not too much different in between both of them. You can understand different in between both of them with the help of example by clicking here.

https://stackoverflow.com/questions/5725296/difference-between-sh-and-bash

Hope it will helps you.
Thanks


Last edited by palak231 on Mon Aug 08, 2022 8:01 am; edited 1 time in total
Back to top
View user's profile Send private message
xanderal
Tux's lil' helper
Tux's lil' helper


Joined: 06 Mar 2019
Posts: 133
Location: Germany

PostPosted: Sat Aug 06, 2022 9:55 am    Post subject: Re: Basic about shell and shell script Reply with quote

Hi Palak,
palak231 wrote:
Well while writing any shell ( https://www.techgeekbuzz.com/blog/shell-commands-in-linux/ ) programs we use /bin/sh and /bin/bash.
No, there is a number of reasons why one might use a different shell, for example since there are things (e.g. in the many different versions of ksh) that don't work (or work differently) in sh or bash so you specify /bin/ksh or /bin/oksh or whatever. So no, we don't just use /bin/sh or /bin/bash.
palak231 wrote:
And there is not too much different in between both of them. You can understand different in between both of them with the help of example by clicking here.

https://stackoverflow.com/questions/5725296/difference-between-sh-and-bash
For one, there is significant difference between them and for two: Just because it's written on stackoverflow, doesn't mean it's true.
In the majority of cases people have /bin/sh only as a symlink to /bin/bash, /bin/dash or something similar. So in a sense, there isn't much of a meaningful difference in most cases.
But if you had read and understood the discussion we had before you might have realised that the majority is different from everyone.

So in summary:
- /bin/sh can be an executable in its own right and not just a symlink.
- When /bin/sh is in fact a symlink people symlink it to a wide variety of posix-compliant shells, i.e. not just /bin/bash (Sidenote: bash is also not always living in /bin/bash but sometimes in e.g. /usr/bin/bash. Figuring out on which systems that is the case, is left as an exercise to the reader :twisted: ).
- That stackoverflow answer is incomplete and should not be referenced without mentioning that caviat.
- It is useful to read a thread before commenting on it.

Last comment: It's cool that you are here. Even if my rant might not look like it: I'm happy you're here ... but please do your research before commenting somewhere. That way you get a learning experience, OP get's a more useful answer and everyone else will likely appreciate it as well.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum