Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[TOPIC] What does the shebang really do?
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Gentoo Chat
View previous topic :: View next topic  
Author Message
Mr. T.
Guru
Guru


Joined: 26 Dec 2016
Posts: 477

PostPosted: Wed Mar 28, 2018 4:16 pm    Post subject: [TOPIC] What does the shebang really do? Reply with quote

Mod edit: Split and judiciously cherry picked from [solved] dash as /bin/sh as the thread had veered decidedly off topic. — JRG

A script begins with a shebang so there will be no problem. Nonetheless, your system must have a bash interpreter (installed by default).


Last edited by Mr. T. on Sat Mar 31, 2018 5:49 am; edited 1 time in total
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 12384

PostPosted: Thu Mar 29, 2018 1:58 am    Post subject: Reply with quote

helecho wrote:
A script begins with a shebang so there will be no problem. Nonetheless, your system must have a bash interpreter (installed by default).
This completely ignores the problem that the OP wanted to address. Bash enables some bashisms even when run as /bin/sh. If a script says it runs in /bin/sh, but actually uses certain Bash extensions, it will run fine on systems where /bin/sh is bash, but fail when /bin/sh points to dash. Dash provides all features that /bin/sh is required to provide, so it should be safe to use for /bin/sh, provided that all scripts that request /bin/sh really do work without Bash extensions. OP wanted to know whether any such bashisms were present.
Back to top
View user's profile Send private message
Mr. T.
Guru
Guru


Joined: 26 Dec 2016
Posts: 477

PostPosted: Thu Mar 29, 2018 6:24 am    Post subject: Reply with quote

@Hu: the original post is not precise enough to support your statement because scripts can be developed by anyone.
You probably assume that scripts developed by Gentoo are reliable. Moreover, an administrator is responsible for
scripts added. Finally, it is reasonable to assume that scripts developed have a correct shebang. Therefore,
I correctly considered the problem, at least partially.
Back to top
View user's profile Send private message
Zucca
Veteran
Veteran


Joined: 14 Jun 2007
Posts: 1299
Location: KUUSANKOSKI, Finland

PostPosted: Thu Mar 29, 2018 9:54 am    Post subject: Reply with quote

helecho wrote:
A script begins with a shebang so there will be no problem.
About shebangs...
Does anyone know if OpenRC is "flexible enough" to support different interpreters via shebang? For example if someone crazy anough would want to write a init script in lua? :D (I'm not.) This, of course, creates a whole new set of problems because functions and variables aren't passed to the lua environment (well, variables maybe).

Moreover, does OpenRC even respect the shebang or does it automatically run the init script using /sbin/openrc-run?
_________________
..: Zucca :..

Code:
ERROR: '--failure' is not an option. Aborting...
Back to top
View user's profile Send private message
Mr. T.
Guru
Guru


Joined: 26 Dec 2016
Posts: 477

PostPosted: Thu Mar 29, 2018 11:36 am    Post subject: Reply with quote

Lua can interact with C code.

Off Topic

I would like to design a package manager implemented in Lua / C to remove Bash (ebuilds, eclasses). Lua is higher level than Bash and provides various facilities.
I like their prototype based programming.

Code:
mt = {__index = function () local z = "non nil"; return z end }
table = {}
setmetatable(table, mt)
print(table.var)


helecho


Last edited by Mr. T. on Thu Mar 29, 2018 12:30 pm; edited 4 times in total
Back to top
View user's profile Send private message
Zucca
Veteran
Veteran


Joined: 14 Jun 2007
Posts: 1299
Location: KUUSANKOSKI, Finland

PostPosted: Thu Mar 29, 2018 12:06 pm    Post subject: Reply with quote

helecho wrote:
implemented in Lua / C to remove Bash (ebuilds, eclasses).
Then you'd need to translate all the ebuilds and eclasses to lua.
It would be easier just to start your own distribution. I'm serious here.

_________________
..: Zucca :..

Code:
ERROR: '--failure' is not an option. Aborting...
Back to top
View user's profile Send private message
Mr. T.
Guru
Guru


Joined: 26 Dec 2016
Posts: 477

PostPosted: Thu Mar 29, 2018 12:35 pm    Post subject: Reply with quote

@Zucca: I would like to fundamentally change the package manager but Gentoo is a malleable meta distribution.
Back to top
View user's profile Send private message
Naib
Watchman
Watchman


Joined: 21 May 2004
Posts: 5290
Location: Removed by Neddy

PostPosted: Thu Mar 29, 2018 12:49 pm    Post subject: Reply with quote

helecho wrote:
@Zucca: I would like to fundamentally change the package manager but Gentoo is a malleable meta distribution.
A few years ago a few of us contemplated a plugin approach to portage and ebuilds... Such ebuilds could be written in the supported languages as long as they adhered to an api. Some means to determine the language of the ebuilds early would be needed as at the moment they are just bash-sourced
_________________
The best argument against democracy is a five-minute conversation with the average voter
Great Britain is a republic, with a hereditary president, while the United States is a monarchy with an elective king
Back to top
View user's profile Send private message
Mr. T.
Guru
Guru


Joined: 26 Dec 2016
Posts: 477

PostPosted: Thu Mar 29, 2018 1:12 pm    Post subject: Reply with quote

@Naib: I have not started my personal work yet but my main motivation is to simplify the interface of the package manager.
In other word, I hope "my" package manager will be easier to use than current Gentoo package managers.
Back to top
View user's profile Send private message
Naib
Watchman
Watchman


Joined: 21 May 2004
Posts: 5290
Location: Removed by Neddy

PostPosted: Thu Mar 29, 2018 1:33 pm    Post subject: Reply with quote

helecho wrote:
@Naib: I have not started my personal work yet but my main motivation is to simplify the interface of the package manager.
In other word, I hope "my" package manager will be easier to use than current Gentoo package managers.
my advice... start with the resolver, this is the bit that is spagetti in portage. The generation of the install scripts (ie basically sourcing the ebuild) is relatively simple
_________________
The best argument against democracy is a five-minute conversation with the average voter
Great Britain is a republic, with a hereditary president, while the United States is a monarchy with an elective king
Back to top
View user's profile Send private message
Mr. T.
Guru
Guru


Joined: 26 Dec 2016
Posts: 477

PostPosted: Thu Mar 29, 2018 2:51 pm    Post subject: Reply with quote

@Naib: I will start from scratch but I will eventually reuse algorithms that come from pkgcore. Portage code is not very attractive.
I need to evaluate how to exploit ebuilds (more precisely data included in ebuilds). The package manager could be redefined
dynamically like emacs (operation, configuration for packages). I do not know the other features.

I believe that package management in a dynamic way would be funny and give flexibility.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 12384

PostPosted: Fri Mar 30, 2018 2:18 am    Post subject: Reply with quote

helecho wrote:
@Hu: the original post is not precise enough to support your statement because scripts can be developed by anyone.
You probably assume that scripts developed by Gentoo are reliable. Moreover, an administrator is responsible for
scripts added. Finally, it is reasonable to assume that scripts developed have a correct shebang. Therefore,
I correctly considered the problem, at least partially.
OP doesn't believe it reasonable to assume the scripts have a correct shebang, and I agree with him. To recap:
  1. A shell script with shebang may specify /bin/sh or /bin/bash (or other choices irrelevant here).
  2. GNU Bash recognizes certain non-POSIX-sh extensions, regardless of the name from which GNU Bash was invoked.
  3. Dash does not recognize some non-POSIX-sh extensions that GNU Bash accepts.
  4. A shell script with a shebang of /bin/sh may or may not contain "bashisms" (non-POSIX-sh extensions).
    1. If /bin/sh is an alias for GNU Bash, then a shell script that contains bashisms will work.
    2. If /bin/sh is not GNU Bash, then a shell script that contains bashisms will not work.
  5. Many systems use GNU Bash for /bin/sh. Testing a script that contains bashisms on such a system will succeed, even though it should not succeed. This pattern has enabled many scripts to develop bashisms that only troubled people who elected to use non-GNU Bash shells for their /bin/sh.
  6. OP wants to use non-GNU Bash for his /bin/sh.
  7. OP worries that such scripts still exist, and will break when he switches.
If I understand your post correctly, you posit that everyone who wrote a shebang picked the correct interpreter for their script. Specifically, that if the script uses any bashisms, then the shebang definitely points to /bin/bash, even though GNU Bash itself does not enforce that the author write this correctly. Additionally, you posit that any subsequent maintenance that introduced a bashism also included a fix to the shebang to require GNU Bash instead of requiring any POSIX-sh shell. History provides counterexamples to these "for all" propositions.
Back to top
View user's profile Send private message
Mr. T.
Guru
Guru


Joined: 26 Dec 2016
Posts: 477

PostPosted: Fri Mar 30, 2018 5:28 am    Post subject: Reply with quote

@Hu: The environment allows to conform to reality so the shebang should not be incorrect.

Code:
larry $ ls -l hello
-rwxr--r-- 1 larry   larry       18 march  30 06:12 hello

larry $ cat hello
# print hello world on screen
echo hello world!

larry $ bash hello
hello world

larry $ nano hello
...
larry $ cat hello
#!/bin/bash

# print hello world on screen
echo hello world!

larry $ ./hello
hello world!


A shebang allows users to run a shell script without specifying the interpreter on the command line.
GNU Bash runs in POSIX mode if the shebang is /bin/sh and specific features are explicitly specified in the documentation.
Back to top
View user's profile Send private message
i4dnf
Apprentice
Apprentice


Joined: 18 Sep 2005
Posts: 251
Location: Bucharest, Romania

PostPosted: Fri Mar 30, 2018 8:52 am    Post subject: Reply with quote

@helecho, I'm terribly sorry to have to ask you this, but have you actually read the post you're replying to?

Hint, relevant part quoted below:
Hu wrote:
To recap:
  1. A shell script with shebang may specify /bin/sh or /bin/bash (or other choices irrelevant here).
  2. GNU Bash recognizes certain non-POSIX-sh extensions, regardless of the name from which GNU Bash was invoked.
  3. Dash does not recognize some non-POSIX-sh extensions that GNU Bash accepts.
  4. A shell script with a shebang of /bin/sh may or may not contain "bashisms" (non-POSIX-sh extensions).
    1. If /bin/sh is an alias for GNU Bash, then a shell script that contains bashisms will work.
    2. If /bin/sh is not GNU Bash, then a shell script that contains bashisms will not work.
  5. Many systems use GNU Bash for /bin/sh. Testing a script that contains bashisms on such a system will succeed, even though it should not succeed. This pattern has enabled many scripts to develop bashisms that only troubled people who elected to use non-GNU Bash shells for their /bin/sh.
  6. OP wants to use non-GNU Bash for his /bin/sh.
  7. OP worries that such scripts still exist, and will break when he switches.


Another hint: https://packages.gentoo.org/packages/dev-util/checkbashisms
_________________
"The only difference between me and a madman is that I am not MAD" (SALVATOR DALI)
Back to top
View user's profile Send private message
Mr. T.
Guru
Guru


Joined: 26 Dec 2016
Posts: 477

PostPosted: Fri Mar 30, 2018 9:54 am    Post subject: Reply with quote

@i4dnf: Hu assumed that the scripts were not properly developed. I partially agree with him because the environment allows to create compliant scripts.
The mentioned problems would only be due to an incorrect shebang (cf. remark below), therefore, we come back to what I said:

helecho wrote:
@Hu: the original post is not precise enough to support your statement because scripts can be developed by anyone.
You probably assume that scripts developed by Gentoo are reliable. Moreover, an administrator is responsible for
scripts added. Finally, it is reasonable to assume that scripts developed have a correct shebang. Therefore,
I correctly considered the problem, at least partially.


Note: A lack of knowledge or incompetence of contributor(s).
Edit: The symbolic link allows to execute a POSIX compliant script (#!/bin/sh) with a POSIX capable interpreter.

helecho
Back to top
View user's profile Send private message
i4dnf
Apprentice
Apprentice


Joined: 18 Sep 2005
Posts: 251
Location: Bucharest, Romania

PostPosted: Fri Mar 30, 2018 10:29 am    Post subject: Reply with quote

Let's try this again:

Hu wrote:
  • A shell script with a shebang of /bin/sh may or may not contain "bashisms" (non-POSIX-sh extensions).
    1. If /bin/sh is an alias for GNU Bash, then a shell script that contains bashisms will work.
    2. If /bin/sh is not GNU Bash, then a shell script that contains bashisms will not work.




And:
Hu wrote:
  • Many systems use GNU Bash for /bin/sh. Testing a script that contains bashisms on such a system will succeed, even though it should not succeed. This pattern has enabled many scripts to develop bashisms that only troubled people who elected to use non-GNU Bash shells for their /bin/sh


So, no, in fact logic (and prior experience) dictates that:
helecho wrote:
it is NOT reasonable to assume that scripts developed have a correct shebang.
(correction and emphasis added)
_________________
"The only difference between me and a madman is that I am not MAD" (SALVATOR DALI)
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 5757
Location: Room 101

PostPosted: Fri Mar 30, 2018 10:53 am    Post subject: Reply with quote

helecho wrote:
GNU Bash runs in POSIX mode if the shebang is /bin/sh [...]

helecho ... then all that's needed is a definition of "POSIX mode"!!! ... because for bash '--posix' doesn't effect the point Hu was making: non-POSIX "bashisms" are accepted by /bin/sh => /bin/bash

Code:
% /bin/bash --posix
bash-4.4$ if [[ -n $HOME ]] ; then echo "non posix" ; fi
non posix
bash-4.4$ echo -e "non posix"
non posix
bash-4.4$ echo {non,posix}
non posix
bash-4.4$ bar="non posix" ; declare -u bar="$bar" ; echo $bar
NON POSIX
bash-4.4$ foo=1 ; let foo=${foo}+1 ; echo $foo
2
bash-4.4$ eselect sh list
Available POSIX shell implementations:
  [1]   bash
  [2]   dash *
  [3]   mksh
bash-4.4$ su -c 'eselect sh set 1'
bash-4.4$ ls -l /bin/sh
lrwxrwxrwx 1 root root 4 2018-03-30 12:52 /bin/sh -> bash*
bash-4.4$ echo 'foo=1 ; let foo=${foo}+1 ; echo $foo' > test.sh
bash-4.4$ /bin/sh ./test.sh
2

helecho wrote:
[...] and specific features are explicitly specified in the documentation.

The docs (man bash) may state that "[b]ash can be configured to be POSIX-conformant by default" and "--posix ... [c]hange the behavior of bash where the default operation differs from the POSIX standard to match the standard" but this means absolutely nothing when it comes to execution (as the above code illustrates clearly).

best ... khay
Back to top
View user's profile Send private message
Mr. T.
Guru
Guru


Joined: 26 Dec 2016
Posts: 477

PostPosted: Fri Mar 30, 2018 11:01 am    Post subject: Reply with quote

@4dfn: Please, do not replace words in a quotation!! It's a good way to create confusion.

needMoreFirmware wrote:
How viable is it?, the wiki states that some scripts on Gentoo have bashisms on them, I checked the init scripts at /etc/init.d/ with checkbashisms for a quick check and they all seemed to be fine, where else should I look for bashisms before trying to replace /bin/sh with dash?, or should I even try it at all?


The original poster asks whether it is safe to set Dash as the system interpreter.

needMoreFirmware wrote:
[...] the wiki states that some scripts on Gentoo [...]


needMoreFirmware wrote:
[...] where else should I look for bashisms [...]


The original poster does not know who developed scripts on his system.

needMoreFirmware wrote:
[...] I checked the init scripts at /etc/init.d/ with checkbashisms for a quick check [...]


The original poster wonders (asks) whether Gentoo scripts are safe.

helecho wrote:
@Hu: the original post is not precise enough to support your statement because scripts can be developed by anyone. [...]


Edit: fixed a "incorrect" word.

helecho


Last edited by Mr. T. on Fri Mar 30, 2018 12:04 pm; edited 1 time in total
Back to top
View user's profile Send private message
Naib
Watchman
Watchman


Joined: 21 May 2004
Posts: 5290
Location: Removed by Neddy

PostPosted: Fri Mar 30, 2018 11:07 am    Post subject: Reply with quote

*sigh* this discussion should have ended by post #3.
_________________
The best argument against democracy is a five-minute conversation with the average voter
Great Britain is a republic, with a hereditary president, while the United States is a monarchy with an elective king
Back to top
View user's profile Send private message
Mr. T.
Guru
Guru


Joined: 26 Dec 2016
Posts: 477

PostPosted: Fri Mar 30, 2018 11:14 am    Post subject: Reply with quote

@khayyam: GNU Bash properly considers the shebang because it adapts its operations: #!/bin/sh unlike a supposed contributor (shebang incorrectly used).

Edit: @Hu @i4dfn: I partially agree with you; it is a matter of perception. I do not want to talk about that for a decade.
Back to top
View user's profile Send private message
needMoreFirmware
n00b
n00b


Joined: 21 Mar 2018
Posts: 21

PostPosted: Fri Mar 30, 2018 12:48 pm    Post subject: Reply with quote

I'm genuinely confused at what is being discussed at this point, my question was simple, I wanted to know whether gentoo's scripts (and only gentoo's) contain an important amount of bashisms, important as in enough to break stability, since this wiki entry seems to indicate that it does contain some, there is no deeper analysis needed on what I did or did not mean.
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 5757
Location: Room 101

PostPosted: Fri Mar 30, 2018 1:12 pm    Post subject: Reply with quote

helecho wrote:
@khayyam: GNU Bash properly considers the shebang because it adapts its operations: #!/bin/sh unlike a supposed contributor (shebang incorrectly used).

helecho ... no, it clearly doesn't, unless you think that bashisms are somehow "posix". Anyhow, this discussion seems to be going nowhere fast, you're not actually engaging with what's presented but expecting that if you throw enough words, or statements, one after another, someone might make some sense of it.

To make that criticism absolutely concrete please answer this question: in what way can it be said that "bash properly considers the shebang because it adapts its operations" when the following (and what was presented previously) shows that this clearly isn't the case?

Code:
% cat test.sh
#!/bin/sh
foo=1 ; let foo=${foo}+1 ; echo $foo
% checkbashisms test.sh
possible bashism in test.sh line 2 (let ...):
foo=1 ; let foo=${foo}+1 ; echo $foo
% su -c 'eselect sh set 1'
% ls -l /bin/sh
lrwxrwxrwx 1 root root 4 2018-03-30 15:02 /bin/sh -> bash
% /bin/sh ./test.sh
2
% su -c 'eselect sh set 2'
% ls -l /bin/sh
lrwxrwxrwx 1 root root 4 2018-03-30 15:04 /bin/sh -> dash
% /bin/sh ./test.sh
test.sh: 2: test.sh: let: not found
1

best ... khay
Back to top
View user's profile Send private message
Naib
Watchman
Watchman


Joined: 21 May 2004
Posts: 5290
Location: Removed by Neddy

PostPosted: Fri Mar 30, 2018 1:22 pm    Post subject: Reply with quote

needMoreFirmware wrote:
I'm genuinely confused at what is being discussed at this point, my question was simple, I wanted to know whether gentoo's scripts (and only gentoo's) contain an important amount of bashisms, important as in enough to break stability, since this wiki entry seems to indicate that it does contain some, there is no deeper analysis needed on what I did or did not mean.
Ignore those scratching their ego's here.

What the wiki is saying is a word of warning because the norm is to have bash act as the sh interpreter and thus it is all too easy for someone to accidentally add bashism to a sh script. With sh -> bash this would work but sh -> dash this would not work.

If we look at gentoo and pretty much just @system... Portage needs bash, the ebuilds and portage makes extensive use of bashism's and this is why the associated scripts have #!/bin/bash and the ebuilds are sourced. OpenRC was developed by a BSD advodate and he wanted full POSIX compliance and thus mandated sh, true sh. All init script will have a #!/bin/sh shebang. They work with bash and should work with dash (if any don't please raise a bug as a bashism may have accidentally crept in)

so in setting sh -> dash your system should boot. HOWEVER... your system isn't just portage and openRC... it is what you want installed. A 3rd party app my have put #!/bin/sh but it was developed on a bash system and thus this might not work and this is is what hte wiki is warning about.

The best bet is to try with dash. Your system will boot and you MIGHT run into a script from another package that fails
_________________
The best argument against democracy is a five-minute conversation with the average voter
Great Britain is a republic, with a hereditary president, while the United States is a monarchy with an elective king
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 5757
Location: Room 101

PostPosted: Fri Mar 30, 2018 1:23 pm    Post subject: Reply with quote

needMoreFirmware wrote:
I'm genuinely confused at what is being discussed at this point, [...]

needMoreFirmware ... and you have a right to, there isn't much of a discussion, helecho is simply trying to bamboozle his/her way into your conciousness.

needMoreFirmware wrote:
[...] my question was simple, I wanted to know whether gentoo's scripts (and only gentoo's) contain an important amount of bashisms, important as in enough to break stability, since this wiki entry seems to indicate that it does contain some, there is no deeper analysis needed on what I did or did not mean.

Any bashism in an initscript would be considered a bug, and should be reported. That said, I've been using /bin/dash as posix shell for some time without issue, and though I have encountered bashisms in the past I don't remember the last time I did so. Having checked with checkbashisms I think you'll probably survive the experience :)

best ... khay
Back to top
View user's profile Send private message
Mr. T.
Guru
Guru


Joined: 26 Dec 2016
Posts: 477

PostPosted: Fri Mar 30, 2018 3:04 pm    Post subject: Reply with quote

@Naib @khayyam: your declarations are dissapointing and disrespectful: it seems to you equal that the interlocutor has a good faith; you advocate an offensive approach.

@khayyam: GNU Bash supports more features than a POSIX shell but is a POSIX capable shell. The documentation explicitly states what differs concerning POSIX features.
My statement you have previously quoted is not a criticism but a comment supporting my previous posts. My words as you interpret them are meaningless because you distort them.
I would even say your statement is meaningless: "non POSIX "bashisms" are accepted by Bash" (see below). Stop fantasizing: I will not certify anything and certainly not POSIX!

khayyam wrote:
helecho ... then all that's needed is a definition of "POSIX mode"!!! ... because for bash '--posix' doesn't effect the point Hu was making: non-POSIX "bashisms" are accepted by /bin/sh => /bin/bash


helecho
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo Chat All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum