View previous topic :: View next topic |
Author |
Message |
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3345 Location: Rasi, Finland
|
Posted: Mon Feb 19, 2018 11:02 am Post subject: Opinions/tips - is this AWK shebang a bad idea? |
|
|
I wanted to be able to run a awk script "user friendly" way by just giving it +x.
However I noticed that if I used the common shebang of #!/usr/bin/awk -f it fails if switches are being given as arguments - awk itself grabs the switch (and usually doesn't recognize it).
So I had a need to be able to have a shebang like #!/usr/bin/awk -f <script> -- [arguments can start with dash here].
That's not possible on "normal ways" so with little thinking I created this monster: awk shebang: | #!/usr/bin/awk {for (i in ARGV) {if (i < 2) continue; A = A " '" ARGV[i] "'"} system("awk -f " FILENAME " -- " A); exit} |
Does anyone see anything that's wrong there? Other than it looks pretty ugly. Is there a simpler way?
I tested and it can even handle arguments that have whitespaces.
Here's a test script I created: Code: | #!/usr/bin/awk {for (i in ARGV) {if (i < 2) continue; A = A " '" ARGV[i] "'"} system("awk -f " FILENAME " -- " A); exit}
BEGIN {
print "Arguments passed:"
for (i in ARGV) printf "'" ARGV[i] "' "
print ""
} |
Do anyone see any problems that might occur by using this complicated shebang? _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
Naib Watchman
Joined: 21 May 2004 Posts: 6051 Location: Removed by Neddy
|
Posted: Mon Feb 19, 2018 12:10 pm Post subject: |
|
|
Have you tried /use/bin/env awk as the shebang? Iirc there were a number of benefits using env over explicit binary referencing _________________
Quote: | Removed by Chiitoo |
|
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3345 Location: Rasi, Finland
|
Posted: Mon Feb 19, 2018 12:55 pm Post subject: |
|
|
Naib wrote: | Have you tried /use/bin/env awk as the shebang? | Good point. At least it makes the script little more portable.
I'm gonna run tests with that too. Thanks. _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3345 Location: Rasi, Finland
|
Posted: Mon Feb 19, 2018 6:08 pm Post subject: |
|
|
Hmmm... Strange. When using env to run that awk shebang the awk process falls into infinite loop. I also tried with #!/usr/bin/env awk -- <rest of the shebang>, but same result. _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
Naib Watchman
Joined: 21 May 2004 Posts: 6051 Location: Removed by Neddy
|
Posted: Mon Feb 19, 2018 7:24 pm Post subject: |
|
|
Ok crud solution *if* the awk arguements could change...
Have the entire thing as a bash script, this consumes the arguements, the awk script is then a docstring within the bash consumer and now $1 can be passed to awk _________________
Quote: | Removed by Chiitoo |
|
|
Back to top |
|
|
khayyam Watchman
Joined: 07 Jun 2012 Posts: 6227 Location: Room 101
|
Posted: Mon Feb 19, 2018 8:19 pm Post subject: |
|
|
Zucca ...
not sure what you mean by "awk itself grabs the switch (and usually doesn't recognize it)", I can pass '-v' (see below). Anyhow, awk "programs" do not have a shebang, by adding one the interpreter is the shell (which then runs 'awk'), so they must be written in a way the interpreter understands (so, much the same as you might provide it "on the commanline"):
Code: | % cat <<- EOF > test.awk
heredocd> #!/usr/bin/awk -f
heredocd> BEGIN { print "Arguments passed:" ;
heredocd> for (i in ARGV) printf "'" ARGV[i] "' " ;
heredocd> print foo }
heredocd> EOF
% chmod u+x test.awk
% ./test.awk -v foo=ba a b c
Arguments passed:
'awk' 'a' 'b' 'c' ba |
Otherwise, you can use AWKPATH for awk "programs" ...
Code: | % echo $AWKPATH
/home/khayyam/.awk
% cat <<- EOF > $AWKPATH/test.awk
heredocd> BEGIN {
heredocd> print "Arguments passed:"
heredocd> for (i in ARGV) printf "'" ARGV[i] "' "
heredocd> print foo
heredocd> EOF
% awk -f test a b c
Arguments passed:
'awk' 'a' 'b' 'c'
% awk -f test -v foo=ba a b c
Arguments passed:
'awk' 'a' 'b' 'c' ba |
HTH & best ... khay |
|
Back to top |
|
|
1clue Advocate
Joined: 05 Feb 2006 Posts: 2569
|
Posted: Mon Feb 19, 2018 8:59 pm Post subject: |
|
|
The shebang is supposed to be the path to the app/shell/language interpreter/whatever which is to be used to execute the rest of the script.
The first line (the one with the shebang) is interpreted by your shell (presumably your login shell but that would depend on if another shell is in use within that). I believe you may get different results if your shell is bash or sh or csh or any other value.
There is a well-defined way to pass arguments to these scripts. I'm positive that passing arguments correctly by the correct mechanism is extensively tested and very consistent across shells and interpreters included with a typical distro.
All that aside, I can see no benefit to doing things the way they are being shown here. Because you can? Code obfuscation? An addiction to one-line complex solutions? |
|
Back to top |
|
|
1clue Advocate
Joined: 05 Feb 2006 Posts: 2569
|
Posted: Mon Feb 19, 2018 9:01 pm Post subject: |
|
|
On top of that, I think the exit value will possibly be the numeric result of the shebang, not the exit result of the script. |
|
Back to top |
|
|
Naib Watchman
Joined: 21 May 2004 Posts: 6051 Location: Removed by Neddy
|
Posted: Mon Feb 19, 2018 9:35 pm Post subject: |
|
|
Shebang is interpreted by the kernel iirc
Edit. Yup just reconfirmed, by the kernel and this is why there are limitations on additional arguments AND equally why gentoo devs had to write their own python elf to wrap around a py script shebang to intercept it correctly
As per the kernel docu, they recommend a wrapper as I suggested
If you want to pass special arguments to your interpreter, you can write a wrapper script for it. See Documentation/admin-guide/java.rst for an example. _________________
Quote: | Removed by Chiitoo |
|
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3345 Location: Rasi, Finland
|
Posted: Mon Feb 19, 2018 10:20 pm Post subject: |
|
|
khayyam wrote: | Zucca ...
not sure what you mean by "awk itself grabs the switch (and usually doesn't recognize it)", I can pass '-v' (see below). | ... and that's the problem. awk takes the -v and foo=ba along with it while I want them to be passed as-is to the script.
Following is the exact way I want it: Code: | ./shebang.bang -v foo=ba a b c
Arguments passed:
'awk' '-v' 'foo=ba' 'a' 'b' 'c' | So no awk picking up that -v there. (ARGV[0] is always the awk interpreter)
And as you could see khayyam wrote: | Code: | % ./test.awk -v foo=ba a b c
Arguments passed:
'awk' 'a' 'b' 'c' ba |
| ... there '-v' as an argument is absent and so is 'foo=ba'.
(Rather strangely ba is then printed at the end. Since awk grabbing -v should result only in predefined variable, foo, inside awk environment.)
Now if you have the common shebang of #!/usr/bin/awk -f then kernel executes: /usr/bin/awk -f ./shebang.bang -v foo=ba a b c
If I have that ugly shebang then kernel executes: /usr/bin/awk -f ./shebang.bang -- -v foo=ba a b c
To be precise, kernel executes the awk command in the shebang which in turn executes the main awk process that actually runs the script.
Btw: you all propably already knew what I just told, but just in case of clarity of what I'm trying to achieve here...
Anyway. A little correction to my original post here: (and usually doesn't recognize it) was wrong. awk grabs those arguments (switches) it can recognize. For example '--help'. If I pass help as an argument for my awk script, then awk actually prints its own help text rather than passing '--help' to my script.
Actually writing code for '--help' switch and didn't get it working was the original reason for me trying to come up with some clever "new" shebang. _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3345 Location: Rasi, Finland
|
Posted: Mon Feb 19, 2018 10:25 pm Post subject: |
|
|
Naib wrote: | As per the kernel docu, they recommend a wrapper as I suggested
If you want to pass special arguments to your interpreter, you can write a wrapper script for it. See Documentation/admin-guide/java.rst for an example. | Yup. That was my intention at first. Then I thought if it was possible only by using shebang.
Yes, it is. But very ugly. But since it's the kernel directly executing the shebang command (where '#!' is seen as magic number by kernel, afaik)... it should be quite portable among different Linux distros...? Right?
1clue wrote: | On top of that, I think the exit value will possibly be the numeric result of the shebang, not the exit result of the script. | Very good point. That (long) shebang is running two concurrewnt awk processes... Handling exit status could be pita. Dang... _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3345 Location: Rasi, Finland
|
Posted: Tue Feb 20, 2018 11:39 am Post subject: |
|
|
I've resolved the exit status issue. ugly shebang: | #!/usr/bin/awk {for (i in ARGV) {if (i < 2) continue; A = A " '" ARGV[i] "'"} exit system("awk -f " FILENAME " -- " A)} | Tested it and works as expected. Basically whatever "system()" returns is also passed to the "exit". So now it's only up the the script writer to have proper exit codes in his/her code. ;) _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
1clue Advocate
Joined: 05 Feb 2006 Posts: 2569
|
Posted: Tue Feb 20, 2018 2:39 pm Post subject: |
|
|
Zucca wrote: | I've resolved the exit status issue. ugly shebang: | #!/usr/bin/awk {for (i in ARGV) {if (i < 2) continue; A = A " '" ARGV[i] "'"} exit system("awk -f " FILENAME " -- " A)} | Tested it and works as expected. Basically whatever "system()" returns is also passed to the "exit". So now it's only up the the script writer to have proper exit codes in his/her code. |
Now, since it has been established that the shebang is executed by the kernel, why don't you test if this bit runs as root? It could be a really bad exploit. |
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3345 Location: Rasi, Finland
|
Posted: Tue Feb 20, 2018 3:21 pm Post subject: |
|
|
1clue wrote: | Now, since it has been established that the shebang is executed by the kernel, why don't you test if this bit runs as root? It could be a really bad exploit. | By using shebang of Code: | #!/usr/bin/awk {print "I am " ENVIRON["USER"]; exit} | ... it returns login name of the current user, not root (and does not execute the script at all).
I think if this was a problem it would have been already exploited all over the place and patched too. _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
R0b0t1 Apprentice
Joined: 05 Jun 2008 Posts: 264
|
Posted: Tue Feb 27, 2018 3:22 am Post subject: |
|
|
Did you ever get awk scripts running cleanly with the interpreter line? I'm not sure I understand your post containing the following: Code: | #!/usr/bin/awk {for (i in ARGV) {if (i < 2) continue; A = A " '" ARGV[i] "'"} exit system("awk -f " FILENAME " -- " A)} | This implies you need to specify code for awk in the interpreter line to have it behave like a normal program with a return code? What is this doing? It's one line? It might be worth opening a request for awk/gawk to support something that works more cleanly. |
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3345 Location: Rasi, Finland
|
Posted: Wed Feb 28, 2018 8:15 pm Post subject: |
|
|
R0b0t1 wrote: | This implies you need to specify code for awk in the interpreter line to have it behave like a normal program with a return code? What is this doing? It's one line? It might be worth opening a request for awk/gawk to support something that works more cleanly. | Basically if I were to wrote an awk script (one that uses #!/usr/bin/awk -f as shebang) that reads command line switches, and one of them being --help, for example. Then if user runs ./some_script.awk --help, the script doesn't even get run, but the awk interpreter will output its help message (same as running awk --help). If any of the switches aren't known to the awk interpreter, then the script is run as expected.
So the ugly shebang I've created actually runs awk -f ./some_script.awk -- --help.
If there really isn't any more clean way, then, yes, a bug report should be filled... _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3345 Location: Rasi, Finland
|
Posted: Tue Mar 12, 2019 11:56 pm Post subject: |
|
|
It seems gawk has a --exec switch to circumvent the problem.
So... Code: | #!/usr/bin/gawk --exec | should be enough, assuming the system has gawk. _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
|