Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Opinions/tips - is this AWK shebang a bad idea?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3345
Location: Rasi, Finland

PostPosted: Mon Feb 19, 2018 11:02 am    Post subject: Opinions/tips - is this AWK shebang a bad idea? Reply with quote

I wanted to be able to run a awk script "user friendly" way by just giving it +x.
However I noticed that if I used the common shebang of #!/usr/bin/awk -f it fails if switches are being given as arguments - awk itself grabs the switch (and usually doesn't recognize it).
So I had a need to be able to have a shebang like #!/usr/bin/awk -f <script> -- [arguments can start with dash here].
That's not possible on "normal ways" so with little thinking I created this monster:
awk shebang:
#!/usr/bin/awk {for (i in ARGV) {if (i < 2) continue; A = A " '" ARGV[i] "'"} system("awk -f " FILENAME " -- " A); exit}

Does anyone see anything that's wrong there? Other than it looks pretty ugly. :P Is there a simpler way?
I tested and it can even handle arguments that have whitespaces.

Here's a test script I created:
Code:
#!/usr/bin/awk {for (i in ARGV) {if (i < 2) continue; A = A " '" ARGV[i] "'"} system("awk -f " FILENAME " -- " A); exit}

BEGIN {
    print "Arguments passed:"
    for (i in ARGV) printf "'" ARGV[i] "' "
    print ""
}


Do anyone see any problems that might occur by using this complicated shebang?
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
Naib
Watchman
Watchman


Joined: 21 May 2004
Posts: 6051
Location: Removed by Neddy

PostPosted: Mon Feb 19, 2018 12:10 pm    Post subject: Reply with quote

Have you tried /use/bin/env awk as the shebang? Iirc there were a number of benefits using env over explicit binary referencing
_________________
Quote:
Removed by Chiitoo
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3345
Location: Rasi, Finland

PostPosted: Mon Feb 19, 2018 12:55 pm    Post subject: Reply with quote

Naib wrote:
Have you tried /use/bin/env awk as the shebang?
Good point. At least it makes the script little more portable.
I'm gonna run tests with that too. Thanks.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3345
Location: Rasi, Finland

PostPosted: Mon Feb 19, 2018 6:08 pm    Post subject: Reply with quote

Hmmm... Strange. When using env to run that awk shebang the awk process falls into infinite loop. I also tried with #!/usr/bin/env awk -- <rest of the shebang>, but same result.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
Naib
Watchman
Watchman


Joined: 21 May 2004
Posts: 6051
Location: Removed by Neddy

PostPosted: Mon Feb 19, 2018 7:24 pm    Post subject: Reply with quote

Ok crud solution *if* the awk arguements could change...
Have the entire thing as a bash script, this consumes the arguements, the awk script is then a docstring within the bash consumer and now $1 can be passed to awk
_________________
Quote:
Removed by Chiitoo
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6227
Location: Room 101

PostPosted: Mon Feb 19, 2018 8:19 pm    Post subject: Reply with quote

Zucca ...

not sure what you mean by "awk itself grabs the switch (and usually doesn't recognize it)", I can pass '-v' (see below). Anyhow, awk "programs" do not have a shebang, by adding one the interpreter is the shell (which then runs 'awk'), so they must be written in a way the interpreter understands (so, much the same as you might provide it "on the commanline"):

Code:
% cat <<- EOF > test.awk
heredocd> #!/usr/bin/awk -f
heredocd> BEGIN { print "Arguments passed:" ;
heredocd>    for (i in ARGV) printf "'" ARGV[i] "' " ;
heredocd>    print foo }
heredocd> EOF
% chmod u+x test.awk
% ./test.awk -v foo=ba a b c
Arguments passed:
'awk' 'a' 'b' 'c' ba

Otherwise, you can use AWKPATH for awk "programs" ...

Code:
% echo $AWKPATH
/home/khayyam/.awk
% cat <<- EOF > $AWKPATH/test.awk
heredocd> BEGIN {
heredocd>    print "Arguments passed:"
heredocd>    for (i in ARGV) printf "'" ARGV[i] "' "
heredocd>    print foo
heredocd> EOF
% awk -f test a b c
Arguments passed:
'awk' 'a' 'b' 'c'
% awk -f test -v foo=ba a b c
Arguments passed:
'awk' 'a' 'b' 'c' ba

HTH & best ... khay
Back to top
View user's profile Send private message
1clue
Advocate
Advocate


Joined: 05 Feb 2006
Posts: 2569

PostPosted: Mon Feb 19, 2018 8:59 pm    Post subject: Reply with quote

The shebang is supposed to be the path to the app/shell/language interpreter/whatever which is to be used to execute the rest of the script.

The first line (the one with the shebang) is interpreted by your shell (presumably your login shell but that would depend on if another shell is in use within that). I believe you may get different results if your shell is bash or sh or csh or any other value.

There is a well-defined way to pass arguments to these scripts. I'm positive that passing arguments correctly by the correct mechanism is extensively tested and very consistent across shells and interpreters included with a typical distro.

All that aside, I can see no benefit to doing things the way they are being shown here. Because you can? Code obfuscation? An addiction to one-line complex solutions?
Back to top
View user's profile Send private message
1clue
Advocate
Advocate


Joined: 05 Feb 2006
Posts: 2569

PostPosted: Mon Feb 19, 2018 9:01 pm    Post subject: Reply with quote

On top of that, I think the exit value will possibly be the numeric result of the shebang, not the exit result of the script.
Back to top
View user's profile Send private message
Naib
Watchman
Watchman


Joined: 21 May 2004
Posts: 6051
Location: Removed by Neddy

PostPosted: Mon Feb 19, 2018 9:35 pm    Post subject: Reply with quote

Shebang is interpreted by the kernel iirc


Edit. Yup just reconfirmed, by the kernel and this is why there are limitations on additional arguments AND equally why gentoo devs had to write their own python elf to wrap around a py script shebang to intercept it correctly

As per the kernel docu, they recommend a wrapper as I suggested


If you want to pass special arguments to your interpreter, you can write a wrapper script for it. See Documentation/admin-guide/java.rst for an example.
_________________
Quote:
Removed by Chiitoo
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3345
Location: Rasi, Finland

PostPosted: Mon Feb 19, 2018 10:20 pm    Post subject: Reply with quote

khayyam wrote:
Zucca ...

not sure what you mean by "awk itself grabs the switch (and usually doesn't recognize it)", I can pass '-v' (see below).
... and that's the problem. awk takes the -v and foo=ba along with it while I want them to be passed as-is to the script.
Following is the exact way I want it:
Code:
./shebang.bang -v foo=ba a b c
Arguments passed:
'awk' '-v' 'foo=ba' 'a' 'b' 'c'
So no awk picking up that -v there. (ARGV[0] is always the awk interpreter)

And as you could see
khayyam wrote:
Code:
% ./test.awk -v foo=ba a b c
Arguments passed:
'awk' 'a' 'b' 'c' ba
... there '-v' as an argument is absent and so is 'foo=ba'.
(Rather strangely ba is then printed at the end. Since awk grabbing -v should result only in predefined variable, foo, inside awk environment.)

Now if you have the common shebang of #!/usr/bin/awk -f then kernel executes: /usr/bin/awk -f ./shebang.bang -v foo=ba a b c
If I have that ugly shebang then kernel executes: /usr/bin/awk -f ./shebang.bang -- -v foo=ba a b c
To be precise, kernel executes the awk command in the shebang which in turn executes the main awk process that actually runs the script.

Btw: you all propably already knew what I just told, but just in case of clarity of what I'm trying to achieve here...

Anyway. A little correction to my original post here: (and usually doesn't recognize it) was wrong. awk grabs those arguments (switches) it can recognize. For example '--help'. If I pass help as an argument for my awk script, then awk actually prints its own help text rather than passing '--help' to my script.
Actually writing code for '--help' switch and didn't get it working was the original reason for me trying to come up with some clever "new" shebang. ;)
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3345
Location: Rasi, Finland

PostPosted: Mon Feb 19, 2018 10:25 pm    Post subject: Reply with quote

Naib wrote:
As per the kernel docu, they recommend a wrapper as I suggested


If you want to pass special arguments to your interpreter, you can write a wrapper script for it. See Documentation/admin-guide/java.rst for an example.
Yup. That was my intention at first. Then I thought if it was possible only by using shebang.
Yes, it is. But very ugly. But since it's the kernel directly executing the shebang command (where '#!' is seen as magic number by kernel, afaik)... it should be quite portable among different Linux distros...? Right?

1clue wrote:
On top of that, I think the exit value will possibly be the numeric result of the shebang, not the exit result of the script.
Very good point. That (long) shebang is running two concurrewnt awk processes... Handling exit status could be pita. Dang...
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3345
Location: Rasi, Finland

PostPosted: Tue Feb 20, 2018 11:39 am    Post subject: Reply with quote

I've resolved the exit status issue.
ugly shebang:
#!/usr/bin/awk {for (i in ARGV) {if (i < 2) continue; A = A " '" ARGV[i] "'"} exit system("awk -f " FILENAME " -- " A)}
Tested it and works as expected. Basically whatever "system()" returns is also passed to the "exit". So now it's only up the the script writer to have proper exit codes in his/her code. ;)
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
1clue
Advocate
Advocate


Joined: 05 Feb 2006
Posts: 2569

PostPosted: Tue Feb 20, 2018 2:39 pm    Post subject: Reply with quote

Zucca wrote:
I've resolved the exit status issue.
ugly shebang:
#!/usr/bin/awk {for (i in ARGV) {if (i < 2) continue; A = A " '" ARGV[i] "'"} exit system("awk -f " FILENAME " -- " A)}
Tested it and works as expected. Basically whatever "system()" returns is also passed to the "exit". So now it's only up the the script writer to have proper exit codes in his/her code. ;)


Now, since it has been established that the shebang is executed by the kernel, why don't you test if this bit runs as root? It could be a really bad exploit.
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3345
Location: Rasi, Finland

PostPosted: Tue Feb 20, 2018 3:21 pm    Post subject: Reply with quote

1clue wrote:
Now, since it has been established that the shebang is executed by the kernel, why don't you test if this bit runs as root? It could be a really bad exploit.
By using shebang of
Code:
#!/usr/bin/awk {print "I am " ENVIRON["USER"]; exit}
... it returns login name of the current user, not root (and does not execute the script at all).
I think if this was a problem it would have been already exploited all over the place and patched too.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
R0b0t1
Apprentice
Apprentice


Joined: 05 Jun 2008
Posts: 264

PostPosted: Tue Feb 27, 2018 3:22 am    Post subject: Reply with quote

Did you ever get awk scripts running cleanly with the interpreter line? I'm not sure I understand your post containing the following:
Code:
#!/usr/bin/awk {for (i in ARGV) {if (i < 2) continue; A = A " '" ARGV[i] "'"} exit system("awk -f " FILENAME " -- " A)}
This implies you need to specify code for awk in the interpreter line to have it behave like a normal program with a return code? What is this doing? It's one line? It might be worth opening a request for awk/gawk to support something that works more cleanly.
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3345
Location: Rasi, Finland

PostPosted: Wed Feb 28, 2018 8:15 pm    Post subject: Reply with quote

R0b0t1 wrote:
This implies you need to specify code for awk in the interpreter line to have it behave like a normal program with a return code? What is this doing? It's one line? It might be worth opening a request for awk/gawk to support something that works more cleanly.
Basically if I were to wrote an awk script (one that uses #!/usr/bin/awk -f as shebang) that reads command line switches, and one of them being --help, for example. Then if user runs ./some_script.awk --help, the script doesn't even get run, but the awk interpreter will output its help message (same as running awk --help). If any of the switches aren't known to the awk interpreter, then the script is run as expected.
So the ugly shebang I've created actually runs awk -f ./some_script.awk -- --help.

If there really isn't any more clean way, then, yes, a bug report should be filled...
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3345
Location: Rasi, Finland

PostPosted: Tue Mar 12, 2019 11:56 pm    Post subject: Reply with quote

It seems gawk has a --exec switch to circumvent the problem.

So...
Code:
#!/usr/bin/gawk --exec
should be enough, assuming the system has gawk.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum