POSIX sh - grab/proces stdout and stderr separately [SOLVED]

[Zucca]

So in bash I can do this kind of trickery:

Code: Select all

cmd1 > >(cmd2) 2> >(cmd3)

Which enables me to process stdout and stderr separately with cmd2 processing stdout and cmd3 processing stderr.

Can I do the ~same in POSIX sh? Without creating any temporary files?

[szatox]

I don't think it has context substitution.
You can try this though:
command1 2>/dev/stdout 1>/dev/stderr | grab_error_instead_of_out

You could also use named pipes AKA fifo. These are not real files, like in they don't actually store any data, but they do reside on the filesystem and can be open from both ends (for read and write respectively)

[geki]

I wonder if this will work. I started bash in posix mode to test.

Code: Select all

$ bash --posix
bash-5.2$ ( echo yea >&1; echo nay >&2; 2>&1 1>&3 | grep nay 1>&2 ) 3>&1 | cat -n
nay
     1	yea

And here something in motion.

Code: Select all

( while true; do echo yea >&1; echo nay >&2; done; 2>&1 1>&3 | grep nay 1>&2 ) 3>&1 | cat -n

Taken from the example of Joe Smith at https://www.quora.com/How-do-I-piping-stderr-in-UNIX/

How does it work... AFAIS.

• print yea to stdout as cmd1
• print nay to stderr as cmd1
• redirect stdout to temporary
• redirect stderr to stdout
• pocess stderr with grep as cmd3
• redirect stdout to stderr for cmd3 processing error stream
• redirect temporary to stdout
• process stdout with cat as cmd2

Fixed description of how it works.

[Zucca]

Yes.
I thought something like that too.

Code: Select all

{ cmd1 | grab_stdout; } 2>&1 | grab_stderr

And it seem to work :

Code: Select all

 { { echo stdout; echo stderr 1>&2; } | rev; } 2>&1 | cat

... although the lines come in reverse order. Maybe a race condition...

EDIT: Nope. cat grabs all.

[Zucca]

Thanks geki.
I'll try to "decipher" what's going on at the code there.
I will conduct some tests afterwards.

[szatox]

This bit is interesting

; 2>&1 1>&3 |

messing with file descriptors without providing any command? So, you're basically trying to swap output streams that belong to the shell itself rather than alter the flow between children?

Also, this thing is mindblowing

1>&3 ) 3>&1

It looks absurd, but apparently the redirection 3>&1 happens before creating subshell, and the subshell inherits fd 3 so it can use it for redirection, since it's already open from the outside... I really expected it to crash and burn, but () actually makes it work even though at the first glance it should make it fail.
Funny how it works

While I do like inlining stuff and using shortcuts like implicit relations, mkfifo /tmp/pipe$$.1 is not that hard to call.

[Zucca]

I'm trying the wrap my head around this.
Apparently you really need subshell. Grouping commands won't do it.

[pingtoo]

I am sorry to interrupt, but I am really curious what is motive behind this.

Is this a existing piece of code just trying understand what is doing? Or trying to fit a complex operation into single line of code as challenge?

I found it from the conversation in this thread that it seems to be difficult to understand the operation itself. So why do it?

I am in favour of trying to express the operation in script that as plain as statement that 5 years old can understand even it will take a few more sentences to help make it easier to understand. So why not use named pipe to solve it?

[geki]

We just do basic standard stream redirection in shell. As stated in first post, process stdout and stderr of a command with other commands separately.
So, from bash

Code: Select all

cmd1 > >(cmd2) 2> >(cmd3)

to posix shell

Code: Select all

( cmd1 2>&1 1>&3 | cmd3 1>&2 ) 3>&1 | cmd2

• cmd1 - command to execute
• cmd2 - process stdout of cmd1
• cmd3 - process stderr of cmd1

[geki]

I'm trying the wrap my head around this.
Apparently you really need subshell. Grouping commands won't do it.

If I am not just wrong you do 2 subshells in your bash example.

Code: Select all

>(...)

is a subshell invocation. Or what is wrong with the subshells? editA ha, this is process substitution. heh.

[Zucca]

what is motive behind this.

• To grab normal informative messages and error messages from a program as separate streams
• process them
• pass to logger with appropriate arguments (and --prio-prefix which needs the processing above)

I hope to get at least informational messages (stdout) and warnings (stderr) passed to a logger process. Then maybe later try to match words like 'error', 'alert' and so on from stderr and mark them as errors.

The problem with just doing

Code: Select all

somecommand 2>&1 | messageprocessing | logger <--switches> --prio-prefix

... is that I cannot reliably distinguish stderr from stdout anymore.

And also there are many other places where I could use some POSIX sh (or close to POSIX compatible) instead of bash if this can be accomplished with plain sh.

[Zucca]

If I am not just wrong you do 2 subshells in your bash example.

Yes. In this case I think it's mandatory.
I only tried with command grouping and it failed. Obviously.

[pingtoo]

Correct me if I am wrong, I may have misinterpreting the command line wrong.

Code: Select all

TMPDIR=/tmp/mytmpd.$$
mkdir $TMPDIR
trap "rm -rf $TMPDIR" 0 1 2 3 15
mknod $TMPDIR/out p
mknod $TMPDIR/err p
(cmd2 < $TMPDIR/out)&
(cmd3 < $TMPDIR/err)&
exec 1>$TMPDIR/out
exec 2>$TMPDIR/err
cmd1()
{
  echo "normal message"
  echo "error message" >2
}
cmd1

I group "cmd2" and "cmd3" in the spirit of your command. They don't have to be sub-shell in my example code.

I believe my example code can work in any unix/linux environment with posix shell.

[Hu]

The nice thing about the process substitution (bash-specific) approach, and any emulation of that in POSIX sh, is that it can work without a writable filesystem on which to place the FIFO nodes. This can be useful in some container-type environments, which often bind almost everything as read-only, or in the early boot of a system before it has remounted the root filesystem as read-write. Also, by not creating FIFO nodes, there is nothing to explicitly clean up at exit, so you do not need a cleanup handler on exit, nor need to deal with cleaning out any garbage left from a prior run that was killed without the opportunity to run its cleanup handler.

[Zucca]

I believe my example code can work in any unix/linux environment with posix shell.

Yes. It should work.
Now try without creating any files.

EDIT: I tried geki's example in few different ways and it really works. Amazing. Thank you!

posix shell

Code: Select all

( cmd1 2>&1 1>&3 | cmd3 1>&2 ) 3>&1 | cmd2

Now I'm left to wonder why bash (--posix) so often complained about bad fd 3 even when I had both ends open...
I guess I still haven't wrapped my head around this completely.
Some people say say starting to write shell code is easy, but to master it all is mind-bendingly hard. I think along those lines too now.

EDIT2: It doesn't seem to work with busybox sh. However since it worked with bash --posix I call this [solved].

EDIT3: Busybox's shell lies somewhere between ash and posix. However what I noticed was that cmd1 stdout and stderr, both get mixed into one "stream" at the very beginning. I don't know if it's a bug.

[Goverp]

what is motive behind this.

• To grab normal informative messages and error messages from a program as separate streams
• process them
• pass to logger with appropriate arguments

...

I've been playing the same game with the same results. Works fine. I guess if you write to syslog you should get decent timestamps (though of course the ts is for when logger sees the message, not when the message was created, but that's always an insoluble problem here).

In my case I was going to write to a private log file (this is for KDE X11/plasma console output). My plan was to add ANSI colours to stderr output, and then merge the two streams on timestamp back into a single colourful log. The standard awk script to stick a timestamp on works well (there's an item somewhere in the forums, but I can't find it quickly), but only gives to the nearest second, which isn't sufficient in these days of multi-GHz processors. There's an awk function to get nanosecond times, but it's packaged in a new improved way that means it's not in portage ATM.

[Zucca]

I'm starting to think busybox sh fd redirections are broken.
So I tried to find any solution that works. Nada. I had hoped some combination would work. Then when one is found try it in bash --posix.
Here's one example:

Code: Select all

( sample 2>&3 | cat 1>&2 ) 3>&1 | cat -n

'sample' being simple function which echoes to stdout and stderr.
What happens there (and with many other derivatives of this) is that cat catches stdout and stderr and nothing is passed via cat -n while the final output is all from stdout. What the wtf?

So if I use this trick and I want to use some shebang I need to either

• assume /bin/sh isn't ever ash or busybox sh
• have /bin/bash --posix or something else posixy compatible as shebang

I don't know... Now I could just leave my script(s) as bash. *deep sigh*

[geki]

Now I'm left to wonder why bash (--posix) so often complained about bad fd 3 even when I had both ends open...
EDIT2: It doesn't seem to work with busybox sh. However since it worked with bash --posix I call this [solved].

I'm starting to think busybox sh fd redirections are broken.

Hmm, I tried BusyBox v1.35.0 (Debian 1:1.35.0-4+b3) built-in shell (ash) and my example seems to work. How do you invoke your commands to get bad fds? How frequently? The last example looks wrong to me.

[Zucca]

Hmm, I tried BusyBox v1.35.0 (Debian 1:1.35.0-4+b3) built-in shell (ash) and my example seems to work. How do you invoke your commands to get bad fds? How frequently? The last example looks wrong to me.

Code: Select all

$ eshowkw -O busybox
Keywords for sys-apps/busybox:
             |                               |   u   |  
             | a   a     p s     l r   a     |   n   |  
             | m   r h   p p   i o i s l m m | e u s | r
             | d a m p p c a x a o s 3 p 6 i | a s l | e
             | 6 r 6 p p 6 r 8 6 n c 9 h 8 p | p e o | p
             | 4 m 4 a c 4 c 6 4 g v 0 a k s | i d t | o
-------------+-------------------------------+-------+-------
[I]1.34.1-r2 | + + + + + + + + ~ ~ ~ ~ ~ ~ ~ | 7 o 0 | gentoo
   1.35.0-r2 | o o o o o o o o o o o o o o o | 7 #   | gentoo
   1.36.1    | ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ | 8 o   | gentoo
     9999    | o o o o o o o o o o o o o o o | 8 o   | gentoo

Hm. IIRC 1.34.x caused segfaults on musl on my other machine. I better keyword 1.36.1... Maybe there are more fixes?
Off to keywordin' 'n' compilin' ...

[Zucca]

Nope.

Code: Select all

$ ( sample 2>&1 1>&3 | cat -n 1>&2 ) 3>&1 | rev
egassem lamron
egassem rorre

cat -n gets nothing.

Code: Select all

 $ busybox | head -n 1; qlist -vI busybox
BusyBox v1.36.1 (2023-07-05 23:17:13 -00) multi-call binary.
sys-apps/busybox-1.36.1

... although... Neither the bash --posix work... anymore?
Yeah... I need to rewind a bit to the point where I managed to get this working. Maybe tomorrow...

EDIT: No. Now. Got it working. On both shells. I had 1's and 2's mixed... so I redirected stderr into stdout, not the other way around. <insert facepalm picture here>

[szatox]

sh-5.2$ ( echo rofl | cat -n >&2 ) 3>&1 | rev
1 rofl
sh-5.2$ ( echo rofl 2>&1 1>&3 | cat -n >&2 ) 3>&1 | rev
lfor

cat -n gets stdout, rev gets stderr.
I have cat -n output to stderr to remove this output from pipe instead of directing it to rev.
Using subshells for those tests does make this code terribly messy, but the redirection itself #worksforme #asintended

[Zucca]

Well. I managed to code a little wrapper for logger in sh. That was fun. Although start-stop-daemon should be a better choice in most cases.