| View previous topic :: View next topic |
| Author |
Message |
jagdpanther
l33t
Joined: 22 Nov 2003
Posts: 729
|
 Posted: Fri Jan 06, 2017 3:34 pm Post subject: How Forward Data Stream Stdin Within Bash or Python Script |
 |
|
There is a closed-source backup program that I use that will optionally call 2nd program, a compressor, with a possible argument, and then send that 2nd program a large amount of data (GB) on Stdout. I want to write a little script (Bash or Python) and forward the stdin being sent to my script to the program I call (7za) from within my script. Data flow looks like this:
Backup ( uncompressed stdout) --> myscript (calls 1 of 2 7za commands) (stdout) --> (compressed stdin) Backup
The backup program allows me to choose my compression program. I want to set arguments on that compression program.
Question: within myscript how do I forward the data stream stdin being sent to my script to the command (7za) I am executing from within myscript?
(I can not just use a pipe because there is a little logic in myscript depending on the backup program argument to compress or decompress.)
The following within myscript does not seem to work
7za a -si -so <0
or
7za x -si -so <0 |
|
| Back to top |
|
 |
cboldt
Veteran
Joined: 24 Aug 2005
Posts: 1046
|
| Posted: Fri Jan 06, 2017 3:55 pm Post subject: |
|
|
I think this gets easy if you include the closed-source program in your script and put the pipeline in your script
| Code: | #!/bin/sh
compress_pgm=/usr/bin/bzip2
compress_parm1=$1
compress_parm2=$2
closed_source_backup=/usr/local/bin/closed-source-backup
$closed_source_backup | $compress_pgm $compress_parm1 $compress_parm2 |
There are many ways to pass parameters from the command line to routines inside the script, the above is simple but doesn't trap errors. Also, the pipeline above doesn't handle the output from the compression program.
If you want to sometimes include the $compress_pgm, and other times not, you could choose between pipelines depending on the contents (or absence of contents) of a script command line parameter. |
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Fri Jan 06, 2017 4:34 pm Post subject: |
|
|
jagdpanther ...
I don't think 7z has implimented '-so' for 7z compression, following the manpage you would expect both of the following to produce the required output, but 7z (the default if no '-t<type>' is provided) fails.
| Code: | # echo foo | 7z a dummy -tgzip -si -so > out.gz
# file out.gz
out.gz: gzip compressed data, last modified: Fri Jan 6 16:25:07 2017, max speed, from Unix
# echo foo | 7z a dummy -t7z -si -so > out.7z
System ERROR:
E_NOTIMPL
# file out.7z
out.7z: empty |
That said, why do you need it to go to stdout, rather than write to 'file.7z'?
best ... khay |
|
| Back to top |
|
|
jagdpanther
l33t
Joined: 22 Nov 2003
Posts: 729
|
| Posted: Fri Jan 06, 2017 7:46 pm Post subject: |
|
|
cboldt, khayyam thanks for the quick replies.
cboldt: There is output from the backup program that I need and not trapping errors would be an issue.
khayyam:
> That said, why do you need it to go to stdout, rather than write to 'file.7z'?
Because the backup program also encrypts the backup, so after the compression, the backup program then encrypts the data and writes a checksum to the archive, block by block. (This is a nice feature, because if part of the backup archive becomes corrupt, which I have simulated by using hexedit to randomy change some bytes within a backup, most of the archive is still recoverable.)
From the backup software manual, about choosing an external compression engine:
| Quote: | provide a mechanism which allows you to utilize an external
compression engine instead of the built in LZO engine |
| Quote: | The only requirement is that the compression engine accepts
stdin as th compression source and send the resulting compressed data to
stdout. Also, you must be able to call the engine for decompression using a -d
option to the original program (i.e.: bzip2 -d). |
> I don't think 7z has implimented '-so' for 7z compression, following the manpage
> you would expect both of the following to produce the required output,
> but 7z (the default if no '-t<type>' is provided) fails.
I read the man page, and didn't test it till I read your post. You are correct 7z and 7za give errors if you try to use -so
| Code: | System ERROR:
E_NOTIMPL |
Guess I look to see if there is another compression program I want to use. The best part about 7z and 7za was that if you send them lots of data, 7z uses multiple threads to compress. In my test of a 2.5 GB directory, tar piped to 7z compressed the directory in just over 2 min (using multiple threads) and the archive was smaller than bzip2 which took about 5 min. |
|
| Back to top |
|
|
cboldt
Veteran
Joined: 24 Aug 2005
Posts: 1046
|
| Posted: Fri Jan 06, 2017 10:33 pm Post subject: |
|
|
Bash allows direction of stderr to one place and let only the (stdout) stream through the pipe.
| Code: | | compress_pgm 2> error-trap.txt | compress_cmd |
I use that technique with a script that writes tripwire rules. Progress messages, which can't be part of the tripwire rule file, are sent to stderr. If the output of the script is sent to another file (e.g., rule_generator > rule.file), the progress messages still appear on the console.
I would guess that your compression program writes error and similar (e.g., progress) messages to stderr, but of course would test to find out.
I've never had occasion to redirect stderr to the console rather than a file (error-trap.txt in the example), in a pipeline, but would look to `tee` if i wanted to send some output to more than one place, as well as researching how to redirect (some) output to console while allowing other output to run through the pipe. |
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Sat Jan 07, 2017 4:38 am Post subject: |
|
|
| jagdpanther wrote: | | Guess I look to see if there is another compression program I want to use. The best part about 7z and 7za was that if you send them lots of data, 7z uses multiple threads to compress. In my test of a 2.5 GB directory, tar piped to 7z compressed the directory in just over 2 min (using multiple threads) and the archive was smaller than bzip2 which took about 5 min. |
jagdpanther ... you might try app-arch/xz-utils (LZMA2) and compare ratio and time. This also supports 'threads' for compression, see '-T,--threads=threads'.
HTH & best ... khay |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Portage & Programming
