Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Creating a self-extracting script to embed files in postings
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Sun Nov 26, 2006 6:36 pm    Post subject: Creating a self-extracting script to embed files in postings Reply with quote

Hi all,

I have created a script which can used as a filter: It takes the output of tar as its input and writes a script as its output which, when run, will extract the files from the tar archive.

The scripts generated that way are very small and can be used as preformatted text in any posting, such as this one. Or they can be pasted directly into an e-mail message (which might be useful if you don't want to use attachments for any reason or if you are using a very dumb email program which does not support attachments).

My script does not assume anything about the contents of the tar archive passed to it by the user, i. e. it can contain text files, or binary files, or both.

The script will compress the archive contents passed to it with BZIP2 and GZIP (both with maximum compression enabled) and compare the compressed sizes against each other (and also against the uncompressed archive). It will then use the shortest of the resulting output versions for embedding into the script.

Which means that adaptive compression will be applied: The shortest resulting file size will be used, whether that means the tar archive has to be gzip-compressed, bzip2-compressed or not be compressed at all. (The latter can be true for archives already containing compressed files and which also happen to have a size which is close to being aligned to the tar output block size.)

The compressed archive will then be encoded as BASE-64 using OpenSSL, and finally a wrapper script will be added which will extract the files from the archive when run.

As a working example of the output of my script, here is a tar archive containing my script, after having been applied to itself:
Code:
#! /bin/sh
which openssl > /dev/null 2>& 1 \
|| { echo "OpenSSL is required for BASE-64 decoding!" >&2; exit 1;}
openssl enc -d -a << END | tar -xvz
H4sIAEnRaUUCA+1WW2/bNhTOq/grThWtXrPKtmSna5wLkCxJN6A3OC4KdC1SWqIt
NrKkUpQdp81++86hpMRutrYvG1BAH5DI5rlf+Mnjce5mPLjgU+Fqrjb+C3QRv25v
myfiy+d2t9fb8LrbPb/X6/a7qOf5fd/bgO7G/4Aix7IBNlSa6q/pfUv+g2LzHnTG
MunkEdtkm3AqYy0UaH4hkylwGB0OgasgknMBE5XOINehTIAnISyU1KVSHiiZabSe
pApCEaSzTIk8J6GOuDZOdEqmaaHbJs4oEmu+FzKOYSyAay1mmRYhGeD32hceLKSO
QOA/odDBkzd/vAQMd4RP/0ZNpgkUVWCB9rleE6WZxkeOKcDrSJAncpGk2qjfyYcH
uuBxvFxPBI1DkYkkzAFdkiGeF7HpRSySKWbpmuM8SpWmFOZCmfDkFK3RW4F+bhsx
FYlQnIouO1lGD9JEc1lGWGlDStUXycoJ5Y16R4dnJ+6jPgiUhWYwNCTyhNHgBWZ8
dva0GlKtS8Mi3TIX53fBw1fDpwPoBDyIRN65zOdJp7jk4ayjVZFcdIpcdeI04LFZ
mvE6eYBDTg4LjYUPgC5MeXKMtQ3A73YfuZ7n+o9G3vbA9wZ9r933dvrb3ptSbSjm
kvo0gL7/mI5oTLhkWiRUH9nDeAlPCpGY0R1hRrhfsVBtxlgoxc8P4BOzRBClYJ8M
hy+GA3C2bDi4Dz4eX0oNHrtmbPisVFQzsB3Phs+fAY3B/i0t4tBsgxKzFJeg5Xit
ezaZbMKhmuYDmHOV8Jl4CBMZizabCn2ey6sqsOkLnL05vxgn3rTf+7jDH4urHdWd
FcViGfU9wayvSfft98hFGlwXRzTDDz/lmKBvv7fhLbOsu2ka7ZbjU5KWmGNwx9t3
vprASi00OOyiuBRBofk4xqr+4a48XL3Qq1eINvu89LFePpwy63TfdkZtbC6zTIsd
H/awlhEOAx+ntz13PJhwGRdKUA11P8kLqTFLTuBPwJLAjXGZjk7ORvBul65Egh2h
r1QvfsQ78OLl6AzD9qgXcY69tobPKi8TWRbu/EvlTAka6/kYBavFrFdicnFFWcBt
FuaKm2Ms7/59MFFHqG/N5qWukdyMjyflilFEnN5pC5cbya7ljMwcy1yLHO/UeSTi
bHWpX9EpLvWn7ubmVuca09mr5nEA7XYb3lGCLuzhXXRvBO/QaYCLsrfnttotxqwF
Xh5RWx6AJB4TkE4M00zSOE4XSAkDZrkuJfAQ3Aj/3KIMfizzLOZLVEZDLS6R0C1S
RX04TJb1ggCfmBcJadXknnHDVzpSaTGNiOFxbGg2XprQd2iQo5cwlOQPh4GhFDLy
2hJayJ91AL3MZFDTNRGsYbqa2G03sys7ZAvru3nF+i72su6QV7tkHJrlIkKyAORP
sQthSsPIRck8kvbHjeAzlK02H0yf4QG+PuRE78LtJuzukrZ7KxorwS/K0y1juoWy
alOSvMgyev+EVdUVmxk23IWSDY0tGd14EjkPmGX8sxDXgpIPInQhkjyP8fZ2QjHv
JAX220dPHq01rqfZ7OoVQwulxMdCqmoEX76YDKWO9u2OnmWdv+plPnecNh7YjHb1
YP3OHKn0AoeVyYxo4oYlhoevDVHgZSupwBywGy5wL+f2Kk/B9Epm4O4ACq7WBGMU
+JXkwyobGJO1A6PKygsFJ8+P2covqG816+1bRv2qZvStftUvZzO025ldM4pqXLTq
UNhacENweZUUbgNyQIt6UzbD+eQNfoFrfB2y2maMa4hRSl5m1Vvz+bHNagLbaNCg
QYMGDRo0aNCgQYMGDRo0aNCgQYMGDRo0aNCgwY+FvwFITYUlACgAAA==
END


In order to get my script, just extract the above text to some file such as guenther_script.b64, and then do a
Code:
sh guenther_script.b64

in order to extract my script bbs-package-tar to the current directory.

Finally an example how to create a tar archive embedded within an article text snippet using my generator script.

Let's assume you want to attach the three files "myscript.sh", "myarchive.tar.gz" and "mypict.jpg" to your article in the forum.

Having installed my script bbs-package-tar in /usr/local/bin or any other path where it can be found, run the following commands then in order to create a script mysnippet.b64:
Code:
tar -c myscript.sh myarchive.tar.gz mypict.jpg | bbs-package-tar > mysnippet.b64


That's it! Easy, wasn't it?

Now write your article, and paste the contents of the mysnippet.b64 into the text and format it as preformatted text.

Note that tar has been run with the -c (create) option only, not with the -z (compress gzip) or -j (compress bzip2) options. This is because my script will will determine the best compression itself, so specifying -z or -j would be counter-productive and must be avoided.

Also note that the latter note only refers to the tar command used in the above command line; the files to be contained within that archive can have been created compressed or not.

For instance, the file myarchive.tar.gz in the example might have been created with the -z option. myscript.sh might be a text file that has not been compressed at all. And mypict.jpg will very likely be a JPEG-image which is already compressed because the JPEG image file format itself uses compressed data.

All those files are included in the tar command line without additional compression, but my script will automatically apply compression if this will result in a smaller size of the text snippet to be created.


Last edited by Guenther Brunthaler on Sun Nov 26, 2006 8:24 pm; edited 1 time in total
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Sun Nov 26, 2006 6:54 pm    Post subject: Using my script versus using SHAR Reply with quote

There is another tool called SHAR (for "SHell ARchive") which can also be used to convert any list of files into a self-extracting script file.

However, my script generates only 5 lines of overhead, where SHAR creates... MANY lines of overhead.

My script however assumes that the tar, gzip, bzip2 and openssl executables are available, or extraction will not work.

SHAR on the contrary, will work without any of those executables installed; it only requires a working sh.

This makes SHAR a better choice for barebone machines with only a minimum set of tools installed, but a workstation machine will quite certainly have all the tools installed which are required by my script.

And because my script has only very low overhead, it will typically be suited better for inclusion of a single file (or a set of rather small files) in postings.
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Sun Nov 26, 2006 7:08 pm    Post subject: -Z compression is not supported Reply with quote

One might ask "why does your script not support compression using the compress tool, as used by tar -cZ?"

The answer is: Initially, I wanted to do that. But then I found out that compress was not installed on my box - although I have an otherwise pretty complete Gentoo installation.

In other words: While compress certainly is a standard tool in non-GNU UNIX systems, it seems not to be installed on every GNU/Linux system.

And that's why I chose not to support it in my script either.

However, it can easily be added by applying the following patch:
Code:
diff -Naur old/bbs-package-tar new/bbs-package-tar
--- old/bbs-package-tar   2006-11-26 20:01:11.000000000 +0100
+++ new/bbs-package-tar   2006-11-26 20:04:51.000000000 +0100
@@ -101,8 +101,10 @@
 TAROPTS="-xv"
 test_packer gzip -9 -xvz
 test_packer bzip2 -9 -xvj
+test_packer compress -c -xvZ
 rename_back gzip
 rename_back bzip2
+rename_back compress
 cat << END
 #! /bin/sh
 which openssl > /dev/null 2>& 1 \\
Back to top
View user's profile Send private message
truc
Advocate
Advocate


Joined: 25 Jul 2005
Posts: 3199

PostPosted: Tue Nov 28, 2006 6:48 am    Post subject: Reply with quote

you already used this trick on the forums before, and I liked it. this can be really handy!

thank you
_________________
The End of the Internet!


Last edited by truc on Wed Nov 29, 2006 1:33 pm; edited 1 time in total
Back to top
View user's profile Send private message
avx
Advocate
Advocate


Joined: 21 Jun 2004
Posts: 2070

PostPosted: Wed Nov 29, 2006 12:16 am    Post subject: Reply with quote

Seen it - tried it - loved it ... do I need to say more?
Well, a big thank you comes to my mind, so here it is: Thank you :)
Back to top
View user's profile Send private message
bludger
Guru
Guru


Joined: 09 Apr 2003
Posts: 389

PostPosted: Wed Nov 29, 2006 1:08 pm    Post subject: Reply with quote

This looks pretty cool.

When I first tried pasting this into a file and extracting it, I got the following error:

gzip: stdin: unexpected end of file
bbs-package-tar
tar: Child returned status 1
tar: Error exit delayed from previous errors

I then saw that there was a space or something after the "END". When I deleted this, it extracted without any error messages. Is there some way of getting around this problem?
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Wed Nov 29, 2006 2:32 pm    Post subject: Reply with quote

bludger wrote:

I then saw that there was a space or something after the "END". When I deleted this, it extracted without any error messages. Is there some way of getting around this problem?


I think it depends on the Browser/Editor combo one is using for the copy/paste. As it seems, in some combinations whitespace is added at the end of each line.

I'm using Firefox and kedit, and at least that tool combination does not add additional whitespace at the end of the lines.

While I suppose OpenSSL may be tolerant to this, the shell unfortunately isn't when using the "here doc"-syntax, as my script does.

I'll have to find a better way how to feed the BASE-64 chunk into OpenSSL. Until I find one, I',m afraid the user who does the copy/paste has to take care of that.
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Wed Nov 29, 2006 2:44 pm    Post subject: Whitespace Problem Reply with quote

Hi all,

I found a working solution to the whitespace-problem: Just remove the "END" line from the last line of the generated script, and it will work no matter how much whitespace is added by the copy/paste to the end of each line!

As it seems, the "here doc"-syntax works without the delimiter as well: In this case, all the data until EOF will be read (and fed to OpenSSL); which is exactly what I want.

However, I do not know whether this behaviour is portable, or just coincidental.

An ideas about the portability?

If it is indeed portable, I will happily modify my script to omit the ending delimiter for the "here doc" text. This will even make the generated script one line shorter - with no additional changes required.

UPDATE:

In http://opengroup.org/onlinepubs/007908799/xcu/chap2.html is stated that a missing delimiter is implementation-defined and should be considered an error.

Which means, this approach to the whitespace-problem just isn't going to be the right one. ;-(
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum