View previous topic :: View next topic |
Author |
Message |
kernelOfTruth Watchman
Joined: 20 Dec 2005 Posts: 6111 Location: Vienna, Austria; Germany; hello world :)
|
Posted: Sat Aug 02, 2008 8:05 pm Post subject: tar, bzip2 multicore goodness |
|
|
Hi everyone,
the basics for using tar are provided here:
http://www.shell-fu.org/lister.php?tag=tar
knowing these basics one can combine p7zip and tar to following command:
Code: | time (nice -20 tar -cp / -X /root/stage4.excl | 7z a -si -tbzip2 /bak/system/stage4-amd64_Final-11-030808.tbz2) |
(this should create a stage4-tarball using bzip2-format with maximal compression and multiple cpu-cores, for you convenience it also shows the time it took to do so)
the command for extraction would be:
Code: | 7z e -so -tbzip2 /bak/system/stage4-amd64_Final-11-030808.tbz2 | tar -xp -C /test/ |
if anything of the above is incorrect please post
I'm currently testing those commands & update this thread accordingly what I experience
update1:
now the sample commands' syntax should be correct _________________ https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa
Hardcore Gentoo Linux user since 2004
Last edited by kernelOfTruth on Sun Aug 03, 2008 8:23 pm; edited 1 time in total |
|
Back to top |
|
|
prizident n00b
Joined: 06 Dec 2006 Posts: 42
|
Posted: Sun Aug 03, 2008 12:54 am Post subject: |
|
|
there is also a tool pbzip2, which also can handle multiple cores |
|
Back to top |
|
|
kernelOfTruth Watchman
Joined: 20 Dec 2005 Posts: 6111 Location: Vienna, Austria; Germany; hello world :)
|
|
Back to top |
|
|
kernelOfTruth Watchman
Joined: 20 Dec 2005 Posts: 6111 Location: Vienna, Austria; Germany; hello world :)
|
Posted: Sun Aug 03, 2008 9:27 pm Post subject: |
|
|
here the output of my first multi-core created stage4 tarball
Quote: | time (nice -20 tar -cp / -X /root/stage4.excl | 7z a -si -tbzip2 /bak/system/stage4-amd64_Final-11-030808.tbz2)
tar: Removing leading `/' from member names
7-Zip 4.58 beta Copyright (c) 1999-2008 Igor Pavlov 2008-05-05
p7zip Version 4.58 (locale=en_US.utf8,Utf16=on,HugeFiles=on,2 CPUs)
Creating archive /bak/system/stage4-amd64_Final-11-030808.tbz2
...
Everything is Ok
real 51m2.510s
user 76m58.764s
sys 2m22.778s |
(this is 51 minutes instead of 80 or more minutes ) _________________ https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa
Hardcore Gentoo Linux user since 2004 |
|
Back to top |
|
|
fangorn Veteran
Joined: 31 Jul 2004 Posts: 1886
|
Posted: Mon Aug 04, 2008 8:00 am Post subject: |
|
|
This is working great. Thank you.
For convenience I packed this into two scripts tbz2 and utbz2. If someone is interested, here they are.
Code: | #!/bin/bash
if [ $# -le 1 ] ; then
echo "Usage: $0 <archive_file> source1 [source2 [...]]"
exit;
fi
dest=$1
shift
nice -20 tar -cp $@ | nice -20 7z a -si -tbzip2 $dest |
Code: | #!/bin/bash
dest=""
if [ $# -lt 1 ] ; then
echo "Usage: $0 <archive_file> [destination_directory]"
exit;
fi
if [ ! -f $1 ] ; then
echo "Usage: $0 <archive_file> [destination_directory] "
exit;
fi
if [ ! -z $2 ] ; then
if [ -d $2 ] ; then
dest="-C "$2
else
echo "Directory $2 does not exist. Do you want to create it (y/n)"
read a
if [ $a = "y" ] || [ $a = "Y" ] ; then
mkdir -p $2
dest="-C "$2
else
exit;
fi
fi
fi
7z e -so -tbzip2 $1 | tar -xp $dest |
_________________ Video Encoding scripts collection | Project page |
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3328 Location: Rasi, Finland
|
Posted: Thu Aug 14, 2008 3:29 pm Post subject: |
|
|
This might make compressing even more effective: Code: | time (nice -20 tar -cp / -X /root/stage4.excl | 7z a -si -tbzip2 -md=32m -mx=9 -mpass=10 -mmt=5 /bak/system/stage4.tbz2) |
I haven't tested much it.
It's slower, yes. On my test 7min --> 12min difference on a test archive. _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
shentino n00b
Joined: 21 Nov 2009 Posts: 63
|
Posted: Mon Aug 27, 2012 6:54 pm Post subject: what if |
|
|
What if each bzip2 block were forked into its own thread for decompression, and then all the thawed blocks were simply reassembled in the correct order? |
|
Back to top |
|
|
mv Watchman
Joined: 20 Apr 2005 Posts: 6747
|
Posted: Mon Aug 27, 2012 7:19 pm Post subject: |
|
|
GNU tar has the option --use-compress-program. So you could just write a script which calls "exec 7z" with appropriate parameters and use that option. I can imagine (depending on the implementation in GNU tar which I did not check) that this could be slightly faster than using the shell for piping. |
|
Back to top |
|
|
mattst88 Developer
Joined: 28 Oct 2004 Posts: 422
|
Posted: Sat Sep 15, 2012 6:06 am Post subject: |
|
|
prizident wrote: | there is also a tool pbzip2, which also can handle multiple cores |
Please use lbzip2 instead.
kernelOfTruth wrote: | yes, the problem with that seems to be:
Quote: | Decompressing non-pbzip2 Created Archives
pbzip2 can only decompress archives in parallel that have been compressed with pbzip2. For example, extracting linux-2.6.23.8.tar.bz2 as found on kernel.org with pbzip2 takes roughly twice as long on a dual core system when compared against bzip2. |
http://gentoo-wiki.com/HOWTO_Speed_up_decompression_with_pbzip2 |
lbzip2 does not have this limitation. Use it instead. _________________ My Wiki page |
|
Back to top |
|
|
Ant P. Watchman
Joined: 18 Apr 2009 Posts: 6920
|
Posted: Sat Sep 15, 2012 5:07 pm Post subject: |
|
|
Anyone tried app-arch/lrzip on these stage4 files? It usually gives 1GB/min at maximum settings for me. |
|
Back to top |
|
|
kernelOfTruth Watchman
Joined: 20 Dec 2005 Posts: 6111 Location: Vienna, Austria; Germany; hello world :)
|
|
Back to top |
|
|
John R. Graham Administrator
Joined: 08 Mar 2005 Posts: 10587 Location: Somewhere over Atlanta, Georgia
|
Posted: Thu Oct 18, 2012 3:52 pm Post subject: |
|
|
Just delivered a .tar.bz packaged with pbzip2 to a far east factory partner that they could not unpack with WinRAR. It appears I had installed pbzip2 since I had last delivered anything to the factory. Another strike against pbzip2.
- John _________________ I can confirm that I have received between 0 and 499 National Security Letters. |
|
Back to top |
|
|
|