Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
TIP: Compressing portage using squashfs: initscript method
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3 ... 10, 11, 12, 13, 14  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
yoshi314
l33t
l33t


Joined: 30 Dec 2004
Posts: 819
Location: PL

PostPosted: Wed Aug 21, 2013 6:10 am    Post subject: Reply with quote

thanks, will test later on today.

edit: works fine now. the initial example looks a bit intimidating, but it's manageable eventually ;)
_________________
~amd64, ~x86
shrink your /usr/portage with squashfs+aufs
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4078

PostPosted: Thu Aug 22, 2013 9:01 pm    Post subject: Reply with quote

There were some serious issues with <=squashmount-2.4 (first restart was buggy, and the fix had a serious typo concerning umount).
However, the current squashmount-2.5 should be ok.
Back to top
View user's profile Send private message
yoshi314
l33t
l33t


Joined: 30 Dec 2004
Posts: 819
Location: PL

PostPosted: Mon Aug 26, 2013 10:36 am    Post subject: Reply with quote

how do i disable automatic rebuilds of squashfs images on shutdown? will commenting out the threshold do, or do i have to set it to 0 or really high value?
_________________
~amd64, ~x86
shrink your /usr/portage with squashfs+aufs
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4078

PostPosted: Mon Aug 26, 2013 8:05 pm    Post subject: Reply with quote

yoshi314 wrote:
how do i disable automatic rebuilds of squashfs images on shutdown? will commenting out the threshold do, or do i have to set it to 0 or really high value?

A negative threshold means "infinity" (even more, it overrides "kill"). If you only mean for the next reboot, you can use something like
Code:
squashmount -n set # -n is the short option for --nosquash
which does nothing else than setting temporarily (in /run/squashmount) the threshold to -1.
Back to top
View user's profile Send private message
yoshi314
l33t
l33t


Joined: 30 Dec 2004
Posts: 819
Location: PL

PostPosted: Mon Sep 02, 2013 10:23 am    Post subject: Reply with quote

after recent update to 2.8 ( i think ) i could not restart existing mounts.

after manually undoing all union and squashfs mounts, script mounted everything successfully. not sure if that meant i experienced some potential data loss or not.
_________________
~amd64, ~x86
shrink your /usr/portage with squashfs+aufs
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4078

PostPosted: Wed Sep 04, 2013 6:11 am    Post subject: Reply with quote

yoshi314 wrote:
after recent update to 2.8 ( i think ) i could not restart existing mounts.

I had this also once, but I was unable to reproduce it. If you know how to reproduce it, please post and the bug can be fixed ;)
(Maybe it happened from an update of an much earlier version? In this case it could be related due to the change of the format in /var/run).
Quote:
after manually undoing all union and squashfs mounts, script mounted everything successfully. not sure if that meant i experienced some potential data loss or not.

It does not mean a data loss; squashmount.2.8 contains options -I and -f to help you in such an emergency (when it can). For instance
Code:
squashmount -I umount
will try to umount even if it "thinks" that the partition is not mounted; it might be necessary to combine this with -f to try more aggressively.
Back to top
View user's profile Send private message
yoshi314
l33t
l33t


Joined: 30 Dec 2004
Posts: 819
Location: PL

PostPosted: Wed Sep 04, 2013 3:57 pm    Post subject: Reply with quote

seems to be happening again :


Code:
[root@box ~]# squashmount status                                   
 * [db]:       not mounted
 * [portage]:  not mounted
 * [overlays]: not mounted


Code:
mount
(....)
/mnt/storage/db.sqfs on /var/db.readonly type squashfs (ro,noatime)
unionfs on /var/db type fuse.unionfs (rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other)
/mnt/storage/portage.sqfs on /mnt/storage/portage.readonly type squashfs (ro,noatime)
unionfs on /usr/portage type fuse.unionfs (rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other)
/mnt/storage/overlays.sqfs on /mnt/storage/overlays.readonly type squashfs (ro,noatime)
unionfs on /mnt/storage/overlays type fuse.unionfs (rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other)


my config is as it used to be :

Code:
@order = ( 'unionfs-fuse' );

push(@mounts, {
   TAG => 'db',
   DIR => '/var/db',
   FILE => '/mnt/storage/db.sqfs',
   BACKUP => '/var/db.sqfs.bak', # keep a backup
   CHANGES => '/var/db.changes',
   READONLY => '/var/db.readonly',

   # Do not resquash everytime but only after 30 megabytes of fresh data:
   THRESHOLD => '30m',
   # Since this directory contains only very small files, we cheat with
   # this size by using that each file takes at least a full block:
   # Hence, the number of files is more important for THRESHOLD than
   # their size. In gentoo, one installed package thus "counts" about
   # 2m in size (although it is actually only 20 very short files):
   BLOCKSIZE => 65536
       }, {
               TAG => 'portage',
               DIR => '/usr/portage',
               FILE => '/mnt/storage/portage.sqfs',
               CHANGES => '/mnt/storage/portage.changes',
               READONLY => '/mnt/storage/portage.readonly',
               THRESHOLD => '20m', # resquash on umount if 40 megabytes changed
          COMPRESSION => 'lzo'
       }, {
               TAG => 'overlays',
               DIR => '/mnt/storage/overlays',
               FILE => '/mnt/storage/overlays.sqfs',
               CHANGES => '/mnt/storage/overlays.changes',
               READONLY => '/mnt/storage/overlays.readonly',
               THRESHOLD => '10m', # resquash on umount if 40 megabytes changed
          COMPRESSION => 'lzo'
       });


      # If /var is on a separate partition, we want that the
      # squashfile is first generated in /var/tmp:
      #TEMPDIR => '/var/tmp',




hm, something's odd :
Code:
squashmount check -f
 * [db]:       error:   DIR is nonempty!
 * [portage]:  error:   DIR is nonempty!
 * [overlays]: error:   DIR is nonempty!


after manual unmounting it reports back fine.
_________________
~amd64, ~x86
shrink your /usr/portage with squashfs+aufs
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4078

PostPosted: Wed Sep 04, 2013 5:02 pm    Post subject: Reply with quote

yoshi314 wrote:
seems to be happening again:

Can you reproduce what you did to make it happen?
Quote:
hm, something's odd :
Code:
squashmount check -f

This is not surprising: squashmount thinks that your original directory is not mounted; since it contains data (namely that from the mount which squashmount is not aware of), it reports the directory as nonempty. The bug happened before: squashmount should not have flagged the directory as umounted (in /run/squashmount) when actually it is mounted.
Back to top
View user's profile Send private message
yoshi314
l33t
l33t


Joined: 30 Dec 2004
Posts: 819
Location: PL

PostPosted: Wed Sep 04, 2013 7:09 pm    Post subject: Reply with quote

mv wrote:
yoshi314 wrote:
seems to be happening again:

Can you reproduce what you did to make it happen?
i just booted the system. journalctl logs say it started up fine, i have squashfs and unionfs mounted. only script seems to be having a different opinion on that.
_________________
~amd64, ~x86
shrink your /usr/portage with squashfs+aufs
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4078

PostPosted: Wed Sep 04, 2013 7:14 pm    Post subject: Reply with quote

yoshi314 wrote:
i just booted the system. journalctl logs say it started up fine, i have squashfs and unionfs mounted. only script seems to be having a different opinion on that.

Ah, systemd. (Normally I run openrc). Then I guess that it is a dependency issue; I will report back.
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4078

PostPosted: Wed Sep 04, 2013 9:38 pm    Post subject: Reply with quote

The bug is finally fixed: systemd was the correct hint for reproducing (now I remembered that I was testing with systemd when the bug occured in my case).

It was a stupid issue: tmpfiles.d was configured to clean /run/squashmount. So if systemd cleaned this directory after starting squashmount (which could happen by accident since there was no corresponding dependency specified in the systemd service file), it cleaned all information which squashmount had stored about mounted dirs.

The issue is fixed in the current squashmount-2.9, and also the emergency behaviour was improved and explicitly documented.
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1206
Location: PB, Germany

PostPosted: Wed Sep 11, 2013 10:41 am    Post subject: Reply with quote

I took a look at the new squashmount. I really appreciate the rewrite in Perl. If things are getting bigger, bash and similars have their restrictions. Even if Perl is not the problem for me, are you going to keep the configuration /etc/squashmount.pl in plain Perl syntax? I think this is quite unusual even for projects written in Perl, at least I know Perl-only users like that, but the configuration is the user interface for non-developers.

So since migrating is not that easy as replacing squash_dir with squashmount, I stay with squash_dir at the moment, trying the switch some time later and watching your squashmount.
I had no issues with different init systems as I just did switch to OpenRC when Gentoo did announce the new default.
_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G | amd64:Acer Z5610 (Core2QuadQ8200),i5-3470 | amd64-prefix:OpenSuse | Lila-Theme
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4078

PostPosted: Wed Sep 11, 2013 6:34 pm    Post subject: Reply with quote

Massimo B. wrote:
going to keep the configuration /etc/squashmount.pl in plain Perl syntax?

Yes, because it is more appropriate: I also thought first about a "plain" configuration file, but this way you can easily make conditions on something (e.g. I use the same config-file on different machines depending on the existence of some symlinks), you can make certain values (like the compression algorithm) easily the default for all your mount points etc. Moreover, for configuring several mount-points in one file, the syntax of an init-file would not be much simpler anyway.
Quote:
So since migrating is not that easy as replacing squash_dir with squashmount

If you want, it can be almost that easy: Write
Code:
push(@mounts, {
TAG => 'portage',
...
}); # do not forget these braces and semicolon!

and put into "..." your old configuration, essentially just replacing "=" by "=>", finishing each line except the last with a comma and putting the names into '...' or "..." (as you perhaps already did in squash_dir); repeat this for every mount point (replacing "portage" by a different name of your choice).
Only take care that some variable names have slightly changed (e.g. previous DIRECTORY is now called simpler DIR) and that some global variables have become perl variables; for a "normal" user this involves only the ORDER variable which must now be set differently:
Code:
@order = ('overlayfs', 'aufs', 'unionfs-fuse'); # Do not forget the semicolon
Everything else is just syntactic sougar which becomes useful if you have more mount-points (e.g. I have 12 mount-points, and such a huge configuration file becomes much shorter and easier to maintain using perl-syntax shortcuts than if I would repeat the above task 12 times, although the latter would be possible).
Quote:
I stay with squash_dir at the moment,

I strongly recommend to switch, especially if you ever wanted to omit resquashing for the next boot or force squashing immediately. All this could be done with squash_dir, but is now much simpler and much more convenient.
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1206
Location: PB, Germany

PostPosted: Mon Sep 16, 2013 7:32 am    Post subject: Reply with quote

Hi. I was wondering about the sub added_hash{} as it only returns the hash it gets. Ok it merges sub_hashes, maybe meant to ease some default like sub standard_mount{}.
Having some little Perl knowledge I asked in #perl about this approach. Hash::Merge::Simple could be more appropriate.

But anyway, having configuration of executable code and calling functions from there also sounds unusual and PHPish to #perl people. Maybe the gained flexibility is not the most save way and not necessarily desirable. This even made me reading the code and ask Perl coders to understand the configuration example, which should not be the point of a user application. I stay tuned, I liked the squash_dir and I appreciate the new port.
_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G | amd64:Acer Z5610 (Core2QuadQ8200),i5-3470 | amd64-prefix:OpenSuse | Lila-Theme
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4078

PostPosted: Mon Sep 16, 2013 8:15 pm    Post subject: Reply with quote

Massimo B. wrote:
Hi. I was wondering about the sub added_hash{} as it only returns the hash it gets. Ok it merges sub_hashes, maybe meant to ease some default like sub standard_mount{}.

This is exactly the idea: That you can merge some "defaults" with other data (like standard_mount()). This is also the way it is used in the examples.
Quote:
Having some little Perl knowledge I asked in #perl about this approach. Hash::Merge::Simple could be more appropriate.

This function is neither in perl-5.18.1 nor in Gentoo at all. Moreover, recursion and other tricks of that function are not needed. added_has() is practically a simplified (non-recursive) variant of Hash::Merge::Simple
Quote:
having configuration of executable code and calling functions from there also sounds unusual

Nobody forces you to use these features. As I have explained (and hopefully also made clear on the manpage), you can use just a "plain" configuration. If you have perl knowledge you can use perl tricks to avoid the repetition or use other perl code for setup (e.g. source other setup files or whatever). The two functions are there to support you when doing this, but there is no need to do it.
It is a bit comparable with writing LaTeX: If you have to type similar texts several times you either can do this, or you setup some macros which avoid the repetition and allow for easy changing in the repetitions. The two provided functions are just IMHO useful "macros" in this context, but you can completely ignore them and just write the repetitions or write your own functions (e.g. use Hash::Merge::Simple if you have installed it).
Back to top
View user's profile Send private message
Yamakuzure
Veteran
Veteran


Joined: 21 Jun 2006
Posts: 1374
Location: Bardowick, Germany

PostPosted: Tue Sep 17, 2013 10:31 am    Post subject: Reply with quote

mv wrote:
Code:
@order = ('overlayfs', 'aufs', 'unionfs-fuse'); # Do not forget the semicolon
Is there a reason to place overlayfs before aufs? Or in other words, are there reasons to prefer overlayfs over aufs? I am currently using aufs and hadto change that particular line, so squashmount won't complain every time that overlayfs isn't available. AFAIK overlayfs is a bit like an AUFS-light, right?
_________________
I *do* know that I easily aggravate people due to my condensed writing. Rule of thumb: If I wrote anything that can be understood in two different ways, and one way offends you, then I meant the other! ;)
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4078

PostPosted: Tue Sep 17, 2013 12:31 pm    Post subject: Reply with quote

overlayfs is first, because it appeared when writing squashmount that it would be in the kernel, soon.
Both are more or less equivalent, although there are some known cases where directories from overlayfs do not behave exactly as required by POSIX - aufs contains a lot of code to cover all these (rare) cases correctly.
Back to top
View user's profile Send private message
Yamakuzure
Veteran
Veteran


Joined: 21 Jun 2006
Posts: 1374
Location: Bardowick, Germany

PostPosted: Tue Sep 17, 2013 1:17 pm    Post subject: Reply with quote

Thank you very much!
_________________
I *do* know that I easily aggravate people due to my condensed writing. Rule of thumb: If I wrote anything that can be understood in two different ways, and one way offends you, then I meant the other! ;)
Back to top
View user's profile Send private message
init_6
Apprentice
Apprentice


Joined: 22 Jun 2008
Posts: 166

PostPosted: Wed Sep 18, 2013 7:28 am    Post subject: Reply with quote

By the old wiki Squashed Portage Tree or mirror I make squashed-portage bash script with systemd support. To install layman -a init6; emerge sys-apps/squashed-portage Use the same as old init.d script

Code:
/usr/sbin/squashed-portage start
/usr/sbin/squashed-portage stop
/usr/sbin/squashed-portage reload


or:

Code:
systemctl start portage.service
systemctl stop portage.service
systemctl reload portage.service
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4078

PostPosted: Wed Sep 18, 2013 9:44 pm    Post subject: Reply with quote

init_6 wrote:
bash script with systemd support.

This is fine, but already squash_dir with systemd support could do much more (squashing only if threshold is reached, ignoring unimportant files, support for multiple mount points, fallbacks to alternatives to aufs, rather secure error handling, etc). The new implementation squashmount has even a lot more features, especially concerning the user interface (conveniently forcing/omitting resquashing immediately/later on, killing previous modification currently/next time/forever, ...) and is (after the birth problems have been fixed) probably also the safest choice concerning various error handing. The price for the features is a somewhat more complex setup of the mount-points, but I hope that after the setup the actual user interface is rather intuitive.

BTW, here is another argument, why it is useful to have a setup file in perl: You can use it to set the order of the tools depending on a test what your current kernel does support. Examples will probably be given in the next release of squashmount. Here just an untested one:
/etc/squasmount.pl wrote:
if(system('modprobe aufs >/dev/null 2>&1')) {
@order = ('aufs', 'unionfs-fuse')
} elsif(system('modprobe overlayfs >/dev/null 2>&1'))
@order = ('overlayfs', 'unionfs-fuse')
} else {
@order = ('unionfs-fuse')
}
Back to top
View user's profile Send private message
slycordinator
Advocate
Advocate


Joined: 31 Jan 2004
Posts: 3057
Location: Redmond, WA

PostPosted: Tue Sep 24, 2013 6:38 pm    Post subject: Reply with quote

On my end squashmount always attempts to create a squashfs file for the "games" directory at /usr/share/games with the LZO compression scheme. And this always fails because I followed the general advice of the kernel config to not include LZO support.
_________________
My political stance/bias
slycordinator != slycoordinator
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4078

PostPosted: Tue Sep 24, 2013 6:47 pm    Post subject: Reply with quote

slycordinator wrote:
On my end squashmount always attempts to create a squashfs file for the "games" directory

You must modify the provided /etc/squashmount.pl to your needs - this is just full of various examples (among them an example for the games directory with COMPRESSION => 'lzo').
If you have already squashed directories which you did not intend to you can use the unsquash program (after(!) umounting the directories with squashmount) to get the original data back.
Back to top
View user's profile Send private message
slycordinator
Advocate
Advocate


Joined: 31 Jan 2004
Posts: 3057
Location: Redmond, WA

PostPosted: Tue Sep 24, 2013 7:19 pm    Post subject: Reply with quote

Any particular reason you chose lzo for that? Were you finding it performed better for the games directory or something?

NM I read why in that file
_________________
My political stance/bias
slycordinator != slycoordinator
Back to top
View user's profile Send private message
emc
Guru
Guru


Joined: 02 Jul 2004
Posts: 390
Location: Cracow, Poland

PostPosted: Wed Oct 09, 2013 11:36 am    Post subject: Reply with quote

I will try it
Back to top
View user's profile Send private message
slycordinator
Advocate
Advocate


Joined: 31 Jan 2004
Posts: 3057
Location: Redmond, WA

PostPosted: Mon Oct 21, 2013 10:18 pm    Post subject: Reply with quote

With squashmount-3.6 installed I get:
Code:
 * umounting/stopping squashmount ...
error parsing /etc/squashmount.pl: Undefined subroutine &CFG::getpwname called at /etc/squashmount.pl line 40.

stopped at /usr/bin/squashmount line 2082.                                                                                                                                   [ !! ]
 * ERROR: squashmount failed to stop

_________________
My political stance/bias
slycordinator != slycoordinator
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3 ... 10, 11, 12, 13, 14  Next
Page 11 of 14

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum