Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Compressing filesystems with squashfs and squashmount
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8, 9, 10  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Tue Sep 01, 2015 8:09 am    Post subject: Reply with quote

Massimo B. wrote:
So this was a rebase again?

Might be; I don't remember.
Quote:
On the mv overlay it happens quite often

I think it was 4-7 times in the lifetime of the overlay; usually to permantly remove a commit which was completely broken or very problematic for various reasons.
Quote:
So I should file this as layman bug?

Why not?
Quote:
What is the correct command for that hard update?

I don't know, I never understood these details of git, and I would guess the author of layman does neither - it seems to me that nobody really understands all these details without trying out a lot of cases.
Usually, I remove and re-add an overlay with layman if it happens to me. (And indeed, I always make backups of my git directories since sometimes I am not able to restore a previous state. E.g. after checking out an old version, I get a "deprecated head", and somehow it seems sometimes not possible to restore the previous head - "git list" does not appear to show it with any option, and even if I had taken a note of the hash of the head in advance, I can check it out again, but not make it the new "head" with all rights again. git simply behaves occassionally mysterious to me, and I guess that I am not the only one...)
Back to top
View user's profile Send private message
Petros
n00b
n00b


Joined: 28 Mar 2015
Posts: 50

PostPosted: Tue Sep 15, 2015 11:01 am    Post subject: Reply with quote

I decided to compress my ~/.config directory with squashmount, but I have some trouble.

My Plasma refuses to boot with a recently (re)squashed .config dir, unless I change the owner of the directory.
Code:
sudo chown petros:petros ~/.config && sudo chown petros:petros ~/.config*


But squashmount.pl has a field in which permmisions are set for my user.
Code:
added_hash($defaults, {
               TAG => '.config',
               DIR => '/home/petros/.config',
               FILE => '/home/petros/.config.sqfs',
               CHANGES => '/home/petros/.config.changes',
               READONLY => '/home/petros/.config.readonly',
               BACKUP => '/home/petros/.config.backup',
               CHMOD => 0770, # squashfile readonly by user
               CHOWN => [ (getpwnam('petros'))[2], # user and group of ...
                       (getgrnam('petros'))[2] ],  # ... new squashfile's owner

                THRESHOLD => '1m'
#               KILL => 1, # normally remove data on every umount/remount
#               # Clean temporary directories, independent of defaults:
#               RM_CHANGES => 1, RM_WORKDIR => 1, RM_READONLY => 1,
                # If you want to cancel this KILL temporarily
                # (e.g. to make modifications on guest-skeleton.sqsf)
                # use something like "squashmount --nokill set"
                # In such a case, we must no postpone resquashing
                # even if $resquash_on_start should be true, becuse
                # CHANGES is a temporary directory:
#               RESQUASH_ON_START => '1'
}),


I tried in the first place with the default setting (after removing #) without any success. I searched for someone else's post but didn't find anything.
Back to top
View user's profile Send private message
cal22cal
n00b
n00b


Joined: 19 Jan 2006
Posts: 36

PostPosted: Sun Sep 20, 2015 2:57 am    Post subject: Reply with quote

Petros wrote:
I decided to compress my ~/.config directory with squashmount, but I have some trouble.

Got the same problem, though.
sys-fs/squashmount-12.1.1 using kernel 4.1.6-gentoo with overlay

Have to manually chown me:me /home/me/.config.changes
Then got the write right.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Sun Sep 20, 2015 10:59 am    Post subject: Reply with quote

CHMOD and CHOWN refer to the permissions of the squashfile not of any directories.

In order to change the permissions of DIR after mounting, one could use a post-mount hook.
Alternatively, squashmount-12.2.0 contains also CHMOD_DIR and CHOWN_DIR (with a syntax similar to CHMOD and CHOWN) which are used to change the mode/ownership of DIR after mounting.
Back to top
View user's profile Send private message
Petros
n00b
n00b


Joined: 28 Mar 2015
Posts: 50

PostPosted: Sun Sep 20, 2015 8:06 pm    Post subject: Reply with quote

mv wrote:

Alternatively, squashmount-12.2.0 contains also CHMOD_DIR and CHOWN_DIR (with a syntax similar to CHMOD and CHOWN) which are used to change the mode/ownership of DIR after mounting.


That solved the problem , at least for a first try / KDE boot. Thank you for your effort, you have done an excellent job :)
Back to top
View user's profile Send private message
Petros
n00b
n00b


Joined: 28 Mar 2015
Posts: 50

PostPosted: Mon Sep 28, 2015 7:34 pm    Post subject: Reply with quote

Maybe my question is irrelevant to squashmount, but how can I start an openrc service (including squashmount) if I am repairing my installation with a liveDVD/chroot?

The only way to emerge a package is to append --nodeps in order to avoid circular dependencies due to innaccesible pkg database.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Mon Sep 28, 2015 8:46 pm    Post subject: Reply with quote

Petros wrote:
Maybe my question is irrelevant to squashmount, but how can I start an openrc service (including squashmount) if I am repairing my installation with a liveDVD/chroot?

Maybe somebody corrects me, but I would say that there is no proper way to do that; copy your service to the livecd (if it supports openrc; e.g. sysrescuecd does) and start it from there.
This problem was one of the reasons why squashmount was rewritten without relying on any init system: In fact, the predecessor of squashmount (squash_dir) had introduced some special command ("START" with capital latters) which were a dirty workaround for that problem, but using just a separate command which is independent of any init-system is much cleaner, of course.
Quote:
The only way to emerge a package is to append --nodeps in order to avoid circular dependencies due to innaccesible pkg database.

Do you mean, because /var/db is squashmount-compressed? It should be possible to call "squashmount mount" in a chroot; the chroot is not necessary if you modify your config-file correspondingly (or create symlinks) for the directories/files to be mounted; the --config option to specify the config-file might be handy in such a situation.
If everything fails, you can still "unsquashfs" your /var/db.mount/db.sfs
Back to top
View user's profile Send private message
Petros
n00b
n00b


Joined: 28 Mar 2015
Posts: 50

PostPosted: Tue Sep 29, 2015 2:21 pm    Post subject: Reply with quote

mv wrote:

Do you mean, because /var/db is squashmount-compressed? It should be possible to call "squashmount mount" in a chroot; the chroot is not necessary if you modify your config-file correspondingly (or create symlinks) for the directories/files to be mounted; the --config option to specify the config-file might be handy in such a situation.
If everything fails, you can still "unsquashfs" your /var/db.mount/db.sfs


Yes, you are right. I suppose decompressing the .sfs files should apply for portage as well; not only db. After all, when gentoo is back and running, squashmount will re-compress these directoriesif .sfs files are missing.

EDIT. FTR I was refering to Gentoo LiveDVD, so yes it supports openrc.
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1759
Location: PB, Germany

PostPosted: Thu Oct 01, 2015 8:11 am    Post subject: Reply with quote

Hi, currently I get these error from squashmount:
Code:
$ squashmount remount
 * [layman]:           error:   umount skipped: dir in use: /var/lib/layman.mount/readonly
                                (this test can be skipped with --lsof=0)
 * [local_portage]:    remounting appears unnecessary
 * [portage]:          remounting appears unnecessary
 * [db]:               error:   umount skipped: dir in use: /var/db.mount/readonly
                                (this test can be skipped with --lsof=0)
I usually need the remount for manual re-squashing.
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Thu Oct 01, 2015 8:37 am    Post subject: Reply with quote

I am also confused why lsof sometimes declares files from READONLY in use, although it is possible to umount that directory. This seems to be a bug somewhere (lsof, kernel, or the overlay module).
As the output says: You can use the option --lsof=0 to skip the check. You can make this the default by setting $lsof = 0 in the configuration, although I do not think that this would be a good idea.
Back to top
View user's profile Send private message
Petros
n00b
n00b


Joined: 28 Mar 2015
Posts: 50

PostPosted: Thu Oct 01, 2015 9:47 am    Post subject: Reply with quote

Massimo B. wrote:
Hi, currently I get these error from squashmount:
Code:
$ squashmount remount
 * [layman]:           error:   umount skipped: dir in use: /var/lib/layman.mount/readonly
                                (this test can be skipped with --lsof=0)
 * [local_portage]:    remounting appears unnecessary
 * [portage]:          remounting appears unnecessary
 * [db]:               error:   umount skipped: dir in use: /var/db.mount/readonly
                                (this test can be skipped with --lsof=0)
I usually need the remount for manual re-squashing.


For the record, what kernel and squashmount version, do you use?
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1759
Location: PB, Germany

PostPosted: Thu Oct 01, 2015 9:54 am    Post subject: Reply with quote

Petros wrote:
For the record, what kernel and squashmount version, do you use?
[I] sys-fs/squashmount [1] (12.2.0@28.09.2015)
sys-kernel/ck-sources-4.0.7
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770
Back to top
View user's profile Send private message
Petros
n00b
n00b


Joined: 28 Mar 2015
Posts: 50

PostPosted: Thu Oct 01, 2015 10:12 am    Post subject: Reply with quote

Massimo B. wrote:
[I] sys-fs/squashmount [1] (12.2.0@28.09.2015)
sys-kernel/ck-sources-4.0.7


Do you use an aufs patch or something? Or you don't use aufs at all in your squashmount configuration?

I am using aufs-sources and aufs and everything is normal.
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1759
Location: PB, Germany

PostPosted: Thu Oct 01, 2015 10:15 am    Post subject: Reply with quote

Right, on that system I use unionfs-fuse. On some other system with overlayfs in the kernel, I don't have that issue.
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770
Back to top
View user's profile Send private message
Petros
n00b
n00b


Joined: 28 Mar 2015
Posts: 50

PostPosted: Thu Oct 01, 2015 11:35 am    Post subject: Reply with quote

Massimo B. wrote:
Right, on that system I use unionfs-fuse. On some other system with overlayfs in the kernel, I don't have that issue.


The other system you are refering to, has the same kernel - version, but only differs to unionfs?

Maybe the problem is unionfs or kernel specific. That's why I am asking you.

Did any kernel upgrade change the squashmount's behaviour?
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Thu Oct 01, 2015 2:29 pm    Post subject: Reply with quote

Petros wrote:
Maybe the problem is unionfs or kernel specific.

The issue can also occur with overlay(fs). In my case, sometimes some font-files from /usr/share/texmf-dist.mount/readonly/ are reported by lsof, but it is too seldom to do any useful testing. It is (if it occurs) always the same two files - I suppose that the corresponding files in /usr/share/texmf-dist/ are used by the X server or some pdf/ps/...-viewer in some circumstances. However, the report is "false" in the sense that the directory can be umounted. It is also strange that only the files in .../readonly/ are supported and not those from /usr/share/texmf-dist/, but this may be a "feature" of overlay.

Edit: Maybe it is necessary to call "lsof" with some options? Currently, squashmount essentially just calls
Code:
lsof -- /usr/share/texmf-dist.mount/readonly
(discarding any output) and tests the exit status...
Back to top
View user's profile Send private message
Petros
n00b
n00b


Joined: 28 Mar 2015
Posts: 50

PostPosted: Mon Oct 12, 2015 7:24 pm    Post subject: Reply with quote

Hey I was wondering what if we could mount some of the directories we compress on to the RAM (/tmp directory).

http://skolima.blogspot.gr/2013/05/squashfs-portage-tree-saving-time-and.html

This guy over here manages to compress (squashfs) his portage, and mount it on RAM. But he doesn't make use any of aufs hacks to redirect the changes. He uses some bash and rsync in order to recompress the changes.

I think it would be a killer feature to combine your work and aufs, with the ability to mount portage/db on RAM. Maybe it coulb be like an option, not a necessity.

What do you think about this?
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Mon Oct 12, 2015 9:41 pm    Post subject: Reply with quote

Petros wrote:
But he doesn't make use any of aufs hacks to redirect the changes.

Which means he actually cannot change it: He has to unpack the whole archive in disk, change the unpacked version, and repack the whole thing. It means a horrible waste of disk/memory space (at least temporary), and he looses all the advantages of having a writable directory. For instance, for things like /usr/share/texmf-dist or /var/db such an approach is not possible at all since these directories must be writable.

Such a thing makes sense only for PORTDIR, and for this there is the much better approach of mgorny of distributing binary diffs to the squashfile with the gentoo repository.
squashmount is ready to support that feature - everything it needs is implemented, and it was already tested with the implementation in portage - until some developers broke it: Some bugs were reported, mgorny complained about the breakage, and as a result mgorny (but actually the users) were "punished" by removing the implementation.
Bug the developers to return that feature!
Quote:
with the ability to mount portage/db on RAM

I do not undesrtand what you mean. If you have enough ram to waste, you can of course use a ramdisk for the CHANGES directory if you want, which means that except for the squashfile itself everything will be in RAM.
Back to top
View user's profile Send private message
Petros
n00b
n00b


Joined: 28 Mar 2015
Posts: 50

PostPosted: Tue Oct 13, 2015 9:13 am    Post subject: Reply with quote

What I mean with the last quote is that it could be nice if portage was on some kind of tmpfs. For example many gentoo system (with lot's of RAM) tend to mount /tmp directory as tmpfs and even redirect their TMPDIR over there, in order to compile on RAM. Anyway that's irrelevant, but I was wondering what if PORTDIR was copied/moved/you named it on /tmp and RAM? Dependencies calculation is going to be even faster?

Ayway instead of having two (I think) branches of each directory on the disk, one coulde be on tmpfs. For example when portage.sqfs is mounted and portage become visibleand accessible, the last could exist (somehow) on RAM and not on the disk. That is what I had on mind. But I guess your comment on the ramdisk, is the answer. Besides portage's size is not to be taken withouth caution.


About the binary diffs I will give it shot with a bug report.

Your patience is really appreciated. Thank you.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Tue Oct 13, 2015 3:06 pm    Post subject: Reply with quote

Petros wrote:
What I mean with the last quote is that it could be nice if portage was on some kind of tmpfs

If your CHANGES directory is empty, this won't help concerning speed, because also squashfs will be cached, and reading the metadata from the squashfile once costs only negligible time (since the metadata directory is kept together). If CHANGES is non-empty, you have presumably just synced so that this directory is cached, too.
Only if you kept the old CHANGES for a long time (e.g. on harddisk and rebooted in the meanwhile) you will experience a slowdown; that's why you should put a THRESHOLD to force resquashing after a certain minimal amount of entries of CHANGES. If you keep your system running for a long time without rebooting (so that squashmount had no change to resquash "automatically", and the time is so long that the cache for CHANGES becomes filled with other data meanwhile), I would recommend to call
Code:
squashmount remount
after a while ('though not in a cron job to avoid problems with surprisingly umounted directories).
Back to top
View user's profile Send private message
Petros
n00b
n00b


Joined: 28 Mar 2015
Posts: 50

PostPosted: Tue Oct 13, 2015 4:05 pm    Post subject: Reply with quote

The THRESHOLD value is quite low; just 1MB, so none of the problems you describe occur. Also I reboot at least once per 24h so....

Thanks :)
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1759
Location: PB, Germany

PostPosted: Mon Oct 19, 2015 6:23 am    Post subject: Reply with quote

mv wrote:
I consider this a layman bug: IMHO, layman should not only pull but actually "hard" update. (At the very least, this should be the default, perhaps with a possibility to opt out, although I do not see any reason for such an option.)
-> Bug #563464: Layman fails to reset and pull
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Mon Oct 19, 2015 3:30 pm    Post subject: Reply with quote

Massimo B. wrote:
Bug #563464: Layman fails to reset and pull

Maybe this should have been announced here, but a few days ago the mv overlay received a "dramatic" restructuring: In order to allow for
Code:
egencache --repo=mv --update-changelogs
commits must not be coupled (which was previously done in the mv overlay whenever possible to save storage and traffic).
To solve this problem either the history had to be rewritten (which is rather error-prune and a lot of manual work or very tricky scripting) or removed.
To do the latter without actual data loss, a new "historical" branch with the previous history was created, and a new master branch started from scratch.
I am not that an expert in git, but I do not know a reliable way to fetch such dramatic changes except than by removing the previous copy and cloning again; I do not think that "reset --hard" would have been sufficient this time.
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1759
Location: PB, Germany

PostPosted: Tue Oct 20, 2015 6:40 am    Post subject: Reply with quote

If I get this right, you mean, after these changes on the repo the git reset --hard origin/master should have failed as well? It did not, I'm currently fine and fixed it as usual by the reset.
Code:
>>> Syncing repository 'mv' into '/var/lib/layman/mv'...
>>> Starting layman sync for mv...
 * Running Git... # ( cd /var/lib/layman/mv  && /usr/bin/git pull --depth=1 )
remote: Total 0 (delta 0), reused 0 (delta 0)
Already up-to-date.

_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1759
Location: PB, Germany

PostPosted: Mon Nov 23, 2015 9:51 am    Post subject: Reply with quote

Massimo B. wrote:
Bug #563464: Layman fails to reset and pull
Concerning this issue I'm quite sure that my squashmount setup is responsible for the failing syncs, as nobody else encounters this errors. I'm using kernel overlay fs and lzo. The mv overlay needs the reset nearly every time now, and the kde overlay is failing completely with lot of ebuilds:
Code:
# emerge -auNDtv world --keep-going
--- Invalid atom in /var/lib/layman/kde/profiles/package.mask/kde-apps-15.08.3: <<<<<<< HEAD
--- Invalid atom in /var/lib/layman/kde/profiles/package.mask/kde-apps-15.08.3: =======
--- Invalid atom in /var/lib/layman/kde/profiles/package.mask/kde-apps-15.08.3: >>>>>>> 0fda00f147613e49cba113025d083fcf2ca17fb9

These are the packages that would be merged, in reverse order:

Calculating dependencies |/var/lib/layman/mv/app-shells/runtitle/runtitle-2.7.ebuild: line 5: syntax error near unexpected token `<<<'
/var/lib/layman/mv/app-shells/runtitle/runtitle-2.7.ebuild: line 5: `<<<<<<< HEAD'
 * ERROR: app-shells/runtitle-2.7::mv failed (depend phase):
 *   error sourcing ebuild
 *
 * Call stack:
 *   ebuild.sh, line 624:  Called die
 * The specific snippet of code:
 *            source "$EBUILD" || die "error sourcing ebuild"
 *
 * If you need support, post the output of `emerge --info '=app-shells/runtitle-2.7::mv'`,
 * the complete build log and the output of `emerge -pqv '=app-shells/runtitle-2.7::mv'`.
 * Working directory: '/usr/lib64/python2.7/site-packages'
 * S: '/var/tmp/portage/app-shells/runtitle-2.7/work/runtitle-2.7'
 \/var/lib/layman/mv/sys-apps/less/less-481.ebuild: line 5: syntax error near unexpected token `<<<'
/var/lib/layman/mv/sys-apps/less/less-481.ebuild: line 5: `<<<<<<< HEAD'
 * ERROR: sys-apps/less-481::mv failed (depend phase):
 *   error sourcing ebuild
.. and lot more

Code:
# cd /var/lib/layman/mv
# git pull
error: Pull is not possible because you have unmerged files.
hint: Fix them up in the work tree, and then use 'git add/rm <file>'
hint: as appropriate to mark resolution and make a commit.
fatal: Exiting because of an unresolved conflict.

_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8, 9, 10  Next
Page 8 of 10

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum