Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
TIP: Compressing portage using squashfs: initscript method
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3 ... , 9, 10, 11  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Sat Sep 15, 2012 1:20 pm    Post subject: Reply with quote

Gordex wrote:
tried it today but didnt get it to work properly :/

Hard to say what you tried, but none of the effects you described were ever reported or did ever occur on one of my systems.
Quote:
Compression-var seems to be ignored..I got gzip compression all the way

Perhaps you have not compiled squashfs-tools with support for any other compression algorithm. If you do not set COMPRESSION, the default is even xz.
Quote:
an error message indicated that it tried to umount the readonly before the aufs when shutdown

The code does it certainly in the correct way. Perhaps you had mounted aufs manually in addion or the umounting failed for some reason. Without more information, it is not possible to say what went wrong.
Quote:
openrc doesnt seem to launch it altho added to default runlevel..

This is the strangest thing, of course, because whether openrc starts it or not is not a question of the script. It indicates that there is something completely broken in your setup - perhaps the names of init.d/... and conf.d/ do not match, or you tried to add squash_dir instead of the symlink or whatever. Again, without more information, it is not possible to say what went wrong.
Back to top
View user's profile Send private message
Gordex
n00b
n00b


Joined: 10 Jul 2008
Posts: 26

PostPosted: Sat Sep 15, 2012 3:07 pm    Post subject: Reply with quote

I tried to doublecheck everything according to the README and got it working somehow now I believe :) thanksalot.. the problem remaining is that COMPRESSION="xz" or lzo .. are still ignored .. I can manually squash it that way (mksquashfs -comp xz ..) but ur script takes gzip here anyway
not sure why
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Sat Sep 15, 2012 3:40 pm    Post subject: Reply with quote

Try to insert into line 495 of /etc/init.d/squash_dir:
Code:
eval "einfo mksquashfs options: ${sqfsdaddopt}"

When compressing, this should display you the -comp-Option actually used. If -comp does not appear here, somewhere COMPRESSION must have been explicitly defined to the empty string in your config, because the default "xz" is set in line 311.
Back to top
View user's profile Send private message
Gordex
n00b
n00b


Joined: 10 Jul 2008
Posts: 26

PostPosted: Sat Sep 15, 2012 4:29 pm    Post subject: Reply with quote

I did but nothin seems to have changed -_-

Code:
 # /etc/init.d/squash_portage start
 * It seems you start squash_portage for the first time with this configuration:
 * The squashed file /usr/portage.sqfs does not exist yet.
 * So I will try to initialize...
 * Squashing /usr/portage to /usr/portage.sqfs
Parallel mksquashfs: Using 4 processors
Creating 4.0 filesystem on /usr/portage.sqfs, block size 131072.
[===========================================================================================================
===========================================================================================================
================================================================================|] 133549/133549 100%
Exportable Squashfs 4.0 filesystem, gzip compressed, data block size 131072
        compressed data, compressed metadata, compressed fragments, no xattrs
        duplicates are removed
Filesystem size 68404.20 Kbytes (66.80 Mbytes)
        26.03% of uncompressed filesystem size (262780.17 Kbytes)
Inode table size 1658360 bytes (1619.49 Kbytes)
        32.97% of uncompressed inode table size (5030387 bytes)
Directory table size 1511461 bytes (1476.04 Kbytes)
        36.17% of uncompressed directory table size (4178856 bytes)
Number of duplicate files found 8488
Number of inodes 157094
Number of files 133523
Number of fragments 2009
Number of symbolic links  0
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 23571
Number of ids (unique uids + gids) 1
Number of uids 1
        portage (250)
Number of gids 1
        portage (250)
 * Removing all in /usr/portage
 * Now its content is only available as squashfs.
 * Name of the squashfs file: /usr/portage.sqfs
 * Mounting /usr/portage.sqfs as /usr/portage ... 


Admin edit: Added line breaks to unbroken line of "=" characters. --pjp
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Sat Sep 15, 2012 5:01 pm    Post subject: Reply with quote

Gordex wrote:
I did but nothin seems to have changed

Now I understand: You were speaking about the "initial" mksquashfs, not about the one which is "normally" used when the directory is re-squashed.
You are right, here the option is ignored by mistake. Thanks for the report. I will fix it.
Back to top
View user's profile Send private message
Gordex
n00b
n00b


Joined: 10 Jul 2008
Posts: 26

PostPosted: Sat Sep 15, 2012 5:20 pm    Post subject: Reply with quote

ah awesome :) I see.. thx

edit: you might add that aufs3 needs to be compiled with useflag fuse..I think thats what causes my earlier problems but not 100% sure
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1753
Location: PB, Germany

PostPosted: Mon Nov 12, 2012 2:00 pm    Post subject: Reply with quote

Hi, again. Is there any rescue if aufs is not available?
Today I started my new kernel, but I forgot to build aufs3 kernel module. Now the emerge aufs3 works but fails to install, probably because /var/db is readonly.
I tried to install the module and the binaries from /var/tmp/portage manually but modprobe aufs3 still says "module not found" eventhough the .ko file is in the right directory (and found by bash_completion).
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Mon Nov 12, 2012 5:38 pm    Post subject: Reply with quote

Massimo B. wrote:
Hi, again. Is there any rescue if aufs is not available?

Yes, but it is inconvenient: You can manually copy the readonly /var/db to some new temporary directory, then stop the squash_...-scripts, rename, /var/db to something else, and then rename the new temporary directory to /var/db. (If /var/db does not even a sane [readonly] content, you can also use unsquashfs to unpack /var/db.sqfs manually).
Of course, this works only if you have sufficient disk space for the unpackad copy of /var/db, but perhaps you can temporarily get the free space by moving something to a backup medium which you do not need for the restauration.
Then emerge aufs (hopefully it works now).
Finally rename both directories back and then reboot (or start squash_... if it works).
When everything is running you should remove the old data of /var/db/pkg/sys-fs/aufs-... and move the new data (from your last emerge) which is now in the renamed-back temporary directory to /var/db/pkg/sys-fs/aufs-...
After this the temporary directory can be deleted.
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1753
Location: PB, Germany

PostPosted: Tue Nov 13, 2012 11:19 am    Post subject: Reply with quote

Thanks for the help. That worked. Though I needed to force ORDER="aufs" in all the configurations, maybe because of the recent change. I think ORDER does mean the order of testing. But eventhough aufs already was in lsmod, squash_dir restart failed because of missing overlayfs.

If aufs3 will be the next default aufs I hope it goes to the kernel soon without requiring the kernel patches. Next kernel switch I need to remember to emerge aufs3 first against the new sources and build the aufs3-patched sources afterwards.
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Tue Nov 13, 2012 5:08 pm    Post subject: Reply with quote

Massimo B. wrote:
I think ORDER does mean the order of testing. But eventhough aufs already was in lsmod, squash_dir restart failed because of missing overlayfs.

Yes, ORDER means the order of trying. So it should work even with the default ORDER but perhaps only print some error messages - but it should not fail if aufs succeeds.
Quote:
If aufs3 will be the next default aufs I hope it goes to the kernel soon without requiring the kernel patches.

Unfortunately, the chances for this are practically zero, since despite a huge interest from the users, the kernel maintainers are not interested in including any type of union filesystem. Some kernel maintainers had announced that perhaps they might give overlayfs a chance. Although I am not so optimistic (since many months it is ignored, practically without any comments - the same has already happened to union mount for many years), maybe this is the first union type filesystem which has a chance to go into the kernel. For this reason, I have changed the default. I will of course change the default immediately back, if it seems more likely that aufs is included or overlayfs is going to be rejected.
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1753
Location: PB, Germany

PostPosted: Thu Nov 22, 2012 8:53 am    Post subject: Reply with quote

Thanks, the discussion about integrating aufs3 was the same with squashfs which finally was integrated into 2.6.29 after half of the Linux world was already using it. [german article]

Some other idea, when I was thinking about compressing binaries such as large firefox using UPX for faster startup: I could also just add things like /usr/lib64/firefox to squash_dir. That should just work with upgrades and re-squashing for now, since I even restart squash_dir once per day by cron for potential re-squashing.

EDIT: I successfully squashed firefox, from 43M to 20M, and startup feels a bit faster...
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1753
Location: PB, Germany

PostPosted: Tue Dec 04, 2012 9:02 am    Post subject: Reply with quote

Hi again. I have a daily cronjob for re-squashing. How save actually is re-squashing by squash_dir restart when currently writing to this filesystem?
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Tue Dec 04, 2012 3:01 pm    Post subject: Reply with quote

Massimo B. wrote:
Hi again. I have a daily cronjob for re-squashing. How save actually is re-squashing by squash_dir restart when currently writing to this filesystem?

It is not save; you can loose any new information and perhaps even get a broken squash file. Even the remounting itself might fail if the filesystem is busy at that time.
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1753
Location: PB, Germany

PostPosted: Wed Dec 19, 2012 10:07 am    Post subject: squash_dir from rescue live-cd chroot Reply with quote

Before the last openrc update
Code:
    Tue Dec 18 21:18:32 2012 <<< sys-apps/openrc-0.9.8.4
    Tue Dec 18 21:18:40 2012 >>> sys-apps/openrc-0.11.8
..it was just possible to do squash_dir start or restart from inside a live-cd chroot. There (using Finnix on PPC) aufs and squashfs modules are provided. It was possible to mount the squash_dir parts rw and even recompress. So portage was completely functional from inside chroot.
Since the latest openrc update I get the following:
Code:
$ squash_dir zap
 * You are attempting to run an openrc service on a
 * system which openrc did not boot.
 * You may be inside a chroot or you may have used
 * another initialization system to boot this system.
 * In this situation, you will get unpredictable results!
 * If you really want to do this, issue the following command:
 * touch /run/openrc/softlevel
/etc/init.d/squash_db is not running...

Doing the touch /run... openrc tries to start bad dependencies:
Code:
$ squash_dir start
Starting db...
 * Caching service dependencies ...                                         [ ok ]
 * Mounting security filesystem ...                                         [ ok ]
 * Mounting debug filesystem ...                                            [ ok ]
 * Mounting cgroup filesystem ...                                           [ ok ]
 * /dev is already mounted
 * Starting udev ...                                                        [ ok ]
 * Populating /dev with existing devices through uevents ...                [ ok ]
 * Waiting for uevents to be processed ...                                  [ ok ]
 * Checking local filesystems  ...
/dev/sda5 is in use.
e2fsck: Cannot continue, aborting.


 * Operational error                                                        [ !! ]
 * Mounting local filesystems ...                                           [ ok ]
 * Mounting /var/db.sqfs as /var/db ...
mount: /var/db.sqfs is already mounted
 * Failed mounting /var/db.sqfs as /var/db.readonly [exit with 1]
rmdir: failed to remove �/var/db.readonly�: Device or resource busy         [ !! ]
 * ERROR: squash_db failed to start
Skipping /etc/init.d/squash_layman due to previous error
Skipping /etc/init.d/squash_local_portage due to previous error
Skipping /etc/init.d/squash_portage due to previous error

Is there any way to avoid the dependencies to get squash_dir in a working state within chroot as it was working before?
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Wed Dec 19, 2012 5:24 pm    Post subject: Re: squash_dir from rescue live-cd chroot Reply with quote

Massimo B. wrote:
squash_dir start or restart from inside a live-cd chroot.

In the new squash_dir-12.6 there is now a START command which executes the action of the "start" command but does not actually start the script. You can use the STOP command (which exists already quite a while) to execute the action of the "stop" command of such an officially non-started script. Of course, you must use these hacks very carefully: Double execution or stopping can have undesired results, especially if the temporary filename feature is used for the readonly directory.
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1753
Location: PB, Germany

PostPosted: Tue Jan 08, 2013 9:14 am    Post subject: Reply with quote

Thanks. I was able to start squash_dir even within chroot from live-cd now.

Next bug report, I tried to squash new directories:
Code:
$ cat /etc/conf.d/squash_icedtea6
DIRECTORY="/usr/lib/icedtea6"
DIR_CHANGE="${DIRECTORY}.changes"
DIR_SQUASH="${DIRECTORY}.readonly"
ORDER="aufs"
COMPRESSION=""
Code:
$ /etc/init.d/squash_icedtea6 start
squash_icedtea6     | * Caching service dependencies ...                                                                  [ ok ]
squash_icedtea6     | * It seems you start squash_icedtea6 for the first time with this configuration:
squash_icedtea6     | * The squashed file /usr/lib64/icedtea6.sqfs does not exist yet.
squash_icedtea6     | * So I will try to initialize...
squash_icedtea6     | * Squashing /usr/lib64/icedtea6 to /usr/lib64/icedtea6.sqfs
squash_icedtea6     |SYNTAX:mksquashfs source1 source2 ...  dest [options] [-e list of exclude...

EDIT: Parameter order has changed somehow, works now with this patch:
Code:
$ diff -u /etc/init.d/squash_dir /etc/init.d/squash_dir.new
--- /etc/init.d/squash_dir   2013-01-08 10:54:05.946782911 +0100
+++ /etc/init.d/squash_dir.new   2013-01-08 10:53:21.788301570 +0100
@@ -247,8 +247,8 @@
       return 2
    }
    einfo "Squashing ${DIRECTORY} to ${FILE_SQFS}"
-   mksquashfs ${COMPRESSION:+-comp "${COMPRESSION}"} ${MKSQUASHFS} \
-      "${DIRECTORY}" "${FILE_SQFS}" || {
+   mksquashfs "${DIRECTORY}" "${FILE_SQFS}" \
+          ${COMPRESSION:+-comp "${COMPRESSION}"} ${MKSQUASHFS} || {
       Error ${?} "mksquashfs failed. Exiting."
       return
    }

_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Tue Jan 08, 2013 8:45 pm    Post subject: Reply with quote

This is fixed in squash_dir-12.8
Back to top
View user's profile Send private message
tabascoz
n00b
n00b


Joined: 14 May 2003
Posts: 45
Location: Rio de Janeiro

PostPosted: Thu Aug 01, 2013 6:10 pm    Post subject: Reply with quote

Hello,

Did someone has written a systemd init script for squash_dir? Since the last update to gnome 3.8 and, thus, the need to switch to systemd , i had been unable to use this great toot.
_________________
Tabascoz
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Fri Aug 02, 2013 8:48 am    Post subject: Reply with quote

tabascoz wrote:
Did someone has written a systemd init script for squash_dir?

AFAIK systemd is too poor to do this. I will not support this bad systemd concept, anyway.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Wed Aug 14, 2013 5:51 pm    Post subject: Reply with quote

mv wrote:
AFAIK systemd is too poor to do this.

It turned out that it can be done with systemd as well - only it is not possible to send the output to the terminal; that's a restriction due to the bad systemd concept. Thus, here is the announcement

squash_dir-13.0 does support systemd, experimentally.

It also contains a script openrc-wrapper which allows to use many native openrc services in other init-systems like systemd. This might be interesting also for other openrc init-scripts.

So this is the implementation: The script is native openrc, but can be fully used in systemd by the openrc-wrapper. The advantage of this solution for gentoo users is that the same configuration can be used for systemd as for openrc, i.e. squash_dir does not have to be configured twice on different places. The disadvantage is of course that native systemd users might not want to configure in /etc/conf.d and to create squash_* symlinks in /etc/init.d which is currently necessary even if openrc is not installed. In the long run, squash_dir might be replaced e.g. by a perl-script which manages all functionality in a single script which can be called externally and which manages its own database of "mounted" directories in e.g. /run, but I will not have time in the near future to implement such a universal solution. Anyway, this would be different project and should have a different name...
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 20048

PostPosted: Fri Apr 06, 2018 5:06 pm    Post subject: Reply with quote

Split off Compressing filesystems with squashfs and squashmount
_________________
Quis separabit? Quo animo?
Back to top
View user's profile Send private message
Khumarahn
Apprentice
Apprentice


Joined: 17 Apr 2009
Posts: 199

PostPosted: Mon Sep 24, 2018 1:23 pm    Post subject: Reply with quote

What is the state of squashfs with squashdelta for portage tree?

Using the lines from teh wiki
Code:

# for daily squashfs snapshots
sync-type = squashdelta
sync-uri = mirror://gentoo/../snapshots/squashfs

results in
Code:

# emerge --sync
!!! Repository 'gentoo' has sync-type attribute set to unsupported value: 'squashdelta'
!!! Installed sync-types are: '['cvs', 'git', 'rsync', 'svn', 'webrsync']'
!!! Repository 'gentoo' has sync-type attribute set to unsupported value: 'squashdelta'
!!! Installed sync-types are: '['cvs', 'git', 'rsync', 'svn', 'webrsync']'
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Mon Sep 24, 2018 7:20 pm    Post subject: Reply with quote

Khumarahn wrote:
What is the state of squashfs with squashdelta for portage tree?

It is abandoned, and AFAIK there are no plans to revive it.
Back to top
View user's profile Send private message
Khumarahn
Apprentice
Apprentice


Joined: 17 Apr 2009
Posts: 199

PostPosted: Mon Sep 24, 2018 7:40 pm    Post subject: Reply with quote

Thanks!!

What is the easiest way to keep portage in squashfs, and recompress in on emerge --sync?
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 20048

PostPosted: Mon Sep 24, 2018 8:49 pm    Post subject: Reply with quote

I've been using tmpfs and squashfs.

Create a tmpfs.
Create a squasfs of the portage tree.

When I sync, the squashfs is extracted to a tmpfs, the sync occurs to the in memory tmpfs and then a new squashfs is created. The tmpfs is then discarded (freeing the memory) and the new portage tree squashfs is mounted.

That process requires enough free memory to temporarily hold the tree during that sync / squashfs process.
Also, /usr/portage/distfiles and /usr/portage/packages are kept elsewhere. These directories were configurable using
DISTDIR & PKGDIR in make.conf. man make.conf now references repos.conf. I've not found new documentation for their use in repos.conf (but since make.conf seems to work, I haven't made it a priority).

If I ever get around to making the script more generic, I had planned to post it.
_________________
Quis separabit? Quo animo?
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3 ... , 9, 10, 11  Next
Page 10 of 11

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum