Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
TIP: Compressing portage using squashfs: initscript method
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3 ... , 17, 18, 19  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
Petros
n00b
n00b


Joined: 28 Mar 2015
Posts: 50

PostPosted: Mon Sep 28, 2015 7:34 pm    Post subject: Reply with quote

Maybe my question is irrelevant to squashmount, but how can I start an openrc service (including squashmount) if I am repairing my installation with a liveDVD/chroot?

The only way to emerge a package is to append --nodeps in order to avoid circular dependencies due to innaccesible pkg database.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 5754

PostPosted: Mon Sep 28, 2015 8:46 pm    Post subject: Reply with quote

Petros wrote:
Maybe my question is irrelevant to squashmount, but how can I start an openrc service (including squashmount) if I am repairing my installation with a liveDVD/chroot?

Maybe somebody corrects me, but I would say that there is no proper way to do that; copy your service to the livecd (if it supports openrc; e.g. sysrescuecd does) and start it from there.
This problem was one of the reasons why squashmount was rewritten without relying on any init system: In fact, the predecessor of squashmount (squash_dir) had introduced some special command ("START" with capital latters) which were a dirty workaround for that problem, but using just a separate command which is independent of any init-system is much cleaner, of course.
Quote:
The only way to emerge a package is to append --nodeps in order to avoid circular dependencies due to innaccesible pkg database.

Do you mean, because /var/db is squashmount-compressed? It should be possible to call "squashmount mount" in a chroot; the chroot is not necessary if you modify your config-file correspondingly (or create symlinks) for the directories/files to be mounted; the --config option to specify the config-file might be handy in such a situation.
If everything fails, you can still "unsquashfs" your /var/db.mount/db.sfs
Back to top
View user's profile Send private message
Petros
n00b
n00b


Joined: 28 Mar 2015
Posts: 50

PostPosted: Tue Sep 29, 2015 2:21 pm    Post subject: Reply with quote

mv wrote:

Do you mean, because /var/db is squashmount-compressed? It should be possible to call "squashmount mount" in a chroot; the chroot is not necessary if you modify your config-file correspondingly (or create symlinks) for the directories/files to be mounted; the --config option to specify the config-file might be handy in such a situation.
If everything fails, you can still "unsquashfs" your /var/db.mount/db.sfs


Yes, you are right. I suppose decompressing the .sfs files should apply for portage as well; not only db. After all, when gentoo is back and running, squashmount will re-compress these directoriesif .sfs files are missing.

EDIT. FTR I was refering to Gentoo LiveDVD, so yes it supports openrc.
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1479
Location: Germany

PostPosted: Thu Oct 01, 2015 8:11 am    Post subject: Reply with quote

Hi, currently I get these error from squashmount:
Code:
$ squashmount remount
 * [layman]:           error:   umount skipped: dir in use: /var/lib/layman.mount/readonly
                                (this test can be skipped with --lsof=0)
 * [local_portage]:    remounting appears unnecessary
 * [portage]:          remounting appears unnecessary
 * [db]:               error:   umount skipped: dir in use: /var/db.mount/readonly
                                (this test can be skipped with --lsof=0)
I usually need the remount for manual re-squashing.
_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G Ram|amd64:HP EliteBook 8560w,i7-2620M,16G Ram|amd64:Acer Z5610 (Core2QuadQ8200),8G Ram|amd64-prefix:OpenSuse
Lila-Theme
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 5754

PostPosted: Thu Oct 01, 2015 8:37 am    Post subject: Reply with quote

I am also confused why lsof sometimes declares files from READONLY in use, although it is possible to umount that directory. This seems to be a bug somewhere (lsof, kernel, or the overlay module).
As the output says: You can use the option --lsof=0 to skip the check. You can make this the default by setting $lsof = 0 in the configuration, although I do not think that this would be a good idea.
Back to top
View user's profile Send private message
Petros
n00b
n00b


Joined: 28 Mar 2015
Posts: 50

PostPosted: Thu Oct 01, 2015 9:47 am    Post subject: Reply with quote

Massimo B. wrote:
Hi, currently I get these error from squashmount:
Code:
$ squashmount remount
 * [layman]:           error:   umount skipped: dir in use: /var/lib/layman.mount/readonly
                                (this test can be skipped with --lsof=0)
 * [local_portage]:    remounting appears unnecessary
 * [portage]:          remounting appears unnecessary
 * [db]:               error:   umount skipped: dir in use: /var/db.mount/readonly
                                (this test can be skipped with --lsof=0)
I usually need the remount for manual re-squashing.


For the record, what kernel and squashmount version, do you use?
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1479
Location: Germany

PostPosted: Thu Oct 01, 2015 9:54 am    Post subject: Reply with quote

Petros wrote:
For the record, what kernel and squashmount version, do you use?
[I] sys-fs/squashmount [1] (12.2.0@28.09.2015)
sys-kernel/ck-sources-4.0.7
_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G Ram|amd64:HP EliteBook 8560w,i7-2620M,16G Ram|amd64:Acer Z5610 (Core2QuadQ8200),8G Ram|amd64-prefix:OpenSuse
Lila-Theme
Back to top
View user's profile Send private message
Petros
n00b
n00b


Joined: 28 Mar 2015
Posts: 50

PostPosted: Thu Oct 01, 2015 10:12 am    Post subject: Reply with quote

Massimo B. wrote:
[I] sys-fs/squashmount [1] (12.2.0@28.09.2015)
sys-kernel/ck-sources-4.0.7


Do you use an aufs patch or something? Or you don't use aufs at all in your squashmount configuration?

I am using aufs-sources and aufs and everything is normal.
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1479
Location: Germany

PostPosted: Thu Oct 01, 2015 10:15 am    Post subject: Reply with quote

Right, on that system I use unionfs-fuse. On some other system with overlayfs in the kernel, I don't have that issue.
_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G Ram|amd64:HP EliteBook 8560w,i7-2620M,16G Ram|amd64:Acer Z5610 (Core2QuadQ8200),8G Ram|amd64-prefix:OpenSuse
Lila-Theme
Back to top
View user's profile Send private message
Petros
n00b
n00b


Joined: 28 Mar 2015
Posts: 50

PostPosted: Thu Oct 01, 2015 11:35 am    Post subject: Reply with quote

Massimo B. wrote:
Right, on that system I use unionfs-fuse. On some other system with overlayfs in the kernel, I don't have that issue.


The other system you are refering to, has the same kernel - version, but only differs to unionfs?

Maybe the problem is unionfs or kernel specific. That's why I am asking you.

Did any kernel upgrade change the squashmount's behaviour?
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 5754

PostPosted: Thu Oct 01, 2015 2:29 pm    Post subject: Reply with quote

Petros wrote:
Maybe the problem is unionfs or kernel specific.

The issue can also occur with overlay(fs). In my case, sometimes some font-files from /usr/share/texmf-dist.mount/readonly/ are reported by lsof, but it is too seldom to do any useful testing. It is (if it occurs) always the same two files - I suppose that the corresponding files in /usr/share/texmf-dist/ are used by the X server or some pdf/ps/...-viewer in some circumstances. However, the report is "false" in the sense that the directory can be umounted. It is also strange that only the files in .../readonly/ are supported and not those from /usr/share/texmf-dist/, but this may be a "feature" of overlay.

Edit: Maybe it is necessary to call "lsof" with some options? Currently, squashmount essentially just calls
Code:
lsof -- /usr/share/texmf-dist.mount/readonly
(discarding any output) and tests the exit status...
Back to top
View user's profile Send private message
Petros
n00b
n00b


Joined: 28 Mar 2015
Posts: 50

PostPosted: Mon Oct 12, 2015 7:24 pm    Post subject: Reply with quote

Hey I was wondering what if we could mount some of the directories we compress on to the RAM (/tmp directory).

http://skolima.blogspot.gr/2013/05/squashfs-portage-tree-saving-time-and.html

This guy over here manages to compress (squashfs) his portage, and mount it on RAM. But he doesn't make use any of aufs hacks to redirect the changes. He uses some bash and rsync in order to recompress the changes.

I think it would be a killer feature to combine your work and aufs, with the ability to mount portage/db on RAM. Maybe it coulb be like an option, not a necessity.

What do you think about this?
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 5754

PostPosted: Mon Oct 12, 2015 9:41 pm    Post subject: Reply with quote

Petros wrote:
But he doesn't make use any of aufs hacks to redirect the changes.

Which means he actually cannot change it: He has to unpack the whole archive in disk, change the unpacked version, and repack the whole thing. It means a horrible waste of disk/memory space (at least temporary), and he looses all the advantages of having a writable directory. For instance, for things like /usr/share/texmf-dist or /var/db such an approach is not possible at all since these directories must be writable.

Such a thing makes sense only for PORTDIR, and for this there is the much better approach of mgorny of distributing binary diffs to the squashfile with the gentoo repository.
squashmount is ready to support that feature - everything it needs is implemented, and it was already tested with the implementation in portage - until some developers broke it: Some bugs were reported, mgorny complained about the breakage, and as a result mgorny (but actually the users) were "punished" by removing the implementation.
Bug the developers to return that feature!
Quote:
with the ability to mount portage/db on RAM

I do not undesrtand what you mean. If you have enough ram to waste, you can of course use a ramdisk for the CHANGES directory if you want, which means that except for the squashfile itself everything will be in RAM.
Back to top
View user's profile Send private message
Petros
n00b
n00b


Joined: 28 Mar 2015
Posts: 50

PostPosted: Tue Oct 13, 2015 9:13 am    Post subject: Reply with quote

What I mean with the last quote is that it could be nice if portage was on some kind of tmpfs. For example many gentoo system (with lot's of RAM) tend to mount /tmp directory as tmpfs and even redirect their TMPDIR over there, in order to compile on RAM. Anyway that's irrelevant, but I was wondering what if PORTDIR was copied/moved/you named it on /tmp and RAM? Dependencies calculation is going to be even faster?

Ayway instead of having two (I think) branches of each directory on the disk, one coulde be on tmpfs. For example when portage.sqfs is mounted and portage become visibleand accessible, the last could exist (somehow) on RAM and not on the disk. That is what I had on mind. But I guess your comment on the ramdisk, is the answer. Besides portage's size is not to be taken withouth caution.


About the binary diffs I will give it shot with a bug report.

Your patience is really appreciated. Thank you.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 5754

PostPosted: Tue Oct 13, 2015 3:06 pm    Post subject: Reply with quote

Petros wrote:
What I mean with the last quote is that it could be nice if portage was on some kind of tmpfs

If your CHANGES directory is empty, this won't help concerning speed, because also squashfs will be cached, and reading the metadata from the squashfile once costs only negligible time (since the metadata directory is kept together). If CHANGES is non-empty, you have presumably just synced so that this directory is cached, too.
Only if you kept the old CHANGES for a long time (e.g. on harddisk and rebooted in the meanwhile) you will experience a slowdown; that's why you should put a THRESHOLD to force resquashing after a certain minimal amount of entries of CHANGES. If you keep your system running for a long time without rebooting (so that squashmount had no change to resquash "automatically", and the time is so long that the cache for CHANGES becomes filled with other data meanwhile), I would recommend to call
Code:
squashmount remount
after a while ('though not in a cron job to avoid problems with surprisingly umounted directories).
Back to top
View user's profile Send private message
Petros
n00b
n00b


Joined: 28 Mar 2015
Posts: 50

PostPosted: Tue Oct 13, 2015 4:05 pm    Post subject: Reply with quote

The THRESHOLD value is quite low; just 1MB, so none of the problems you describe occur. Also I reboot at least once per 24h so....

Thanks :)
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1479
Location: Germany

PostPosted: Mon Oct 19, 2015 6:23 am    Post subject: Reply with quote

mv wrote:
I consider this a layman bug: IMHO, layman should not only pull but actually "hard" update. (At the very least, this should be the default, perhaps with a possibility to opt out, although I do not see any reason for such an option.)
-> Bug #563464: Layman fails to reset and pull
_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G Ram|amd64:HP EliteBook 8560w,i7-2620M,16G Ram|amd64:Acer Z5610 (Core2QuadQ8200),8G Ram|amd64-prefix:OpenSuse
Lila-Theme
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 5754

PostPosted: Mon Oct 19, 2015 3:30 pm    Post subject: Reply with quote

Massimo B. wrote:
Bug #563464: Layman fails to reset and pull

Maybe this should have been announced here, but a few days ago the mv overlay received a "dramatic" restructuring: In order to allow for
Code:
egencache --repo=mv --update-changelogs
commits must not be coupled (which was previously done in the mv overlay whenever possible to save storage and traffic).
To solve this problem either the history had to be rewritten (which is rather error-prune and a lot of manual work or very tricky scripting) or removed.
To do the latter without actual data loss, a new "historical" branch with the previous history was created, and a new master branch started from scratch.
I am not that an expert in git, but I do not know a reliable way to fetch such dramatic changes except than by removing the previous copy and cloning again; I do not think that "reset --hard" would have been sufficient this time.
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1479
Location: Germany

PostPosted: Tue Oct 20, 2015 6:40 am    Post subject: Reply with quote

If I get this right, you mean, after these changes on the repo the git reset --hard origin/master should have failed as well? It did not, I'm currently fine and fixed it as usual by the reset.
Code:
>>> Syncing repository 'mv' into '/var/lib/layman/mv'...
>>> Starting layman sync for mv...
 * Running Git... # ( cd /var/lib/layman/mv  && /usr/bin/git pull --depth=1 )
remote: Total 0 (delta 0), reused 0 (delta 0)
Already up-to-date.

_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G Ram|amd64:HP EliteBook 8560w,i7-2620M,16G Ram|amd64:Acer Z5610 (Core2QuadQ8200),8G Ram|amd64-prefix:OpenSuse
Lila-Theme
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1479
Location: Germany

PostPosted: Mon Nov 23, 2015 9:51 am    Post subject: Reply with quote

Massimo B. wrote:
Bug #563464: Layman fails to reset and pull
Concerning this issue I'm quite sure that my squashmount setup is responsible for the failing syncs, as nobody else encounters this errors. I'm using kernel overlay fs and lzo. The mv overlay needs the reset nearly every time now, and the kde overlay is failing completely with lot of ebuilds:
Code:
# emerge -auNDtv world --keep-going
--- Invalid atom in /var/lib/layman/kde/profiles/package.mask/kde-apps-15.08.3: <<<<<<< HEAD
--- Invalid atom in /var/lib/layman/kde/profiles/package.mask/kde-apps-15.08.3: =======
--- Invalid atom in /var/lib/layman/kde/profiles/package.mask/kde-apps-15.08.3: >>>>>>> 0fda00f147613e49cba113025d083fcf2ca17fb9

These are the packages that would be merged, in reverse order:

Calculating dependencies |/var/lib/layman/mv/app-shells/runtitle/runtitle-2.7.ebuild: line 5: syntax error near unexpected token `<<<'
/var/lib/layman/mv/app-shells/runtitle/runtitle-2.7.ebuild: line 5: `<<<<<<< HEAD'
 * ERROR: app-shells/runtitle-2.7::mv failed (depend phase):
 *   error sourcing ebuild
 *
 * Call stack:
 *   ebuild.sh, line 624:  Called die
 * The specific snippet of code:
 *            source "$EBUILD" || die "error sourcing ebuild"
 *
 * If you need support, post the output of `emerge --info '=app-shells/runtitle-2.7::mv'`,
 * the complete build log and the output of `emerge -pqv '=app-shells/runtitle-2.7::mv'`.
 * Working directory: '/usr/lib64/python2.7/site-packages'
 * S: '/var/tmp/portage/app-shells/runtitle-2.7/work/runtitle-2.7'
 \/var/lib/layman/mv/sys-apps/less/less-481.ebuild: line 5: syntax error near unexpected token `<<<'
/var/lib/layman/mv/sys-apps/less/less-481.ebuild: line 5: `<<<<<<< HEAD'
 * ERROR: sys-apps/less-481::mv failed (depend phase):
 *   error sourcing ebuild
.. and lot more

Code:
# cd /var/lib/layman/mv
# git pull
error: Pull is not possible because you have unmerged files.
hint: Fix them up in the work tree, and then use 'git add/rm <file>'
hint: as appropriate to mark resolution and make a commit.
fatal: Exiting because of an unresolved conflict.

_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G Ram|amd64:HP EliteBook 8560w,i7-2620M,16G Ram|amd64:Acer Z5610 (Core2QuadQ8200),8G Ram|amd64-prefix:OpenSuse
Lila-Theme
Back to top
View user's profile Send private message
schorsch_76
Guru
Guru


Joined: 19 Jun 2012
Posts: 450

PostPosted: Mon Nov 23, 2015 12:26 pm    Post subject: Reply with quote

This Problem with the mv overlay is most probably cause by rebase on the git repository. As a rule of thumb: "Never ever rebase any published branches".

As a git development model suitable for all of such issues:
http://nvie.com/posts/a-successful-git-branching-model/
_________________
// valid again: I forgot about the git access. Now 1.2GB big. Start: 2015-06-25
git daily portage tree
Web: https://portage.schorsch-tech.de
git clone https://portage.schorsch-tech.de/portage.git
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 5754

PostPosted: Mon Nov 23, 2015 6:01 pm    Post subject: Reply with quote

schorsch_76 wrote:
This Problem with the mv overlay is most probably cause by rebase on the git repository

There was no rebase since the one announced on Oct 19.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 5754

PostPosted: Wed Nov 25, 2015 9:37 am    Post subject: Reply with quote

schorsch_76 wrote:
As a rule of thumb: "Never ever rebase any published branches".

BTW, this rule might be useful for projects with many people involved and with kind of systematic QA before pushing to public - iit does not apply for experimental overlays managed by a single person where normally every change should go pulbic immediately (perhaps after a very crude testing).
But even for these other projects, it is only a rule of thumb. There might be good reasons why a bad history slipped through which should be eliminated forever:
A file unrelated to the project and copied by mistake from somewhere else by one developer (and slipped through QA) might be such a reason.
Not unnecessarily wasting disk space for known-to-be-broken commits is another reason.
A change of commit policy (as e.g. explained several postings ago) is yet another reason.
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1479
Location: Germany

PostPosted: Mon Jan 18, 2016 2:47 pm    Post subject: Reply with quote

Anyway, in the meanwhile I can't really work anymore concerning squashmount and git repos like the overlays. Curiously the big portage git repo doesn't have these issues. But as for the layman overlays almost every eix-sync is failing with tons of broken ebuilds like
Code:
ebuild failed with status 1
     Reading category  80|166 ( 48): kde-apps...
cannot properly execute /var/lib/layman/kde/kde-apps/okteta/okteta-9999.ebuild
     Reading category  80|166 ( 48): kde-apps.../var/lib/layman/kde/eclass/cmake-utils.eclass: line 67: syntax error near unexpected token `<<<'
/var/lib/layman/kde/eclass/cmake-utils.eclass: line 67: `<<<<<<< HEAD'
or
Code:
-- invalid line 141 in /var/lib/layman/kde/profiles/package.mask: "<<<<<<< ..."
    cannot read category

-- invalid line 142 in /var/lib/layman/kde/profiles/package.mask: "======= ..."
    cannot read category

-- invalid line 144 in /var/lib/layman/kde/profiles/package.mask: ">>>>>>> ..."
    cannot read category

As a short solution I currently need to flush all overlays before eix-sync which is not a long-time solution:
Code:
overlays="flavour kde mv sunrise";layman -d $overlays;layman -a $overlays

Can you take a look what is wrong with my setup? (I did not mention all mount points such as firefox, libreoffice, texlive etc.)
/etc/squashmount.pl:

use Sys::Hostname;
my $hostname = ($ENV{'HOSTNAME'} // hostname());

@order = qw(overlay unionfs-fuse);

my $defaults = {

      COMPRESSION => 'lzo',
      THRESHOLD   => '10m',
};
my $non_binary = {
      COMPOPT_XZ => undef # "-Xbcj x86" is slower for pure text archives
};

@mounts = (
      standard_mount('layman',        '/var/lib/layman',      $defaults),
      standard_mount('local_portage', '/usr/local/portage',   $defaults),

      standard_mount('portage', '/usr/portage', $defaults, $non_binary, {
                  UMOUNT => ((@umount) ? undef : '-i'),
                  THRESHOLD => '100m',
                  FILL => qr{^local/(?!(?:\.git|profiles|metadata)(?:/|$))}
            }),
      standard_mount('db', '/var/db', $defaults, $non_binary, {
                  UMOUNT => ((@umount) ? undef : '-i'),
                  THRESHOLD => '40m',
                  BACKUP => '.bak'
            })
);

'EOF';

_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G Ram|amd64:HP EliteBook 8560w,i7-2620M,16G Ram|amd64:Acer Z5610 (Core2QuadQ8200),8G Ram|amd64-prefix:OpenSuse
Lila-Theme
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 5754

PostPosted: Mon Jan 18, 2016 9:35 pm    Post subject: Reply with quote

Massimo B. wrote:
Code:
"<<<<<<< ..."
    cannot read category

-- invalid line 142 in /var/lib/layman/kde/profiles/package.mask: "======= ..."
    cannot read category

-- invalid line 144 in /var/lib/layman/kde/profiles/package.mask: ">>>>>>> ..."
    cannot read category

Such lines are caused by "git pull" when there are local changes or when the overlay was rebased. It has only to do with git/layman and nothing to do with squashmount.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3 ... , 17, 18, 19  Next
Page 18 of 19

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum