app-portage/cfg-update - installation instructions

[F.Ultra]

I think SSHFS and FUSE would be our best option here. That would give us access to remote filesystems as if they were on the localhost itself.

I like this! The traditional route taken by portage & co is to expose the management server (for example the portage tree) via NFS and then the managed servers have cronjobs that connect to the management server in order to do their magic.

This requires two things:

#1 The management server has to be positioned in your network so the managed servers can contact it, more or less dictades a place in the DMZ or even externally.
#2 The managed servers has to have extra software installed, for example the cronjobs etc.

With your thoughts about using SSHFS the situation is reversed, aka the management server can be put into the safer network and we only have to allow it to acess the various networks where the managed servers are, and also the software that we use to manage the servers only have to be installed on the management server! (i.e cfg-update and say paludis only need to be installed on the management server)

Another implification is that the management can be performed when the admin says so and not when some remote cronjob are executed. And it is far easier to let the management server reach out to x number of remote serverns per run (in case we have 10000 servers we do not want to start 10000 parelell sessions) than it is to tune the timings of the remote crons.

[xentric]

After lot's of testing I have finally submitted the new version 1.8.2
- all reported bugs and annoyances are fixed
- backups are now stored in a single location (/var/lib/cfg-update/backups)
- alias for emerge has been replaced by hooks for Portage and Paludis
- added support for package manager Paludis
- added support for mergetool imediff2
- cfg-update can now update multiple hosts from a single location! (see /etc/cfg-update.hosts)

Warning: if you have used Paludis <0.20.0, you need to run cfg-update --check-packages and follow the instructions! Due to a "bug" in the older versions of Paludis you risk losing custom settings in the configuration files, belonging to packages that were installed with Paludis <0.20.0, when using cfg-update.

Thanks go out to the users who have contributed new ideas and especially zxy for helping me with the Paludis issues!

Have fun...

[XenoTerraCide]

great! excellent work. will be testing soon.

[devsk]

Awesome!! I think single location for backup, was the most waited fix. Thank you so much!

[zxy]

xentric good work on latest version, i tested it with paludis on two machines for now and it works without problems.

Thanks and keep up the good work.

[swimmer]

Very well done xentric!

The single backup location solution is much more cleaner and nicer ... the transition to that went flawless - thanks for that!

Greetz
swimmer

PS: Are you playing Q3 from time to time?

[xentric]

PS: Are you playing Q3 from time to time?

No, I finally have a new computer that can handle Q3 but I haven't played any games on it yet
I'm just enjoying the fast compile times at the moment, plus I can finally run vmware at decent speeds!

[swimmer]

Eventually yesterday I found out that the xentric that is playing all the time on the xs4all Q3-server is not dutch but german

I apologize

Greetz
swimmer

[creidiki]

Just remembered to check this thread to see if paludis support was in, tried it, works great, excellent work

Btw, is it possible to kill the warning about the alias?

[ppurka]

Btw, is it possible to kill the warning about the alias?

Yeah. Do what it says! Remove the alias

[creidiki]

O dear, I never even noticed it added that last time I tried cfg-update o.o

[bdm]

Not sure if this is a bug. But I did 10 files and they all seem to be screwed up.

Here's what I now get when I go into the console:

Code: Select all

bash: /etc/bash/bashrc: line 48: syntax error near unexpected token `>>'
bash: /etc/bash/bashrc: line 48: `>>>>>>>>>>>>>>>>>>>> File 1'
dircolors: `/etc/DIR_COLORS':90: unrecognized keyword >>>>>>>>>>>>>>>>>>>>
dircolors: `/etc/DIR_COLORS':91: unrecognized keyword >>>>>>>>>>>>>>>>>>>>
dircolors: /etc/DIR_COLORS:94: invalid line;  missing second token
dircolors: `/etc/DIR_COLORS':98: unrecognized keyword >>>>>>>>>>>>>>>>>>>>
dircolors: `/etc/DIR_COLORS':100: unrecognized keyword >>>>>>>>>>>>>>>>>>>>
dircolors: /etc/DIR_COLORS:103: invalid line;  missing second token
dircolors: `/etc/DIR_COLORS':119: unrecognized keyword >>>>>>>>>>>>>>>>>>>>
dircolors: `/etc/DIR_COLORS':126: unrecognized keyword >>>>>>>>>>>>>>>>>>>>
dircolors: /etc/DIR_COLORS:133: invalid line;  missing second token
dircolors: `/etc/DIR_COLORS':151: unrecognized keyword >>>>>>>>>>>>>>>>>>>>
dircolors: `/etc/DIR_COLORS':153: unrecognized keyword >>>>>>>>>>>>>>>>>>>>
dircolors: /etc/DIR_COLORS:154: invalid line;  missing second token
dircolors: `/etc/DIR_COLORS':158: unrecognized keyword >>>>>>>>>>>>>>>>>>>>
dircolors: `/etc/DIR_COLORS':167: unrecognized keyword >>>>>>>>>>>>>>>>>>>>
dircolors: /etc/DIR_COLORS:176: invalid line;  missing second token
dircolors: `/etc/DIR_COLORS':178: unrecognized keyword >>>>>>>>>>>>>>>>>>>>
dircolors: `/etc/DIR_COLORS':184: unrecognized keyword >>>>>>>>>>>>>>>>>>>>
dircolors: /etc/DIR_COLORS:190: invalid line;  missing second token
dircolors: `/etc/DIR_COLORS':197: unrecognized keyword >>>>>>>>>>>>>>>>>>>>
dircolors: `/etc/DIR_COLORS':198: unrecognized keyword >>>>>>>>>>>>>>>>>>>>
dircolors: /etc/DIR_COLORS:200: invalid line;  missing second token
dircolors: `/etc/DIR_COLORS':213: unrecognized keyword >>>>>>>>>>>>>>>>>>>>
dircolors: `/etc/DIR_COLORS':216: unrecognized keyword >>>>>>>>>>>>>>>>>>>>
dircolors: /etc/DIR_COLORS:224: invalid line;  missing second token
dircolors: `/etc/DIR_COLORS':227: unrecognized keyword >>>>>>>>>>>>>>>>>>>>
dircolors: `/etc/DIR_COLORS':228: unrecognized keyword >>>>>>>>>>>>>>>>>>>>
dircolors: /etc/DIR_COLORS:230: invalid line;  missing second token
dircolors: `/etc/DIR_COLORS':232: unrecognized keyword >>>>>>>>>>>>>>>>>>>>
dircolors: `/etc/DIR_COLORS':240: unrecognized keyword >>>>>>>>>>>>>>>>>>>>
dircolors: `/etc/DIR_COLORS':241: unrecognized keyword >>>>>>>>>>>>>>>>>>>>
bash: /etc/bash/bashrc: line 48: syntax error near unexpected token `>>'
bash: /etc/bash/bashrc: line 48: `>>>>>>>>>>>>>>>>>>>> File 1'

And each file I updated has a buncxh of File1 File2 >>>>> blah blah.

Any ideas?

[xentric]

Any ideas?

Can you post the complete and unedited contents of /etc/bash/bashrc?

[XenoTerraCide]

some thoughts... does the latest cfg-update scan just the config files for the package it's currenly on? if not would that be plausible? also I was wondering if the scan could be backgrounded... because _usually_ it would finish before any files would install.

[xentric]

some thoughts... does the latest cfg-update scan just the config files for the package it's currenly on? if not would that be plausible?

This question makes me wonder how many seconds the index update takes on your machine?
It is possible, but is it worth the trouble of changing this?

also I was wondering if the scan could be backgrounded... because _usually_ it would finish before any files would install.

I like the idea, but the _usually_ always worries me a bit
If it doesn't finish before the first files are being installed, there's a possibility that the wrong checksums end up in the index. Worst case is that an unmodified file would show up as being modified... so you potentially miss out on automatic updates which is the whole point of using the index. The chances of this not-so-optimal situation increase when updating the index takes a long time.

That's why I want to know how long it takes on your system to update the index...
(You can time it with "time cfg-update --force --index")

[Rick]

I'm getting this right now:

Code: Select all

# time cfg-update --force --index
>>> cfg-update-1.8.2-r1: Creating checksum index...
cfg-update --force --index  1.66s user 1.33s system 15% cpu 19.506 total

Trying it for a second time naturally drops it to 2s.
Either way, I know enough ebuilds that finish well within 20 seconds so I have my doubts it would be safe. The only safe way to do it is threading it with a semaphore or something like that.

[Devport]

What about meta packages which dont install anything but may come with a configuration file - its probably possible that they would finish faster than the indexing.

[Rick]

Even without meta packages, most python packages are just download and unpack, most of the small ones are done within 5-10 seconds here.

[XenoTerraCide]

from running twice in a row

Code: Select all

slave1 ~ # time cfg-update --force --index
>>> cfg-update-1.8.2-r1: Creating checksum index...

real    0m16.891s
user    0m0.730s
sys     0m0.580s
slave1 ~ # time cfg-update --force --index
>>> cfg-update-1.8.2-r1: Creating checksum index...

real    0m1.220s
user    0m0.690s
sys     0m0.420s

I'm just thinking of possible ways it could be improved... maybe it could create a lock file and if it wasn't finished by the time that portage was ready to write files portage would have to wait. btw, I've not seen so many meta packages that write config file. they are meta because all they do is pull in other packages.

[xentric]

I'm getting this right now:

Code: Select all

# time cfg-update --force --index
>>> cfg-update-1.8.2-r1: Creating checksum index...
cfg-update --force --index  1.66s user 1.33s system 15% cpu 19.506 total

Trying it for a second time naturally drops it to 2s.
Either way, I know enough ebuilds that finish well within 20 seconds so I have my doubts it would be safe. The only safe way to do it is threading it with a semaphore or something like that.

I get similar results, about 20 secs on a Pentium3 850Mhz.

I have changed the way the index is built because of 2 things:
1. There's a fixed limit on the number of files that can be handled in tools like grep (got bug report) so I had to change it and now it uses echo/xargs for creating the index.
2. cfg-update now supports updating of remote machines from a single location. The index used to be about 17Mb (on my machine), as it was a full dump of all checksums. The new version only stores the checksums of files in the protected directories, which is slower, but it results in a much smaller index of about 120Kb. The smaller size is needed for accessing the index file over a network connection...

The 20 secs indexing per package drops to about 1 second as soon as ._cfg0000_ files are found because that will make cfg-update skip the index update. So is the 20 secs really a problem?

However, I could add a variable that controls the indexing method:
FAST_INDEXING = yes # (default) indexing will take about 6 secs but this index isn't usable for remote updating.
FAST_INDEXING = no # indexing will take about 20 secs and index can be used by a remote cfg-update session.

Until the next version is released, you can speed up the index with this change to /usr/bin/cfg-update
comment out line 730-733:

Code: Select all

        for (my $i = 0; $i < @dir; ++$i) {
# The following command needs to use echo and xargs because cat and grep both have limitations on the number of...
            `echo $pkg_db/*/*/CONTENTS $debug | xargs grep "^obj $dir[$i]" | cut -d" " -f2-3 $debug >> $index_file`;
        }

and add an extra line under it, so it looks like this:

Code: Select all

#        for (my $i = 0; $i < @dir; ++$i) {
# The following command needs to use echo and xargs because cat and grep both have limitations on the number of...
#            `echo $pkg_db/*/*/CONTENTS $debug | xargs grep "^obj $dir[$i]" | cut -d" " -f2-3 $debug >> $index_file`;
#        }
        `echo $pkg_db/*/*/CONTENTS $debug | xargs grep "^obj " | cut -d" " -f2-3 $debug >> $index_file`;

[iBormuth]

Hmm, vimdiff can show up to 4 files next to each other.
What do you think of the following?

Code: Select all

--- /usr/bin/cfg-update	2007-07-10 17:58:07.000000000 +0200
+++ cfg-update	2007-07-10 17:56:56.000000000 +0200
@@ -793,7 +793,7 @@
         if ($opt_d >= 1) { print "$tab"."stage4 disabled...\n"; }
     }
 # Check if tool can handle manual 3-way merges...
-    if ($merge_tool =~ /\/xxdiff$|^xxdiff$|\/kdiff3$|^kdiff3$|\/meld$|^meld$|\/tkdiff$|^tkdiff$/) {
+    if ($merge_tool =~ /\/xxdiff$|^xxdiff$|\/kdiff3$|^kdiff3$|\/meld$|^meld$|\/tkdiff$|^tkdiff$|\/vimdiff$|^vimdiff$|\/gvimdiff$|^gvimdiff$/) {
         $tool_supports_3way = "yes";
         if ($opt_d >= 1) { print "$tab"."tool_supports_3way                = $tool_supports_3way\n"; }
     } else {
@@ -835,8 +835,10 @@
     if (($_[1] =~ /\/meld$|^meld$/         ) && ($threeway_update =~ "no" )) { $cmd = "$_[1] $file1 $file2 $debug"; }
     if (($_[1] =~ /\/tkdiff$|^tkdiff$/     ) && ($threeway_update =~ "yes")) { $cmd = "$_[1] $file1 $file2 -a $file4 -o $file5 $debug"; }
     if (($_[1] =~ /\/tkdiff$|^tkdiff$/     ) && ($threeway_update =~ "no" )) { $cmd = "$_[1] $file1 $file2 -o $file5 $debug"; }
-    if  ($_[1] =~ /\/vimdiff$|^vimdiff$/   )                                 { $cmd = "$_[1] -c 'saveas $file1' -c next -c 'setlocal nomodifiable readonly' -c prev $file1 $file2 --nofork $debug"; }
-    if  ($_[1] =~ /\/gvimdiff$|^gvimdiff$/ )                                 { $cmd = "$_[1] -c 'saveas $file1' -c next -c 'setlocal nomodifiable readonly' -c prev $file1 $file2 --nofork 2>\&1>/dev/null"; }
+    if (($_[1] =~ /\/vimdiff$|^vimdiff$/   ) && ($threeway_update =~ "yes")) { $cmd = "$_[1] $file1 $file2 $file4 +2b '+setlocal ma ro' +3b '+setlocal ma ro' +1b $debug"; }
+    if (($_[1] =~ /\/vimdiff$|^vimdiff$/   ) && ($threeway_update =~ "no" )) { $cmd = "$_[1] $file1 $file2 +2b '+setlocal ma ro' +1b $debug"; }
+    if (($_[1] =~ /\/gvimdiff$|^gvimdiff$/ ) && ($threeway_update =~ "yes")) { $cmd = "$_[1] $file1 $file2 $file4 +2b '+setlocal ma ro' +3b '+setlocal ma ro' +1b --nofork $debug"; }
+    if (($_[1] =~ /\/gvimdiff$|^gvimdiff$/ ) && ($threeway_update =~ "yes")) { $cmd = "$_[1] $file1 $file2 +2b '+setlocal ma ro' +1b --nofork $debug"; }
     if  ($_[1] =~ /\/gtkdiff$|^gtkdiff$/   )                                 { $cmd = "$_[1] -o $file5 $file1 $file2 $debug"; }
     if  ($_[1] =~ /\/imediff2$|^imediff2$/ )                                 { $cmd = "$_[1] -c -o $file5 $file1 $file2"; }
     if  ($_[1] =~ /\/sdiff$|^sdiff$/       )     

You merge to the first column.
Just type ':diffg 2' or ':diffg 3' to get the corresponding block from another column.
Type :wqa to quit (column 2+3 are readonly).

Did I miss anything?

[bell]

Hallo,
cfg-update rocks!
I have a future-Request for this great tool.
cfg-update should respect env-variables ROOT and PORTAGE_CONFIGROOT
Thats needed for update config files on mounted gentoo disks without chroot or on gentoo-crosscompiled targets.
Current i dont know any way for updating config files on my crossdev/emerge-crosscompiled ARM target.

man emerge
..
ENVIRONMENT OPTIONS
ROOT = [path]
Use ROOT to specify the target root filesystem to be used for merging pack-
ages or ebuilds. This variable can be set in make.conf(5) when PORTAGE_CON-
FIGROOT has a value other than /.
Defaults to /.

PORTAGE_CONFIGROOT = [path]
Use PORTAGE_CONFIGROOT to specify the location for various portage configu-
ration files (see FILES for a detailed list of configuration files). This
variable can be set via the --config-root option.
Defaults to /.

[xentric]

Current i dont know any way for updating config files on my crossdev/emerge-crosscompiled ARM target.

cfg-update supports updating multiple (remote)hosts from a single location. As long as the complete system of the host is available via a mountpoint, the backup directory is in the same location behind the mountpoint and it has an indexfile in the same location behind the mountpoint, you should be OK.

So if the BACKUP_PATH on your localhost is /var/lib/cfg-update/backups, just create the directory /your_mount_point/var/lib/cfg-update/backups.
Now comes the -I'm not sure this will work- part, cfg-update also expects an indexfile on the remote host, but because you probably update that host with Portage from your localhost you can try copying or soft-linking the /var/lib/cfg-update/checksum.index file from your localhost to /your_mount_point/var/lib/cfg-update/checksum.index

I think that if you then add your mountpoint (just leave mount_cmd and unmount_cmd empty) to the bottom of /etc/cfg-update.hosts it could work...

cfg-update --check = check if all hosts (mountpoints) are ready for updating
cfg-update -l = list the config updates on the localhost
cfg-update -u = update the config updates on the localhost
cfg-update -lh 1 = list the config updates on the first hosts
cfg-update -uh 1 = update the config updates on the first hosts
cfg-update -lh = list the config updates on all hosts
cfg-update -uh = update the config updates on all hosts

(I'm not sure if the indexfile on your localhost also contains MD5sums for the files on your mounted ARM system... If it doesn't contain the checksums from your ARM machine cfg-update will mark all updates as modified files, thus forcing you to update manually. But as soon as you update a file for a second time cfg-update will at least try the automatic 3-way merge with the backup of the first previous update. So eventually cfg-update will update more files automatically on your ARM machine even though it cannot use the checksum-index!)

Please let me know if this works... or if you have any questions!

[bell]

Thanks, it works "as mounted host" with cfg-update -[lu]h 1.
It shows the "Error: host not mounted", but i can update the config files.

But for uniformity with portage is nice to support ROOT (=mountpoint) and PORTAGE_CONFIGROOT = (Path to cfg-update config files without /etc/* at end).
PORTAGE_CONFIGROOT is from environment
ROOT is from environment or if not set from fgrep ROOT= $PORTAGE_CONFIGROOT/etc/make.conf

[XenoTerraCide]

FEATURE REQUEST.

when cfg-update detects

Code: Select all

* YOU ARE ABOUT TO UPDATE A MODIFIED FILE WHICH PROBABLY CONTAINS
* CUSTOM SETTINGS. YOU ARE FORCED TO UPDATE MANUALLY!
  Press [y] - to merge the current file and the ._cfg0000_* file with vimdiff
  Press [s] - to skip this update (to investigate first, and try again later)
  Press [q] - to quit cfg-update immediately

add these to the initial list. I don't need to check my files when I've just done a re-merge. (e.g. the program should treat me like I know what I'm doing).

Code: Select all

  Press [1] - to replace the current file with the ._cfg0000_* file
  Press [2] - to keep the current file and remove the ._cfg0000_* file