| View previous topic :: View next topic |
| Author |
Message |
dleverton
Guru
Joined: 28 Aug 2006
Posts: 517
|
 Posted: Wed Jan 31, 2007 9:26 pm Post subject: [PALUDIS] collision-protect and undo-prelink hooks |
 |
|
I have written two hooks for the Paludis package manager. One implements the collision-protect feature of Portage, which prevents different packages from trying to overwrite each other's files. The other prevents prelinked binaries from being left behind when unmerging the package (normally, they would be left because they have been modified since they were installed).
WARNING: these scripts come with NO WARRANTY, use them at your own risk. They are NOT SUPPORTED by the Paludis developers. Feel free to ask me if you have problems with them, but I can't promise to be able to solve them. For Paludis questions not related to these hooks, see the Paludis support thread.
To use the hooks, make sure you are using Paludis 0.26.0_alpha13 or later, and save them with a .hook extension into one of the following directories, and make sure they are executable:
- /etc/paludis/hooks/auto to install the hook for the default configuration
- /etc/paludis-suffix/hooks/auto to install the hook for the configuration used by --config-suffix suffix
- ~/.paludis/hooks/auto to install the hook for the default configuration for your user account
- ~/.paludis-suffix/hooks/auto to install the hook for the configuration used by --config-suffix suffix for your user account
- /usr/share/paludis/hooks/auto to install the hooks for all configurations and users
Most people should probably use either /etc/paludis/hooks/auto or /usr/share/paludis/hooks/auto.
These hooks are also available as part of the paludis-extras overlay, if you would prefer to install them that way rather than manually.
WARNING: it seems that some browsers insert extra spaces at the ends of lines when copying the code, and this will break the scripts. Please make sure that none of the lines in the files end with spaces when you save them.
collision-protect.hook
You can have some extra control over the protection by setting the COLLISION_IGNORE variable in /etc/paludis/bashrc. It should be a space-seperated list of files or directories (including wildcards) for which collisions will be ignored (you can disable it completely by including / in the list). As with anything in bashrc, you can make it conditional on package name, version, phase of the moon, or whatever else you can express in bash code.
You may have trouble with collision-protect if you are using Paludis 0.26 alpha 2 or earlier, and a packages tries to install files with spaces in their names, because Paludis doesn't like those much (see docs). You can work around it by including a wildcard pattern for the troublesome files in COLLISION_IGNORE, for example | Code: | | COLLISION_IGNORE="/usr/share/projectM/presets/*" |
ChangeLog
2006-12-15: Initial release.
2006-12-15: Be more robust against stupid file names.
2006-12-17: Big rework, should be faster (and not make bash segfault  ) on large packages.
2006-12-20: Don't spew useless error messages on nvidia-drivers (and possibly other OpenGLish things).
2006-12-28: Work round Portage not recording /usr/share/{gcc,binutils}-data/*/*/info/dir in the VBD for some unknown reason.
2006-12-31: Make previous fix less sloppy.
2006-12-31: Seems previous fix isn't necessary for binutils.
2007-01-04: Remove redundant --no-colour.
2007-01-15: Clean up a couple of comments (no code changes).
2007-01-23: Implement COLLISION_IGNORE (see above).
2007-01-31: Clean up the code a bit, and allow COLLISION_IGNORE to contain files as well as directories.
2007-02-01: Fix braino.
2007-02-13: Fix glitch in the (disabled by default) timing code.
2007-02-23: Port to Paludis 0.20+. Installation instructions have changed, see above.
2007-03-02: Work around locale-related retardedness in the sort command.
2007-07-22: Suppress some irrelevant warnings from trunk Paludis.
2007-08-31: Make config protect handling PMS-compliant, handle wildcards in COLLISION_IGNORE better.
2007-10-10: Work properly when files are missing from the installed version of the package.
2008-03-30: Update to new style of installation. Instructions have changed, see above.
| Code: | #! /bin/bash
source "${PALUDIS_EBUILD_DIR}/echo_functions.bash"
hook_run_merger_check_post() {
load_bashrc() {
local CONFIG_PROTECT CONFIG_PROTECT_MASK
for f in ${PALUDIS_BASHRC_FILES}; do
[[ -f "${f}" ]] && source "${f}"
done
}
load_bashrc
if [[ " ${COLLISION_IGNORE} " == *" / "* ]]; then
einfo_unhooked "\${COLLISION_IGNORE} contains /, skipping collision check"
return 0
fi
einfo_unhooked "Checking for collisions..."
# Uncomment this and the corresponding bit at the end if you
# want to know how long it takes.
## "times" doesn't return real time, and "time" is too awkward to
## use in this context.
#local start=( $( date '+%s %N' ) )
# Some packages try to update the Info directory, but Paludis
# manages that so it doesn't really matter. For some reason, the
# GCC Info directory gets installed with Paludis but not Portage,
# so handle that too.
local prunes=( -wholename ./usr/share/info/dir -prune -or
-wholename ./usr/share/info/dir.gz -prune -or
-wholename ./usr/share/info/dir.bz2 -prune -or
-wholename ./usr/share/gcc-data/*/info/dir -prune -or
-wholename ./usr/share/gcc-data/*/info/dir.gz -prune -or
-wholename ./usr/share/gcc-data/*/info/dir.bz2 -prune -or )
# Don't expand wildcards in ${COLLISION_IGNORE} etc
local oldset="${-}"
set -f
local ignore
for ignore in ${COLLISION_IGNORE}; do
# No need for a slash/wildcard here: if it's a directory, find
# will prune it and all its contents, if not it'll just prune
# that file.
prunes+=( -wholename ".${ignore%/}" -prune -or )
done
local mask
for mask in ${CONFIG_PROTECT_MASK}; do
prunes+=( -wholename ".${mask}" ! -type d -print -or
-wholename ".${mask%/}/*" ! -type d -print -or )
done
local protect
for protect in ${CONFIG_PROTECT}; do
prunes+=( -wholename ".${protect}" ! -type d -prune -or
-wholename ".${protect%/}/*" ! -type d -prune -or )
done
[[ ${oldset} == *f* ]] || set +f
# Use ls -d 2>/dev/null to find files that already exist. The -d'\n'
# is needed to stop xargs from interpreting special characters. Use
# cd to handle ${ROOT}, rather than adding it with sed, to avoid the
# chance of evil characters messing things up.
find_existing() {
sed -e 's,^,.,' |
( cd "${ROOT}" && xargs -r -d'\n' ls -d 2>/dev/null ) |
sed -e 's,^\.,,'
}
# Need silent to suppress warnings about the temporary directory in the VDB.
local installed="$( ${PALUDIS_COMMAND} --log-level silent \
--best-version "${CATEGORY}/${PN}:${SLOT}" )"
cd "${IMAGE}"
# Not using bash arrays here because it makes it harder to pass to
# external commands.
if [[ -n "${installed}" ]]; then
local inst_files="$( ${PALUDIS_COMMAND} --log-level silent \
--contents "=${installed}" |
sed -ne 's/ -> .*//;s/^ \+//p' |
sort | find_existing )"
local new_files="$( find . "${prunes[@]}" ! -type d -print |
sed -e 's,^\.,,' | sort )"
local collisions="$( comm -2 -3 <( echo "${new_files}" ) \
<( echo "${inst_files}" ) |
find_existing )"
else
local collisions="$( find . "${prunes[@]}" ! -type d -print |
sed -e 's,^\.,,' | find_existing )"
fi
# Handle the case that a package installs files to the same directory
# as the previous version but under a different name, via symlinking.
# For example, a package that's made multilib-aware (lib -> lib64) or
# an X program that was moved from /usr/X11R6 to /usr.
if [[ -n "${collisions}" && -n "${installed}" ]]; then
# Two files are "the same" for our purposes if they have the same
# name and their directories (after resolving symlinks) have the
# same device and inode numbers.
dev_inode_names() {
# first sed: Find the directory names.
# xargs stat: Find the dev/inode numbers of said directories.
# second sed: Find the filenames.
# ${2} "${1}": Optionally add the full path/filename to the end.
paste <( sed -e 's,^\(.*\)/[^/]*$,.\1,' <<<"$1" |
( cd "${ROOT}" && xargs -r -d'\n' \
stat -L -c '%D %i' 2>/dev/null ) ) \
<( sed -e 's,^.*/,,' <<<"$1" ) \
<( ${2} "${1}" )
}
# Find the info of all the potential collisions. Sort it up to
# but not including the first slash (i.e. up to the start of the
# full pathname).
local dins="$( dev_inode_names "${collisions}" echo |
LC_ALL=C sort -t/ -k1 )"
# comm: Chop off the pathname part and compare with the pathless
# info of the files from the previously installed version.
# paste: Stick the pathnames back on.
# sed: Find those files that were /not/ in the installed version
# (aka the lines from comm that do not start with tabs) and
# chop off the dev/inode information, leaving only the
# pathnames.
collisions="$( paste <( LC_ALL=C comm -2 <( cut -d/ -f1 <<<"${dins}" ) \
<( dev_inode_names \
"${inst_files}" : |
LC_ALL=C sort ) ) \
<( echo "${dins}" ) |
sed -ne 's,^[^\t][^/]*\(/.*\)$,\1,p' )"
fi
if [[ -n "${collisions}" ]]; then
local okay=false file
while read file; do
eerror "${file} already exists in ${ROOT}"
done <<<"${collisions}"
else
local okay=true
fi
# Uncomment this and the corresponding bit at the beginning if you
# want to know how long it takes.
#local end=( $( date '+%s %N' ) )
#local elapsed=$(( (${end[0]} - ${start[0]}) * 1000
# + (10#${end[1]} / 1000000)
# - (10#${start[1]} / 1000000) ))
#einfo_unhooked "Elapsed time: $( printf %d.%03d \
# $(( ${elapsed} / 1000 )) \
# $(( ${elapsed} % 1000 )) ) seconds"
if ${okay}; then
einfo_unhooked "Hooray! No collisions"
return 0
else
eerror "Package collides with existing files"
return 1
fi
}
hook_auto_names() {
echo merger_check_post
}
|
undo-prelink.hook
To install the hook, save it as $PALUDIS_DIR/hooks/common/undo-prelink.bash, then symlink it to $PALUDIS_DIR/hooks/install_post/undo-prelink.bash, $PALUDIS_DIR/hooks/install_fail/undo-prelink.bash, $PALUDIS_DIR/hooks/merger_check_pre/undo-prelink.bash and $PALUDIS_DIR/hooks/unmerger_unlink_pre/undo-prelink.bash.
ChangeLog
2007-01-04: Initial release.
2007-01-04: Replace call to cut with pattern match.
2007-01-05: Support ${ROOT} (d'oh).
2007-01-14: Print a message at the start, so it doesn't look like it's died with large packages.
2007-01-15: Big speed-up on large packages.
2007-01-16: Fix i18n bug (and prune a bit of redundant code).
2007-01-23: Make it work when reinstalling the same version that's already installed. (Usage instructions have changed, see above.)
2007-02-05: Work properly on packages that contain files with no newlines.
2007-02-12: Be less case-sensitive. Also simplify the code a bit.
2007-02-13: Simplify the code a lot, maybe even a small speedup. Also fixed a glitch in the (disabled by default) timing code.
2007-02-24: Port to Paludis 0.20+. Installation instructions have changed, see above.
2007-07-22: Suppress some irrelevant warnings from trunk Paludis.
2008-03-30: Update to new style of installation. Instructions have changed, see above.
| Code: | #! /bin/bash
source "${PALUDIS_EBUILD_DIR}/echo_functions.bash"
get_file_list() {
# Need to use silent to suppress warnings about temporary
# directories in the VDB. Don't forget to add . to the front of
# the filenames (see below).
${PALUDIS_COMMAND} --log-level silent \
--contents "=${CATEGORY}/${PF}:${SLOT}" |
sed -ne '/ -> /d;s/^ \+/./p'
}
file_list_file="${PALUDIS_HOOKS_TMPDIR:-${ROOT}/var/tmp/paludis}/undo-prelink.${PALUDIS_PID}"
hook_run_install_post() {
rm -f "${file_list_file}"
}
hook_run_install_fail() {
hook_run_install_post
}
# If we're reinstalling the same version that's installed
# already, we won't be able to use --contents in the unmerge
# hook, so do it now and save to a file.
hook_run_merger_check_post() {
if ${PALUDIS_COMMAND} --log-level silent \
--has-version "=${CATEGORY}/${PF}"; then
get_file_list >"${file_list_file}"
fi
return 0
}
hook_run_unmerger_unlink_pre() {
einfo_unhooked "Checking for prelinked binaries..."
# Uncomment this and the corresponding bit below if you want
# to know how long it takes.
## "times" doesn't return real time, and "time" is too awkward
## to use in this context.
#local start=( $( date '+%s %N' ) )
# cd to ${ROOT} instead of mangling the filenames so as not to
# get caught by evil characters.
cd "${ROOT}"
# xargs needs special options to avoid mishandling special
# characters. It turns out to be much simpler and just as
# fast if not faster to give everything to prelink and let it
# figure out what's a binary, than to filter out beforehand.
# Only caveat is that we need to discard errors.
if [[ -e "${file_list_file}" ]]; then
cat "${file_list_file}"
else
get_file_list
fi | xargs -r -d'\n' prelink -u 2>/dev/null
# Uncomment this and the corresponding bit above if you want
# to know how long it takes.
#end=( $( date '+%s %N' ) )
#elapsed=$(( (${end[0]} - ${start[0]}) * 1000
# + (10#${end[1]} / 1000000) - (10#${start[1]} / 1000000) ))
#einfo_unhooked "Elapsed time: $( printf %d.%03d \
# $(( ${elapsed} / 1000 )) \
# $(( ${elapsed} % 1000 )) ) seconds"
return 0
}
hook_auto_names() {
echo install_post install_fail merger_check_post unmerger_unlink_pre
}
|
Last edited by dleverton on Sun Mar 30, 2008 6:51 pm; edited 18 times in total |
|
| Back to top |
|
 |
dleverton
Guru
Joined: 28 Aug 2006
Posts: 517
|
| Posted: Mon Feb 05, 2007 6:18 pm Post subject: |
|
|
| Updated undo-prelink to fix a bug with files that don't contain newlines. |
|
| Back to top |
|
|
zxy
Veteran
Joined: 06 Jan 2006
Posts: 1160
Location: in bed in front of the computer
|
| Posted: Mon Feb 05, 2007 9:33 pm Post subject: |
|
|
New update added to paludis-extras.... (undo-prelink ver 1.2)
_________________
Nature does not hurry, yet everything is accomplished.
Lao Tzu |
|
| Back to top |
|
|
dleverton
Guru
Joined: 28 Aug 2006
Posts: 517
|
| Posted: Mon Feb 12, 2007 6:07 pm Post subject: |
|
|
| Updated undo-prelink to fix a case-sensitivity issue. Thanks to zxy and his mysterious informant for reporting. |
|
| Back to top |
|
|
pdouble
Tux's lil' helper
Joined: 24 Aug 2002
Posts: 89
Location: USA
|
| Posted: Mon Feb 12, 2007 6:53 pm Post subject: |
|
|
| dleverton wrote: | | Updated undo-prelink to fix a case-sensitivity issue. Thanks to zxy and his mysterious informant for reporting. |
You asked (via zxy):
| dleverton wrote: |
That's odd... do you know what version of coreutils was installed? Or maybe it
was a BSD system? (Does prelink even work on BSD? If so, good to know the
hook works with only minor changes.) Anyway, I'll post a fixed version
shortly.
|
Not BSD. coreutils 6.7-r1. |
|
| Back to top |
|
|
dleverton
Guru
Joined: 28 Aug 2006
Posts: 517
|
| Posted: Mon Feb 12, 2007 8:22 pm Post subject: |
|
|
| pdouble wrote: | | Not BSD. coreutils 6.7-r1. |
Hmm, I don't see anything in the code that would cause that, but I'm not brave enough to actually install it before it goes stable. Is the stat command in your $PATH the actual stat from coreutils, or somewhere else?
In any case, though, I'll leave the fix in until/unless I find a more appropriate way to solve the problem. |
|
| Back to top |
|
|
pdouble
Tux's lil' helper
Joined: 24 Aug 2002
Posts: 89
Location: USA
|
| Posted: Mon Feb 12, 2007 9:00 pm Post subject: |
|
|
| dleverton wrote: | | pdouble wrote: | | Not BSD. coreutils 6.7-r1. |
Hmm, I don't see anything in the code that would cause that, but I'm not brave enough to actually install it before it goes stable. Is the stat command in your $PATH the actual stat from coreutils, or somewhere else?
In any case, though, I'll leave the fix in until/unless I find a more appropriate way to solve the problem. |
Interestingly enough, /usr/bin/stat is from:
| Code: |
* sys-apps/stat
gentoo: 2.5 3.3 {:0}
installed: 3.3* {:0}
Homepage: http://www.gnu.org/directory/stat.html
Description: A command-line stat() wrapper
License: ( GPL-2 )
Source origin: sys-apps/stat-3.3::gentoo
Installed time: Wed Feb 7 22:14:03 2007
|
Not sure why this package exists, or if I should use it. There doesn't seem to be any packages that depend on it. I replaced with coreutils-6.7-r1 and the type is now "regular file", all lowercase. Not very comforting, it's possible coreutils may end up using the code from stat-3.3 in the future. |
|
| Back to top |
|
|
dleverton
Guru
Joined: 28 Aug 2006
Posts: 517
|
| Posted: Mon Feb 12, 2007 10:08 pm Post subject: |
|
|
| pdouble wrote: | Interestingly enough, /usr/bin/stat is from:
| Code: |
* sys-apps/stat
gentoo: 2.5 3.3 {:0}
installed: 3.3* {:0}
Homepage: http://www.gnu.org/directory/stat.html
Description: A command-line stat() wrapper
License: ( GPL-2 )
Source origin: sys-apps/stat-3.3::gentoo
Installed time: Wed Feb 7 22:14:03 2007
|
|
Ah, that explains it. In that case I'll leave it as it is, recognising both case variations. (Although, I'm starting to think I shouldn't use stat at all... but I can't think of anything better right now.) |
|
| Back to top |
|
|
pdouble
Tux's lil' helper
Joined: 24 Aug 2002
Posts: 89
Location: USA
|
| Posted: Mon Feb 12, 2007 10:17 pm Post subject: |
|
|
| dleverton wrote: |
Ah, that explains it. In that case I'll leave it as it is, recognising both case variations. (Although, I'm starting to think I shouldn't use stat at all... but I can't think of anything better right now.) |
I think the line using stat is inst_files to determine which files to check. Would this work?
| Code: |
inst_files="$( paludis -k coreutils | while read F; do [ -f "$F" ] && echo $F; done )"
|
|
|
| Back to top |
|
|
pdouble
Tux's lil' helper
Joined: 24 Aug 2002
Posts: 89
Location: USA
|
| Posted: Mon Feb 12, 2007 10:26 pm Post subject: |
|
|
After I think about it some more I think you can simply the script down to:
| Code: |
#! /bin/bash
[[ "${HOOK}" == ebuild_merge_pre ]] &&
! ${PALUDIS_COMMAND} --has-version "=${CATEGORY}/${PF}" && return
einfo_unhooked "Checking for prelinked binaries..."
(
cd "${ROOT}"
${PALUDIS_COMMAND} --contents "=${CATEGORY}/${PF}" |
while read file; do
[ -f "$F" -a -x "$F" ] && prelink -u "${file}" 2>/dev/null;
done
)
# Need this as last command may return error, and this is OK, we don't want paludis complaining about the hook
exit 0
|
prelink detects ELF binaries and the prelink state itself, so I don't think there is a need to use sed / objdump / grep to do it. Although there may be some bugs in prelink that you are trying to avoid. |
|
| Back to top |
|
|
dleverton
Guru
Joined: 28 Aug 2006
Posts: 517
|
| Posted: Mon Feb 12, 2007 11:16 pm Post subject: |
|
|
| pdouble wrote: | After I think about it some more I think you can simply the script down to:
... |
I did originally do something like that, but bash loops are rather slow, and pipelines are much faster, at least for large numbers of files, so I'd like to run the loop on as few files as possible.
As for running prelink on all binaries, instead of using objdump to see whether they're already prelinked, I'll try that and see how it affects the speed. Hmm, in fact it might allow me to eliminate the loop entirely....
Thanks for the suggestions, anyway. |
|
| Back to top |
|
|
dleverton
Guru
Joined: 28 Aug 2006
Posts: 517
|
| Posted: Tue Feb 13, 2007 12:13 am Post subject: |
|
|
| dleverton wrote: | | As for running prelink on all binaries, instead of using objdump to see whether they're already prelinked, I'll try that and see how it affects the speed. Hmm, in fact it might allow me to eliminate the loop entirely.... |
Wow, that turned out to be a lot simpler, and even slightly faster in some cases. I put up a new version.
pdouble: thanks again for the advice.
zxy: I also fixed a small glitch in collision-protect, in the timing code, but since that's commented by default I don't think it's worth doing a new version in the overlay. Up to you, though. |
|
| Back to top |
|
|
dleverton
Guru
Joined: 28 Aug 2006
Posts: 517
|
| Posted: Fri Feb 23, 2007 9:21 pm Post subject: |
|
|
| Ported collision-protect to Paludis 0.19+ (current -scm, and the upcoming 0.20). undo-prelink will follow soon. |
|
| Back to top |
|
|
dleverton
Guru
Joined: 28 Aug 2006
Posts: 517
|
| Posted: Sat Feb 24, 2007 1:48 pm Post subject: |
|
|
| dleverton wrote: | | undo-prelink will follow soon. |
Or maybe not, sorry. Seems this is rather difficult with the new merger interface.... |
|
| Back to top |
|
|
dleverton
Guru
Joined: 28 Aug 2006
Posts: 517
|
| Posted: Sat Feb 24, 2007 6:10 pm Post subject: |
|
|
| dleverton wrote: | | dleverton wrote: | | undo-prelink will follow soon. |
Or maybe not, sorry. Seems this is rather difficult with the new merger interface.... |
Difficult, but not impossible.  Posted. |
|
| Back to top |
|
|
dleverton
Guru
Joined: 28 Aug 2006
Posts: 517
|
| Posted: Fri Mar 02, 2007 3:58 pm Post subject: |
|
|
New collision-protect because sort is retarded.
I also removed the Paludis 0.18 versions from the post, although they're still in the overlay for now. Let me know if you really need them, but I suggest you upgrade instead. |
|
| Back to top |
|
|
dleverton
Guru
Joined: 28 Aug 2006
Posts: 517
|
| Posted: Sun Jul 22, 2007 12:03 pm Post subject: |
|
|
| Added new versions of both that disable some irrelevant warnings from trunk Paludis. |
|
| Back to top |
|
|
dleverton
Guru
Joined: 28 Aug 2006
Posts: 517
|
| Posted: Fri Aug 31, 2007 2:56 pm Post subject: |
|
|
| New collision-protect, improved CONFIG_PROTECT and COLLISION_IGNORE handling. |
|
| Back to top |
|
|
dleverton
Guru
Joined: 28 Aug 2006
Posts: 517
|
| Posted: Wed Oct 10, 2007 3:37 pm Post subject: |
|
|
| Fixed a bug in collision-protect when some files are missing from the installed package. |
|
| Back to top |
|
|
dleverton
Guru
Joined: 28 Aug 2006
Posts: 517
|
| Posted: Sat Oct 27, 2007 6:32 pm Post subject: |
|
|
| Added a note to the initial post about collision-protect and files with spaces in the names (no code changes). |
|
| Back to top |
|
|
dleverton
Guru
Joined: 28 Aug 2006
Posts: 517
|
| Posted: Sun Mar 30, 2008 6:50 pm Post subject: |
|
|
| Updated both hooks to new installation method, please see updated installation instructions and make sure you are using paludis 0.26.0_alpha13 or later. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Unsupported Software
