| View previous topic :: View next topic |
| Author |
Message |
Atom2
Apprentice
Joined: 01 Aug 2011
Posts: 185
|
 Posted: Sat Jul 19, 2014 11:05 am Post subject: epatch_user and /etc/portage/patches |
 |
|
Hi guys,
I wonder whether anybody of those in the know might be able to clear my confusion: According to the documentation here | Quote: | | The epatch_user function applies source code patches that are found in /etc/portage/patches/<category>/<package>[-<version>[-<revision>]], whatever directory is found first. |
In the quote I have marked in bold the part where I am getting confused. What does this really mean - i.e. which directory is actually found first?
Let me explain what I want to achieve: I do have one patch to my hardened-sources which has to do with my XEN dom0 not shutting down but rather rebooting when I issueSo this patch I need to carry with every new version of my kernel sources and I have therefore created a directory | Code: | | /etc/portage/sys-kernel/hardened-sources |
which includes that specific patch file named "shutdown.patch". It is automatically applied whenever I emerge new kernel sources.
From time to time there's an additional patch which fixes certain issues for a specific kernel version only. Lately for instance there was a fix to the netback.c XEN driver which solved an issue of running out of buffers. This found its way into 3.16, but is missing in the 3.15 kernel sources - although a git commit against the 3.15 kernel does exist.
So my idea was to create a new directory | Code: | | /etc/portage/patches/hardened-sources-3.15 |
and include that git patch into this directory in the hope that both patches - the generic for *all* hardened-sources under /etc/portage/patches/sys-kernel/hardened-sources and the one under /etc/portage/patches/sys-kernel/hardened-sources-3.15 - would automatically be applied. To my disappointment this approach did not work out. I was evern more surprised to find out, that only the generic patch under /etc/portage/patches/sys-kernel/hardened-sources was applied, but the more specific one under /etc/portage/patches/hardened-sources-3.15 was ignored, although the latter directory clearly was more specific to the sources I was emerging.
Refering back to the documentation this cryptic "whatever directory is found first" came up which I do not really understand. I simply know nothing about ebuild_user's search strategy and therefore can't predictably forsee which directory is found first and therefore which patches are applied. So I'd be happy if somebody could shed some light on this.
Having said that, the ideal solution would really be if epatch_user automatically applied all relevant patches to sources. That is: Those for the generic package, then those for the specific package-version and lastly those for the specific package-version-revision. This would really allow for the greatest degree of flexibility by moving from the general patch to the most specific patch for a package. I don't know whether this is possible at all using the standard ebuilds or whether that would require more effort over and above what the ebuild_user function supports.
Any input/thoughts very much appreciated.
Thanks and regards Atom2 |
|
| Back to top |
|
 |
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Sat Jul 19, 2014 1:32 pm Post subject: Re: epatch_user and /etc/portage/patches |
|
|
| Atom2 wrote: | | Refering back to the documentation this cryptic "whatever directory is found first" came up which I do not really understand. I simply know nothing about ebuild_user's search strategy and therefore can't predictably forsee which directory is found first and therefore which patches are applied. So I'd be happy if somebody could shed some light on this. |
Atom2 ... the shorter 'hardended-sources' matches first ... you can see why with the following:
| Code: | % mkdir -p test/hardened-sources{,-3.15,-999999}
% ls -l test/
total 12
drwx------ 2 khayyam users 4096 2014-07-19 15:23 hardened-sources/
drwx------ 2 khayyam users 4096 2014-07-19 15:23 hardened-sources-3.15/
drwx------ 2 khayyam users 4096 2014-07-19 15:23 hardened-sources-999999/ |
... 'nothing' (for want of a better word) is logically prior to 'something'.
| Atom2 wrote: | | Having said that, the ideal solution would really be if epatch_user automatically applied all relevant patches to sources. That is: Those for the generic package, then those for the specific package-version and lastly those for the specific package-version-revision. This would really allow for the greatest degree of flexibility by moving from the general patch to the most specific patch for a package. I don't know whether this is possible at all using the standard ebuilds or whether that would require more effort over and above what the ebuild_user function supports. |
I don't see a way to do that without changing the eutils.eclass or using symlinks, and the latter wouldn't be very flexible (though its scriptable).
best ... khay |
|
| Back to top |
|
|
Atom2
Apprentice
Joined: 01 Aug 2011
Posts: 185
|
| Posted: Sat Jul 19, 2014 2:37 pm Post subject: Re: epatch_user and /etc/portage/patches |
|
|
khay,
as always, many thanks for your reply. | khayyam wrote: | | Code: | % mkdir -p test/hardened-sources{,-3.15,-999999}
% ls -l test/
total 12
drwx------ 2 khayyam users 4096 2014-07-19 15:23 hardened-sources/
drwx------ 2 khayyam users 4096 2014-07-19 15:23 hardened-sources-3.15/
drwx------ 2 khayyam users 4096 2014-07-19 15:23 hardened-sources-999999/ |
... 'nothing' (for want of a better word) is logically prior to 'something'. |
Your answer is obviously based upon a pre-supposed lexical search- (or probably rather: sort-) algorithm because only using that algorithm 'nothing' (for want of a better word) is lexically (not: logically) sorted (and by no means: found) prior to 'something'. If the sentence in the documentation had been "whatever directory is lexically first" it would have been a no-brainer for me and I'd fully concur with your reply. But just suppose the following: | Code: | # mkdir -p test/hardened-sources{-999999,,-3.15}
# ls -l --sort=none test/
total 0
drwxr-xr-x 1 root root 0 Jul 19 16:17 hardened-sources-999999
drwxr-xr-x 1 root root 0 Jul 19 16:17 hardened-sources
drwxr-xr-x 1 root root 0 Jul 19 16:17 hardened-sources-3.15 |
In this case the output of ls -l - after disabling the default lexical sort algorithm - depends on which directory has been created first. So natuarlly (i.e. without any special post-find treatment) the directory created first is displayed first because it was found first.
I was really getting into thinking by the use of the term 'found first' given that Unix/linux documentation is normally very precise and leaves no (to very little) wiggle room.
Regards Atom2
Last edited by Atom2 on Sat Jul 19, 2014 8:28 pm; edited 1 time in total |
|
| Back to top |
|
|
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21602
|
| Posted: Sat Jul 19, 2014 3:34 pm Post subject: |
|
|
When in doubt, read the source: | /usr/portage/eclass/eutils.eclass: | epatch_user() {
[[ $# -ne 0 ]] && die "epatch_user takes no options"
# Allow multiple calls to this function; ignore all but the first
local applied="${T}/epatch_user.log"
[[ -e ${applied} ]] && return 2
# don't clobber any EPATCH vars that the parent might want
local EPATCH_SOURCE check base=${PORTAGE_CONFIGROOT%/}/etc/portage/patches
for check in ${CATEGORY}/{${P}-${PR},${P},${PN}}{,:${SLOT}}; do
|
The first directory found by $check that exists is used. Expanding those shell expressions shows that more specific matches come first, except in the case of slots, where the slot-specific is found only if no slot-nonspecific is found. |
|
| Back to top |
|
|
toralf
Developer
Joined: 01 Feb 2004
Posts: 3922
Location: Hamburg
|
| Posted: Sat Jul 19, 2014 5:17 pm Post subject: Re: epatch_user and /etc/portage/patches |
|
|
| khayyam wrote: |
Atom2 ... the shorter 'hardended-sources' matches first |
Sure ? I do have /etc/portage/patches/sci-misc/boinc, but sometimes I need a /etc/portage/patches/sci-misc/boinc-7.4.8 or so, which rules over the shorter one ! |
|
| Back to top |
|
|
Atom2
Apprentice
Joined: 01 Aug 2011
Posts: 185
|
| Posted: Sat Jul 19, 2014 8:17 pm Post subject: |
|
|
| Hu wrote: | | When in doubt, read the source |
Hu, thanks for that pointer. I have now read the sources and tried to understand how it works and I think I have figured it out. So I am going to document it here for anybody else's benefit:
The comment in /usr/portage/eclass/eutils.eclass says: | /usr/portage/eclass/eutils.eclass: | # Applies user-provided patches to the source tree. The patches are
# taken from /etc/portage/patches/<CATEGORY>/<P-PR|P|PN>[:SLOT]/, where the first
# of these three directories to exist will be the one to use, ignoring
# any more general directories which might exist as well. They must end
# in ".patch" to be applied. |
According to documentation at the Gentoo Development Guide, Secion Variables those variables refer to the following: | Code: | CATEGORY refers to the package's category
"sys-kernel" in my case
PN refers to the package name
"hardened-sources" in my case
PR to be the package revision
"r1" in my case
P refers to the package name and version w/o revision
"hardened-sources-3.15.5" in my case
SLOT refers to the slot, that is "version-revision" for slotted packages or 0 otherwise (see equery output below)
"3.15.5-r1" in my case |
| Code: | # equery list -p hardened-sources
* Searching for hardened-sources ...
[-P-] [ ] sys-kernel/hardened-sources-3.2.55-r7:3.2.55-r7
[-P-] [ ] sys-kernel/hardened-sources-3.2.59-r5:3.2.59-r5
[-P-] [ ] sys-kernel/hardened-sources-3.2.61-r1:3.2.61-r1
[IP-] [ ] sys-kernel/hardened-sources-3.13.10:3.13.10
[IP-] [ ] sys-kernel/hardened-sources-3.14.5-r2:3.14.5-r2
[-P-] [ ] sys-kernel/hardened-sources-3.14.12-r1:3.14.12-r1
[IP-] [ ] sys-kernel/hardened-sources-3.15.5-r1:3.15.5-r1
#
# equery list -p vim
* Searching for vim ...
[-P-] [ ] app-editors/vim-7.3.762:0
[-P-] [ ~] app-editors/vim-7.3.1214:0
[-P-] [ ~] app-editors/vim-7.4.52:0
[-P-] [ ~] app-editors/vim-7.4.155:0
[-P-] [ ~] app-editors/vim-7.4.169:0
[-P-] [ ~] app-editors/vim-7.4.193:0
[IP-] [ ] app-editors/vim-7.4.273:0
[-P-] [ ~] app-editors/vim-7.4.326:0
[-P-] [ -] app-editors/vim-9999:0 |
Trying further to decipher | /usr/portage/eclass/eutils.eclass: | | for check in ${CATEGORY}/{${P}-${PR},${P},${PN}}{,:${SLOT}}; do |
this seems to expand (for details see under keyword "brace expansion" in man bash) to the following list (for clarity's sake shown as one entry per line): | Code: | sys-kernel/hardened-sources-3.15.5-r1
sys-kernel/hardened-sources-3.15.5-r1:3.15.5-r1
sys-kernel/hardened-sources-3.15.5
sys-kernel/hardened-sources-3.15.5:3.15.5-r1
sys-kernel/hardened-sources
sys-kernel/hardened-sources:3.15.5-r1 |
out of which only the first match - i.e. the first existing directory - will be taken. In my case that was | Code: | | /etc/portage/patches/sys-kernel/hardened-sources |
as I had (ignoring the slotted versions for the moment) not used either of | Code: | /etc/portage/patches/sys-kernel/hardened-sources-3.15.5-r1
or
/etc/portage/patches/sys-kernel/hardened-sources-3.15.5 |
as my patch directory but rather mistakenly | Code: | | /etc/portage/patches/sys-kernel/hardened-sources-3.15 |
(NOTE the missing .5 at the end.
That was due to a misunderstanding of the meaning for version and revision. I was mistakenly under the impression that the version of a kernel package refers to the official version of the the kernel (i.e. the 3.15 kernel) and the remaining part of the version (i.e. .5-r1) was the revision - which obviously is not true.
Having now understood how it works, I still consider it somehow disappointing that patches are not applied incrementally starting from the general package directory through to the most specific patch directory available and applying all patches in sequence.
Regards Atom2
Last edited by Atom2 on Sat Jul 19, 2014 9:13 pm; edited 1 time in total |
|
| Back to top |
|
|
Atom2
Apprentice
Joined: 01 Aug 2011
Posts: 185
|
| Posted: Sat Jul 19, 2014 8:24 pm Post subject: Re: epatch_user and /etc/portage/patches |
|
|
| toralf wrote: | | khayyam wrote: |
Atom2 ... the shorter 'hardended-sources' matches first |
Sure ? I do have /etc/portage/patches/sci-misc/boinc, but sometimes I need a /etc/portage/patches/sci-misc/boinc-7.4.8 or so, which rules over the shorter one ! |
No, the shorter hardened-sources directory does not match first (it only did in my case because I made a mistake - see my previous post above).
In actual fact, as one would reasonably expect, only the most specific (which, however, is not necessarily the longest) directoy is considered - please again see my previous post where I tried to get to the gist of the matter whilst at the same time documenting it for everybody else's (and obviously also my own) benefit in this thread.
Regards Atom2 |
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Sat Jul 19, 2014 9:01 pm Post subject: |
|
|
Atom2, toralf, et al ...
at this moment in time my lawyer is drawing up a lack of coffee defense and I'm not at liberty to discuss the issue :)
That said, yes, I did assume a lexical search, and based on what you had said I'd asumed that as hardened-sources was used in preference to hardened-sources-${PV} that this was what epatch_user was doing. I only considered why hardened-sources-${PV} was not given preference ... and came to the above conclusion. Spank me sideways why didn't I take a look see ... I guess I made the assumption I didn't need to ... but yes "when in doubt", hehe.
best ... khay |
|
| Back to top |
|
|
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21602
|
| Posted: Sat Jul 19, 2014 9:03 pm Post subject: |
|
|
Overriding, rather than extending, is the better behavior. How else do you set up a patch that applies to a specific version, but not to later versions where upstream accepted the patch and you should no longer apply it locally?
If you need to patch multiple versions in a cascade, you could link (whether hard or soft) the common patches into the required directories. For elaborate scenarios, you could use Portage bashrc and a custom function similar in spirit to epatch_user, but with the semantics you need for a given package. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Sun Jul 20, 2014 4:51 am Post subject: |
|
|
| Atom2 wrote: | | for check in ${CATEGORY}/{${P}-${PR},${P},${PN}}{,:${SLOT}} |
It seems to me that this is a bug: The last part should be {:${SLOT},} (going from more specific to more general).
I would suggest that you write a bug report  |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Sun Jul 20, 2014 3:46 pm Post subject: |
|
|
I agree with mv; the ordering shown is incorrect. eg: cat/pkg:slot should be considered more specific than cat/pkg.
edit: as usual, it's the crappy redundant braces that cause the problem; I imagine whoever wrote the above was just happy to get the bracketing to line up ;) Simplify and it's a lot easier to deal with (and understand):
| Code: | | for check in "$CATEGORY"/{$P-$PR,$P,$PN}{:$SLOT,} |
eg:
| Code: | $ CATEGORY=sys-kernel PN=hardened-sources P=hardened-sources-3.15.5 PR=r1 SLOT=3.15.5-r1
$ printf '%s\n' "$CATEGORY"/{$P-$PR,$P,$PN}{:$SLOT,}
sys-kernel/hardened-sources-3.15.5-r1:3.15.5-r1
sys-kernel/hardened-sources-3.15.5-r1
sys-kernel/hardened-sources-3.15.5:3.15.5-r1
sys-kernel/hardened-sources-3.15.5
sys-kernel/hardened-sources:3.15.5-r1
sys-kernel/hardened-sources
|
Though one might also want to be a bit smarter with PR, that's a better place to go from, imo.
To see what I mean, and run the printf again.
shows the (desired) ordering better for our purposes here:
| Code: | sys-kernel/hardened-sources-3.15.5-r1:SLOT
sys-kernel/hardened-sources-3.15.5-r1
sys-kernel/hardened-sources-3.15.5:SLOT
sys-kernel/hardened-sources-3.15.5
sys-kernel/hardened-sources:SLOT
sys-kernel/hardened-sources |
It's worth quoting "$CATEGORY" as zmedico told me a while back that some of the cross-compilation people use funny things in there; crossdev doesn't seem so bad, but I'm pretty sure he said something about "anything allowed" there, in the context of automated builds. The others are restricted, as are normal category names, but emerge (and thus ebuild.sh) have to deal with other things too, at least in category. Personally I'd quote all of them if it were my code:
| Code: | | for check in "$CATEGORY"/{"$P-$PR","$P","$PN"}{":$SLOT",} |
just because it means I never have to worry (aka: robustness), and the quotes delimit much more usefully (more so with reasonable syntax highlighting) than mindless bracing. |
|
| Back to top |
|
|
Ottre
Tux's lil' helper
Joined: 23 Dec 2012
Posts: 129
|
| Posted: Mon Sep 08, 2014 7:06 am Post subject: |
|
|
Update for people who are interested in user patches.
This feature was proposed for EAPI5 and rejected, proposed again for EAPI6 and approved.
Now mgorny has made the changes to src_prepare():
| Code: |
__eapi6_src_prepare() {
if [[ $(declare -p PATCHES) == "declare -a"* ]]; then
eapply "${PATCHES[@]}"
elif [[ -n ${PATCHES} ]]; then
eapply ${PATCHES}
fi
eapply_user
}
|
And here is the eapply_user() function that it calls. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Portage & Programming
