[geki-overlay] split build of boost.org libraries

[geki]

___
overlay
https://github.com/geki-yaba/gekis-playground

___
howto
https://github.com/geki-yaba/gekis-play ... i/Boost.md

___
hi there!
I have started splitting up the boost build.
I hope the gentoo boost maintainers may see the use of it somewhen.

general improvements - can be achieved in various ways!

[ - move code from ebuild to eclass
[ => single source to maintain
[ => ebuilds ( and if it is just of the one boost package ) just include the minimal necessary descriptions

[ - add something like boost-utils eclass ( yes, the eselect module could do it, too, and ... )
[ => for packages to get specific version(s) - see: 306151 & hackerly ebuild diff

[ - make libraries optional by ...

optional ideas

[ - create tarballs for patches
[ => clean filesdir
[ => uniform way to handle patches in eclass - apply all patches in directory instead of filling src_prepare with 'epatch $patch'

[ - split headers for RDEPEND <-> DEPEND of packages

not supported

[ - unversioned libraries in LIBDIR
[ - no eselect necessary
[ - either add include/library path to your projects or use boost-utils eclass.
[ => automagic is evil(tm)

___
you may remove boost & eselect-boost and its left-overs located at

/usr/include/boost
/usr/lib/libboost_*

( the eselect code from the ebuild is commented out in the eclass. )

now, you may check your system for boost libraries used.

# libraries
for f in $(find /usr/lib/ -type f -name '*.so*'); do echo $f; ldd $f | grep boost; done
# binaries
for f in $(find /usr/bin/ -type f -perm -111); do echo $f; ldd $f | grep boost; done
# binaries under LIBDIR
for f in $(find /usr/lib/ -type f -perm -111 ! -name '*.so*'); do echo $f; ldd $f | grep boost; done

___
a line of RDEPEND from lince ebuild.

dev-libs/boost[date_time,filesystem,signals,thread]

add 'boost-utils' to 'inherit' at the top of the ebuild
add this line to src_configure before configure ; it checks for the respective library path of the boost-headers version!

add_boost_paths

truely fixes these bugs: 306151 & hackerly ebuild diff

even runtime is resolved properly!

ana portage # ldd /usr/lib64/liblince-1.so.0.0.5|grep boost
libboost_date_time-mt-1_42.so.1.42.0 => /usr/lib/libboost_date_time-mt-1_42.so.1.42.0 (0x00007ff922bbf000)
libboost_signals-mt-1_42.so.1.42.0 => /usr/lib/libboost_signals-mt-1_42.so.1.42.0 (0x00007ff9229ac000)
libboost_system-mt-1_42.so.1.42.0 => /usr/lib/libboost_system-mt-1_42.so.1.42.0 (0x00007ff921c8c000)
libboost_filesystem-mt-1_42.so.1.42.0 => /usr/lib/libboost_filesystem-mt-1_42.so.1.42.0 (0x00007ff921a7b000)
libboost_thread-mt-1_42.so.1.42.0 => /usr/lib/libboost_thread-mt-1_42.so.1.42.0 (0x00007ff92186a000)

voila!

___
one thing may already be said, there is no increase in [bug=307921#c1]compile-time[/bug].
there are two or three intra-dependencies within boost libraries from ~15 libraries available.
- they compile that fast it does not hurt.

___
please help testing! thank you.

note: gentoo boost maintainers have done a great ebuild. nearly 1:1 copy as eclass for splitted version. thanks go to them!

[asturm]

How much time did you spend on achieving that?

Although I'm impressed, this is too hot for me - no way (or, time ) I'm going to figure out which parts of the boost distribution are needed by the various packages that depend on it. Could you please mask it for the not-so-experimental guys?

[Ant P.]

If I've got enough time I'll try breaking my system with this.

I noticed you don't have the icu useflag (possibly others) that's in gentoo's boost ebuild - is that intentional?

[geki]

genstorm
yesterday 5 hours thrashing by getting into it.
today, too, thrashing about 5 hours; testing eselect; failed; added boost-utils eclass with nice effect.

like I said - simple & straight forward for I am not a boost packager.

something like this to see boost dependencies

for f in $(find /usr/lib/ -type f); do echo $f; ldd $f | grep boost; done
for f in $(find /usr/bin -type f -perm -111); do echo $f; ldd $f | grep boost; done

masking, hmm... another day - it is late already.

Ant_P
the useflag [icu] is only interesting for 'dev-libs/boost-regex'.

yes, only some libraries need these useflags.

[geki]

btw, one should update the overlay, it has just been repackaged with latest fixes!

[DaggyStyle]

question, what is the pros over the current way?
also, if current state is one block, why not use one build with useflags?

[geki]

DaggyStyle
I think I have stated why it is useful in the first post already. most of the libraries are not used.

talking about system packages, right now, paludis, and when I start on java-config that one, too, would depend on the full blown boost package.
that is self-explaining: silly for that it uses only a minimal subset.

you may also check a desktop installation for boost libraries used.

for f in $(find /usr/lib/ -type f -name '*.so*'); do echo $f; ldd $f | grep boost; done
for f in $(find /usr/bin/ -type f -perm -111); do echo $f; ldd $f | grep boost; done

or something like this.

like I said already, I just want to show its use to split. I do not say that it has to be this way.
so, another way is to add useflags to the gentoo boost package.
though, instead of adding useflags, I could think of another idea.

what I have in '/etc/make.conf':

APACHE2_MODULES="actions alias auth_basic ..."

how about that way?

but then again, how do you fix the two bugs I have mentioned?

and whenever you need a new boost library you have to merge the boost package, instead of selectively adding the boost library package. hmm ...

[DaggyStyle]

sorry, you post was too long and I've lost concentration.
in that case,why not use one ebuild with multiple use flags?

[yoshi314]

great idea. i think it's only natural for gentoo to split up big multi-feature packages like that.

i wish we had the same thing for openoffice (it actually distributes its apps in separate tarballs, which suggests that upstream is supporting such builds).

[geki]

yoshi314
well, the way I do it is like qt is handled. though, there are other ways to split. each has its pros & cons.

openoffice may be splitted in 5 to 10 years, maybe. first ideas started some years ago.
though, it moves slowly, managing and re-organizing its modules is a pita.

[geki]

I did some minor fixes and cleanup.

if anyone sees the DEBUG message, tell me, otherwise I am going to bin it.

# FIXME: this is not triggered!
# Remove (unversioned) symlinks
# And check for what we remove to catch bugs
# got a better idea how to do it? tell me!
for f in $(ls -1 ${LIBRARY_TARGETS} | grep -v "${MAJOR_PV}") ; do
echo "DEBUG: $f"

if [ ! -h "${f}" ] ; then
eerror "Ups, tried to remove '${f}' which is a a real file instead of a symlink"
die "slotting/naming of the libs broken!"
fi

rm "${f}" || die
done

[Kingoftherings]

This is a great idea. About 80% of the Boost libraries are headers only anyway, but the remaining parts that need to be built take forever.

Now we just need ebuilds that specify which parts of Boost they need.

[SpectreDragon]

hm I get something like this for apparently every boost ebuild except boost-headers, am I doing something wrong?

Code: Select all

>>> Emerging (2 of 9) dev-libs/boost-system-1.42.0 from geki-overlay
 * boost_1_42_0.tar.bz2 RMD160 SHA1 SHA256 size ;-) ... [ ok ]
 * checking ebuild checksums ;-) ... [ ok ]
 * checking auxfile checksums ;-) ... [ ok ]
 * checking miscfile checksums ;-) ... [ ok ]
 * CPV:  dev-libs/boost-system-1.42.0
 * REPO: geki-overlay
 * USE:  amd64 elibc_glibc kernel_linux threads userland_GNU
>>> Unpacking source...
tar xjpf /var/tmp/portage/dev-libs/boost-system-1.42.0/distdir/boost_1_42_0.tar.bz2 --exclude=boost_1_42_0/boost --exclude=boost_1_42_0/libs
tar xjpf /var/tmp/portage/dev-libs/boost-system-1.42.0/distdir/boost_1_42_0.tar.bz2 boost_1_42_0/libs/system
>>> Source unpacked in /var/tmp/portage/dev-libs/boost-system-1.42.0/work
>>> Preparing source in /var/tmp/portage/dev-libs/boost-system-1.42.0/work/boost_1_42_0 ...
 * Applying remove-toolset-1.42.0.patch ... [ ok ]
 * Applying skip-headers.diff ... [ ok ]
>>> Source prepared.
>>> Configuring source in /var/tmp/portage/dev-libs/boost-system-1.42.0/work/boost_1_42_0 ...
 * Writing new user-config.jam
 * found boost version 1_42
 * ERROR: dev-libs/boost-system-1.42.0 failed:
 *   path not found!
 * 
 * Call stack:
 *     ebuild.sh, line   48:  Called src_configure
 *   environment, line 5094:  Called boost_src_configure
 *   environment, line  945:  Called add_boost_paths '=142'
 *   environment, line  826:  Called add_boost_library_path '1_42'
 *   environment, line  818:  Called die
 * The specific snippet of code:
 *           die "path not found!";
 * 
 * If you need support, post the output of 'emerge --info =dev-libs/boost-system-1.42.0',
 * the complete build log and the output of 'emerge -pqv =dev-libs/boost-system-1.42.0'.
 * This ebuild used the following eclasses from overlays:
 *   /var/lib/layman/openoffice-geki/eclass/boost.eclass
 *   /var/lib/layman/openoffice-geki/eclass/boost-utils.eclass
 * This ebuild is from an overlay named 'geki-overlay': '/var/lib/layman/openoffice-geki/'
 * The complete build log is located at '/var/tmp/portage/dev-libs/boost-system-1.42.0/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/dev-libs/boost-system-1.42.0/temp/environment'.
 * S: '/var/tmp/portage/dev-libs/boost-system-1.42.0/work/boost_1_42_0'

[i92guboj]

sorry, you post was too long and I've lost concentration.
in that case,why not use one ebuild with multiple use flags?

You can always do that later. Just wrap all the boost-* ebuilds into a boost-meta ebuild and add lots of use flags if that's your call. Not that your are going to gain anything doing that, though. The modular approach has only advantages, just like in kde or qt to name a couple of examples. That way, other ebuilds can depend only on the needed split package, instead of depending upon the whole monster that boost is.

[geki]

SpectreDragon
I was adding include & library path in boost_src_configure. so the first time were failing like that.

thanks for report and fixed!

i92guboj
I just want to point out the other way around. have a look at dbus.
it was split-packaged and then merged into one package again.
many ways possible.

but, yes, having the boost headers split for DEPEND over RDEPEND and selective libraries is just sweet.

[mv]

sorry, you post was too long and I've lost concentration.
in that case,why not use one ebuild with multiple use flags?

You can always do that later. Just wrap all the boost-* ebuilds into a boost-meta ebuild and add lots of use flags if that's your call.

This would make much more sense and be easier to maintain. For instance, poppler (and some others which I forgot now) have been changed from split ebuild into an ebuild with useflag.

Not that your are going to gain anything doing that, though.

For instance, you gain that an

Code: Select all

emerge -f @world

(e.g. with the trickyfetch script) would be much faster, because there is no need to checksum the same tarball everytime. Also maintenance (revbump) is much easier, since one revbump changes everything and you do not have the problem that you need to check whether different versions are compatible (or even have to block each other which can cause unnecessary headache for the user).

The modular approach has only advantages, just like in kde or qt to name a couple of examples.

I can see only disadvantages. It only makes sense if there are reasons to recompile rather often only some of the split ebuilds and not any others. For a mega-project like kde this makes sense (revdep-rebuild), for qt the approach is already doubtfully.

That way, other ebuilds can depend only on the needed split package, instead of depending upon the whole monster that boost is.

Which is a completely meaningless argument now that portage has USE-dependencies.

[geki]

For instance, you gain that an

Code: Select all

emerge -f @world

(e.g. with the trickyfetch script) would be much faster, because there is no need to checksum the same tarball everytime.

doing multiple checksum and portage idle my split-boost is 1/3 of merge time compared to full boost. what do I miss in your statement?

Also maintenance (revbump) is much easier, since one revbump changes everything and you do not have the problem that you need to check whether different versions are compatible (or even have to block each other which can cause unnecessary headache for the user)

I still have SLOTs and instead of one ebuild I have one eclass. revbump causes one library to remerge over complete boost. hmm ....

I can see only disadvantages. It only makes sense if there are reasons to recompile rather often only some of the split ebuilds and not any others. ... for qt the approach is already doubtfully.

to the latter, you must have a high-end cpu. I get sick each time I have to update qt4. I am glad it is split. to the former, yes, I use just some. and the general desktop will. remember, I just have to maintain one eclass instead of one ebuild. no difference in maintenance.

That way, other ebuilds can depend only on the needed split package, instead of depending upon the whole monster that boost is.

Which is a completely meaningless argument now that portage has USE-dependencies.

o ha, try that on a not so brand-new hardware.
just have a look at bjam analysing the headers on install stage. if you got that time, you can do it the other way around, too.

[SpectreDragon]

SpectreDragon
I was adding include & library path in boost_src_configure. so the first time were failing like that.

thanks for report and fixed!
...

thanks. next would be..

Code: Select all

...

libs/program_options/src/utf8_codecvt_facet.cpp:15:47: error: ../../detail/utf8_codecvt_facet.cpp: No such file or directory
gcc.compile.c++ bin.v2/libs/program_options/build/gcc-4.4/gentoorelease/pch-off/value_semantic.o

    "x86_64-pc-linux-gnu-g++"  -ftemplate-depth-128 -march=core2 -O2 -pipe -msse3 -mssse3 -fno-strict-aliasing -I/usr/include/boost-1_42 -finline-functions -Wno-inline -Wall -fPIC  -DBOOST_ALL_NO_LIB=1 -DBOOST_PROGRAM_OPTIONS_DYN_LINK=1 -DNDEBUG  -I"." -c -o "bin.v2/libs/program_options/build/gcc-4.4/gentoorelease/pch-off/value_semantic.o" "libs/program_options/src/value_semantic.cpp"

...failed updating 1 target...
 * ERROR: dev-libs/boost-program_options-1.42.0 failed:
 *   building boost failed
 * 
 * Call stack:
 *     ebuild.sh, line   48:  Called src_compile
 *   environment, line 5096:  Called boost_src_compile
 *   environment, line  923:  Called die
 * The specific snippet of code:
 *       ${BJAM} ${NUMJOBS} -q -d+2 gentoorelease ${OPTIONS} threading=${threading} ${LINK_OPTS} runtime-link=shared || die "building boost failed";
 * 
 * If you need support, post the output of 'emerge --info =dev-libs/boost-program_options-1.42.0',
 * the complete build log and the output of 'emerge -pqv =dev-libs/boost-program_options-1.42.0'.
 * This ebuild used the following eclasses from overlays:
 *   /var/lib/layman/openoffice-geki/eclass/boost.eclass
 *   /var/lib/layman/openoffice-geki/eclass/boost-utils.eclass
 * This ebuild is from an overlay named 'geki-overlay': '/var/lib/layman/openoffice-geki/'
 * The complete build log is located at '/var/tmp/portage/dev-libs/boost-program_options-1.42.0/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/dev-libs/boost-program_options-1.42.0/temp/environment'.
 * S: '/var/tmp/portage/dev-libs/boost-program_options-1.42.0/work/boost_1_42_0'

looks like if there is something missing?

[mv]

For instance, you gain that an

Code: Select all

emerge -f @world

(e.g. with the trickyfetch script) would be much faster, because there is no need to checksum the same tarball everytime.

doing multiple checksum and portage idle my split-boost is 1/3 of merge time compared to full boost. what do I miss in your statement?

You mean emerging the related split ebuilds is three times faster than emerging one monolithic ebuild (with the corresponding USE-flags set, of course)? In this case, I would say that the build system of the monolithic ebuild is broken.
However, it is more likely that something in your measurement is false (e.g. you have used ccache and did not empty it before emerging the split ebuilds or something similar).

I still have SLOTs and instead of one ebuild I have one eclass. revbump causes one library to remerge over complete boost. hmm ....

So you install redundant identical data in different slots. And the eselect of course changes slots of every split ebuild simultaneously so that it makes absolutely no sense to have one of the split ebuild installed in one version but not in another. Once again: What is the advantage of not installing all in one ebuild?

I can see only disadvantages. It only makes sense if there are reasons to recompile rather often only some of the split ebuilds and not any others. ... for qt the approach is already doubtfully.

to the latter, you must have a high-end cpu. I get sick each time I have to update qt4. I am glad it is split.

For upgrading qt, it would need the same (actually even less) time if qt were not split (but of course only those parts corresponding to the USE-flags are compiled). The only case where you see a difference is if you have to reemerge the same version of a split ebuild but not the other installed ones - this usually can only happen during revdep-rebuild, if only one of the split ebuilds has a dependency on some library but the others not. But how often does this really occur?

remember, I just have to maintain one eclass instead of one ebuild. no difference in maintenance.

There will be enormous difference if you want to avoid installing redundant data. If you do not want to avoid this, split ebuild makes no sense - USE-flags-ebuild is preferrable.

That way, other ebuilds can depend only on the needed split package, instead of depending upon the whole monster that boost is.

Which is a completely meaningless argument now that portage has USE-dependencies.

o ha, try that on a not so brand-new hardware.

What are you talking about? Do you seriously believe that portage needs considerable more CPU-time to track USE-dependencies instead of a split ebuild?

just have a look at bjam analysing the headers on install stage. if you got that time, you can do it the other way around, too.

I do not understand how this is related with the question of whether split ebuilds or a monolithic ebuild with USE-flags (to determine the parts to be built) is preferrable.

[Genone]

'inherit boost-utils' - add this line to src_configure before configure ; it checks for the minimum version required!

You might want to rephrase that, in the current form it sounds (to me at least) like the inherit call should be added in src_configure, I had to read it three times until I realized you meant the following line. Fortunately I couldn't believe anyone would really call inherit inside a function

[geki]

mv
well, boost in portage is what is.

and to most of the other things I states my pov already. I know the advantages and disadvantages of various ways. how it is done it does not really matter but for the headers being DEPEND or RDEPEND, then. just thinking about intranet distribution. and yes, we have the bandwidth it does not really matter. so we may turn around and round and ... in the end it is how the maintainers want it to be.

just one final thing.

There will be enormous difference if you want to avoid installing redundant data. If you do not want to avoid this, split ebuild makes no sense - USE-flags-ebuild is preferrable.

that you should show me. at the moment I avoid more than the other way.

Genone
let's see.

[geki]

and another round of dependency fixes.

[mv]

There will be enormous difference if you want to avoid installing redundant data. If you do not want to avoid this, split ebuild makes no sense - USE-flags-ebuild is preferrable.

that you should show me.

Let us say you have boost-filesystem:1.41 and boost-filesystem:1.42 but actually the filesystem-related code did not change in the upgrade boost-1.41 ->boost-1.42 (i.e. only other parts of boost were changed). Then you could omit boost-filesystem:1.42 and depend instead on boost-filesystem:1.41: This way, the user would neither have to upgrade to boost-filesystem:1.42 nor would he have to install boost-filesystem:1.42 redundantly if he wants to have both slots. Of course, eselect-boost would need to be smart enough to set correct symlinks.
If ebuilds are really maintained this way, this is a lot of work for the maintainer, but it really saves compilation time for the user: The maintainer has to check for every upgrade which packages really changed. Moreover, it would be sane to change the boost-filesystem:1.41 ebuild (without revbump) for the upgrade to use the tarball of boost-1.42: This way, the user who wants only the most current boost would not have to keep the old boost-1.41 tarball around (or download it again) when he needs to re-emerge boost-filesystem:1.41.

at the moment I avoid more than the other way.

I do not see anything which you avoid in the moment - I mean compared to ebuild with USE-flags, of course. Just in case that this was a misunderstanding: I do not argue, of course, that a split ebuild has advantages over a monolithic ebuild without USE-flags. However, a monolithic ebuild with USE-flags has even more advantages.
This is what happened e.g. to poppler: First, it was split, but then the maintainer decided that it is more reasonable to have a monolothic ebuild with USE-flags instead and to have USE-dependencies. I can understand this both, from a user perspective (concerning problems with upgrade blockers and the mentioned repeated checksumming) as well as from the maintainer's perspective (you have to bump all ebuilds, and unless you do the mentioned work with comparing versions for every bump, there is no real point in having split ebuilds). Of course, this is only possible since USE-dependencies are available - otherwise it would be a nightmare for users.
In my opinion there is just a few cases where - in the presence of USE-dependencies - it really makes sense to have split ebuilds: This is mainly if a package has many split ebuilds which in turn need to have separate USE-flags or are such that it is not an exception that a user wants to install or uninstall one of the split ebuilds independently from the others. A collection of programs like KDE is an example where both cases apply. A collection of libraries like boost/qt/poppler does not belong to this class in my opinion, since most users will install these only by means of dependencies, and these are not likely to change frequently.

[geki]

well, you talk about my implementation. that is just not the point in this thread. what I want to show is what can be improved. full stop.
the implementation is up to the maintainer alone and his/her liking.


p.s.: I added some notes to the first post. anything else is uninteresting to discuss here.

[SpectreDragon]

using the splitted boost every ebuild depending on some boost libraries fails to detect it :-/
is there anything I have to do in order to get this fixed?

some examples:

from your openoffice ebuild

Code: Select all

...
checking which boost to use... external
checking boost/shared_ptr.hpp usability... no
checking boost/shared_ptr.hpp presence... no
checking for boost/shared_ptr.hpp... no
configure: error: boost/shared_ptr.hpp not found. install boost
make: *** [stamp/build] Error 1

kde-base/akonadi

Code: Select all

...
-----------------------------------------------------------------------------
-- The following OPTIONAL packages could NOT be located on your system.
-- Consider installing them to enable more features from this software.
-----------------------------------------------------------------------------
   * Boost (1.34.0 or higher)  <http://www.boost.org>
     Boost C++ Libraries
     Required by several critical KDEPIM apps.
   * The Boost Topological_sort header  <http://www.boost.org/>
     topological_sort.hpp (part of Boost_graph)
     Necessary to build kleopatra

-----------------------------------------------------------------------------

-- <<< Gentoo configuration >>>
Build type: Gentoo
Install path: /usr

CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
Boost_INCLUDE_DIR (ADVANCED)
   used as include directory in directory /var/tmp/portage/kde-base/akonadi-4.4.1/work/akonadi-4.4.1/akonadi/kcal
   used as include directory in directory /var/tmp/portage/kde-base/akonadi-4.4.1/work/akonadi-4.4.1/akonadi/akonadi_next
   used as include directory in directory /var/tmp/portage/kde-base/akonadi-4.4.1/work/akonadi-4.4.1/akonadiconsole