| View previous topic :: View next topic |
| Shall portage support Function Multi-Versioning? |
| yes |
|
15% |
[ 3 ] |
| no |
|
40% |
[ 8 ] |
| don't care |
|
45% |
[ 9 ] |
|
| Total Votes : 20 |
|
| Author |
Message |
skunk
l33t
Joined: 28 May 2003
Posts: 646
Location: granada, spain
|
 Posted: Fri Apr 06, 2018 3:35 pm Post subject: Support for Function Multi-Versioning (FMV) |
 |
|
[Moderator note: changed title from support for fmv to Support for Function Multi-Versioning (FMV) because FMV in computing commonly means "Full Motion Video", which is completely unrelated to the topic of this thread; changed poll question from shall portage support fmv? to Shall portage support Function Multi-Versioning? for the same reason. -Hu]
hey!
since i use my desktop (ivybridge) to build packages for my laptop (skylake) i would really like to see function multi-versioning support in portage... |
|
| Back to top |
|
 |
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20067
|
| Posted: Fri Apr 06, 2018 4:43 pm Post subject: |
|
|
| Quote: | | optimized for Intel |
So, no.
Although I voted "don't care" as long as the extraneous code isn't clogging up my system.
_________________
Quis separabit? Quo animo? |
|
| Back to top |
|
|
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
| Posted: Fri Apr 06, 2018 6:36 pm Post subject: |
|
|
Someone should remind Intel that GCC exists.
You're not going to get much benefit by building fat binaries for two *lake chips in any case. If you're that hard pressed for speed, better to use hardware without Meltdown. |
|
| Back to top |
|
|
skunk
l33t
Joined: 28 May 2003
Posts: 646
Location: granada, spain
|
| Posted: Fri Apr 06, 2018 6:50 pm Post subject: |
|
|
| pjp wrote: | | Quote: | | optimized for Intel |
So, no.
Although I voted "don't care" as long as the extraneous code isn't clogging up my system. |
i wonder if you've cared to read the whole article or just the title... 
the concept is about building application that runs on old generation cpu's and may leverage on more modern cpu extensions as well.
in my case i would be able to build an application that may use avx2 extensions and run on both my desktop (without avx2) and my laptop (with avx2).
and yes, even amd or arm processors would benefit from such a technique, it's not just another intel thing...
i know the most benefit are for binary distributions, however it would be just another nice (imho) choice even for gentoo...  |
|
| Back to top |
|
|
skunk
l33t
Joined: 28 May 2003
Posts: 646
Location: granada, spain
|
| Posted: Fri Apr 06, 2018 6:56 pm Post subject: |
|
|
| Ant P. wrote: | Someone should remind Intel that GCC exists.
You're not going to get much benefit by building fat binaries for two *lake chips in any case. If you're that hard pressed for speed, better to use hardware without Meltdown. |
uh? those are gcc-6 and glibc-2-26 new features... 
i'll be happy to buy a power9 desktop when they'll start to sell them at a reasonable price and a risc-v laptop when there will be one available...
no intel/amd/arm fanboy here, just someone who buys hardware that suits my needs...  |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54214
Location: 56N 3W
|
| Posted: Fri Apr 06, 2018 7:52 pm Post subject: |
|
|
skunk,
I'm a don't care leaning towards no. This feature cannot be implemented without a performance penalty.
I can't spare the RAM and in some cases the HDD space for the bloat.
Being a Gentoo user everything is tailored to the individual target anyway.
I can see binary distros and purveyors of binaries liking it but in my opinion, its rather like systemd.
Use it if you want to, just don't force it on me.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20067
|
| Posted: Fri Apr 06, 2018 9:19 pm Post subject: |
|
|
| skunk wrote: | | i wonder if you've cared to read the whole article or just the title... |
I didn't presume that the title would be false or inaccurate, and as I don't use Intel CPUs, the rest seemed not relevant. So, yeah.
_________________
Quis separabit? Quo animo? |
|
| Back to top |
|
|
Tony0945
Watchman
Joined: 25 Jul 2006
Posts: 5127
Location: Illinois, USA
|
| Posted: Fri Apr 06, 2018 10:22 pm Post subject: |
|
|
Don't see any need. Run the appropriate march/mtune in your make.conf. This is for binary packages not source packages. If you're building for distribution to Ubuntu or Arch, yeah! There may be a small set of Gentoo users building binaries for Windows or cross-comopling. So, no, no special effort. Developer effort should go into fixing bugs or speeding up portage.
EDIT: spelling
Last edited by Tony0945 on Mon Apr 09, 2018 1:02 am; edited 1 time in total |
|
| Back to top |
|
|
geki
Advocate
Joined: 13 May 2004
Posts: 2387
Location: Germania
|
| Posted: Sat Apr 07, 2018 7:29 am Post subject: |
|
|
My POV as software developer:
Luckily, most (if not all) software, that cares for performance and is in need thereof, does this or similar already. AFAI watch compiles flying by (i.e.: ffmpeg, mesa and others). 
So, nothing really to be done; from this POV.
_________________
hear hear |
|
| Back to top |
|
|
bunder
Bodhisattva
Joined: 10 Apr 2004
Posts: 5934
|
| Posted: Sat Apr 07, 2018 9:31 am Post subject: |
|
|
I would rather upgrade the ivybridge to something newer than skylake and use distcc. 
_________________
| Neddyseagoon wrote: | | The problem with leaving is that you can only do it once and it reduces your influence. |
banned from #gentoo since sept 2017 |
|
| Back to top |
|
|
The_Document
Apprentice
Joined: 03 Feb 2018
Posts: 275
|
| Posted: Sun Apr 08, 2018 10:17 pm Post subject: |
|
|
| No cause its likely to break things. Developers should do it right and implement modern instruction sets right into their code. |
|
| Back to top |
|
|
krinn
Watchman
Joined: 02 May 2003
Posts: 7470
|
| Posted: Mon Apr 09, 2018 4:41 am Post subject: |
|
|
wonderful option for binaries distro
totally useless for gentoo, space & memory eater for no benefits. |
|
| Back to top |
|
|
Goverp
Veteran
Joined: 07 Mar 2007
Posts: 1994
|
| Posted: Mon Apr 09, 2018 7:25 am Post subject: |
|
|
a) Gentoo is about choice
b) Gentoo is a meta-distribution
Ergo, people should be able to build FMV distributions using Gentoo.
But I voted "Don't care"
_________________
Greybeard |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Mon Apr 09, 2018 9:34 am Post subject: |
|
|
| geki wrote: | My POV as software developer:
|
That's exactly what the whole story is about.
| Quote: | | Luckily, most (if not all) software, that cares for performance and is in need thereof, does this or similar already. |
This is plainly false: Only software that cares for performance and has bundled all libraries (at least the ones which could gain from the optimization) can do this.
For libraries which are not bundled but which are pulled in as dependencies, this is exactly what portage would need to care about.
Typical example: numpy (which would profit from an optimized cblas/lapack) |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Mon Apr 09, 2018 9:51 am Post subject: |
|
|
| NeddySeagoon wrote: | | This feature cannot be implemented without a performance penalty. |
A performance penalty only at startup (depending on the glibc implementation perhaps even only once at startup of the machine).
| Quote: | | I can't spare the RAM and in some cases the HDD space for the bloat. |
RAM cost is only a few bytes in glibc. HDD cost is none except if you target multiple architectures. In the latter case, it is only the code for the additional architectures itself which take HDD space.
| Quote: | | I can see binary distros and purveyors of binaries |
Gentoo users are such purveyors if they build binary packages for several machines (which not necessarily all have the same processor features). I think it is a rather frequent setting to have one "build" machine and to distribute the binaries to other systems. Currently, you have to choose the lowest common denominator on these systems.
| Quote: | | Use it if you want to, just don't force it on me. |
Nothing would be forced: From the user side an implementation would probably be rather similar to how ABI_X86=.... currently behaves, i.e. you select "your" architecture. If you do not select a second/third/forth/... architecture nothing changes for you (also no disk space overhead), except that perhaps a different directory path is used for the generated library (lib/arch/ instead of lib/)
Edit: Typos and reformulations to make things clearer. |
|
| Back to top |
|
|
skunk
l33t
Joined: 28 May 2003
Posts: 646
Location: granada, spain
|
| Posted: Mon Apr 09, 2018 10:27 am Post subject: |
|
|
| mv wrote: | | Quote: | | I can see binary distros and purveyors of binaries |
Gentoo users are such purveyors if they build binary packages for several machines (which not necessarily all have the same processor features). I think it is a rather frequent setting to have one "build" machine and to distribute the binaries to other systems. Currently, you have to choose the lowest common denominator on these systems. |
that's exactly what i mean as use case, moreover in my specifically case:- i need to build packages for a modern mobile weak but power efficient processor and i want to use an older but powerful processor for that for obvious reasons
- my job consist in deploying private clouds for my customers using linux containers and want to be able to move those containers around between an heterogeneous set of hosts without having to rebuild the container images on the destination host
of course, as mv pointed out, i'm sure it could be absolutely opt in without jamming it into anybody's throat... |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54214
Location: 56N 3W
|
| Posted: Mon Apr 09, 2018 2:04 pm Post subject: |
|
|
skunk,
I use (cross) distcc in these situations.
My main amd64 /no-multilib/ system builds for arm, arm64, x86 and another low power amd64 system.
I understand the problem but adding bloat is not a part of the solution on a source based distro,
Going back to lowest common denominator builds, all systems wait at the same speed, so its only a few performance critical apps where in actually makes a difference.
e.g. libreoffice spends most of its time waiting for keystrokes.
Video processing can use all the speed it can get, but you don't do that on out of date hardware if you can avoid it.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Mon Apr 09, 2018 3:03 pm Post subject: |
|
|
@skunk: Do you plan to implement it, e.g. in a google SOC project?
Otherwise, I am afraid that this whole poll is rather useless. A poll alone will not inspire an implementation. An implementation is rather tedious; much harder than ABI_X86= since in a sense it is a strict superset of this.
When I think how many years ABI_X86 took... not only the pure implementation itself, but also the fights how it should be implemented and many competing half-finished implementation attempts... |
|
| Back to top |
|
|
skunk
l33t
Joined: 28 May 2003
Posts: 646
Location: granada, spain
|
| Posted: Wed Apr 11, 2018 2:30 pm Post subject: |
|
|
ok guys, i was just asking some opinions about something i've believed could have been useful...
i'll keep doing the lowest common denominator strategy with my servers farm and on my desktop/laptop.
thank you |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Fri May 11, 2018 2:41 pm Post subject: |
|
|
I think there's a definite use-case or three for it, but not as any sort of default, nor do I believe it should require any changes to portage.
eg: I definitely believe skunk should be able to build suitable binpkgs for his cloud, that might adjust to the local CPU.
I don't understand why it can't be a simple addition to CFLAGS or CPPFLAGS, LDFLAGS et al, for those applications which know how to use it.
I'd imagine most don't, so I don't see the point in it being anything that happens at mangler level, but rather in a ROOT, with suitable config-root.
What am I missing mv? (why do you say it needs as much work at mangler level, or more, than ABI_X86, which sucks imo.) |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Fri May 11, 2018 4:36 pm Post subject: |
|
|
| steveL wrote: | | What am I missing mv? (why do you say it needs as much work at mangler level, or more, than ABI_X86, which sucks imo.) |
If you want that a library supports this thing, you have to compile it several times with different CFLAGS and install the result into various /usr/lib/$ARCH directories (or whatever the convention for the location is); the point is that the build system of the library has probably no special support for this.
Only the calling package must be aware of the mechanism, and it needs to be passed somehow at compile time the possible $ARCH values (and directory names) for which the libraries are available.
So a way to implement it would be analogous to ABI_X86: The ebuilds for the libraries inherit from some eclass which builds them multiple times, depending on some USE (or EXPAND_USE) variable, and the calling package has the same USE-variables and corresponding USE-dependencies on the relevant libraries.
I am not claiming that this is the only way how it can be done, but given that this is the approach chosen for multilib support and the underlying problem is in principle the same (different libraries are needed for different processor versions/modes), it is natural to use the same approach. |
|
| Back to top |
|
|
krinn
Watchman
Joined: 02 May 2003
Posts: 7470
|
| Posted: Fri May 11, 2018 6:45 pm Post subject: |
|
|
| mv wrote: | | you have to compile it several times with different CFLAGS |
Maybe i didn't get it fully, but i think the idea is not building a cross compiler or different arch, but running optimizations (not gcc ones, but code specific ones).
So the idea is using the dumbest generic optimization for gcc, while getting optimizations from the libraries and enhancement from the code in use.
ie: building vlc with -march=i686 -mmmx -msse ending up with
vlc for any i686
vlc-lib normal (i686) code
vlc-lib-part-mmx-enable
vlc-lib-part-sse-enable
...
This way, the "package" in its binary form provide all the libs and the program (vlc), and depending on what cpu features the host have, allow it to use vlc-lib-part-"cpufeature" to enhance it.
And user could run vlc on any i686, i686 with mmx, i686 with mmx+sse ; as of today, binary packager has to make a trade off, if he build it with sse, non-sse cpu cannot run it.
This is easier for x86-64 as "root" code available is bigger (mmx, sse... anything that the "root" x86-64 was supporting, must be opteron, except the specific amd feature part), still it start to get odd even for x86-64 because of sse4, sse4.1, avx... support |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Fri May 11, 2018 8:02 pm Post subject: |
|
|
| krinn wrote: | | Maybe i didn't get it fully, but i think the idea is not building a cross compiler or different arch |
$ARCH refers here to things like +ssl-ssl2+mmx (that's why there is a large number of possible $ARCH values and the thing with USE_EXPAND would not be so simple).
Packages with bundled libs require no package manager involvement, of course: They do the multiple translations in their own build system if necessary.
The difficulty is when the libs are a separate package like the already mentioned blas or lapack libs which are used by e.g. numpy. |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Sat May 12, 2018 12:01 am Post subject: |
|
|
| mv wrote: | | $ARCH refers here to things like +ssl-ssl2+mmx (that's why there is a large number of possible $ARCH values and the thing with USE_EXPAND would not be so simple). |
Cheers for the explanation.
Like krinn, I thought this was about enabling extra code paths that can be "dynamically patched" in, equivalently to kernel NOP => traps.
I'm sure glibc does something similar (or did.)
I didn't realise it's about building X variants of the whole library (that sucks, afaic.)
| Quote: | | The difficulty is when the libs are a separate package like the already mentioned blas or lapack libs which are used by e.g. numpy. |
I agree that's an important use-case, that we (Gentoo and its users, ie: users, in the medium-term) should address. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Sat May 12, 2018 6:18 am Post subject: |
|
|
| steveL wrote: | | about enabling extra code paths that can be "dynamically patched" |
How could one do this for an external library? Especially for blas and lapack (which are probably the main example) where more or less every function requires such an adapted code path?
The most "natural" way to solve this (which after the decision about the processor type needs no time at all) is simply to load the "correct" library. Previously this would have required that the library is dynamically loaded, with all the disadvantages like lacking symbol checking at build time. If I understood correctly, the main feature of function multi-versioning is that this is somehow supported now by the linker. |
|
| Back to top |
|
|
|
Gentoo Chat
