| View previous topic :: View next topic |
| Author |
Message |
Anard
Apprentice
Joined: 01 Oct 2020
Posts: 251
|
 Posted: Tue Dec 29, 2020 1:01 pm Post subject: emerge for another system |
 |
|
Hi.
I use Gentoo on a desktop PC (intel i5) and a laptop (intel core2duo) with few different configurations.
It works great on the desktop, and laptop too, but compilation times are 4 or 5x longer on laptop, (now upgrading GCC? it takes about 4h30 vs less than an hour on the desktop).
I've tried to install crossdev on the desktop but I think there are much things I didn't understand : when trying to create the target I have errors when emerging binutils :
| Code: | anard@gentoo-imack ~ $ sudo crossdev -t x86-hp-linux-gnu
Mot de passe :
-
* crossdev version: 20201122
* Host Portage ARCH: amd64
* Host Portage System: x86_64-pc-linux-gnu (i686-pc-linux-gnu x86_64-pc-linux-gnu)
* Target Portage ARCH: *
* Target System: x86-hp-linux-gnu
* Stage: 4 (C/C++ compiler)
* USE=multilib: no
* Target ABIs: default
* binutils: binutils-[latest]
* gcc: gcc-[latest]
* headers: linux-headers-[latest]
* libc: glibc-[latest]
* CROSSDEV_OVERLAY: /var/db/repos/localrepo-crossdev
* PORT_LOGDIR: /var/log/portage
* PORTAGE_CONFIGROOT: /
* Portage flags:
* leaving sys-devel/binutils in /var/db/repos/localrepo-crossdev
* leaving sys-devel/gcc in /var/db/repos/localrepo-crossdev
* leaving sys-kernel/linux-headers in /var/db/repos/localrepo-crossdev
* leaving sys-libs/glibc in /var/db/repos/localrepo-crossdev
* leaving sys-devel/gdb in /var/db/repos/localrepo-crossdev
* leaving metadata/layout.conf alone in /var/db/repos/localrepo-crossdev
!!! WARNING - Cannot auto-configure CHOST x86-hp-linux-gnu;
!!! You should edit /usr/x86-hp-linux-gnu/etc/portage/make.conf /usr/x86-hp-linux-gnu/etc/portage/profile/make.defaults /usr/x86-hp-linux-gnu/etc/portage/profile/use.force
!!! by hand to complete your configuration.
!!! No ARCH is known for this target.
* Log: /var/log/portage/cross-x86-hp-linux-gnu-binutils.log
* Emerging cross-binutils ...
* error: binutils failed :(
*
* If you file a bug, please attach the following logfiles:
* /var/log/portage/cross-x86-hp-linux-gnu-info.log
* /var/log/portage/cross-x86-hp-linux-gnu-binutils.log.xz
* /var/tmp/portage/cross-x86-hp-linux-gnu/binutils*/temp/binutils-config.logs.tar.xz
|
I also tried to select another version of binutils and gcc without results.
My wish is simply to update my working laptop system with the performance of the desktop. How could I chroot in the laptop from desktop via ssh to update my system ?
Thanks for help
_________________
"iMack" : GA-H97M-D3H, Intel i7 4790, 16Go DDR3, Sapphire RX570 4Go, 2x SSD 256Go, HDD 500Go + Zpool 3x2To / Clover - macOS Mojave / Gentoo-Xfce |
|
| Back to top |
|
 |
alamahant
Advocate
Joined: 23 Mar 2019
Posts: 3961
|
| Posted: Tue Dec 29, 2020 1:21 pm Post subject: |
|
|
Hi
Maybe you can use SSHFS to mount the remote root filedystem and then chroot into it.
I hope it will work.......
Just checked it.It works just fine...
I am not sure though ifall xattrs etc are preserved.....
Try it.........
_________________
 |
|
| Back to top |
|
|
Anard
Apprentice
Joined: 01 Oct 2020
Posts: 251
|
| Posted: Tue Dec 29, 2020 1:49 pm Post subject: |
|
|
Thx. It seems to work but ... not sure it enconter a problem on /dev/null permissions...
| Code: | gentoo-imack ~ # mkdir /mnt/portable
gentoo-imack ~ # sshfs gentoo-portable:/ /mnt/portable/
Password:
gentoo-imack ~ # ls /mnt/portable/
bin dev home lib64 media opt root sbin tmp var
boot etc lib lost+found mnt proc run sys usr
gentoo-imack ~ # chroot /mnt/portable/
bash: /dev/null: Permission non accordée
gentoo-imack / #
|
Anyway, when I cat /etc/portage/make.conf, it's laptop's make.conf which is shown, so maybe all OK ?
_________________
"iMack" : GA-H97M-D3H, Intel i7 4790, 16Go DDR3, Sapphire RX570 4Go, 2x SSD 256Go, HDD 500Go + Zpool 3x2To / Clover - macOS Mojave / Gentoo-Xfce |
|
| Back to top |
|
|
alamahant
Advocate
Joined: 23 Mar 2019
Posts: 3961
|
| Posted: Tue Dec 29, 2020 1:54 pm Post subject: |
|
|
Yes there are some weird permission issues.....
I think one of the masters will be able to help you........
_________________
|
|
| Back to top |
|
|
Anard
Apprentice
Joined: 01 Oct 2020
Posts: 251
|
| Posted: Tue Dec 29, 2020 2:12 pm Post subject: |
|
|
After reading https://wiki.gentoo.org/wiki/Chroot it seems to work better...
Trying to emerge portage from desktop... 
| Code: | | (chroot) portable from gentoo-imack / # emerge -auDN portage |
EDIT :
No.  After a very long pre-check :
| Code: | Would you like to merge these packages? [Yes/No]
>>> Verifying ebuild manifests
>>> Running pre-merge checks for sys-devel/gcc-9.3.0-r1
Exception in callback AsynchronousTask._exit_listener_cb(<bound method...7f373fea39b0>>)
handle: <Handle AsynchronousTask._exit_listener_cb(<bound method...7f373fea39b0>>)>
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/_emerge/AbstractEbuildProcess.py", line 216, in _init_ipc_fifos
st = os.lstat(p)
File "/usr/lib/python3.7/site-packages/portage/__init__.py", line 229, in __call__
rval = self._func(*wrapped_args, **wrapped_kwargs)
FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sys-devel/gcc-9.3.0-r1/.ipc_in'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.7/asyncio/events.py", line 88, in _run
self._context.run(self._callback, *self._args)
File "/usr/lib/python3.7/site-packages/_emerge/AsynchronousTask.py", line 201, in _exit_listener_cb
listener(self)
File "/usr/lib/python3.7/site-packages/_emerge/EbuildPhase.py", line 158, in _async_start_exit
self._start_lock()
File "/usr/lib/python3.7/site-packages/_emerge/EbuildPhase.py", line 178, in _start_lock
self._start_ebuild()
File "/usr/lib/python3.7/site-packages/_emerge/EbuildPhase.py", line 221, in _start_ebuild
self._start_task(ebuild_process, self._ebuild_exit)
File "/usr/lib/python3.7/site-packages/_emerge/CompositeTask.py", line 113, in _start_task
task.start()
File "/usr/lib/python3.7/site-packages/_emerge/AsynchronousTask.py", line 30, in start
self._start()
File "/usr/lib/python3.7/site-packages/_emerge/AbstractEbuildProcess.py", line 169, in _start
self._start_post_builddir_lock(start_ipc_daemon=start_ipc_daemon)
File "/usr/lib/python3.7/site-packages/_emerge/AbstractEbuildProcess.py", line 187, in _start_post_builddir_lock
self._start_ipc_daemon()
File "/usr/lib/python3.7/site-packages/_emerge/AbstractEbuildProcess.py", line 249, in _start_ipc_daemon
input_fifo, output_fifo = self._init_ipc_fifos()
File "/usr/lib/python3.7/site-packages/_emerge/AbstractEbuildProcess.py", line 218, in _init_ipc_fifos
os.mkfifo(p)
PermissionError: [Errno 1] Operation not permitted
Complété
(chroot) portable from gentoo-imack / #
|
What I did was :
| Code: | gentoo-imack ~ # mkdir /mnt/portable
gentoo-imack ~ # sshfs gentoo-portable:/ /mnt/portable
gentoo-imack ~ # mount --rbind /dev /mnt/portable/dev
gentoo-imack ~ # mount --make-rslave /mnt/portable/dev
gentoo-imack ~ # mount -t proc /proc /mnt/portable/proc
gentoo-imack ~ # mount --rbind /sys /mnt/portable/sys
gentoo-imack ~ # mount --make-rslave /mnt/portable/sys
gentoo-imack ~ # mount --rbind /tmp /mnt/portable/tmp
gentoo-imack ~ # cp /etc/resolv.conf /mnt/mychroot/etc
gentoo-imack ~ # chroot /mnt/portable /bin/bash
gentoo-imack / # env-update && . /etc/profile
>>> Regenerating /etc/ld.so.cache...
gentoo-imack / # export PS1="(chroot) portable from $PS1"
(chroot) portable from gentoo-imack / # emerge -auDN portage
|
_________________
"iMack" : GA-H97M-D3H, Intel i7 4790, 16Go DDR3, Sapphire RX570 4Go, 2x SSD 256Go, HDD 500Go + Zpool 3x2To / Clover - macOS Mojave / Gentoo-Xfce |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 23532
|
| Posted: Tue Dec 29, 2020 4:56 pm Post subject: |
|
|
| Although it requires some compromises elsewhere, I think you might find it less frustrating to reconfigure the desktop's /etc/portage/make.conf so that anything it builds will run on the laptop. Tell the desktop not to use CPU features that the laptop lacks. Then, make the desktop build the package and save a tbz2 of the built package. Tell the laptop to use the desktop as a binhost. You could have the laptop do some smaller packages for itself, and use the binhost only for large packages. If your uses are similar enough between the two systems, you could have the laptop download everything from the desktop, and build nothing for itself. Effectively, you would make your own binary distribution. |
|
| Back to top |
|
|
alamahant
Advocate
Joined: 23 Mar 2019
Posts: 3961
|
| Posted: Tue Dec 29, 2020 5:23 pm Post subject: |
|
|
Best to listen to Hu
but i managed to overcome the permission issue by
| Code: |
sshfs root@192.168.2.54:/ /mnt -o default_permissions,allow_other,idmap=user,workaround=truncate:rename
|
I was able to update a Debian vm by chrooting in the mounted sshfs folder.
I dont have a Gentoo vm so I cant gurantee it will work.......
_________________
|
|
| Back to top |
|
|
nick_gentoo
Tux's lil' helper
Joined: 07 Jan 2019
Posts: 140
|
| Posted: Tue Dec 29, 2020 7:25 pm Post subject: |
|
|
| Hi, related to the original question, isn't this what distcc is able to do? |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 23532
|
| Posted: Tue Dec 29, 2020 8:11 pm Post subject: |
|
|
| In part, yes. However, distcc does not work with some package build systems, and even when it does, it only distributes the compilation phase of C/C++ programs. It doesn't distribute preprocessing or linking, and it doesn't work on programs that are not C/C++. Given how many key packages have significantly rusted in the last few years, distcc is less useful on a broad scale than it once was. A good implementation of distrust might help with the latter problem. |
|
| Back to top |
|
|
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3654
|
| Posted: Wed Dec 30, 2020 1:18 am Post subject: |
|
|
| Quote: | | and even when it does, it only distributes the compilation phase of C/C++ programs. It doesn't distribute preprocessing or linking, |
It had pump mode for ages, and in this mode it does distribute more than just the compilation phase, at the cost of requiring the sources to be actually static which would break builds in such corner cases as linux kernel for example.
Still, I had the best experience with root NFS-mounted as a network share to the beefy machine, and running updates in chroot. Essentially "borrowing" the CPU. As long as both machines are capable of running the resulting code, of course - which may require setting cflags manually ("native" will break anything containing more exotic instructions, since the beefy CPU you build with is probably newer, faster, more advanced, and supports some instruction sets that are illegal in the other machine)
Oh, right, that was before xattrs too, so perhaps there is a better way to mount your root now.
| Quote: | | I've tried to install crossdev on the desktop but I think there are much things I didn't understand : when trying to create the target I have errors when emerging binutils |
Creating a cross-compiller for the same architecture is not exactly straight-forward. Fortunately, in many cases it's also unnecessary, since what you actually need is the same toolchain with another value for -march= and CFLAGS. |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 23532
|
| Posted: Wed Dec 30, 2020 2:12 am Post subject: |
|
|
| I specifically ignored pump mode because, as I understand it, it was even more fragile than general distcc. |
|
| Back to top |
|
|
Tony0945
Watchman
Joined: 25 Jul 2006
Posts: 5127
Location: Illinois, USA
|
| Posted: Wed Dec 30, 2020 3:59 am Post subject: |
|
|
I have used these two scripts to build for my Phenom II on Ryzen | Code: | /usr/local/bin $ cat buildfam10
#! /bin/bash
if [ -L /etc/portage/make.conf ]
then
rm /etc/portage/make.conf
ln -s /etc/portage/make.conf.amdfam10 /etc/portage/make.conf
else
echo "/etc/portage/make.conf is not a symlink "
fi
tony@MSI /usr/local/bin $ cat buildnative
#! /bin/bash
if [ -L /etc/portage/make.conf ]
then
rm /etc/portage/make.conf
ln -s /etc/portage/make.conf.native /etc/portage/make.conf
else
echo "/etc/portage/make.conf is not a symlink "
fi |
To convert "mv /etc/portage/make.conf /etc/portage/make.conf.native && ln -s /etc/portage/make.conf.native /etc/portage/make.conf"
I then copied the make.conf from the Phenom II to /etc/portage/make.conf.amdfam10 on the build box.
Inside the make.conf.amdfam10 I set PKGDIR="/usr/local/portage/amdfam10/packages"
and added "--buildpkg=y --buildpkgonly" to EMERGE_DEFAULT_OPTS
Then with a single command I can flip from building amdfam10 to building for ryzen (--march=native) and back.
This won't work with static linking. Also "eix" and possibly some other packages ignore CFLAGS and determine march from what one is building on. IMHO, that is a bug in the build system. After building I use rsync to update the package dir on the target. Then I re-emerge then with --usepkgonly which is very quick since it only untars the package and updates the portage database. |
|
| Back to top |
|
|
nick_gentoo
Tux's lil' helper
Joined: 07 Jan 2019
Posts: 140
|
| Posted: Wed Dec 30, 2020 8:27 am Post subject: |
|
|
| Hu wrote: | | In part, yes. However, distcc does not work with some package build systems, and even when it does, it only distributes the compilation phase of C/C++ programs. It doesn't distribute preprocessing or linking, and it doesn't work on programs that are not C/C++. Given how many key packages have significantly rusted in the last few years, distcc is less useful on a broad scale than it once was. A good implementation of distrust might help with the latter problem. |
Thank you for the details! I also knew most of this, but it's good to be reminded of all the limitations.
I am in a situation similar to Tony, but using now distcc to compile on a Ryzen box for a Phenom I, and did not care so much lately for the speed gains. I think now I like your variant better.
| Hu wrote: | | Effectively, you would make your own binary distribution. |
I didn't expect it, but I impressed some friends last week when I told them that I run Gentoo. Now I know I can top this next year: "Guys, now I am running my own binary distribution"  |
|
| Back to top |
|
|
eccerr0r
Watchman
Joined: 01 Jul 2004
Posts: 10023
Location: almost Mile High in the USA
|
| Posted: Wed Dec 30, 2020 8:59 am Post subject: |
|
|
Distcc also does not help on packages that compile itself or isn't supported, gcc is one, and rust is the other case.
Unfortunately my solution is still distcc for my Atom and suck up the 17 hour rust builds :( (rust-bin? Meh.)
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
| Back to top |
|
|
Anard
Apprentice
Joined: 01 Oct 2020
Posts: 251
|
| Posted: Wed Dec 30, 2020 10:28 am Post subject: |
|
|
Thx for your interest.
I just tried to upgrade my laptop with distcc... compilation times have grown so much  Even when distcc is used, CPUs on the desktop are not used at 100%, but laptop's are fully used. I'd like to avoid compiling on it also because its temperature rise (65-80°C) during compilation, during long hours and I'm afraid to burn its components.
17 hours to emerge rust ? I must consider happy : 5-7 hours in normal time, through distcc, it raised to 8 hours.
I think I should study Tony0945's solution, even not sure I'm able to. Particularly, I need to understand how to make package.use, package.accept_keywords, package.license and /var/lib/portage/world also adapted if target is different.
Should I edit the script to apply it on full /etc/portage /var/lib/portage folders ? Or I risk imprevisible results ?
_________________
"iMack" : GA-H97M-D3H, Intel i7 4790, 16Go DDR3, Sapphire RX570 4Go, 2x SSD 256Go, HDD 500Go + Zpool 3x2To / Clover - macOS Mojave / Gentoo-Xfce |
|
| Back to top |
|
|
flysideways
Guru
Joined: 29 Jan 2005
Posts: 508
|
| Posted: Wed Dec 30, 2020 2:19 pm Post subject: |
|
|
There is guidance for specifying the hosts in the Gentoo distcc wiki and the distcc man page.
Specify the number of jobs on each machine with ipaddress/n ipaddress/n, with the good helper listed first and n set to at least its number of cores. Specify n as 1 for the machine being helped.
| Code: | | distcc-config --set-hosts "ipaddress/n ipaddress/n" |
can be used dynamically while emerging to specify the load. |
|
| Back to top |
|
|
Tony0945
Watchman
Joined: 25 Jul 2006
Posts: 5127
Location: Illinois, USA
|
| Posted: Wed Dec 30, 2020 3:48 pm Post subject: |
|
|
| Anard wrote: | | I think I should study Tony0945's solution, even not sure I'm able to. Particularly, I need to understand how to make package.use, package.accept_keywords, package.license and /var/lib/portage/world also adapted if target is different. |
That is indeed a problem. In my case, AFAIK, the Ryzen is a superset of the K10. II don't think it would work to build for say an i7. Also I ave no substantial differences in keywords or flags, both machines being personal workstations with different physical locations. Most importantly, the k10 is not exactly weak. I use this technique inly for a few packages, not everything.
Another cross-compilation technique I use is for a 32-bit k-6 three. Here I do build anythiong because the CPU is slow compared to modern CPU's, the drive is a slow (fast for its time) PATA drive, and memory is less than 1G. I build for this one one the k10 with an entirely separate partition that I chroot into https://wiki.gentoo.org/wiki/Project:AMD64/32-bit_Chroot_Guide. This partition believes it is an i486 and builds with "-march=k6". It has it's own grub legacy installation installed on the partition which I can also chain load from boot. It's a mirror of the target. Just a much faster version of the target with a full 4G memory. Only eix has given me trouble so far, so I have to build eix natively.
I would like to build on the raspberry pi, especially now that Sakaki will no longer be making builds. I'm pretty sure I need crossdev for that.
However, either of my two techniques should work if the build box is a superset of the target box or only a few obscure instructions need to be excluded from CFLAGS.
I did post a question about using crossdev with emphasis on Mingw earlier this year, but never got any replies. My problem was the crossdev products bleeding into the host environment. I too would see a better crossdev wiki page with explanation of why we do certain things, rather than "do this, run that, then run that also". That's a general complaint about the wiki. If you don't understand the "why" you can't really change the "what" or recognize why it doesn't fit your case. Okay, I'm really asking for a textbook with examples. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Installing Gentoo
