View previous topic :: View next topic |
Author |
Message |
deterenkelt n00b
Joined: 09 Dec 2014 Posts: 9
|
Posted: Tue Dec 09, 2014 12:32 pm Post subject: Cross-compiling for an old CPU—chroot or qemu and nfs? |
|
|
I have an old netbook with Intel Atom N270 (1 core, 1.6GHz, +HT) inside. To compile all the packages I use my desktop, where I set up a chroot directory with i686 CHOST. All the configuration is contained in a git repositiory, so the files such make.conf are easy to mirror from one system to another. This works fine for years, but it has two drawbacks:
- using chroot makes repo syncing a bit more complicated than between two hosts;
- desktop CPU doesn’t have movbe instruction that Atom has. So I thought about compiling mounting netbook’s rootfs via NFS in qemu. But this will have another potential drawbacks:
- the rootfs must not be in use by the OS on it (that is not an issue, I suppose I can alter my initramfs to run an NFS server for this).
- the link speed netbook offers is only 100 Mbit/sec, i.e. 12 MB/sec which is equal to USB 2.0 (wi-fi is not an option, because netbook is placed stationary in a point where wi-fi signal is not very good).
- the main issue I see so far is the ccache usage. Considering that I’ll be rebuilding toolchain at least twice at upgrade time if all goes well (to get a new stable toolchain independent from the old one), the kernel (rarely, but possible) and especially -9999 packages, such as mpv that I compile at least once in a month (ffmpeg, too). So, during complie and installation I/O may be used too much for a 100 Mbit link, do you think it may be an issue so there would be no sense in switching to qemu+nfs setup in terms of speed?
I didn’t see any examples of qemu usage where the rootfs was mounted via NFS outside the host machine—I mean, qemu offers a built-in NFS server and 10.0.0.0/24 network IIRC. Could it be able to connect to an NFS server that is in the same LAN as the host, considering that the VM network interface is bridged with the host’s LAN interface?
About ccache: if the PC ↔ netbook link speed would be too low for ccache usage, will it be possible to somehow mount ccache folder on the host directly via NFS on loopback or local IP? I mean, so the requests to ccache folder were coming directly to the host’s NFS server from the qemu running on this machine? And not like PC (ccache) ↔ netbook ↔ PC (qemu)? I’m just not so sure, how this works…
—————
Don’t offer distcc, it’s useless for the whole @world compilation. |
|
Back to top |
|
|
xaviermiller Bodhisattva
Joined: 23 Jul 2004 Posts: 8706 Location: ~Brussels - Belgique
|
Posted: Tue Dec 09, 2014 12:38 pm Post subject: |
|
|
crossdev + distcc _________________ Kind regards,
Xavier Miller |
|
Back to top |
|
|
deterenkelt n00b
Joined: 09 Dec 2014 Posts: 9
|
Posted: Tue Dec 09, 2014 12:56 pm Post subject: |
|
|
xaviermiller wrote: | crossdev + distcc |
Did you ever try to cross-compile a thousand packages from @world set via distcc?
Moreover, I don’t think that cross-compiled toolchain will work if I build it with -mmovbe on core-i7-avx |
|
Back to top |
|
|
xaviermiller Bodhisattva
Joined: 23 Jul 2004 Posts: 8706 Location: ~Brussels - Belgique
|
Posted: Tue Dec 09, 2014 1:26 pm Post subject: |
|
|
Yes, I do that method for a raspberry pi and a netbook (same processor as yours).
It works really well, except for a few packages that cannot use distcc.
The method is:
- install distcc on both machines, set-up distcc server (edit the config file, and start)
- install crossdev, and create the cross environment on the server
- in the host, check the crossdev wiki, you need to replace the cc, gcc, g++ symlinks to a wrapper (otherwile you will get 64 bits binaries from the server)
- add distcc, distcc-pump in the FEATURES on the host _________________ Kind regards,
Xavier Miller |
|
Back to top |
|
|
WWWW Tux's lil' helper
Joined: 30 Nov 2014 Posts: 143
|
Posted: Tue Dec 09, 2014 2:18 pm Post subject: |
|
|
I want this badly.
Does this work for any target regardless of arch (arm, mips, etc) and bitness (x32, x64)? |
|
Back to top |
|
|
xaviermiller Bodhisattva
Joined: 23 Jul 2004 Posts: 8706 Location: ~Brussels - Belgique
|
Posted: Tue Dec 09, 2014 3:04 pm Post subject: |
|
|
Yes and yes _________________ Kind regards,
Xavier Miller |
|
Back to top |
|
|
deterenkelt n00b
Joined: 09 Dec 2014 Posts: 9
|
Posted: Tue Dec 09, 2014 3:57 pm Post subject: |
|
|
xaviermiller wrote: | Yes, I do that method for a raspberry pi and a netbook (same processor as yours).
It works really well, except for a few packages that cannot use distcc.
The method is:
- install distcc on both machines, set-up distcc server (edit the config file, and start)
- install crossdev, and create the cross environment on the server
- in the host, check the crossdev wiki, you need to replace the cc, gcc, g++ symlinks to a wrapper (otherwile you will get 64 bits binaries from the server)
- add distcc, distcc-pump in the FEATURES on the host |
I know how to set up distcc, right? It is just the last time I tried to cross compile @world via distcc two or three years ago, about 1/3 of all pacakges didn’t compile because of sandbox, and when the sandbox was disabled (though I wasn’t sure how safe it is back then) there were other packages that didn’t want to cross-compile. Tens of them. I was really tired of it and set up the chroot. I wanted to make compilation successful independently of how packages may or may not build well through distcc.
I have a brother that may take the netbook and for him the process of compilation was complicated only by one command—a wrapper for emerge that actually connects to my desktop PC and builds a binary package in chroot. Then he simply emerges it on the netbook using -K. I’m not gonna explain 4 hours for not a programmer, how he can fine-tune package compilation in distcc, what to check and how to finally compile it taking into account all the refinements of cross compilation. He simply wouldn’t listen and would be angry at me, because I’m not able to get it simply working. Yes, he can simply remove distcc from features, but hey, what does the big PC stands for here?
Back then qemu couldn’t offer CPU emulation, so I had to set up chroot. It took some time, but now it guarantees me that compilation will be fast and as successful, as it would be done on the netbook itself. Now I just think whether the setup with qemu and nfs might be simpler and, if possible, faster, or, at least at the same speed considering the I/O. One qemu VM would be a lot easier to set up and maintain than a chroot or especially, cross toolchain. |
|
Back to top |
|
|
xaviermiller Bodhisattva
Joined: 23 Jul 2004 Posts: 8706 Location: ~Brussels - Belgique
|
Posted: Tue Dec 09, 2014 4:39 pm Post subject: |
|
|
cross-distcc is really useful and stable for years. I don't remember such sandbox problems. Are you sure you don't mix with crossdev ?
cross-distcc = native compilation calling distcc, which is at the serve side a cross-gcc. _________________ Kind regards,
Xavier Miller |
|
Back to top |
|
|
deterenkelt n00b
Joined: 09 Dec 2014 Posts: 9
|
Posted: Tue Dec 09, 2014 5:54 pm Post subject: |
|
|
xaviermiller wrote: | cross-distcc is really useful and stable for years. I don't remember such sandbox problems. Are you sure you don't mix with crossdev ? |
First you advise me
xaviermiller wrote: | crossdev + distcc |
Then you say that there is some other ‘cross-distcc’? Excuse me, I don’t get it.
xaviermiller wrote: | cross-distcc = native compilation calling distcc, which is at the serve side a cross-gcc. |
I know that distcc itself is not suited for cross compilation, because environment must be brought in accordance with the one of the host for which compilation is being done. This is when crossdev and -march mismatch kick in. |
|
Back to top |
|
|
NeddySeagoon Administrator
Joined: 05 Jul 2003 Posts: 54098 Location: 56N 3W
|
Posted: Tue Dec 09, 2014 8:18 pm Post subject: |
|
|
deterenkelt,
I use it for the Raspberry Pi.
There are lots of things that will not build in a pure cross environment. Perl any Python among them.
The problem being that they build some code, then try to run it on the build host in the course of the build.
There are other reasons why things break in a pure cross build too.
With cross distcc. The cross compiler on the helper does less of the work than a pure cross build but this gets round almost all of the issues.
gcc cannot bootstrap with cross distcc.
distcc will not distriubute if the distcc caller is using -march=native so you need to fix that on the N270.
I have one of these netbooks too. I let it update itself about once a year. It takes about a week of 24/7 running. _________________ Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
Back to top |
|
|
deterenkelt n00b
Joined: 09 Dec 2014 Posts: 9
|
Posted: Wed Dec 10, 2014 11:29 am Post subject: |
|
|
Can somebody who uses qemu and root on nfs answer, please?.. ;_; |
|
Back to top |
|
|
xaviermiller Bodhisattva
Joined: 23 Jul 2004 Posts: 8706 Location: ~Brussels - Belgique
|
Posted: Wed Dec 10, 2014 11:37 am Post subject: |
|
|
I tried qemu for ARM emulation and a chroot, it was horrible. But it could be better with an Atom emulation.
Or you can compile for a generic i686 architecture, removing the features your powerful CPU don't understand. There you can use a standard chroot, if your kernel has support for 32 bits. _________________ Kind regards,
Xavier Miller |
|
Back to top |
|
|
NeddySeagoon Administrator
Joined: 05 Jul 2003 Posts: 54098 Location: 56N 3W
|
Posted: Wed Dec 10, 2014 5:13 pm Post subject: |
|
|
deterenkelt,
I tried QEMU for SPARC on a 3.4GHz Phenom.
I got a 100Hz emulation wth one core running flat out emulating the SPARC CPU. As the real system ran at 400Hz it just wasn't worth it.
The killer here is the guest CPU has to be emulated in software.
Thats also what killed the Intel IA64. To run 32 bit x86 software it uses a softmare CPU emulator.
When AMB brought out the AMD64, nobody wanted IA64 any more.
In theory, for Atom on AMD64, you only need to emulate the Atom instructions that the AMD64 doesn't have.
That can both be difficult and slow. QEMU could just pick up the illegal instruction exceptions and deal with them but thats at least two contexet switches per illegal instruction exception. _________________ Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
Back to top |
|
|
szatox Advocate
Joined: 27 Aug 2013 Posts: 3104
|
Posted: Wed Dec 10, 2014 8:32 pm Post subject: |
|
|
deterenkelt, first question is if that insctruction supported by atom but not suported by your desktop is used by your toolchain.
If yes... you can consider disabling it globaly, or going for cross compilation as mentioned earlier.
If not (or you decide to disable it), you can go the way I use for my damn-old-laptop. And unlike you with your atom, I do mean it's damn-old. 15 years make it almost a medieval relict.
Anyway, here's the trick:
1) Connect both machines over fast and secure LAN. Your home network, hidden behind NAT will do.
2) on slower machine export / over NFS with no_root_squash option enabled
3) on stronger machine mount weaker's / over NFS to some temporary directory (let's call it $TMP)
4) on stronger machine mount --bind /usr/portage to $TMP/usr/portage
5) on stronger machine mount --bind /var/tmp to $TMP/var/tmp (or better mount ramdisk instead - it's strong machine after all, it can handle compilation in RAM)
mount --bind /proc $TMP/proc
6) chroot $TMP
7) env-update && . /etc/profile && PS1=(chroot)$PS1
and..... you have just provided your atom with you'r desktop's CPU, RAM and even a bit of often-accessed storage. And you don't even have to emerge --sync, because you have done that earlier that day updating your desktop, haven't you?
Oh, remember to umount all stuff before you unplug after update. NFS doesn't like being interrupted and if you mount anything on top of it, you WILL have to reboot to fix the mess |
|
Back to top |
|
|
szatox Advocate
Joined: 27 Aug 2013 Posts: 3104
|
Posted: Wed Dec 10, 2014 8:33 pm Post subject: |
|
|
szatox wrote: | deterenkelt, first question is if that insctruction supported by atom but not suported by your desktop is used by your toolchain.
If yes... you can consider disabling it globaly, or going for cross compilation as mentioned earlier.
If not (or you decide to disable it), you can go the way I use for my damn-old-laptop. And unlike you with your atom, I do mean it's damn-old. 15 years make it almost a medieval relict.
Anyway, here's the trick:
1) Connect both machines over fast and secure LAN. Your home network, hidden behind NAT will do.
2) on slower machine export / over NFS with no_root_squash option enabled
3) on stronger machine mount weaker's / over NFS to some temporary directory (let's call it $TMP)
4) on stronger machine mount --bind /usr/portage to $TMP/usr/portage
5) on stronger machine mount --bind /var/tmp to $TMP/var/tmp (or better mount ramdisk instead - it's strong machine after all, it can handle compilation in RAM)
mount --bind /proc $TMP/proc
6) chroot $TMP
7) env-update && . /etc/profile && PS1=(chroot)$PS1
and..... you have just provided your atom with you'r desktop's CPU, RAM and even a bit of often-accessed storage. And you don't even have to emerge --sync, because you have done that earlier that day updating your desktop, haven't you?
Oh, remember to umount all stuff before you unplug after update. NFS doesn't like being interrupted and if you mount anything on top of it, you won't be able to fix the mess without rebooting. |
|
|
Back to top |
|
|
deterenkelt n00b
Joined: 09 Dec 2014 Posts: 9
|
Posted: Thu Dec 11, 2014 1:55 pm Post subject: |
|
|
NeddySeagoon, thanks, that was interesting. It turns out that use of Atom instruction set would be too resource eating. Did you use to mount rootfs with bandwidth speed around 100Mbit ethernet? I’m curoius about how much throughput may it take duriing the @world set compilation (though it’s not much important). |
|
Back to top |
|
|
xaviermiller Bodhisattva
Joined: 23 Jul 2004 Posts: 8706 Location: ~Brussels - Belgique
|
Posted: Thu Dec 11, 2014 1:58 pm Post subject: |
|
|
with portage temporary files in tmpfs + the cache, there should not have too much network activity. _________________ Kind regards,
Xavier Miller |
|
Back to top |
|
|
deterenkelt n00b
Joined: 09 Dec 2014 Posts: 9
|
Posted: Thu Dec 11, 2014 2:02 pm Post subject: |
|
|
szatox wrote: | and..... you have just provided your atom with you'r desktop's CPU, RAM and even a bit of often-accessed storage. And you don't even have to emerge --sync, because you have done that earlier that day updating your desktop, haven't you? |
I’m not sure, if exporting root via NFS when netbook is still running is a good idea. Couldn’t it bring a potential failure later?.. I see that /proc/ sys and /dev are mounted on the big PC, as well as /var/tmp, so the major threat is gone, but I’m not sure about log filesystem and the other.
In terms of comparison with qemu I see this is better (if I’ll run an NFS server from the initramfs to be sure the rootfs is safe). But I’d insist on ability to not reboot the netbook into initramfs just to build one package in the future. |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|