Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Cross-compiling for an old CPU—chroot or qemu and nfs?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
deterenkelt
n00b
n00b


Joined: 09 Dec 2014
Posts: 9

PostPosted: Tue Dec 09, 2014 12:32 pm    Post subject: Cross-compiling for an old CPU—chroot or qemu and nfs? Reply with quote

I have an old netbook with Intel Atom N270 (1 core, 1.6GHz, +HT) inside. To compile all the packages I use my desktop, where I set up a chroot directory with i686 CHOST. All the configuration is contained in a git repositiory, so the files such make.conf are easy to mirror from one system to another. This works fine for years, but it has two drawbacks:
    - using chroot makes repo syncing a bit more complicated than between two hosts;
    - desktop CPU doesn’t have movbe instruction that Atom has.
So I thought about compiling mounting netbook’s rootfs via NFS in qemu. But this will have another potential drawbacks:
    - the rootfs must not be in use by the OS on it (that is not an issue, I suppose I can alter my initramfs to run an NFS server for this).
    - the link speed netbook offers is only 100 Mbit/sec, i.e. 12 MB/sec which is equal to USB 2.0 (wi-fi is not an option, because netbook is placed stationary in a point where wi-fi signal is not very good).
    - the main issue I see so far is the ccache usage. Considering that I’ll be rebuilding toolchain at least twice at upgrade time if all goes well (to get a new stable toolchain independent from the old one), the kernel (rarely, but possible) and especially -9999 packages, such as mpv that I compile at least once in a month (ffmpeg, too).
So, during complie and installation I/O may be used too much for a 100 Mbit link, do you think it may be an issue so there would be no sense in switching to qemu+nfs setup in terms of speed?

I didn’t see any examples of qemu usage where the rootfs was mounted via NFS outside the host machine—I mean, qemu offers a built-in NFS server and 10.0.0.0/24 network IIRC. Could it be able to connect to an NFS server that is in the same LAN as the host, considering that the VM network interface is bridged with the host’s LAN interface?

About ccache: if the PC ↔ netbook link speed would be too low for ccache usage, will it be possible to somehow mount ccache folder on the host directly via NFS on loopback or local IP? I mean, so the requests to ccache folder were coming directly to the host’s NFS server from the qemu running on this machine? And not like PC (ccache) ↔ netbook ↔ PC (qemu)? I’m just not so sure, how this works…

—————
Don’t offer distcc, it’s useless for the whole @world compilation.
Back to top
View user's profile Send private message
xaviermiller
Bodhisattva
Bodhisattva


Joined: 23 Jul 2004
Posts: 8706
Location: ~Brussels - Belgique

PostPosted: Tue Dec 09, 2014 12:38 pm    Post subject: Reply with quote

crossdev + distcc
_________________
Kind regards,
Xavier Miller
Back to top
View user's profile Send private message
deterenkelt
n00b
n00b


Joined: 09 Dec 2014
Posts: 9

PostPosted: Tue Dec 09, 2014 12:56 pm    Post subject: Reply with quote

xaviermiller wrote:
crossdev + distcc

Did you ever try to cross-compile a thousand packages from @world set via distcc?
Moreover, I don’t think that cross-compiled toolchain will work if I build it with -mmovbe on core-i7-avx
Back to top
View user's profile Send private message
xaviermiller
Bodhisattva
Bodhisattva


Joined: 23 Jul 2004
Posts: 8706
Location: ~Brussels - Belgique

PostPosted: Tue Dec 09, 2014 1:26 pm    Post subject: Reply with quote

Yes, I do that method for a raspberry pi and a netbook (same processor as yours).
It works really well, except for a few packages that cannot use distcc.

The method is:
- install distcc on both machines, set-up distcc server (edit the config file, and start)
- install crossdev, and create the cross environment on the server
- in the host, check the crossdev wiki, you need to replace the cc, gcc, g++ symlinks to a wrapper (otherwile you will get 64 bits binaries from the server)
- add distcc, distcc-pump in the FEATURES on the host
_________________
Kind regards,
Xavier Miller
Back to top
View user's profile Send private message
WWWW
Tux's lil' helper
Tux's lil' helper


Joined: 30 Nov 2014
Posts: 143

PostPosted: Tue Dec 09, 2014 2:18 pm    Post subject: Reply with quote

I want this badly.

Does this work for any target regardless of arch (arm, mips, etc) and bitness (x32, x64)?
Back to top
View user's profile Send private message
xaviermiller
Bodhisattva
Bodhisattva


Joined: 23 Jul 2004
Posts: 8706
Location: ~Brussels - Belgique

PostPosted: Tue Dec 09, 2014 3:04 pm    Post subject: Reply with quote

Yes and yes :)
_________________
Kind regards,
Xavier Miller
Back to top
View user's profile Send private message
deterenkelt
n00b
n00b


Joined: 09 Dec 2014
Posts: 9

PostPosted: Tue Dec 09, 2014 3:57 pm    Post subject: Reply with quote

xaviermiller wrote:
Yes, I do that method for a raspberry pi and a netbook (same processor as yours).
It works really well, except for a few packages that cannot use distcc.

The method is:
- install distcc on both machines, set-up distcc server (edit the config file, and start)
- install crossdev, and create the cross environment on the server
- in the host, check the crossdev wiki, you need to replace the cc, gcc, g++ symlinks to a wrapper (otherwile you will get 64 bits binaries from the server)
- add distcc, distcc-pump in the FEATURES on the host

I know how to set up distcc, right? It is just the last time I tried to cross compile @world via distcc two or three years ago, about 1/3 of all pacakges didn’t compile because of sandbox, and when the sandbox was disabled (though I wasn’t sure how safe it is back then) there were other packages that didn’t want to cross-compile. Tens of them. I was really tired of it and set up the chroot. I wanted to make compilation successful independently of how packages may or may not build well through distcc.

I have a brother that may take the netbook and for him the process of compilation was complicated only by one command—a wrapper for emerge that actually connects to my desktop PC and builds a binary package in chroot. Then he simply emerges it on the netbook using -K. I’m not gonna explain 4 hours for not a programmer, how he can fine-tune package compilation in distcc, what to check and how to finally compile it taking into account all the refinements of cross compilation. He simply wouldn’t listen and would be angry at me, because I’m not able to get it simply working. Yes, he can simply remove distcc from features, but hey, what does the big PC stands for here?

Back then qemu couldn’t offer CPU emulation, so I had to set up chroot. It took some time, but now it guarantees me that compilation will be fast and as successful, as it would be done on the netbook itself. Now I just think whether the setup with qemu and nfs might be simpler and, if possible, faster, or, at least at the same speed considering the I/O. One qemu VM would be a lot easier to set up and maintain than a chroot or especially, cross toolchain.
Back to top
View user's profile Send private message
xaviermiller
Bodhisattva
Bodhisattva


Joined: 23 Jul 2004
Posts: 8706
Location: ~Brussels - Belgique

PostPosted: Tue Dec 09, 2014 4:39 pm    Post subject: Reply with quote

cross-distcc is really useful and stable for years. I don't remember such sandbox problems. Are you sure you don't mix with crossdev ?

cross-distcc = native compilation calling distcc, which is at the serve side a cross-gcc.
_________________
Kind regards,
Xavier Miller
Back to top
View user's profile Send private message
deterenkelt
n00b
n00b


Joined: 09 Dec 2014
Posts: 9

PostPosted: Tue Dec 09, 2014 5:54 pm    Post subject: Reply with quote

xaviermiller wrote:
cross-distcc is really useful and stable for years. I don't remember such sandbox problems. Are you sure you don't mix with crossdev ?

First you advise me
xaviermiller wrote:
crossdev + distcc

Then you say that there is some other ‘cross-distcc’? Excuse me, I don’t get it.

xaviermiller wrote:
cross-distcc = native compilation calling distcc, which is at the serve side a cross-gcc.

I know that distcc itself is not suited for cross compilation, because environment must be brought in accordance with the one of the host for which compilation is being done. This is when crossdev and -march mismatch kick in.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54098
Location: 56N 3W

PostPosted: Tue Dec 09, 2014 8:18 pm    Post subject: Reply with quote

deterenkelt,

I use it for the Raspberry Pi.

There are lots of things that will not build in a pure cross environment. Perl any Python among them.
The problem being that they build some code, then try to run it on the build host in the course of the build.
There are other reasons why things break in a pure cross build too.

With cross distcc. The cross compiler on the helper does less of the work than a pure cross build but this gets round almost all of the issues.
gcc cannot bootstrap with cross distcc.

distcc will not distriubute if the distcc caller is using -march=native so you need to fix that on the N270.
I have one of these netbooks too. I let it update itself about once a year. It takes about a week of 24/7 running.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
deterenkelt
n00b
n00b


Joined: 09 Dec 2014
Posts: 9

PostPosted: Wed Dec 10, 2014 11:29 am    Post subject: Reply with quote

Can somebody who uses qemu and root on nfs answer, please?.. ;_;
Back to top
View user's profile Send private message
xaviermiller
Bodhisattva
Bodhisattva


Joined: 23 Jul 2004
Posts: 8706
Location: ~Brussels - Belgique

PostPosted: Wed Dec 10, 2014 11:37 am    Post subject: Reply with quote

I tried qemu for ARM emulation and a chroot, it was horrible. But it could be better with an Atom emulation.

Or you can compile for a generic i686 architecture, removing the features your powerful CPU don't understand. There you can use a standard chroot, if your kernel has support for 32 bits.
_________________
Kind regards,
Xavier Miller
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54098
Location: 56N 3W

PostPosted: Wed Dec 10, 2014 5:13 pm    Post subject: Reply with quote

deterenkelt,

I tried QEMU for SPARC on a 3.4GHz Phenom.
I got a 100Hz emulation wth one core running flat out emulating the SPARC CPU. As the real system ran at 400Hz it just wasn't worth it.
The killer here is the guest CPU has to be emulated in software.

Thats also what killed the Intel IA64. To run 32 bit x86 software it uses a softmare CPU emulator.
When AMB brought out the AMD64, nobody wanted IA64 any more.

In theory, for Atom on AMD64, you only need to emulate the Atom instructions that the AMD64 doesn't have.
That can both be difficult and slow. QEMU could just pick up the illegal instruction exceptions and deal with them but thats at least two contexet switches per illegal instruction exception.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
szatox
Advocate
Advocate


Joined: 27 Aug 2013
Posts: 3104

PostPosted: Wed Dec 10, 2014 8:32 pm    Post subject: Reply with quote

deterenkelt, first question is if that insctruction supported by atom but not suported by your desktop is used by your toolchain.
If yes... you can consider disabling it globaly, or going for cross compilation as mentioned earlier.

If not (or you decide to disable it), you can go the way I use for my damn-old-laptop. And unlike you with your atom, I do mean it's damn-old. 15 years make it almost a medieval relict.

Anyway, here's the trick:

1) Connect both machines over fast and secure LAN. Your home network, hidden behind NAT will do.
2) on slower machine export / over NFS with no_root_squash option enabled
3) on stronger machine mount weaker's / over NFS to some temporary directory (let's call it $TMP)
4) on stronger machine mount --bind /usr/portage to $TMP/usr/portage
5) on stronger machine mount --bind /var/tmp to $TMP/var/tmp (or better mount ramdisk instead - it's strong machine after all, it can handle compilation in RAM)
mount --bind /proc $TMP/proc
6) chroot $TMP
7) env-update && . /etc/profile && PS1=(chroot)$PS1

and..... you have just provided your atom with you'r desktop's CPU, RAM and even a bit of often-accessed storage. And you don't even have to emerge --sync, because you have done that earlier that day updating your desktop, haven't you?

Oh, remember to umount all stuff before you unplug after update. NFS doesn't like being interrupted and if you mount anything on top of it, you WILL have to reboot to fix the mess
Back to top
View user's profile Send private message
szatox
Advocate
Advocate


Joined: 27 Aug 2013
Posts: 3104

PostPosted: Wed Dec 10, 2014 8:33 pm    Post subject: Reply with quote

szatox wrote:
deterenkelt, first question is if that insctruction supported by atom but not suported by your desktop is used by your toolchain.
If yes... you can consider disabling it globaly, or going for cross compilation as mentioned earlier.

If not (or you decide to disable it), you can go the way I use for my damn-old-laptop. And unlike you with your atom, I do mean it's damn-old. 15 years make it almost a medieval relict.

Anyway, here's the trick:

1) Connect both machines over fast and secure LAN. Your home network, hidden behind NAT will do.
2) on slower machine export / over NFS with no_root_squash option enabled
3) on stronger machine mount weaker's / over NFS to some temporary directory (let's call it $TMP)
4) on stronger machine mount --bind /usr/portage to $TMP/usr/portage
5) on stronger machine mount --bind /var/tmp to $TMP/var/tmp (or better mount ramdisk instead - it's strong machine after all, it can handle compilation in RAM)
mount --bind /proc $TMP/proc
6) chroot $TMP
7) env-update && . /etc/profile && PS1=(chroot)$PS1

and..... you have just provided your atom with you'r desktop's CPU, RAM and even a bit of often-accessed storage. And you don't even have to emerge --sync, because you have done that earlier that day updating your desktop, haven't you?

Oh, remember to umount all stuff before you unplug after update. NFS doesn't like being interrupted and if you mount anything on top of it, you won't be able to fix the mess without rebooting.
Back to top
View user's profile Send private message
deterenkelt
n00b
n00b


Joined: 09 Dec 2014
Posts: 9

PostPosted: Thu Dec 11, 2014 1:55 pm    Post subject: Reply with quote

NeddySeagoon, thanks, that was interesting. It turns out that use of Atom instruction set would be too resource eating. Did you use to mount rootfs with bandwidth speed around 100Mbit ethernet? I’m curoius about how much throughput may it take duriing the @world set compilation (though it’s not much important).
Back to top
View user's profile Send private message
xaviermiller
Bodhisattva
Bodhisattva


Joined: 23 Jul 2004
Posts: 8706
Location: ~Brussels - Belgique

PostPosted: Thu Dec 11, 2014 1:58 pm    Post subject: Reply with quote

with portage temporary files in tmpfs + the cache, there should not have too much network activity.
_________________
Kind regards,
Xavier Miller
Back to top
View user's profile Send private message
deterenkelt
n00b
n00b


Joined: 09 Dec 2014
Posts: 9

PostPosted: Thu Dec 11, 2014 2:02 pm    Post subject: Reply with quote

szatox wrote:
and..... you have just provided your atom with you'r desktop's CPU, RAM and even a bit of often-accessed storage. And you don't even have to emerge --sync, because you have done that earlier that day updating your desktop, haven't you?

I’m not sure, if exporting root via NFS when netbook is still running is a good idea. Couldn’t it bring a potential failure later?.. I see that /proc/ sys and /dev are mounted on the big PC, as well as /var/tmp, so the major threat is gone, but I’m not sure about log filesystem and the other.

In terms of comparison with qemu I see this is better (if I’ll run an NFS server from the initramfs to be sure the rootfs is safe). But I’d insist on ability to not reboot the netbook into initramfs just to build one package in the future.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum