Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Any option to disable premerge check?
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2  
Reply to topic    Gentoo Forums Forum Index Installing Gentoo
View previous topic :: View next topic  
Author Message
lefsha
Veteran
Veteran


Joined: 30 Aug 2004
Posts: 1069
Location: Burgas, Bulgaria

PostPosted: Sun Dec 01, 2019 10:27 pm    Post subject: Reply with quote

Hu wrote:
This check is useful for many users, though it is not perfect.


Great and detailed answer Hu! Appreciate it! Unfortunately I also gave my reasons why I do believe
it is not OK. A very detailed explanation see above.

In a very brief form - there is no guarantee whatsoever, that any check which is positive now
will stay that way in the future. Inform user, but don't prevent him doing what he is after.

If some one calls for help without reason, people will usually listen to him once or twice, but
then they will stop it. And if real troubles will come people won't listen anymore.

Setting variable I_KNOW_WHAT_I_AM_DOING is a very very bad decision, but
at the moment observing the process abort is worse for me.

Trying to save my ass portage actually put it on fire, because trying to mitigate
a harmless issue I might fall into a real issue.

Such a thing like process abort is very serious and have to be made ONLY if nothing
else is possible or too late. A brilliant mitigation of process abort is
the check prior installation whether or not existing files belongs to a different package.
If not emerge is proceeding with installation. Just perfect! That is how the system should
behave!

P.S. I am talking very obvious things and frankly can't see why it's not crystal clear to others.
I give up on that.
_________________
Lefsha
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 44193
Location: 56N 3W

PostPosted: Sun Dec 01, 2019 10:36 pm    Post subject: Reply with quote

John R. Graham,
Code:

arm64-build ~ # time emerge -uDUpv @world --keep-going --with-bdeps=y --jobs=6 --exclude="spamdyke" --verbose-conflicts --backtrack=200

real    100m19.185s
user    100m10.000s
sys     0m5.708s

Almost 2 hours :)

lshw won't run on arm64. I did keyword it and try. Here's some of cpuinfo
/proc/cpuinfo:
processor       : 0
BogoMIPS        : 200.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x43
CPU architecture: 8
CPU variant     : 0x1
CPU part        : 0x0a1
CPU revision    : 1
...
processor       : 95
BogoMIPS        : 200.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x43
CPU architecture: 8
CPU variant     : 0x1
CPU part        : 0

_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
lefsha
Veteran
Veteran


Joined: 30 Aug 2004
Posts: 1069
Location: Burgas, Bulgaria

PostPosted: Mon Dec 02, 2019 12:53 am    Post subject: Reply with quote

I tried with RAM. It's even worse...

Code:
\/var/db/repos/gentoo/eclass/kernel-2.eclass: line 328: cannot create temp file for here-document: No space left on device


The same 10Gb available for merge. The check passed. The result is expectable. Exactly what I have said.
_________________
Lefsha
Back to top
View user's profile Send private message
lefsha
Veteran
Veteran


Joined: 30 Aug 2004
Posts: 1069
Location: Burgas, Bulgaria

PostPosted: Mon Dec 02, 2019 1:19 am    Post subject: Reply with quote

NeddySeagoon wrote:
Your logic is not clear because your underlying assumptions are not stated.


Nope. I've made no assumptions. I was asking you to demonstrate the mistake in my explanation.
I see none.

NeddySeagoon wrote:
If you have a single partition for Gentoo then what you say is correct.


No I don't have a single partition. I do have multiple datasets, if you really wish to know it.
But I fail to understand how it is related.

NeddySeagoon wrote:
All processes compete for the same free disk space.


I just have demonstrated the same effect with RAM. No difference. Why should be any
if the logic is broken?

NeddySeagoon wrote:

If you have several partitions that may no longer be true. Lets take my install as a worked example.

When emerge fetches sources, they go to /usr/portage/distfiles.
The build phase happens in /var/tmp/portage.


Great. And what it should change?

Think more careful. I do wish to compile palemoon = 9Gb and firefox = 9Gb and chromium = 9Gb

Portage will do following comparison: 10Gb > 9Gb - OK, 10Gb > 9Gb - OK, 10Gb > 9Gb - OK,

Me, as a human will do following comparison 10Gb < 3*9=27Gb - FAILED.

Surely I do exaggerate, but it's enough to add just more, then 1Gb to make RAM disk full.

And again, please consider, that your configuration is a VERY SPECIAL one.
Portage has to work at ANY configuration. More over I do have standard = default configuration
in that respect, which is available to EVERY user by default. No change required.

NeddySeagoon wrote:

Why do you assume that downloads by you or by portage will eat into build space?
What does your
Code:
df -hT
show?


I don't need to assume that. It's enough to assume, that build take all build space.
No need to download anything. RAM disk breaks the same way. It is much worse - there is no way to increase the available space
Just use --jobs=16 and you are done.


My system layout is in the transition from Arch to Gentoo.
It's quite simple though.

Code:
Filesystem           Type      Size  Used Avail Use% Mounted on
udev                 devtmpfs   10M     0   10M   0% /dev
root                 zfs        19G  9.0G   11G  47% /
tmpfs                tmpfs     2.0G  1.1M  2.0G   1% /run
shm                  tmpfs     9.8G  6.7M  9.8G   1% /dev/shm
root/home            zfs        20G   10G   11G  50% /home
root/srv             zfs        11G  128K   11G   1% /srv
none                 tmpfs     9.8G   20K  9.8G   1% /run/user/1000
none                 tmpfs     9.8G     0  9.8G   0% /run/user/0

_________________
Lefsha
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 10386
Location: Somewhere over Atlanta, Georgia

PostPosted: Mon Dec 02, 2019 6:42 pm    Post subject: Reply with quote

NeddySeagoon wrote:
Code:

arm64-build ~ # time emerge -uDUpv @world --keep-going --with-bdeps=y --jobs=6 --exclude="spamdyke" --verbose-conflicts --backtrack=200

real    100m19.185s
user    100m10.000s
sys     0m5.708s

Almost 2 hours :)
Ah yes: ARM. Should've said modern desktop hardware.

For what it's worth, on my big gun workstation:
Code:
# lshw -short -class cpu
H/W path                     Device      Class          Description
===================================================================
/0/8e                                    processor      Intel(R) Xeon(R) CPU E5-2643 v3 @ 3.40GHz
/0/a2                                    processor      Intel(R) Xeon(R) CPU E5-2643 v3 @ 3.40GHz
~ # time emerge -uDvp --changed-use --with-bdeps=y --backtrack=200 @world

These are the packages that would be merged, in reverse order:

Calculating dependencies... done!

Total: 0 packages, Size of downloads: 0 KiB

real    3m29.461s
user    3m28.772s
sys     0m0.673s
- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 44193
Location: 56N 3W

PostPosted: Mon Dec 02, 2019 6:52 pm    Post subject: Reply with quote

lefsha,

Code:
Filesystem           Type      Size  Used Avail Use% Mounted on
root                 zfs        19G  9.0G   11G  47% /
root/home            zfs        20G   10G   11G  50% /home


You have two filesystems. Anything you download as your normal user goes to root/home and does not interfere with anything portage wants to do.
On your system, portage does all its work in root.


Code:
Filesystem           Type      Size  Used Avail Use% Mounted on
shm                  tmpfs     9.8G  6.7M  9.8G   1% /dev/shm

As you have 9.8G of tmpfs, why do you not build there?
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
lefsha
Veteran
Veteran


Joined: 30 Aug 2004
Posts: 1069
Location: Burgas, Bulgaria

PostPosted: Wed Dec 04, 2019 8:53 am    Post subject: Reply with quote

NeddySeagoon wrote:
As you have 9.8G of tmpfs, why do you not build there?


I do. I have reported above about that. The same issue.
Premerge check is unable to do math like 2+2.

Neddy, independent from configuration I am able to succeed with build
while premerge normally won't let me go and I am able to fail if premerge check
gives the green light in regard to available space.

What is the point in the feature, which doesn't work properly or it can be demonstrated,
that it doesn't work. Any one can do, not just me.

I see 2 reasonable options:
1. Disable not working feature.
2. Develop a new one which really works.
Both are complimentary.

I stop the discussion here. I can't even file the bug, because I see no support from others.
Every one tries to teach me or tell me where I am wrong. I was not asking for that.

I guess the audience some day should understand, that not only newbies are writing here.
And people should not always behave like - we will try to help to clean-up your mess.

Next time I would specifically mention, that this is not help request, but help offer.
Whether it would be taken or not is a different story.
_________________
Lefsha
Back to top
View user's profile Send private message
Goverp
l33t
l33t


Joined: 07 Mar 2007
Posts: 729

PostPosted: Thu Dec 05, 2019 9:51 am    Post subject: Reply with quote

lefsha wrote:

...
Think more careful. I do wish to compile palemoon = 9Gb and firefox = 9Gb and chromium = 9Gb

Portage will do following comparison: 10Gb > 9Gb - OK, 10Gb > 9Gb - OK, 10Gb > 9Gb - OK,

Me, as a human will do following comparison 10Gb < 3*9=27Gb - FAILED.
...

Coming late to this discussion, I'd suggest bypassing the checks is the wrong cure. What's wanted (and I suffer from this lack) is a means to tell portage to run some ebuilds one-at-a-time. For my specific example, I've a 4-core box, and normally run emerge -jobs 3 (sort-of leaving a core for desktop stuff), but qtwebengine, firefox (and an annoyingly increasing number of big applications) really need all the resources to themselves, otherwise the emerge time grows too long and the chances of running out of disk or memory rise too high.

I've wondered if package.env setup specifying "--jobs 1" for some packages would work, but doubt it - I assume portage would only notice after it had already started 3 jobs in parallel... But a package.env setting for "run this emerge by itself" would definitely help.
_________________
Greybeard
Back to top
View user's profile Send private message
Ionen
Tux's lil' helper
Tux's lil' helper


Joined: 06 Dec 2018
Posts: 84

PostPosted: Thu Dec 05, 2019 10:29 am    Post subject: Reply with quote

Goverp wrote:
But a package.env setting for "run this emerge by itself" would definitely help.
It's not as nice but you can always use --exclude "firefox qtwebengine ..." on your --jobs X, then && emerge ... again to build remaining things without --jobs. Could always use simple scripts to automate it. At most it may be annoying if portage wants to rebuild them right away after upgrading other things (exclusions sometimes make dependency checks confused and very slow)
Back to top
View user's profile Send private message
Section_8
Guru
Guru


Joined: 22 May 2004
Posts: 578
Location: Arlington, TX, US

PostPosted: Thu Dec 05, 2019 12:59 pm    Post subject: Reply with quote

Quote:
What's wanted (and I suffer from this lack) is a means to tell portage to run some ebuilds one-at-a-time.
I think you can do this using package.env: https://wiki.gentoo.org/wiki//etc/portage/package.env by setting MAKEOPTS="-j1" for qtwebengine, etc.
Back to top
View user's profile Send private message
lefsha
Veteran
Veteran


Joined: 30 Aug 2004
Posts: 1069
Location: Burgas, Bulgaria

PostPosted: Thu Dec 05, 2019 4:07 pm    Post subject: Reply with quote

Section_8 wrote:
by setting MAKEOPTS="-j1" for qtwebengine, etc.


It's a misunderstanding. --jobs=N and MAKEOPTS=-jN are 2 different beasts!

MAKEOPTS is rather helpful to speed-up emerge of a single package and it should be used
always, besides for some packages, where it fails due to the internal dependencies.
It also won't take more RAM or disk space comparing to:

--jobs=N - which is a parallel build on MANY packages at the same time.

Surely one can combine both.
_________________
Lefsha
Back to top
View user's profile Send private message
lefsha
Veteran
Veteran


Joined: 30 Aug 2004
Posts: 1069
Location: Burgas, Bulgaria

PostPosted: Thu Dec 05, 2019 4:29 pm    Post subject: Reply with quote

Goverp wrote:
I've wondered if package.env setup specifying "--jobs 1" for some packages would work, but doubt it - I assume portage would only notice after it had already started 3 jobs in parallel... But a package.env setting for "run this emerge by itself" would definitely help.


Agreed. That one will make the check working again.
Actually not all ebuilds do have space check. It would be nice automatically to exclude those which require special treatment
and not by specific name, cause that could change over time, but based on ebuild flag. Something like:

Code:
emerge --update --exclude-big --jobs=8 @world
emerge --update --jobs=1 @world


It doesn't look nice, but may be an acceptable workaround.
Or as you say automatically imply --jobs=1 for those where disk space check take place.

I have not seen it before, but in the future we might meet a package, which fails to build with -jN, where N>1
Running a single build with 1 thread is a nightmare. qtwebengine takes more than 24h if building in 1 thread.

Therefore if space is available why not let people run multiple jobs?
Or may be it can be dynamically adjusted where --jobs=N is the maximum allowed number of jobs.
The actual number of jobs will be decided at runtime where a certain package will wait for time slot
when required disk space is available. May be it's the best solution.
_________________
Lefsha
Back to top
View user's profile Send private message
Ionen
Tux's lil' helper
Tux's lil' helper


Joined: 06 Dec 2018
Posts: 84

PostPosted: Thu Dec 05, 2019 4:32 pm    Post subject: Reply with quote

lefsha wrote:
MAKEOPTS is rather helpful to speed-up emerge of a single package and it should be used
always, besides for some packages, where it fails due to the internal dependencies.
It also won't take more RAM or disk space comparing to:

--jobs=N - which is a parallel build on MANY packages at the same time.
Higher -jX takes plenty more ram, some rustc/cc1plus process can go in the 1-3+GB ram usage each, and if you have multiple of them ran at once (because of -jX), it adds up. If you _also_ have --jobs X on top, then it adds up even more.

That aside, biggest advantage of --jobs is that configuration/merge process tend to not use resources fully, and on like 80% of packages you often spend more time on that than the actual compilation if have a decent cpu. Lowering the -jX can help manage resources (except disk space) while still letting those happen concurrently.
Edit: not that I'd go as low as -j1 unless I had only 2GB ram and didn't use distcc
Back to top
View user's profile Send private message
lefsha
Veteran
Veteran


Joined: 30 Aug 2004
Posts: 1069
Location: Burgas, Bulgaria

PostPosted: Thu Dec 05, 2019 4:50 pm    Post subject: Reply with quote

Ionen,

RAM and a disk space even if mapped to RAM are 2 different things.
It has no relation to the above mentioned problem.

No one will use -j1 unless it is required for a given package. Unfortunately there are some, which fail with -jN N>1
If memory is a concern one can remove -pipe from CFLAGS.
_________________
Lefsha
Back to top
View user's profile Send private message
Ionen
Tux's lil' helper
Tux's lil' helper


Joined: 06 Dec 2018
Posts: 84

PostPosted: Thu Dec 05, 2019 4:55 pm    Post subject: Reply with quote

There's been several threads/bug reports of people getting cc1plus processes killed by the kernel due to running out of memory (often with 2-4GB ram) while using like -j3-4. The solution was to lower the -jX (sometimes a low as -j1) to let the package build just fine while not going into swap abuse territory. It's a real issue for some, and removing -pipe isn't always enough.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 44193
Location: 56N 3W

PostPosted: Thu Dec 05, 2019 6:38 pm    Post subject: Reply with quote

-pipe is not a command to gcc, its a request to use RAM to pass intermediate files between phases if the RAM is available.
The -pipe request will not be honoured when RAM is not available.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Installing Gentoo All times are GMT
Goto page Previous  1, 2
Page 2 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum