fork

[szatox]

He was referring to emerge hogging all resources during an update. AKA the most common cause of Gentoo machines getting hit by the killer.
Basically "make user's applications nice to the system instead of having system forcefully put them back in their place"

[eccerr0r]

correct, this is a userspace problem at least for the most common reason for oom-killer for killing Gentoo users applications.
I suspect portage builds is the #1 consumer of memory and cpu cycles for an average Gentoo user. I don't think I have many other software that uses so many cpu cycles or memory.

And yes I was assuming people knew I meant make = ninja or waf of course :p No, really, it was generic to any build scheduling system. Most of them don't even care about system status and assume the OS will take care of it, while really the userland scheduler knows more about the nature of the jobs than the OS does. However the OS better return accurate status information to the build system to make decisions.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

[Gentoopc]

[Gentoopc]

[eccerr0r]

Of course /proc/meminfo contains latest kernel memory information but you still need to consider how much cache/buffers are really needed to be kept.
But this is not the only part of the equation, knowing your own size/how much you expect to use, not just memory but also cpu and disk i/o are also important.
Kernel reporting of cpu utilization is tricky because if a process doesn't use the whole time slot and something else uses it, how should this be counted? What if the process creates a LWT, should it be counted as 1 or 2 processes? What if it's part of a cgroup, should that be counted?

Even a userland scheduler still has problems knowing exactly what each subprocess will do. Like if the build script requests to compile foo.c, how much memory will it take? It likely won't know either. Plus if it needs to run rust or something that's inherently multithreaded, it should NOT try to schedule more than 1 or 2 jobs at a time as it's unneeded except when dist-rust comes around...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

[Gentoopc]

[pietinger]

[pingtoo]

[szatox]

What are you trying to do by hacking kernel?
If you want to change behavior of some application based on free memory, why not just read it from /proc/meminfo instead of dealing with kernel's internal variables?
BTW, this is also how free gets this data before printing it in your terminal. You can find it running free under strace like this:

[Gentoopc]

[Gentoopc]

[pingtoo]

[eccerr0r]

Not sure why you want hardware memory, really all that matters is how much memory is available at that instant. If you have a firefox process hogging half your memory and your build is hogging the other half, is it a good idea to fork another process that might consume another GiB of RAM even if BIOS said you have 64GiB RAM?
Well in this case if you have swap available, yes it's fine but if not, no. But if you had 2GiB you may be in trouble from the get go.

All of this data is in /proc/meminfo anyway subtracting out kernel memory which you shouldn't bank on anyway.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

[Gentoopc]

[NeddySeagoon]

Gentoopc,

I don't quite follow the logic here.

You allow fork to run.
The new process dynamically allocates a chunk of RAM.
Now you no longer have enough RAM to keep the kernel alive.
The OOM cannot run so the kernel panics.

What did I miss?
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[Gentoopc]

[NeddySeagoon]

Gentoopc,

That's my point expressed in a different way.

How do you know, in advance of the fork, how much RAM the process needs to know if it should be allowed or not?

The OOM is a last ditch thing called by the kernel when it cannot get the RAM it needs to keep working.
The choice it faces, is to kill a user process or panic.

I can see this heading towards a very unkind description of early versions of Windows.
It only ever used half of RAM and kept the other half in case it was needed.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[Gentoopc]

[Gentoopc]

[eccerr0r]

also keep in mind unused ram is wasted ram.

fork() does not use much ram, it's what that program runs in that fork, usually that exec(). If it's just running a shell or something I'd prefer that it just runs that shell instead of what, hanging because there *might* not be enough ram because it might spawn a g++ on a 2GB process?
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

[pingtoo]

So this is really a kernel process scheduling design exercise.

this is my understand of your thinking, A run request (fork) was submitted, but before the actual fork execute, you want to examine current available ram, and decide execution of the fork or postpone it (put in queue). is this is the idea?

First of all, in kernel there is no actual fork() function you can call, the fork(2) is a syscall that kernel expose to user space. The most close to the idea of fork() in kernel is kernel_clone()

You can use si_mem_available() to get available memory at the moment, do your calculation and decide if you let fork to execute. However since the call does not perform any kind of control, there could be race condition that after the call return the actual available is more or last than the return value

[szatox]

[Gentoopc]

[Gentoopc]

[Gentoopc]