| View previous topic :: View next topic |
| Author |
Message |
Gentoopc
Apprentice
Joined: 25 Dec 2017
Posts: 296
|
 Posted: Sun Oct 01, 2023 11:41 am Post subject: |
 |
|
| Gentoopc wrote: |
English is not my native language. it's hard for me to explain. to put it quite simply, I understand that when fork() generates a process, it will be the root node, and from here the process tree will begin to grow. it's figurative. I understand that this is not static, it's all dynamic, the drawing of the process tree is constantly changing. I want to teach this process tree to grow so that killer() is not needed. this is an easy task, any junior will solve it. what's complicated about that? if there are no hardware resources to create new processes, then why wait for them? to call killer()? this is nonsense,! if it is very difficult to free up resources to create new processes, then it is necessary to teach processes to be created correctly when there are resources for this, if there are none, it is necessary to teach them to wait until they are released. even this approach is better than just producing processes until the computer freezes. therefore, it does not matter which kellir() process is destroyed, the main thing is to destroy and think that this is the standard of the operating system and protect it in every way from attempts to change it. |
|
|
| Back to top |
|
 |
grknight
Retired Dev
Joined: 20 Feb 2015
Posts: 1744
|
| Posted: Sun Oct 01, 2023 12:26 pm Post subject: |
|
|
I will say this simply once more.
fork already fails when there is not enough resources with an error ENOMEM. But this is only ever triggered, in Linux, if the kernel's sysctl vm.overcommit_memory is disabled by setting it to 2.
If you wanted to reserve 5% of RAM that no process can touch, vm.overcommit_ratio=95 with it.
Documented with /usr/src/linux/Documentation/vm/overcommit-accounting.rst
The reason why fork, and malloc, will freely receive allocations now is the default policy says "tell processes there is RAM except if it is a stupidly high request and when it does run out, kill a process to gain some back".
Now programs that check for errors and handle them well can work with this setting. Poorly written programs may die.
This XY problem of changing fork is unlikely to fix anything that isn't already considered. |
|
| Back to top |
|
|
pingtoo
Veteran
Joined: 10 Sep 2021
Posts: 1000
Location: Richmond Hill, Canada
|
| Posted: Sun Oct 01, 2023 12:39 pm Post subject: |
|
|
Gentoopc,
No worries, English also is not my first language.
It is a little bit hard to understand what you mean by "teach processes to be created correctly when there are resources for this". Since process is not really a sentient being, Just like there really is no way to "teach" something like "cup" to "watch for leak".
If you are thinking of create a "monitor" for each "fork"ed process than it is not realistic because this goes against the OS design philosophy. It is the job of kernel to manage available resource so individual process no need to manage on its own.
So far in my response is anything that you believe is contradicted to your logic? Because it is my understand that you intention is trying to modify kernel source code such that you can control how and when a "process" is born. Am I missing something?
Please quote the paragraph(s) you wish for further discuss or else it is hard to guess the context and hard to response without offending. |
|
| Back to top |
|
|
Gentoopc
Apprentice
Joined: 25 Dec 2017
Posts: 296
|
| Posted: Sun Oct 01, 2023 1:42 pm Post subject: |
|
|
| grknight wrote: | I will say this simply once more.
|
I will also repeat to you once again that this approach with killing processes does not suit me. the crooked logic of the kernel cannot be corrected and kill() will not save it. I'm just looking for other ways. not the fact that I will find it, but I'm just trying. I'm just trying to find another way. you can live as you lived with kill(). |
|
| Back to top |
|
|
Gentoopc
Apprentice
Joined: 25 Dec 2017
Posts: 296
|
| Posted: Sun Oct 01, 2023 1:53 pm Post subject: |
|
|
you don't understand me. It is ok. the process itself is not reasonable, but it is possible to build a system in which the birth of processes will not just be hassle-free and stupid, but systematized. I will not control the system of processes, they will just work so that control will not be necessary. I just need everything to work like clockwork. everything in the core should be debugged and honed. then killer() will not be needed. I only see the big picture. the details are still unclear to me. but I know one thing, that if RAM is approaching the critical mark of shortage, then all the functions of the spoiling processes should work in a different mode. but that's what you should understand. |
|
| Back to top |
|
|
grknight
Retired Dev
Joined: 20 Feb 2015
Posts: 1744
|
| Posted: Sun Oct 01, 2023 1:59 pm Post subject: |
|
|
| Gentoopc wrote: | | grknight wrote: | I will say this simply once more.
|
I will also repeat to you once again that this approach with killing processes does not suit me. the crooked logic of the kernel cannot be corrected and kill() will not save it. I'm just looking for other ways. not the fact that I will find it, but I'm just trying. I'm just trying to find another way. you can live as you lived with kill(). |
No you do not understand. vm.overcommit_memory=2 effectively removes kill(). fork() and malloc() fail and there is no kernel kill() |
|
| Back to top |
|
|
pingtoo
Veteran
Joined: 10 Sep 2021
Posts: 1000
Location: Richmond Hill, Canada
|
| Posted: Sun Oct 01, 2023 2:07 pm Post subject: |
|
|
| Gentoopc wrote: | | you don't understand me. It is ok. |
Ok, I wish you well. Sorry not able to be of any help.
| wrote: | | but it is possible to build a system in which the birth of processes will not just be hassle-free and stupid |
Unlikely  due to human nature it is unlikely hassle-free and since it is NOT hassle-free is must be stupid.
| Gentoopc wrote: | | that if RAM is approaching the critical mark of shortage, then all the functions of the spoiling processes should work in a different mode |
Since it is a wishful thinking, May I counter with another wish  , Could it be that OS will produce a runtime environment for each "process" that will never be "RAM shortage"? therefor no need for the killer() function? |
|
| Back to top |
|
|
eccerr0r
Watchman
Joined: 01 Jul 2004
Posts: 9714
Location: almost Mile High in the USA
|
| Posted: Sun Oct 01, 2023 2:46 pm Post subject: |
|
|
The underlying problem is that the OS cannot predict how much RAM each process will eventually need. Should all processes be stopped because it *could* eat 4GB of RAM when there's only 3GB available, including processes not related to the build thread? But it could end up being a 1GB process that would perfectly fit and you'd be utilizing your cpu instead of letting it lie fallow? Note that the build engine, make/waf/ninja also has control of what to and not to submit, not just the kernel.
Swap is still the best general solution of temporary overflows. Processes are localized, they tend to work on a segment of memory at a time and everything else could be swapped to disk. The problem becomes when there's still too much "localized" memory that needs to be worked on, leading to thrashing.
As a Gentoo user we are sort of developers and need to have at least a general understanding of how much memory each build takes, know that if we have rust or qtwebengine show up on emerge.log we have potential problems - and act accordingly.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
| Back to top |
|
|
sublogic
Apprentice
Joined: 21 Mar 2022
Posts: 226
Location: Pennsylvania, USA
|
| Posted: Mon Oct 02, 2023 1:02 am Post subject: |
|
|
| Gentoopc wrote: | | English is not my native language. it's hard for me to explain. to put it quite simply, I understand that when fork() generates a process, it will be the root node, and from here the process tree will begin to grow. |
I think you want to turn set /proc/sys/vm/overcommit_memory to 2, as grknight and others said.
There is a process tree, in the sense that every process has exactly one parent, but the tree structure may not be informative. More importantly, fork() is far from the only mechanism that consumes memory.- After a process calls fork(), there are two copies of that process. The two copies are almost identical but they can tell which one is the original (the parent) and which one is the new process (the child).
- The fork requires little memory because the two processes share all of their program code and most of their data.
- Both processes have the right to change their data. This requires a copy operation to new memory, one page at a time (4kB), so that each process can have a private copy. So after the fork, memory requirements can grow over time as the two processes diverge.
- A very common pattern is for the child process to quickly exec() a new program. If so, there is no need to duplicate pages.
- An exec() may increase or decrease memory requirements, depending on the size of the new program.
- At any time any process can request more memory, typically by calling malloc(). It is impossible to predict ahead of time when and how much.
When a process forks, #2 above, the kernel knows that its memory requirements may grow later from (program)+(data) to (program)+(2*data). By default, it allows the fork even if (program)+(2*data) exceeds available memory, because of the very common pattern of fork() quickly followed by exec(), #4 above.
If an exec() requires more memory than available, the exec fails. A well written program will recover or fail gracefully. A poorly written one may crash. [Aside: or does the kernel allow some leeway ? have to look at the code! ]
If a memory request, #6 above, exceeds available memory, by default the kernel may allow it anyway on the assumption that not all the memory will be actually used.
If you write "2" to /proc/sys/vm/overcommit_memory, the kernel becomes more strict. fork() fails if the kernel can't guarantee (program)+(2*data). exec() fails if the new program is too big. malloc() and similar functions fail if the requested memory cannot be guaranteed ahead of time. Well-written programs will recover or fail gracefully; others may crash. There is nothing the kernel can do about that.
| Quote: | | it's figurative. I understand that this is not static, it's all dynamic, the drawing of the process tree is constantly changing. I want to teach this process tree to grow so that killer() is not needed. this is an easy task, any junior will solve it. what's complicated about that? |
The complication is that the tree can grow by adding new branches, but also by existing branches becoming bigger.
| Quote: | | if there are no iron resources to create new processes, then why wait for them? to call killer()? this is nonsense,! |
Then set overcommit to 2.
| Quote: | | if it is very difficult to free up resources to create new processes, then it is necessary to teach processes to be created correctly when there are resources for this, if there are none, it is necessary to teach them to wait until they are released. |
Set overcommit to 2. fork() won't wait, it will fail. The program is free to wait a bit and try again. If you modify fork() to block and force all callers to wait, you risk deadlocks. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Kernel & Hardware
