No such file or directory [SOLVED]

[lefsha]

Frankly, I am pretty tired to see that message over and over again, when I try to compile what ever mostly bulky packages.

I was blaming concurrent compilation or multiprocessor (multi-thread) archiver. Despite it take almost forever
I was trying to get rid of all of them, but nothing has helped me.

I don't remember when it all has began. I have tested my file system and I cannot see anything wrong with it.

Here is the example:

Code: Select all

cc1plus: fatal error: /var/tmp/portage/sys-devel/gcc-13.2.1_p20230826/work/gcc-13-20230826/gcc/lower-subreg.cc: No such file or directory

That file indeed doesn't exist. /var/tmp/portage is a temporal dir mounted on tmpfs where all builds take place.

Almost certain, when I recompile gcc again that file will be present and everything will be OK.
If not, I have to repeat it again and it will be build successfully, just because it is already compiled before and used as system compiler.

It has nothing to do with gcc, not even exact version of it. And it has nothing to do with the hardware, because all systems I have
demonstrate the same behavior.

It has all began with extremely long compiling time by gcc, not clang, but only gcc. It could take days for gcc to build up gcc or something heavier,
because it had dropped multiple jobs for just 1 and that job was doing nothing 99% of the time.

Now it seems like the issue has gone, but instead I got that crazy behavior of missing files.

Funny enough if I compile anything outside of portage, by using own script I do never experience that issue.
I do use the same archivers, the same compilers, but when compiling fails it doesn't fail because of missing file.

I cannot even file a bug, because I don't understand WHAT I should write there and WHAT application to blame.

Maybe some one else had that issue and can comment on it.

As mentioned above the workaround is dead simple - recompile the sources again it most of the time it will work. If not do that 2 times.

I have tested my DRAM by running long 10+ hours memory test. No error. FS, have been tested too, but actually it all
happens in memory - tmpfs and it happens on 4 separate systems of different age. One of them uses musl instead of glibc.

No other issues while running programs were observed.


Thx

[Zucca]

I guess the build log is so big that it doesn't fit into a regular pastebin? If so you could compress the log and encode it with base64 so it may then fit.

Anyways without seeing any logs I have one wild guess: you've ran out of inodes of your tmpfs mount. It's not likely, but could lead into missing files.

Can you paste the output of df -hi /var/tmp/portage ? Replace the path with your portage temp directory if it differs.

[lefsha]

Thanks. I will check that possibility. Sounds like an option.
I would just guess IF number of inodes runs out the FS should stop working in a way no more other files could be created.

The first stage is source unpacking. That is the time when a certain file can be created or not created
once number of inodes runs out.

The next stage is compilation. That process creates numerous object files. It takes usually quite long
till I see the error of some file is not available. During that time huge number of other files are created.

But usually it is true I try to clean-up the /var/tmp/portage before second try and packages which cannot
be successfully build within emerge -e @world process can be build separately. Something prevents them
to be build together with others.

[Hu]

If the filesystem runs out of inodes, it returns ENOSPC until some files are deleted. Any application which detects errors should notice this and report a failure.

[lefsha]

If the filesystem runs out of inodes, it returns ENOSPC until some files are deleted. Any application which detects errors should notice this and report a failure.

Exactly. It should then happen during the unpacking stage, cause that missing file should be created at that time.
It does mean the archiver doesn't care IF certain file is created or not. It cannot disappear once it was successfully
created.

If let say I would have a problem with *.o files during the compilation running out of inodes would be the explanation.

I hope not only me using tmfs for compilation. But looks like no one has faced that issue. Otherwise it would be already reported.

There must be something else. Something similar, but different.

[CaptainBlood]

Any good reason not to temporary try to workaround with:

Code: Select all

cat /etc/portage/package.env/sys-devel\:\:gcc.conf 
sys-devel/gcc no-tmpfs.conf

Code: Select all

cat /etc/portage/env/no-tmpfs.conf 
PORTAGE_TMPDIR="/var/no-tmpfs"

Note

Code: Select all

mkdir /var/no-tmpfs

is required.
Thks 4 ur attention, interest & support

[Zucca]

@lefsha, yes. It's very unlikely that the problem is caused by lack of inodes. For example I have 858k inodes on my tmpfs which portage uses and I haven't adjusted it. How much you have?

Also try to catch the build log when this happens and pastebin it. Without it, we're only making these wild guesses and cannot really help.

[Hu]

I use a tmpfs for all packages small enough to fit in the tmpfs I can afford to give them. I once ran out of inodes when some failed builds were not cleaned promptly, and the failure mode was very obvious that ENOSPC was the cause.

[lefsha]

I guess I have solved that issue!

Yes, I do use kind of tmpfs, but I am very stingy on memory, therefore I am using zram, which requires
to select file system in memory to make it compressed with zstd in my case.

My stinginess (does such thing exist? ) brought me to use ext2 as a corresponding file system.

I had something like that in my config:

Code: Select all

type2=/var/tmp
flag2=ext2
size2=57344
opts2="noatime,nosuid,nodev"
mode2=777
maxs2=1
algo2=zstd
labl2=var_tmp

The code size of ext2 is much smaller, than ext4. Because I don't use ext4 at all I was thinking, that ext2 would be fully enough to me.
Long time everything was fine. But once the problem with missing files has began. It was sporadic and w/o any logic.
Besides if you compile a large package or many in parallel it can happen.

The limit of inodes numbers is 100% not the issue. I did some cheating on that issue.
During the compilation of the large package which I know WILL fail I have manually from the different tab in mate-terminal just using mc (is not forbidden to use? - yes - I'm old)
just copied the content of the archive into the work folder... and it did work. Few missing files were added and compilation went well! (sic!)

I don't know why, but I did look at the config above and thought how about replacing ext2 with ext4... I hope God won't kill me for wasting more kB inside the kernel...
So I changed the config by replacing ext2 with ext4 and the problem has gone... tadam...

Likely ext2 was not updated for years and became incompatible in a strange way for new kernels.

I didn't check further WHAT exactly is wrong with ext2, but for now it works. I was able to rebuild the world w/o errors... Trust me, that is better, than... well you know...

P.S. Any better alternative for file system in memory vs ext4?

[Zucca]

ext2 as a corresponding file system.

Then there should be, at least some, traces on dmesg if it was ext2 issue.

[lefsha]

True. The issue has appeared again. Or likely never disappeared. It must have something to do with kernel or system config.
I don't know what is the problem. Most likely not hardware, because at least 3 devices have that issue.

[lefsha]

Then there should be, at least some, traces on dmesg if it was ext2 issue.

I have no idea where to look at. If I compile the same package it can go wrong or it can be compiled well
with no change in between.

I had disabled ZRAMFS and used standard ZFS for temporal folder. IDK what is going on.

It never happens even now with manual compilation w/o using portage.

If I simply git clone from github something = anything and do cmake or make it all works or fails,
but never because some file is missing.

[pingtoo]

Your description is a little bit of hard to make sense for me, so I hope you don't mind I ask few questions.

Then there should be, at least some, traces on dmesg if it was ext2 issue.

I have no idea where to look at. If I compile the same package it can go wrong or it can be compiled well
with no change in between.

If there is no changes why did you recompile?

I had disabled ZRAMFS and used standard ZFS for temporal folder. IDK what is going on.

It never happens even now with manual compilation w/o using portage.

Have you compare what were different when doing manually vs using portage? for example is built location different? compiler options different?

If I simply git clone from github something = anything and do cmake or make it all works or fails,
but never because some file is missing.

Missing file(s) symptom usually due to storage system having issue (hardware failure or software (filesystem) ) or sometime it is build system issue due to misconfiguration build options or (rarely but happen) race condition. but without review the entire build environment there not much anyone can help because there are no basis to start with. (here "environment" mean start from the command(s) used start the build sequence + the working environment, So it is not refer to environment variables)

Please also consider there are possibility that there are more than few conditions that cause the unwanted end result. Address one condition may not give wanted result because there are more need to address.

[lefsha]

If there is no changes why did you recompile?

Because it failed to compile. I meant to say, that between 2 compilations
1st failed and 2nd successful or Ns successful I did not change anything besides restarting compilation over and over again.
Thought it was clear.

Have you compare what were different when doing manually vs using portage? for example is built location different? compiler options different?

I have heavily simplified compiling flags and not using anymore LTO and PGO. It is definitely not related to them, simply because it works sometimes.

The mistakes is always happening during unpacking of archives, not during compilation phase. make or gcc simply cannot find some files.
Most of the time headers - *.h sometimes different files. Well, what usually source code archive is made of.

If I check manually - those files are indeed missing. So the compiler is correct blaming they are absent. if I do unpack manually - all files are present.
Only inside of Portage sometimes it is not working. May be python is the issue. Who knows. I wait for some day it will disappear the same way like it came to me.
Something like COVID...

Missing file(s) symptom usually due to storage system having issue (hardware failure or software (filesystem) ) or sometime it is build system issue due to misconfiguration build options or (rarely but happen) race condition. but without review the entire build environment there not much anyone can help because there are no basis to start with. (here "environment" mean start from the command(s) used start the build sequence + the working environment, So it is not refer to environment variables)
Please also consider there are possibility that there are more than few conditions that cause the unwanted end result. Address one condition may not give wanted result because there are more need to address.

That is why I though the problem was ext2. I did it within RAM. Not only 1 hardware unit is affected. Using disk makes it happen more rare, but it does happen anyway.
I do also think the issue is somehow related to race conditions.

That is why I have downgraded from parallel version of zlib, gzip and bzip2 to old single threaded version. It didn't help.

P.S. Recently it was reported, that GCC is broken with march=native, because it cannot build even GLIBC with it. That is real case. Software is getting worse and worse every year.

[pingtoo]

I am still not convince this is related to current Portage.

In your description it "feel" there is "lucky" involve. But we know this cannot be true. Computer is dumb, it cannot do thing randomly. So something is influence the outcome.

One thing I would like to point out that comparing emerge to manual build process, emerge have phase "patch" and "configuration", I mean these phase was executed even before calling package's build configuration script. So we will not able to know if your manual build process is comparable to emerge build process.

Have you try to unpack the "problem" package archive multiple times into different destination (but same file system) and compare? Were there any difference?

Is the "problem" package archive is the same used by emerge?

Have you try to compare manual unpack vs Portage unpack only?

1. for example unpack to /var/tmp/manual
2. /usr/bin/ebuild $(equery w <problem package archive>) unpack
3. directory compare /var/tmp/manual /var/tmp/portage/<proglem package>/work/<package version>

(Note: I could be wrong in the Portage's unpack destination, so please verify)

You may have to do few times in order to capture when the error condition occur, this will help us understand more about this condition.

[lefsha]

I am still not convince this is related to current Portage.

Well, I don't know what is the reason for sure. I just say it doesn't exist outside of portage.

In your description it "feel" there is "lucky" involve. But we know this cannot be true. Computer is dumb, it cannot do thing randomly. So something is influence the outcome.

Wrong. There is NO luck. Where did you get it? Of course computer CAN do things randomly. It can be proven. You simply compile the same package over and over again
and you compare the LOG files. They will be different.

In the most trivial case with a single CPU, which execute instructions one by one in always the same ORDER you will get always exactly the same LOG file.
Since many many years NO CPU executes code in the single thread in the exact the same sequence.

One thing I would like to point out that comparing emerge to manual build process, emerge have phase "patch" and "configuration", I mean these phase was executed even before calling package's build configuration script. So we will not able to know if your manual build process is comparable to emerge build process.

Well, manual means: "git clone, make, make install". The unpacking part is absent. The archive usually has all missing files, but working dir doesn't.
If phase "patch" can remove certain files, then obviously it is responsible for the fault. But because it is random there is no such a patch which is removing some files.

My build process doesn't exist. It all goes according to Makefile or CMakeLists.txt and it is mostly irrelevant in that case. It is not the stage when it goes wrong.

Have you try to unpack the "problem" package archive multiple times into different destination (but same file system) and compare? Were there any difference?

There is no problem package. It can be any package. Large size packages are more often affected, than others.

What I did is unpacking the source separately in some dir. Then start the emerge. Let it start do the job and then manually copy unpacked elsewhere archive to the working dir
with mc and avoid changing existing files just add those which are not existing - missing. That helps really. Doing so I was able to ensure, the successful compilation IF
those missing files were not Makefile or the like. Usually header files were missing. Using that trick helps to solve the issue.

Is the "problem" package archive is the same used by emerge?

Well, no. I do "my standard" directly from git repo. I never use the standard portage archives simply because it doesn't make sense.
If ebuild does exist and package archive does exist too, then there is no point of using custom compilation using the same source.

I am using git sources in 2 cases. 1st - if ebuild is missing and if ebuild is not able to build the package. Quite often the source from git does the better job.
But as you can see manual unpacking almost always helps. But that is a time game. I should copy missing files into working dir before it will be required by the building process,
but after all other steps are done. Works best with large packages and long compilation time.

Have you try to compare manual unpack vs Portage unpack only?

Yes of course. See above. Sometimes there is a difference sometimes not. That is exactly the point where it fails. I don't know why. I don't even understand why it could be that way.
But that is proven. If you don't compare unpacking process upfront you will never know before it fails to compile.

But unpacking by portage and manually some files are missing some are different. Not always like that, because I cannot compare all archives.

You may have to do few times in order to capture when the error condition occur, this will help us understand more about this condition.

There is no obvious or regular condition for that. It happens more frequently if multiple packages are build at the same time or if using single thread I try to use all CPU's cores.

[logrusx]

Without using any kind of tmpf and using ext4, I run into similar issue yesterday with firefox, being unable to rm some firefox-bin directory. At first I thought it might be a mistake in the ebuild, but I didn't see any reference to such a command whatsoever. I thought it might be because of a terminated and resumed ebuild, but that might actually have been the same case as yours. After I resumed the build, it failed on another file, so I decided I'll keep the current version of firefox, but them I realized I hadn't synced portage tree and I did that hoping a new ebuild will be available and let it work and firefox compiled successfully without encountering any of the previous errors.

After a second attempt it succeeded, that's why I attributed it to a terminated and then resumed build.

Unfortunately I don't have good enough enough recollection of the timeline to go through my systemd logs and look for something specific. Grepping for filesystem, inode, ext4, kernel didn't yield anything. I'll keep an eye on this issue.

EDIT: one thing I recalled is it happened during the install phase.

Best Regards,
Georgi