system unresponsive every 24h for 15 minutes

DanielG · n00b Joined: 07 May 2004 Posts: 18

Hi guys,

I'm somewhat out of ideas with the behavior I'm seeing on my gentoo server.

Every evening there are a few long-running php-fpm instances that cause a complete communication breakdown. This takes approx. 15 minutes, then things return to normal.
As soon as connectivity is back up, my monitoring shows a load1 value of over 180.

According to the logs, the long-running PHP calls originate from the Nextcloud I'm hosting.

Has anyone seen some similar behavior?
Is this a question for PHP, Nextcloud or still gentoo?

If there is any info you could point me to, I'd greatly appreciate it.

Thanks and regards
Daniel
_________________
Loving Gentoo since 2005!

mike155 · Posted: Sun Nov 20, 2022 9:13 pm Post subject:

I think I would collect more data. Either run 'top' during that event. Or, if that doesn't work, run a small script 'moni.sh':

DanielG · n00b Joined: 07 May 2004 Posts: 18

Thanks a lot for your suggestion.

For two days I did now set up such a log writer.
The interesting thing is:
During the "phenomenon", there are no lines written to the /tmp/moni.log file.

Once the server responds again, moni.log grows again...

Due to the PHP-FPM logs I'm going to take this to the NextCloud community.
But if anyone has further ideas, any input is appreciated!

Thanks!
_________________
Loving Gentoo since 2005!

Banana · Veteran Joined: 21 May 2004 Posts: 1392 Location: Germany

any cronjobs or backups running in this timeframe?
_________________
My personal space
My delta-labs.org snippets do expire

PFL - Portage file list - find which package a file or command belongs to.

DanielG · n00b Joined: 07 May 2004 Posts: 18

Hi,

backups are running at 2 am without any issues.
Cron jobs are not running as far as I can see from the output of crontab -l
_________________
Loving Gentoo since 2005!

Banana · Veteran Joined: 21 May 2004 Posts: 1392 Location: Germany

hmm. Maybe Nextcloud itself does have one?
https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/background_jobs_configuration.html
_________________
My personal space
My delta-labs.org snippets do expire

PFL - Portage file list - find which package a file or command belongs to.

ocbMaurice · Posted: Sun Nov 27, 2022 4:19 am Post subject:

Any chance your system is heavily swapping in and out memory when executing those scripts?

DanielG · n00b Joined: 07 May 2004 Posts: 18

So far I did not observe any excessive swapping during the periods in question.
Also I reduced running the Nextcloud cron jobs to once every 15 minutes. This works fine for the whole day and the start of the non-responsiveness does not coincide with an execution of any cron job.

What I however was able to find out: The CPU load is almost 100% iowait while the system isn't responsive.
As my server is a virtual server I contacted the hosting company to check whether any other guest system could be causing any issues here.

Or is there any other thing that could be causing high iowait and would be within the scope of my gentoo installation?
_________________
Loving Gentoo since 2005!

spica · Apprentice Joined: 04 Jun 2021 Posts: 287

100% iowait can be the disk is busy.
If you know the exact time when the problem starts, you can try to check what process consumes I\O using iotop utility (run it before the problem starts)