Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[Solved] Massive problems (freezing) with eix-sync
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
linuxfluesterer
Tux's lil' helper
Tux's lil' helper


Joined: 06 Jan 2016
Posts: 130
Location: Germany

PostPosted: Wed Nov 07, 2018 1:32 pm    Post subject: [Solved] Massive problems (freezing) with eix-sync Reply with quote

Hallo friends.
Since 3 month about I'm now using and learning about Gentoo. I'm very glad for this opportunity to come on with your help here.
But sadly, since beginning, I have a massive problem, when I want to update my repositories with
Code:
eix-sync

The thing is, that the first part (listing of all sections/atoms) it needs a very long time (20,30 minutes and more) until I receive the lines with hash... or, what is much harder, I do NOT receive such a hash line and the process is not getting finished because my pc box hangs (freezes). Then I can no more strike nay keys with an effect.

And, my pc box is not of small capacity: here is my:
Code:
emerge --info
Portage 2.3.51 (python 3.6.5-final-0, default/linux/amd64/17.0/desktop/plasma/systemd, gcc-7.3.0, glibc-2.27-r6, 4.18.15-gentoo_tuxedo x86_64)
=================================================================
System uname: Linux-4.18.15-gentoo_tuxedo-x86_64-Intel-R-_Core-TM-_i7-7700T_CPU_@_2.90GHz-with-gentoo-2.4.1
KiB Mem:    16099552 total,  13570168 free
KiB Swap:    8392700 total,   8392700 free
Timestamp of repository gentoo: Wed, 07 Nov 2018 12:00:02 +0000
Head commit of repository gentoo: b0a6e480d81788b9e79daa052bf93c0cffa2ce1f
sh bash 4.4_p12
ld GNU ld (Gentoo 2.30 p5) 2.30.0
app-shells/bash:          4.4_p12::gentoo
dev-lang/perl:            5.24.3-r1::gentoo
dev-lang/python:          2.7.15::gentoo, 3.6.5::gentoo
dev-util/cmake:           3.9.6::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.4.1-r2::gentoo
sys-apps/sandbox:         2.13::gentoo
sys-devel/autoconf:       2.13::gentoo, 2.69-r4::gentoo
sys-devel/automake:       1.15.1-r2::gentoo
sys-devel/binutils:       2.30-r4::gentoo
sys-devel/gcc:            7.3.0-r3::gentoo
sys-devel/gcc-config:     1.8-r1::gentoo
sys-devel/libtool:        2.4.6-r3::gentoo
sys-devel/make:           4.2.1-r4::gentoo
sys-kernel/linux-headers: 4.13::gentoo (virtual/os-headers)
sys-libs/glibc:           2.27-r6::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000
    sync-rsync-extra-opts:
    sync-rsync-verify-jobs: 2
    sync-rsync-verify-max-age: 24
    sync-rsync-verify-metamanifest: no

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=broadwell -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=broadwell -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--jobs=8 --load-average=6"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync multilib-strict news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="ftp://ftp.halifax.rwth-aachen.de/gentoo/"
LANG="de_DE.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="de en"
MAKEOPTS="-j8 -l8"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/mnt/tmpfs/portage/"
USE="X a52 aac acl acpi activities aes alsa amd64 avx avx2 berkdb bluetooth branding bzip2 cairo cdda cdr cli crypt cups cxx dbus declarative dri dts dvd dvdr emboss encode exif f16c fam ffmpeg flac fma3 fortran gdbm gif glamor gpm gtk iconv ipv6 jpeg kde kipi kwallet lcms ldap libnotify libtirpc mad mmx mmxext mng mp3 mp4 mpeg multilib ncurses nls nptl ogg opengl openmp pam pango pclmul pcre pdf phonon plasma png policykit popcnt ppds pulseaudio qml qt5 readline sdl seccomp semantic-desktop sockets spell sse sse2 sse3 sse4_1 sse4_2 ssl ssse3 startup-notification svg systemd tcpd tiff truetype udev udisks unicode upower usb vorbis widgets wxwidgets x264 xattr xcb xcomposite xml xv xvid zlib" ABI_X86="64" ALSA_CARDS="hda-intel" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon plan sheets stage words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput keyboard mouse" KERNEL="linux" L10N="de en" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6 php7-1" POSTGRES_TARGETS="postgres9_5 postgres10" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" RUBY_TARGETS="ruby23" USERLAND="GNU" VIDEO_CARDS="intel" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LC_ALL, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS


I have no idea, what could be the reason for. So, sometimes I need to re-sync again, though I read, that it is a bad attitude (against the netiquette of Gentoo providers) to sync more than once a day..
I assume, the downloads are done quickly, but the the syncing process on my pc, the unpacking and checking against the md5 keys happen on my pc box.
If this is the case, then is it correct, when I should simply execute the second step after downloading the repo entries onto my pc to compare my (old) repo against the new repo?
And if so, which ist the correct command to execute this step to compare old and new repo entries?

Any help is appreciated.

-Linuxfluesterer


Last edited by linuxfluesterer on Thu Nov 15, 2018 12:39 pm; edited 1 time in total
Back to top
View user's profile Send private message
linuxfluesterer
Tux's lil' helper
Tux's lil' helper


Joined: 06 Jan 2016
Posts: 130
Location: Germany

PostPosted: Wed Nov 07, 2018 1:55 pm    Post subject: Reply with quote

Update....
In between (since a suggestion in my older thread of October), I changed the value of MAKEOPTS in /etc/portage/make.conf to -j8 -l8

While syncing it is impossible to start actions with key commands. It seems the keys are not accepted, because the system is busy with syncing.
After some time the small clock in taskbar freezes the time also.
Here's my dmesg:https://paste.pound-python.org/show/mb4uZUWEt2hdLKWHSZHd/

-Linuxfluesterer
Back to top
View user's profile Send private message
mike155
Veteran
Veteran


Joined: 17 Sep 2010
Posts: 1540
Location: Frankfurt, Germany

PostPosted: Wed Nov 07, 2018 3:52 pm    Post subject: Reply with quote

1) Please run the 2 commands below and post the output of the time command
Code:
/usr/bin/time emerge --sync
/usr/bin/time eix-update

If you don't have /usr/bin/time on your machine: please install it: 'emerge sys-process/time'.

Below is the output I get:

Code:
emerge --sync:

  12.28user 18.72system 0:41.73elapsed 74%CPU (0avgtext+0avgdata 40872maxresident)k
  0inputs+0outputs (7major+79620minor)pagefaults 0swaps

eix-update:

  12.88user 1.45system 0:20.40elapsed 70%CPU (0avgtext+0avgdata 114776maxresident)k
  0inputs+0outputs (18major+27981minor)pagefaults 0swaps


2) Please post the output of
Code:
df -k
mount | grep "/dev"

3) How often do you run fstrim?
Back to top
View user's profile Send private message
linuxfluesterer
Tux's lil' helper
Tux's lil' helper


Joined: 06 Jan 2016
Posts: 130
Location: Germany

PostPosted: Wed Nov 14, 2018 4:02 pm    Post subject: Reply with quote

mike155 wrote:
1) Please run the 2 commands below and post the output of the time command
Code:
/usr/bin/time emerge --sync
/usr/bin/time eix-update

If you don't have /usr/bin/time on your machine: please install it: 'emerge sys-process/time'.

Below is the output I get:

Code:
emerge --sync:

  12.28user 18.72system 0:41.73elapsed 74%CPU (0avgtext+0avgdata 40872maxresident)k
  0inputs+0outputs (7major+79620minor)pagefaults 0swaps

eix-update:

  12.88user 1.45system 0:20.40elapsed 70%CPU (0avgtext+0avgdata 114776maxresident)k
  0inputs+0outputs (18major+27981minor)pagefaults 0swaps


2) Please post the output of
Code:
df -k
mount | grep "/dev"

3) How often do you run fstrim?


Thank you for reply. Sorry for so late reply, but in last days I was busy with installing a Sabayon system for my friend.
To the first two commands, I cannot serve you because both the commands don't come to an end. While executing, the pc freezes, I can only hard reset.
Here is my result for:
Code:
df -k
Dateisystem    1K-Blöcke  Benutzt Verfügbar Verw% Eingehängt auf
/dev/nvme0n1p4 116501828 51288592  64002560   45% /
devtmpfs         8047888        0   8047888    0% /dev
tmpfs            8049664    75200   7974464    1% /dev/shm
tmpfs            8049664     1180   8048484    1% /run
tmpfs            8049664        0   8049664    0% /sys/fs/cgroup
tmpfs           10485760        0  10485760    0% /mnt/tmpfs
tmpfs            8049664    56828   7992836    1% /tmp
/dev/nvme0n1p4 116501828 51288592  64002560   45% /home
tmpfs            1609932        8   1609924    1% /run/user/1000

and
Code:

mount | grep "/dev"
/dev/nvme0n1p4 on / type btrfs (rw,noatime,ssd,space_cache,subvolid=257,subvol=/@root)
devtmpfs on /dev type devtmpfs (rw,noatime,size=8047888k,nr_inodes=2011972,mode=755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,relatime)
/dev/nvme0n1p4 on /home type btrfs (rw,noatime,ssd,space_cache,subvolid=258,subvol=/@home)


To answer your last question: On both my Gentoo systems (both almost similar, differing in NVME size and Ram only), I haven't made an fstrim since installation yet.

Then today (before I read your post) I was optimistic to try to execute an eix-sync command again (two times) with result of freezing both times :( .
But I have isolated the output of journalctl which is found here: https://paste.pound-python.org/show/XXGgSLprmcpgwPZxZrPS/

I'll give you this for information, but on my other system I will make an fstrim in my evening, when I'm back home.
Hope, my informations could help you (and finally me also).

Thank you for understanding.

-Linuxfluesterer
Back to top
View user's profile Send private message
mike155
Veteran
Veteran


Joined: 17 Sep 2010
Posts: 1540
Location: Frankfurt, Germany

PostPosted: Wed Nov 14, 2018 6:09 pm    Post subject: Reply with quote

Delete everything form your NVME disk that you don't need : snapshots, temporary files, backups, ...

Then run
Code:
fstrim -v /
fstrim -v /home

This will probably solve your problem. In future, you should run fstrim regularly.
Back to top
View user's profile Send private message
linuxfluesterer
Tux's lil' helper
Tux's lil' helper


Joined: 06 Jan 2016
Posts: 130
Location: Germany

PostPosted: Wed Nov 14, 2018 8:29 pm    Post subject: Reply with quote

mike155 wrote:
Delete everything form your NVME disk that you don't need : snapshots, temporary files, backups, ...

Then run
Code:
fstrim -v /
fstrim -v /home

This will probably solve your problem. In future, you should run fstrim regularly.


Thank you for this answer. Before, when I was working with Sabayon on a standard SSD (2.5") I regularly executed
Code:
fstrim -v /
fstrim -v /home
about 2 times a week.

But I did not think about that necessary for (ssd/nvme) media which still have plenty of space.
In between I have made an fstrim on my smaller nvme with 128GBytes and after that the eix-sync command went through and needed about 9 min 30 seconds.
Tomorrow I will upgrade this one and start a trimming of my bigger system to prove the result of my smaller system. I'm positive for that now.

So then, thank you very much. When everything will work fine tomorrow, I will mark this thread (and the other also) as [Solved].

Good evening!

-Linuxfluesterer
Back to top
View user's profile Send private message
linuxfluesterer
Tux's lil' helper
Tux's lil' helper


Joined: 06 Jan 2016
Posts: 130
Location: Germany

PostPosted: Thu Nov 15, 2018 12:39 pm    Post subject: Reply with quote

mike155 wrote:
Delete everything form your NVME disk that you don't need : snapshots, temporary files, backups, ...

Then run
Code:
fstrim -v /
fstrim -v /home

This will probably solve your problem. In future, you should run fstrim regularly.


As I said in my last post, I will follow your suggestions on my bigger system today, what I just am doing. After trimming / and /home also the eix-sync went smoothly and needed just some seconds only (about 21).
Now I'M upgrading with:
Code:
time emerge -auUD --backtrack=30 --verbose --with-bdeps=y @world
which is fast also.
So, the problem seems to be solved.

Thank you very much. I will mark this and the older thread to this topic as [Solved] now.

But in addition too, I will open another thread to the topic: Where (which directory) are the downloaded repo informations?

-Linuxfluesterer
Back to top
View user's profile Send private message
Josef.95
Advocate
Advocate


Joined: 03 Sep 2007
Posts: 3691
Location: Germany

PostPosted: Thu Nov 15, 2018 5:40 pm    Post subject: Reply with quote

For this job, your sys-apps/util-linux[systemd] package provides
fstrim.service
and
fstrim.timer
Code:
systemctl enable fstrim.timer
should help.
Back to top
View user's profile Send private message
linuxfluesterer
Tux's lil' helper
Tux's lil' helper


Joined: 06 Jan 2016
Posts: 130
Location: Germany

PostPosted: Thu Nov 15, 2018 6:40 pm    Post subject: Reply with quote

Josef.95 wrote:
For this job, your sys-apps/util-linux[systemd] package provides
fstrim.service
and
fstrim.timer
Code:
systemctl enable fstrim.timer
should help.


Great! I just started this job. How (where) will I get informed that fstrim has been executed? And, if I want to trim every 3 or 4 days, how can I tell it to the timer? Or is that too often?

-Linuxfluesterer
Back to top
View user's profile Send private message
Tony0945
Advocate
Advocate


Joined: 25 Jul 2006
Posts: 3192
Location: Illinois, USA

PostPosted: Thu Nov 15, 2018 7:23 pm    Post subject: Reply with quote

On my (openrc) SSD box I have a cron job that periodically trims:
Code:
~ # crontab -l|grep fstrim
15 2 * * tue  /sbin/fstrim -va  | logger

It runs at 2:15AM every tuesday. Adjust as required (man 5 crontab)
Once a week seems adequate for me.
Back to top
View user's profile Send private message
Josef.95
Advocate
Advocate


Joined: 03 Sep 2007
Posts: 3691
Location: Germany

PostPosted: Thu Nov 15, 2018 7:34 pm    Post subject: Reply with quote

linuxfluesterer wrote:
How (where) will I get informed that fstrim has been executed?

Hm for example, you can look via
systemctl list-timers
and
systemctl status fstrim.timer
and in journal
journalctl --unit=fstrim.service

linuxfluesterer wrote:
And, if I want to trim every 3 or 4 days, how can I tell it to the timer?
You can write your own service and timer, for example see in https://wiki.gentoo.org/wiki/SSD#systemd_timer
Back to top
View user's profile Send private message
linuxfluesterer
Tux's lil' helper
Tux's lil' helper


Joined: 06 Jan 2016
Posts: 130
Location: Germany

PostPosted: Thu Nov 15, 2018 9:08 pm    Post subject: Reply with quote

Josef.95 wrote:
linuxfluesterer wrote:
How (where) will I get informed that fstrim has been executed?

Hm for example, you can look via
systemctl list-timers
and
systemctl status fstrim.timer
and in journal
journalctl --unit=fstrim.service

linuxfluesterer wrote:
And, if I want to trim every 3 or 4 days, how can I tell it to the timer?
You can write your own service and timer, for example see in https://wiki.gentoo.org/wiki/SSD#systemd_timer


And again: Thank you!

-Linuxfluesterer
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum