[SOLVED] Broken Portage

[farmer.ro]

All of a sudden i wanted to

Code: Select all

emerge --sync

and got

Code: Select all

emerge --sync
!!! Error loading '/var/cache/edb/mtimedb': [Errno 21] Is a directory: b'/var/cache/edb/mtimedb'
>>> Syncing repository 'gentoo' into '/usr/portage'...
>>> Starting rsync with rsync://81.91.253.252/gentoo-portage...
>>> Checking server timestamp ...
Welcome to starling.gentoo.org / rsync.gentoo.org

Server Address : 81.91.253.252, 2a01:90:200:10::1a
Contact Name   : mirror-admin@gentoo.org
Hardware       : 2 x Intel(R) Xeon(R) CPU E5649 @ 2.53GHz, 3959MB RAM
Sponsor        : Qube Managed Services Limited, Zurich, Switzerland, EU

Please note: common gentoo-netiquette says you should not sync more
than once a day.  Users who abuse the rsync.gentoo.org rotation
may be added to a temporary ban list.

MOTD autogenerated by update-rsync-motd on Wed Dec 16 19:40:44 UTC 2015

receiving incremental file list
timestamp.chk

Number of files: 1 (reg: 1)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 1
Total file size: 32 bytes
Total transferred file size: 32 bytes
Literal data: 32 bytes
Matched data: 0 bytes
File list size: 41
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 104
Total bytes received: 132

sent 104 bytes  received 132 bytes  472.00 bytes/sec
total size is 32  speedup is 0.14

>>>
>>> Timestamps on the server and in the local repository are the same.
>>> Cancelling all further sync action. You are already up to date.
>>>
>>> In order to force sync, remove '/usr/portage/metadata/timestamp.chk'.
>>>

=== Sync completed for gentoo
q: Updating ebuild cache in /usr/portage ... 
q: Finished 38230 entries in 0.114523 seconds
!!! Error loading '/var/cache/edb/mtimedb': [Errno 21] Is a directory: b'/var/cache/edb/mtimedb'

Performing Global Updates
(Could take a couple of minutes if you have a lot of binary packages.)
  .='update pass'  *='binary update'  #='/var/db update'  @='/var/db move'
  s='/var/db SLOT move'  %='binary move'  S='binary SLOT move'
  p='update /etc/portage/package.*'
/usr/portage/profiles/updates/1Q-2011...............................
............................................
/usr/portage/profiles/updates/2Q-2011...............................
.................................................
......................................................................................
......................................................................................
......................................................................................
......................................................................................
......................................................................................
......................................................................................
......................................................................................
......................................................................................
......................................................................................
......................................................................................
......................................................................................
................................................
......................................................................................
......................................................................................
......................................................................................
......................................................................................
......................................................................................
......................................................................................
......................................................................................
/usr/portage/profiles/updates/3Q-2011..........................
/usr/portage/profiles/updates/4Q-2011..............................
...............................................................................
/usr/portage/profiles/updates/1Q-2012.......................
/usr/portage/profiles/updates/2Q-2012........................
/usr/portage/profiles/updates/3Q-2012..............................
.......................
/usr/portage/profiles/updates/4Q-2012..............................
...............................
/usr/portage/profiles/updates/1Q-2013..............................
....................................................................
/usr/portage/profiles/updates/2Q-2013...............
/usr/portage/profiles/updates/3Q-2013............
/usr/portage/profiles/updates/4Q-2013......
/usr/portage/profiles/updates/1Q-2014.......
/usr/portage/profiles/updates/2Q-2014..
/usr/portage/profiles/updates/3Q-2014................
/usr/portage/profiles/updates/4Q-2014.............
/usr/portage/profiles/updates/1Q-2015..............................
.....................................
/usr/portage/profiles/updates/2Q-2015..............................
......................................................................................
......................................................................................
.......................................
/usr/portage/profiles/updates/3Q-2015...........................
/usr/portage/profiles/updates/4Q-2015..............................
...............
/usr/portage/profiles/updates/1Q-2016..............................
.................
/usr/portage/profiles/updates/2Q-2016..............................
..................................
/usr/portage/profiles/updates/3Q-2016................


Traceback (most recent call last):
  File "/usr/lib/python-exec/python3.4/emerge", line 50, in <module>
    retval = emerge_main()
  File "/usr/lib64/python3.4/site-packages/_emerge/main.py", line 1185, in emerge_main
    return run_action(emerge_config)
  File "/usr/lib64/python3.4/site-packages/_emerge/actions.py", line 3109, in run_action
    return action_sync(emerge_config)
  File "/usr/lib64/python3.4/site-packages/_emerge/actions.py", line 1981, in action_sync
    retvals = syncer.auto_sync(options={'return-messages': False})
  File "/usr/lib64/python3.4/site-packages/portage/emaint/modules/sync/sync.py", line 98, in auto_sync
    emaint_opts=options)
  File "/usr/lib64/python3.4/site-packages/portage/emaint/modules/sync/sync.py", line 251, in _sync
    self._do_pkg_moves()
  File "/usr/lib64/python3.4/site-packages/portage/emaint/modules/sync/sync.py", line 266, in _do_pkg_moves
    self.emerge_config.target_config.mtimedb.commit()
  File "/usr/lib64/python3.4/site-packages/portage/util/mtimedb.py", line 125, in commit
    f.close()
  File "/usr/lib64/python3.4/site-packages/portage/util/__init__.py", line 1352, in close
    os.rename(f.name, real_name)
  File "/usr/lib64/python3.4/site-packages/portage/__init__.py", line 250, in __call__
    rval = self._func(*wrapped_args, **wrapped_kwargs)
IsADirectoryError: [Errno 21] Is a directory: b'/var/cache/edb/mtimedb.3802' -> b'/var/cache/edb/mtimedb'

Code: Select all

env-update
!!! Error loading '/var/cache/edb/mtimedb': [Errno 21] Is a directory: b'/var/cache/edb/mtimedb'
>>> Regenerating /etc/ld.so.cache..

i am clueless on what to do now.

Broke some lines to make the forum-layout behave. —[profile=215889]Chiitoo[/profile]

[farmer.ro]

it seems to me that there might be some corruption on the file system, i never had this problem before, and i am looking for a solution, i might have to build Gentoo from scratch again any ideas?

[lexflex]

Hi,

If you think the filesystem is corrupt you should run fsck and check (and possible repair) it.

it seems to me that there might be some corruption on the file system, i never had this problem before, and i am looking for a solution,
i might have to build Gentoo from scratch again

Also, can you check mtimedb , what does this give:

Code: Select all

ls /var/cache/edb/  -all

Alex.

[Yamakuzure]

As it lies in /var/cache, I would think the file can be regenerated, by simply deleting the offending /var/cache/edb/mtimedb and syncing again.

[farmer.ro]

Code: Select all

ls /var/cache/edb/  -all
total 16
drwxr-xr-x 3 root    portage 4096 Aug 25 10:12 .
drwxr-xr-x 4 root    root    4096 Aug 25 10:12 ..
drwxrwsr-x 3 root    portage 4096 Aug 25 10:03 dep
drwxr-xr-x 8 portage portage 4096 Aug 25 08:50 mtimedb
pfffirkwark package.use # cd /var/cache
pfffirkwark cache # ls
edb  ldconfig

about deleting the /var/cache/edb folder was the first thing that came up to me, but it seems that it refuses to delete the folder with any; "rm -rf" "rmdir -p" "rm -r"

Code: Select all

# rm -rf edb
rm: cannot remove 'edb/mtimedb': Is a directory

Code: Select all

# cd /var/cache/edb
# ls
mtimedb
# cd /var/cache/edb/mtimedb
# ls
ls: cannot access 'build-info': Structure needs cleaning
ls: cannot access 'image': Structure needs cleaning
ls: cannot access 'work': Structure needs cleaning
ls: cannot access 'distdir': Structure needs cleaning
build-info  distdir  image  work

still clueless on how to make "emerge --sync" work again

it definitely seems to be the "/var/cache/edb" folder because:

Error when getting information for file '/var/cache/edb/mtimedb/distdir': Structure needs cleaning.

Could it be that my Gentoo Box got malware infected? or just some corrupted data?

when trying to acces the edb folder my computer seems to freak out, the problem is that i can not seem to delete this folder, i will try it in a chroot live cd now.

[farmer.ro]

Code: Select all

Traceback (most recent call last):
  File "/usr/lib/python-exec/python2.7/emerge", line 50, in <module>
    retval = emerge_main()
  File "/usr/lib64/python2.7/site-packages/_emerge/main.py", line 1185, in emerge_main
    return run_action(emerge_config)
  File "/usr/lib64/python2.7/site-packages/_emerge/actions.py", line 3109, in run_action
    return action_sync(emerge_config)
  File "/usr/lib64/python2.7/site-packages/_emerge/actions.py", line 1981, in action_sync
    retvals = syncer.auto_sync(options={'return-messages': False})
  File "/usr/lib64/python2.7/site-packages/portage/emaint/modules/sync/sync.py", line 98, in auto_sync
    emaint_opts=options)
  File "/usr/lib64/python2.7/site-packages/portage/emaint/modules/sync/sync.py", line 251, in _sync
    self._do_pkg_moves()
  File "/usr/lib64/python2.7/site-packages/portage/emaint/modules/sync/sync.py", line 266, in _do_pkg_moves
    self.emerge_config.target_config.mtimedb.commit()
  File "/usr/lib64/python2.7/site-packages/portage/util/mtimedb.py", line 125, in commit
    f.close()
  File "/usr/lib64/python2.7/site-packages/portage/util/__init__.py", line 1352, in close
    os.rename(f.name, real_name)
  File "/usr/lib64/python2.7/site-packages/portage/__init__.py", line 250, in __call__
    rval = self._func(*wrapped_args, **wrapped_kwargs)
OSError: [Errno 21] Is a directory
(chroot) livecd mtimedb # 

output after running "emerge --sync" seems to be a problem with python, anyone have ideas on how to fix ?

i am on Gentoo ~amd64, and i suspect the culprit being python, since i can not:

Code: Select all

 (chroot) livecd # eselect python list
Available Python interpreters, in order of preference:
  [1]   python3.5
  [2]   python2.7
  [3]   python3.4
(chroot) livecd # eselect python set 2
(chroot) livecd # eselect python update
Switching to python3.5
(chroot) livecd # eselect python update
Switching to python3.5
(chroot) livecd # python-updater
bash: python-updater: command not found
(chroot) livecd # python updater
/usr/lib/python-exec/python3.5/python: can't open file 'updater': [Errno 2] No such file or directory

[farmer.ro]

past few days i have been trying to remove openssh and ssh packages, which i also might have removed some of the dependencies.

now i start to think that by removing essential base parts of the system like ssh and its dependencies, i might have broken my python therefor portage seems broken too.

i guess it is best not to remove base parts of the system because it might break your system.

at this stage i do not think it is a bug with the latest ~amd64 python and portage, but rather silly me breaking my system.

[nativemad]

at this stage i do not think it is a bug with the latest ~amd64 python and portage, but rather silly me breaking my system.

I don't think so!
It looks like your FS got screwed... /var/cache/edb/mtimedb should be a file. If it is not, then the python-code in portage fails.
Another thing that looks like an FS problem is that you can not delete the folder that easily!
What you normally can do is rename it! After that you should be able to sync again - but be aware and don't forget the corrupted FS! - There might be other places where you might get into trouble of *any* kind! The safest thing would be backup/rebuild/restore imho.


HTH, cheers

[Genone]

Definitely looks like filesystem corruption. Run fsck to check it, and also dmesg (or whatever systemd is using) for any suspicious messages.

If you're lucky it's just a single corrupted inode for whatever reason, worst case your disc is dying.

[farmer.ro]

i have partitioned my encrypted lvm luks file system thanks to a third party linux guide, could that be the case my file system died? or was it just bad luck?

i did numerous "dd if=/dev/urandom of=/dev/sda" because of my paranoia, did this break the 120 GB Solid State Drive i am using?

Question: How does one get a corrupted file system?

Question: How does one prevent a corrupted file system?

Question: Do i need to buy a new hard drive?

Question: How can i know my Solid State Drive is broken?

*edit* the cable connectors on my Solid State Drive (which is placed sideways inside my computer case) where a bit loose, could this have caused the file system corruption? or did i got trolled by someone who might have wanted to break my system?

[Genone]

i have partitioned my encrypted lvm luks file system thanks to a third party linux guide, could that be the case my file system died? or was it just bad luck?

I've no experience with such setups, but it's generally possible as any component between application and hardware can cause this in theory.

i did numerous "dd if=/dev/urandom of=/dev/sda" because of my paranoia, did this break the 120 GB Solid State Drive i am using?

Unlikely, unless "numerous" means hundreds or thousands of times, or the SSD was already in a bad state.

Question: How does one get a corrupted file system?

Lots of potential reasons: Hardware or firmware failures, kernel or driver bugs, misbehaving or broken software tools, power outages, ....

Question: How does one prevent a corrupted file system?

Well, perfect solution would be to have a 100% reliable power supply, use a RAID 1 setup, keep your filesystem setup simple and only mount the filesystem readonly. That's not very practical though.

Question: Do i need to buy a new hard drive?

Not necessarily. It's quite possible that the cause for the corruption is software. But it could also be hardware, impossible to say without a more detailed analysis.

Question: How can i know my Solid State Drive is broken?

Check system logs and then maybe SMART data.

*edit* the cable connectors on my Solid State Drive (which is placed sideways inside my computer case) where a bit loose, could this have caused the file system corruption?

Quite possible.

[nativemad]

There are multiple scenarios why a filesystem can get corrupted.
Mostly it's because of an immediate shutdown during write because of a powerloss or kerneloops or similar. - Thats why one should cleanly unmount usb devices before unplugging them... To prevent you from these corruptions, you will need a ups and stop using crappy hardware and/or stop doing nasty things with the kernel! -As we all know this will never happen, it is best to keep recent backups!
Another thing might be the cabling, or the disk itself. Have a look at the smartmontools to find such errors. Sometimes it's enough if you dd over a bad block.
Of course also ssds can and will fail one day, but that shouldn't happen after a few dd's over the full disk.
If smatmonctl still fails its tests even after a dd-run over the full disk, then it's time to buy a new drive.

HTH

[farmer.ro]

So if i understand correctly a physical hardware malfunction of the Solid State Drive is possibly not the reason, it is more that it was a software malfunction, in that case i guess bad luck?

i encrypted my partitions with:

Code: Select all

# parted -a optimal /dev/sda
GNU Parted 2.3
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.

(parted)mklabel gpt

(parted)unit mib

(parted)mkpart primary 1 3
(parted)name 1 grub
(parted)set 1 bios_grub on

(parted)mkpart primary 3 131
(parted)name 2 boot
(parted)mkpart primary 131 -1
(parted)name 3 lvm

(parted)set 2 boot on
(parted)q

The only change I made was not to make a swap and root partition but instead make a single partition and named it lvm. Next I've setup LUKS:

Code: Select all

# Load the dm-crypt module (probably not nessesary)
modprobe dm-crypt

# Crypt the partition we named lvm (in my case that would be /dev/sda3)
cryptsetup -c aes-cbc-essiv:sha256 -v luksFormat -s 256 /dev/sda3

# Open the luks volume
cryptsetup luksOpen /dev/sda3 sda3-luks

# Setup a LVM physical volume 
lvm pvcreate /dev/mapper/sda3-luks

# Setup a volume group
lvm vgcreate vg0 /dev/mapper/sda3-luks

After that setup the actual volumes:

Code: Select all

# Fist setup the swap volume
lvm lvcreate -L 2048M vg0
# Then check how many 'extends' are free (Free PE) using lvm vgdisplay
# use that number to use up the rest of the space:
lvm lvcreate -L 109G vg0

For the rest i basically followed the handbook except where you would normaly do stuff for /dev/sda3 (swap) or /dev/sda4 (root) you would now use /dev/mapper/vg0-lvol0 (swap) and /dev/mapper/vg0-lvol1

is that an O.K setup ?

i also had an up time of 2 days on XFCE could that have caused corruption?

did someone pwned my box remotely?

it also could have been loose cable connectors?
i am a bit confused at this point what might have caused the file system corruption, but from what i understand now, its most likely some unlucky software malfunction or loose cable connectors?

i also powered down my box a few times when i could not login; so i pulled the power switch on the back of my box

**edit** i also must mention that XFCE is throwing a lot of errors with keywords like "gobject not found, panel widget error gtk" i did not found the error log, but i just ignored these errors because i assumed xfce was running fine?

[nativemad]

I believe if somebody remotely could control your box, there would be other easier/better targets to annoy you then portages timestamp file.
Every setup should be ok if it boots and unmounts cleanly during shutdown (which should be the case with an lvm root).
Two days uptime on xfce is really nothing and shouldn't harm at all. Aren't reboots just to upgrade kernels and/or hardware!?!

Pulling the powerplug is exactly how one gets corrupted filesystems and in bad cases defective disks (although ssds are much less prone to hw-defects).
It is hard to tell now which caused what!? Did it got stuck at logon because of the filesystem or did the filesystem got corrupt due to the powerloss!?

[Genone]

So if i understand correctly a physical hardware malfunction of the Solid State Drive is possibly not the reason, it is more that it was a software malfunction, in that case i guess bad luck?

No, what we said was that it's unlikely that your dd runs broke the SSD. Hardware can fail for many reasons. Once more, check the systen logs to get started about the actual problem, and/or let fsck check the filesystem. There is no point in guessing or making unbased assumptions.

i also powered down my box a few times when i could not login; so i pulled the power switch on the back of my box

You mean you actually cut the power without shutdown? That's a rather bad idea for a desktop PC.

[farmer.ro]

Thanks for the responses, i will look into file system check software when i got time, and meanwhile hope the corruption wont happen again.