| View previous topic :: View next topic |
| Author |
Message |
pavel.stratil
Tux's lil' helper
Joined: 23 Oct 2005
Posts: 134
|
 Posted: Sun Aug 02, 2015 1:03 pm Post subject: /dev not populated [solved] |
 |
|
Hi all, I'm in dire need of help with the following problem:
- I volunteered to revive an old poorly maintained and mainly unused server, that didnt fire up correctly after power outage
- it showed that mainly unused doesnt mean unused at all and theres a fairly critical mail server sitting on it 
- what probably happened: system had security maintainance some time ago, but we accidently (or halfassedly) pulled some new udev and other sytem packages.
- After logging in into the server, i noticed several problems, all revolving about boot, udev & friends. Here's the thing:
[*] /dev/* basically unpopulated, incl. missing /dev/urandom, /dev/sd*, /dev/md*, etc. Interesgingly, root partition works, even though /dev/md3 doesnt exist.
[*] INIT: Id "c*" respawning too fast: disabled for 5 minutes messages showing up, also PTY allocation request failed on channel 0, when trying to ssh to the box
[*] Tons of errors on boot, services wont start (I assume due to missing devices to mount)
[*] Tons of "you still use x even if it was deprecated long time ago" messages
Here some state information about the machine:
[*] uname -a http://pastebin.ca/raw/3085622
[*] zcat /proc/config http://pastebin.ca/raw/3085624
[*] lspci -vhttp://pastebin.ca/raw/3085625, lspci -n http://pastebin.ca/raw/3085685
[*] dmesg http://pastebin.ca/raw/3085626
[*] cat /etc/fstab http://pastebin.ca/raw/3085629
[*] ls /dev http://pastebin.ca/raw/3085632 (after fresh boot)
[*] ls /dev/mapper http://pastebin.ca/raw/3085634 (after fresh boot)
[*] ls /sys/block http://pastebin.ca/raw/3085638
[*] cat /proc/mdstat http://pastebin.ca/raw/3085641
[*] after renaming /dev/.tmp.md* to /dev/md* :
[*] pvscan http://pastebin.ca/raw/3085647
[*] vgscan http://pastebin.ca/raw/3085650
[*] lvscan http://pastebin.ca/raw/3085651
[*] rc-update http://pastebin.ca/raw/3085694
[*] cat /boot/grub/grub.conf http://pastebin.ca/raw/3085709
[*] cat /boot/grub/device.map http://pastebin.ca/raw/3085708
**1. /dev/* unpopulated**
1.a recreating /dev/urandom with
| Code: | mknod -m 644 /dev/urandom c 1 9
chown root:root /dev/urandom
/etc/init.d/urandom start |
1.b noticed that /dev/md* hide as /dev/.tmp.md, reactivating volumegroume,
| Code: | mv /dev/.tmp.md0 /dev/md0 # /boot
mv /dev/.tmp.md3 /dev/md3 # /
mv /dev/.tmp.md4 /dev/md4 # volumegroup
vgchange -a y # reactivating volumegroup
/etc/init.d/localmount restart # remounting everything ... getting into a semiworking state, alls up and running |
rebooting again however, brings the machine into its sorry state again. Sofar I tried to upgrade the kernel to 4.0.5 but that results into a kernel panic that the old virtual console wont show me, because the developpers intended to make my life a hell by disabling scrolling.
**2. INIT: Id "c*" respawning too fast: disabled for 5 minutes + not getting a ssh connection working**
I don't like the idea of touching /etc/inittab, cause I dont know what I'm doing then, so I don't like the idea of even trying http://www.linuxforums.org/forum/installation/158358-i-need-help-troubleshooting-respawning-too-fast-disabled-5-minutes.html. I also believe, that this has more to do with my unpopulated /dev (on my laptop, i have tons of tty devices, the server's got two). But here I'm at loss on what to do.
...
Anyways, now I'm on an unlucky combo of udev 208-r1 and an anctient kernel. There are right now 3 relevant kernels (see grub.conf paste):
[*] title=Gentoo Linux 2.6.32-gentoo-udev - udev friendly configuration with DEVTMPFS, that complains that it can't find (hd0,2)
[*] title=Gentoo Linux 2.6.32-gentoo-r18 - the only bootable kernel, without DEVTMPFS but with empty /dev that needs to be repopulated manually
[*] title=Gentoo Linux 4-gentoo-r18 - didnt have much luck sofar with upgrading to this (a panic I cant yet scroll to due to lovely remote console)
Looking for ideas how to
[*] improve the dev population problem (on nonudev kernel, i need at least SSH)
[*] boot into udev friendly 2.6.32 kernel
[*] ... once i get my head around the scrolling issue, moving to a recent kernel.
http://kmuto.jp/debian/hcl/index.rhtmlx
_________________
Senbonzakura
Last edited by pavel.stratil on Sat Aug 08, 2015 6:01 pm; edited 1 time in total |
|
| Back to top |
|
 |
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54220
Location: 56N 3W
|
| Posted: Sun Aug 02, 2015 1:34 pm Post subject: |
|
|
pavel.stratil,
There are more urgent problems than /dev
| Code: | Personalities : [linear] [raid0] [raid1]
md1 : active raid1 sda1[1]
136448 blocks [2/1] [_U]
md3 : active raid1 sda3[1]
40009792 blocks [2/1] [_U]
md4 : active raid1 sda4[1]
447233408 blocks [2/1] [_U]
unused devices: <none> |
It appears that sdb has fallen out of all three raid sets.
Indeed, there is no sign of another drive in dmesg.
You need to do a backup nowish or even sooner.
The root partition is mounted due to the root=/dev/md3 in grub.conf. /dev is on the root filesystem, so it can't be read until root is mounted.
| dmesg: | | [ 0.000000] Linux version 2.6.32-gentoo-r18 (root@innocence) (gcc version 4.3.4 ... |
So the kernel was built with gcc-4.3.4
This install is so old that a new install will be faster. If you want the educational experience, do the update ... after you nave fixed the hardware or found some hardware to migrate to.
You can get yor ptys back
| /etc/fstab: | | devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620 0 0 |
as I also have a fully static /dev (no DEVTMPFS in the kernel)
You can turn that into a mount command if you want to do it manually
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
pavel.stratil
Tux's lil' helper
Joined: 23 Oct 2005
Posts: 134
|
| Posted: Sat Aug 08, 2015 6:01 pm Post subject: |
|
|
Thanks lots Neddy, I was so hung up on solving the problem I didn't even notice it until you pointed it out! For some reason my /dev was utterly destroyed, basically /dev/tty* were files, not nodes (eek). In the end I got my terminal back by removing the ttys, making the multiplexer and adding the ttys again.
| Code: | mknod -m 666 /dev/ptmx c 5 2
mknod -m 666 /dev/tty c 5 0
mknod -m 666 /dev/tty0 c 4 0
mknod -m 666 /dev/tty1 c 4 1 |
Here for feature refference of anyone who might need this.
Marking this as solved
_________________
Senbonzakura |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Kernel & Hardware
