Init doesn't start after migration to new disk

Message

nitm · Post by **nitm** » Sat Jan 03, 2015 1:27 pm

Hi,

I'm trying to migrate my old system from a hard disk to my new ssd.
I've copied the data using 'rsync -avHhx --progress /old /new' as I've done many times in the past.
And now when I try to boot from the new disk the boot process stops at 'Freeing memory blabla'.

The disk and the root partition are found correctly.
The root partition is mounted readonly.
And I'm able to boot the system if I use init=/bin/bash.
So I think that the init process doesn't start for some unknown reason.

I have no idea what is going wrong with my new system.

This is my old fstab:

Code: Select all

/dev/sda1               /boot           ext3            noauto,noatime          1 2
/dev/sda6               /               ext4            noatime         0 1
/dev/sda5               none            swap            sw              0 0
/dev/sda7               /usr/portage    ext4            noatime         0 2
/dev/sda10              /var            ext4            noatime         0 2
/dev/sda9               /home           ext4            noatime         0 2

And this is the new one:

Code: Select all

/dev/sdb1               /boot           ext3            noauto,relatime                 1 2
/dev/sdb5               /               ext4            defaults,relatime,discard       0 1
/dev/sdb2               none            swap            sw,discard                      0 0
/dev/sdb6               /home           ext4            defaults,relatime,discard       0 2

And the output of fdisk:

Code: Select all

# fdisk -l /dev/sda /dev/sdb

Disk /dev/sda: 596.2 GiB, 640135028736 bytes, 1250263728 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x4830482f

Device     Boot     Start       End   Sectors   Size Id Type
/dev/sda1              63    192779    192717  94.1M 83 Linux
/dev/sda2          194560  61634559  61440000  29.3G  b W95 FAT32
/dev/sda3  *     61634560 140150783  78516224  37.5G  7 HPFS/NTFS/exFAT
/dev/sda4       140151060 618807734 478656675 228.2G  5 Extended
/dev/sda5       140151123 148151429   8000307   3.8G 82 Linux swap / Solaris
/dev/sda6       148151493 188153279  40001787  19.1G 83 Linux
/dev/sda7       188153343 193037039   4883697   2.3G 83 Linux
/dev/sda8       193037103 202804559   9767457   4.7G 83 Linux
/dev/sda9       202804623 602806994 400002372 190.8G 83 Linux
/dev/sda10      602807058 618807734  16000677   7.6G 83 Linux

Disk /dev/sdb: 477 GiB, 512110190592 bytes, 1000215216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x2952c192

Device     Boot     Start        End   Sectors   Size Id Type
/dev/sdb1            2048     206847    204800   100M 83 Linux
/dev/sdb2          206848   16984063  16777216     8G 82 Linux swap / Solaris
/dev/sdb3  *     16984064  184756223 167772160    80G  7 HPFS/NTFS/exFAT
/dev/sdb4       184756224 1000214527 815458304 388.9G  5 Extended
/dev/sdb5       184758272  268644351  83886080    40G 83 Linux
/dev/sdb6       268646400  792934399 524288000   250G 83 Linux

As you can see, I want to remove the partitions for /usr/portage and /var. I've copied the data separately, so they should be in tact.

This is the kernel I'm using.

Code: Select all

 # uname -a
Linux xlad 3.17.2-gentoo #3 SMP PREEMPT Sat Dec 6 21:16:24 EET 2014 x86_64 Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz GenuineIntel GNU/Linux

Any help with this issue will be highly appreciated.

Post by Hu » Sat Jan 03, 2015 4:41 pm

Please post the output of emerge --info and the contents of your new bootloader configuration. You can run the emerge --info from the hard disk if that is easier. We just need to see a few major points, like whether you are using systemd or openrc.

nitm · Post by **nitm** » Sat Jan 03, 2015 5:15 pm

OpenRc, no systemd here

This is the relevant part of the menu.lst, but the grub on the ssd doesn't find the menu.lst file for some reason, so I enter the values manually.

Code: Select all

title Gentoo_3.17.2
root (hd1,0)
kernel (hd1,0)/kern3.17.2 root=/dev/sdb5 rootfstype=ext4

Code: Select all

 # emerge --info
Portage 2.2.15 (python 2.7.9-final-0, default/linux/amd64/13.0/desktop, gcc-4.8.4, glibc-2.17, 3.17.2-gentoo x86_64)
=================================================================
System uname: Linux-3.17.2-gentoo-x86_64-Intel-R-_Core-TM-2_Quad_CPU_Q6600_@_2.40GHz-with-gentoo-2.2
KiB Mem:     8185412 total,   2591524 free
KiB Swap:    4000148 total,   4000148 free
Timestamp of tree: Mon, 29 Dec 2014 14:30:01 +0000
sh bash 4.3_p30-r1
ld GNU ld (Gentoo 2.24 p1.4) 2.24
app-shells/bash:          4.3_p30-r1
dev-java/java-config:     2.2.0
dev-lang/perl:            5.20.1-r4
dev-lang/python:          2.7.9-r1
dev-util/cmake:           3.0.2
dev-util/pkgconfig:       0.28-r2
sys-apps/baselayout:      2.2
sys-apps/openrc:          0.13.6
sys-apps/sandbox:         2.6-r1
sys-devel/autoconf:       2.13, 2.69
sys-devel/automake:       1.9.6-r3, 1.11.6, 1.14.1
sys-devel/binutils:       2.24-r3
sys-devel/gcc:            4.1.2, 4.5.4, 4.6.4, 4.7.4, 4.8.4, 4.9.2
sys-devel/gcc-config:     1.8
sys-devel/libtool:        2.4.4
sys-devel/make:           4.1-r1
sys-kernel/linux-headers: 3.17-r1 (virtual/os-headers)
sys-libs/glibc:           2.17
Repositories: gentoo proaudio enlightenment sera obfuscated-local-repo
Installed sets: @cb, @e17, @system
ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=core2 -O2 -fweb -frename-registers -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/themes/oxygen-gtk/gtk-2.0"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=core2 -O2 -fweb -frename-registers -pipe"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--quiet-build=n"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://mirrors.telepoint.bg/gentoo/"
INSTALL_MASK="winemenubuilder.exe winemenubuilder.exe.so"
LC_ALL="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j5"
PKGDIR="/home/portage_overlay/distfiles/"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/var/lib/layman/pro-audio /var/lib/layman/enlightenment /var/lib/layman/sera /home/portage_overlay"
USE="X a52 aac acl acpi alsa amd64 berkdb branding bzip2 cairo cdda cdr cli consolekit cracklib crypt cxx dbus dri dts dvd dvdr dvdread emboss encode exif fam firefox flac fontconfig gdbm gif glamor gpm gtk gtk2 gtk3 iconv jack jpeg lcms libnotify mad midi mmx mng modules mp3 mp4 mpeg multilib ncurses nls nptl ogg opengl openmp pam pango pcre pdf png policykit ppds pulseaudio readline sdl session spell sse sse2 ssl ssse3 startup-notification svg tcpd tiff truetype udev udisks unicode upower usb vdpau vorbis wxwidgets x264 xcb xinerama xml xv xvid zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="keyboard mouse wacom joystick evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="bg en en_GB" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7" RUBY_TARGETS="ruby19 ruby20" SANE_BACKENDS="plustek" USERLAND="GNU" VIDEO_CARDS="nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
USE_PYTHON="2.7"
Unset:  CPPFLAGS, CTARGET, LANG, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, SYNC

Post by **NeddySeagoon** » Sat Jan 03, 2015 6:19 pm

nitm,

I bet its trying to find init on sda5, which is swap, although, root would fail to mount then.

Anyway, a few tests. Be aware that at boot time, greb uses the BIOS hdd detection order, which might not be the same as the kernel.
Even worse, some BIOSes always present the boot disk as hd0. All this uncertainty can make setting up a multi disc system a pain.
We need to track it every step of the way.

First step, boot to the grub menu, then press e. We are goind to edit the in memory copy of grub.conf.
Choose the line

Code: Select all

root (hd1,0)

then erase the last few charaters so it reads

Code: Select all

root (hd1,

Press the tab key. Grub will list all the partitions on whatever it sees as (hd1, Count them.
/dev/sda has 10 and /dev/sdb has 6. How do the names tie up between the BIOS and Grub?

Once you have worked that out, you can edit

Code: Select all

kernel (hd1,0)/kern3.17.2 root=/dev/sdb5 rootfstype=ext4

to get you going.

If you have one of these BIOSes that always presents the boot drive as hd0, the kernel may see it as sda too - however, not almays.
Experimenting with the grub editor is harmless. It will either boot or not.

Once it boots, edit the on disk grub.conf.

I take it you installed grub to the SSD?
A dd copy from the original disk will not work

nitm · Post by **nitm** » Sat Jan 03, 2015 6:33 pm

I don't care much about the grub's menu at the moment. I'll fix it once the hdd is out of the system.

I'm entering this manually, as you've guessed, to make the system boot:

Code: Select all

root (hd1,0)
kernel (hd1,0)/kern3.17.2 root=/dev/sdb5 rootfstype=ext4

Also in the log I can see that the kernel finds all sdb partitions correctly.
I see a line like:

Code: Select all

blabla bal  sdb1 sdb2 sdb3 sdb4 <sdb5 sdb6>

I can paste the log if there is a way to store/retrieve it somehow.

The same problem happens if I remove the hdd and adjust all sdbs to be sdas in grub and fstab.

Post by **NeddySeagoon** » Sat Jan 03, 2015 6:42 pm

nitm,

Which drive is the BIOS booting from?

That you have to enter

Code: Select all

kernel (hd1,0)/kern3.17.2 root=/dev/sdb5 rootfstype=ext4

to boot, suggests that the BIOS is seeing drives in the order you expect, provided the file /boot/kern3.17.2 only exists on the SSD. Is that true?

szatox · Post by **szatox** » Sat Jan 03, 2015 7:00 pm

perhaps it would be a good idea to mount root by label or by guid rather than sdXY ?

Also, what happens when you specify /sbin/init as init?

nitm · Post by **nitm** » Sat Jan 03, 2015 10:33 pm

szatox wrote:Also, what happens when you specify /sbin/init as init?

I've considered trying this but I've forgot to do it...Will try in a minute.

About the BIOS and drives:
When there are two drives in the system the boot happens from the hdd (which is hd0).
But as I've said, I've tried to removed the hdd and see what happens and it is the same problem.
I think something is wrong with the data on the ssd, but I don't know what.

Post by **NeddySeagoon** » Sat Jan 03, 2015 11:00 pm

nitm,

So the /boot/grub/grub.conf that is in use is the one on the rotating rust, not the one on the SSD.
We still don't know how kern3.17.2 is enumerating the drives.

Run

Code: Select all

$ sudo blkid
/dev/sda1: UUID="9392926d-6408-6e7a-8663-82834138a597" TYPE="linux_raid_member" PARTUUID="0553caf4-01"
/dev/sda2: UUID="b6633d8e-41ef-4485-9bbe-c4c2d69f4e8c" TYPE="swap" PARTUUID="0553caf4-02"
/dev/sda5: UUID="5e3cadd4-cfd2-665d-9690-1ac76d8f5a5d" TYPE="linux_raid_member" PARTUUID="0553caf4-05"

Find the PARTUUID for your sdb5.

Change the

Code: Select all

root=/dev/sdb5

to root=PARTUUID=<your_PARTUUID> which will remove the ambiguity.
There are no " marks in the root= statement. No initrd is required either, unlike when you use the filesystem UUID.

nitm · Post by **nitm** » Sat Jan 03, 2015 11:08 pm

Using init=/sbin/init fails the same way.

See a screenshot with the output: http://i776.photobucket.com/albums/yy46/fuscated/pics_in_forums/DSC_9501.jpg

Also if I unplug my keyboard and the plug it in the kernel detects it and prints some info about it.
So the kernel has not panicked.

krinn · Post by **krinn** » Sat Jan 03, 2015 11:15 pm

next to freeing memory, userland tool comes into action.
do you have needed kernel parameters for your device manager? (tmpfs and friends)

Code: Select all

[    3.022848] VFS: Mounted root (ext3 filesystem) readonly on device 8:18.
[    3.034015] devtmpfs: mounted <---
[    3.034602] Freeing unused kernel memory: 332k freed
[    5.216381] udevd[237]: starting version 1.9

Post by **NeddySeagoon** » Sat Jan 03, 2015 11:21 pm

nitm,

If init was missing, the kernel would have panicked. That it hasn't, suggests that the console has been redirected.
Thats not likely either as your keyboad plugging/unplugging shows.

How do you log in?
At the console or with a GUI login?

Can you get access winh ssh from another system to pick up some logs?
dmesg would be good.
Also /var/log/Xorg.0.log if you use a GUI login.

They are both too big to fit in a post, so an a pastebin please.

nitm · Post by **nitm** » Sat Jan 03, 2015 11:24 pm

Don't know. As far as I've read udev should mount the devtmpfs. I know there is an automount kernel configuration option, but it is not set.
I know there is a kernel option to tell the kernel to mount devtmpfs, but I've not used it.

krinn: Can you tell me what do you mean by kernel parameters, so I can try them?

The strange thing is that the same system works without problems on the hdd, but fails on the ssd.

krinn · Post by **krinn** » Sat Jan 03, 2015 11:33 pm

CONFIG_CHECK="~BLK_DEV_BSG ~DEVTMPFS ~!IDE ~INOTIFY_USER ~!SYSFS_DEPRECATED ~!SYSFS_DEPRECATED_V2 ~SIGNALFD ~EPOLL ~FHANDLE ~NET"

nitm · Post by **nitm** » Sat Jan 03, 2015 11:39 pm

I don't know if there is an easier method to check them but here is the output from my kernel:

Code: Select all

 $ zcat /proc/config.gz | grep BLK_DEV_BSG 
CONFIG_BLK_DEV_BSG=yobfuscated@xlad ~/ 
# CONFIG_BLK_DEV_BSGLIB is not set
$ zcat /proc/config.gz | grep DEVTMPFS
CONFIG_DEVTMPFS=y
# CONFIG_DEVTMPFS_MOUNT is not set
$ zcat /proc/config.gz | grep IDE
CONFIG_HAVE_IDE=y
# CONFIG_IDE is not set
# CONFIG_INPUT_IDEAPAD_SLIDEBAR is not set
# CONFIG_SND_RIPTIDE is not set
# CONFIG_SND_TRIDENT is not set
# CONFIG_USB_CHIPIDEA is not set
# CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
$ zcat /proc/config.gz | grep INOTIFY_USER
CONFIG_INOTIFY_USER=y
$ zcat /proc/config.gz | grep SYSFS_DEPRECATED
# CONFIG_SYSFS_DEPRECATED is not set
$ zcat /proc/config.gz | grep SYSFS_DEPRECATED_V2
$ zcat /proc/config.gz | grep SIGNALFD
CONFIG_SIGNALFD=y
$ zcat /proc/config.gz | grep EPOLL
CONFIG_EPOLL=y
$ zcat /proc/config.gz | grep FHANDLE
CONFIG_FHANDLE=y
$ zcat /proc/config.gz | grep NET
# CONFIG_NET_NS is not set
CONFIG_NET=y

Post by **NeddySeagoon** » Sat Jan 03, 2015 11:43 pm

nitm,

What happens if you press Ctrl-Alt-F2.
If there is a login prompt waiting for you, does it work?

nitm · Post by **nitm** » Sat Jan 03, 2015 11:53 pm

I think, I've tried it and nothing happens. There is no login on alt-f2.

Post by **NeddySeagoon** » Sun Jan 04, 2015 12:02 am

nitm,

What about ssh ?

What is in /etc/fstab on the SSD. If its the original fstab, /var on sda will get mounted over /var on the SSD.
That will be ugly as files in use just vanish.

krinn · Post by **krinn** » Sun Jan 04, 2015 12:14 am

recent udev/eudev change let them stall for minutes while waiting for firmware loading.
if your udev is stuck at loading a firmware... so "stuck", no error message, while usb keyboard and friends aren't yet init (so no ctrl+alt+f2...)
it appears stupid to suggest that, but i think you should try "wait" to see if udev isn't just playing its "omg ! firmware let me get stuck for nothing"

nitm · Post by **nitm** » Sun Jan 04, 2015 12:34 am

NeddySeagoon wrote:What about ssh ?

I'll try it, but I doubt it will work.

NeddySeagoon wrote:What is in /etc/fstab on the SSD. If its the original fstab, /var on sda will get mounted over /var on the SSD.

I've posted it in a previous post. It is not the same. I've adjusted it to follow the layout of the new disk.

@krinn: I'll try waiting, but why doesn't it happen when booting from the hdd?

krinn · Post by **krinn** » Sun Jan 04, 2015 12:36 am

ah yes, maybe you forget something while copying it?
try mount your / partition somewhere and check /dev in it for console and null presence. here's from my / so you can get the perms.

Code: Select all

ls -l
total 0
crw------- 1 root root 5, 1 16 sept.  2009 console
crw-rw-rw- 1 root root 1, 3 16 sept.  2009 null

nitm · Post by **nitm** » Sun Jan 04, 2015 1:24 am

Hm, the dev in the target system is empty. I've told rsync to stay on the filesystem.
Can I unmount /dev while the system is running just to sync /dev?

nitm · Post by **nitm** » Sun Jan 04, 2015 7:18 am

krinn: It turned out that this is the exact problem I'm having. After I've created these two device files manually, I was able to boot the system just fine.

Thank you all for helping me.
Do I need to mark the topic as solved?

Post by **NeddySeagoon** » Sun Jan 04, 2015 10:24 am

nitm,

You can edit the topic in your original past if you wish.

For completness. If the kernel is allowed to mount DEVTMPFS, /dev can be empty.
If not, they are needed by userspace which will mount DEVTMPFS over the top of your almost empty /dev but it needs those /dev nodes first.