[SOLVED] eth0 'does not exist' after a hard shutdown

Message

dahoste · Post by **dahoste** » Fri Jul 06, 2007 3:21 pm

Lightning actually struck my house. My server, which was 'protected' by a UPS, suffered a hard shutdown. Upon rebooting, eth0 and eth1 are not found. Specifically, I get the following message by the init script:

Code: Select all

network interface eth0 does not exist
Please verify hardware or kernel module (driver)

I like to take error messages for their word, so I did exactly that:

- I booted from a liveCD and verified that the machine (including the two ethernet devices) were still functional. There did not appear to be any hardware issues. I was able to bring up both eth0 and eth1 and ping through them.

- I double checked the kernel config, rebuilt the kernel, and booted the new image. No change - devices still not found.

lspci happily reports both ethernet devices.

dmesg shows both eth0 and eth1 as discovered and functional, and there are no associated error messages.

ifconfig shows only lo.

In other words, everything *except* the init script (and therefore ifconfig) is indicating that the ethernet devices are present, functional, and ready to use.

So now I start grasping at straws:

- I changed the kernel config to use modules for the network drivers, instead of being built-in. Build kernel, reboot. Modules show up in 'lsmod' and appear to load fine - no error messages. Modprobe doesn't complain either. eth0 and eth1 still 'do not exist'.

- I upgraded the kernel (I tend to get lazy and lag behind kernel updates), thinking maybe just building from a totally different source directory will make the problem go away. It didn't.

- I rebuild baselayout, since it seems to magically both break and fix things. It did neither.

Keep in mind that all of this is on what was a perfectly functioning server only moments before the crash. Same hardware, same ethernet cards, same kernel, same kernel config. No emerge updates were performed. The server simply suffered a hard shutdown, and then was rebooted. Booting a liveCD seems to indicate that everything in the system is still fully functional.

When I rebooted the first time, obviously I got some journal replays from the filesystem (I'm using resierfs), but that seemed to complete normally and other than the ethernet issue I didn't see any other error messages.

Does anyone have any idea what I can try to just snap the system out of its mistaken conclusion that eth0 and eth1 don't exist? Is it a script related issue? Is there something wrong with /dev that I can easily remedy?

It feels like it's one simple, quick step away from being back to normal, but if I don't discover that one step then I'm going to be forced to completely re-install everything.

Thanks in advance.

(p.s. I can't really post file contents or log details, since I can't get the machine connected to anything, and the machine is my firewall/proxy/dhcp/dns/everything server. I'm typing this on my laptop plugged directly into my verizon box. But it's not a config problem. Same config, same hardware, only now it doesn't work.)

kevstar31 · Post by **kevstar31** » Fri Jul 06, 2007 4:38 pm

What is the output of: ls /etc/init.d

dahoste · Post by **dahoste** » Fri Jul 06, 2007 4:50 pm

Well, I can't really post it, but net.eth0 and net.eth1 are both there, as symlinks to net.lo, if that's what you're wondering.

The scripts run, and they're the ones that report that each interface 'does not exist'.

dahoste · Post by **dahoste** » Fri Jul 06, 2007 6:27 pm

PROGRESS: turns out that for some reason the system no longer identifies the ethernet devices as eth0 and eth1, but rather eth2 and eth4.

????

What caused that? I'm off to search the forums for something relating to that little tidbit.

dahoste · Post by **dahoste** » Fri Jul 06, 2007 11:17 pm

Ok, this is solved. I'll supply a full-postmortem once I put all my hardware back together and make sure this isn't a phantom fix.

But basically, udev remembers the MAC addr of ethernet devices, and caches them with the ethernet interface name that they were assigned. What I neglected to mention in the first post (to my eternal shame) was that in fact everything about the hardware *wasn't* the same. After the lightning, I suspected that the nic card was fried, so I swapped in another (identical make/model). That was the lynchpin of this whole scenario. The original nic was in fact fried, so my first reboot attempt failed --- ironically, if I remember correctly, with the same message as I subsequently saw on all of the following reboot attempts ('does not exist'). I swapped in a new identical nic, but since I kept seeing the same message, I figured it wasn't the nic!

Like I said -- full details in a follow-up post once the dust settles. I've got fires to put out after a 48 hour downtime.

yuwy · Post by **yuwy** » Sat Jul 07, 2007 11:32 am

hmm yea, someone mentioned the stupidity of the eth interface increment when it should
start back at zero if there are no additional cards.

dahoste · Post by **dahoste** » Sat Jul 07, 2007 2:49 pm

I can see a use for caching the MAC addr to eth mapping, but it seems like that should be opt-in, rather than the default. I've *never* expected a box to call it's first ethernet device anything besides eth0.

mariourk · Post by **mariourk** » Thu Jul 26, 2007 9:09 am

Where does dbus store the cached MAC-addresses?
I've been breaking my head over the same problem for days. Thanks to you
I now know that the only nic in my pc is eth2 instead of eth0. I would like
to reset the whole mess so eth2 becomes eth0 again.

Anyone knows how?

dahoste · Post by **dahoste** » Fri Jul 27, 2007 2:10 am

Code: Select all

/etc/udev/rules.d/70-persistent-net.rules

I think you can just remove the lines in that file to get udev to 'rediscover' the ethernet devices.

After you edit it, I think you can just start udev and then restart net.eth0.