[solved?] USB dead/resets, 2.6.35 up to .38, 2.6.34 is fine

[TheTuxKeeper]

Hi,

since the first time i tried to upgrade from 2.6.34 to 2.6.35 i had "resets" of all usb ports or even a complete loss of usb with a reboot as only option. I have found too many posts with usb errors and none that fit to mine.
Since i want to try kms and perhaps btrfs, i really want to upgrade to 2.6.38. That's why i'm posting now

What is causing my complete usb to stop working (not only single ports) and how can i fix it? since 2.6.34 is working for several months now, it has to be something that changed in .35. i even read at the changelog on kernel.org, but dind't find a hint or just didn't see it since i'm not a kernel dev

Right now i'm (still) at gentoo-sources-2.6.34-r10. i'm always using gentoo-sources.
More infos on my configuration:
- it's a sony vaio vpc eb2 notebook
- emerge --info output
- dmesg with 2.6.38 (only usb stuff) (it starts at around 60 since the kernel was looking for the kms firmware that wasn't there )
- config from 2.6.38
- config from 2.6.38 only usb options (grep -Fi usb)
- lspci output
- lsusb output

I already tried to disable autosuspend (i think only with 2.6.37), but it didn't change anything:

Code: Select all

for suspend in /sys/bus/usb/devices/[12]*/power/autosuspend; do echo "-1" > $suspend; done
for suspend in /sys/bus/usb/devices/usb*/power/autosuspend; do echo "-1" > $suspend; done

i haven't tried it with no or just one plugged in usb device (will do some more tests later). will also create a new set of logs then, but right now i'm not in the mood to kill my usb

thanks for your help, hope i can upgrade my kernel in near future

[NeddySeagoon]

TheTuxKeeper,

boot your choice of working kernel and post the content of /proc/bus/usb/devices so we can look at your USB device tree,

If you want a couple of stabs in the dark, turn off

Code: Select all

 CONFIG_USB_OHCI_HCD=y

is for nin intel USB1.1 root hubs only. You don't have any of those. Sometimes it prevents

Code: Select all

CONFIG_USB_UHCI_HCD=y

working too.

You probably don't need

Code: Select all

CONFIG_USB_UHCI_HCD=y

as your lspci only lists USB2 root hubs. That probably means it supports Transaction Translation, which you have off.

Code: Select all

# CONFIG_USB_EHCI_ROOT_HUB_TT is not set
# CONFIG_USB_EHCI_TT_NEWSCHED is not set

So, in make menuconfig, turn off both OHCI_HCD and UHCI_HCD support and turn on Transaction Translation.

[TheTuxKeeper]

thanks for the fast answer

here the config change:

Code: Select all

tuxkeeper-gentoo ~ # diff -u kernel-config-x86_64-2.6.38-gentoo-sony1-kms_onlyusb.txt kernel-config-x86_64-2.6.38-gentoo-sony2-kms_onlyusb.txt
--- kernel-config-x86_64-2.6.38-gentoo-sony1-kms_onlyusb.txt    2011-03-29 21:35:26.126552980 +0200
+++ kernel-config-x86_64-2.6.38-gentoo-sony2-kms_onlyusb.txt    2011-03-29 23:06:37.058366785 +0200
@@ -154,17 +154,14 @@
 # CONFIG_USB_C67X00_HCD is not set
 # CONFIG_USB_XHCI_HCD is not set
 CONFIG_USB_EHCI_HCD=y
-# CONFIG_USB_EHCI_ROOT_HUB_TT is not set
+CONFIG_USB_EHCI_ROOT_HUB_TT=y
 # CONFIG_USB_EHCI_TT_NEWSCHED is not set
 # CONFIG_USB_OXU210HP_HCD is not set
 # CONFIG_USB_ISP116X_HCD is not set
 # CONFIG_USB_ISP1760_HCD is not set
 # CONFIG_USB_ISP1362_HCD is not set
-CONFIG_USB_OHCI_HCD=y
-# CONFIG_USB_OHCI_BIG_ENDIAN_DESC is not set
-# CONFIG_USB_OHCI_BIG_ENDIAN_MMIO is not set
-CONFIG_USB_OHCI_LITTLE_ENDIAN=y
-CONFIG_USB_UHCI_HCD=y
+# CONFIG_USB_OHCI_HCD is not set
+# CONFIG_USB_UHCI_HCD is not set
 # CONFIG_USB_SL811_HCD is not set
 # CONFIG_USB_R8A66597_HCD is not set
 # CONFIG_USB_WHCI_HCD is not set

Perhaps i have to cleanup my config. both uhci and ohci enabled are obviously not needed (if you read the help text, but i've already known the difference of those two ). but i didn't know the transaction translation stuff

/proc/bus/usb/devices output

Will rebuild the kernel and check whether there will be still usb resets. hope it will be this simple, just once

edit:
forgot to set CONFIG_USB_EHCI_TT_NEWSCHED, will try that later. right now i booted with the new config and 2.6.38, hope it works longer than a few hours

[NeddySeagoon]

TheTuxKeeper,

Code: Select all

CONFIG_USB_EHCI_TT_NEWSCHED 

is a go faster option, its not essential.

[TheTuxKeeper]

Weren't the uhci/ohci options.

Code: Select all

Mar 30 02:12:52 tuxkeeper-gentoo kernel: [10011.764594] usb 2-1: usb-storage timed out on ep0out len=0/0
Mar 30 02:12:52 tuxkeeper-gentoo kernel: [10011.764604] hub 2-1:1.0: cannot reset port 6 (err = -110)
Mar 30 02:12:52 tuxkeeper-gentoo kernel: [10011.764607] hub 2-1:1.0: port 6 not enabled, trying reset again...
Mar 30 02:12:53 tuxkeeper-gentoo kernel: [10012.772899] usb 2-1: usb-storage timed out on ep0out len=0/0
Mar 30 02:12:53 tuxkeeper-gentoo kernel: [10012.772907] hub 2-1:1.0: cannot reset port 6 (err = -110)
Mar 30 02:12:53 tuxkeeper-gentoo kernel: [10012.772910] hub 2-1:1.0: port 6 not enabled, trying reset again...
Mar 30 02:12:54 tuxkeeper-gentoo kernel: [10013.781210] usb 2-1: usb-storage timed out on ep0out len=0/0
Mar 30 02:12:54 tuxkeeper-gentoo kernel: [10013.781218] hub 2-1:1.0: cannot reset port 6 (err = -110)
Mar 30 02:12:54 tuxkeeper-gentoo kernel: [10013.781221] hub 2-1:1.0: port 6 not enabled, trying reset again...
Mar 30 02:12:55 tuxkeeper-gentoo kernel: [10014.789508] usb 2-1: usb-storage timed out on ep0out len=0/0
Mar 30 02:12:55 tuxkeeper-gentoo kernel: [10014.789515] hub 2-1:1.0: cannot reset port 6 (err = -110)
Mar 30 02:12:55 tuxkeeper-gentoo kernel: [10014.789518] hub 2-1:1.0: port 6 not enabled, trying reset again...
Mar 30 02:12:56 tuxkeeper-gentoo kernel: [10015.797824] usb 2-1: usb-storage timed out on ep0out len=0/0
Mar 30 02:12:56 tuxkeeper-gentoo kernel: [10015.797831] hub 2-1:1.0: cannot reset port 6 (err = -110)
Mar 30 02:12:56 tuxkeeper-gentoo kernel: [10015.797833] hub 2-1:1.0: port 6 not enabled, trying reset again...
Mar 30 02:12:56 tuxkeeper-gentoo kernel: [10015.797835] hub 2-1:1.0: Cannot enable port 6.  Maybe the USB cable is bad?
[...]
Mar 30 02:13:16 tuxkeeper-gentoo kernel: [10035.963892] hub 2-1:1.0: state 7 ports 8 chg 0040 evt 0000
Mar 30 02:13:21 tuxkeeper-gentoo kernel: [10041.005358] hub 2-1:1.0: hub_port_status failed (err = -110)
Mar 30 02:13:17 tuxkeeper-gentoo kernel: [10036.972129] usb 2-1: khubd timed out on ep0in len=0/4
Mar 30 02:13:18 tuxkeeper-gentoo kernel: [10037.980430] usb 2-1: khubd timed out on ep0in len=0/4

But usb works again. seems like it "restarted" itself and it's only "hub 2-1:1.0" or "usb 2-1" that has/causes problems (seems that 2* are my external/visible usb ports and 1* are the internal things like BT and camera).
perhaps my usb hd could be the cause. i got an error that the device didn't exist when i wanted to remount it (with the uuid), but forgot to take a look into /dev.
i plugged my (still or again) working keyboard in the port of my usb hd and it stopped working, even plugging it into the old port and it still didn't work. i get many "khubd time out" messages.
i rebooted without the usb hd, only with mouse and keyboard plugged in.

The usb port of my external hd was this one (right now my keyboard is plugged in):
T: Bus=02 Lev=02 Prnt=02 Port=05 Cnt=02 Dev#= 9 Spd=1.5 MxCh= 0

[TheTuxKeeper]

so far no usb reset. now i plugged in my external hd an let amarok play some music from it. i deactivated everything else what would touch files on the hd. if that will kill usb, then i'll try to let the hd only plugged in and not mounted.
any other suggestions what could help to find more hints?
are there any debug kernel options which could help?

[NeddySeagoon]

TheTuxKeeper,

Your USB HDD does not appear to be USB bus powered, so its probably not a problem.
If it is bus powered , its almost certainly an issue.

usbview (emerge usbview) shows you have powered hubs connected to both of your USB root hubs.
Some things do not like to operate via a hub. The hub with your USB HDD attached has a mix of USB1 and USB2 devices attached.
It may be forcing your USB2 USB HDD to operate at USB1 speeds. USB2 devices are supposed to fall back to USB1 but many do not.

Just because your root hubs support transaction translation for mixed USB1/2 operation your hub may not.
Does the USB HDD alone work?
Do the two other USB1 devices work together?

[TheTuxKeeper]

usbview (emerge usbview) shows you have powered hubs connected to both of your USB root hubs.
Some things do not like to operate via a hub. The hub with your USB HDD attached has a mix of USB1 and USB2 devices attached.
It may be forcing your USB2 USB HDD to operate at USB1 speeds. USB2 devices are supposed to fall back to USB1 but many do not.

No, it's working with more than usb 1.1 (i've always dstat running, so i see it quite often). it's always >5 mb/s.
My mouse receiver and my keyboard are usb powered that's right. I can see 100 mA for each (MaxPower)

Just because your root hubs support transaction translation for mixed USB1/2 operation your hub may not.
Does the USB HDD alone work?
Do the two other USB1 devices work together?

I'm little confused, perhaps because it sounds like you mean an external hub
i don't have any external hub, it's all builtin (though those might be/are hubs too). lsusb shows them as "Bus 001 Device 002: ID 8087:0020 Intel Corp. Integrated Rate Matching Hub".
lsusb -v shows "bDeviceProtocol 1 Single TT" for the Intel hubs. I googled and found that this means, there's only one TT for all ports. But that shouldn't be a problem with just a mouse and a keyboard as the only usb1.1 devices.

My external hd was mounted the whole day and nothing really happened. I thought i let amarok play some music to add some minimal load to the external hd/usb port, but perhaps it crashed since it wasn't there any more
I got ~20 messages like this every few minutes (twice a minute to every 10 minutes):

Code: Select all

Mar 31 09:32:18 tuxkeeper-gentoo kernel: [89971.628696] ehci_hcd 0000:00:1d.0: IAA with nothing to reclaim?

perhaps that was while amarok still ran and read mp3 from the external hd. after that nothing came until i started a "find /external/hd -type f | wc -l". a few minutes later the "IAA" message started to pop up every minute (find started ~19:35 and i killed it at ~19:38 ) :

Code: Select all

Mar 31 10:37:25 tuxkeeper-gentoo kernel: [93871.717932] ehci_hcd 0000:00:1d.0: IAA with nothing to reclaim?
Mar 31 19:35:33 tuxkeeper-gentoo kernel: [126105.500415] ehci_hcd 0000:00:1d.0: IAA with nothing to reclaim?
Mar 31 19:36:06 tuxkeeper-gentoo kernel: [126139.168700] ehci_hcd 0000:00:1d.0: IAA with nothing to reclaim?
Mar 31 19:37:16 tuxkeeper-gentoo kernel: [126208.282336] ehci_hcd 0000:00:1d.0: IAA with nothing to reclaim?
Mar 31 19:38:30 tuxkeeper-gentoo kernel: [126282.475383] ehci_hcd 0000:00:1d.0: IAA with nothing to reclaim?
Mar 31 19:48:44 tuxkeeper-gentoo kernel: [126895.221652] usb 1-1.2: usb auto-resume
Mar 31 19:48:44 tuxkeeper-gentoo kernel: [126895.258220] usb 1-1.2: finish resume
Mar 31 19:48:44 tuxkeeper-gentoo kernel: [126895.259218] usb 1-1.2: disable remote wakeup, status -32
Mar 31 19:48:46 tuxkeeper-gentoo kernel: [126897.474426] usb 1-1.2: usb auto-suspend
Mar 31 19:51:55 tuxkeeper-gentoo kernel: [127086.269343] usb 1-1.2: usb auto-resume
Mar 31 19:51:55 tuxkeeper-gentoo kernel: [127086.306510] usb 1-1.2: finish resume
Mar 31 19:51:55 tuxkeeper-gentoo kernel: [127086.307522] usb 1-1.2: disable remote wakeup, status -32
Mar 31 19:51:57 tuxkeeper-gentoo kernel: [127088.322147] usb 1-1.2: usb auto-suspend
Mar 31 19:59:59 tuxkeeper-gentoo kernel: [127569.243661] usb 1-1.2: usb auto-resume
Mar 31 19:59:59 tuxkeeper-gentoo kernel: [127569.280231] usb 1-1.2: finish resume
Mar 31 19:59:59 tuxkeeper-gentoo kernel: [127569.281193] usb 1-1.2: disable remote wakeup, status -32
Mar 31 20:00:01 tuxkeeper-gentoo kernel: [127571.355771] usb 1-1.2: usb auto-suspend

The "IAA" message seems to be reproducable, i started the find again and a second later the message appeared again.
found nothing else than unrelated stuff or kernel diffs when i searched that message.
Could it be that the usb bus (or something else) fails on high load ?

[NeddySeagoon]

TheTuxKeeper,

The kernel seems to be powering down the USB port.

I wasn't sure if your hubs were internal or not. usbview shows them as separate devices, which is unusual.

[TheTuxKeeper]

TheTuxKeeper,

The kernel seems to be powering down the USB port.

i disabled that for all ports now. will do the "find" test again.

Code: Select all

tuxkeeper-gentoo ~ # cat /sys/bus/usb/devices/*/power/autosuspend_delay_ms 
-1000
-1000
-1000
-1000
-1000
-1000
-1000
-1000
-1000

Code: Select all

tuxkeeper-gentoo ~ # cat /sys/bus/usb/devices/1-1.2/manufacturer
USB 2.0 Camera_FLV14

seems to be my inactive builtin webcam that did those suspends and resumes

Code: Select all

Mar 31 21:28:28 tuxkeeper-gentoo kernel: [132869.585448] usb 1-1.2: usb auto-resume
Mar 31 21:28:28 tuxkeeper-gentoo kernel: [132869.622409] usb 1-1.2: finish resume
Mar 31 21:28:28 tuxkeeper-gentoo kernel: [132869.623527] usb 1-1.2: disable remote wakeup, status -32

that's after i disabled autosuspend.

I wasn't sure if your hubs were internal or not. usbview shows them as separate devices, which is unusual.

ok, sorry i didn't make this clear at the begining

[TheTuxKeeper]

Now i did some "high load" tests. aacgain over a few thousand files an rsync backup (1:1 copy of my gentoo) and bang it "crashed" again.

the same log entries as always.

1. some "IAA with nothing to reclaim?" if there a little load
2. then "usb-storage timed out on ep0out len=0/0" with "cannot reset port 6 (err = -110)" and "port 6 not enabled, trying reset again.." following
3. at the end "khubd timed out on ep0in len=0/4" and "hub_port_status failed (err = -110)"

after unplugging the external hd those entries repeat nearly endlessly:

Code: Select all

Mar 31 23:03:26 tuxkeeper-gentoo kernel: [138557.852638] hub 2-1:1.0: state 7 ports 8 chg 0040 evt 0040
Mar 31 23:03:27 tuxkeeper-gentoo kernel: [138558.860673] usb 2-1: khubd timed out on ep0in len=0/4
Mar 31 23:03:28 tuxkeeper-gentoo kernel: [138559.868955] usb 2-1: khubd timed out on ep0in len=0/4
Mar 31 23:03:29 tuxkeeper-gentoo kernel: [138560.877284] usb 2-1: khubd timed out on ep0in len=0/4
Mar 31 23:03:30 tuxkeeper-gentoo kernel: [138561.885587] usb 2-1: khubd timed out on ep0in len=0/4
Mar 31 23:03:31 tuxkeeper-gentoo kernel: [138562.893816] usb 2-1: khubd timed out on ep0in len=0/4
Mar 31 23:03:31 tuxkeeper-gentoo kernel: [138562.893825] hub 2-1:1.0: hub_port_status failed (err = -110)
Mar 31 23:03:31 tuxkeeper-gentoo kernel: [138562.893833] hub 2-1:1.0: state 7 ports 8 chg 0040 evt 0040

dmesg/kern.log output (usb stuff only)

edit:
i did an unbind and bind of the hub of the ports with mouse, keyboard and external hd and it works again. But that's not a real solution ...

Code: Select all

tuxkeeper-gentoo ~ # cd /sys/bus/pci/drivers/ehci_hcd
tuxkeeper-gentoo ehci_hcd # l
total 0
drwxr-xr-x  2 root root    0 Mär 31 23:10 ./
drwxr-xr-x 21 root root    0 Mär 31 23:10 ../
lrwxrwxrwx  1 root root    0 Mär 31 23:12 0000:00:1a.0 -> ../../../../devices/pci0000:00/0000:00:1a.0/
lrwxrwxrwx  1 root root    0 Mär 31 23:12 0000:00:1d.0 -> ../../../../devices/pci0000:00/0000:00:1d.0/
--w-------  1 root root 4,0K Mär 31 23:12 bind
lrwxrwxrwx  1 root root    0 Mär 31 23:12 module -> ../../../../module/ehci_hcd/
--w-------  1 root root 4,0K Mär 31 23:12 new_id
--w-------  1 root root 4,0K Mär 31 23:12 remove_id
--w-------  1 root root 4,0K Mär 31 23:12 uevent
--w-------  1 root root 4,0K Mär 31 23:12 unbind
tuxkeeper-gentoo ehci_hcd # echo '0000:00:1d.0' > unbind 
tuxkeeper-gentoo ehci_hcd # echo '0000:00:1d.0' > bind 

dmesg/kern.log output (unbind + bind)

edit2:
I tried some kernelparameters:
1) acpi=off
=> no usb resets not even when transfering ~90gb (multiple .ts mpeg streams) with 30 mb/s avg to /dev/null. But i have only one core available and HTseems to be off too. so not really an option/solution ...

2) acpi=strict (just tried to disable acpi not completely)
=> seemed to be little better but after a while the "IAA" messages appeared and some hours later usb was dead. But while it worked the transfer to the usb hd stopped for ~15s every 1-2 minutes (even only <1mb reads from amarok got stuck). So that didn't help either

[lkraav]

also ran into this today, running 2.6.38.2-zen.

i was also working with an external usb hdd when this happened. had an mmc memory card inserted as well. last night was using a smartcard reader and pcscd crashed at one point, unsure if this had an effect.

machine: http://en.gentoo-wiki.com/wiki/Acer_Travelmate_8172

[TheTuxKeeper]

I found a "fix". I have to boot windows before rebooting into linux.
now nearly two days uptime without any problems ... only disabled autosuspend is still in the kernelparameter list

[TheTuxKeeper]

After a new hard disk in my notebook (that's why i couldn't test for some time) and a few weeks without problems, i can say that it seems to be fixed though i sometimes can see "IAA with nothing to reclaim" in the log. but no usb reset so far.
I'm running sys-kernel/gentoo-sources-2.6.38-r7 but -r3 and -r4 were good too. usb autosuspend is on default values (no kernel parameters set). didn't test with .39 since i have no time for that right now.

EDIT: looks like it's still happening from time to time. perhaps it's the usb hub that is connected (it's also a usb to parallel, serial, net, ..., a usb "docking station")

[Cocker68]

I did have exactly the same problem as TheTuxKeeper: A huge amount of messages reading "IAA with nothing to reclaim", after an hour followed by a total hangup of the USB, which made reboot necessary. (Kernel 2.6.39-gentoo-r3)

It turned out, that the one to blame was a 1TB-disk built into an external case, which apparently was not able to deal with it properly. I now built the disk into a different case, and all hickups are gone. Both cases are powered by their own supply.

Greetings
- Cocker :wq