My router started to crash randomly over 2 months ago or so.
Its a home built ITX router and it was more or less 3 years old on the day the problem started,
I knew from the logs that I had not upgraded the system in over 30 days so I suspected hardware early on.
Everything pointed to the iwlwifi module in my logs so some problem with the Intel AC 7260.
System would get random kernel panics but befor then one of my wifi cards acting as Access point would stop working, that being the intel AC 7260, my other card that uses the ATK5 module works just fine.
I hunted the problem for almost a month, I could trigger the intel 7260 using iwlwifi to crash by having multiple devices connected and transferring data, usually it took 30-40 minutes with synthetic loads and 2 days or so during normal loads.
I could not make my other card crash that uses atk5 module.
I tested different kernels, the system had been running 4.5 for a very long time, 4.9 had the same problem, 4.10 did not crash but devices did disconnect and refused to reconnect so in the end a reset was required and the machine was overall even less stable and useful.
The only real software relating to the wifi that was upgraded over a month before the problem began was hostapd, from 2.5 to 2.6.
I downgraded and everything seems to work, a howl day of stress testing and no failure. Two days later I got a Kernel panic.
I knew from my logs from the start that I had PCI-E buss errors, IRQ errors, timeouts and all related to iwlwifi.
So now a new card had to be ordered. My card uses miniPCI-E, so I could only find one card in Sweden in stock and it was another Intel AC 7260.
I ordered it and had it the next day or so. Replaced the old one and power the router on, ran stress test for a howl day and then some, its been running stable now for 14 days and so far no crash. Before I replaced it It ran for about 2 days during normal use before crashing.
So far I have had no errors of the kind I had before in my logs.
But just the other day this new card showed its first error, tough I am on kernel 4.9 now and a new firmware so I cant exclude it being a firmware issue.
But intels 7260 cards seems problematic to say the least, I figured intel could do better but considering the problem people seem to have even with Windows machines running this card where replacements seems to at least fix the issue for some time the only conclusion I can come to is that intels Wifi card are crap, poor quality builds.
First error after 12 days with a new card.... no crash but annoying. This error is nothing like with the old card spits out in the same machine.
Code: Select all
May 15 06:28:47 enterprise kernel: [998495.236877] iwlwifi 0000:05:00.0: Queue 4 stuck for 10000 ms.
May 15 06:28:47 enterprise kernel: [998495.236894] iwlwifi 0000:05:00.0: Current SW read_ptr 80 write_ptr 81
May 15 06:28:47 enterprise kernel: [998495.236921] iwl data: 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
May 15 06:28:47 enterprise kernel: [998495.236937] iwlwifi 0000:05:00.0: FH TRBs(0) = 0x00000000
May 15 06:28:47 enterprise kernel: [998495.236952] iwlwifi 0000:05:00.0: FH TRBs(1) = 0x801020af
May 15 06:28:47 enterprise kernel: [998495.236966] iwlwifi 0000:05:00.0: FH TRBs(2) = 0x00000000
May 15 06:28:47 enterprise kernel: [998495.236982] iwlwifi 0000:05:00.0: FH TRBs(3) = 0x803000c0
May 15 06:28:47 enterprise kernel: [998495.236995] iwlwifi 0000:05:00.0: FH TRBs(4) = 0x00000000
May 15 06:28:47 enterprise kernel: [998495.237008] iwlwifi 0000:05:00.0: FH TRBs(5) = 0x80504050
May 15 06:28:47 enterprise kernel: [998495.237022] iwlwifi 0000:05:00.0: FH TRBs(6) = 0x00000000
May 15 06:28:47 enterprise kernel: [998495.237035] iwlwifi 0000:05:00.0: FH TRBs(7) = 0x007090f5
May 15 06:28:47 enterprise kernel: [998495.237086] iwlwifi 0000:05:00.0: Q 0 is active and mapped to fifo 3 ra_tid 0x0000 [193,193]
May 15 06:28:47 enterprise kernel: [998495.237137] iwlwifi 0000:05:00.0: Q 1 is active and mapped to fifo 2 ra_tid 0x0000 [0,0]
May 15 06:28:47 enterprise kernel: [998495.237187] iwlwifi 0000:05:00.0: Q 2 is active and mapped to fifo 1 ra_tid 0x0000 [176,176]
May 15 06:28:47 enterprise kernel: [998495.237237] iwlwifi 0000:05:00.0: Q 3 is active and mapped to fifo 0 ra_tid 0x0000 [0,0]
May 15 06:28:47 enterprise kernel: [998495.237287] iwlwifi 0000:05:00.0: Q 4 is active and mapped to fifo 5 ra_tid 0x0000 [80,81]
May 15 06:28:47 enterprise kernel: [998495.237338] iwlwifi 0000:05:00.0: Q 5 is inactive and mapped to fifo 0 ra_tid 0x0000 [0,0]
May 15 06:28:47 enterprise kernel: [998495.237388] iwlwifi 0000:05:00.0: Q 6 is inactive and mapped to fifo 0 ra_tid 0x0000 [0,0]
May 15 06:28:47 enterprise kernel: [998495.237438] iwlwifi 0000:05:00.0: Q 7 is inactive and mapped to fifo 0 ra_tid 0x0000 [0,0]
May 15 06:28:47 enterprise kernel: [998495.237488] iwlwifi 0000:05:00.0: Q 8 is inactive and mapped to fifo 0 ra_tid 0x0000 [0,0]
May 15 06:28:47 enterprise kernel: [998495.237539] iwlwifi 0000:05:00.0: Q 9 is active and mapped to fifo 7 ra_tid 0x0000 [246,246]
May 15 06:28:47 enterprise kernel: [998495.237589] iwlwifi 0000:05:00.0: Q 10 is inactive and mapped to fifo 0 ra_tid 0x0000 [0,0]
May 15 06:28:47 enterprise kernel: [998495.237639] iwlwifi 0000:05:00.0: Q 11 is inactive and mapped to fifo 0 ra_tid 0x0000 [0,0]
May 15 06:28:47 enterprise kernel: [998495.237689] iwlwifi 0000:05:00.0: Q 12 is inactive and mapped to fifo 0 ra_tid 0x0000 [0,0]
May 15 06:28:47 enterprise kernel: [998495.237740] iwlwifi 0000:05:00.0: Q 13 is inactive and mapped to fifo 0 ra_tid 0x0000 [0,0]
May 15 06:28:47 enterprise kernel: [998495.237790] iwlwifi 0000:05:00.0: Q 14 is inactive and mapped to fifo 0 ra_tid 0x0000 [0,0]
May 15 06:28:47 enterprise kernel: [998495.237840] iwlwifi 0000:05:00.0: Q 15 is active and mapped to fifo 5 ra_tid 0x0000 [0,0]
May 15 06:28:47 enterprise kernel: [998495.237890] iwlwifi 0000:05:00.0: Q 16 is inactive and mapped to fifo 1 ra_tid 0x0000 [232,232]
May 15 06:28:47 enterprise kernel: [998495.237941] iwlwifi 0000:05:00.0: Q 17 is inactive and mapped to fifo 1 ra_tid 0x0000 [215,215]
May 15 06:28:47 enterprise kernel: [998495.237991] iwlwifi 0000:05:00.0: Q 18 is inactive and mapped to fifo 0 ra_tid 0x0000 [0,0]
May 15 06:28:47 enterprise kernel: [998495.238042] iwlwifi 0000:05:00.0: Q 19 is inactive and mapped to fifo 0 ra_tid 0x0000 [0,0]
May 15 06:28:47 enterprise kernel: [998495.238092] iwlwifi 0000:05:00.0: Q 20 is inactive and mapped to fifo 0 ra_tid 0x0000 [0,0]
May 15 06:28:47 enterprise kernel: [998495.238142] iwlwifi 0000:05:00.0: Q 21 is inactive and mapped to fifo 0 ra_tid 0x0000 [0,0]
May 15 06:28:47 enterprise kernel: [998495.238193] iwlwifi 0000:05:00.0: Q 22 is inactive and mapped to fifo 0 ra_tid 0x0000 [0,0]
May 15 06:28:47 enterprise kernel: [998495.238243] iwlwifi 0000:05:00.0: Q 23 is inactive and mapped to fifo 0 ra_tid 0x0000 [0,0]
May 15 06:28:47 enterprise kernel: [998495.238294] iwlwifi 0000:05:00.0: Q 24 is inactive and mapped to fifo 0 ra_tid 0x0000 [0,0]
May 15 06:28:47 enterprise kernel: [998495.238344] iwlwifi 0000:05:00.0: Q 25 is inactive and mapped to fifo 0 ra_tid 0x0000 [0,0]
May 15 06:28:47 enterprise kernel: [998495.238394] iwlwifi 0000:05:00.0: Q 26 is inactive and mapped to fifo 0 ra_tid 0x0000 [0,0]
May 15 06:28:47 enterprise kernel: [998495.238445] iwlwifi 0000:05:00.0: Q 27 is inactive and mapped to fifo 0 ra_tid 0x0000 [0,0]
May 15 06:28:47 enterprise kernel: [998495.238496] iwlwifi 0000:05:00.0: Q 28 is inactive and mapped to fifo 0 ra_tid 0x0000 [0,0]
May 15 06:28:47 enterprise kernel: [998495.238552] iwlwifi 0000:05:00.0: Q 29 is inactive and mapped to fifo 0 ra_tid 0x0000 [0,0]
May 15 06:28:47 enterprise kernel: [998495.238602] iwlwifi 0000:05:00.0: Q 30 is inactive and mapped to fifo 0 ra_tid 0x0000 [0,0]
May 15 06:28:47 enterprise kernel: [998495.238947] iwlwifi 0000:05:00.0: Microcode SW error detected. Restarting 0x2000000.
May 15 06:28:47 enterprise kernel: [998495.238953] iwlwifi 0000:05:00.0: CSR values:
May 15 06:28:47 enterprise kernel: [998495.238956] iwlwifi 0000:05:00.0: (2nd byte of CSR_INT_COALESCING is CSR_INT_PERIODIC_REG)
May 15 06:28:47 enterprise kernel: [998495.238963] iwlwifi 0000:05:00.0: CSR_HW_IF_CONFIG_REG: 0X40489204
May 15 06:28:47 enterprise kernel: [998495.238969] iwlwifi 0000:05:00.0: CSR_INT_COALESCING: 0X80000040
May 15 06:28:47 enterprise kernel: [998495.238974] iwlwifi 0000:05:00.0: CSR_INT: 0X00000000
May 15 06:28:47 enterprise kernel: [998495.238980] iwlwifi 0000:05:00.0: CSR_INT_MASK: 0X00000000
May 15 06:28:47 enterprise kernel: [998495.238986] iwlwifi 0000:05:00.0: CSR_FH_INT_STATUS: 0X00000000
May 15 06:28:47 enterprise kernel: [998495.238991] iwlwifi 0000:05:00.0: CSR_GPIO_IN: 0X00000000
May 15 06:28:47 enterprise kernel: [998495.238997] iwlwifi 0000:05:00.0: CSR_RESET: 0X00000000
May 15 06:28:47 enterprise kernel: [998495.239002] iwlwifi 0000:05:00.0: CSR_GP_CNTRL: 0X080403c5
May 15 06:28:47 enterprise kernel: [998495.239008] iwlwifi 0000:05:00.0: CSR_HW_REV: 0X00000144
May 15 06:28:47 enterprise kernel: [998495.239014] iwlwifi 0000:05:00.0: CSR_EEPROM_REG: 0X00000000
May 15 06:28:47 enterprise kernel: [998495.239019] iwlwifi 0000:05:00.0: CSR_EEPROM_GP: 0X80000000
May 15 06:28:47 enterprise kernel: [998495.239025] iwlwifi 0000:05:00.0: CSR_OTP_GP_REG: 0X803a0000
May 15 06:28:47 enterprise kernel: [998495.239030] iwlwifi 0000:05:00.0: CSR_GIO_REG: 0X001f0044
May 15 06:28:47 enterprise kernel: [998495.239036] iwlwifi 0000:05:00.0: CSR_GP_UCODE_REG: 0X00000000
May 15 06:28:47 enterprise kernel: [998495.239042] iwlwifi 0000:05:00.0: CSR_GP_DRIVER_REG: 0X00000000
May 15 06:28:47 enterprise kernel: [998495.239047] iwlwifi 0000:05:00.0: CSR_UCODE_DRV_GP1: 0X00000000
May 15 06:28:47 enterprise kernel: [998495.239053] iwlwifi 0000:05:00.0: CSR_UCODE_DRV_GP2: 0X00000000
May 15 06:28:47 enterprise kernel: [998495.239058] iwlwifi 0000:05:00.0: CSR_LED_REG: 0X00000060
May 15 06:28:47 enterprise kernel: [998495.239064] iwlwifi 0000:05:00.0: CSR_DRAM_INT_TBL_REG: 0X88231c36
May 15 06:28:47 enterprise kernel: [998495.239069] iwlwifi 0000:05:00.0: CSR_GIO_CHICKEN_BITS: 0X27800200
May 15 06:28:47 enterprise kernel: [998495.239075] iwlwifi 0000:05:00.0: CSR_ANA_PLL_CFG: 0Xd55555d5
May 15 06:28:47 enterprise kernel: [998495.239081] iwlwifi 0000:05:00.0: CSR_MONITOR_STATUS_REG: 0X3d0801bd
May 15 06:28:47 enterprise kernel: [998495.239086] iwlwifi 0000:05:00.0: CSR_HW_REV_WA_REG: 0X0001001a
May 15 06:28:47 enterprise kernel: [998495.239092] iwlwifi 0000:05:00.0: CSR_DBG_HPET_MEM_REG: 0Xffff0000
May 15 06:28:47 enterprise kernel: [998495.239095] iwlwifi 0000:05:00.0: FH register values:
May 15 06:28:47 enterprise kernel: [998495.239109] iwlwifi 0000:05:00.0: FH_RSCSR_CHNL0_STTS_WPTR_REG: 0X23166e00
May 15 06:28:47 enterprise kernel: [998495.239123] iwlwifi 0000:05:00.0: FH_RSCSR_CHNL0_RBDCB_BASE_REG: 0X023166d0
May 15 06:28:47 enterprise kernel: [998495.239136] iwlwifi 0000:05:00.0: FH_RSCSR_CHNL0_WPTR: 0X00000010
May 15 06:28:47 enterprise kernel: [998495.239150] iwlwifi 0000:05:00.0: FH_MEM_RCSR_CHNL0_CONFIG_REG: 0X00801114
May 15 06:28:47 enterprise kernel: [998495.239163] iwlwifi 0000:05:00.0: FH_MEM_RSSR_SHARED_CTRL_REG: 0X000000fc
May 15 06:28:47 enterprise kernel: [998495.239177] iwlwifi 0000:05:00.0: FH_MEM_RSSR_RX_STATUS_REG: 0X03030000
May 15 06:28:47 enterprise kernel: [998495.239190] iwlwifi 0000:05:00.0: FH_MEM_RSSR_RX_ENABLE_ERR_IRQ2DRV: 0X00000000
May 15 06:28:47 enterprise kernel: [998495.239204] iwlwifi 0000:05:00.0: FH_TSSR_TX_STATUS_REG: 0X07ff0001
May 15 06:28:47 enterprise kernel: [998495.239217] iwlwifi 0000:05:00.0: FH_TSSR_TX_ERROR_REG: 0X00000000
May 15 06:28:47 enterprise kernel: [998495.239322] iwlwifi 0000:05:00.0: Start IWL Error Log Dump:
May 15 06:28:47 enterprise kernel: [998495.239326] iwlwifi 0000:05:00.0: Status: 0x00000000, count: 6
May 15 06:28:47 enterprise kernel: [998495.239329] iwlwifi 0000:05:00.0: Loaded firmware version: 17.459231.0
May 15 06:28:47 enterprise kernel: [998495.239333] iwlwifi 0000:05:00.0: 0x00000084 | NMI_INTERRUPT_UNKNOWN
May 15 06:28:47 enterprise kernel: [998495.239336] iwlwifi 0000:05:00.0: 0x008002F1 | trm_hw_status0
May 15 06:28:47 enterprise kernel: [998495.239339] iwlwifi 0000:05:00.0: 0x00000000 | trm_hw_status1
May 15 06:28:47 enterprise kernel: [998495.239342] iwlwifi 0000:05:00.0: 0x00000B30 | branchlink2
May 15 06:28:47 enterprise kernel: [998495.239345] iwlwifi 0000:05:00.0: 0x000164C0 | interruptlink1
May 15 06:28:47 enterprise kernel: [998495.239348] iwlwifi 0000:05:00.0: 0x00015614 | interruptlink2
May 15 06:28:47 enterprise kernel: [998495.239351] iwlwifi 0000:05:00.0: 0x00000000 | data1
May 15 06:28:47 enterprise kernel: [998495.239354] iwlwifi 0000:05:00.0: 0x00000080 | data2
May 15 06:28:47 enterprise kernel: [998495.239358] iwlwifi 0000:05:00.0: 0x07030000 | data3
May 15 06:28:47 enterprise kernel: [998495.239361] iwlwifi 0000:05:00.0: 0x294178AA | beacon time
May 15 06:28:47 enterprise kernel: [998495.239364] iwlwifi 0000:05:00.0: 0x60C0587F | tsf low
May 15 06:28:47 enterprise kernel: [998495.239367] iwlwifi 0000:05:00.0: 0x000000E8 | tsf hi
May 15 06:28:47 enterprise kernel: [998495.239370] iwlwifi 0000:05:00.0: 0x00000000 | time gp1
May 15 06:28:47 enterprise kernel: [998495.239373] iwlwifi 0000:05:00.0: 0x60C05880 | time gp2
May 15 06:28:47 enterprise kernel: [998495.239376] iwlwifi 0000:05:00.0: 0x00000000 | uCode revision type
May 15 06:28:47 enterprise kernel: [998495.239379] iwlwifi 0000:05:00.0: 0x00000011 | uCode version major
May 15 06:28:47 enterprise kernel: [998495.239382] iwlwifi 0000:05:00.0: 0x000701DF | uCode version minor
May 15 06:28:47 enterprise kernel: [998495.239385] iwlwifi 0000:05:00.0: 0x00000144 | hw version
May 15 06:28:47 enterprise kernel: [998495.239388] iwlwifi 0000:05:00.0: 0x40489204 | board version
May 15 06:28:47 enterprise kernel: [998495.239391] iwlwifi 0000:05:00.0: 0x09F1011C | hcmd
May 15 06:28:47 enterprise kernel: [998495.239394] iwlwifi 0000:05:00.0: 0x00922060 | isr0
May 15 06:28:47 enterprise kernel: [998495.239397] iwlwifi 0000:05:00.0: 0x00000000 | isr1
May 15 06:28:47 enterprise kernel: [998495.239400] iwlwifi 0000:05:00.0: 0x0000000A | isr2
May 15 06:28:47 enterprise kernel: [998495.239403] iwlwifi 0000:05:00.0: 0x0041E8C1 | isr3
May 15 06:28:47 enterprise kernel: [998495.239406] iwlwifi 0000:05:00.0: 0x00000000 | isr4
May 15 06:28:47 enterprise kernel: [998495.239409] iwlwifi 0000:05:00.0: 0x000C0110 | last cmd Id
May 15 06:28:47 enterprise kernel: [998495.239412] iwlwifi 0000:05:00.0: 0x00000000 | wait_event
May 15 06:28:47 enterprise kernel: [998495.239416] iwlwifi 0000:05:00.0: 0x00004302 | l2p_control
May 15 06:28:47 enterprise kernel: [998495.239419] iwlwifi 0000:05:00.0: 0x00000820 | l2p_duration
May 15 06:28:47 enterprise kernel: [998495.239422] iwlwifi 0000:05:00.0: 0x00000003 | l2p_mhvalid
May 15 06:28:47 enterprise kernel: [998495.239425] iwlwifi 0000:05:00.0: 0x0000F000 | l2p_addr_match
May 15 06:28:47 enterprise kernel: [998495.239428] iwlwifi 0000:05:00.0: 0x00000007 | lmpm_pmg_sel
May 15 06:28:47 enterprise kernel: [998495.239431] iwlwifi 0000:05:00.0: 0x11011741 | timestamp
May 15 06:28:47 enterprise kernel: [998495.239434] iwlwifi 0000:05:00.0: 0x00341020 | flow_handler
May 15 06:28:47 enterprise kernel: [998495.239440] ieee80211 phy1: Hardware restart was requested
May 15 06:28:47 enterprise kernel: [998495.383865] iwlwifi 0000:05:00.0: L1 Disabled - LTR Enabled
May 15 06:28:47 enterprise kernel: [998495.384094] iwlwifi 0000:05:00.0: L1 Disabled - LTR Enabled
May 15 06:28:47 enterprise kernel: [998495.611327] iwlwifi 0000:05:00.0: L1 Disabled - LTR Enabled
May 15 06:28:47 enterprise kernel: [998495.611558] iwlwifi 0000:05:00.0: L1 Disabled - LTR Enabled
Gentoo server: R5 5600G 4.4Ghz, 32GB ram, 8TB SSD, 76TB HDD, NIC: 10Gbit + 2.5Gbit