Troubleshooting apcupsd with USB

[whiskeypriest]

Troubleshooting apcupsd with USB

In September 2004 the HOWTO: apcupsd with USB and its ensuing thread finally broke the three-page mark, necessitating (at least in my mind) the need for a separate topic to discuss problems with installation and execution. Therefore, if you're having problems please read through the remainder of that thread first; if you don't find an answer there, please post to this thread.

[whiskeypriest]

Taken up from this thread:

Excellent HOW-TO!

Most everything went smoothly on the setup; just a couple of comments:

1) I had problems building usb support as a module with my 2.6 kernel. After making usb built-in, everything went smoothly.

2) The UPS unit does not appear to shut off via the 'killpower' option to apccontrol after the shutdown sequence has completed. Is this a limitation of my UPS (APC Back-Ups RS 1000) or a problem with the shutdown/halt scripts?

I waited a good 6 mins after the host shutdown for the UPS to 'killpower', but I had to turn it off by pressing the UPS power button. I'm assuming this has to happen for the UPS to correctly bring the computers back up when AC power returns.

UPDATE: I just noticed that the halt script looks for /etc/apcupsd/powerfail in order to killpower to the UPS. I just did a quick test and the file is not created when there is a loss of power. Looking through the apcupsd manual, it looks this should be taken care of by the /etc/apcupsd/poweroff or onbattery script:
http://www2.apcupsd.com/3.10.x-manual/shutdown.html

You raise an excellent question; unfortunately, it's one I don't have an answer to. I've observed identical behavior in all my UPS units (my stable currently includes one Back-UPS CS 350, one Back-UPS RS 1000, and two Back-UPS XS 800's). It's my off-the-cuff opinion that this functionality is probably limited to the higher-end Smart-UPS units; if I had one, I'd be happy to confirm or deny this theory. I'll do some digging and see if I can't find a definitive answer.

This problem is a prime example of where my knowledge on this subject ends: I'd love to know why this doesn't work, as well as several other things about apcupsd – for example, how to get multiple USB-driven units playing nicely together on a single machine.

So consider this an exhortation to anyone who has a more exhaustive knowledge of the subject than I do: please feel free to contribute. I'm not the final authority on this subject, just the guy who wrote a HOWTO on the basics.

[suhlhorn]

WP: I suspect you're right about the killpower functionality being limited to the Smart-UPS units. This is in-line with the Back-UPS [CRX]S units not being able to update the eeprom, from what I can tell.

Are you a member of the apcupsd-user list? If so, maybe you can ask the user list about this one. If not, I might join so I can figure this out...just for my curiosity.

-stephen

[whiskeypriest]

I'm not currently subscribed to the apcupsd-user list, but I did run a quick search of the archives over at SF and came up with this recent thread, which seems to confirm our suspicions.

It looks as though the fine folks over at the apcupsd project are working at enabling this functionality for USB UPS units in the 3.10.16 release; with any luck, I'd imagine we'll see this in portage before the turn of the year. It also appears that they're looking for testers right now, if anyone's interested in wading into the fray...

[suhlhorn]

Interesting...

Seeing as how my power problem is 1000% better than it was 1 week ago, I'm pretty content to wait for the next release which should clear this up.

BTW- Just wanted to thank you (WP) again for the excellent how-to. It was very clear and I had everything running in few hours time.

Thanks!
-stephen

[whiskeypriest]

Yeah, you folks in Florida have taken quite a beating this year. Hope the worst of it's behind you, and thanks for saying thanks.

We'll see what shakes out with 3.10.16...

[whiskeypriest]

I do have one questions, though. How can I get my APC unit to turn my machine back on when power is restored? Right now, when the APC shuts down my system, gentoo hangs on the "Power Down" step of the shut down process. It doesn't actually power down. I guess what I'm asking is how can I get gentoo to shutdown without having to push the power button? I changed my BIOS last state setting so my system should turn back on when power is restored.

What you’re asking is actually two separate things, so let me take them in turn:

• Currently, USB APC units are not capable of shutting themselves down after calling the system halt; as such, the machine must be manually powered back on after a shutdown unless the UPS unit has completely exhausted itself…though it sounds like you have the correct BIOS settings for when this functionality is eventually implemented. See above for further information.
• As for your system hanging at “Power Down” this is usually a power management issue, correctable with the proper kernel configuration. See the KERNEL CONFIGURATION section of the HOWTO for further details. From a troubleshooting perspective, my first questions to you would be, “Did the system power itself down completely before you followed this HOWTO? Does it power itself down completely on a shutdown command without apcupsd calling it?” If so, I’d recommend returning your power management configuration to what it was previously. Otherwise, give the aforementioned kernel configurations a try.

[dmolavi]

i'm using an APC BackUPS ES 500, and was, until a few weeks ago, using apcupsd-3.10.10-r2 on my machine. It behaved as expected, including shutting down and powering down the machine when the appropriate time came, and restarting the machine with the return of mains.

However, upon upgrade to 3.10.13, the system hangs on the "Power Down" part, and needs to be manually restarted.

What changed in apcupsd that would make this happen? I'm thinking of downgrading to 3.10.10-r2 again and waiting for a fix. My kernel config has not changed in months...

EDIT: here is some more info:

Code: Select all

root: ~ # ls /dev/usb/hid/h*
/dev/usb/hid/hiddev0

root: ~ # cat /proc/bus/usb/devices
T:  Bus=01 Lev=01 Prnt=01 Port=00 Cnt=01 Dev#=  2 Spd=1.5 MxCh= 0
D:  Ver= 1.10 Cls=00(>ifc ) Sub=00 Prot=00 MxPS= 8 #Cfgs=  1
P:  Vendor=051d ProdID=0002 Rev= 1.06
S:  Manufacturer=APC
S:  Product=Back-UPS ES 500 FW:2.e2.D USB FW:e2
S:  SerialNumber=JB0312030497  
C:* #Ifs= 1 Cfg#= 1 Atr=e0 MxPwr=  2mA
I:  If#= 0 Alt= 0 #EPs= 1 Cls=03(HID  ) Sub=00 Prot=00 Driver=hid
E:  Ad=81(I) Atr=03(Int.) MxPS=   6 Ivl=10ms

(just  a snip w/ the UPS data)

root: /proc/bus/usb # cat drivers 
         usbdevfs
         hub
 96-111: hiddev
         hid

root: /usr/src/linux # grep -i usb .config
# CONFIG_INPUT_IFORCE_USB is not set
# USB support
CONFIG_USB=y
CONFIG_USB_DEBUG=y
CONFIG_USB_DEVICEFS=y
# CONFIG_USB_BANDWIDTH is not set
CONFIG_USB_EHCI_HCD=y
CONFIG_USB_UHCI_ALT=y
# CONFIG_USB_OHCI is not set
# CONFIG_USB_SL811HS_ALT is not set
# CONFIG_USB_SL811HS is not set
# CONFIG_USB_AUDIO is not set
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_BLUETOOTH is not set
# CONFIG_USB_MIDI is not set
# CONFIG_USB_STORAGE is not set
# CONFIG_USB_STORAGE_DEBUG is not set
# CONFIG_USB_STORAGE_DATAFAB is not set
# CONFIG_USB_STORAGE_FREECOM is not set
# CONFIG_USB_STORAGE_ISD200 is not set
# CONFIG_USB_STORAGE_DPCM is not set
# CONFIG_USB_STORAGE_HP8200e is not set
# CONFIG_USB_STORAGE_SDDR09 is not set
# CONFIG_USB_STORAGE_SDDR55 is not set
# CONFIG_USB_STORAGE_JUMPSHOT is not set
# CONFIG_USB_ACM is not set
# CONFIG_USB_PRINTER is not set
CONFIG_USB_HID=y
CONFIG_USB_HIDINPUT=y
CONFIG_USB_HIDDEV=y
# CONFIG_USB_AIPTEK is not set
# CONFIG_USB_WACOM is not set
# CONFIG_USB_KBTAB is not set
# CONFIG_USB_POWERMATE is not set
# CONFIG_USB_DC2XX is not set
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_SCANNER is not set
# CONFIG_USB_MICROTEK is not set
# CONFIG_USB_HPUSBSCSI is not set
# CONFIG_USB_PEGASUS is not set
# CONFIG_USB_RTL8150 is not set
# CONFIG_USB_KAWETH is not set
# CONFIG_USB_CATC is not set
# CONFIG_USB_CDCETHER is not set
# CONFIG_USB_USBNET is not set
# CONFIG_USB_USS720 is not set
# USB Serial Converter support
# CONFIG_USB_SERIAL is not set
# CONFIG_USB_RIO500 is not set
# CONFIG_USB_AUERSWALD is not set
# CONFIG_USB_TIGL is not set
# CONFIG_USB_BRLVGER is not set
# CONFIG_USB_LCD is not set
# Support for USB gadgets
# CONFIG_USB_GADGET is not set

root: /usr/src/linux # grep -i apm .config
# CONFIG_APM is not set
root: /usr/src/linux # grep -i acpi .config
# ACPI Support
# CONFIG_ACPI is not set

Note that none of this (the kernel config) has changed since the upgrade to apcupsd 3.10.13.

[whiskeypriest]

I'm not sure what changed to elicit this sort of behavior from your system; mine still powers itself down completely when I run the apcupsd tests. I do, however, have a couple questions for you:

• Which kernel are you using? I'm assuming it's a 2.4 iteration from your configuration options, but I'd like to be sure.
• If you issue...

  Code: Select all

  # shutdown –h now

  ...does your system power itself down without user intervention?

I ask the second question because I notice that neither APM or ACPI are enabled in your kernel configuration. It's always been my understanding that either APM or ACPI was necessary for the system to power itself down without assistance.

If issuing a shutdown from the command line gives you the same issue as apcupsd, I'd try reconfiguring your kernel for either APM or ACPI as outlined in the KERNEL CONFIGURATION section of the HOWTO. Otherwise, please post back here.

[dmolavi]

issuing a 'shutdown -h now' requires me to power off the machine manually. this is really odd, since my (2.4) config hasn't been altered. i will enable the various ACPI/APM items in the kernel and post back with my results.

EDIT: it was a good excuse to go from 2.4.26 -> 2.4.27. so i upgraded, enabled APM, did a 'shutdown -h now' and it shutdown here's hoping that apcupsd will behave again

btw, speaking of apcupsd, why doesn't the cgi read the css file anymore? see http://www.nukedgallery.net/cgi-bin/apc ... ltimon.cgi for a demo.

[whiskeypriest]

If the system powers itself down when you issue the shutdown command, apcupsd should follow suit. You can always test it to be certain.

As for your issue with the css file, I'm afraid I can't offer any assistance...again, mine performs as expected and this is not an area of the program into which I've delved too deeply. The only recommendation I can give is to restart Apache, then restart apcupsd and see if that doesn't solve the problem.

Good luck with it...

[dmolavi]

as it's a production system, i don't want to have to power it down too much to verify functionality. i have noticed that apcupsd is behaving a bit more normally.

as for the css issue, beats me why it's not working. i saw something in the 3.10.15 changelog that might indicate a fix, so perhaps it will be fixed in the next release in portage.

thanks for your help

[whiskeypriest]

Understood about the production system...testing is simply a recommendation.

One other thought about your css issue: you could always back up your configuration files (apcupsd.conf, hosts.conf, et al.), unmerge apcupsd, wipe the relevant directories and remerge the whole thing. Do whatever you feel is most appropriate; this is just another shot in the dark.

Again, best of luck…

[ectospasm]

It's always been my understanding that either APM or ACPI was necessary for the system to power itself down without assistance.

This is not true... before I followed the HOWTO, I had neither APM nor ACPI configured in my kernel, and a "halt" or "shutdown -h now" command powered off my machine with no further user intervention. If this shouldn't have been the case, maybe it's hardware specific? I dunno...

[whiskeypriest]

That’s a good possibility. I’ve heard the same story enough times to believe that APM/ACPI are not necessary on all machines, which is why I usually advocate the “ain’t-broke-don’t-fix” policy where this issue is concerned. My machines range from the moderately-recent (e.g. P4) to the moderately-ancient (e.g. MMX) and all have required either APM or ACPI before powering themselves down unassisted…but that’s solely my experience, which is not quite so universal as I might have believed.

So far as the shutdown issue you mentioned in the HOWTO thread, did you reset your TIMEOUT value to zero in the apcupsd.conf file after testing? If all other parameters were above their triggers (and it sounds as though they were) this is the first item I’d check. I know checking this value seems to be my first response to a great many posts, but frankly it’s been the culprit more times than not where erratic shutdown behavior is concerned. I’ve forgotten to reset it after testing a few times myself, only to be reminded during the next electrical storm…anyway, if this isn’t the problem please post back here and we’ll see what we can do.

Thanks for the information (both about the AMD64 hardware and the power management) and I’m glad you found the guide to be useful.

[ectospasm]

I thought of that too, but since during the tests you described in the HOWTO set TIMEOUT to 30s, and with a 10-20min shutdown delay, TIMEOUT 0 could not be the culprit (I double-checked, just to make sure, and it was set to zero). And my BATTERYLEVEL and MINUTES are set to 5 and 3, respectively. My ONBATTERYDELAY is set to 6, but that still shouldn't affect the problem I experienced.

Here's my /etc/apcupsd/apcupsd.conf with comments removed:

Code: Select all

$ egrep -v "^#" /etc/apcupsd/apcupsd.conf
UPSCABLE usb
UPSTYPE usb
LOCKFILE /var/lock
ONBATTERYDELAY 6
BATTERYLEVEL 5
MINUTES 3
TIMEOUT 0
ANNOY 300
ANNOYDELAY 60
NOLOGON disable
KILLDELAY 0
NETSERVER on 
NISIP 0.0.0.0
NISPORT 3551
EVENTSFILE /var/log/apcupsd.events
EVENTSFILEMAX 10
UPSCLASS standalone
UPSMODE disable
STATTIME 0
STATFILE /var/log/apcupsd.status
LOGSTATS off
DATATIME 0

And my apcupsd version:

Code: Select all

# apcupsd --version
apcupsd 3.10.15 (04 August 2004) gentoo

[whiskeypriest]

Sorry, I misunderstood: I thought your system was shutting down immediately on power failure, but (as I understand you now) your system delayed shutdown for 10-20 minutes on power failure. The specs you posted initially indicate that you should have had roughly 75 minutes of runtime in the event of a power failure, hence your problem.

I’m away from my working installations right now, but have a couple quick questions/suggestions. For clarification: when running the final (full power failure, “real-world” test) did apcupsd initiate the shutdown on power failure, but the system delayed shutting itself down for ~20 minutes after the shutdown was issued? Or is it that the system ran on-battery for ~20 minutes before apcupsd issued the shutdown, when it should have run for ~75 minutes?

If the TIMEOUT value isn’t the issue (and everything else in your .conf file seems optimal for your setup) I can only think of three things which might be causing either of the above behaviors:

• apcupsd hasn’t been restarted, and is not utilizing the values in your current apcupsd.conf file.
• There’s an issue with apcupsd-3.10.15. Unless a new one’s been released since I synced yesterday afternoon, I have the latest stable version as 3.10.13. You might try backing up your configuration files and downgrading to that version.
• There’s an issue with your optional battery pack and/or its interaction with apcupsd. Unfortunately, I have absolutely no experience or insight with regards to the additional battery packs; that said, would it be possible that either a mechanism internal to the battery pack (some sort of “failover” switch?) or its interface with apcupsd is allowing for both power sources to be considered for monitoring purposes, but only one (e.g. the UPS unit itself) is used when meeting the parameters for a shutdown? This might explain the second scenario I described above, assuming that’s the one you’re actually experiencing. Just flailing with this…since you’re the owner of the unit, feel free to disabuse me of my misconceptions if I’m way the hell off base.

That’s about all I’ve got for the time being. If you can clarify what’s actually occurring and/or eliminate the first two suggestions I’ve listed, then we’ll do some further digging.

Let me know...

[ectospasm]

For clarification: when running the final (full power failure, “real-world” test) did apcupsd initiate the shutdown on power failure, but the system delayed shutting itself down for ~20 minutes after the shutdown was issued? Or is it that the system ran on-battery for ~20 minutes before apcupsd issued the shutdown, when it should have run for ~75 minutes?

It's the latter: apcupsd signals that power has failed, but doesn't initiate the shutdown until ~20 minutes later. Actually, here's the snippet from /var/log/apcupsd.events:

Code: Select all

Tue Oct 05 22:10:54 CDT 2004  Power failure.
Tue Oct 05 22:11:00 CDT 2004  Running on UPS batteries.
Tue Oct 05 22:43:17 CDT 2004  Reached remaining time percentage limit on batteries.
Tue Oct 05 22:43:17 CDT 2004  Initiating system shutdown!
Tue Oct 05 22:43:17 CDT 2004  User logins prohibited
Tue Oct 05 22:43:19 CDT 2004  apcupsd exiting, signal 15
Tue Oct 05 22:43:19 CDT 2004  apcupsd shutdown succeeded

I was wrong, it lasted for 00:32:17 before it shutdown. It's a reasonable time, but when the maximum Battery Run Time (as reported by "apcaccess status" and multimon.cgi) is 93.5 minutes, it makes me wish I had more time.

The battery pack is, as far as I can tell, completely dumb. The connector suggests that all it does is positive, negative, and ground (it's similar to the connector used for the internal battery of the main UPS unit). This is probably not a smart idea, but I'll disconnect the pack to see if the TIMELEFT drops significantly... The UPS beeped, and lo and behold, the TIMELEFT now says 23.4 min... after reconnecting, it's back up to 93.5 minutes (after peaking at 96.0 minutes).

The unit's output is only at 35% (I would have thought it would be more). This is not a serious problem, and I think I'm gonna test it again, just to double check that's what's going on. I should have been smarter and used the logs to determine uptime after power failure. I'll do that from now on.

[/code]

[whiskeypriest]

I'm used to seeing some loss of runtime when the battery actually comes under load, but I agree that ~60 minutes is a bit much. It doesn't sound like the auxiliary battery is at issue either. I'm at a loss.

Out of curiosity, you weren't doing anything that might have spiked the processors during that last test, were you? Obviously, my system has a bit less runtime if I'm compiling something monstrous or playing UT2K4 than it does when I'm basically idle. Just another thought.

Let me know how the second round of testing goes...

[ectospasm]

I'm used to seeing some Out of curiosity, you weren't doing anything that might have spiked the processors during that last test, were you? Obviously, my system has a bit less runtime if I'm compiling something monstrous or playing UT2K4 than it does when I'm basically idle. Just another thought.

I run dnetc, which keeps my processors at 100% (don't want to lose those spare cycles now, do we??). AFAIK there wasn't any heavy disk activity when the anomaly occurred. I'm gonna start the second test now, and I'm even gonna tax it more by burning a PHLAK ISO... That will probably bring it down faster, so I'll need to keep an eye on things... Seems burning only increases my UPS load by 3%...

[ectospasm]

This time it only lasted for 12min. I'm gonna ask apcupsd-users if anyone has seen this problem. Is there a way to search old apcupsd-users posts? I couldn't find one on the apcupsd site. I don't want to ask a question that's been answered too many times in the past...

[whiskeypriest]

Try this link...should take you to the search page for apcupsd-users at SF.

[ectospasm]

Try this link...should take you to the search page for apcupsd-users at SF.

Thanks, I found it on my own. I need to learn how to look harder. Of course, I couldn't find any posts with my own problem.

Incidentally, I tried the test again, but with my own doshutdown script. Here's the script in its current incarnation:

Code: Select all

# cat /etc/apcupsd/doshutdown
#!/bin/sh

BCHARGE=`/usr/sbin/apcaccess status | grep "BCHARGE"` 
TIMELEFT=`/usr/sbin/apcaccess status | grep "TIMELEFT"` 
echo `date` ${BCHARGE} >> /var/log/apcupsd.events
echo `date` ${TIMELEFT} >> /var/log/apcupsd.events
exit 0

Basically, what this does is write the BCHARGE and TIMELEFT values of "apcaccess status" to /var/log/apcupsd.events, just before shutdown. It returns an exit condition of 0, so apccontrol knows to continue doing what it needs to do. An earlier incantation of the script (the version that was used in my last test) gave these results:

Code: Select all

Thu Oct 07 23:32:54 CDT 2004  Power failure.
Thu Oct 07 23:33:00 CDT 2004  Running on UPS batteries.
Fri Oct 08 00:23:57 CDT 2004  Reached remaining time percentage limit on batteries.
Fri Oct 08 00:23:57 CDT 2004  Initiating system shutdown!
Fri Oct 08 00:23:57 CDT 2004  User logins prohibited
Fri Oct 8 00:23:57 CDT 2004
BCHARGE  : 031.0 Percent
TIMELEFT :   0.0 Minutes
Fri Oct 08 00:23:58 CDT 2004  apcupsd exiting, signal 15
Fri Oct 08 00:23:58 CDT 2004  apcupsd shutdown succeeded

You'll notice that the TIMELEFT variable is at 0.0min, so that's a reason why it's shutting down so fast. This time, however, it lasted over 50min, which I find to be very acceptible. I think it's just another case of scrutinizing a problem causes it to disappear. (-:

Thanks for your help... I'll keep an eye on things and see if things don't improve...

[whiskeypriest]

Glad you found a workable solution, and I also hope things improve for you.

Sorry if my last reply seemed abrupt, but I was pressed for time and wanted to get the information to you A.S.A.P. Didn’t have time for a full-blown search at SF myself, otherwise I might have saved you the trouble.

I had asked about your system load at test-time to make certain there wasn’t a discrepancy between the system’s state while monitoring versus its state while testing. However, you’ve established that wasn’t the issue either.

Please keep me posted if you find anything new on this…I’m now curious as to whether there’s a disconnect between my reported runtime and the system’s actual life under battery. After the initial installation, I’m accustomed to simply testing each new iteration of apcupsd with a TIMEOUT value of 30 and letting it go if everything performs as expected. I’ll test my main system tonight with a full power-failure and report back here with the results.

[ectospasm]

Things haven't gotten better, they've gotten worse (as far as I can tell). Here's the latest data from an actual outage (i.e. it wasn't a test, the power actually failed):

Code: Select all

Sun Oct 10 10:17:32 CDT 2004  Power failure.
Sun Oct 10 10:17:38 CDT 2004  Running on UPS batteries.
Sun Oct 10 10:32:03 CDT 2004  Reached remaining time percentage limit on batteries.
Sun Oct 10 10:32:03 CDT 2004  Initiating system shutdown!
Sun Oct 10 10:32:03 CDT 2004  User logins prohibited
Sun Oct 10 10:32:04 CDT 2004 BCHARGE : 090.0 Percent
Sun Oct 10 10:32:04 CDT 2004 TIMELEFT : 0.0 Minutes
Sun Oct 10 10:32:07 CDT 2004  apcupsd exiting, signal 15
Sun Oct 10 10:32:07 CDT 2004  apcupsd shutdown succeeded

It only lasted about 15 minutes. That doesn't really bother me that much; what does bother me is that the time percentage left does not seem to match the remaining battery percentage. If the battery is at 90.0% charge, I should have much more than 0.0 minutes left, especially since this is a brand new UPS and brand new expansion battery pack.

I'm thinking I should mail apcupsd-users, but wanted your thoughts on it whiskeypriest.