Best FS for tolerance of power loss

[mounty1]

My day-job involves a small PC which is a TV device similar to Apple TV, Telstra TV (in Australia) etc. One problem we face is that it will be installed in situations in which power is unreliable and can be lost without warning. This is very much less than ideal but as the PC has solid-state storage, I was thinking to make the file system F2FS rather than EXT4 as it is now. But I don't know if F2FS still caches like EXT4, and if that can be disabled. Then maybe LFS would be more suitable to this application. Can anyone advise?

[NeddySeagoon]

mounty1,

My day-job involves a small PC which is a TV device similar to Apple TV, Telstra TV (in Australia) etc. One problem we face is that it will be installed in situations in which power is unreliable and can be lost without warning. This is very much less than ideal but as the PC has solid-state storage, I was thinking to make the file system F2FS rather than EXT4 as it is now. But I don't know if F2FS still caches like EXT4, and if that can be disabled. Then maybe LFS would be more suitable to this application. Can anyone advise?

PM because I'm responding with my systems engineering hat on rather than answering the question and I don't want to take your topic out of the unanswered posts search.

Reading assumptions into what you didn't write, I need to ask why you need a read/write root filesystem in the first place?
Maybe the solution to your problem is to have root mounted read only except for software updates, rather than a power loss tolerant filesystem.
Logs can go to tmpfs if you want to read them and /dev/null if you don't.

[mounty1]

I appreciate that you meant to PM but you replied in chat.

The FS has to be RW because we do change some files. We could segregate these onto a separate RW partition but that task would require some maintenance:

• MySQL database
• Firefox hierarchy (incl. cookies)
• Application logs
• JVM logs

Each of those could be made to flush more granularly, although I don't know now how to make Firefox do so, so we'd rather just do it at the filesystem level.

[figueroa]

Seems like an obvious application for a UPS. I have six UPSs in my home, and I have insisted on them at the school I support that is remote to me. At the school, when we've had surprise extended power outages, ext4 has served us well.

[Zucca]

Was someone porting HAMMER from DragonFlyBSD to Linux?
From what I've read HAMMER can survive unexpected power losses pretty well.
But since that's not (probably) an option here, I'd go the route of mounting root ro, like Neddy suggested.

People say XFS is very stable fs. XFS has tons of tuning knobs, but I'm no expert at XFS tuning. Maybe someone else pops up to deliver some knowledge..?

[NeddySeagoon]

mounty1,

If write speed in not an issue, ext4 with journaling everything. The default is metadata only. Now everything is written twice. Once to the journal and again for real.
Use the mount option sync. This is horrendous as write caching is turned off.
The drive may do some but LInux won't.

You will always lose whatever is in 'dirty' buffers, regardless of the filesystem in use.

I would still go for a read only code space, so the system will always come back after a power outage and keep the data in a separate filesystem.
Maybe two filesystems. One that is updated rarely and usually mounted read only. Part of the update process would be a remount read/write.
Another that has to be read/write most of the time.

Its really damage limitation, not prevention. As figueroa says, you really need battery backup to force a clean shutdown when the power goes out.

[C5ace]

Had the same problems. Bought UPS from Umart.com.au for $119.00. Battery lasts about 30 minuts with monitor, router and phone.

[wjb]

The problem with adding a UPS to a business system is that it generally means additional paperwork is required to cover off safety issues. Depends on the local regs obviously, but I'd be surprised if you could get away with nothing.

Edit

For Australia, a quick google would suggest you start with "AS 62040.1:2019" and see where that takes you.

[C5ace]

The problem with adding a UPS to a business system is that it generally means additional paperwork is required to cover off safety issues. Depends on the local regs obviously, but I'd be surprised if you could get away with nothing.

No testing and tagging of personal computer and connected equipment power cords are rewired in Australia.

[Fitzcarraldo]

mounty1,

If the small PC is like an Apple TV then its power consumption is presumably around 3 to 6 Watts. Is that the case? If it is, I can recommend the iLEPO ECO PLUS 412P mini UPS (output 15 Watts max.) if the small PC requires a PSU for 5V/9V/12V/15V/24V: https://www.amazon.co.uk/dp/B07R4FGWDX/

I have one protecting my hub, as I cannot connect it to my main UPS because the hub is in a different room to my server. It weights 400 grammes and is a little bigger than my hand. It's a good piece of kit.

[Zucca]

Well... UPSs base their working principle on batteries.
I think it's more the batteries which are the cause of concern in some environments. UPS batteries are normally sealed lead acid batteries. But during changing a trace amount of pure oxygen and hydrogen can be produced. Mixture of those two is very flammable, as many of you probably know. However normally sealed batteries do their job and seal any leaks (I think it's very rare for sealed battery to leak hydrogen).
But in case of short circuit the batteries can deliver huge currents. This may be a reason for some places don't to allow UPSs.

Side note: I once worked on a place where we had two rooms for UPS system.
First room was the smart part: switched mode power supply and all the sensory parts which look for any fluctuations in the input power from normal.
The second room was the power reserve: half full of non-sealed lead acid batteries, in series (20x12V), with 100Amp fuses. There was also active ventilation. When an electrician went there to change some of the batteries something happened. Probably a spark and then right after BOOM. That guy was lucky to have his fingers still intact. His hearing took some damage, but what I heard he got treated quickly and the damage wasn't permanent.

[Hu]

One problem we face is that it will be installed in situations in which power is unreliable and can be lost without warning.

Is power unreliable because the local power company is unreliable, or is it unreliable because you have users who think that abruptly turning off a power strip is a good idea? If the former, solutions posted earlier in the thread look reasonable to me. If the latter, perhaps some user training, in conjunction with some basic hardware mods (like a locked cabinet) to reinforce the lesson, would be viable. Teach users to stop turning off power suddenly, instead of trying to mitigate the consequences.

[figueroa]

I've had several sealed lead acid batteries crack and bulge to the point of being able to see the cells. Didn't leak a drop. Maybe they did leak some gas. I'm not aware of any regulation in the US against using UPSs in common business environments. The fire inspectors seem to think they are a much better deal than using ordinary surge suppressors or power strips.

I do know that AT&T and Comcast routinely install UPSs in homes and businesses as part of certain packages to ensure 24/7 phone and/or network operation.

[Zucca]

I've had several sealed lead acid batteries crack and bulge to the point of being able to see the cells. Didn't leak a drop. Maybe they did leak some gas.

This is common. Not once I've needed to pry out the batteries using something like crowbar or even disassemble the covers of a UPS to get the batteries out. The interesting part is can they leak gas when the rupture happens? I don't have any knowledge on that.
(Of course manufacturers say they don't leak because there's no liquid, but gel (or some acid substrate infused in fiberglass), and that the batteries are... yes, sealed.
But anyway. Some companies might consider big batteries a possible... threat. But I bet those are in few.)

EDIT: Back to topic:

mounty1, as Fitzcarraldo asked and I assume too that your setup there uses quite little power.
There are UPSs on the market, which use super/ultra capacitors instead of batteries. Capacitors can tolarate millions of charge-discharge cycles, but their energy capacity is much lower than regular battery of similar size. However that short amount of time your setup has power should be enough to safely power it off. Or at least sync and remount ro all the filesystems.
EDIT: Although... Prices are somewhat high, but those are practically maintenance free.
Linky for example - 12-24Volt versions.

EDIT: Typofix.

[mounty1]

Thanks for all the replies but I didn't make clear some details of the application.

The device is this which we buy in bulk for about AU$200 each. They are installed in staff accommodation in remote mine camps and the users are FIFO (fly in, fly out) miners who just wanna watch their programming OK. They are fitted behind the TV set and there's no space for a UPS and bearing in mind that these rooms can easily reach 50 degrees Celcius in the daytime, a UPS with its attendant risk of battery leakage (however slight) is a non-starter. There's also the matter of how you mount a relatively heavy and bulky UPS on the wall behind the TV. The PC has a 12 V 3 A power pack so it can theoretically draw 36 W although typically I suppose 20 W. When we install, we are doing so in the 100s or even 1000s so every cost item, including the technician's time in each room, has to be factored-in.

Even a capacitative UPS is a significant complication as we'd need one we could fit inline with the power pack, i.e., has the same coaxial power connections as the PC.

As for power loss, that's out of our control and is just a feature of these remote locations. We have to live with it.

[Goverp]

Ah, it's got "64GB eMMC 5.1 internal storage". I'd consider f2fs; it's a journalling file system, so IIUC it will write everything (but allowing for cache, of course).

You can consider the tips for enabling checksums in the Arch wiki on f2fs. I'm not sure what they're protecting against though.

A particular drawback is that fsck.f2fs doesn't understand the rules: it will only run against an unmounted filesystem, not a r/o one. So if you want an f2fs root filesystem, you need an initramfs to do the fsck. Alternatively, have say ext4 /root which you avoid updating, and f2fs system, say /home, sitting below it, so that your boot processing can fsck before mounting. But probably initramfs is the way forward. I guess as that looks pretty modern, it's UEFI/GPT setup, so you can put the necessary kernel and (if you do external) initramfs on the ESP. Which is FAT, and pretty simple, so I'd hope it would survive a half-arsed update getting cut halfway through. You can make that sort-of atomic by loading the new kernel as "vmlinuz.new" and then using mv (current)->.old, mv .new->(current)

FWIW, I recently had my fsf2 root refuse to boot. I had to find a way to fsck it. As documented in the Arch wiki, you need contemporary version of fsck.f2fs to do it, so I decided to build an initramfs that both ran fsck and provided a busybox recovery environment. Once I got that, fsck indeed fixed the disk - lots of scary messages, but I think that's just an inappropriate message level, but it just worked.

IIUC, the issue with non-contemporary fsck is much reduced now - the fs metadata records the kernel version used at the last fsck, and it used to decide that a new kernel version required an fsck (which takes a minute on my 480 GB laptop SSD); that no longer happens.

[C5ace]

Thanks for all the replies but I didn't make clear some details of the application.

The device is this which we buy in bulk for about AU$200 each. They are installed in staff accommodation in remote mine camps and the users are FIFO (fly in, fly out) miners who just wanna watch their programming OK. They are fitted behind the TV set and there's no space for a UPS and bearing in mind that these rooms can easily reach 50 degrees Celcius in the daytime, a UPS with its attendant risk of battery leakage (however slight) is a non-starter. There's also the matter of how you mount a relatively heavy and bulky UPS on the wall behind the TV. The PC has a 12 V 3 A power pack so it can theoretically draw 36 W although typically I suppose 20 W. When we install, we are doing so in the 100s or even 1000s so every cost item, including the technician's time in each room, has to be factored-in.

Even a capacitative UPS is a significant complication as we'd need one we could fit inline with the power pack, i.e., has the same coaxial power connections as the PC.

As for power loss, that's out of our control and is just a feature of these remote locations. We have to live with it.

You probably need is very small UPS that replaces the original power supply, plugs into 240Volts wall socket, supplies 12V 3Ah constantly and for a minute or so after power loss. Then sends a shut down signal to to Ubuntu.

Alternatively, if space permits ask the manufacturer to place a small 12V battery inside or to the back of the case. Maybe something like this:
https://www.alibaba.com/product-detail/ ... 3889vwSuTY

The question is how many do you need at which cost?

[szatox]

Can you partition it?
Make a readonly, squashed installer on one partition, and put the real thing on another. User data can be separated too (preferably in initramfs if you can afford lose them at the end of the session), and overlayed on top of your system.

If things go very wrong, just use the installer to do a "factory reset" on the other partition. Could even download new image from company's server.

Now, if you don't care about updates, you can just as well put the whole system on partition one and have the rest for user's data. Again, overlay allows you to merge those 2 areas, in a way that is easy for you and transparent to the applications. If the user data FS gets damaged beyond repair, just reformat and reboot.

[steve_v]

Really, all this talk of UPS and onboard batteries doesn't sound very productive to me. It's added cost, complexity, another part to wear out, and if you're talking lithium technology it's a potential fire hazard. There are zillions of embedded devices out there that do just fine without them.
Hint: They usually don't have a rw root partition, and often writes go not only to a separate partition, but to entirely separate media.

In reality ext4 survives power-loss as well as any other though, and that means 98% of the time all you get is a fsck on startup.
The 2% can be handled by keeping / mounted ro (or even using an image as the root FS and loading it into a ramdisk from the initrd, i.e like a liveCD distro) and providing a mechanism to repair, restore, or recreate the volatile user storage area in the unlikely event it gets hosed.
If you don't need blazing-fast write performance (and from the sound of your application I don't see why you would), you likely want to mount that volatile partition sync and disable disk caching too.


I'll second the squashfs system restore as well, I've used this in the (distant) past with great success.
As an added bonus, it's also an ideal candidate for tying into a proper (i.e. tested system image) update mechanism.

if you don't care about updates

Please don't be one of those tools who unleashes an internet-connected device on the world and never provides updates.

[mounty1]

Really, all this talk of UPS and onboard batteries doesn't sound very productive to me.

Indeed, and as I've stated, it ain't gonna happen so please can we move on from talk of capacitative UPSs or any other type.

I reckon the best bet is to turn off all caching on ext4. I do have experience of using F2FS as a root filesystem on two of my home machines, and it is problematical, esp. regarding fsck.f2fs not checking RO-mounted systems.

[figueroa]

How many boot failures do you have as a result of unreliable power?

In other words, is this a use-case problem in need of a solution, or conjecture? In the event of a reboot failure, is remote intervention even possible? In case of failure, can you just ship an immediate replacement and fix the failed device in the shop?

Can you have two bootable partitions? One to run on, and one to restore from backup in the event of resulting corruption? If not, I think a read-only OS is called for that is made RW only when updates are needed. And then, you still have the possibility of power failure during updates.

Can you do what you need to do with Debian-stable? (few updates) Push out out updates with a "stage4" tarball equivalent? Install the update on the alternate OS partition, make any necessary tweaks, swap which is next to boot (GRUB or whatever), reboot, done. Script the entire process.

I can see the beauty of booting from removable media (like live USB -- computer on a stick). Keep spare media with every device. Therefore, is this the right device?

[steve_v]

I reckon the best bet is to turn off all caching on ext4.

Well, it's certainly the laziest solution. Partitioning and a ro root would be more reliable, but if you intend to run bog-standard Ubuntu that's up to you.
Be sure write-caching is disabled on the drive as well, and if your chosen storage has a DRAM buffer it doesn't do anything funky if it looses power. I'm not sure what the situation is right now, but historically cheap SSDs could be pretty abysmal for this, in some cases even bricking the device completely.
The device you mention gives very little detail on the onboard storage beyond "it's eMMC". If that's what you'll be using then given that the box is marketed as an appliance it's probably fairly resilient to power loss... Probably. But without knowing what the controller is or testing it for yourself, who really knows. Some eMMC controllers have PLP, some don't.

On the topic of bugger-all detailed specs, I'd also be testing to ensure that the "keeps NEO Z83-4U Ubuntu operational even under the most strenuous tasks." thermal solution accounts for the kind of ambient temperatures you're liable to find at an Aussie mine site... Proper industrial kit is usually good for at least 50C continuous, but with Chineseum it's anyone's guess.

is this a use-case problem in need of a solution, or conjecture?

If this site is anything like the mining sites I've been involved in, poor power quality and availability is not conjecture at all. It's a complete pain in the ass. As for whether that creates problems for this particular device, the only way to be sure is to try it.



Admittedly my day job is more industrial control systems than entertainment units, but personally I wouldn't be very comfortable deploying anything with consumer-grade storage to a mine at all, at least not without extensive testing, power filtering and surge protection, and a non-volatile recovery area. At the very least I'd be looking to an industrial SSD, a read-only system partition, and swapping out the PSU for something that isn't 100% cost-down Chineseium.
But hey, I'm just an industrial sparky, and my stuff has to work all the time whilst being operated by monkeys and hit with rocks... With the occasional lightning strike thrown in for good measure.
What level of reliability is actually needed here is not my call.

[Zucca]

While browsing the net I stumbled upon this article. Quite related I think. It seems to cover some of the proposed solutions as suggested on this topic.

However. We're looking for fault tolerant filesystem? Right? After deciding which, then move to maybe mounting it ro and other measures.

Like I mentioned earlier xfs is very mature filesystem which is still in active development. Tuning it properly might give you very fault tolerant fs. I'm no expert in this, but xfs has its reputation.

Also btrfs has its DUP data profile where it stores all the data twice... Then you'd have a possibility recover your data if only the other half is corrupted, BUT it may very well be useless since SSD controllers tend to perform data deduplication (and because of that btrfs doesn't use DUP profile on single SSDs by default). If we'd only have the specs of that eMMC controller. ;)

[steve_v]

Quite related I think.

Pretty much exactly what I was just getting at, but better

We're looking for fault tolerant filesystem? Right? After deciding which, then move to maybe mounting it ro and other measures.

Indeed. But then almost any mature journaling filesystem choice is fault-tolerant, and comes with an option to force sync writes and/or full journaling in case of an unreliable power supply...
So assuming that the underlying storage doesn't lie about when data has actually made it to non-volatile media, which filesystem to use is a matter of personal preference as much as anything else. vOv

The more important question to my mind is whether or not that assumption is valid, because that's what's going to dictate the need (or not) for special filesystem features or recovery mechanisms.
Any discussion of filesystem reliability will inevitably end up talking about the hardware, for obvious reasons. Filesystems don't exist in a vacuum, and the system is only as reliable as the weakest link.

If we'd only have the specs of that eMMC controller.

If only we did, in which case we could find out whether or not taking additional measures to protect the filesystem from failures at the firmware/hardware level is warranted.
Since we don't know, IMO the sensible thing to do is to take at least the easiest and least expensive of those measures anyway - i.e. a r/o root filesystem. Deciding whether to go further is a cost/benefit calculation, and that's difficult to do from here.

[C5ace]

Thanks for all the replies but I didn't make clear some details of the application.

The device is this which we buy in bulk for about AU$200 each. They are installed in staff accommodation in remote mine camps and the users are FIFO (fly in, fly out) miners who just wanna watch their programming OK. They are fitted behind the TV set and there's no space for a UPS and bearing in mind that these rooms can easily reach 50 degrees Celcius in the daytime, a UPS with its attendant risk of battery leakage (however slight) is a non-starter. There's also the matter of how you mount a relatively heavy and bulky UPS on the wall behind the TV. The PC has a 12 V 3 A power pack so it can theoretically draw 36 W although typically I suppose 20 W. When we install, we are doing so in the 100s or even 1000s so every cost item, including the technician's time in each room, has to be factored-in.

Even a capacitative UPS is a significant complication as we'd need one we could fit inline with the power pack, i.e., has the same coaxial power connections as the PC.

As for power loss, that's out of our control and is just a feature of these remote locations. We have to live with it.

Why not use a small smart TV and a wireless keyboard and mouse?

I bought a last year from Aldi a 32" one for $222.00 Aussi. Works like a charm. Ethernet, WiFi, USB PVR, USB Playback, Browser. etc. etc. No worries about disk corruption on powerloss. https://bauhn.com.au/32-hd-smart-tv-april20/.

JbHifi has this on sale for $275.00: https://www.jbhifi.com.au/products/blau ... android-tv.