Best FS for tolerance of power loss

mounty1 · l33t Joined: 06 Jul 2006 Posts: 934 Location: Queensland

My day-job involves a small PC which is a TV device similar to Apple TV, Telstra TV (in Australia) etc. One problem we face is that it will be installed in situations in which power is unreliable and can be lost without warning. This is very much less than ideal but as the PC has solid-state storage, I was thinking to make the file system F2FS rather than EXT4 as it is now. But I don't know if F2FS still caches like EXT4, and if that can be disabled. Then maybe LFS would be more suitable to this application. Can anyone advise?
_________________
Michael Mounteney

NeddySeagoon

mounty1,

mounty1 · l33t Joined: 06 Jul 2006 Posts: 934 Location: Queensland

I appreciate that you meant to PM but you replied in chat.

The FS has to be RW because we do change some files. We could segregate these onto a separate RW partition but that task would require some maintenance:

MySQL database
Firefox hierarchy (incl. cookies)
Application logs
JVM logs

Each of those could be made to flush more granularly, although I don't know now how to make Firefox do so, so we'd rather just do it at the filesystem level.
_________________
Michael Mounteney

figueroa · Posted: Sat Mar 20, 2021 3:58 am Post subject:

Seems like an obvious application for a UPS. I have six UPSs in my home, and I have insisted on them at the school I support that is remote to me. At the school, when we've had surprise extended power outages, ext4 has served us well.
_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi

Zucca · Posted: Sat Mar 20, 2021 7:10 am Post subject:

Was someone porting HAMMER from DragonFlyBSD to Linux?
From what I've read HAMMER can survive unexpected power losses pretty well.
But since that's not (probably) an option here, I'd go the route of mounting root ro, like Neddy suggested.

People say XFS is very stable fs. XFS has tons of tuning knobs, but I'm no expert at XFS tuning. Maybe someone else pops up to deliver some knowledge..?
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--

NeddySeagoon · Posted: Sat Mar 20, 2021 10:10 am Post subject:

mounty1,

If write speed in not an issue, ext4 with journaling everything. The default is metadata only. Now everything is written twice. Once to the journal and again for real.
Use the mount option sync. This is horrendous as write caching is turned off.
The drive may do some but LInux won't.

You will always lose whatever is in 'dirty' buffers, regardless of the filesystem in use.

I would still go for a read only code space, so the system will always come back after a power outage and keep the data in a separate filesystem.
Maybe two filesystems. One that is updated rarely and usually mounted read only. Part of the update process would be a remount read/write.
Another that has to be read/write most of the time.

Its really damage limitation, not prevention. As figueroa says, you really need battery backup to force a clean shutdown when the power goes out.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

C5ace · Posted: Sat Mar 20, 2021 10:31 am Post subject:

Had the same problems. Bought UPS from Umart.com.au for $119.00. Battery lasts about 30 minuts with monitor, router and phone.
_________________
Observation after 30 years working with computers:
All software has known and unknown bugs and vulnerabilities. Especially software written in complex, unstable and object oriented languages such as perl, python, C++, C#, Rust and the likes.

wjb · Posted: Sat Mar 20, 2021 1:16 pm Post subject:

The problem with adding a UPS to a business system is that it generally means additional paperwork is required to cover off safety issues. Depends on the local regs obviously, but I'd be surprised if you could get away with nothing.

Edit

For Australia, a quick google would suggest you start with "AS 62040.1:2019" and see where that takes you.

C5ace · Posted: Sat Mar 20, 2021 3:31 pm Post subject:

Fitzcarraldo · Posted: Sat Mar 20, 2021 4:20 pm Post subject:

mounty1,

If the small PC is like an Apple TV then its power consumption is presumably around 3 to 6 Watts. Is that the case? If it is, I can recommend the iLEPO ECO PLUS 412P mini UPS (output 15 Watts max.) if the small PC requires a PSU for 5V/9V/12V/15V/24V: https://www.amazon.co.uk/dp/B07R4FGWDX/

I have one protecting my hub, as I cannot connect it to my main UPS because the hub is in a different room to my server. It weights 400 grammes and is a little bigger than my hand. It's a good piece of kit.
_________________
Clevo W230SS: amd64, VIDEO_CARDS="intel modesetting nvidia".
Compal NBLB2: ~amd64, xf86-video-ati. Dual boot Win 7 Pro 64-bit.
OpenRC udev elogind & KDE on both.
Fitzcarraldo's blog

Zucca · Posted: Sat Mar 20, 2021 4:30 pm Post subject:

Well... UPSs base their working principle on batteries.
I think it's more the batteries which are the cause of concern in some environments. UPS batteries are normally sealed lead acid batteries. But during changing a trace amount of pure oxygen and hydrogen can be produced. Mixture of those two is very flammable, as many of you probably know. However normally sealed batteries do their job and seal any leaks (I think it's very rare for sealed battery to leak hydrogen).
But in case of short circuit the batteries can deliver huge currents. This may be a reason for some places don't to allow UPSs.

Side note: I once worked on a place where we had two rooms for UPS system.
First room was the smart part: switched mode power supply and all the sensory parts which look for any fluctuations in the input power from normal.
The second room was the power reserve: half full of non-sealed lead acid batteries, in series (20x12V), with 100Amp fuses. There was also active ventilation. When an electrician went there to change some of the batteries something happened. Probably a spark and then right after BOOM. That guy was lucky to have his fingers still intact. His hearing took some damage, but what I heard he got treated quickly and the damage wasn't permanent.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--

Hu · Moderator Joined: 06 Mar 2007 Posts: 21631

figueroa · Posted: Sat Mar 20, 2021 5:19 pm Post subject:

I've had several sealed lead acid batteries crack and bulge to the point of being able to see the cells. Didn't leak a drop. Maybe they did leak some gas. I'm not aware of any regulation in the US against using UPSs in common business environments. The fire inspectors seem to think they are a much better deal than using ordinary surge suppressors or power strips.

I do know that AT&T and Comcast routinely install UPSs in homes and businesses as part of certain packages to ensure 24/7 phone and/or network operation.
_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi

Zucca · Posted: Sat Mar 20, 2021 5:34 pm Post subject:

mounty1 · l33t Joined: 06 Jul 2006 Posts: 934 Location: Queensland

Thanks for all the replies but I didn't make clear some details of the application.

The device is this which we buy in bulk for about AU$200 each. They are installed in staff accommodation in remote mine camps and the users are FIFO (fly in, fly out) miners who just wanna watch their programming OK. They are fitted behind the TV set and there's no space for a UPS and bearing in mind that these rooms can easily reach 50 degrees Celcius in the daytime, a UPS with its attendant risk of battery leakage (however slight) is a non-starter. There's also the matter of how you mount a relatively heavy and bulky UPS on the wall behind the TV. The PC has a 12 V 3 A power pack so it can theoretically draw 36 W although typically I suppose 20 W. When we install, we are doing so in the 100s or even 1000s so every cost item, including the technician's time in each room, has to be factored-in.

Even a capacitative UPS is a significant complication as we'd need one we could fit inline with the power pack, i.e., has the same coaxial power connections as the PC.

As for power loss, that's out of our control and is just a feature of these remote locations. We have to live with it.
_________________
Michael Mounteney

Goverp · Advocate Joined: 07 Mar 2007 Posts: 2006

Ah, it's got "64GB eMMC 5.1 internal storage". I'd consider f2fs; it's a journalling file system, so IIUC it will write everything (but allowing for cache, of course).

You can consider the tips for enabling checksums in the Arch wiki on f2fs. I'm not sure what they're protecting against though.

A particular drawback is that fsck.f2fs doesn't understand the rules: it will only run against an unmounted filesystem, not a r/o one. So if you want an f2fs root filesystem, you need an initramfs to do the fsck. Alternatively, have say ext4 /root which you avoid updating, and f2fs system, say /home, sitting below it, so that your boot processing can fsck before mounting. But probably initramfs is the way forward. I guess as that looks pretty modern, it's UEFI/GPT setup, so you can put the necessary kernel and (if you do external) initramfs on the ESP. Which is FAT, and pretty simple, so I'd hope it would survive a half-arsed update getting cut halfway through. You can make that sort-of atomic by loading the new kernel as "vmlinuz.new" and then using mv (current)->.old, mv .new->(current)

FWIW, I recently had my fsf2 root refuse to boot. I had to find a way to fsck it. As documented in the Arch wiki, you need contemporary version of fsck.f2fs to do it, so I decided to build an initramfs that both ran fsck and provided a busybox recovery environment. Once I got that, fsck indeed fixed the disk - lots of scary messages, but I think that's just an inappropriate message level, but it just worked.

IIUC, the issue with non-contemporary fsck is much reduced now - the fs metadata records the kernel version used at the last fsck, and it used to decide that a new kernel version required an fsck (which takes a minute on my 480 GB laptop SSD); that no longer happens.
_________________
Greybeard

C5ace · Posted: Sun Mar 21, 2021 10:37 am Post subject:

szatox · Advocate Joined: 27 Aug 2013 Posts: 3135

Can you partition it?
Make a readonly, squashed installer on one partition, and put the real thing on another. User data can be separated too (preferably in initramfs if you can afford lose them at the end of the session), and overlayed on top of your system.

If things go very wrong, just use the installer to do a "factory reset" on the other partition. Could even download new image from company's server.

Now, if you don't care about updates, you can just as well put the whole system on partition one and have the rest for user's data. Again, overlay allows you to merge those 2 areas, in a way that is easy for you and transparent to the applications. If the user data FS gets damaged beyond repair, just reformat and reboot.

steve_v · Guru Joined: 20 Jun 2004 Posts: 388 Location: New Zealand

Really, all this talk of UPS and onboard batteries doesn't sound very productive to me. It's added cost, complexity, another part to wear out, and if you're talking lithium technology it's a potential fire hazard. There are zillions of embedded devices out there that do just fine without them.
Hint: They usually don't have a rw root partition, and often writes go not only to a separate partition, but to entirely separate media.

In reality ext4 survives power-loss as well as any other though, and that means 98% of the time all you get is a fsck on startup.
The 2% can be handled by keeping / mounted ro (or even using an image as the root FS and loading it into a ramdisk from the initrd, i.e like a liveCD distro) and providing a mechanism to repair, restore, or recreate the volatile user storage area in the unlikely event it gets hosed.
If you don't need blazing-fast write performance (and from the sound of your application I don't see why you would), you likely want to mount that volatile partition sync and disable disk caching too.

I'll second the squashfs system restore as well, I've used this in the (distant) past with great success.
As an added bonus, it's also an ideal candidate for tying into a proper (i.e. tested system image) update mechanism.

mounty1 · l33t Joined: 06 Jul 2006 Posts: 934 Location: Queensland

figueroa · Posted: Mon Mar 22, 2021 3:51 am Post subject:

How many boot failures do you have as a result of unreliable power?

In other words, is this a use-case problem in need of a solution, or conjecture? In the event of a reboot failure, is remote intervention even possible? In case of failure, can you just ship an immediate replacement and fix the failed device in the shop?

Can you have two bootable partitions? One to run on, and one to restore from backup in the event of resulting corruption? If not, I think a read-only OS is called for that is made RW only when updates are needed. And then, you still have the possibility of power failure during updates.

Can you do what you need to do with Debian-stable? (few updates) Push out out updates with a "stage4" tarball equivalent? Install the update on the alternate OS partition, make any necessary tweaks, swap which is next to boot (GRUB or whatever), reboot, done. Script the entire process.

I can see the beauty of booting from removable media (like live USB -- computer on a stick). Keep spare media with every device. Therefore, is this the right device?
_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi

steve_v · Guru Joined: 20 Jun 2004 Posts: 388 Location: New Zealand

Zucca · Posted: Mon Mar 22, 2021 11:06 am Post subject:

While browsing the net I stumbled upon this article. Quite related I think. It seems to cover some of the proposed solutions as suggested on this topic.

However. We're looking for fault tolerant filesystem? Right? After deciding which, then move to maybe mounting it ro and other measures.

Like I mentioned earlier xfs is very mature filesystem which is still in active development. Tuning it properly might give you very fault tolerant fs. I'm no expert in this, but xfs has its reputation.

Also btrfs has its DUP data profile where it stores all the data twice... Then you'd have a possibility recover your data if only the other half is corrupted, BUT it may very well be useless since SSD controllers tend to perform data deduplication (and because of that btrfs doesn't use DUP profile on single SSDs by default). If we'd only have the specs of that eMMC controller. ;)
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--

steve_v · Guru Joined: 20 Jun 2004 Posts: 388 Location: New Zealand

C5ace · Posted: Mon Mar 22, 2021 1:06 pm Post subject: