| View previous topic :: View next topic |
| Author |
Message |
axl
Veteran
Joined: 11 Oct 2002
Posts: 1144
Location: Romania
|
 Posted: Fri Mar 17, 2017 1:05 pm Post subject: Question about the safety of data |
 |
|
Hello folks.
I've been watching these videos about zfs/btrfs. Which seem to raise a question in my mind that I couldn't possibly answer alone at this point. I've tried to figure it out alone, but I am unsure of it. I really hope this is the offtopic forum because here goes. Let me describe my setup:
2 x 4 Tb drives. I've put them in a raid1 array, formatted xfs and that's all she wrote.
OK. According to the videos I've seen, this setup is vulnerable to data alteration over time. I don't really know how that's a thing. Isn't a FS self consistent with itself? According to this theory, if a place on a hard-disk just decides to have 1's instead of 0's in a particular sector, in a raid1 array, with xfs on top of it (AGAIN just because ONE of the disks decided on it's own that in that particular sector it's a 1 and not a 0, the other disk would have 0) and somehow the end result of my data would be a zero, and not a 1. this theory is based in the idea that harddrives return other data that you put into them and they dont return an error for it.
i guess _some_ harddrives degrade over time. flash does it all the time.
so, going down the food chain. since i haven't used degradable storage options, if a drive decides to return other data than you actually input into the drive, would xfs know? i'm given to understand... no. If you have a file with letters numbers 10 in it, and disk somehow changes data inside the file, without updating the rest of xfs. data just degrades on the drive and somehow ends up 01. would xfs know? i'm given to understand no. assuming ofc, the drive doesn't complain to bios. which again, i'm given to understand, most cheap drives dont.
second question, going down the food chain. would md raid 1 know ? i'm given to understand no.
third question. would btrfs know ? i'm given to understand yes. same for zfs. is that correct ? did i understand correctly ? |
|
| Back to top |
|
 |
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54237
Location: 56N 3W
|
| Posted: Fri Mar 17, 2017 1:52 pm Post subject: |
|
|
axl,
| axl wrote: | | OK. According to the videos I've seen, this setup is vulnerable to data alteration over time. I don't really know how that's a thing. Isn't a FS self consistent with itself? According to this theory, if a place on a hard-disk just decides to have 1's instead of 0's in a particular sector, in a raid1 array, with xfs on top of it (AGAIN just because ONE of the disks decided on it's own that in that particular sector it's a 1 and not a 0, the other disk would have 0) and somehow the end result of my data would be a zero, and not a 1. this theory is based in the idea that harddrives return other data that you put into them and they dont return an error for it. |
It doesn't happen. The drive has error correction and checksums.
The probability that a one bit error beats the error correction and checksums is | Douglas Adams wrote: | | infinitely improbable |
.
On a software raid set a much bigger problem is the write hole.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
axl
Veteran
Joined: 11 Oct 2002
Posts: 1144
Location: Romania
|
| Posted: Fri Mar 17, 2017 3:03 pm Post subject: |
|
|
| NeddySeagoon wrote: | It doesn't happen. The drive has error correction and checksums.
The probability that a one bit error beats the error correction and checksums is | Douglas Adams wrote: | | infinitely improbable |
. |
I assumed as much. But according to some people, some drives don't go out of whack when data doesn't checksum. wait. for me to understand, how does a standard harddrive checksum? where does it keep the checksums? on the drive?
I use md5sum to see if a file changed over time. am not entirely sure how a drive does it. or xfs. or md. the proponents of zfs/btrfs make it sound like everything else CAN fail and the user, or worse, the kernel not know it. can that be possible?
not probable. possible?
let's assume my setup. i have 2 x 4Tb drives. in raid1. with xfs on top. i boot a blank kernel, without raid support, dont initialize raid, dont mount etc. now. i go at random points on drive a and change random 1's into 0's. some of them. i don't know how many. a few. 20 of them lets say. i use a C script, seek data, change data on drive a, not on drive b. now reboot. would the raid know? would raid know if just 20 bytes of data from drive A would be different from drive B ? would xfs know ? would the kernel complain? how would it look for me as a user when i try to access that data through nfs let's say.
according to what i thought i knew, xfs should put zeros in that file because it's unsure if that file had ones or zeros. xfs always puts zeros when it's unsure. and has some sort of checksum verification.
md/raid on the other hand... not really sure how checksums are handled. if any. any?! really? i really have to check. will use an old system with 2 pata drives of 250gb each. change only one char on drive a. see how long till something catches one that something is wrong.
| Quote: | | On a software raid set a much bigger problem is the write hole. |
again, the proponents of zfs/btrfs (as I understand it) are saying that zfs/btrfs protects you when your drive remembers wrong. that's the best way i can put it. he's old, and he just remembers wrong. they also say zfs/btrfs protects you against dataloss when power outage is involved ... but that's not what i am worried about.
I am not concerned with power outage. I am not concerned with incomplete writes. in my setup, the system always shuts down properly.
I am only concerned with the idea/possibility that a pair of drives with supposedly identical data, could actually have different data over time.
and it's not only an idea, or a possibility. it came to bite me on the ass quite so many times to be honest. sd cards come to mind.
the whole reason i came to think about this subject is because i was reading threads about btrfs and raspberry pi. how it would cut space in half but provide some sort of data error checking. and data recovery. xfs on flash memory for gentoo on a raspberry pi is a flash memory that will die fast. and in my experience, xfs doesn't say anything until it's too late.
meanwhile, i remembered a lot of files over times, in xfs, that kernel said nothing about them... but hey this file is somehow degraded over time. it's not the same. it has zeros.
i know what anyone would say. what i would say. you were writing data while the server rebooted. while the power went off. no. that array mostly has videos. and over time, once in a while, every file would get a series of zeros instead of real data. and in any movie player it would look just like a mpeg artifact. an error. but that error wasn't always there.
we all assumed that hdds check checksums. against... what exactly i dont know. themselves? the very disk that can go wrong is the very medium where you keep the checksums? no. can't be that. a memory of some kind? no... can't be that either. so... bullshit. enterprise drives do. because of special components. not cheapware. (scsi/sas) not normal sata.
does raid1 checksum? I THINK IT SHOULD. and is. and i will be surprised if it doesn't. but in a raid1 setup, when drive a says one thing and drive b says another thing, and no bios complains... what does the kernel do?
and xfs. let's assume raid / md module under kernel doesnt' checksum. and one time u call file x and it comes 1. and call it again comes out 0. md didn't catch it. just served raw data.
how would xfs know which is right? we say the filesystems are self consistent (like a crazy dude checking it's own md5 sums) but xfs isn't. is it?
u're welcome to insult me. for every stupid thing i said. i just want to understand. |
|
| Back to top |
|
|
Roman_Gruber
Advocate
Joined: 03 Oct 2006
Posts: 3846
Location: Austro Bavaria
|
| Posted: Fri Mar 17, 2017 3:03 pm Post subject: |
|
|
a bit off topic.
when you care for data security, than you should not consider exotic file sytems like btfs, xfs, zfs.
I use ext4 because I assume it is hte mostly used one file system as of now. So I assume bugs would have been discovered more likely and fixed as in other file systems.
I would more be worried about data corruption. And that is likely to happen with exotic filesystems. Point finger at reiserfs.
Also I use 3 different brand and models, and production years for my backups. 3 SSDs. I slowly get rid of HDDS. but the market for used HDDs has vanished these days.
data safety: hardware vs software vs user.
| Quote: | | how does a standard harddrive checksum? |
The firmware of the harddrive (ssd) does this for you.
| Quote: | | e a and change random 1's into 0's. |
A usual question for file system, ...
I think there are thesis for file systems / or called papers for those values.
on how many bit errors can be corrected for a given data set (depends on teh structure)
keyword: bit error correction in file systems / harddrives
| Quote: | | flash memory that will die fast |
Is this the same hoax. As guys tells you that a SSD dies quite fast?
How comes that my samsung hdds died earlier. as my now exactly 5 years old SSD from plextor, which i used for everything on my gentoo notebook on a daily basis. Many years with even the distfiles on the same SSD.
Those values are jsut estimates. Mathematical models.
I had at least three broken 2.5" HDDS with SATA (yay samsung is the crap) in comparission with none broken USB Stick, SSDs, SD-card from my camera, No broken memory for my 4 year old smartphone.
| Quote: | again, the proponents of zfs/btrfs (as I understand it) are saying that zfs/btrfs protects you when your drive remembers wrong. that's the best way i can put it. he's old, and he just remembers wrong. they also say zfs/btrfs protects you against dataloss when power outage is involved ... but that's not what i am worried about.
|
any file system tries to correct errors, or has tools for it. There is information stored with the data for this purpose. => fsck.XY
data loss when power is lost => ext 4 journal. works as it should
| Quote: | i know what anyone would say. what i would say. you were writing data while the server rebooted. while the power went off. no. that array mostly has videos. and over time, once in a while, every file would get a series of zeros instead of real data. and in any movie player it would look just like a mpeg artifact. an error. but that error wasn't always there.
|
Sorry but thats sounds like bullshit.
Recent example why I think so.
I took photos and videos in 2008.
copied those "media" to 320GB HDD.
did the same to 1.5TB 3.5" HDD (several kinds)
had it on my 1TB 2.5" notebook HDD
had it 2.5" SSDs from 250GB HDDS, 320GB, with different kernels / type of ext4 tools.
Moved the picture from my not touched sdcards (2008 !) with my notebook cardreader to my HDD.
I moved all the pictures to my hdd, than my ssd. I did a duplicate file finder.
any same named picture, from all those "backups" had the same checksum. had the same visual content.
also for the videos.
I did that quite recently => You can check my recent post why I asked why handbrake dropped audio from my video! Why duplicate file finder quite randomyl does not generate a list of duplicate files.
--
Summary checksum of SD-cards, ext3, ext4 2.5" / 3.5" Hdds were all the same.
Only explanation. You got data corruption. And as wrote earlier. Do not use exotic file systems.
| Quote: | does raid1 checksum? I THINK IT SHOULD. and is. and i will be surprised if it doesn't. but in a raid1 setup, when drive a says one thing and drive b says another thing, and no bios complains... what does the kernel do?
+ rest of it ...
|
I want to talk only about LVM2 + ext4.
Usually you determine with tune2fs, fstab, when and how often the file system should be checked.
I still remember the messages, and they still exists with openrc. File system should be checked. ... tune2fs cries...
Anyway I want to refer to the specs for those raids.
Also i want to refer to xfs spec sheet, code.
I would not assume that those software raids does any checksumming and such. hardware raid controllers may do, again the specifications may reveal it. Same with your file system questions.
--
i use lvm2 so i can be independend of the hardware layer.
i use ext4 because it did most of the time fixed it errors when i forgot to plug in my power in my notebook. happened again a few hours ago. |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Fri Mar 17, 2017 3:55 pm Post subject: |
|
|
Great site, Neddy.
Makes me think "TF for sysadmins".. ;-)
Although every first-line Gentoo user is an admin of their own machine/s, we all rely on the services of dedicated system administators. |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54237
Location: 56N 3W
|
| Posted: Fri Mar 17, 2017 4:02 pm Post subject: |
|
|
axl,
You cannot change user data on a HDD without changing the error correcting data and checksums too.
You send the drive a sectors worth of user data. The drive adds the error correcting data and checksum.
On read, the drive reads the data, applies error correction and checks the checksum.
If all is well, you get your data back. If not the drive does retries. When it runs out of retries, you get a read error back and no data.
Any file system that invents the data in the face of a read error is broken by design.
When you write random blocks on the drives in a two spindle raid1, the raid layer won't notice until you do a check.
Then it can tell that they are different, it can't tell which, if either, is correct.
The sector data on the drive is a continuous stream of flux reversals.
User data, error correction data and checksum. Its differentiated by context.
Much like to grub, the kernel and initrd files are data to be read.
Its only when grub jumps to the kernel start address the kernel becomes instructions to the CPU.
Instructions are different lengths too. Its still only context that separates instructions and data.
HDD sector reads either all works or it doesn't.
If the checksum becomes corrupt, the check fails, even if the user data is correct. You still get a read error.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
axl
Veteran
Joined: 11 Oct 2002
Posts: 1144
Location: Romania
|
| Posted: Fri Mar 17, 2017 4:51 pm Post subject: |
|
|
cool. i like your reply. let me answer point by point.
| Roman_Gruber wrote: | a bit off topic.
when you care for data security, than you should not consider exotic file sytems like btfs, xfs, zfs. |
let me stop you right there. i've been using xfs since 2.4. back then you had to manually patch your kernel and recompile it. and it was slackware. in 2001 gentoo already had xfs in gentoo sources. which is one of the reasons i adopted gentoo.
so going back and saying i wont use an experimental fs... is kinda hypocritical at this point. would be. for me. many years of xfs. only xfs.
(as a sidenote, i actually posted something about mfs)
| Quote: | I use ext4 because I assume it is hte mostly used one file system as of now. So I assume bugs would have been discovered more likely and fixed as in other file systems.
I would more be worried about data corruption. And that is likely to happen with exotic filesystems. Point finger at reiserfs. |
again. 2001. back then, i tried reiserfs for cyrus. cyrus imapd is sort of like squid. tones of little files. perfect for reiserfs you would say. except if you want to loose data.
reiserfs, is great at small files ... you want to loose. better than ntfs which is the greatestestestest fs at deleting files. no kidding, benchmarks suggest ntfs is great at deleting files.
| Quote: | Also I use 3 different brand and models, and production years for my backups. 3 SSDs. I slowly get rid of HDDS. but the market for used HDDs has vanished these days.
data safety: hardware vs software vs user.
| Quote: | | how does a standard harddrive checksum? |
The firmware of the harddrive (ssd) does this for you.
A usual question for file system, ...
I think there are thesis for file systems / or called papers for those values.
on how many bit errors can be corrected for a given data set (depends on teh structure)
keyword: bit error correction in file systems / harddrives
|
i only have to suffer 3 intels 750 series and the 4 Tb array i was talking about which is on WD reds. so... yeah. i can't complain.
but again, going back to raspberry pi and those hateful flash drives I CAN COMPLAIN A LOT.
because i'm not talking about hardware. i'm considering faulty hardware. i'm talking about software.
so the assumption is that the drive will checksum the data and report to the kernel / bios that data is faulty.
which, again most drives don't. they dont even check. against what? it's only an idea 
it becomes painfully obvious with flash drives where there is nothing to check against.
| Quote: | | Quote: | | flash memory that will die fast |
Is this the same hoax. As guys tells you that a SSD dies quite fast?
How comes that my samsung hdds died earlier. as my now exactly 5 years old SSD from plextor, which i used for everything on my gentoo notebook on a daily basis. Many years with even the distfiles on the same SSD.
Those values are jsut estimates. Mathematical models.
I had at least three broken 2.5" HDDS with SATA (yay samsung is the crap) in comparission with none broken USB Stick, SSDs, SD-card from my camera, No broken memory for my 4 year old smartphone.
|
no it's not the same thing, pay attention. it's about data integrity in random fs's and software raid and what zfs/btrfs are supposed to actually be.
| Quote: |
| Quote: | again, the proponents of zfs/btrfs (as I understand it) are saying that zfs/btrfs protects you when your drive remembers wrong. that's the best way i can put it. he's old, and he just remembers wrong. they also say zfs/btrfs protects you against dataloss when power outage is involved ... but that's not what i am worried about.
|
any file system tries to correct errors, or has tools for it. There is information stored with the data for this purpose. => fsck.XY
data loss when power is lost => ext 4 journal. works as it should |
but does it? i mean, xfs/jfs/ext3/4 and that handicaped reiser/ntfs are journaled filesystems. which were all the rage back at kernel 2.6. because the journal could be consistent with itself. which meant the journal was checksumed in some sort of way. but the drive has 4 Tb. you can't checksum all of that data at boot.
| Quote: | | Quote: | i know what anyone would say. what i would say. you were writing data while the server rebooted. while the power went off. no. that array mostly has videos. and over time, once in a while, every file would get a series of zeros instead of real data. and in any movie player it would look just like a mpeg artifact. an error. but that error wasn't always there.
|
Sorry but thats sounds like bullshit.
Recent example why I think so.
I took photos and videos in 2008.
copied those "media" to 320GB HDD.
did the same to 1.5TB 3.5" HDD (several kinds)
had it on my 1TB 2.5" notebook HDD
had it 2.5" SSDs from 250GB HDDS, 320GB, with different kernels / type of ext4 tools.
Moved the picture from my not touched sdcards (2008 !) with my notebook cardreader to my HDD.
I moved all the pictures to my hdd, than my ssd. I did a duplicate file finder.
any same named picture, from all those "backups" had the same checksum. had the same visual content.
also for the videos.
I did that quite recently => You can check my recent post why I asked why handbrake dropped audio from my video! Why duplicate file finder quite randomyl does not generate a list of duplicate files.
--
Summary checksum of SD-cards, ext3, ext4 2.5" / 3.5" Hdds were all the same.
Only explanation. You got data corruption. And as wrote earlier. Do not use exotic file systems.
| Quote: | does raid1 checksum? I THINK IT SHOULD. and is. and i will be surprised if it doesn't. but in a raid1 setup, when drive a says one thing and drive b says another thing, and no bios complains... what does the kernel do?
+ rest of it ...
|
|
true. the only explanation is data corruption. so what do you do? some people partitioned the flash in two and made it btrfs. others made it raid1 2 partitions on the same disk. am pretty sure no one knows the answer to this question.
| Quote: |
I want to talk only about LVM2 + ext4.
Usually you determine with tune2fs, fstab, when and how often the file system should be checked.
I still remember the messages, and they still exists with openrc. File system should be checked. ... tune2fs cries...
Anyway I want to refer to the specs for those raids.
Also i want to refer to xfs spec sheet, code.
I would not assume that those software raids does any checksumming and such. hardware raid controllers may do, again the specifications may reveal it. Same with your file system questions.
--
i use lvm2 so i can be independend of the hardware layer.
i use ext4 because it did most of the time fixed it errors when i forgot to plug in my power in my notebook. happened again a few hours ago. |
uhm, xfs doesn't do any periodical fsck at all. it's not like windows ntfs. once 50 boots you have to fsck. xfs is the most versatile fs i've seen so far. again. i'm not doubting the fs. just the drives. 
uhm, lvm is pretty cool. but i'm not sure it does checksums either. this is a big selling point for btrfs and zfs. your data, is the exactly same data u left here.
that's the feeling i'm getting. |
|
| Back to top |
|
|
axl
Veteran
Joined: 11 Oct 2002
Posts: 1144
Location: Romania
|
| Posted: Fri Mar 17, 2017 5:06 pm Post subject: |
|
|
| NeddySeagoon wrote: | axl,
You cannot change user data on a HDD without changing the error correcting data and checksums too.
You send the drive a sectors worth of user data. The drive adds the error correcting data and checksum.
On read, the drive reads the data, applies error correction and checks the checksum.
If all is well, you get your data back. If not the drive does retries. When it runs out of retries, you get a read error back and no data.
Any file system that invents the data in the face of a read error is broken by design.
When you write random blocks on the drives in a two spindle raid1, the raid layer won't notice until you do a check.
Then it can tell that they are different, it can't tell which, if either, is correct.
The sector data on the drive is a continuous stream of flux reversals.
User data, error correction data and checksum. Its differentiated by context.
Much like to grub, the kernel and initrd files are data to be read.
Its only when grub jumps to the kernel start address the kernel becomes instructions to the CPU.
Instructions are different lengths too. Its still only context that separates instructions and data.
HDD sector reads either all works or it doesn't.
If the checksum becomes corrupt, the check fails, even if the user data is correct. You still get a read error. |
where exactly is the checksum. and how does it relate to cheap sata drives and cheap china bioses ?
does it actually have a checksum? no it doesn't. most of your data does't. buy ecc memory. admin spotting
just wrong assumptions. |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54237
Location: 56N 3W
|
| Posted: Fri Mar 17, 2017 6:07 pm Post subject: |
|
|
axl,
Google is your friend. Read about Partial Response Maximum Likelihood encoding on magnetic media. (That's a PDF)
Also read Data Layout That page is mostly rubbish but the diagram is correct.
What has the BIOS got to do with anything?
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
Roman_Gruber
Advocate
Joined: 03 Oct 2006
Posts: 3846
Location: Austro Bavaria
|
| Posted: Sat Mar 18, 2017 12:18 pm Post subject: |
|
|
Thanks for all the reply
| Quote: | | it becomes painfully obvious with flash drives where there is nothing to check against. |
Nope.
The software "stack" / "level" is always the same. When we assume it'S the same linux kernel and file system compiled for just another architecture!
I used xfs for a month to classify it as low performer, and "crap". Sorry that's my opinion. And that point xfs had less features as my ext3 / ext4 at that point of time. Something was missing and the reason why i switched back after a month using xfs on /.
I do remember data corruption on my computers, but that was pre Windows 95 / pre Windows 2000 era, with those earlier file systems and "microsoft" platforms. I also remember the same with REDHAT (suse 6.2 era) with reiserfs.
--
I regularly loose power on my notebook. The reason is quite simple. I do not want to keep my 2nd hand notebook on power when I leave the house. So I plug the AC cord and randomly forgets to replug when I come back. So you can say at least average twice a month over 1.5 years I loose power and my ext4 + lvm2 + luks never made any fuss here. (Note: I do not know what the preowner of my notebook did with that hardware and thats just a safety measure to unplug it...)
I do remember those days when I was bugged with y/n questions after a power failure.
--
My previous example above with my "media" has shown you hopefully, how mature ext3 / ext4. my sdcards uses fat 16 or fat32 (camera demands it). For different drives / manufactuerers / moved data.
--
Sidenote: When you want to use a rasperri pi, you should use F2FS. AFAIK developed by samsung for flash based smartphones. F2FS is faster than the "standard fs" (forgot which one) on my Nexus 4 / Nexus 7. And you should make regular backups. over network, or just use a cardreader to regularly dump the contents.
Flash memories of sd cards are most of the time very very slow.
| Quote: | | reiser/ntfs are journaled filesystems. which were all the rage back at kernel 2.6. because the journal could be consistent with itself. |
Nope. Back in my Suse 6.2 days I emailed that reiserguy to tell him his file system has a bug. He asked me for money. I fixed the error myself and ditched that file system.
Why should i pay for his buggy file system in the first place!
Thats the issue with reiserfs/ntfs it is not that "durable" as ext4.
The only thing which other file systems may have is a duplicate file recognition feature on the fs level to only have the file once on the phyiscal disk. |
|
| Back to top |
|
|
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
| Posted: Sat Mar 18, 2017 7:17 pm Post subject: |
|
|
| Some timely reading material, since the Dunning-Kruger syndrome in this thread is approaching nauseating levels. |
|
| Back to top |
|
|
frostschutz
Advocate
Joined: 22 Feb 2005
Posts: 2977
Location: Germany
|
| Posted: Sat Mar 18, 2017 7:45 pm Post subject: |
|
|
| Quote: | | This means there was a data error on the drive. |
Or a bug in the filesystem, or a power outage, or ... a non-silent drive error. Many possibilities. Without a second opinion impossible to falsify.
I'm also checking for bitflips on my RAID arrays, if a single disk decided it had silent corruptions, there would be parity mismatches. But there are none. Haven't been in years since I started actively testing for them. And I'm like using the cheapest SATA controllers out there (asmedia) and the cheapest drives (WD Greens).
ZFS is not an option for me as it's not in the Linux kernel.
btrfs is not an option because every day in IRC I see someone who has lost data to it.
I hear XFS has plans to add checksums, we will see how it goes...
The most common cause of corruption... some program fondling the files, some user error - no filesystem protects you against that.
I don't believe in silent bitrot. I do believe in rigorous disk monitoring, selftests and getting rid of hardware that is obviously broken (read failure, reallocated/pending/uncorrectable sector). |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54237
Location: 56N 3W
|
| Posted: Sat Mar 18, 2017 7:57 pm Post subject: |
|
|
Ant P.,
That illustration was not bitrot on the drive. The drive returned correct information, then it was corrupted during transmission over the interface, since changing the data cable fixed the problem.
Actually the corruption was detected during a read operation. It could have occurred during the write. Write corruptions would only be detected by the filesystem reading the data back and doing the checksum test. That sounds like it would halve the IO capacity of the drive, if that were done as a part of the write operation.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
axl
Veteran
Joined: 11 Oct 2002
Posts: 1144
Location: Romania
|
| Posted: Sun Mar 19, 2017 12:20 am Post subject: |
|
|
| Roman_Gruber wrote: | Sidenote: When you want to use a rasperri pi, you should use F2FS. AFAIK developed by samsung for flash based smartphones. F2FS is faster than the "standard fs" (forgot which one) on my Nexus 4 / Nexus 7. And you should make regular backups. over network, or just use a cardreader to regularly dump the contents.
Flash memories of sd cards are most of the time very very slow. |
i keep hearing about f2fs. didn't have time to have a proper look at it. thanks for the suggestion. |
|
| Back to top |
|
|
axl
Veteran
Joined: 11 Oct 2002
Posts: 1144
Location: Romania
|
| Posted: Sun Mar 19, 2017 12:46 am Post subject: |
|
|
yes. thank you. "bitrot".
everything I was trying to say can be easily summed up with that word.
and given that i finished migrating from md array/xfs to btrfs, i could post some impressions.
it went smooth. took double than i expected. copying all that data from drive A to drive B takes about 8 hours. at first the 2 drives were completely separate and i was using rsync to keep them synced. after, i created an array out of drive A, and copied all data on drive A, then added drive B to the array. And you could see as drive B was added to the array that data was pouring out of drive A onto drive B. simple.
but with btrfs it was different. when you add another drive to a btrfs fs or mountpoint and resync, data is read from the source, then written on the source, then on the copy. and it takes double time. again, resync on a md array usually means copy from A to B. on btrfs it means read from A, write to A, write to B.
performance also took a big hit. reading wise, used to read with 140-100Mbs. now it's 80-100Mbs. writing i dont really have a frame of reference.
also discovered seekwatcher while doing the migration but didn't have BLK_DEV_IO_TRACE enabled in kernel so... couldn't play with it. am considering switching back from btrfs to the old md/xfs design just to graph the migration. gonna stick with btffs for a few days for now.
I don't _think_ the types of drives I have really develop "bitrot". or if they do, i doubt they develop a lot of it. i keep md5 sums now of most of my files. the good thing about it is that the files stored on those drives are just phat video files that don't change a lot. they just sit there. and that machine doesn't get power outages. AT ALL. it has an UPS connected to the serial port, it would shut down before it runs out of power.
still, not sure what to make of this. just discovered the idea of bitrot a few months back. didn't even cross my mind before. not sure if btrfs/zfs are here to address an issue that is a non-issue, or it's something that we didn't even knew we face. more research required. what I DO want to try is this migration back and forth, and this time, graph it with seekwatcher. |
|
| Back to top |
|
|
axl
Veteran
Joined: 11 Oct 2002
Posts: 1144
Location: Romania
|
| Posted: Sun Mar 19, 2017 1:26 am Post subject: |
|
|
| Roman_Gruber wrote: | Thanks for all the reply
| Quote: | | it becomes painfully obvious with flash drives where there is nothing to check against. |
Nope.
The software "stack" / "level" is always the same. When we assume it'S the same linux kernel and file system compiled for just another architecture!
I used xfs for a month to classify it as low performer, and "crap". Sorry that's my opinion. And that point xfs had less features as my ext3 / ext4 at that point of time. Something was missing and the reason why i switched back after a month using xfs on /.
I do remember data corruption on my computers, but that was pre Windows 95 / pre Windows 2000 era, with those earlier file systems and "microsoft" platforms. I also remember the same with REDHAT (suse 6.2 era) with reiserfs.
--
I regularly loose power on my notebook. The reason is quite simple. I do not want to keep my 2nd hand notebook on power when I leave the house. So I plug the AC cord and randomly forgets to replug when I come back. So you can say at least average twice a month over 1.5 years I loose power and my ext4 + lvm2 + luks never made any fuss here. (Note: I do not know what the preowner of my notebook did with that hardware and thats just a safety measure to unplug it...)
I do remember those days when I was bugged with y/n questions after a power failure.
--
My previous example above with my "media" has shown you hopefully, how mature ext3 / ext4. my sdcards uses fat 16 or fat32 (camera demands it). For different drives / manufactuerers / moved data.
| Quote: | | reiser/ntfs are journaled filesystems. which were all the rage back at kernel 2.6. because the journal could be consistent with itself. |
Nope. Back in my Suse 6.2 days I emailed that reiserguy to tell him his file system has a bug. He asked me for money. I fixed the error myself and ditched that file system.
Why should i pay for his buggy file system in the first place!
Thats the issue with reiserfs/ntfs it is not that "durable" as ext4.
The only thing which other file systems may have is a duplicate file recognition feature on the fs level to only have the file once on the phyiscal disk. |
I've considered if I should reply to this bit or not. It's gentoo chat. And I kinda want to rant a bit. But ... well here goes.
As far as I remember, I didn't choose xfs. I had to go along with it because at the time the only other choice was jfs and reiserfs. it was during the 2.3/2.4 kernel era. I had 8 drives of 250Gb each in a liniar raid. and at that time, xfs was the only one that could handle 2 Tb. and actually not loose that data. ext3 didn't exist. ext2 couldn't handle all that data. that's how i ended up with xfs.
we can both agree that reiserfs is very fast competent fast fs for small files, if you want to loose small files. every experience i had with reiserfs was disastrous.I lost important files only twice, but it was enough not to try it again.
xfs on the other hand didn't loose important chunks of data (for me). just an occasional file here and there is zeroed out. ext2/3/4 are good. i gave it a try here and there. use it sometimes for vm's or pi's or other exotic stuff, but main horse (the array in question) was just fine being xfs. I mentioned the brand of the drives. Mentioned the model. the fs. uhm, i kept xfs because of performance. every benchmark i could possibly conceive shows xfs is faster than any incarnation of ext.
that's why i'm excited about finding seekwatcher. there's some scenarios I want to see. actually see.
https://www.youtube.com/watch?v=UQ12PS5x53U
this is a complete 908 sec read out of things being written and read from an intel 750 series 400Gb drive. dd if=/dev/nvme0n1 of=/dev/null. while on other consoles dd if=/dev/zero of=/file bs=1G count=$random; rm /file on another console.
this tool is awesum. I plan, like i said before to switch back from btrfs to raid/xfs and again to btrfs and map out the migration, but also, i plan to map out (in a vm environment) some tests between all the fs's. just for my own curiosity. seeing them in motion. i really love this tool. give it a search. seekwatcher. it's in portage |
|
| Back to top |
|
|
axl
Veteran
Joined: 11 Oct 2002
Posts: 1144
Location: Romania
|
| Posted: Sun Mar 19, 2017 1:46 am Post subject: |
|
|
| frostschutz wrote: | | Quote: | | This means there was a data error on the drive. |
Or a bug in the filesystem, or a power outage, or ... a non-silent drive error. Many possibilities. Without a second opinion impossible to falsify.
I'm also checking for bitflips on my RAID arrays, if a single disk decided it had silent corruptions, there would be parity mismatches. But there are none. Haven't been in years since I started actively testing for them. And I'm like using the cheapest SATA controllers out there (asmedia) and the cheapest drives (WD Greens).
ZFS is not an option for me as it's not in the Linux kernel.
btrfs is not an option because every day in IRC I see someone who has lost data to it.
I hear XFS has plans to add checksums, we will see how it goes...
The most common cause of corruption... some program fondling the files, some user error - no filesystem protects you against that.
I don't believe in silent bitrot. I do believe in rigorous disk monitoring, selftests and getting rid of hardware that is obviously broken (read failure, reallocated/pending/uncorrectable sector). |
I agree with your assessment of zfs. It's not in the kernel, it's not for me. I went through this with xfs. Dont want to do it again if btrfs is an alternative. On the other hand solaris has done it for years.
But, now, all it's base belongs to ORACLE. which... no. just no. i will not use that software. on general principle. rather have my stuff rot than use oracle software. (love mariadb and i never used java).
I hope xfs implements that feature. SGI is a weird company. Done some amazing things, including xfs. I feel invested. I really do love sgi and xfs. It never failed once.
About your comment with WD greens. I have WD reds. no problem whatsoever. I dont expect problems. Last 8 pata drives of 250Gb each, were retired because of obsolete speed, not because of failure. They were also WD's. in a liniar raid. i mentioned that in another post.
which is weird for me. same raid. liniar to raid1. 2 Tb to 4Tb.
this array was born in the 90's. went through so many changes. at first the concern was space. the size was never enough. after that it was speed, and just only recently it became about data integrity. just goes to show how expectations grow over time. |
|
| Back to top |
|
|
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
| Posted: Sun Mar 19, 2017 6:09 pm Post subject: |
|
|
| axl wrote: | I've considered if I should reply to this bit or not. It's gentoo chat. And I kinda want to rant a bit. But ... well here goes.
As far as I remember, I didn't choose xfs. I had to go along with it because at the time the only other choice was jfs and reiserfs. it was during the 2.3/2.4 kernel era. I had 8 drives of 250Gb each in a liniar raid. and at that time, xfs was the only one that could handle 2 Tb. and actually not loose that data. ext3 didn't exist. ext2 couldn't handle all that data. that's how i ended up with xfs. |
Don't let anyone shame you for using XFS, it's the right choice if you want something that's lasted 20 years and will probably last another 20.
It's the only filesystem project that bothered to write a test suite, that says a lot. |
|
| Back to top |
|
|
axl
Veteran
Joined: 11 Oct 2002
Posts: 1144
Location: Romania
|
| Posted: Sun Mar 19, 2017 9:11 pm Post subject: |
|
|
| Ant P. wrote: | | axl wrote: | I've considered if I should reply to this bit or not. It's gentoo chat. And I kinda want to rant a bit. But ... well here goes.
As far as I remember, I didn't choose xfs. I had to go along with it because at the time the only other choice was jfs and reiserfs. it was during the 2.3/2.4 kernel era. I had 8 drives of 250Gb each in a liniar raid. and at that time, xfs was the only one that could handle 2 Tb. and actually not loose that data. ext3 didn't exist. ext2 couldn't handle all that data. that's how i ended up with xfs. |
Don't let anyone shame you for using XFS, it's the right choice if you want something that's lasted 20 years and will probably last another 20.
It's the only filesystem project that bothered to write a test suite, that says a lot. |
i think at the end of the day, the stuff we talk about has to fit our needs. when you get used to something like linux, or gentoo, or xfs, it's something that suited you for some time.
that's basically the reason i try my best to stay out of any kind of flame wars. everyone has their favorite thing. that suits them. i've read opinions of people that could swear on jfs or ext. and it suits them.
and that's what you want. everybody be happy.
well in other news, being excited about seekwatcher, i decided to start the process early to move back to raid/xfs. i explained at length my setup.
SO, to put some constructive info into the chat, tomorrow I'll be able to post a video of how it looks when you remove a drive from a raid 1 btrfs.
I went the long way. when you have a raid1 array with btrfs, and you want to remove one of the drives, you have 2 options. either software, or hardware. or software 1 / software 2 and hardware.
The long way around is to convert the entire fs back to single, from a raid1 profile. Which takes a long time. For my setup, about 16-18 hours. And just for the sake of completion, the other 2 ways are to signal kernel drive is defective, force it out of the array reformat and reuse, or just disconnect the drive physically and remount degraded blah blah. I took the long way around.
And this process of making the drives single, I'll be able to post a video of seekwatcher tomorrow. I'm so excited. I'm 40, but i'm still a geek.
the fact that you can run a machine in just console is amazing. linux is amazing. gentoo is amazing. but to be able to switch fs's around. create raids then move data, destroy old fs, start over. map it out and upload to youtube... jeez this is exciting.
how will it be in 20 years? will people complain that crc-fs's are just too slow. do you remember those early 2000 years when fs's were just fast?
anyway. am so excited for blktrace and seekwatcher. i've been wondering for years if such a tool actually exists. it does. can't wait to see the animation tomorrow. and then i'll post 2 others. copy data from single btrfs to xfs, and raid 1 adding a new drive. i expect all 3 to be pretty linear.
sorry. just the inner geek had to explode. and i _think_ i want to explode near my flock. and for better or worse u are my flock. gentoo user since 2001. sorry. |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54237
Location: 56N 3W
|
| Posted: Sun Mar 19, 2017 11:24 pm Post subject: |
|
|
axl,
You are just a bairn :)
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
axl
Veteran
Joined: 11 Oct 2002
Posts: 1144
Location: Romania
|
| Posted: Sun Mar 19, 2017 11:32 pm Post subject: |
|
|
| I don't mind being called childish, it's a quality I hold dear at heart. But i have to ask, where in god's green earth did you first encounter that word? bairn. it's the first time i've came across it. |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54237
Location: 56N 3W
|
| Posted: Mon Mar 20, 2017 12:00 am Post subject: |
|
|
axl,
Its a Scottish word.
It does not mean that you are childlish, it means that you are young compared to the person using the phrase "just a bairn".
I'm 63.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Mon Mar 20, 2017 1:46 am Post subject: |
|
|
Heh, interesting post.
I've just been looking into (no)delalloc for ext4, as I'm used to ext3 and wanted to see what the new things were about.
The ext4 default setup does not look good.
Ted Tso's article did not reassure, although at least setting nodelalloc provides better performance than ext3.
I don't trust auto_da_alloc, except on mounts for tmp directories, because "growing files at the end" is very common (O_APPEND, shell '>>').
If you view the latest comment here (Oct 21, 2016 by damnyoulinux), you'll see ext4's delalloc still causes issues.
So in summary, I've turned it off with tune2fs -o nodelalloc /dev/xyz for every filesystem with data I care about (and added auto_da_alloc for tmp disk-mounts in /etc/fstab.) |
|
| Back to top |
|
|
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21631
|
| Posted: Mon Mar 20, 2017 3:35 am Post subject: |
|
|
| That comment is an impressive wall of text, but failed to mention anything that could be used to confirm or refute the diagnosis. It failed to identify the kernel version or even the distribution. It failed to mention what application(s) are misbehaving, other than to assert that they are popular. It openly admitted users routinely power off the system improperly. It asserts corruption, but then describes it as truncated files. Truncated files are definitely a sign of lost data, but corruption to me implies that the file has wrong data, not absent data. I agree with the latter concern: an obvious authoritative reference would be helpful. |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Mon Mar 20, 2017 5:50 pm Post subject: |
|
|
Well, Tso's article is pretty authoritative in that it comes from the upstream author of ext2/3/4, so the problem is definitely real.
It's fine to provide options that are only safe when you use an UPS. It is not fine to pretend to users that they are getting the same data=ordered treatment as ext3, while doing nothing of the sort in the default setup.
It is made even worse when you then a) blame the users, and b) blame everyone else (for your defaults, in your software, and the mess they caused.)
IMO delalloc should be off by default, and auto_da_alloc should switch on when delalloc is on, unless the user overrides.
Further, auto_da_alloc still needs work, as bleating about O_APPEND is simply dumb (especially for a FS coder.)
Sorry I haven't got more references, I'm in the middle of an install, and might be referring to other discussions or articles I read yesterday.
Just seems to me that the systemdbust mentality (blame the users, blame other programmers) is infecting other Linux developers which is not good.
I preferred: "Don't break the user experience." |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Gentoo Chat
