Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
flash filesystem over a block device?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Off the Wall
View previous topic :: View next topic  
Author Message
eccerr0r
Advocate
Advocate


Joined: 01 Jul 2004
Posts: 3598
Location: USA

PostPosted: Fri Feb 15, 2013 4:23 pm    Post subject: flash filesystem over a block device? Reply with quote

There are plenty of warnings out there that file systems like jffs2 and ubifs were not meant for block devices like SD / MMC cards, but why couldn't there be something for these cards anyway?

What I'm wondering is some of these SD/MMC and other flash cards (even USB) probably, to be cheap, do not even have flash block management, and if using FAT file system will happily burn out the first erase block, rendering the device unreadable and possibly useless... So if the blocks on the device are indeed directly mapped to flash chip blocks, why wouldn't a flash filesystem help (granted, need a layer of conversion from memory -> block, which may be slow.

I suppose tell-tale signs of this are that the device size is pretty much exactly the size of the underlying media (no spare blocks).
But this can be due to disabling bad blocks so this is not a good way to tell...

And if the device actually does have wear leveling, we'd just be wasting disk bandwidth and disk space, and risk of journal handling data loss - from both the internal and file system (which is double risk)

Anyone know more about this?
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed to be advocating?
Back to top
View user's profile Send private message
erm67
Tux's lil' helper
Tux's lil' helper


Joined: 01 Nov 2005
Posts: 130
Location: somewhere in Berlusconia.

PostPosted: Fri Feb 15, 2013 5:18 pm    Post subject: Re: flash filesystem over a block device? Reply with quote

eccerr0r wrote:
There are plenty of warnings out there that file systems like jffs2 and ubifs were not meant for block devices like SD / MMC cards, but why couldn't there be something for these cards anyway?

What I'm wondering is some of these SD/MMC and other flash cards (even USB) probably, to be cheap, do not even have flash block management, and if using FAT file system will happily burn out the first erase block, rendering the device unreadable and possibly useless... So if the blocks on the device are indeed directly mapped to flash chip blocks, why wouldn't a flash filesystem help (granted, need a layer of conversion from memory -> block, which may be slow.

I suppose tell-tale signs of this are that the device size is pretty much exactly the size of the underlying media (no spare blocks).
But this can be due to disabling bad blocks so this is not a good way to tell...

And if the device actually does have wear leveling, we'd just be wasting disk bandwidth and disk space, and risk of journal handling data loss - from both the internal and file system (which is double risk)

Anyone know more about this?

use exfat :-)


The problem is that to properly do wear leveling the filesystem should have access to the physical sectors not only to the remapped logical sectors accessible on a sd/mmc or usb device.
Most devices use dynamic wear leveling, i.e. writes every time to a new block:
Quote:
The first type of real leveling is called dynamic wear leveling and it uses a map to link Logical Block Addresses (LBAs) from the OS to the physical Flash memory. Each time the OS writes replacement data, the map is updated so the original physical block is marked as invalid data, and a new block is linked to that map entry. Each time a block of data is re-written to the Flash memory it is written to a new location. However, blocks that never get replacement data sit with no additional wear on the Flash memory. The name comes from only the dynamic data is being recycled.

_________________
Truck!!
A posse ad esse non valet consequentia
Πάντα ῥεῖ
Back to top
View user's profile Send private message
eccerr0r
Advocate
Advocate


Joined: 01 Jul 2004
Posts: 3598
Location: USA

PostPosted: Sat Feb 16, 2013 12:37 am    Post subject: Re: flash filesystem over a block device? Reply with quote

erm67 wrote:
The problem is that to properly do wear leveling the filesystem should have access to the physical sectors not only to the remapped logical sectors accessible on a sd/mmc or usb device.
Most devices use dynamic wear leveling, i.e. writes every time to a new block:

I'm wondering - of the available blocks to the user, why can't one make your own remap table, and make some assumptions about what blocks are in the same erase block (or it could even be determined by experimentation). Yes, these will be wasted blocks on the available space to make your own mapping, but I'm not sure why this wouldn't work. Unless doing this isn't possible unless you have special blocks on the disk that have extended wear capabilities? Is this necessary? I thought most flash filesystems assume all blocks on a particular device has the same write endurance... I do know there are some devices out there that have a few blocks of extended endurance cells that they do the mappings with, but are typical flash filesystems even aware of this space if it exists?

Again the problem is not more wasted space, the hope is to increase endurance by cycling through clean blocks to increase overall device lifetime of a device that doesn't know how to do this intrinsically...
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed to be advocating?
Back to top
View user's profile Send private message
salahx
Guru
Guru


Joined: 12 Mar 2005
Posts: 368

PostPosted: Sat Feb 16, 2013 5:34 am    Post subject: Reply with quote

Actually, "cooked" flash devices (like mmC and USB thumb drivers) generally DO do wear leveling, and often have special optimizations for FAT. The problem is its not possible to access the "raw" flash from such a device, everything goes through the hardware FTL (Flash Transition Layer). The hardware does not report how th flash is layed out (although one can take a guess using the timing data from flashbench).

That having been said, there is a new linux filesystem (f2fs) that does try to be smarter with "cooked" flash devices.
Back to top
View user's profile Send private message
eccerr0r
Advocate
Advocate


Joined: 01 Jul 2004
Posts: 3598
Location: USA

PostPosted: Sat Feb 16, 2013 6:06 am    Post subject: Reply with quote

The question remains: does it matter if it isn't raw? Unlike running stuff like reiserfs over reiserfs which was a problem at a point, running another translation over another translation should be fine, there's no way to confuse one over the other for either until the data can't be read. In this respect it's also not necessary to have redundant bits to check the integrity of the block - the underlying cooked structure will provide it. The upper layer simply deals with wear translations.

The whole problem is that some of these cooked devices may have very poor wear leveling algorithms. What if you just trick it to distribute the load (say, if it was built for FAT, ignore the area where it normally uses for the FAT (that gets overwritten a lot and usually is "wear leveled") and make your own wear map and translation). In this data area, you may get an incorrect map once in a where you may reuse a freshly erased block, but chances are you won't because of your own routine which will more or less be "random" as far as the underlying wear leveling system sees it. And random is good, it distributes the wear across the device. I'd even venture to guess that it doesn't matter what the underlying block size is, just make sure you rotate through blocks used.

But yes this will slow things down a lot, especially if you get the block size wrong (see the 4K sector problem). But it should still work, albeit it will cause more wear than needed - hopefully better than what the "stupid" wear leveling system does.

(but yes, DON'T use something like this on an intelligent wear level system like on most SSDs and probably compact flash these days, these controllers are doing the right thing. I'm targeting Secure Digital/Multi Media Cards and cheap USB flash media which seem to fail way too often... Then again it may not be media wear that's causing them to fail...
Speaking of compact flash, anyone wear one of these out yet?)
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed to be advocating?
Back to top
View user's profile Send private message
salahx
Guru
Guru


Joined: 12 Mar 2005
Posts: 368

PostPosted: Sat Feb 16, 2013 7:07 am    Post subject: Reply with quote

The problem is we have no idea how the underlying FTL is deciding to do things (or even the underlying physical geometry of the flash). By trying to be "smart" it can actually make things worse. If quite possible (and even likely) a given model of cooked flash device may have several revisions in which the physical layout and/or programming of the device has change (but the capacity has stayed the same) but no change in the hardware ID. Especially for cheap flash devices, which are probably stitched together from the "rejects" in the flash factory and recycled Chinese menus.
Back to top
View user's profile Send private message
eccerr0r
Advocate
Advocate


Joined: 01 Jul 2004
Posts: 3598
Location: USA

PostPosted: Sat Feb 16, 2013 11:42 pm    Post subject: Reply with quote

Give an example of how it can be worse other than if your sector/erase block boundaries are wrong (see above).

And compare it to if someone was using their SSD with the same pattern that would have been written with a flash translation filesystem?
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed to be advocating?
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Off the Wall All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum