| View previous topic :: View next topic |
| Author |
Message |
Moriah
Veteran
Joined: 27 Mar 2004
Posts: 1815
Location: Warsaw KY US
|
 Posted: Sat Apr 28, 2012 6:07 pm Post subject: Anyone ever heard of a dedup block device? |
 |
|
Does anyone know of a dedup block device, instead of a dedup filesystem? That way, you could put any filesystem on top of it. You could use a fixed block size, such as 4K, a good cryptographic hashing algorithm, and an indirect addressing table that mapped to LBA's instead of filesystem chunks. It might even be possible to implement such a thing as an add-on to LVM.
_________________
The MyWord KJV Bible tool is at http://www.elilabs.com/~myword
Foghorn Leghorn is a Warner Bros. cartoon character. |
|
| Back to top |
|
 |
energyman76b
Advocate
Joined: 26 Mar 2003
Posts: 2022
Location: Germany
|
| Posted: Sat Apr 28, 2012 7:55 pm Post subject: |
|
|
aren't some ssd controllers already doing this? to reduce write tear?
edit:
http://storagemojo.com/2011/06/27/de-dup-too-much-of-good-thing/
_________________
| AidanJT wrote: |
Libertardian denial of reality is wholly unimpressive and unconvincing, and simply serves to demonstrate what a bunch of delusional fools they all are.
|
Satan's got perfectly toned abs and rocks a c-cup.
Last edited by energyman76b on Sat Apr 28, 2012 7:56 pm; edited 1 time in total |
|
| Back to top |
|
|
notageek
Tux's lil' helper
Joined: 05 Jun 2008
Posts: 78
Location: Bangalore, India
|
| Posted: Sat Apr 28, 2012 7:55 pm Post subject: |
|
|
| Quote: | | Anyone ever heard of a dedup block device? |
No.
_________________
What looks like a cat, flies like a bat, brays like a donkey, and plays like a monkey? |
|
| Back to top |
|
|
Moriah
Veteran
Joined: 27 Mar 2004
Posts: 1815
Location: Warsaw KY US
|
| Posted: Sat Apr 28, 2012 10:40 pm Post subject: |
|
|
energyman76b: I had expected that SSD's did that sort of thing; I was really asking about such a technique in an open sourced implementation that could sit between a physical drive and a filesystem on a Linux box. I was particularly interested in any design considerations that would be "special" for a block device dedup as opposed to a filesystem dedup. I wanted to leverage existing filesystems on top of a deduping substrate.
The article you linked to mentioned one: delibreately duplicated metadata blcoks should not be deduped or else you defeat the whole point of the deliberate duplication.
BTW I have been using OCZ SSD's for several years, and the only failures I have seen have been purely electrical at the SATA interface level, not at the stored data level.
Does it sound feasible to implement a deduping block device as a plugin to LVM? My own personal interests are for disk based backup servers and for virtual disks for virtual machines.
_________________
The MyWord KJV Bible tool is at http://www.elilabs.com/~myword
Foghorn Leghorn is a Warner Bros. cartoon character. |
|
| Back to top |
|
|
energyman76b
Advocate
Joined: 26 Mar 2003
Posts: 2022
Location: Germany
|
| Posted: Sat Apr 28, 2012 10:46 pm Post subject: |
|
|
and how shall one discern between data to be dedup and data that shall not be dedup?
This 'deduplication' frenzy sounds stupid.
_________________
| AidanJT wrote: |
Libertardian denial of reality is wholly unimpressive and unconvincing, and simply serves to demonstrate what a bunch of delusional fools they all are.
|
Satan's got perfectly toned abs and rocks a c-cup. |
|
| Back to top |
|
|
Moriah
Veteran
Joined: 27 Mar 2004
Posts: 1815
Location: Warsaw KY US
|
| Posted: Sat Apr 28, 2012 10:49 pm Post subject: |
|
|
The article you linked to had a suggestion in the reader comments section about implementing a UNIQUE command analogous to the TRIM command.
It is not stupid if it gets 8-20 times effective compression ration; most other techniques only get 2-3 times. Of course, speed and reliability have to be taken into account, but those are application considerations. Acceptable parameters for a video game would be less demanding than for nuclear reactor control. 
_________________
The MyWord KJV Bible tool is at http://www.elilabs.com/~myword
Foghorn Leghorn is a Warner Bros. cartoon character. |
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 16029
Location: Colorado
|
| Posted: Sat Apr 28, 2012 11:06 pm Post subject: |
|
|
| energyman76b wrote: | | This 'deduplication' frenzy sounds stupid. |
A virtualization server with many tens of operating systems can save a lot of disk space. Multiple that across many servers, and the physical cost is significant, but probably less than cost savings of electricity and/or cooling.
_________________
lolgov. 'cause where we're going, you don't have civil liberties.
In Loving Memory
1787 - 2008 |
|
| Back to top |
|
|
energyman76b
Advocate
Joined: 26 Mar 2003
Posts: 2022
Location: Germany
|
| Posted: Sat Apr 28, 2012 11:21 pm Post subject: |
|
|
| pjp wrote: | | energyman76b wrote: | | This 'deduplication' frenzy sounds stupid. |
A virtualization server with many tens of operating systems can save a lot of disk space. Multiple that across many servers, and the physical cost is significant, but probably less than cost savings of electricity and/or cooling. |
and when the one copy of some important structure/file goes down, all ten virtual servers go down instead of one single instance. Which makes virtualization look stupid in the first place.
_________________
| AidanJT wrote: |
Libertardian denial of reality is wholly unimpressive and unconvincing, and simply serves to demonstrate what a bunch of delusional fools they all are.
|
Satan's got perfectly toned abs and rocks a c-cup. |
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 16029
Location: Colorado
|
| Posted: Sun Apr 29, 2012 4:13 pm Post subject: |
|
|
Unless you would run a business on a single critical server, you'd never have those 10 virtuals on a single piece of hardware.
_________________
lolgov. 'cause where we're going, you don't have civil liberties.
In Loving Memory
1787 - 2008 |
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 16029
Location: Colorado
|
| Posted: Sun Apr 29, 2012 10:38 pm Post subject: |
|
|
| energyman76b wrote: | | http://storagemojo.com/2011/06/27/de-dup-too-much-of-good-thing/ |
| Quote: | De-dup is a Good Thing, right?
Researchers found that at least 1 Sandforce SSD controller – the SF1200 – does block-level deduplication by default. Many file systems write critical metadata to multiple blocks in case one copy gets corrupted. But what if, unbeknownst to you, your SSD de-duplicates that block, leaving your file system with only 1 copy?
Yup, corruption of 1 block could wipe out your entire file system. And since all the “copies” point to the same corrupted block, there’s no way to recover. |
That's just a condition of the HW design, and obviously something to consider when implementing that HW with fs dedup.
_________________
lolgov. 'cause where we're going, you don't have civil liberties.
In Loving Memory
1787 - 2008 |
|
| Back to top |
|
|
BoneKracker
Veteran
Joined: 14 Mar 2006
Posts: 1488
Location: U.S.A.
|
| Posted: Sun Apr 29, 2012 11:22 pm Post subject: Re: Anyone ever heard of a dedup block device? |
|
|
| Moriah wrote: | | Does anyone know of a dedup block device, instead of a dedup filesystem? That way, you could put any filesystem on top of it. You could use a fixed block size, such as 4K, a good cryptographic hashing algorithm, and an indirect addressing table that mapped to LBA's instead of filesystem chunks. It might even be possible to implement such a thing as an add-on to LVM. |
Hurry up and write a patent application, but throw in something to different from the existing art. Patent "de-duplicating holographic data storage mechanism (implemented in hardware or software)". Look at what's already been done, and just change the terminology and throw in some spatial bullshit. Right around the time you're having your mid-life crisis, somebody will pay you millions of dollars for it, which will enable you to get hair plugs, buy a sports car, and a trade in your wife for a new one.
_________________
Oldthinkers unbellyfeel INGSOC.
-- Headline of a document on Winston Smith's terminal in his cubicle at the Ministry of Truth, seen briefly in the background in one scene of the movie rendition of Nineteen Eighty-Four. |
|
| Back to top |
|
|
Moriah
Veteran
Joined: 27 Mar 2004
Posts: 1815
Location: Warsaw KY US
|
| Posted: Mon Apr 30, 2012 1:38 am Post subject: |
|
|
Too late. 
I'm already past my mid-life crisis, but way back then I did develop a patent, and indeed, it looks like it might be worth some money now. 
I've already had my sportscar -- a Lotus which I dated my wife in, I don't need hair plugs, and I want to keep my wife. 
What I had in mind was a block psuedo-device implemented in software, possibly as an extension to LVM, and making it open source so everyone who wanted to could use it.
_________________
The MyWord KJV Bible tool is at http://www.elilabs.com/~myword
Foghorn Leghorn is a Warner Bros. cartoon character. |
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 16029
Location: Colorado
|
| Posted: Mon Apr 30, 2012 2:11 am Post subject: |
|
|
| Moriah wrote: | | What I had in mind was a block psuedo-device implemented in software, possibly as an extension to LVM, and making it open source so everyone who wanted to could use it. |
Dual-license. Free for non-commercial use, then add some features only a business would need, then charge for it. You could even charge low fees for small businesses so they could use it "nearly free."
All kidding aside, sounds like a cool idea, though I'm not a fan of lvm, mainly because it seems more complicated than necessary.
_________________
lolgov. 'cause where we're going, you don't have civil liberties.
In Loving Memory
1787 - 2008 |
|
| Back to top |
|
|
BoneKracker
Veteran
Joined: 14 Mar 2006
Posts: 1488
Location: U.S.A.
|
| Posted: Mon Apr 30, 2012 2:15 am Post subject: |
|
|
Well ah say... ah was about as far off track there as a barnyard dog barkin' up a Cincinnati whore's leg, wun't I?
_________________
Oldthinkers unbellyfeel INGSOC.
-- Headline of a document on Winston Smith's terminal in his cubicle at the Ministry of Truth, seen briefly in the background in one scene of the movie rendition of Nineteen Eighty-Four. |
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 16029
Location: Colorado
|
| Posted: Mon Apr 30, 2012 2:36 am Post subject: |
|
|
_________________
lolgov. 'cause where we're going, you don't have civil liberties.
In Loving Memory
1787 - 2008 |
|
| Back to top |
|
|
|
Off the Wall
