Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
3TB USB HD is showing up as 801GB
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
Diovanti
n00b
n00b


Joined: 27 Feb 2003
Posts: 63

PostPosted: Thu Apr 14, 2011 8:34 pm    Post subject: 3TB USB HD is showing up as 801GB Reply with quote

I just got a 3GB Hitachi HD and put it into a USB external hard drive enclosure. The problem is linux cannot see its full capicity. The drive in the enclosure has been tested with a windows 7 machine connecting with esata and it works fine under windows.
Code:

Linux nirvana 2.6.36.2 #1 SMP Thu Mar 3 19:37:29 Local time zone must be set--see zic m x86_64 AMD Athlon(tm) 64 Processor 3200+ AuthenticAMD GNU/Linux

[610730.148026] scsi 14:0:0:0: Direct-Access     Hitachi  HDS5C3030ALA630       PQ: 0 ANSI: 2 CCS
[610730.148831] sd 14:0:0:0: Attached scsi generic sg4 type 0
[610730.149306] sd 14:0:0:0: [sdd] 1565565872 512-byte logical blocks: (801 GB/746 GiB)
[610730.151917] sd 14:0:0:0: [sdd] Write Protect is off
[610730.151924] sd 14:0:0:0: [sdd] Mode Sense: 00 38 00 00
[610730.151928] sd 14:0:0:0: [sdd] Assuming drive cache: write through
[610730.154882] sd 14:0:0:0: [sdd] Assuming drive cache: write through
[610730.177121]  sdd: unknown partition table
[610730.180915] sd 14:0:0:0: [sdd] Assuming drive cache: write through
[610730.180923] sd 14:0:0:0: [sdd] Attached SCSI disk


The 'sdd: unknown partition table' is a result of what happens when I partition the drive in windows and the total size of partition exceeds' 800gb. I can clear the disklabel using parted and create a new GPT label that linux will read, but I cant make it more than 800GB. It will say the 'Error: The location is outside of the device /dev/sdd'

Any ideas how to get this to work. The external drive is going to be hooked up to an ARM machine running debian in the future so I hope any workaround can be applied to that as well.
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 7470

PostPosted: Thu Apr 14, 2011 10:24 pm    Post subject: Reply with quote

cat /usr/src/linux/.config | grep LBDAF ?
Back to top
View user's profile Send private message
Jon Wilder
Tux's lil' helper
Tux's lil' helper


Joined: 04 Apr 2011
Posts: 112
Location: Fresno CA

PostPosted: Fri Apr 15, 2011 12:36 am    Post subject: Reply with quote

Diovanti wrote:
I just got a 3GB Hitachi HD


Hopefully you meant a 3TB drive. ;)

Under "Enable the block layer" in the kernel config menu, do you have "Support for large (2TB+) block devices and files" compiled in?

Code:

Top level menu
Enable the block layer------>
         [*]Support for large (2TB+) block devices and files"


If not, you'll need to enable that in the kernel menu, recompile the kernel, then reboot to that kernel.
Back to top
View user's profile Send private message
Diovanti
n00b
n00b


Joined: 27 Feb 2003
Posts: 63

PostPosted: Fri Apr 15, 2011 8:36 am    Post subject: Reply with quote

Jon Wilder wrote:
Diovanti wrote:
I just got a 3GB Hitachi HD


Hopefully you meant a 3TB drive. ;)

Under "Enable the block layer" in the kernel config menu, do you have "Support for large (2TB+) block devices and files" compiled in?

Code:

Top level menu
Enable the block layer------>
         [*]Support for large (2TB+) block devices and files"


If not, you'll need to enable that in the kernel menu, recompile the kernel, then reboot to that kernel.


Its a 3TB HD :) Here's a screenshot from my menuconfig 2.6.36.2 vanilla (I don't see the option for large block listed, grep showes nothing as well):

Code:

 .config - Linux Kernel v2.6.36.2 Configuration
 ──────────────────────────────────────────────────────────────────────────────
  ┌──────────────────────── Enable the block layer ─────────────────────────┐
  │  Arrow keys navigate the menu.  <Enter> selects submenus --->.                                                                             │
  │  Highlighted letters are hotkeys.  Pressing <Y> includes, <N> excludes,                                                                    │
  │  <M> modularizes features.  Press <Esc><Esc> to exit, <?> for Help, </>                                                                    │
  │  for Search.  Legend: [*] built-in  [ ] excluded  <M> module  < >                                                                                 │
  │ ┌─────────────────────────────────────────────────────────────────────┐ │
  │ │    --- Enable the block layer                                                                                                                                 │ │
  │ │    [*]   Block layer SG support v4                                                                                                                          │ │
  │ │    [ ]   Block layer data integrity support                                                                                                             │ │
  │ │          IO Schedulers  --->                                                                                                                                      │ │
  │ │                                                                                                                                                                             │ │
  │ │                                                                                                                                                                             │ │
  │ │                                                                                                                                                                             │ │
  │ │                                                                                                                                                                             │ │
  │ │                                                                                                                                                                             │ │
  │ └─────────────────────────────────────────────────────────────────────┘ │
  ├─────────────────────────────────────────────────────────────────────────┤
  │                    <Select>    < Exit >    < Help >                                                                                                                     │
  └─────────────────────────────────────────────────────────────────────────┘



P.S. If the drive is hooked up to windows via usb without being partitioned it shows up as ~800GB. When using esata it shows up as 3TB. However after partitioning it in windows for the full 3TB it will then show up as the full 3TB using usb in windows as well. Unfortunately my linux systems don't have esata so I need to use usb :(
Back to top
View user's profile Send private message
madchaz
l33t
l33t


Joined: 01 Jul 2003
Posts: 993
Location: Quebec, Canada

PostPosted: Fri Apr 15, 2011 1:11 pm    Post subject: Reply with quote

screengrabs of menuconfig are pretty useless. Try the cat/grep command krinn posted to see if you have the option. It is required to access disks that are that large.
_________________
Someone asked me once if I suffered from mental illness. I told him I enjoyed every second of it.
www.madchaz.com A small candle of a website. As my lab specs on it.
Back to top
View user's profile Send private message
Anon-E-moose
Watchman
Watchman


Joined: 23 May 2008
Posts: 6098
Location: Dallas area

PostPosted: Fri Apr 15, 2011 2:47 pm    Post subject: Reply with quote

You won't find the large block option on 64 bit systems

Code:
config LBDAF
    bool "Support for large (2TB+) block devices and files"
    depends on !64BIT



The bios on the linux box may also be limiting how much of the disk you see.
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland
Back to top
View user's profile Send private message
Diovanti
n00b
n00b


Joined: 27 Feb 2003
Posts: 63

PostPosted: Fri Apr 15, 2011 2:57 pm    Post subject: Reply with quote

Anon-E-moose wrote:
You won't find the large block option on 64 bit systems

Code:
config LBDAF
    bool "Support for large (2TB+) block devices and files"
    depends on !64BIT



The bios on the linux box may also be limiting how much of the disk you see.


Yeah, I'm on 64bit. Is there a way to tell linux the disk geometry manually without going with what the bios says?
Back to top
View user's profile Send private message
Anon-E-moose
Watchman
Watchman


Joined: 23 May 2008
Posts: 6098
Location: Dallas area

PostPosted: Fri Apr 15, 2011 3:47 pm    Post subject: Reply with quote

Diovanti wrote:
Is there a way to tell linux the disk geometry manually without going with what the bios says?


Don't know.

I would google whatever the motherboard is on the linux box...with 3tb drives...and see what shows up.

You could also check with the mb manufacturer and see if they have a bios update that addresses large disks.
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 6920

PostPosted: Fri Apr 15, 2011 7:35 pm    Post subject: Reply with quote

If it works without the USB in the way, then the USB adaptor is junk.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54244
Location: 56N 3W

PostPosted: Fri Apr 15, 2011 9:26 pm    Post subject: Reply with quote

Diovanti,

Code:
[610730.149306] sd 14:0:0:0: [sdd] 1565565872 512-byte logical blocks: (801 GB/746 GiB)

Thats wrong. 3Tb drives have a 4k native sector size. Something somewhere is assuming 512b sectors.

Take care when you partition the drive, partitions are aligned on 4k boundarys. If not, the drive will fake 512b sectors by doing read/modify/writes which is 20 to 30 times slower than properly aligned writes.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
chithanh
Developer
Developer


Joined: 05 Aug 2006
Posts: 2158
Location: Berlin, Germany

PostPosted: Sat Apr 16, 2011 8:16 am    Post subject: Reply with quote

Almost all drives will present 512 byte logical sectors to the OS. Very few exceptions exist (e.g. Samsung S1 Mini) which use 4k logical sectors.

You could try to upgrade your kernel.
Back to top
View user's profile Send private message
Diovanti
n00b
n00b


Joined: 27 Feb 2003
Posts: 63

PostPosted: Sun Apr 17, 2011 8:15 am    Post subject: Reply with quote

chithanh wrote:
Almost all drives will present 512 byte logical sectors to the OS. Very few exceptions exist (e.g. Samsung S1 Mini) which use 4k logical sectors.

You could try to upgrade your kernel.


Upgraded kernel to 2.6.38.2 Still shows bogus capacity.

Ant P. wrote:
If it works without the USB in the way, then the USB adaptor is junk.


I tried hooking up the drive to a more modern system directly using SATA. I booted a Ubuntu 10.10 64bit live cd and it recognized the full capacity. I partitioned it with gparted starting from sector 2048 (so it wouldn't complain about not being aligned) to the end of the disk. I then formatted it NTFS and copied some files to the disk. It worked.

Then I umounted the disk and attached it to the same computer as usb. It was detected as 801GB again and dmesg said 'invalid partition table'. So, it seems there is a problem with my enclosure : http://www.newegg.com/Product/Product.aspx?Item=N82E16817182179

However, Windows 7 detects my ntfs partition correctly when using USB Does anyone have any idea what kind of magic voodoo it does to detected the drive correctly and if that can be applied to linux? (I plan to use this disk on Linux and Windows so I need it to be NTFS.)
Back to top
View user's profile Send private message
chithanh
Developer
Developer


Joined: 05 Aug 2006
Posts: 2158
Location: Berlin, Germany

PostPosted: Sun Apr 17, 2011 8:56 am    Post subject: Reply with quote

It appears to me that this is a kernel problem. If the enclosure is a recent model, you can try to toggle USB Attached SCSI support in your kernel, maybe that will make a difference.
Code:
CONFIG_USB_UAS=y
And you could report a bug on https://bugzilla.kernel.org/
Back to top
View user's profile Send private message
Diovanti
n00b
n00b


Joined: 27 Feb 2003
Posts: 63

PostPosted: Sun Apr 17, 2011 9:43 am    Post subject: Reply with quote

Also one thing I should be a little more clear on: Windows 7 will show the drive's true capacity using usb only when it has already been partitioned previously using a esata/sata connection. If the drive is blank, it will show up to windows as ~800 GB

Maybe the drive's firmware is reporting 800GB? Perhaps for some compatibility reasons for older BIOSes? There's no jumpers on this thing, though. The drives model is: Hitachi HDS5C3030ALA630. I didn't see anyting about this on the manufacturers website. I found another post where someone was having a smilar model Hatachi drive showing up as 801GB as well: http://hardforum.com/showthread.php?t=1573971

chithanh wrote:
It appears to me that this is a kernel problem. If the enclosure is a recent model, you can try to toggle USB Attached SCSI support in your kernel, maybe that will make a difference.
Code:
CONFIG_USB_UAS=y
And you could report a bug on https://bugzilla.kernel.org/


I just made the uas modules and modprobed it. Still dosen't work.

[/code]
[ 5348.811223] usbcore: registered new interface driver uas
[ 5395.851973] hub 1-0:1.0: state 7 ports 10 chg 0000 evt 0200
[ 5395.851987] ehci_hcd 0000:00:02.1: GetStatus port:9 status 001002 0 ACK POWER sig=se0 CSC
[ 5395.851996] hub 1-0:1.0: port 9, status 0100, change 0001, 12 Mb/s
[ 5395.852000] usb 1-9: USB disconnect, address 3
[ 5395.852010] usb 1-9: unregistering device
[ 5395.852013] usb 1-9: unregistering interface 1-9:1.0
[ 5395.854601] usb 1-9: usb_disable_device nuking all URBs
[ 5395.958015] hub 1-0:1.0: debounce: port 9: total 100ms stable 100ms status 0x100
[ 5414.061991] hub 1-0:1.0: state 7 ports 10 chg 0000 evt 0200
[ 5414.062004] ehci_hcd 0000:00:02.1: GetStatus port:9 status 001803 0 ACK POWER sig=j CSC CONNECT
[ 5414.062021] hub 1-0:1.0: port 9, status 0501, change 0001, 480 Mb/s
[ 5414.166006] hub 1-0:1.0: debounce: port 9: total 100ms stable 100ms status 0x501
[ 5414.217214] ehci_hcd 0000:00:02.1: port 9 high speed
[ 5414.217219] ehci_hcd 0000:00:02.1: GetStatus port:9 status 001005 0 ACK POWER sig=se0 PE CONNECT
[ 5414.268008] usb 1-9: new high speed USB device using ehci_hcd and address 4
[ 5414.319224] ehci_hcd 0000:00:02.1: port 9 high speed
[ 5414.319228] ehci_hcd 0000:00:02.1: GetStatus port:9 status 001005 0 ACK POWER sig=se0 PE CONNECT
[ 5414.383446] usb 1-9: default language 0x0409
[ 5414.384322] usb 1-9: udev 4, busnum 1, minor = 3
[ 5414.384324] usb 1-9: New USB device found, idVendor=152d, idProduct=2336
[ 5414.384327] usb 1-9: New USB device strings: Mfr=1, Product=2, SerialNumber=5
[ 5414.384330] usb 1-9: Product: JM20336 SATA, USB Combo
[ 5414.384333] usb 1-9: Manufacturer: JMicron
[ 5414.384335] usb 1-9: SerialNumber: 13231874C888
[ 5414.384540] usb 1-9: usb_probe_device
[ 5414.384545] usb 1-9: configuration #1 chosen from 1 choice
[ 5414.385079] usb 1-9: adding 1-9:1.0 (config #1, interface 0)
[ 5414.385620] usb-storage 1-9:1.0: usb_probe_interface
[ 5414.385628] usb-storage 1-9:1.0: usb_probe_interface - got id
[ 5414.386473] scsi8 : usb-storage 1-9:1.0
[ 5414.386687] drivers/usb/core/inode.c: creating file '004'
[ 5415.388845] scsi 8:0:0:0: Direct-Access Hitachi HDS5C3030ALA630 PQ: 0 ANSI: 2 CCS
[ 5415.389606] sd 8:0:0:0: Attached scsi generic sg4 type 0
[ 5415.390076] sd 8:0:0:0: [sdd] 1565565872 512-byte logical blocks: (801 GB/746 GiB)
[ 5415.392739] sd 8:0:0:0: [sdd] Write Protect is off
[ 5415.392746] sd 8:0:0:0: [sdd] Mode Sense: 00 38 00 00
[ 5415.393323] sd 8:0:0:0: [sdd] Asking for cache data failed
[ 5415.393327] sd 8:0:0:0: [sdd] Assuming drive cache: write through
[ 5415.397319] sd 8:0:0:0: [sdd] Asking for cache data failed
[ 5415.397323] sd 8:0:0:0: [sdd] Assuming drive cache: write through
[ 5415.437740] sdd:
[ 5415.443460] sd 8:0:0:0: [sdd] Asking for cache data failed
[ 5415.443466] sd 8:0:0:0: [sdd] Assuming drive cache: write through
[ 5415.443470] sd 8:0:0:0: [sdd] Attached SCSI disk

nirvana linux-2.6.38.2 # parted /dev/sdd
GNU Parted 2.3
Using /dev/sdd
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Error: Can't have a partition outside the disk!
(parted)
[/code]
Back to top
View user's profile Send private message
chithanh
Developer
Developer


Joined: 05 Aug 2006
Posts: 2158
Location: Berlin, Germany

PostPosted: Sun Apr 17, 2011 9:48 am    Post subject: Reply with quote

Ok in that case I take that back and it could be a problem of the USB enclosure. Did you try to contact the manufacturer? If it doesn't work properly on Windows, then they can hardly refuse support.
Back to top
View user's profile Send private message
Anon-E-moose
Watchman
Watchman


Joined: 23 May 2008
Posts: 6098
Location: Dallas area

PostPosted: Sun Apr 17, 2011 10:52 am    Post subject: Reply with quote

A lot of USB external enclosures, mention that they are good up to 2TB.
It probably requires a firmware upgrade (if offered) for the enclosure.
Something to keep in mind if one wants to use 2TB+ disks in them.


Note: when disks get over 2TB then many systems have problems with the MBR.
GPT (GUID Partition Table) will be the next big thing for VLD (Very Large Disks)
but very few motherboard and enclosure manufacturers support that yet.


Edit to add: from this link http://ubuntuforums.org/archive/index.php/t-1657961.html
Quote:
Unfortunately, my old RX35-AT-SC enclosures (I have two) are not compatible with 3TB drives.

_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland


Last edited by Anon-E-moose on Sun Apr 17, 2011 9:03 pm; edited 1 time in total
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54244
Location: 56N 3W

PostPosted: Sun Apr 17, 2011 11:10 am    Post subject: Reply with quote

The enclosure should not care about data on the drive, including filesystem metadata.
The OS handles the filesystem, the drive gets commands at the block level.

800G is a strange size to get reported too. Its not the 137G limit of early LBA so the enclosure is clearly seeing 48 bit LBA, or it couldn't report the drive size as 800G. The drives response to the identify command is a 512b data block ... just like any block read off the disk.
The enclosure shouldn't care about that either.

As I said earlier, a 3T drive will use 4k physical sectors, 512b blocks will be faked. That should not be an issue for the enclosure.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Anon-E-moose
Watchman
Watchman


Joined: 23 May 2008
Posts: 6098
Location: Dallas area

PostPosted: Sun Apr 17, 2011 11:54 am    Post subject: Reply with quote

NeddySeagoon wrote:
The enclosure should not care about data on the drive,


It may not care about the data, per se, but if it doesn't understand the geometry of the disk,
then there will be a problem, and the electronics/interface/firmware on the enclosure is what handles that.

PC's had that problem long ago where they couldn't handle 8+ gig disks.
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54244
Location: 56N 3W

PostPosted: Sun Apr 17, 2011 2:02 pm    Post subject: Reply with quote

Anon-E-moose,

Disk Geometry has been a fiction since drives reached 4G. Everything is Logical Block Addressing (LBA) now.
Geometry is only used by old tools and operating systems and as you say, the most you can address with CHS is about 8G
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
frostschutz
Advocate
Advocate


Joined: 22 Feb 2005
Posts: 2977
Location: Germany

PostPosted: Sun Apr 17, 2011 2:22 pm    Post subject: Reply with quote

Can you actually put >2TB of data on the drive in windows through USB?

The enclosure has to support it (the usb<->sata translation unfortunately isn't "dumb" enough to ignore the size), otherwise it shouldn't work at all.

If there's any way to give your Linux box eSATA capability, do that and forget about USB... it's much faster than USB2 too...
Back to top
View user's profile Send private message
Anon-E-moose
Watchman
Watchman


Joined: 23 May 2008
Posts: 6098
Location: Dallas area

PostPosted: Sun Apr 17, 2011 2:28 pm    Post subject: Reply with quote

LBA still has it's problems and limits.
So no we haven't gotten away from geometry, we've just expanded it out some :lol:


There are many articles mentioning these problems, here's part of one

Quote:
moving to a capacity greater than 2.1 TB requires more work than merely upping the areal density. In fact, most PCs just aren't built to cope with hard drive capacities beyond that limit thanks to the original logical block addressing (LBA) standard set by Microsoft and IBM twenty years ago.

We've heard this before song and dance before: the limited LBA was originally designed for DOS, and only allotted 512-byte sectors for each sector, locking capacities to a 2.1 TB limit. According to Craig, at the time no one imagined that technology would allow for capacities over 2.1 TB when the LBA standard was developed in 1980.

With that said, Long LBA addressing will need to be applied to get around the capacity lock, requiring 64-bit versions of Windows 7 and Windows Vista, and modified versions of Linux. As for Windows XP, Long LBA addressing leaves the older OS out of the picture. Craig said that in-house tests have shown that only 990 MB of a 3 TB drive is available in XP, with the remaining 2.1 TB literally unseen by the OS.

But the current LBA isn't the only hurdle. Craig said that a new GUID partition table (GPT) will need to be released, as current master boot record partitions are locked to 2.1 TB. GPT and Long LBA addressing are now part of the new UEFI system, however this new BIOS replacement hasn't become standard, and is only used in a minimum number of motherboards. As it stands now, a 3 TB drive is pointless until the industry is overhauled, including motherboards, RAID controllers, drivers, and operating systems.



Anyway enough of my prattling.
Be well my friend
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland
Back to top
View user's profile Send private message
DONAHUE
Watchman
Watchman


Joined: 09 Dec 2006
Posts: 7651
Location: Goose Creek SC

PostPosted: Sun Apr 17, 2011 6:55 pm    Post subject: Reply with quote

for example $6 solution
_________________
Defund the FCC.
Back to top
View user's profile Send private message
Diovanti
n00b
n00b


Joined: 27 Feb 2003
Posts: 63

PostPosted: Sun Apr 17, 2011 8:56 pm    Post subject: Reply with quote

Thanks for all the replies. The enclosure just isn't compatible with a 3TB drive. I've ordered one specifically that offers 3TB support.

While I could add e-sata to my gentoo box, this drive is going to be soon hooked up to a plug computer (Guruplug) and all it has are two USB ports.
Back to top
View user's profile Send private message
mbjr
Guru
Guru


Joined: 17 Jan 2004
Posts: 531
Location: Budapest/Hungary

PostPosted: Mon Dec 23, 2013 7:13 pm    Post subject: Reply with quote

*bump*

Did the new enclosure solve it?
I have 3 different enclosures from different manufacturers, an older one back from a year ago and and one I have just purchased to solve this particular problem :>
_________________
mb
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum