SOLVED: DD - File size limit exceeded

[Evo]

It seems to me dd should always work if you play the dvd for a short time in a media player. But about half seem to fail. Is there anything I can do to help this?

An example of an error I get is

Code: Select all

chris@chris-laptop:~$ dd if=/dev/hdc of=/media/WD\ COMBO/SAMURAI_CHAMPLOO_VOL2.iso bs=2048
File size limit exceeded
chris@chris-laptop:~$

The original dvd is 5.5gb and the output is 4gb. Thoughts? Also I would like to make md5 checksum of the original disc to compare with the image. How would go about this?

Thanks for your help!

[desultory]

As to the first issue, it could be an issue which you could resolve with ulimit or it could be a filesystem issue.
As to the second, md5sum /dev/hdc > sum, then edit the resulting file to match the name of the image, then md5sum -c sum.

[Evo]

I'm trying to use dd to make an image of a dvd, but am getting File size limit exceeded error when I hit 4gbs. I am writing the file to a ext3 external hard drive. I used mke2fs -j -O dir_index /dev/sda to format the drive. Any ideas how I can fix this?

[yabbadabbadont]

I checked the coreutils ebuild (which includes dd) and large file support is enabled by default. Does anyone know what the actual size limit is for a single file on ext2/3 filesystem?

EDIT: Ok, found it in the kernel docs. Doesn't appear to be a filesystem limitation.

Code: Select all

Limitations
-----------

There are various limits imposed by the on-disk layout of ext2.  Other
limits are imposed by the current implementation of the kernel code.
Many of the limits are determined at the time the filesystem is first
created, and depend upon the block size chosen.  The ratio of inodes to
data blocks is fixed at filesystem creation time, so the only way to
increase the number of inodes is to increase the size of the filesystem.
No tools currently exist which can change the ratio of inodes to blocks.

Most of these limits could be overcome with slight changes in the on-disk
format and using a compatibility flag to signal the format change (at
the expense of some compatibility).

Filesystem block size:     1kB        2kB        4kB        8kB

File size limit:          16GB      256GB     2048GB     2048GB
Filesystem size limit:  2047GB     8192GB    16384GB    32768GB

[Evo]

I tried the same operation with sudo, but the same result...

[Evo]

The problem doesn't seem to be ulimit.

chris@chris-laptop:~$ ulimit
unlimited

[yabbadabbadont]

Are you just dd'ing from the unmounted device for your dvd drive?

e.g. dd if=/dev/hdc of=/home/user/filename.iso

[Evo]

I think it mounts on it's own.

Code: Select all

chris@chris-laptop:~$ mount
/dev/hda1 on / type ext3 (rw,errors=remount-ro)
proc on /proc type proc (rw)
/sys on /sys type sysfs (rw)
varrun on /var/run type tmpfs (rw)
varlock on /var/lock type tmpfs (rw)
procbususb on /proc/bus/usb type usbfs (rw)
udev on /dev type tmpfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
devshm on /dev/shm type tmpfs (rw)
lrm on /lib/modules/2.6.15-26-386/volatile type tmpfs (rw)
/dev/sda1 on /media/WD COMBO type vfat (rw,nosuid,nodev,quiet,shortname=mixed,uid=1000,gid=1000,umask=077,iocharset=utf8)
/dev/hdc on /media/cdrom0 type iso9660 (ro,noexec,nosuid,nodev,user=chris)
chris@chris-laptop:~$

[yabbadabbadont]

It shouldn't be mounted when you run the dd command. But that shouldn't have anything to do with the file size problem.

[Evo]

It seems it has something to do with the external hard drive, because I dd completed when I output to my home. Any ideas what is going on? I know FAT only allows 4gb in size, but I formated the external drive with ext3.

[Evo]

Perhaps I formated the external drive incorrectly?

Code: Select all

chris@chris-laptop:~$ fdisk /dev/sda
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.


The number of cylinders for this disk is set to 24321.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): p

Disk /dev/sda: 200.0 GB, 200049647616 bytes
255 heads, 63 sectors/track, 24321 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System

Command (m for help):

[Evo]

I'm trying to format my external hard drive. I made a primary partition using fdisk and formatted it using mke2fs -j -O dir_index /dev/sda. However, I get a strange output when I fdisk p and more importantly I cannot copy more than 4gb sized files to the hard drive.

Code: Select all

chris@chris-laptop:~$ fdisk /dev/sda
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.


The number of cylinders for this disk is set to 24321.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): p

Disk /dev/sda: 200.0 GB, 200049647616 bytes
255 heads, 63 sectors/track, 24321 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System

Command (m for help): 

[desultory]

The reason for the strange output is that there are no partitions, as such the output you posted is not strange at all. A filesystem was created on the raw device, not in a partition.
As for the size issue, is that not getting [topic=492368]enough[/topic] [topic=492580]attention[/topic]?

[Evo]

Thanks for the reply, the other thread was solved as it turned out not be dd related at all.

How do I format the parition after I make it? /dev/sda is the external hard drive. mke2fs -j -O dir_index /dev/sda1?

Also I'm trying to repair an external hard drive that says its having cylinder redundancy check errors I think. I just want to format it to FAT filesystem so it can be used in windows.

fdisk and create primary or extended partition?
can i format fat in linux?

[slycordinator]

If you want to format it as fat32, do "mkfs.vfat /dev/sda1" (replace sda1 with the actual partition).

[Gentree]

Code: Select all

File size limit exceeded

Seems pretty explicit. What's the fs?

[desultory]

Before partitioning the drive overwrite the entire drive so that the drive will allow itself to reallocate its spare sectors to compensate for the sectors which are apparently malfunctioning.

[NeddySeagoon]

desultory,

Drives do spare sector allocation on the basis of a read, not a write. Its only when data is read that the drive can say to itself "well, thats doesn't look right" and allocate a spare sector.

[desultory]

Even so, at least have the format done so as to check the drive for errors.

[NeddySeagoon]

desultory,

You can run badblocks as a part of mkfs operation. I think its -c or -cc bepending on the leel of checking you require.

[desultory]

Which would be what I was making reference to, though mkfs and some mkfs backends do not indicate supporting -cc as mkfs.ext2, for example, does.

[jmbsvicetto]

[mod]I've merged three threads on the same issue together.

Evo, please read the Forums Guidelines - you should not cross-post a question in multiple forums. Please don't do that again. If you would like for a thread to be moved to another forum, please ask so here.[/mod]