EXT4 does not give full drive?

FizzyWidget · Posted: Sat Dec 24, 2011 5:24 pm Post subject: EXT4 does not give full drive?

I will try and elaborate on this

I have in my system a 1TB (930GB) hard drive, when formatted with XFS it gives me the full 930GB, when i use EXT4 it gives me 917GB, so iam losing 16GB right off the bat, even if i remove the reserved blocks and the 5% reserved space for root, I still don't get the full amount.

Is there some option i need to do when formatting the drive? Would a smaller block size help or would that just give me more inodes?

I only ask as i am thinking of moving away from XFS due to the fact it has started to play up on my home partition and a few times i have had the machine lock up on me, when i have rebooted XFS has always said it needs looking at as it couldn't write metadata to the disc and denied access to prevent further damage.

I have been lucky and not lost anything but I dont wish to temp fate and feel using a different filesystem might be the best course for me

The drive contains a mixture of little to large files, the usual txt picture music and video files, so need a system suited to store those size of files, ranging fome a few ytes to 20GB+ (bluray)
_________________
I know 43 ways to kill with a SKITTLE, so taste my rainbow bitch.

titanofold · Posted: Sat Dec 24, 2011 5:39 pm Post subject:

I forget which specific thing it is, but Ext2/3/4 reserves some space. This is configurable.
_________________
The best things in life are free.
Guy-1: Surely, you will fold with me...
Guy-2: Alright, but don't call me Shirley

titanofold · Posted: Sat Dec 24, 2011 5:41 pm Post subject:

FizzyWidget

titanofold

FizzyWidget · Posted: Sat Dec 24, 2011 5:55 pm Post subject:

Thanks, guess its the inodes then, if i use a smaller blocksize i will get more of them and less space? Do i bite the bullet i wonder and lose 16GB or take the risk of XFS taking a big dump :/
_________________
I know 43 ways to kill with a SKITTLE, so taste my rainbow bitch.

mv · Watchman Joined: 20 Apr 2005 Posts: 6747

ext4 needs space for the inode tables. You can decrease the inode ratio by using mkfs.ext4 with e.g. -T largefile or -T largefile4 (see /etc/mke2fs.conf for details).

However, you should be aware that if you run out of inodes you will get a "disk full" error even if there should still be plenty of space.
This shouldn't be an issue if you really have lots of large files on the disk.

FizzyWidget · Posted: Sat Dec 24, 2011 6:00 pm Post subject:

will be loads of little files and loads of big files, guess i will just have to take the loss, bit annoying though, guess someone on the ext4 dev team didnt think that one through

Now to format and move over 500GB again

_________________
I know 43 ways to kill with a SKITTLE, so taste my rainbow bitch.

mv · Watchman Joined: 20 Apr 2005 Posts: 6747

FizzyWidget · Posted: Sat Dec 24, 2011 6:06 pm Post subject:

not many will be that small, normal size jpgs/png/psd mp3 movies, will see what is what after i have moved everything across, I dont think it should be that much of an issue
_________________
I know 43 ways to kill with a SKITTLE, so taste my rainbow bitch.

Ant P. · Watchman Joined: 18 Apr 2009 Posts: 6920

256 bytes per inode, 1 inode per 16KB, that's where your 1.56% of disk space has gone.

If you need an extra 12GB of that and wasting up to 64KB per file isn't an issue then use `mkfs.ext4 -T huge`. XFS/reiserfs/btrfs/* aren't inode-based but they have their own unique ways of losing space, sometimes with existing data in it.

FizzyWidget · Posted: Sat Dec 24, 2011 6:18 pm Post subject:

I dont want to waste anything, was curious as to were the space went, and what would be the best options when formating the drive for an ext4 system
_________________
I know 43 ways to kill with a SKITTLE, so taste my rainbow bitch.

Ant P. · Watchman Joined: 18 Apr 2009 Posts: 6920

Alternatively you could make one filesystem for the large files and put a loopback partition inside it for the other stuff.

NeddySeagoon · Posted: Sat Dec 24, 2011 9:53 pm Post subject:

Dark Foo,

All filesystems need some space for themselves to describe where your data is stored and which space is free.
Some filesystems. like FAT and extX allocate it when the filesystem is made, others assign it as files are added/deleted.
You should not assume that because you can see empty space, you can use it for your data.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

FizzyWidget · Posted: Sat Dec 24, 2011 9:58 pm Post subject:

NeddySeagoon · Posted: Sat Dec 24, 2011 10:06 pm Post subject:

Dark Foo,

If you use LVM2 to divide the disk, you can move empty space around as you need to.
You could make say two partitions, one for big files and one for small files with differing block sizes and inode ratios.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

FizzyWidget · Posted: Sat Dec 24, 2011 10:07 pm Post subject:

i am going to see what the inode count is like after i have added the 500G or so, wont take long on a GB network, i could leave it over night and see what it was in the morning.
_________________
I know 43 ways to kill with a SKITTLE, so taste my rainbow bitch.

Hu · Moderator Joined: 06 Mar 2007 Posts: 21631

You can anticipate with a fairly high level of accuracy how many inodes are needed. You need one inode for each file, directory, and so on. Thus, the output of find . | wc -l at the top of your source hierarchy should tell you how many inodes are used in the source, which will tell you also how many inodes you will use in the destination. You can tune the find expression if there are areas of the source hierarchy which are not copied into the destination.

FizzyWidget · Posted: Sat Dec 24, 2011 11:55 pm Post subject:

FizzyWidget · Posted: Sun Dec 25, 2011 11:29 am Post subject:

Well after transfering near 172,000 files here is what the inode situation is like

NeddySeagoon · Posted: Sun Dec 25, 2011 3:42 pm Post subject:

Dark Foo,

The reserved space for root is intended to prevent normal users from crashing the box by filling up /.
When / is full, nothing can open new files. That means lock files and logs. When that happens, nobody can log in either - not even root.
To regain control, you need to boot with a liveCD and make some space.
The reserved space stops that happening. The 5% was set many years ago when drives were much smaller than they are now.

You only need reserved space on / and /var if thats separate.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

FizzyWidget · Posted: Sun Dec 25, 2011 4:53 pm Post subject:

yes / is about 1GB and that only has about a few hundred meg on it if that, and /var is 10GB as is /usr, so your saying its safe to fully remove the reserved blocks from /home and /storage, well any mount point other than / and /var ?
_________________
I know 43 ways to kill with a SKITTLE, so taste my rainbow bitch.

NeddySeagoon · Posted: Sun Dec 25, 2011 9:07 pm Post subject:

Dark Foo,

Its safe to fully remove the reserved blocks from /home, /storage and /usr.
For other file systems it depends on your setup.

Keep some reserved space for root on any filesystem that the system depends on root being able to write on for normal operation that is also writable by (or on behalf of) ordinary users.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

FizzyWidget · Posted: Sun Dec 25, 2011 9:13 pm Post subject:

my set up is

/
/opt
/usr
/var
/home
/storage

same on laptop except /storage - don't have that on there, but nice to now i can claw some space back on the other mount points

More I learn about Linux the more I think I will like it

_________________
I know 43 ways to kill with a SKITTLE, so taste my rainbow bitch.

NeddySeagoon · Posted: Sun Dec 25, 2011 9:20 pm Post subject:

Dark Foo,

/
/opt
/usr

Should only be writable by root anyway, so having the reserved space there makes no difference.
By definition, root can use it and users can't write there at all.

/var is where your logs and lock files go, so you need the reserved space here, in case a user does something silly.

/home and /storage are not used for system operations, so you don't need to reserve space for root here either.
roots home dir is /root.

Where is your /tmp ?
If /tmp is on /, you must reserve space for root on / since users can write to /tmp, potentially filling /, which is a very bad thing.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.