| View previous topic :: View next topic |
| Author |
Message |
SarahS93
l33t
Joined: 21 Nov 2013
Posts: 706
|
 Posted: Sun May 19, 2024 5:10 pm Post subject: create file into file into file - space goes wrong?! |
 |
|
i will do (for testing) create a container and into it a second container and into that a third one.
| Code: |
file1
'- file2
'- file3
|
in "in/" is a file that i will copy into third container (file3).
every container has a luks-layer and filesystem-layer (ext4)
(dont know if ext4 good for that, i do try it)
so i do calculate "filesize that i will copy into container 3" +3GB, for every container 1gb overhead, thats to much but no much enough!?
i do
| Code: | stat -c %s in/* > temp/size
echo $(( "$(cat temp/szie)"+1073741824)) > temp/szie_plus_1gb
echo $(( "$(cat temp/szie)"+2147483648)) > temp/szie_plus_2gb
echo $(( "$(cat temp/szie)"+3221225472)) > temp/szie_plus_3gb
|
in temp/ there are the 4 files and the value into that file:
| Code: | size 73000908424
temp/szie_plus_1gb 74074650248
temp/szie_plus_2gb 75148392072
temp/szie_plus_3gb 76222133896 |
than i do
| Code: | fallocate -l $(cat temp/size_plus_3gb) temp/file1
cryptsetup .... luksFormat temp/file1
cryptsetup .... luksOpen temp/file1 file13
mkfs.ext4 -q -m 0 /dev/mapper/file13
mkdir -p temp/file13_decrypt
mount /dev/mapper/file13 temp/file13_decrypt |
"fdisk /dev/mapper/file13" shows me "76222133760" byte.
"df -H" shows me "75G"
"du -B1" shows me 74417565696
in the next step i will create file2 with "75148392072" byte size, but that will not work, there are only "74417565696" byte space left, i dont know why?!
where does the free space go to?!
what do i wrong?! |
|
| Back to top |
|
 |
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54418
Location: 56N 3W
|
| Posted: Sun May 19, 2024 6:44 pm Post subject: |
|
|
SarahS93,
A filesystem is a 'container'.
When you make a filesystem an a block device the filesystem will use all of the block device but some of the space is used by the filesystem itself to track the user data.
ext4 has a root directory, an inode table, a journal and backups of some of its structures too, so not all of the space on the block device is available for user data.
A twist is that the extX filesystems also reserve 5% for use by root, so root and ordinary users get different answers for free space.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
SarahS93
l33t
Joined: 21 Nov 2013
Posts: 706
|
| Posted: Sun May 19, 2024 8:01 pm Post subject: |
|
|
with "mkfs.ext4 -q -m 0 /dev/mapper/file13" i do set the 5% to 0%.
should i use an other filesystem in my case?
or is my calculation so wrong?
| Code: | 75148392072 byte i need
74417565696 byte are there
730826376 byte are missing |
|
|
| Back to top |
|
|
SarahS93
l33t
Joined: 21 Nov 2013
Posts: 706
|
| Posted: Sun May 19, 2024 8:39 pm Post subject: |
|
|
do i calculate the overhead for every container with +2 and not with +1 gb, it works.
but 2gb overhead for an container with ext4 filesytem? |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54418
Location: 56N 3W
|
| Posted: Mon May 20, 2024 11:37 am Post subject: |
|
|
SarahS93,
The | Quote: | | 730826376 byte are missing |
are used by the filesystem to keep track of user data. They are not missing.
That's called the filesystem metadata.
You can turn off the journal if you wish, reduce the number of i-nodes and maybe a few other things to reduce the metadata size.
Be warned that each file stored in an ext4 filesyestem needs a least one i-node and the i-node count is fixed when the filesystem is created.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
Genone
Retired Dev
Joined: 14 Mar 2003
Posts: 9553
Location: beyond the rim
|
| Posted: Mon May 20, 2024 12:01 pm Post subject: |
|
|
| If you intend to only put a single file onto that filesystem you probably want to tweak the filesystem parameters like blocksize, inodesize, groupsize and so on to minimize wasted space. |
|
| Back to top |
|
|
SarahS93
l33t
Joined: 21 Nov 2013
Posts: 706
|
| Posted: Tue May 21, 2024 3:09 pm Post subject: |
|
|
is there a formula to calculate from the overhead from a ext4 filesystem?
if i need 70.000.000.000 byte in the ext4 filesystem for one single file, how can i calculate the overhead to know how big the container must be? |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54418
Location: 56N 3W
|
| Posted: Tue May 21, 2024 3:20 pm Post subject: |
|
|
SarahS93,
The answer is not fixed. You can change the space required by the filesystem by changing the parameters to
Will there be a journal or not?
How many i-nodes?
Superblock backup copies?
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
Genone
Retired Dev
Joined: 14 Mar 2003
Posts: 9553
Location: beyond the rim
|
| Posted: Tue May 21, 2024 3:42 pm Post subject: |
|
|
| SarahS93 wrote: | is there a formula to calculate from the overhead from a ext4 filesystem?
if i need 70.000.000.000 byte in the ext4 filesystem for one single file, how can i calculate the overhead to know how big the container must be? |
You probably want to use the largest blocksize and least amount of inodes and groups possible. Not aware of any "formula" as nobody is doing nonsense like this. |
|
| Back to top |
|
|
lagalopex
Guru
Joined: 16 Oct 2004
Posts: 564
|
| Posted: Tue May 21, 2024 4:56 pm Post subject: |
|
|
Why use a filesystem at all, when only one "file" is stored?
You can do it the other way around, tune your parameters, mkfs a filesystem that is big enough, create the file and then shrink the fs to the minimum. (e.g. resize2fs -M) |
|
| Back to top |
|
|
SarahS93
l33t
Joined: 21 Nov 2013
Posts: 706
|
| Posted: Tue May 21, 2024 8:50 pm Post subject: |
|
|
I will try the resize way....
But how to use without a File System? How should this work? |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3402
Location: Rasi, Finland
|
| Posted: Tue May 21, 2024 9:18 pm Post subject: |
|
|
| lagalopex wrote: | Why use a filesystem at all, when only one "file" is stored?
You can do it the other way around, tune your parameters, mkfs a filesystem that is big enough, create the file and then shrink the fs to the minimum. (e.g. resize2fs -M) |
This makes no sense (to me at least).
Without filesystem you'd- first get the exact file length/size.
- create partition (or lv if using lvm) of the same size
- then simply dd (or just cat, pv if you want progress bar) the file and redirect the output into the partition.
That's the "no filesystem" route.
I believe some databases can be created in such way to get rid of filesystem overhead (citation needed, yes). The database needs to know the partition/lv size in case it needs more space.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
pingtoo
l33t
Joined: 10 Sep 2021
Posts: 991
Location: Richmond Hill, Canada
|
| Posted: Tue May 21, 2024 9:25 pm Post subject: |
|
|
| Zucca wrote: | | I believe some databases can be created in such way to get rid of filesystem overhead (citation needed, yes). The database needs to know the partition/lv size in case it needs more space. |
It is known as raw device. Oracle does it for example. |
|
| Back to top |
|
|
SarahS93
l33t
Joined: 21 Nov 2013
Posts: 706
|
| Posted: Tue May 21, 2024 10:31 pm Post subject: |
|
|
i do for testing
| Code: | mkdir test
cd test
mkdir -p file1_mount
mkdir -p file2_mount
mkdir -p file3_mount
fallocate -l 150G file1
mkfs.ext4 -q -m 0 file1
mount file1 file1_mount
fallocate -l 140G file1_mount/file2
mkfs.ext4 -q -m 0 file1_mount/file2
mount file1_mount/file2 file2_mount
fallocate -l 130G file2_mount/file3
mkfs.ext4 -q -m 0 file2_mount/file3
mount file2_mount/file3 file3_mount |
copy my 68,0 GiB (73.000.908.424 byte) file into file3_mount
next
| Code: | umount file3_mount
e2fsck -f file2_mount/file3
resize2fs -M file2_mount/file3
umount file2_mount
e2fsck -f file1_mount/file2
resize2fs -M file1_mount/file2
umount file1_mount
e2fsck -f file1
resize2fs -M file1
mount file1 file1_mount
mount file1_mount/file2 file2_mount
mount file2_mount/file3 file3_mount
df -B1 file*_mount
/dev/loop0 78191632384 75163779072 3011076096 97% /temp/file1_mount
/dev/loop1 77205852160 74082242560 3106832384 96% /temp/file2_mount
/dev/loop2 75871485952 73000947712 2853761024 97% /temp/file3_mount |
are there any way to reduce the 3% + 4% and 3% of not used space? |
|
| Back to top |
|
|
lagalopex
Guru
Joined: 16 Oct 2004
Posts: 564
|
| Posted: Wed May 22, 2024 4:28 am Post subject: |
|
|
| Zucca wrote: | | lagalopex wrote: | Why use a filesystem at all, when only one "file" is stored?
You can do it the other way around, tune your parameters, mkfs a filesystem that is big enough, create the file and then shrink the fs to the minimum. (e.g. resize2fs -M) |
This makes no sense (to me at least). |
Which point exactly makes no sense for you? 
In the initial post it was luks+ext4:
one file with luks->ext4->one file with luks->ext4->one file with luks->ext4->one file
Luks does not need a file, so why not skip on ext4:
one file with luks->luks->luks->ext4->one file
And if the last "file" is not needed to be a file as well, it also can use the block device provided by luks.
| SarahS93 wrote: | | are there any way to reduce the 3% + 4% and 3% of not used space? |
As already mentioned: Increase block size, cluster size, inode ratio. Disable journal. /etc/mke2fs.conf also contains some presets. |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3402
Location: Rasi, Finland
|
| Posted: Wed May 22, 2024 7:01 am Post subject: |
|
|
| lagalopex wrote: | Luks does not need a file, so why not skip on ext4:
one file with luks->luks->luks->ext4->one file |
Ok. I got it now. :D
To be honest, SarahS93, nested lvm lv's will probably only give you headache.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
logrusx
Veteran
Joined: 22 Feb 2018
Posts: 1627
|
| Posted: Wed May 22, 2024 7:11 am Post subject: |
|
|
| pingtoo wrote: | | Zucca wrote: | | I believe some databases can be created in such way to get rid of filesystem overhead (citation needed, yes). The database needs to know the partition/lv size in case it needs more space. |
It is known as raw device. Oracle does it for example. |
There's still metadata required to organize the database. This is what's referred to as overhead which word to me seems like is widely used inappropriately here. Oracle is just eliminating the filesystem metadata which they apparently do not need.
SarahS93,
I get the sense you're trying to do something other than the presented topic. What's the reason behind that test?
Best Regards,
Georgi |
|
| Back to top |
|
|
SarahS93
l33t
Joined: 21 Nov 2013
Posts: 706
|
| Posted: Sat May 25, 2024 6:33 am Post subject: |
|
|
ok, i have now do this
| Code: |
fallocate -l 75G file
cryptsetup .... luksFormat temp/file
cryptsetup .... luksOpen temp/file file13
cryptsetup .... luksFormat /dev/mapper/file13
cryptsetup .... luksOpen /dev/mapper/file13 file23
cryptsetup .... luksFormat /dev/mapper/file23
cryptsetup .... luksOpen /dev/mapper/file23 file33
mkfs.ext4 -q -m 0 /dev/mapper/file33
mkdir -p temp/decrypt
mount /dev/mapper/file33 temp/decrypt
lsblk
loop0 /dev/loop0 loop
└─file13 /dev/mapper/file13 crypt
└─file23 /dev/mapper/file23 crypt
└─file33 /dev/mapper/file33 crypt /mnt/sdb1/temp/decrypt
mount
/dev/mapper/file33 on /mnt/sdb1/temp/decrypt type ext4 (rw,relatime)
mkdir temp/decrypt/test
mkdir: das Verzeichnis „temp/decrypt/test“ kann nicht angelegt werden: Ungültige Nachricht |
i can not use and write into decrypt/ |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 21835
|
| Posted: Sat May 25, 2024 3:19 pm Post subject: |
|
|
You may get better help if you answer the questions put to you, particularly about why you are trying to do this.
In the meantime, for the benefit of those who do not speak your native language, please show the error message in English, by reproducing the error with LC_MESSAGES=C. I might (or might not) be able to tell you what your problem is if I could read the localized error message, but I cannot. |
|
| Back to top |
|
|
sublogic
Apprentice
Joined: 21 Mar 2022
Posts: 224
Location: Pennsylvania, USA
|
| Posted: Sat May 25, 2024 10:36 pm Post subject: |
|
|
Are you showing us everything ?
| SarahS93 wrote: | | Code: | fallocate -l 75G file
cryptsetup .... luksFormat temp/file
cryptsetup .... luksOpen temp/file file13 |
|
You must have run the fallocate command from a directory named temp, and done a cd .. before running the cryptsetups --otherwise there would not have been a temp/file, only a ./file .
| SarahS93 wrote: | | Code: | cryptsetup .... luksFormat /dev/mapper/file13
cryptsetup .... luksOpen /dev/mapper/file13 file23
cryptsetup .... luksFormat /dev/mapper/file23
cryptsetup .... luksOpen /dev/mapper/file23 file33
mkfs.ext4 -q -m 0 /dev/mapper/file33
mkdir -p temp/decrypt
mount /dev/mapper/file33 temp/decrypt
lsblk
loop0 /dev/loop0 loop
└─file13 /dev/mapper/file13 crypt
└─file23 /dev/mapper/file23 crypt
└─file33 /dev/mapper/file33 crypt /mnt/sdb1/temp/decrypt
mount
/dev/mapper/file33 on /mnt/sdb1/temp/decrypt type ext4 (rw,relatime) |
|
Okay, so you started in directory /mnt/sdb1/temp, moved up to /mnt/sdb1 and used a relative path temp/decrypt, without a leading slash, in your mkdir and mount commands. The lsblk and mount commands agree that file33 is mounted on /mnt/sdb1/temp/decrypt.
Now, did you change directory again before running the next command ?
| SarahS93 wrote: | | Code: | mkdir temp/decrypt/test
mkdir: das Verzeichnis „temp/decrypt/test“ kann nicht angelegt werden: Ungültige Nachricht |
|
If you were still in /mnt/sdb1 it should have worked. Otherwise, try using the absolute path, mkdir /mnt/sdb1/temp/decrypt/test. |
|
| Back to top |
|
|
SarahS93
l33t
Joined: 21 Nov 2013
Posts: 706
|
| Posted: Sun May 26, 2024 12:34 am Post subject: |
|
|
i think the problem is an other one
in syslog i see something strange
cd decrypt/
| Code: | ls
kernel: EXT4-fs error (device dm-12): htree_dirblock_to_tree:1080: inode #2: comm ls: Directory block failed checksum |
| Code: | mkdir test
kernel: EXT4-fs error (device dm-12): __ext4_find_entry:1689: inode #2: comm mkdir: checksumming directory block 0 |
dont know what going wrong
is the way with fallocate wrong in interaction with luks in luks in luks ?
if i do unmount decrypt/ and run e2fsck /dev/mapper/file33
| Code: | e2fsck /dev/mapper/file33
e2fsck 1.47.0 (5-Feb-2023)
ext2fs_check_desc: Defekter Gruppendeskriptor: defekter Block für die Block-Bitmap
e2fsck: Gruppen-Deskriptoren scheinen defekt zu sein... Datensicherungs-Blöcke werden versucht ...
e2fsck: Ungültige magische Zahl im Superblock beim Benutzen der Datensicherungs-Blöckee2fsck: es wird zum originalen Superblock zurück gekehrt
Hinweis: Wenn mehrere Inodes oder Bitmap-Blöcke
neu geordnet werden müssen, oder ein Teil der Inode-Tabelle
verschoben werden muss, könnte es helfen, e2fsck erst einmal
mit der Option „-b 32768“ zu starten. Das Problem könnte
im primären Blockgruppenbezeichner liegen und seine
Sicherungskopie in Ordnung sein.
Block-Bitmap für Gruppe 448 ist nicht in der Gruppe. (Block 4792790775515546958)
Zurücksetzen<jy>? ja
Inode-Bitmap für Gruppe 448 ist nicht in der Gruppe. (Block 5635464355302573537)
Zurücksetzen<jy>? ja
Inode-Tabelle für Gruppe 448 ist nicht in der Gruppe. (Block 1318184797462194134)
WARNUNG: GROSSER DATENVERLUST IST MÖGLICH.
Zurücksetzen<jy>? ja
Die Prüfsumme eines oder mehrerer Gruppendeskriptoren ist ungültig. Reparieren<jy>? ja
Die Prüfsumme des Gruppendeskriptors 448 ist 0x182e, sie sollte 0x6c60 sein. REPARIERT.
Block-Bitmap für Gruppe 449 ist nicht in der Gruppe. (Block 17646299346509250498)
Zurücksetzen<jy>? ja
Inode-Bitmap für Gruppe 449 ist nicht in der Gruppe. (Block 1550928633865306464)
Zurücksetzen<jy>? ja
Inode-Tabelle für Gruppe 449 ist nicht in der Gruppe. (Block 721865938269672767)
WARNUNG: GROSSER DATENVERLUST IST MÖGLICH.
Zurücksetzen<jy>? ja
Die Prüfsumme des Gruppendeskriptors 449 ist 0x9b3d, sie sollte 0x468d sein. REPARIERT.
Block-Bitmap für Gruppe 450 ist nicht in der Gruppe. (Block 17646299346509250498)
Zurücksetzen<jy>? ja
Inode-Bitmap für Gruppe 450 ist nicht in der Gruppe. (Block 1550928633865306464)
Zurücksetzen<jy>? ja
Inode-Tabelle für Gruppe 450 ist nicht in der Gruppe. (Block 721865938269672767)
WARNUNG: GROSSER DATENVERLUST IST MÖGLICH.
Zurücksetzen<jy>? ja
Die Prüfsumme des Gruppendeskriptors 450 ist 0x9b3d, sie sollte 0x2ac0 sein. REPARIERT.
Block-Bitmap für Gruppe 451 ist nicht in der Gruppe. (Block 17646299346509250498)
Zurücksetzen<jy>? ja
Inode-Bitmap für Gruppe 451 ist nicht in der Gruppe. (Block 1550928633865306464)
Zurücksetzen<jy>? ja
Inode-Tabelle für Gruppe 451 ist nicht in der Gruppe. (Block 721865938269672767)
WARNUNG: GROSSER DATENVERLUST IST MÖGLICH.
Zurücksetzen<jy>? ja
Die Prüfsumme des Gruppendeskriptors 451 ist 0x9b3d, sie sollte 0x0efb sein. REPARIERT.
Block-Bitmap für Gruppe 452 ist nicht in der Gruppe. (Block 17646299346509250498)
Zurücksetzen<jy>? ja
Inode-Bitmap für Gruppe 452 ist nicht in der Gruppe. (Block 1550928633865306464)
Zurücksetzen<jy>? ja
Inode-Tabelle für Gruppe 452 ist nicht in der Gruppe. (Block 721865938269672767)
WARNUNG: GROSSER DATENVERLUST IST MÖGLICH.
Zurücksetzen<jy>? ja
Die Prüfsumme des Gruppendeskriptors 452 ist 0x9b3d, sie sollte 0xf25a sein. REPARIERT.
Block-Bitmap für Gruppe 453 ist nicht in der Gruppe. (Block 17646299346509250498)
Zurücksetzen<jy>? ja
Inode-Bitmap für Gruppe 453 ist nicht in der Gruppe. (Block 1550928633865306464)
Zurücksetzen<jy>?
........ |
after that repair, i do mount file33 again at decrypt/
i can do ls and can do mkdir test now
what have i do wrong!?
:edit
i unmount all and do all steps new again
at "mkfs.ext4 -q -m 0 /dev/mapper/file33" i see
/dev/mapper/file33 enthält Daten von „OpenPGP Public Key”
Trotzdem fortfahren? (j,n) |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3402
Location: Rasi, Finland
|
| Posted: Sun May 26, 2024 6:07 am Post subject: |
|
|
Please prepend your commands with LC_MESSAGES=C, like Hu said, so that the messages produced by commands are in English.
Also perhaps you could tell us what are you trying to achieve. This method seem overly complicated. Maybe we could present you an alternative way?
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3185
|
| Posted: Sun May 26, 2024 10:10 am Post subject: |
|
|
He's making a Fort Knox in a snow globe; there's no practical reason behind it.
Humor him or move along.
_________________
Make Computing Fun Again |
|
| Back to top |
|
|
SarahS93
l33t
Joined: 21 Nov 2013
Posts: 706
|
| Posted: Sun May 26, 2024 7:04 pm Post subject: |
|
|
how have i to use "LC_MESSAGES=C" ?
The message at syslog, if i try to "mkdir temp/decrypt/test"
| Code: | kernel: EXT4-fs error (device dm-12): __ext4_find_entry:1689: inode #2: comm mkdir: checksumming directory block 0
kernel: EXT4-fs error (device dm-12): __ext4_find_entry:1689: inode #2: comm mkdir: checksumming directory block 0 |
is in englisch. |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3402
Location: Rasi, Finland
|
| Posted: Sun May 26, 2024 9:30 pm Post subject: |
|
|
| SarahS93 wrote: | | how have i to use "LC_MESSAGES=C" ? |
Put it in front (prepend) of the command you're going to run. In this case before e2fsck. | Code: | | LC_MESSAGES=C e2fsck /dev/mapper/file33 |
... then the output should be in english.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
|
Portage & Programming
