Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
NTFS filesystem compatibility and FAT32
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
LIsLinuxIsSogood
l33t
l33t


Joined: 13 Feb 2016
Posts: 968

PostPosted: Thu Apr 11, 2019 8:47 pm    Post subject: NTFS filesystem compatibility and FAT32 Reply with quote

1) What is the best format for a USB drive that will be used to go back and forth between windows and linux? I've experienced some occasional problems with permissions/writing to it as the wrong user. when using FAT32 but I'm willing to continue putting up with that aspect if it is the better choice.
2) Is there some better alternative...perhaps NTFS? From experience what are the drawbacks/benefits of working with NTFS or FAT (from a linux system). I can always adjust how much I will make use of each based on the feedback I received here.
3) For that matter, I have another laptop that the windows system became "corrupt" and I fixed it but now I won't really boot that machine because I don't know enough about MS Windows boot process to fix the boot mechanism...and I am giving up (dont want to fix it now). But since I will keep the partition as a data parition (indefinitely) so would using some of that space which is formatted as NTFS be wise/unwise from another Linux operating system partition?
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 42596
Location: 56N 3W

PostPosted: Thu Apr 11, 2019 9:12 pm    Post subject: Reply with quote

LIsLinuxIsSogood,

There is no right answer. Windows filesystems and Linux filesystems use different and incompatible permissions.
Whatever windows filesystem you choose, you will end up faking permissions when you mount it in Linux.

There is an ext2 driver for Windows but other than it exists, I know nothing of it.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
LIsLinuxIsSogood
l33t
l33t


Joined: 13 Feb 2016
Posts: 968

PostPosted: Thu Apr 11, 2019 9:27 pm    Post subject: Reply with quote

Neddy,
To make matters a little clearer, for the situation of my flash storage devices they will not be used exclusively on my own machines, but on other windows machines that probably will not have those drivers installed. Hence I wanted it to work on both platforms, and I previously had problems with using certain other filesystem types so I was just assuming that I should limit myself to NTFS or FAT. If there is another filesystem that interfaces between the two that might work too. I know lots of people use the cloud to store stuff. I prefer the old fashioned way.


Ty
Back to top
View user's profile Send private message
Fitzcarraldo
Veteran
Veteran


Joined: 30 Aug 2008
Posts: 1635
Location: United Kingdom

PostPosted: Thu Apr 11, 2019 9:43 pm    Post subject: Reply with quote

One of the data partitions on my main laptop (Clevo W230SS running Gentoo amd64) is formatted as NTFS. Never had any trouble using it. The directory the Linux Cloud client syncs with my Cloud server is also on that partition, and I actually store most of my important data on that NTFS partition.

A 6TB WD Elements external USB 3.0 drive, that came formatted as NTFS, is connected to my server, and is used for automatic backups of the users' files on the server; a cronjob launches a Bash script that runs rsync. That works too. My backup script mounts the external USB NTFS drive as follows:

Code:
mount -t ntfs-3g -o async,noexec,nodev,noatime,nodiratime $DEVICE /media/usbhdd


As explained in the blog post Automatic backup of users’ files on a NAS device to an external USB HDD, I throttle rsync in order to avoid buffer overflow when writing to the external HDD:

Code:
--bwlimit=22500


My older laptop (Compal NBLB2) uses Gentoo ~amd64 dual booting with Windows 7 Pro, the latter on an NTFS partition. When that was my main laptop, the Linux Thunderbird e-mail client and the Windows Thunderbird e-mail client used the same e-mail folders and e-mails, which were on the NTFS partition. I never had any trouble with that.

The 1TB external USB 2.0 HDD connected to my family's PC (running Lubuntu) for automatic backup purposes is also formatted as NTFS, and there are no problems with using that either.

FAT32 cannot store files larger than 4GB, so that rules it out for me irrespective of its other limitations.

You might find the following superuser page useful reading: What are NTFS file ownership and file permissions? [closed].
_________________
Clevo W230SS: amd64 OpenRC elogind nvidia-drivers & xf86-video-intel.
Compal NBLB2: ~amd64 OpenRC elogind xf86-video-ati. Dual boot Win 7 Pro 64-bit.
KDE on both.

Fitzcarraldo's blog
Back to top
View user's profile Send private message
CooSee
Guru
Guru


Joined: 20 Nov 2004
Posts: 458
Location: Earth

PostPosted: Fri Apr 12, 2019 4:25 am    Post subject: Reply with quote

hello,

you can even use EXT4 and look for TotalCommnder for Windows and look for File system plugins for TC.

todo so, you can open your USB-Stick or Disk on Windows and be able to use it.

good luck
_________________
beQuiet! Silent Base 800 Black - MSI C236A - Xeon E3-1245v5 - 32GB RAM Kingston - Sapphire Nitro+ Radeon RX590 8G - Samsung PM961 M.2 128GB - Samsung 840 EVO 120GB - Creative Sound Blaster RX PCIe - Logitech Z623 2.1 - G110 Keyboard - Mouse G400
Back to top
View user's profile Send private message
LIsLinuxIsSogood
l33t
l33t


Joined: 13 Feb 2016
Posts: 968

PostPosted: Mon Apr 15, 2019 6:41 am    Post subject: Reply with quote

So some of the suggestions were on track, I like idea of integrating tools from different operating systems with the file manager extension or whatever and even using the one system to mount the filesystem of the other but that would be fine, if I had windows machines of my own, but I do not. I really only use the windows os when working on school/work computers. And generally my linux hosts at all other times. One major concern is the backup of course, when making changes to files, regardless if they are being used on the two different operating systems. I appreciate hearing how using a sync tool like rsync (thanks for the tip fitzcarraldo) over external HDD via USB. Do you know if the specific sync overflow issue mentioned exists in other configurations like when using the tool over a local network connection?

Or if it is just primary a problem with USB backups only? Then that's good to know as well. My main other concern is losing linux file permissions and that kind of thing when going back and forth. So does that hapen when using rsync from ext4 to NTFS? I feel that having chosen FAT filesystem thus far is a problem for the thumb drives that will also in addition to large files may contain other problems with it. But I feel NTFS would better suit me as EXT4 is out of the question since it is not found on windows unaltered. (I understand if there is no perfect answer, and I may have to resort to using cloud storage to manage these files...that would be ok but not ideal.)

By the way since I haven't purchased a new HD in some time, and I may have to is there a brand/make that anyone might suggest looking at? Or a good site that recommends hardware for this?
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 5595

PostPosted: Mon Apr 15, 2019 11:26 am    Post subject: Re: NTFS filesystem compatibility and FAT32 Reply with quote

LIsLinuxIsSogood wrote:
1) What is the best format for a USB drive that will be used to go back and forth between windows and linux?

UDF
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 42596
Location: 56N 3W

PostPosted: Mon Apr 15, 2019 5:34 pm    Post subject: Reply with quote

Ant P.,

UDF is an interesting one.
By default, Linux and Window use different versions of UDF. Read compatibility between versions is a requirement but write compatibility is not.
It should work provided the formatting is done in Linux.

Volumes formatted in Windows are likely to be read only in Linux.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
axl
Guru
Guru


Joined: 11 Oct 2002
Posts: 510
Location: Romania

PostPosted: Mon Apr 15, 2019 6:45 pm    Post subject: Reply with quote

With the advances in virtualization technology it's now become easier to just run the other os in a VM instead. And share files via normal services as if they were 2 different machines. Just a suggestion.
Back to top
View user's profile Send private message
Jaglover
Watchman
Watchman


Joined: 29 May 2005
Posts: 6972
Location: Saint Amant, Acadiana

PostPosted: Mon Apr 15, 2019 6:48 pm    Post subject: Reply with quote

Didn't they invent exFAT to overcome FAT32 limitations.
_________________
Please learn how to denote units correctly!
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 5595

PostPosted: Mon Apr 15, 2019 7:20 pm    Post subject: Reply with quote

exFAT was invented to sneak software patent licensing shakedowns into the SDXC storage format. Ostensibly it exists because FAT32 had a 128GB partition size limit, but that's an arbitrary restriction in Windows and does not apply elsewhere; IIRC you can even create full size FAT32 partitions in Linux and Windows will use them without complaint. The Windows 98 manual spends quite a few words hyping up the fact that the filesystem is good for up to 2TB. That was an unthinkable amount back then but, well, here we are.

The other limitation of FAT32 is 4GB files, but UDF also solves those. As Neddy correctly points out, it's better to make UDF partitions from Linux - it's actually the lowest common denominator.
Back to top
View user's profile Send private message
Fitzcarraldo
Veteran
Veteran


Joined: 30 Aug 2008
Posts: 1635
Location: United Kingdom

PostPosted: Tue Apr 16, 2019 7:57 am    Post subject: Reply with quote

LIsLinuxIsSogood wrote:
Do you know if the specific sync overflow issue mentioned exists in other configurations like when using the tool over a local network connection? Or if it is just primary a problem with USB backups only?


Sometimes you have to use the '--bwlimit' option when using rsync over a network:
e.g.1 https://serverfault.com/questions/884445/limit-rsync-bandwidth-nfs-copy
e.g.2 https://superuser.com/questions/58103/make-rsync-use-less-bandwidth

wrote:
My main other concern is losing linux file permissions and that kind of thing when going back and forth. So does that hapen when using rsync from ext4 to NTFS?


You can see from the examples below that NTFS preserves mtime but not Linux ownership, and adds +x to Linux permissions.

Code:
$ ls -la file1.jpg
-rw-r--r-- 1 bilbo bilbo 106481 Jun 13  2015 file1.jpg
$ cp -p file1.jpg /media/NTFS/file2.jpg
$ cp -p /media/NTFS/file2.jpg ./file3.jpg
$ ls -la file3.jpg
-rwxrwxrwx 1 bilbo bilbo 106481 Jun 13  2015 file3.jpg

Code:
$ ls -la file4.jpg
-rw-r--r-- 1 bilbo users 427421 Apr 16 06:51 file4.jpg
$ cp -p file4.jpg /media/NTFS/file5.jpg
$ cp -p /media/NTFS/file5.jpg ./file6.jpg
$ ls -la file6.jpg
-rwxrwxrwx 1 bilbo bilbo 427421 Apr 16 06:51 file6.jpg


Also, if you examine the attributes of the files on the NTFS partition/disk, the permissions are shown as '-rwxrwxrwx' and the ownership as 'root:root':

Code:
$ ls -la /media/NTFS/file2.jpg
-rwxrwxrwx 1 root root 106481 Jun 13  2015 /media/NTFS/file2.jpg

Code:
$ ls -la /media/NTFS/file5.jpg
-rwxrwxrwx 1 root root 427421 Apr 16 06:51 /media/NTFS/file5.jpg


As the limitation is due to NTFS, the situation is the same if you use rsync:

Code:
$ ls -la file7.png
-rw-r--r-- 1 bilbo bilbo 198797 Jan 11  2018 file7.png
$ rsync --times --perms --protect-args /home/bilbo/file7.png /media/NTFS/file8.png
$ rsync --times --perms --protect-args /media/NTFS/file8.png /home/bilbo/file9.png
$ ls -la file9.png
-rwxrwxrwx 1 bilbo bilbo 198797 Jan 11  2018 file9.png

Code:
$ ls -la file10.png
-rw-r--r-- 1 bilbo users 96990 Apr 10  2018 file10.png
$ rsync --times --perms --protect-args /home/bilbo/file10.png /media/NTFS/file11.png
$ rsync --times --perms --protect-args /media/NTFS/file11.png /home/bilbo/file12.png
$ ls -la file12.png
-rwxrwxrwx 1 bilbo bilbo 96990 Apr 10  2018 file12.png

Code:
$ ls -la /media/NTFS/file8.png
-rwxrwxrwx 1 root root 198797 Jan 11  2018 /media/NTFS/file8.png

Code:
$ ls -la /media/NTFS/file11.png
-rwxrwxrwx 1 root root 96990 Apr 10  2018 /media/NTFS/file11.png


LIsLinuxIsSogood wrote:
By the way since I haven't purchased a new HD in some time, and I may have to is there a brand/make that anyone might suggest looking at? Or a good site that recommends hardware for this?

Are you referring to an external USB drive or to an internal HDD? If the latter, what platter diameter? For 3.5" HDDs I use WD RED. For external USB HDDs I use WD and Toshiba. After an external Seagate USB HDD started sputtering and emitting smoke a few years ago, I avoid Seagate HDDs.
_________________
Clevo W230SS: amd64 OpenRC elogind nvidia-drivers & xf86-video-intel.
Compal NBLB2: ~amd64 OpenRC elogind xf86-video-ati. Dual boot Win 7 Pro 64-bit.
KDE on both.

Fitzcarraldo's blog
Back to top
View user's profile Send private message
LIsLinuxIsSogood
l33t
l33t


Joined: 13 Feb 2016
Posts: 968

PostPosted: Fri Apr 19, 2019 7:51 am    Post subject: Reply with quote

Thanks fitzcarraldo for your information (EDIT: by the way fz I'm also curious to know about the rsync errors, I didn't look at the link yet, but do they tend to actually display an error and stop execution of program or does it appear later more catastrophically when you go to access files and find out there were mistakes made by the application?) this verifies my own ideas that the NTFS filesystem is just an overall problematic way to go. So it does seem the udf/vfat could be the solution...I was wondering if exFAT is in fact something different than what I have here, and also what is the udf doing here as well, I didn't know a F/S could reside on the device without designated space, or is this maybe a remnant of previous data?

Code:
root@JRlinux1:/# lsblk /dev/sda -o NAME,FSTYPE
NAME   FSTYPE
sda    udf
└─sda1 vfat


Here the output is even more confusing showing the same size for sda1/sda but of course the different filesystem types...
Code:
root@JRlinux1:/# lsblk /dev/sda -o NAME,FSTYPE,SIZE,RQ-SIZE,TYPE,VENDOR
NAME   FSTYPE  SIZE RQ-SIZE TYPE VENDOR
sda    udf    58.1G     128 disk SanDisk
└─sda1 vfat   58.1G     128 part
Back to top
View user's profile Send private message
Fitzcarraldo
Veteran
Veteran


Joined: 30 Aug 2008
Posts: 1635
Location: United Kingdom

PostPosted: Sat Apr 20, 2019 6:13 am    Post subject: Reply with quote

LIsLinuxIsSogood,

VFAT is older and less sophisticated than NTFS. You're mistaken if you think Linux file attributes are going to be preserved correctly if you use VFAT. Read the Wikipedia articles on FAT (particularly the section on VFAT) and NTFS, as well as other sources on the Web, such as:

Brien M. Posey, MCSE wrote:
VFAT is an extension of the FAT file system and was introduced with Windows 95.

(In the context of that article, by 'FAT' he meant FAT16.)

Brien M. Posey, MCSE wrote:
FAT32 is actually an extension of FAT and VFAT, first introduced with Windows 95 OEM Service Release 2 (OSR2). FAT32 greatly enhances the VFAT file system but it does have its drawbacks.


Brien M. Posey, MCSE wrote:
Now that I've discussed the FAT, VFAT, and FAT32 file systems, let's examine the NTFS file system. NTFS stands for New Technology File System. Microsoft created NTFS to compensate for the features it felt FAT was lacking.


Linux reports drives formatted as FAT32 to be 'VFAT':

Code:
$ sudo lsblk -no FSTYPE /dev/sdb1
vfat


It would not be difficult for you to partition and format a pen drive and try out each filesystem yourself to find out what happens to file attributes when you copy files to and from a Linux filesystem to a Windows filesystem. GUI tools like GParted make it even easier to partition and format a drive with a Windows filesystem. The only one of the Windows filesystems GParted cannot format presently is exFAT (see https://gparted.sourceforge.io/features.php), but that can be done from the command line; you need to install sys-fs/fuse, sys-fs/exfat-utils and sys-fs/fuse-exfat in Gentoo. Then, once you have created a partition on a pen drive, format it. For example:

Code:
$ sudo mkfs.exfat /dev/sdb1 # Assuming pen drive is sdb.
$ sudo mkdir /media/exfat
$ sudo mount -t exfat /dev/sdb1 /media/exfat


As you will see when you try them yourself, neither VFAT (FAT12, FAT16, FAT32) nor exFAT preserve Linux attributes either.

LIsLinuxIsSogood wrote:
(EDIT: by the way fz I'm also curious to know about the rsync errors, I didn't look at the link yet, but do they tend to actually display an error and stop execution of program or does it appear later more catastrophically when you go to access files and find out there were mistakes made by the application?)

Depends which rsync options you use -- see https://rsync.samba.org/documentation.html -- and what the error is, but usually rsync displays an error message and continues. It's also possible to make rsync log to a log-file (the aforementioned documentation, blogs and other sources such as Stack Exchange will point you in the right direction).
_________________
Clevo W230SS: amd64 OpenRC elogind nvidia-drivers & xf86-video-intel.
Compal NBLB2: ~amd64 OpenRC elogind xf86-video-ati. Dual boot Win 7 Pro 64-bit.
KDE on both.

Fitzcarraldo's blog
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum