| View previous topic :: View next topic |
| Author |
Message |
LIsLinuxIsSogood
Veteran
Joined: 13 Feb 2016
Posts: 1179
|
 Posted: Mon Aug 27, 2018 5:16 am Post subject: Copying large backup files via rsync, cp, and scp |
 |
|
I have a situation that I require some help with about the type of tools to use when transferring large files like backups. My attempts with rsync to copy the large file from a USB drive that is formatted in NTFS has been problematic. The file is backup of my virtual gentoo install in the .fsa format of fsarchiver, and is just over the size limit but somehow managed to find its way onto that drive (confused how, since it is a 5.6 GB backup and resides on a ntfs-3g,).
Here's what I get if I try to use rsync:
| Code: | playboy /mnt/restore/Backups/baremetal/Fsarchiver # rsync --progress /mnt/bafckups/Backups/Gentoo_Virtual.fsa ./COPY_OF_Gentoo_Virtual.fsa
Gentoo_Virtual.fsa
5,908,034,623 100% 28.02MB/s 0:03:21 (xfr#1, to-chk=0/1)
rsync: read errors mapping "/mnt/backups/Backups/Gentoo_Virtual.fsa": Value too large for defined data type (75)
WARNING: Gentoo_Virtual.fsa failed verification -- update discarded (will try again).
Gentoo_Virtual.fsa
5,908,034,623 100% 28.37MB/s 0:03:18 (xfr#2, to-chk=0/1)
rsync: read errors mapping "/mnt/backups/Backups/Gentoo_Virtual.fsa": Value too large for defined data type (75)
ERROR: Gentoo_Virtual.fsa failed verification -- update discarded.
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1189) [sender=3.1.3]
|
The odd thing is that while there are three different copies, of this file and all of them seem to be incomplete which I assume is the case. Is there some way to test or compare these to the file on the USB flash drive? So far using cmp I get the following output as an error code...
| Code: | cmp: /mnt/backups/Backups/Gentoo_Virtual.fsa: Value too large for defined data type
|
Or else I just get the expected problem of the files differing, here
| Code: | Gentoo_Virtual.fsa /mnt/backups/Backups/Gentoo_Virtual.fsa differ: byte 1830092801, line 6709504
|
What could account for the fact that cmp sometimes works only and not other times? The times it works seem totally random. Is this a known issue with ntfs always? If so I will really limit the use of ntfs for just windows files from now on, but the problem with that is I get nervous when mounting a linux format on windows since I've had experience with the native disk scan in Win10 or 7 wiping/corrupting my disk with no warning. Are there tools out there that could be better for interfacing between files on the linux systems with windows filesystems on a usb or external drive? |
|
| Back to top |
|
 |
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54214
Location: 56N 3W
|
| Posted: Mon Aug 27, 2018 7:46 am Post subject: |
|
|
LIsLinuxIsSogood,
Compare the md5sums
| Code: | | md5sum /path/to/file |
Then there is the sha*sum family if you are nervous about md5.
NTFS can store files much bigger than 5.9GB
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
LIsLinuxIsSogood
Veteran
Joined: 13 Feb 2016
Posts: 1179
|
| Posted: Mon Aug 27, 2018 8:01 am Post subject: |
|
|
Here's the error I keep seeing, is the data type here referring to the specific file extension or something else?
| Code: | playboy /mnt/restore/Backups/baremetal/Fsarchiver # mount /dev/sde2 /mnt/backups/
playboy /mnt/restore/Backups/baremetal/Fsarchiver # md5sum /mnt/backups/Backups/Gentoo_Virtual.fsa
md5sum: /mnt/backups/Backups/Gentoo_Virtual.fsa: Value too large for defined data type
|
|
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54214
Location: 56N 3W
|
| Posted: Mon Aug 27, 2018 8:18 am Post subject: |
|
|
LIsLinuxIsSogood,
Do you have a mix of 32 bit and 64 bit installs?
While the file size limits are not a problem on any recent filesystems, beware ISO and vfat, the are extra kernel options for big files on 32 bit installs.
This means that things that just work on 64 bit, may not work when accessed from a 32 bit install.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
LIsLinuxIsSogood
Veteran
Joined: 13 Feb 2016
Posts: 1179
|
| Posted: Mon Aug 27, 2018 8:37 am Post subject: |
|
|
Hey Neddy,
Not sure what is meant by a mix. Does that refer to the typical multilib setup, or is it something else. I do have several chroots and therefore different libraries in different locations, but i dont think that matters here since this is not inside any of those. Are you asking if i have installed some things outside of the portage tree, or is this something that could be tested for missing or inconsistent libraries? Which kernel option would correspond to md5 checking capabilities?
There is another possibility i am considering that the file maybbe corrupt that i am trying to copy. |
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Mon Aug 27, 2018 6:23 pm Post subject: |
|
|
LIsLinuxIsSogood ...
see here. The problem seems to be the ntfs-3g doesn't have LFS (Large File Support).
best ... khay |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54214
Location: 56N 3W
|
| Posted: Mon Aug 27, 2018 6:31 pm Post subject: |
|
|
LIsLinuxIsSogood,
Building on what khayyam said, you could try the kernel ntfs driver. Its read only which is all you need.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
|
Other Things Gentoo
