Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Copying large backup files via rsync, cp, and scp
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
LIsLinuxIsSogood
l33t
l33t


Joined: 13 Feb 2016
Posts: 889

PostPosted: Mon Aug 27, 2018 5:16 am    Post subject: Copying large backup files via rsync, cp, and scp Reply with quote

I have a situation that I require some help with about the type of tools to use when transferring large files like backups. My attempts with rsync to copy the large file from a USB drive that is formatted in NTFS has been problematic. The file is backup of my virtual gentoo install in the .fsa format of fsarchiver, and is just over the size limit but somehow managed to find its way onto that drive (confused how, since it is a 5.6 GB backup and resides on a ntfs-3g,).

Here's what I get if I try to use rsync:
Code:
playboy /mnt/restore/Backups/baremetal/Fsarchiver # rsync --progress /mnt/bafckups/Backups/Gentoo_Virtual.fsa ./COPY_OF_Gentoo_Virtual.fsa
Gentoo_Virtual.fsa
  5,908,034,623 100%   28.02MB/s    0:03:21 (xfr#1, to-chk=0/1)
rsync: read errors mapping "/mnt/backups/Backups/Gentoo_Virtual.fsa": Value too large for defined data type (75)
WARNING: Gentoo_Virtual.fsa failed verification -- update discarded (will try again).
Gentoo_Virtual.fsa
  5,908,034,623 100%   28.37MB/s    0:03:18 (xfr#2, to-chk=0/1)
rsync: read errors mapping "/mnt/backups/Backups/Gentoo_Virtual.fsa": Value too large for defined data type (75)
ERROR: Gentoo_Virtual.fsa failed verification -- update discarded.
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1189) [sender=3.1.3]


The odd thing is that while there are three different copies, of this file and all of them seem to be incomplete which I assume is the case. Is there some way to test or compare these to the file on the USB flash drive? So far using cmp I get the following output as an error code...

Code:
cmp: /mnt/backups/Backups/Gentoo_Virtual.fsa: Value too large for defined data type


Or else I just get the expected problem of the files differing, here

Code:
Gentoo_Virtual.fsa /mnt/backups/Backups/Gentoo_Virtual.fsa differ: byte 1830092801, line 6709504


What could account for the fact that cmp sometimes works only and not other times? The times it works seem totally random. Is this a known issue with ntfs always? If so I will really limit the use of ntfs for just windows files from now on, but the problem with that is I get nervous when mounting a linux format on windows since I've had experience with the native disk scan in Win10 or 7 wiping/corrupting my disk with no warning. Are there tools out there that could be better for interfacing between files on the linux systems with windows filesystems on a usb or external drive?
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 41413
Location: 56N 3W

PostPosted: Mon Aug 27, 2018 7:46 am    Post subject: Reply with quote

LIsLinuxIsSogood,

Compare the md5sums
Code:
md5sum /path/to/file


Then there is the sha*sum family if you are nervous about md5.

NTFS can store files much bigger than 5.9GB
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
LIsLinuxIsSogood
l33t
l33t


Joined: 13 Feb 2016
Posts: 889

PostPosted: Mon Aug 27, 2018 8:01 am    Post subject: Reply with quote

Here's the error I keep seeing, is the data type here referring to the specific file extension or something else?

Code:
playboy /mnt/restore/Backups/baremetal/Fsarchiver # mount /dev/sde2 /mnt/backups/
playboy /mnt/restore/Backups/baremetal/Fsarchiver # md5sum /mnt/backups/Backups/Gentoo_Virtual.fsa
md5sum: /mnt/backups/Backups/Gentoo_Virtual.fsa: Value too large for defined data type
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 41413
Location: 56N 3W

PostPosted: Mon Aug 27, 2018 8:18 am    Post subject: Reply with quote

LIsLinuxIsSogood,

Do you have a mix of 32 bit and 64 bit installs?
While the file size limits are not a problem on any recent filesystems, beware ISO and vfat, the are extra kernel options for big files on 32 bit installs.

This means that things that just work on 64 bit, may not work when accessed from a 32 bit install.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
LIsLinuxIsSogood
l33t
l33t


Joined: 13 Feb 2016
Posts: 889

PostPosted: Mon Aug 27, 2018 8:37 am    Post subject: Reply with quote

Hey Neddy,
Not sure what is meant by a mix. Does that refer to the typical multilib setup, or is it something else. I do have several chroots and therefore different libraries in different locations, but i dont think that matters here since this is not inside any of those. Are you asking if i have installed some things outside of the portage tree, or is this something that could be tested for missing or inconsistent libraries? Which kernel option would correspond to md5 checking capabilities?

There is another possibility i am considering that the file maybbe corrupt that i am trying to copy.
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6216
Location: Room 101

PostPosted: Mon Aug 27, 2018 6:23 pm    Post subject: Reply with quote

LIsLinuxIsSogood ...

see here. The problem seems to be the ntfs-3g doesn't have LFS (Large File Support).

best ... khay
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 41413
Location: 56N 3W

PostPosted: Mon Aug 27, 2018 6:31 pm    Post subject: Reply with quote

LIsLinuxIsSogood,

Building on what khayyam said, you could try the kernel ntfs driver. Its read only which is all you need.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum