Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
rsync 3.1.0 **very slow** during backup (even new files)
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 5737
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Sun Oct 20, 2013 6:47 pm    Post subject: rsync 3.1.0 **very slow** during backup (even new files) Reply with quote

Hi,

after the upgrade to rsync 3.1.0 (which supposedly integrates several data-integrity related improvements from previous versions)

backup jobs are really slow:

e.g. when backing up my recoll (desktop search) directory

the xapiandb subdirectory takes several minutes or up to half an hour and more to copy (even when deleting the already existing old folder with contents on the receiver backup drive)

Quote:
xapiandb/flintlock
0 100% 0.00kB/s 0:00:00 (xfr#12, to-chk=16/31)
xapiandb/iamchert
28 100% 0.06kB/s 0:00:00 (xfr#13, to-chk=15/31)
xapiandb/position.DB
10,171,236,352 100% 53.26MB/s 0:03:02 (xfr#14, to-chk=14/31)
xapiandb/position.baseA
155,223 100% 520.91kB/s 0:00:00 (xfr#15, to-chk=13/31)
xapiandb/position.baseB
155,223 100% 462.15kB/s 0:00:00 (xfr#16, to-chk=12/31)
xapiandb/postlist.DB
876,380,160 27% 4.13MB/s 879,001,600 27% 3.18MB/s


postlist.DB is around 9.5 GiB large so this is pretty annoying :?

can this be sped up or is it Working as intended™ ?


Similar experience on your part ?


Many thanks in advance :)


edit:


/home is on btrfs filesystem

btrfs to btrfs

and

btrfs to ext4

shows this behavior

haven't tested yet with other filesystem types ...
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.3.0-r2
2.6.37.2_plus_v1: BFS, CFS,THP,compaction, zcache or TOI
Hardcore Linux user since 2004 :D
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4283

PostPosted: Sun Oct 20, 2013 7:31 pm    Post subject: Reply with quote

What are your options? For instance, are you using --inplace? (you should!)
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 5737
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Sun Oct 20, 2013 7:42 pm    Post subject: Reply with quote

mv wrote:
What are your options? For instance, are you using --inplace? (you should!)


aha !

wasn't aware I had forgotten to add those options

but:

Quote:
WARNING: you should not use this option to update files that are
being accessed by others, so be careful when choosing to use
this for a copy.

This option is useful for transferring large files with
block-based changes or appended data, and also on systems that
are disk bound, not network bound. It can also help keep a
copy-on-write filesystem snapshot from diverging the entire con-
tents of a file that only has minor changes.


thing is: oftentimes when backing up I'm using firefox or chromium so those directories / files might be inconsistent or incomplete

so I either have to live with slow speed or inconsistent files, and thus avoid working with the computer while backing up - hm ...


I'll give that option a try and see how it changes behavior


thanks mv !


more info:

http://www.manpagez.com/man/1/rsync/
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.3.0-r2
2.6.37.2_plus_v1: BFS, CFS,THP,compaction, zcache or TOI
Hardcore Linux user since 2004 :D
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4283

PostPosted: Mon Oct 21, 2013 7:12 am    Post subject: Reply with quote

kernelOfTruth wrote:
oftentimes when backing up I'm using firefox or chromium so those directories / files might be inconsistent or incomplete

Not using --inplace will not save you from this problem either. If you want to avoid such problems you have to use the snapshot function of the filesystem (if available), but even then the consistency depends on the running application - this is why good databases take a considerable effort in performing operations "atomically".
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum