| View previous topic :: View next topic |
| Author |
Message |
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
 Posted: Tue Jun 01, 2004 11:10 am Post subject: HOWTO:Local Rsync Mirror |
 |
|
If you have a couple of gentoo boxes on a LAN be a good gentoo netizen and setup a local rsync mirror. You can sync the server box across the net and then sync all your other machines faster across the LAN. This not only allows gentoo to grow faster by not abusing mirrors, but your LAN is way faster than your internet connection!!
I recommend you also set up a local package cache for downloaded package files, ebuild and HOWTO is here  https://forums.gentoo.org/viewtopic.php?t=173226
Let me start by saying there is conflicting info in the forums and gentoo docs about how to setup a local rsync mirror. The howto at http://www.gentoo.org/doc/en/rsync.xml is for OFFICIAL public mirrors! It can confuse a user wanting a LOCAL LAN mirror! There is an app-admin/gentoo-rsync-mirror ebuild that is currently outdated and possibly insecure!! If you think that ebuild has been updated and the problems fixed you should check the difference between it and this howto to see if it has been updated to include all fixes and features!
You really only need a config file to get up and running, but this version adds features and security.
Every gentoo box has rsync installed because it's part of the base system. Lets add the parts that make it a server.
1. /etc/rsyncd.conf
| Code: |
uid = nobody
gid = nobody
use chroot = yes
#limit access to private LAN's
hosts allow=192.168.0.0/255.255.0.0 10.0.0.0/255.0.0.0
hosts deny=*
max connections = 15
pid file = /var/run/rsyncd.pid
motd file = /etc/rsync/rsyncd.motd
#This will give you a separate log file
#log file = /var/log/rsync.log
#This will log every file transferred - up to 85,000+ per user, per sync
#transfer logging = yes
log format = %t %a %m %f %b
syslog facility = local3
timeout = 300
#If you need this, UPGRADE portage please!
#[gentoo-x86-portage]
#this entry is for compatibility
#path = /usr/portage
#comment = Gentoo Linux Portage tree
#exclude=distfiles/
[gentoo-portage]
#modern versions of portage use this entry
path = /usr/portage
comment = Gentoo Linux Portage tree mirror
exclude = distfiles/
|
2. /etc/init.d/rsyncd
| Code: |
#!/sbin/runscript
# Copyright 1999-2002 Gentoo Technologies, Inc.
# Distributed under the terms of the GNU General Public License, v2
or later
# $Header: /var/www/www.gentoo.org/raw_cvs/gentoo/xml/htdocs/doc/en/rsync.xml,v 1.26 2004/05/03 12:30:06 neysx Exp $
depend() {
need net
}
# FYI: --sparce seems to cause problems.
RSYNCOPTS="--daemon --timeout=300"
start() {
ebegin "Starting rsync daemon"
start-stop-daemon --start --quiet --pidfile /var/run/rsyncd.pid --nicelevel 15 --exec /usr/bin/rsync -- ${RSYNCOPTS}
eend $?
}
stop() {
ebegin "Stopping rsync daemon"
start-stop-daemon --stop --quiet --pidfile /var/run/rsyncd.pid
eend $?
}
|
3. /etc/rsync/rsyncd.motd
| Code: |
Welcome to My Local Rsync Mirror!!
|
Then start rsyncd
| Code: |
# /etc/init.d/rsyncd start
|
rsyncd should be started at boot - add to default runlevel
| Code: |
# rc-update add rsyncd default
|
Now you have a local server up and running. You must point your client boxes to this new server.
Edit /etc/make.conf on the client boxes and change the SYNC option to your new server.
The SYNC section should look like this :
| Code: |
#SYNC="rsync://rsync.gentoo.org/gentoo-portage"
SYNC="rsync://YourMirrorHere.com/gentoo-portage"
|
Replace YourMirrorHere.com with your server's Name or IP address.
Remember to open port 873 on the rsync server box or any firewall between your clients and the rsync server box.
Then kick back and admire your work, knowing your client boxes can sync faster over the LAN and your saving Gentoo/yourself alot of bandwidth!!
NOTE 1: My /etc/init.d/rsyncd above has --nicelevel set to 15. This purposely slows down the rsync server so you won't notice other boxes syncing - This is what you want if you spend time logged in on the server box. If you only use your rsync server as a server, set the nicelevel to 0 to make the rsync server run at normal speed (change the line with --nicelevel 15 to --nicelevel 0 in the /etc/init.d/rsyncd above. You could also set the nicelevel to max 19 to slow it down even more ).
NOTE 2: The default script installed with rsync is similiar but omits nicelevel, and timeout. You could use the default script and and add "--timeout=300" to /etc/conf.d/rsyncd , but that would still leave out setting the nicelevel which can only be done by editing the file. By replacing the script with my verson, I've kept all the changes in one file, which I find easier when merging future updates......
UPDATE* Latest versions of rsync support bandwidth limiting. Large shared installations might consider this option.
--bwlimit=KBPS limit I/O bandwidth; KBytes per second
If you need to setup this local rsync server on a non-gentoo box, this will sync your data with the official gentoo rsync servers:
| Code: |
#!/bin/bash
RSYNC="/usr/bin/rsync"
OPTS="--quiet --recursive --links --perms --times --devices --delete --timeout=300"
#Uncomment the following line only if you have been granted access to rsync1.us.gentoo.org
#SRC="rsync://rsync1.us.gentoo.org/gentoo-portage"
#If you are waiting for access to our master mirror, select one of our mirrors to mirror from:
SRC="rsync://rsync2.de.gentoo.org/gentoo-portage"
DST="/space/gentoo/rsync/"
echo "Started update at" `date` >> $0.log 2>&1
logger -t rsync "re-rsyncing the gentoo-portage tree"
${RSYNC} ${OPTS} ${SRC} ${DST} >> $0.log 2>&1
echo "End: "`date` >> $0.log 2>&1
|
Let me say again, most of this info is taken from the Official Howto BUT modified for LOCAL use. The gentoo-rsync-mirror ebuild is designed for OFFICIAL PUBLIC mirrors only.
Version 1.8
--safe-links no longer needed or supported in daemon mode
added note about bwlimiting
Version 1.7
Added script note
Version 1.6
Updated rsync.conf location
added non-gentoo
Version 1.5
Added firewall port
Version 1.4
Cleaned up [gentoo-x86-portage]
section in case someone actually needs it
Version 1.3
Added trailing / to 'exclude distifles'
Version 1.2
Added nicelevel note
Version 1.1
added hosts allow/deny
Version 1.0
Initial Post
Last edited by flybynite on Sat Aug 19, 2006 8:24 pm; edited 10 times in total |
|
| Back to top |
|
 |
barran
Tux's lil' helper
Joined: 14 Jan 2003
Posts: 142
Location: Århus, DK
|
| Posted: Tue Jun 01, 2004 12:52 pm Post subject: |
|
|
Why did you make your own init-script? There is one included in the rsync ebuild and it is working smoothly for me also having a local mirror I made myself.
You can adjust settings in /etc/conf.d/rsyncd.
I'm just curious. |
|
| Back to top |
|
|
dhurt
Apprentice
Joined: 14 May 2003
Posts: 278
Location: Davis, CA
|
| Posted: Tue Jun 01, 2004 4:26 pm Post subject: |
|
|
The init script he used was from the original Gentoo howto.
Is the main difference between your method and the method posted here:
https://forums.gentoo.org/viewtopic.php?t=59134
Is that you use the Gentoo Documentations init script and you use host blocking? Why was the previous setup insecure? Thanks for the update.
_________________
"And isn't sanity really just a one-trick pony, anyway? I mean, all you get is one trick, rational thinking, but when you're good and crazy, ooh ooh ooh, the sky's the limit!" -- The Tick |
|
| Back to top |
|
|
senectus
Guru
Joined: 17 Jul 2003
Posts: 534
|
| Posted: Tue Jun 01, 2004 9:19 pm Post subject: |
|
|
woohoo it works fine thanks...
Now to get the package cache working 
_________________
2800+XP A7N8X FX6600GT
www.modmeup.net |
Belief is 9/10 of YOUR reality.
Wise man say: A skilled troll is a master baiter. |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Wed Jun 02, 2004 12:08 am Post subject: |
|
|
| barran wrote: | Why did you make your own init-script? There is one included in the rsync ebuild and it is working smoothly for me also having a local mirror I made myself.
You can adjust settings in /etc/conf.d/rsyncd.
I'm just curious. |
I guess there are two or three reasons for the init script change.
1. Mine uses the start-stop-daemon, which is the official gentoo standard. I believe the default rsync init script will be changed to use the start-stop-daemon whenever a developer can spare some time to fix it.
2. My script has some nice features I wanted to make this a sweet install for users, like setting the --nicelevel which the default script can't do.
3. I copied it exactly as the official howto suggests!! |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Wed Jun 02, 2004 12:18 am Post subject: |
|
|
| ender2431 wrote: |
Is the main difference between your method and the method posted here:
https://forums.gentoo.org/viewtopic.php?t=59134
Is that you use the Gentoo Documentations init script and you use host blocking? Why was the previous setup insecure? Thanks for the update. |
Yes, my init script is different, read the above post for why mine is better.
My config is also more secure!!
An official gentoo rsync server was compromised a while ago. Part of the hack was kernel related, part was rsync's fault.
The main problem with the older howto you mention is it contains:
#uid = nobody
#gid = nobody
use chroot = no
which is the part that makes it insecure due to a known rsync exploit. A fix is in the works but I don't think it is in portage yet.
Look at the latest rsync ebuilds and you will see this warning:
| Quote: |
ewarn "Please make sure you do NOT disable the rsync server running"
ewarn "in a chroot. Please check /etc/rsync/rsyncd.conf and make sure"
ewarn "it says: use chroot = yes"
|
This warning is also in the Official Howto:
| Quote: |
For security reasons, the use of a chrooted environment is required!
|
I added the Hosts Allow/Deny because it is appropriate and another level of security for a LAN only server. |
|
| Back to top |
|
|
cogitate
n00b
Joined: 05 Dec 2003
Posts: 46
Location: Ontario, Canada
|
| Posted: Wed Jun 02, 2004 6:22 pm Post subject: |
|
|
the deny should come before the allow
| Code: | # Deny all
hosts deny *
# Allow from internal
hosts allow 192.168.0.0/255.255.255.0 10.0.0.0/255.0.0.0
|
I'm not sure if it really matters, but I got a deny error first time I tried it. |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Thu Jun 03, 2004 6:46 pm Post subject: |
|
|
| cogitate wrote: | the deny should come before the allow
| Code: | # Deny all
hosts deny *
# Allow from internal
hosts allow 192.168.0.0/255.255.255.0 10.0.0.0/255.0.0.0
|
I'm not sure if it really matters, but I got a deny error first time I tried it. |
I believe your are incorrect because your config syntax is wrong. You forgot both the '=' as in 'hosts deny=', 'hosts allow=', and have one too many 255's in your netmask!!!
Try to copy the config again....... |
|
| Back to top |
|
|
dhurt
Apprentice
Joined: 14 May 2003
Posts: 278
Location: Davis, CA
|
| Posted: Thu Jun 03, 2004 9:22 pm Post subject: |
|
|
The netmask he is using is just more restrictive. It allows only IPs on the 192.168.0.X Class C Subnet. The 255.255.0.0 is a Class B subnet. do you really have more than 254 computers/ip's in use on your network?
_________________
"And isn't sanity really just a one-trick pony, anyway? I mean, all you get is one trick, rational thinking, but when you're good and crazy, ooh ooh ooh, the sky's the limit!" -- The Tick |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Fri Jun 04, 2004 1:36 am Post subject: |
|
|
I'm afraid you missed the point that because of the syntax error, it doesn't matter what the netmask is because his options are being ignored. He is left with the defaults which are to allow anybody and that's why it's working for him now .....
Since he didn't change the 10.x.x.x netmask and yes, neither he, nor I, have 2^24 (16,777,216) computers - I guessed he just copied them wrong and didn't change the netmask on purpose |
|
| Back to top |
|
|
dhurt
Apprentice
Joined: 14 May 2003
Posts: 278
Location: Davis, CA
|
| Posted: Fri Jun 04, 2004 4:02 am Post subject: |
|
|
| flybynite wrote: | | [quote="and have one too many 255's in your netmask!!! |
Yeah, I realize his syntax error, I was just referencing this comment. The only reason I mentioned something is that is how I have mine set (minus the "=" typo) and it works. Thanks for your work in this project here flybynite and with your portage web cache system. I am amazed at how beautiful it works here on my lan.  Cheers!!!
This is just awesome (8.50 M/s):
| Code: |
# emerge -f openoffice
>>> emerge (1 of 1) app-office/openoffice-1.1.1-r1 to /
>>> Downloading http://gentoo.mirrors.pair.com/distfiles/OOo_1.1.1p1_source.tar.bz2
--20:59:36-- http://gentoo.mirrors.pair.com/distfiles/OOo_1.1.1p1_source.tar.bz2
=> `/usr/portage/distfiles/OOo_1.1.1p1_source.tar.bz2'
Resolving xenophobia... 192.168.0.20
Connecting to xenophobia[192.168.0.20]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 192,388,698
100%[====================================>] 192,388,698 8.50M/s ETA 00:00
20:59:59 (7.85 MB/s) - `/usr/portage/distfiles/OOo_1.1.1p1_source.tar.bz2' saved [192388698/192388698]
|
_________________
"And isn't sanity really just a one-trick pony, anyway? I mean, all you get is one trick, rational thinking, but when you're good and crazy, ooh ooh ooh, the sky's the limit!" -- The Tick |
|
| Back to top |
|
|
KpR2000
n00b
Joined: 18 Aug 2003
Posts: 55
|
| Posted: Sun Jun 06, 2004 11:13 am Post subject: |
|
|
Hi,
I have configured my server (700Mhz) with the above hints.
emerge sync works fine at my client computer. But I did not get full speed over a full duplex 100 Mb connection:
wrote 226 bytes read 437 bytes 442.00 bytes/sec
Can I do speed improvements?
Greetings |
|
| Back to top |
|
|
Suicidal
l33t
Joined: 30 Jul 2003
Posts: 958
Location: /dev/null
|
| Posted: Sun Jun 06, 2004 2:33 pm Post subject: |
|
|
| Quote: | | The netmask he is using is just more restrictive. It allows only IPs on the 192.168.0.X Class C Subnet. The 255.255.0.0 is a Class B subnet. do you really have more than 254 computers/ip's in use on your network? |
The netmask for my entire network is 255.252.0.0 or /14 but it is comprised of over 40 subnets spanning 3 class b subnet and alomst 3000 hosts. It really doesnt atter as long as his hosts are within the masks range.
Personally on my home subnet i dont use 192.168.*.* because that is what a hacker would expect to find. I use a range within the 10.*.*.* range in case they are able to get past my firewall and nat. It would make it much more difficult to find a host within that range compared to a simple class c range. |
|
| Back to top |
|
|
Satori80
Tux's lil' helper
Joined: 24 Feb 2004
Posts: 137
|
| Posted: Sun Jun 06, 2004 10:30 pm Post subject: |
|
|
| KpR2000 wrote: | Hi,
I have configured my server (700Mhz) with the above hints.
emerge sync works fine at my client computer. But I did not get full speed over a full duplex 100 Mb connection:
wrote 226 bytes read 437 bytes 442.00 bytes/sec
Can I do speed improvements?
Greetings |
First, that's too small an amount of data to accurately judge the speed of the sync.
Secondly, unless there is something noticeably wrong with your LAN configuration, your hard drive will be the bottleneck here, not the network.
If you like, you can look into caching the distfiles on the server if you have an abundance of RAM. Do a search on proxy and cache, as there have been several posts about it in this section.
The only other thing you could do is upgrade to a faster drive or even use a RAID to improve speed, but I doubt on a one user system it'd be worth the effort and money unless you already notice that the disk(s) is/are slow in day to day use. |
|
| Back to top |
|
|
Satori80
Tux's lil' helper
Joined: 24 Feb 2004
Posts: 137
|
| Posted: Sun Jun 06, 2004 10:53 pm Post subject: |
|
|
| I want to be sure I'm clear on this. Using this configuration one would use emerge sync to update the mirror? |
|
| Back to top |
|
|
dhurt
Apprentice
Joined: 14 May 2003
Posts: 278
Location: Davis, CA
|
| Posted: Sun Jun 06, 2004 11:22 pm Post subject: |
|
|
Yes, and then use the same command to replicate to properly configured clients
_________________
"And isn't sanity really just a one-trick pony, anyway? I mean, all you get is one trick, rational thinking, but when you're good and crazy, ooh ooh ooh, the sky's the limit!" -- The Tick |
|
| Back to top |
|
|
Satori80
Tux's lil' helper
Joined: 24 Feb 2004
Posts: 137
|
| Posted: Mon Jun 07, 2004 12:21 am Post subject: |
|
|
Okay, I've set up everything as per the instructions here (afaik), emerge sync on the machine running rsyncd. All goes well there. Changed my main desktop to sync off my LAN mirror and i get this:
| Code: | >>> Starting retry 1 of 3 with rsync://<snip ip-address>/gentoo-portage
>>> checking server timestamp ...
<snip my motd>
receiving file list ...
link_stat "metadata/timestamp.chk" (in gentoo-portage) failed: No such file or directory
0 files to consider
client: nothing to do: perhaps you need to specify some filenames or the --recursive option?
rsync error: some files could not be transferred (code 23) at main.c(653)
>>> retry ...
|
I've gone over the configs several times, yet obviously I'm missing something. Any ideas?
EDIT: Maybe I wasn't clear enough in my 1st question... one would use emerge sync to get the portage tree for rsyncd? Not an rsync command? |
|
| Back to top |
|
|
dhurt
Apprentice
Joined: 14 May 2003
Posts: 278
Location: Davis, CA
|
| Posted: Mon Jun 07, 2004 12:28 am Post subject: |
|
|
Sounds like you are not sharing the portage directory correctly on the server, could you post your /etc/rsync/rsyncd.conf?
Responce to your edit question, yes you run the "emerge sync" command on all your computers. Just run your server computer first and then emerge sync on the clients.
_________________
"And isn't sanity really just a one-trick pony, anyway? I mean, all you get is one trick, rational thinking, but when you're good and crazy, ooh ooh ooh, the sky's the limit!" -- The Tick |
|
| Back to top |
|
|
Satori80
Tux's lil' helper
Joined: 24 Feb 2004
Posts: 137
|
| Posted: Mon Jun 07, 2004 12:35 am Post subject: |
|
|
| Code: | # Copyright 1999-2004 Gentoo Technologies, Inc.
# Distributed under the terms of the GNU General Public License v2
# $Header: /home/cvsroot/gentoo-x86/app-admin/gentoo-rsync-mirror/files/rsyncd.conf,v 1.4 2004/05/02 22:58:16 mholzer Exp $
uid = nobody
gid = nobody
use chroot = no
max connections = 20
pid file = /var/run/rsyncd.pid
motd file = /etc/rsync/rsyncd.motd
transfer logging = yes
log format = %t %a %m %f %b
syslog facility = local3
timeout = 300
#[gentoo-x86-portage]
#this entry is for compatibility
#path = /opt/gentoo-rsync/portage
#comment = Gentoo Linux Portage tree
[gentoo-portage]
#modern versions of portage use this entry
path = /etc/portage
comment = Gentoo Linux Portage tree mirror
exclude = distfiles
|
|
|
| Back to top |
|
|
Satori80
Tux's lil' helper
Joined: 24 Feb 2004
Posts: 137
|
| Posted: Mon Jun 07, 2004 12:56 am Post subject: |
|
|
From /var/log/messages:
| Code: |
Jun 6 19:33:59 ns1 rsyncd[3041]: rsync: name lookup failed for ip-address deleted: Name or service not known
Jun 6 19:33:59 ns1 rsyncd[3041]: rsync on gentoo-portage/metadata/timestamp.chk from UNKNOWN (ipaddress deleted)
Jun 6 19:34:00 ns1 CRON[3043]: (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons )
Jun 6 19:34:12 ns1 rsyncd[2587]: rsync error: received SIGUSR1 or SIGINT (code 20) at rsync.c(249)
Jun 6 19:34:17 ns1 rsyncd[3133]: rsyncd version 2.6.0 starting, listening on port 873
Jun 6 19:34:23 ns1 rsyncd[3136]: rsync: name lookup failed for ip-address deleted: Name or service not known
Jun 6 19:34:23 ns1 rsyncd[3136]: rsync on gentoo-portage/metadata/timestamp.chk from UNKNOWN (ip-address deleted) |
It would seem this goes back to a DDNS problem I never got solved. :/
Also, when I log into the server it says it's name is "ns1.(none)" wich is another issue i've never been able to sort out.
Thanks a lot for your help. I'll have to get on this other stuff 1st apparently. |
|
| Back to top |
|
|
dhurt
Apprentice
Joined: 14 May 2003
Posts: 278
Location: Davis, CA
|
| Posted: Mon Jun 07, 2004 1:50 am Post subject: |
|
|
Your config is wrong as well, or should be:
This should not be correct in your rsync config:
| Code: |
path = /etc/portage
|
but:
| Code: |
path = /usr/portage
|
Or whatever you have listed in your /etc/make.conf under:
| Code: |
PORTDIR=/usr/portage
|
_________________
"And isn't sanity really just a one-trick pony, anyway? I mean, all you get is one trick, rational thinking, but when you're good and crazy, ooh ooh ooh, the sky's the limit!" -- The Tick
Last edited by dhurt on Mon Jun 07, 2004 5:10 am; edited 1 time in total |
|
| Back to top |
|
|
Satori80
Tux's lil' helper
Joined: 24 Feb 2004
Posts: 137
|
| Posted: Mon Jun 07, 2004 4:58 am Post subject: |
|
|
lol! Damn. Figures it was something simple and stupid like that.
Thanks again. |
|
| Back to top |
|
|
KpR2000
n00b
Joined: 18 Aug 2003
Posts: 55
|
| Posted: Mon Jun 07, 2004 7:43 am Post subject: |
|
|
| Quote: |
First, that's too small an amount of data to accurately judge the speed of the sync.
|
Another test shows still such a bad result.
| Quote: |
Secondly, unless there is something noticeably wrong with your LAN configuration, your hard drive will be the bottleneck here, not the network.
|
ok, hdparm -tT /dev/hdd3 sais:
/dev/hdd3:
Timing buffer-cache reads: 240 MB in 2.00 seconds = 119.72 MB/sec
Timing buffered disk reads: 50 MB in 3.11 seconds = 16.10 MB/sec
It's a 2" HD... But should it not enough for the rsync action?
| Quote: |
If you like, you can look into caching the distfiles on the server if you have an abundance of RAM. Do a search on proxy and cache, as there have been several posts about it in this section.
|
I think it is not a good idea to cache, because my ram is limited to 256Mb
About the ftp protocol I get full speed out of the machine. What is wrong here?
Thx in advance[/quote] |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Thu Jun 10, 2004 1:16 am Post subject: |
|
|
KpR2000:
The easiest fix for the name lookup failures in your logs is to list the ip's and hostsnames in /etc/hosts.
I noticed that you seem to be comparing your rsync server speed with someones distfile cache speed in this thread. Two different things.
Now that you have your config file fixed, what speeds are you getting? |
|
| Back to top |
|
|
CarpJA
n00b
Joined: 29 Apr 2004
Posts: 2
|
| Posted: Thu Jun 10, 2004 5:00 am Post subject: |
|
|
| I too am having speed issues... Downloading the file list from my local mirror takes several minutes as opposed to less than a minute for an internet mirror. Has anyone ecountered this and solved it? |
|
| Back to top |
|
|
|
Documentation, Tips & Tricks
