View previous topic :: View next topic |
Author |
Message |
fourhead l33t
Joined: 03 Sep 2003 Posts: 875 Location: Cologne, Germany
|
Posted: Fri Jan 21, 2005 9:33 pm Post subject: |
|
|
Great how-to I'm setting this up ony my virtual Debian server right now. I copied my local /usr/portage to /var/portage on the Debian server and edited the config files accordingly. My only question now is - how do I rsync the portage tree that is in /var/portage on the Debian box? There's no "emerge sync" of course. I'm doing this service mainly for a few friends and some people on my university, so I was thinking I could sync the tree on the Debian box perhaps two or four times a day, and me & my friends can sync to this server. But again, how do I sync the Debian box with an "real" official portage mirror?
Tom |
|
Back to top |
|
|
Gentree Watchman
Joined: 01 Jul 2003 Posts: 5350 Location: France, Old Europe
|
Posted: Fri Jan 21, 2005 11:03 pm Post subject: |
|
|
man rsync !
I am pretty sure you have rsync on the debain system so just look at /etc/make.conf to see the rsync command used by gentoo and adapt the paths etc. to do the same thing on your Debox.
HTH
PS also do regular tar of both the portage tree and distfiles if you have room. This will be a great help to you when portage rsyncs with broken packages and deletes the working ones from your system's portage. _________________ Linux, because I'd rather own a free OS than steal one that's not worth paying for.
Gentoo because I'm a masochist
AthlonXP-M on A7N8X. Portage ~x86 |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Sat Jan 22, 2005 6:04 am Post subject: |
|
|
This script below should get you started. I took this from the official gentoo howto on rsync at http://www.gentoo.org/doc/en/rsync.xml
The original reason I wrote this howto is because that official howto was confusing and gave bad info for home/casual users. I created this howto and suggested changes to the official howto. Last I looked, most of the problems with the official howto are fixed. Many users could still be confused so my howto is still relavent.
I would also recommend setting up http-replicator. Read all about it and know ~15,000 of your fellow enthusiasts use it also!
https://forums.gentoo.org/viewtopic.php?t=173226
Code: |
#!/bin/bash
RSYNC="/usr/bin/rsync"
OPTS="--quiet --recursive --links --perms --times --devices --delete --timeout=300"
#Uncomment the following line only if you have been granted access to rsync1.us.gentoo.org
#SRC="rsync://rsync1.us.gentoo.org/gentoo-portage"
#If you are waiting for access to our master mirror, select one of our mirrors to mirror from:
SRC="rsync://rsync2.de.gentoo.org/gentoo-portage"
DST="/space/gentoo/rsync/"
echo "Started update at" `date` >> $0.log 2>&1
logger -t rsync "re-rsyncing the gentoo-portage tree"
${RSYNC} ${OPTS} ${SRC} ${DST} >> $0.log 2>&1
echo "End: "`date` >> $0.log 2>&1
|
|
|
Back to top |
|
|
lysergicacid Guru
Joined: 25 Nov 2003 Posts: 352 Location: The Universe,Virgo Super Cluster,Milky Way,Earth
|
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Tue Jan 25, 2005 4:43 am Post subject: Re: hey guys :) |
|
|
lysergicacid wrote: | anyone know how id setup the box i just setup to be my rsync mirror to also be my distfiles server plz ? |
The absolute best way is to use http-replicator which will do what you want plus has many more benefits:
https://forums.gentoo.org/viewtopic.php?t=173226 |
|
Back to top |
|
|
user808 n00b
Joined: 11 Jan 2005 Posts: 7
|
Posted: Sat Jan 29, 2005 7:35 am Post subject: |
|
|
Nice, thanks for the guide... I was having problems untill I figured out that the rsync server runs on port 873 and had to poke a hole in firewall. |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Sat Jan 29, 2005 10:09 am Post subject: |
|
|
user808 wrote: | Nice, thanks for the guide... I was having problems untill I figured out that the rsync server runs on port 873 and had to poke a hole in firewall. |
Thanks! The point about opening the port is a good one, I've added it to the HOWTO. |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Tue Feb 08, 2005 9:31 am Post subject: |
|
|
Ymerej pointed out to me that gentoo has made a few changes to the default scripts and wondered if the scripts in this howto are out of date.
This howto is still current and will work fine. I will update this howto to use the new "style", but there won't be any operational differences so feel free to use this howto without worry |
|
Back to top |
|
|
SchrodingerPenguin n00b
Joined: 08 Jul 2004 Posts: 35 Location: Mandalong, NSW, Australia
|
Posted: Mon Feb 14, 2005 10:49 am Post subject: |
|
|
Lots of thanks to flybynite for a useful and easy-to-implement howto!
I have just followed the instructions as given at the beginning of this thread, and right at this instant my brother's laptop is syncing from mine. This will certainly save doubling up download time from the internet.
I had everything work fine, and I copied the config files exactly as given.
Thanks again for a great bit of help.[/b] _________________ 2B or !(2B) |
|
Back to top |
|
|
jleejj n00b
Joined: 18 Jan 2004 Posts: 57 Location: Eugene, OR
|
Posted: Sun Feb 20, 2005 6:44 pm Post subject: |
|
|
Why does the rsync daemon run with root priviledges. When I do a:
Code: | ps -ef | grep rsync |
I get:
Code: | root 3134 1 0 11:15 ? 00:00:00 /usr/bin/rsync --daemon --safe-links --timeout=300 |
Shouldn't I see this process owned by "nobody" as per the rsyncd.conf file settings? Also, is there a way to check that a process is really running inside a chroot? I honestly don't know that much daemons so if I am confused feel free to set me straight, but I am concerned about the security of the current setup. |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Tue Feb 22, 2005 7:05 am Post subject: |
|
|
jleejj wrote: | Why does the rsync daemon run with root priviledges. When I do a:
Code: | ps -ef | grep rsync |
I get:
Code: | root 3134 1 0 11:15 ? 00:00:00 /usr/bin/rsync --daemon --safe-links --timeout=300 |
Shouldn't I see this process owned by "nobody" as per the rsyncd.conf file settings? Also, is there a way to check that a process is really running inside a chroot? |
These are questions probably better asked on an rsync mailing list. I'm not an expert on rsync, I just use it
But I was curious and I get this:
Code: |
$ ps aux | grep rsync
root 6674 0.0 0.1 1712 496 ? Ss Feb19 0:00 rsync --daemon
nobody 24262 41.2 1.7 8184 6680 ? R 00:44 0:06 rsync --daemon
tom 24273 0.0 0.1 1432 420 pts/2 S+ 00:45 0:00 grep rsync
$
|
The difference is the daemon starts as root but changes into "nobody" when a user actually connects. Only root can change into another user so that makes sense. In my example, You see the program running as "nobody" because a user is actually connected.....
Verifying the chroot, I don't know how to do.. But here is the man page info:
Quote: |
use chroot
If "use chroot" is true, the rsync server will chroot to the
"path" before starting the file transfer with the client. This
has the advantage of extra protection against possible implemen-
tation security holes, but it has the disadvantages of requiring
super-user privileges, of not being able to follow symbolic
links outside of the new root path when reading, and of implying
the --numeric-ids option because /etc/passwd becomes inaccessi-
ble. When "use chroot" is false, for security reasons symlinks
may only be relative paths pointing to other files within the
root path, and leading slashes are removed from absolute paths.
The default for "use chroot" is true.
|
This man page is for a newer version of rsync. Older versions don't have "use chroot" default to true, that is why it's still in the config.
So thats what I find. Rsync runs as root initially in order to be able to change to "nobody" and run in a "chroot" when a user is actually connected, just like its supposed to... |
|
Back to top |
|
|
transitbus Tux's lil' helper
Joined: 11 Dec 2003 Posts: 80 Location: In the corner
|
Posted: Thu Feb 24, 2005 11:34 pm Post subject: |
|
|
I have to confess I have been bad! I have 10 clients I was updating by going to official mirrors. Thanks for making this tutorial, I have beeen meaning to do this forever but never got around to it. Now I can do my part to relieve the load. |
|
Back to top |
|
|
Jimmy Jazz Guru
Joined: 04 Oct 2004 Posts: 325 Location: Strasbourg
|
Posted: Fri Mar 04, 2005 12:42 pm Post subject: share distfiles on local networks |
|
|
Hello,
i'm trying to use a rsync server in order to distribute distfiles to internal gentoo clients
It works great for gentoo portage tree but not for the distfiles tree .
i added the following lines in the /etc/rsync/rsynd.conf file,
Code: |
[gentoo-portage]
path = /usr/portage
comment = Gentoo Linux Portage tree mirror
exclude = distfiles/
[gentoo-packages]
#For distributing Portage packages (distfiles) to internal clients
path = /usr/portage/distfiles
comment = Gentoo Linux Packages mirror
|
and for every clients, i added in /etc/make.conf,
Code: |
GENTOO_MIRRORS="rsync://MY_LOCAL_SERVER_NAME/gentoo-packages
|
what have i done wrong ?
Jj _________________ « La seule condition au triomphe du mal, c'est l'inaction des gens de bien » E.Burke
Code: |
+----+----+----+
| |::::| |
| |::::| |
+----+----+----+ |
motto: WeLCRO
WritE Less Code, Repeat Often |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Fri Mar 04, 2005 1:58 pm Post subject: Re: share distfiles on local networks |
|
|
Jimmy Jazz wrote: | Hello,
i'm trying to use a rsync server in order to distribute distfiles to internal gentoo clients
what have i done wrong ?
Jj |
Just using the wrong tool for the job
Check out my other post on http-replicator. So many advantages, seriously....
https://forums.gentoo.org/viewtopic-t-173226-highlight-.html |
|
Back to top |
|
|
Jimmy Jazz Guru
Joined: 04 Oct 2004 Posts: 325 Location: Strasbourg
|
Posted: Fri Mar 04, 2005 6:12 pm Post subject: Re: share distfiles on local networks |
|
|
@flybynite
i really thought rsyncd would do the job as well.
Also, i will read your post and install it
Thx for your howtos
Jj _________________ « La seule condition au triomphe du mal, c'est l'inaction des gens de bien » E.Burke
Code: |
+----+----+----+
| |::::| |
| |::::| |
+----+----+----+ |
motto: WeLCRO
WritE Less Code, Repeat Often |
|
Back to top |
|
|
Jimmy Jazz Guru
Joined: 04 Oct 2004 Posts: 325 Location: Strasbourg
|
Posted: Fri Mar 04, 2005 9:34 pm Post subject: xinetd & rsyncd |
|
|
Hello,
by the way i 'm using rsyncd with xinetd to make it even more secure, it works well and it is fast enough
stop rsyncd
Code: |
rc_update del rsyncd
/etc/init.d/rsyncd stop
|
restrict the access to your server
Code: |
#cat /etc/xinetd.conf
# Sample configuration file for xinetd
defaults
{
only_from = localhost YOUR_LOCAL_NETWORK
instances = 60
log_type = SYSLOG authpriv info
log_on_success = HOST PID
log_on_failure = HOST
cps = 25 30
}
includedir /etc/xinetd.d
|
add rsyncd file
Code: |
# cat /etc/xinetd.d/rsyncd
# default: off
# xinetd service description for rsyncd
service rsyncd
{
disable = no
socket_type = stream
wait = no
user = root
server = /usr/bin/rsync
protocol = tcp
port = 873
bind = YOUR_SERVER_IP_INTERFACE
cps = 60 10
server_args = --daemon --safe-links --timeout=300
nice = 5
instances = 8
type = UNLISTED
}
|
xinetd needs to read the new configuration
Code: |
kill -HUP XINETD_PROCESS
|
check if portmap is not restricted to listen localhost only
Code: |
#cat /etc/conf.d/portmap
# Listen on localhost only by default
#PORTMAP_OPTS="-l"
|
Jj _________________ « La seule condition au triomphe du mal, c'est l'inaction des gens de bien » E.Burke
Code: |
+----+----+----+
| |::::| |
| |::::| |
+----+----+----+ |
motto: WeLCRO
WritE Less Code, Repeat Often |
|
Back to top |
|
|
Bob P Advocate
Joined: 20 Oct 2004 Posts: 3355 Location: Jackass! Development Labs
|
Posted: Thu Mar 31, 2005 9:07 pm Post subject: |
|
|
Thanks for posting this helpful guide. In following it, I very easily set-up an RSYNC server on my home LAN a few days ago, and did my part to decrease the load on the gentoo mirrors.
I'd like to point out one discrepancy that I've noticed between the guide and the current location of rsyncd.conf on my computers. It seems that the default location of the rscyncd.conf files on my Gentoo boxes is different from the location suggested by the guide:
The guide states that the configuration files are located at /etc/rsync/rsyncd.conf. On both my 2004.3 and 2005.0 boxes there is no /etc/rsync directory, and the rsyncd.conf file is located at /etc/rsyncd.conf. After taking that discrepancy into account, the process of creating an rsync server works fine with the guide as it is written.
My RSYNC server has been running fine for a few days now. I've decided to rebuild the Gentoo installation on the PC that I was using as the rsync client because of a separate software project I'm working on. The rsync client seemed to have worked for a brief period, but now appears broken and issues the following error message:
Code: |
receiving file list...
115784 files to consider
rsync error: received SIGUSR1 or SIGINT (code 20) at rsync.c(249)
rsync: connection unexpectedly closed (2722197 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(189)
>>> retry...
|
then rsync reiterates for the specified number of attempts, and issues this error:
Code: |
!!! Rsync has not successfully finished. It is recommended that you keep
!!! trying or that you use the 'emerge-webrsync' option if you are unable
!!! to use rsync due to firewall or other restrictions. This should be a
!!! temporary problem unless complications exist with your network
!!! (and possibly your system's filesystem) configuration.
|
what seems most odd about this situation was that i have not made any changes to the rsync server. the only thing that i had changed was to reinstall gentoo on a client box upon which rsync had worked fine before.
it seems that i have the rsync settings properly configured in make.conf on the client PC:
Code: |
SYNC="rsync://<lan-IP-address>/gentoo-portage"
RSYNC_RETRIES="3"
RSYNC_TIMEOUT=180
|
so i am at a loss to explain why the rsync appears to be broken. i had considered that i may have encountered a timeout problem, because the client box is slow. so i've increased the timeout values on both the client and the server to a generous 600 seconds. unfortunately, that hasn't improved the situation.
diskspace does not appear to be a problem.
i'd appreciate it if anyone has a helpful idea. thanks. _________________ .
Stage 1/3 | Jackass! | Rockhopper! | Thanks | Google Sucks |
|
Back to top |
|
|
Jimmy Jazz Guru
Joined: 04 Oct 2004 Posts: 325 Location: Strasbourg
|
Posted: Fri Apr 01, 2005 1:09 pm Post subject: |
|
|
@Bob P
Quote: |
Code: |
receiving file list...
115784 files to consider
rsync error: received SIGUSR1 or SIGINT (code 20) at rsync.c(249)
rsync: connection unexpectedly closed (2722197 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(189)
>>> retry...
|
then rsync reiterates for the specified number of attempts, and issues this error:
Code: |
!!! Rsync has not successfully finished. It is recommended that you keep
!!! trying or that you use the 'emerge-webrsync' option if you are unable
!!! to use rsync due to firewall or other restrictions. This should be a
!!! temporary problem unless complications exist with your network
!!! (and possibly your system's filesystem) configuration.
|
what seems most odd about this situation was that i have not made any changes to the rsync server. the only thing that i had changed was to reinstall gentoo on a client box upon which rsync had worked fine before.
it seems that i have the rsync settings properly configured in make.conf on the client PC:
Code: |
SYNC="rsync://<lan-IP-address>/gentoo-portage"
RSYNC_RETRIES="3"
RSYNC_TIMEOUT=180
|
|
Hello,
that seems more a server side problem who has certainly disconnected since .
Have you increased your --timeout option to 300 or more ?
Look after the /etc/conf.d/rsyncd file or /etc/xinet.d/ directory if your are using xinetd and try to modify the --timeout for a more appropriate value.
Jj _________________ « La seule condition au triomphe du mal, c'est l'inaction des gens de bien » E.Burke
Code: |
+----+----+----+
| |::::| |
| |::::| |
+----+----+----+ |
motto: WeLCRO
WritE Less Code, Repeat Often |
|
Back to top |
|
|
Bob P Advocate
Joined: 20 Oct 2004 Posts: 3355 Location: Jackass! Development Labs
|
Posted: Fri Apr 01, 2005 11:21 pm Post subject: |
|
|
Jimmy Jazz wrote: | that seems more a server side problem who has certainly disconnected since .
Have you increased your --timeout option to 300 or more ?
Look after the /etc/conf.d/rsyncd file or /etc/xinet.d/ directory if your are using xinetd and try to modify the --timeout for a more appropriate value.
Jj |
yes, i thought i had mentioned that i've tried increasing the server and client timeouts to a generous value of 600.
what is really strange is that the server works fine with the other boxes on the network that are rsyncing. one client box (2005.0) is having problems. i've rebuilt rsync on the server and the effected client with no change in results. i'm still trying to figure out what the error messages mean. _________________ .
Stage 1/3 | Jackass! | Rockhopper! | Thanks | Google Sucks |
|
Back to top |
|
|
Jimmy Jazz Guru
Joined: 04 Oct 2004 Posts: 325 Location: Strasbourg
|
Posted: Sat Apr 02, 2005 9:18 am Post subject: |
|
|
@Bob P
Quote: |
yes, i thought i had mentioned that i've tried increasing the server and client timeouts to a generous value of 600.
|
Sorry i was absent-minded and i thought you were using xinetd and in that case rsyncd.conf isn't used at all. You need to add it manually for the service.
The strange thing is that rsync caught an INT signal.
Could you trace the process with strace ?
Code: |
ps aux | grep rsync
strace -pRSYNC_PID
|
And perhaps, try ethereal as well.
Jj _________________ « La seule condition au triomphe du mal, c'est l'inaction des gens de bien » E.Burke
Code: |
+----+----+----+
| |::::| |
| |::::| |
+----+----+----+ |
motto: WeLCRO
WritE Less Code, Repeat Often |
|
Back to top |
|
|
Bob P Advocate
Joined: 20 Oct 2004 Posts: 3355 Location: Jackass! Development Labs
|
Posted: Sun Apr 03, 2005 1:53 am Post subject: |
|
|
well, i've solved the problem. it turns out that there was a corrupted baselayout ebuild in the portage tree. i'm not absolutely certain that this was the problem, as i used the shotgun approach of updating the ebuild while rebuilding the entire installation with the --emptytree command. baselayout was the only ebuild that changed, and the net result is that the "rsync over LAN" problem magically disappeared. it seems that the problem was indeed on the client and not the server. thanks for your help.
one thing that i would point out that may be worth updating in the guide: the location of /etc/rsync/rsyncd.conf is now /etc/rsyncd.conf.
thanks again! _________________ .
Stage 1/3 | Jackass! | Rockhopper! | Thanks | Google Sucks |
|
Back to top |
|
|
bino n00b
Joined: 22 Feb 2004 Posts: 17
|
Posted: Thu Jun 30, 2005 12:12 pm Post subject: |
|
|
Bob P,
How do I do an emptytree? I'm having the same problem you had with a local rsync mirror. I googled a bit but can't find anything...
Thanks for any help!
Bino |
|
Back to top |
|
|
zecora l33t
Joined: 17 Aug 2004 Posts: 627 Location: Minneapolis, MN
|
Posted: Fri Jul 01, 2005 1:22 am Post subject: |
|
|
Code: | Warning: unexpected read size of -864025635 in map_ptr
rsync: connection unexpectedly closed (2776821 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(189)
End: Thu Jun 30 20:10:45 CDT 2005
|
I get that msg when i try to tail and see the progress. What is that @#$%?
Here is my /etc/rsync.conf
Code: | uid = nobody
gid = nobody
use chroot = yes
#limit access to private LAN's
hosts allow=192.168.2.1/255.255.255.0 10.0.0.0/255.255.0.0
hosts deny=ALL
max connections = 15
pid file = /var/run/rsyncd.pid
motd file = /etc/rsyncd.motd
#This will give you a separate log file
#log file = /var/log/rsync.log
#This will log every file transferred - up to 85,000+ per user, per sync
#transfer logging = yes
log format = %t %a %m %f %b
syslog facility = local3
timeout = 300
#If you need this, UPGRADE portage please!
#[gentoo-x86-portage]
#this entry is for compatibility
#path = /usr/portage
#comment = Gentoo Linux Portage tree
#exclude=distfiles/
[gentoo-portage]
#modern versions of portage use this entry
path = /usr/portage
comment = Gentoo Linux Portage tree mirror
exclude = distfiles/ |
Here is my /etc/init.d/rsync
Code: | #!/sbin/runscript
# Copyright 1999-2002 Gentoo Technologies, Inc.
# Distributed under the terms of the GNU General Public License, v2
or later
# $Header: /var/www/www.gentoo.org/raw_cvs/gentoo/xml/htdocs/doc/en/rsync.xml,v$
depend() {
need net
}
# FYI: --sparce seems to cause problems.
RSYNCOPTS="--daemon --safe-links --timeout=300"
start() {
ebegin "Starting rsync daemon"
start-stop-daemon --start --quiet --pidfile /var/run/rsyncd.pid --nicelevel 0 -$
eend $?
}
stop() {
ebegin "Stopping rsync daemon"
start-stop-daemon --stop --quiet --pidfile /var/run/rsyncd.pid
eend $?
}
|
I am just wondering why it wont finish? Plz help |
|
Back to top |
|
|
bino n00b
Joined: 22 Feb 2004 Posts: 17
|
Posted: Fri Jul 01, 2005 1:39 pm Post subject: |
|
|
I just tried it today and it fixed the io.c problem...
So I guess people should try a reboot on the client and perhaps re-running emerge sync on the server (my server does this automatically at night)
HTH
bino |
|
Back to top |
|
|
zecora l33t
Joined: 17 Aug 2004 Posts: 627 Location: Minneapolis, MN
|
Posted: Fri Jul 01, 2005 3:11 pm Post subject: |
|
|
bino wrote: | I just tried it today and it fixed the io.c problem...
So I guess people should try a reboot on the client and perhaps re-running emerge sync on the server (my server does this automatically at night)
HTH
bino |
How does it do it automatically? Plz do tell. |
|
Back to top |
|
|
|