Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
HOWTO:Local Rsync Mirror
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4, 5, 6  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
bino
n00b
n00b


Joined: 22 Feb 2004
Posts: 17

PostPosted: Fri Jul 01, 2005 4:40 pm    Post subject: Reply with quote

Sorry, I'll make it clearer. My server has a crontab that runs emerge sync in the middle of the night. So I was trying to explain that the sync run in the night might have fixed the problem. Hope that helps
Bino
Back to top
View user's profile Send private message
chratnox
n00b
n00b


Joined: 04 Aug 2004
Posts: 11
Location: Netherlands

PostPosted: Mon Jul 04, 2005 9:56 am    Post subject: Reply with quote

First off, very nice howto. However, I'm not gonna use it since it nearly kills my p4 2.6 (read: apache/mysql get slow because of it).

Anyway, when you're speaking about security, wouldn't it be better to run it as an other user then 'nobody'?

A snippet from the gentoo Security handbook: (Here)
Quote:

#It is not a good idea to use nobody or nogroup -
#for every service not running as root
#(just add the user apache with group apache)
User apache
Group apache


Just my 2 pennies ;)
Back to top
View user's profile Send private message
piyo
n00b
n00b


Joined: 20 Mar 2004
Posts: 12

PostPosted: Sun Jul 24, 2005 2:29 am    Post subject: Reply with quote

JSharku wrote:
Just a quick note on packages and distfiles; it's better to put the following in your rsyncd.conf:
Code:

# excluding packages is optional, if you don't use --buildpkg you don't need it
exclude = distfiles/ packages/

instead of
Code:

exclude = distfiles packages

NOTE THE TRAILING /'s
If you don't add the slashes rsync will exclude anything ending in either distfiles or packages...


Just a FYI, but this forum thread "emerge depclean says "You have no system list..."" shows what happens when there are no "/".
Back to top
View user's profile Send private message
CarlUman
Apprentice
Apprentice


Joined: 07 Jul 2004
Posts: 158
Location: SE Iowa

PostPosted: Thu Jul 28, 2005 3:21 am    Post subject: Reply with quote

I'm working on this but when I start the process it doesn't write to the .pid file. So when I try /etc/init.d/rsyncd stop it will not stop because there is no pid file. I created the /var/run/rsyncd.pid file but got the same results. Shouldn't I be able to see a rsyncd process in top? Maybe for some reason the process isn't even starting even though there is no error on startup.

Thanks for any help
Carl
Back to top
View user's profile Send private message
CarlUman
Apprentice
Apprentice


Joined: 07 Jul 2004
Posts: 158
Location: SE Iowa

PostPosted: Thu Jul 28, 2005 11:41 pm    Post subject: Reply with quote

I tried rsync --deamon and nothing happens. Nothing in ps or top. What is up?
Back to top
View user's profile Send private message
vico
n00b
n00b


Joined: 06 Jul 2005
Posts: 21

PostPosted: Sun Aug 07, 2005 12:35 am    Post subject: Reply with quote

hello every body :)
i have a question about the config , if the server architecture and the rest of the boxes are different ,
do i have to change my make.conf ? or something else.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Tue Aug 09, 2005 9:16 pm    Post subject: Reply with quote

vico wrote:
hello every body :)
i have a question about the config , if the server architecture and the rest of the boxes are different ,
do i have to change my make.conf ? or something else.



Not if your following the howto!
Back to top
View user's profile Send private message
BertsInn
n00b
n00b


Joined: 02 Jun 2003
Posts: 45
Location: Houston, Texas

PostPosted: Sun Aug 14, 2005 3:20 pm    Post subject: Rsyncd changes Reply with quote

A recent portage update to rsync has changed both the /etc/rsyncd.conf and /etc/init.d/rsyncd files. No surprise that this caused failures. What is the best way to return to working state? Just a matter of copying back your examples?
_________________
Governments Fiscal Policy - Low on cash - print more
Back to top
View user's profile Send private message
Bob P
Advocate
Advocate


Joined: 20 Oct 2004
Posts: 3355
Location: Jackass! Development Labs

PostPosted: Mon Aug 15, 2005 10:55 am    Post subject: Reply with quote

you definitely need to restore your rsyncd.conf file if it was overwritten by etc-update following an emerge. i think you should probably be okay using the new init.d file.
_________________
.
Stage 1/3 | Jackass! | Rockhopper! | Thanks | Google Sucks
Back to top
View user's profile Send private message
Salemixu
n00b
n00b


Joined: 01 Jul 2004
Posts: 72
Location: Spain

PostPosted: Thu Sep 08, 2005 12:17 pm    Post subject: Reply with quote

The Rsync Mirro has been working for quite some time now, but today i hasn't worked, it says this when i sync from the client:
Code:

>>> starting rsync with rsync://192.168.1.2/gentoo-portage...
>>> checking server timestamp ...
rsync: connection unexpectedly closed (0 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(189)
>>> retry ...


I redid what it says in this tutorial, so the configuration files are the same, and i have put two copies of rsyncd.conf, one in /etc and the other on /etc/rsync. Then i have noticed that happens the same as CarlUman, the rsync process is not loaded never, no matter if i run it by init.d or manuall by command line.
I have updated portage and updated the rsync package of the server side and still it doesn't load :(
Any hints?
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Sun Sep 18, 2005 6:43 am    Post subject: Reply with quote

The rsync troubleshooter says:
http://samba.anu.edu.au/rsync/issues.html

Q: Why does my transfer die with something like the following error?

rsync: error writing 4 unbuffered bytes - exiting: Broken pipe
rsync error: error in rsync protocol data stream (code 12) at io.c(463)

or

rsync: connection unexpectedly closed (24 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(342)

A: This error tells you that the local rsync was trying to talk to the remote rsync, but the connection to that rsync is now gone. The thing you must figure out is why, and that can involve some investigative work.

If the remote rsync is a daemon, your first step should be to look at the daemon's log file to see if it logged an error explaining why it aborted the transfer.

Beginning with version 2.6.3, rsync now does a better job of retreiving the error from the remote process than older versions of rsync, so you may wish to try upgrading and see if you now get sent the error message from the remote rsync.

There are several common causes for a remote rsync process going away:

* The destination disk is full (remember that you need at least the size of the largest file that needs to be updated available in free disk space for the transfer to succeed).
* A network error caused the connection to be dropped.
* The remote rsync executable wasn't found.
* Your remote-shell setup isn't working right or isn't "clean" (i.e. it is sending spurious text to rsync).
Back to top
View user's profile Send private message
Salemixu
n00b
n00b


Joined: 01 Jul 2004
Posts: 72
Location: Spain

PostPosted: Sun Sep 18, 2005 11:36 am    Post subject: Reply with quote

Now i tried again without modifying anything more, and it works :/
Perhaps it is fixes because of the reboots i have made in some days.
Back to top
View user's profile Send private message
gronlund2
n00b
n00b


Joined: 22 Jul 2005
Posts: 4

PostPosted: Mon Sep 19, 2005 6:10 pm    Post subject: rsync remote command to temporarily disable cvs access Reply with quote

Hi.
I'm setting up a cvs backup/mirror. I need to temporarily disable cvs access on the remote host that is to be backed-up since a check-in during rsync may leave the back-upped repository in an inconsistent state.
So I need to execute a remote command in my back-up script (to disable cvs on the remote machine). Judging from the manuals it's a real pain to do this with ssh. If I generate keys and distribute the public key, I still need to supply a pass-phrase. And using ssh-agent seems like a real mess to me.
Is there any easy way to execute a remote command in a script (i.e. automatically supply a password)?

Can I run rsync on the to be backed-up server instead? (No need to execute the remote command then.)

Grateful for your help.

/G2
Back to top
View user's profile Send private message
thecooptoo
Veteran
Veteran


Joined: 27 Apr 2003
Posts: 1353
Location: UK

PostPosted: Sat Mar 04, 2006 10:51 am    Post subject: Reply with quote

I did this
Code:
grenada etc # ls -la |grep rsync
drwxr-xr-x    2 root root     4096 Feb 26 21:31 rsync
lrwxrwxrwx    1 root root       17 Mar  4 10:19 rsyncd.conf -> rsync/rsyncd.conf

and then set up the server
Code:
grenada etc # grep ^[a-z] /etc/rsyncd.conf
uid = nobody
gid = nobody
use chroot = yes
max connections = 20
pid file = /var/run/rsyncd.pid
motd file = /etc/rsync/rsyncd.motd
transfer logging = yes
log format = %t %a %m %f %b
syslog facility = local3
timeout = 300
path = /opt/gentoo-rsync/portage
comment = Gentoo Linux Portage tree
path = /opt/gentoo-rsync/portage
comment = Gentoo Linux Portage tree mirror
exclude = distfiles
hosts allow=192.168.0.0/255.255.0.0 10.0.0.0/255.0.0.0
hosts deny=*
grenada etc #       

when i connect from the client
Code:
dads distfiles # emerge sync
>>> starting rsync with rsync://192.168.0.1/gentoo-portage...
>>> checking server timestamp ...
This is rsync[number].[country].gentoo.org.

receiving file list ...
link_stat "metadata/timestamp.chk" (in gentoo-portage) failed: No such file or directory
0 files to consider
client: nothing to do: perhaps you need to specify some filenames or the --recursive option?
rsync error: some files could not be transferred (code 23) at main.c(653)
>>> retry ...


ive got onthe client
Code:
dads distfiles # cat /usr/portage/metadata/timestamp.chk
Sat Mar  4 10:36:59 GMT 2006
dads distfiles #
dads distfiles # grep ^[SYNC]  /etc/make.conf
CHOST="i686-pc-linux-gnu"
CFLAGS="-march=athlon -O3 -pipe"
CXXFLAGS="${CFLAGS}"
SYNC="rsync://192.168.0.1/gentoo-portage"

and on the server FWIW
Code:
grenada etc # cat /usr/portage/metadata/timestamp.chk
Sat Mar  4 10:38:33 GMT 2006
grenada etc #


_________________
join the optout - http://nhsconfidentiality.org
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Mon Mar 06, 2006 1:21 am    Post subject: Reply with quote

thecooptoo wrote:

path = /opt/gentoo-rsync/portage
comment = Gentoo Linux Portage tree
path = /opt/gentoo-rsync/portage
comment = Gentoo Linux Portage tree mirror
exclude = distfiles




Your not using the info from this howto. I lost track of this thread for a while and yes, it does need some updating.

Is the server a gentoo box? If so, the portage dir probably isn't located in /opt/gentoo-rsync. If it isn't a gentoo box, the portage tree probably isn't in /opt/gentoo-rsync either.
Back to top
View user's profile Send private message
radr
Tux's lil' helper
Tux's lil' helper


Joined: 15 Jul 2002
Posts: 77
Location: MT

PostPosted: Tue Mar 07, 2006 6:44 am    Post subject: Thanx Reply with quote

i been successfully using this since early 2k5. never failed me
one sync to rule them all (heh)

and space saved on a 23 machine LAN = priceless

for reference my local sync mirror has a gentoo install that has lasted since late 2k1 (anyone remember pre 1.4 gentoo?)
_________________
Dual Opteron 280/1M (dual core = 4 procs) , 16G reg ECC ram, 4TB disc, NvIDIA GF 6800 PCI-E, SB X-FI
=Theres NO Substitute for Horsepower=
Back to top
View user's profile Send private message
ulises.dumount
n00b
n00b


Joined: 16 Feb 2006
Posts: 44

PostPosted: Thu Mar 16, 2006 2:42 pm    Post subject: Proxy problems! Reply with quote

I follow all the steps in this post but my problem is that i have a LAN and http-replicator working in a pc at this LAN, but all the PCs are working with a proxy to surf websites, and when i need to do a "emerge sync" i get this error:

emerge sync
>>> starting rsync with rsync://192.168.1.72/gentoo-portage...
>>> checking server timestamp ...
bad response from proxy - HTTP/1.1 502 Proxy Error ( El puerto SSL (nivel de sockets seguro) especificado no está permitido. El servidor ISA no está configurado para permitir peticiones de SSL de este puerto. La mayoría de los exploradores de Web utilizan el puerto 443 para peticiones de SSL. )
rsync: failed to connect to 192.168.1.23: Success
rsync error: error in socket IO (code 10) at clientserver.c(88)
>>> retry ...

It seems like it want to connect to internet allways when i try to sync, but the proxy in this LAN didn't give me acces to internet via SSL in that port or something like that.

Any ideas??
_________________
Tres hurras para Gentoo!!
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Fri Mar 17, 2006 12:16 am    Post subject: Re: Proxy problems! Reply with quote

ulises.dumount wrote:

It seems like it want to connect to internet allways when i try to sync, but the proxy in this LAN didn't give me acces to internet via SSL in that port or something like that.

Any ideas??


Is this error when trying to sync the box running the local rsync server or is it when a client tries to connect to your local rsync server?

If it is a client, then it looks to me that all requests are going to the other proxy. You need to change the settings so that the rsync request doesn't go through the proxy. I don't know anything about how your proxy is configured, but there should be a way to turn off the proxy for requests that stay local to your LAN.

If it is the box running the local rsync server, then can you emerge sync normally through your internet proxy? I'm suggesting that the proxy your going through to the internet just won't allow you to emerge sync because of the port it uses. I've run into that before and you need to look into web-rsync which uses the http port to sync the local rsync server box. Then you can have clients connect to the local rsync server on your lan in the normal fashion.
Back to top
View user's profile Send private message
ulises.dumount
n00b
n00b


Joined: 16 Feb 2006
Posts: 44

PostPosted: Fri Mar 17, 2006 3:40 pm    Post subject: Yes Reply with quote

Yes, this error is when a client tries to connect to my local rsync server. All the pcs are behind a proxy and are configured to go through it, all the clients are configured to emerge packages from my http_replicator in my pc. But the problem is when the clients want to sync because they want to go to internet to sync. In my http replicator i do it with "emerge-webrsync" and it goes through the proxy well. But i don't want all my clients to use the emerge-webrsync all the time they want to actualize!!!

Any ideas??
_________________
Tres hurras para Gentoo!!
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Fri Mar 17, 2006 6:39 pm    Post subject: Reply with quote

How did you configure the clients to use the proxy?

Try this on the clients:

Code:

export RSYNC_PROXY=""
emerge sync
Back to top
View user's profile Send private message
ulises.dumount
n00b
n00b


Joined: 16 Feb 2006
Posts: 44

PostPosted: Fri Mar 17, 2006 6:49 pm    Post subject: Nop Reply with quote

export RSYNC_PROXY="192.168.1.23"

emerge sync

>>> starting rsync with rsync://192.168.1.72/gentoo-portage...
>>> checking server timestamp ...
invalid proxy specification: should be HOST:PORT
rsync error: error in socket IO (code 10) at clientserver.c(88)
>>> retry ...

The same problem.
_________________
Tres hurras para Gentoo!!
Back to top
View user's profile Send private message
ulises.dumount
n00b
n00b


Joined: 16 Feb 2006
Posts: 44

PostPosted: Fri Mar 17, 2006 7:37 pm    Post subject: Solved!! Reply with quote

I found a partial solution!!! This is it:

Always when i need to update the clients i need to do three steps:

1- export RSYNC_PROXY=""

2- emerge sync

3- export RSYNC_PROXY="yourproxy.com"

And the emerge works great with a http replicator working in the other pc of my LAN. I only need to configure the clients to not proxy the ip of my http replicator, but for now i do a scrip with these 3 steps in every pc client of my LAN.

Thanks a lot!!
_________________
Tres hurras para Gentoo!!
Back to top
View user's profile Send private message
melinux
n00b
n00b


Joined: 25 May 2006
Posts: 59
Location: Malta

PostPosted: Thu May 25, 2006 12:29 pm    Post subject: Reply with quote

Is there a problem with the latest rsync?
When I start it I get this error:
* Starting rsyncd ...
rsync: --safe-links: unknown option (in daemon mode)
(Type "rsync --daemon --help" for assistance with daemon mode.)
rsync error: syntax or usage error (code 1) at options.c(857) [client=2. [ !! ]


My rsync is the following:
net-misc/rsync-2.6.8-r2 USE="xinetd -acl -build -ipv6 -static"

It seems the option --safe-links has been removed. Does anyone know the best way to fix this in the init script- besides removing the --safe-links and timeout params of course (which I am temporarily doing)? I am using the init script from this thread.
Back to top
View user's profile Send private message
KB7OEB
n00b
n00b


Joined: 28 Dec 2003
Posts: 58
Location: Phoenix,Arizona

PostPosted: Sun Jul 30, 2006 4:17 am    Post subject: Reply with quote

Do real portage mirrors use ssh while syncing? I'm thinking its running encryption and causing the slow down.
Back to top
View user's profile Send private message
zxy
Veteran
Veteran


Joined: 06 Jan 2006
Posts: 1160
Location: in bed in front of the computer

PostPosted: Thu Aug 17, 2006 5:25 pm    Post subject: Reply with quote

@melinux
I have the same problem with the same version of rsync.
_________________
Nature does not hurry, yet everything is accomplished.
Lao Tzu
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3, 4, 5, 6  Next
Page 5 of 6

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum