Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
emerge rsync suggestions
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Gentoo Chat
View previous topic :: View next topic  
Author Message
Vlad
Apprentice
Apprentice


Joined: 09 Apr 2002
Posts: 264
Location: San Diego, California

PostPosted: Sun Jul 21, 2002 6:57 am    Post subject: emerge rsync suggestions Reply with quote

First off, let me say I'm happy to see that there are plenty of mirrors to rsync to. I'm glad to see that the community is helping to support gentoo as it experiences "growing pains." The system, however, needs some work...

For example; a lot of the hosts are often busy/full. Wether this is because of dialup users "clogging" the servers, or just tons of people hitting them at once is beyond me; but either way, it's annoying. I'm sure people have been seeing this (in various forms):

Code:
@ERROR: max connections (50) reached - try again later


I can understand that. The servers busy. No problem, I'll just try it again. And here's where the problem starts; after a dozen or so attempts at "emerge rsync," the only host I get redirected to is not only busy, it's the same host. Something has to be done. Perhaps "randomize" what mirror a client is redirected to. I have no clue how it's done now, but it certainly doesn't work very well.

Other times, emerge rsync will send me to a mirror that is not responding (for one reason or another), and simply hang like so:

Code:

localhost # emerge rsync
>>> starting rsync with rsync://rsync.gentoo.org/gentoo-portage...


And again, when I abort and try to emerge rsync, the same problem occurs. Over, and over, and over. Eventually I get through, or I just later. This isn't exactly an inherent "flaw" in the system, more like a trivial annoyance (especially when you're trying to emerge rsync for the first time...). Anywho, something needs to be done to better randomize what rsync mirror a client is sent to.

Vlad
Back to top
View user's profile Send private message
rasat
n00b
n00b


Joined: 30 May 2002
Posts: 56
Location: -Kenya -Mauritius -Singapore

PostPosted: Sun Jul 21, 2002 7:58 am    Post subject: Reply with quote

.
My ADSL connection is in Singapore where I stay often. No problem with the emerge rsync. I get easily connected. I don't think it has anything to do with the location. Maybe with the download site config.

Here is my /etc/make.conf

# Download sites
# The main Gentoo Linux source mirror; specify as many space-separated mirrors
# as you like.
# GENTOO_MIRRORS="http://www.ibiblio.org/gentoo"
# The main Gentoo Linux Portage server; specify a single Portage server.

SYNC="rsync://rsync.gentoo.org/gentoo-portage"

# An alternate download server if you prefer anoncvs. emerge dev-util/cvs
# for cvs support in emerge. The password for this CVS server is empty --
# just hit Enter.
#SYNC="cvs://:pserver:anonymous@gentoo.org:/home/anoncvs"

Markku
Back to top
View user's profile Send private message
klieber
Administrator
Administrator


Joined: 17 Apr 2002
Posts: 3657
Location: San Francisco, CA

PostPosted: Mon Jul 22, 2002 7:30 pm    Post subject: Re: emerge rsync suggestions Reply with quote

Vlad wrote:
I can understand that. The servers busy. No problem, I'll just try it again. And here's where the problem starts; after a dozen or so attempts at "emerge rsync," the only host I get redirected to is not only busy, it's the same host. Something has to be done. Perhaps "randomize" what mirror a client is redirected to. I have no clue how it's done now, but it certainly doesn't work very well.


Currently, rsync.gentoo.org is simply a round robin DNS setup, which is why you're having this problem. When your computer tries to resolve "rsync.gentoo.org" to an IP address, chances are your DNS server is caching that information. So, if that server is busy, and you try to hit rsync.gentoo.org again, your name server will realize that it already has that information in its cache and send it to you instead of going over the internet to request the address from the gentoo.org authoritative name server.

So, that's all well and good, but it doesn't help solve your problem. :) Here's what you can do instead:

Code:
host rsync.gentoo.org


('host' is part of the bind-utils package if you don't already have it on your system)

That will return all the IP addresses associated with rsync.gentoo.org (currently 16). Pick one of those IP addresses, stick it in your hosts file as "rsync.gentoo.org" and you should be good to go.

Granted, that doesn't obviate the need for a better way to assign rsync servers, but its at least a viable work-around in the mean time.

--kurt
_________________
The problem with political jokes is that they get elected
Back to top
View user's profile Send private message
delta407
Bodhisattva
Bodhisattva


Joined: 23 Apr 2002
Posts: 2876
Location: Chicago, IL

PostPosted: Mon Jul 22, 2002 7:38 pm    Post subject: Reply with quote

Perhaps a mirror list could be distributed (via http or something) that would eliminate the DNS caching issues, but it's not Gentoo's fault. Yell at your ISP; their nameserver is not adhering to the standards, so...
_________________
I don't believe in witty sigs.
Back to top
View user's profile Send private message
klieber
Administrator
Administrator


Joined: 17 Apr 2002
Posts: 3657
Location: San Francisco, CA

PostPosted: Mon Jul 22, 2002 9:53 pm    Post subject: Reply with quote

delta407 wrote:
Yell at your ISP; their nameserver is not adhering to the standards, so...

Actually, caching is an accepted, standard and vital part of DNS. That's the whole purpose of TTL settings on bind zone files -- to define how long external name servers can cache your information before they have to come back to you and refresh it.

Anyway, that's getting a bit outside the realm of this topic, but please don't go yelling at your ISP, because they'll yell right back at you. :)

--kurt
_________________
The problem with political jokes is that they get elected
Back to top
View user's profile Send private message
mikki
n00b
n00b


Joined: 14 Jul 2002
Posts: 51
Location: On Top

PostPosted: Mon Jul 22, 2002 11:50 pm    Post subject: Reply with quote

Using a dns round-robin approach has another issue, kind of. Sometimes hosts are chosen that are in far-off lands (relative to the client). Some regionalization might be in order, at least by continent. Not that the list is all that long, but it seems a bit wasteful...
_________________
Mikki
Back to top
View user's profile Send private message
klieber
Administrator
Administrator


Joined: 17 Apr 2002
Posts: 3657
Location: San Francisco, CA

PostPosted: Mon Jul 22, 2002 11:56 pm    Post subject: Reply with quote

mikki wrote:
Using a dns round-robin approach has another issue, kind of. Sometimes hosts are chosen that are in far-off lands (relative to the client). Some regionalization might be in order, at least by continent. Not that the list is all that long, but it seems a bit wasteful...


Change your rsync server to rsync.us.gentoo.org and you'll get only US-based rsync servers.

--kurt
_________________
The problem with political jokes is that they get elected
Back to top
View user's profile Send private message
Vlad
Apprentice
Apprentice


Joined: 09 Apr 2002
Posts: 264
Location: San Diego, California

PostPosted: Tue Jul 23, 2002 10:41 pm    Post subject: Reply with quote

Thanks for the replies. I'll change my config to use the us based rsync server (I was completely unaware that this was even an option). Many thanks to kurt for the explanation and resolution :)

Vlad
Back to top
View user's profile Send private message
klieber
Administrator
Administrator


Joined: 17 Apr 2002
Posts: 3657
Location: San Francisco, CA

PostPosted: Tue Jul 23, 2002 10:53 pm    Post subject: Reply with quote

Vlad wrote:
(I was completely unaware that this was even an option).


A poorly-documented one, I might add.

For folks in other countries, you can determine if there are country-specific rsync servers for you by doing something like the following:

Code:
host rsync.<your country code here>.gentoo.org


If you get a response, then there's an rsync server for your country.

I know there are ones for the UK, Spain, Italy, Germany, The Netherlands, Norway and Sweden, but there are others as well.

If you find that your country isn't listed, well have some national pride!!! Set one up! :) I know there are no servers in Isreal, Japan and Australia, yet there are regulars here from all of those countries. :)

--kurt
_________________
The problem with political jokes is that they get elected
Back to top
View user's profile Send private message
delta407
Bodhisattva
Bodhisattva


Joined: 23 Apr 2002
Posts: 2876
Location: Chicago, IL

PostPosted: Tue Aug 06, 2002 3:21 am    Post subject: Reply with quote

klieber wrote:
delta407 wrote:
Yell at your ISP; their nameserver is not adhering to the standards, so...

Actually, caching is an accepted, standard and vital part of DNS. That's the whole purpose of TTL settings on bind zone files -- to define how long external name servers can cache your information before they have to come back to you and refresh it.


Right, I know. But, if the gentoo.org zone file is configured properly then the round-robin DNS will work -- if it is configured properly and the ISP insists on sending back the cached version (which, if it is truly round-robin DNS, it ought not do) then their DNS server is set up to cache things in a non-RFC-compliant manner.

But, I digress, because it is off-topic.
_________________
I don't believe in witty sigs.
Back to top
View user's profile Send private message
quadbox
Tux's lil' helper
Tux's lil' helper


Joined: 11 Jun 2002
Posts: 92
Location: Brisbane, Australia

PostPosted: Sat Aug 10, 2002 3:38 am    Post subject: Reply with quote

Just to point out, for anyone else that couldn't find it: The afore mentioned package "bind-utils", is infact called "bind-tools" in the portage tree.

Quote:
I know there are no servers in Australia


Lamentably so...
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo Chat All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum