[LONG] Proposal for an alternative portage tree sync method

[wizeman]

Warning: long

Hi

Recently I've been having some problems doing the 'emerge sync' routine, which I think are related to my ISP connection.
The only alternative, of course, is the far from optimal 'emerge-webrsync'.

Both of these methods have problems.

Specifically, the rsync method has these problems:

1. Doesn't work behind restrictive firewalls
   
2. Due to the large number of files in the portage tree, it takes a lot of time to build/transfer this list
   
3. Involves a lot of disk trashing, also due to the large number of files
   
4. Puts a lot of stress on the server
   

The emerge-webrsync method is also not without problems:

1. Transfers the whole (compressed) portage tree at once
   
2. .. which is not an optimal (read: viable) option for daily updates
   
3. Involves a lot of disk thrashing anyway, due to the subsequent local rsync
   

Alternative methods could involve CVSup, which was already considered before, however:

1. Doesn't work behind restrictive firewalls
   
2. It's written in Modula-3, which is not available on all architectures (AMD64 for instance, although it's getting more portable)
   
3. Also involves disk thrashing (it's less of a bottleneck than rsync, however)
   
4. Also puts (some?) stress on the server
   

So I would like to propose an idea for an alternative portage syncing method.

This method is based on zsync (ebuild available here).
zsync basically synchronizes a local file using the rsync algorithm, but through a normal HTTP server.
To do this, it only needs a pre-generated .zsync file on the server.

One of the most useful features of zsync, however, is that the file to be synchronized can be a common gzipped file on the server-side, and zsync only downloads the (compressed) ranges which are really necessary to synchronize the file.
But the main algorithm, of course, is still the rsync algorithm.
You can read more about this, including performance comparisons to rsync, on the technical paper.

Because zsync is designed to be easy to use, the operations needed to synchronize the portage tree would be very simple.

The method I present here would involve these operations on the server-side:

1. Create a compressed (gzipped) iso file of the portage tree on the server-side.
   This file would weight at about 27 MB nowadays.
   
2. Create the .zsync file (very fast, and about 700 KB with a 4K block size).
   
3. Move both files to the public HTTP directory.
   

On the client side, only this would be necessary:

1. Use zsync to synchronize the portage ISO (this file would be about 260 MB).
   
2. Unmount /usr/portage (old ISO file is still unaltered on disk).
   
3. Mount (-o ro,loop) /usr/portage (with the new ISO file).
   
4. Remove old ISO file.
   

As you can see, the ISO file is mounted in /usr/portage read-only, which I believe is not a problem if the user changes DISTDIR and PKGDIR in /etc/make.conf.

This method currently also has its share of problems:

1. zsync is, according to the author, in alpha stage (although it works great already!).
   I believe gentoo users would be an excellent testing crowd
   
2. For low-disk space servers/routers it may not be possible to store a temporary 260 MB ISO.
   This can be circumvented by the fact that zsync allows to be piped the input file.
   So we could umount /usr/portage, compress iso file, pipe it to zsync (with zcat), remove the old compressed iso file and mount /usr/portage.
   This of course, would make the portage tree unavailable during synchronization.
   

This method, however, has the following advantages:

1. It uses the network-efficient rsync algorithm.
   The current tests performed by the author suggest its bandwidth usage is more or less comparable to rsync.
   
2. Works behind restrictive firewalls, as it only uses HTTP.
   
3. It is much lighter on the servers.
   
4. The disk usage pattern is very efficient, because it works sequentially on only 2 files.
   
5. zsync is made in "simple" C, which is easily portable to all architectures (it already works for AMD64, at the very least) and fast.
   And it's being actively developed.
   
6. It's simple!
   

--------------------------------------------------------------------------------------------------------

So if you think this is a worthy idea, how about setting up an experimental zsync portage mirror?
I would do this myself, but my upstream bandwidth is *very* limited

If you are willing to give it a try, here's a preliminary HOWTO:

Necessary steps to make the portage tree available on the mirror:

1. emerge zsync.
   
2. mkdir /var/www/localhost/htdocs/portsync (modify as necessary).
   

Then periodically:

[garfield]

This is interesting, but I'm sure it will help your course if you provide data about
- time for sync of entire tree
- bandwidth for sync of entire tree (use jnettop or the like to measure this)
- load during sync on (most importantly) server and (less importantly) client
for both the rsync and zsync solution. Numbers for websync could also be included.
You link to some tests, but I didn't see tests that was comparable to a portage sync (+100k small files, and ½-2k changing per day). Also the rsync was run through a tunnel I believe. And he talks about compression, but it wasn't very clear to me what it means in the end (read: I just browsed his page quickly). Finally, it would be nice to know more about how well the server could maintain an updated database.

[wizeman]

Well, I would provide the data but.. I would need someone to setup a zsync mirror (with good bandwidth) for testing

The zsync author even offered to setup a .zsync of the portage tarball on his server, but that wouldn't work as well as this, because users would have to remove the entire portage tree and extract the new tarball.

The ISO way would be much better, as it can be mounted directly (no extraction/disk trashing need).

However, the author is more of a debian/freebsd user himself, so he can't sync the portage tree at least daily..

Also some clarifications to the original post may be necessary:

Users would only have to download the changed portions of the ISO, compressed . Some people didn't quite understand this.
Worst case scenario (user doesn't have the portage tree) would only be a 27 MB download. Of course, if only 800 KB of the uncompressed ISO was changed, then only the compressed portions that make up that 800 KB would be necessary to be download. This is because the .zsync file has an internal mapping of the uncompressed to the compressed portions.
Someone mentioned the ordering of the ISO file creation.. that might make users download a bit more, but I think it's possible to sort the files during ISO creation, if I'm not mistaken.
Anyway, don't forget the rsync algorithm works on overlapping blocks, so as long the content is there, it should be possible to recreate it in another place without downloading.

Also another advantage would be speed. Think how fast disks work sequentially as opposed to having to work on 110,000 scattered files!

But of course, only if someone sets this up we'll be able to see how well it works.. *hint* *hint*

It has the potential to be much better than *at least* emerge-webrsync.. so I don't see the problem

[Betelgeuse]

Is 1 mbit enough? I can set this up for a few users. You can find me as Betelgeuse@freenode.

[Betelgeuse]

Ok. I now have a working server. I made a cron job for updating the server.
http://a.bo.cx/gentoo/sync.sh
It requires zsync and cdrtools. I use esync from esearch to update the tree but one could also use emerge sync.

[wizeman]

Ok, thanks to Betelgeuse, I have some field results:

I created an ISO of my portage tree which is updated as of 2005/03/20 01:36.
Betelgeuse 'emerge sync'ed his tree as of now - 2005/03/22 19:00 (here) and then created a new ISO and a new .zsync file.

This is the result:

[Betelgeuse]

I only have 1 mbit upstream so I don't want tons of users. I can provide the url to users on request.

[wizeman]

As Betelgeuse suggested, I'm going to provide more comparisons between the different sync methods, including the time it takes to 'emerge metadata' where necessary (I was forgetting about this part ).

Between the tests I will reset /usr/portage to a snapshot of 20050319 (and emerge metadata).

Of course these results are only empirical, and it can vary a lot between different systems, however it can be a good comparison.

Lets see how it goes

[jd5419]

I have a server in california with a pretty fast upstream if you want i can setup some zrsync stuff just tell me how and or i'll give you a shell to setup whatever necessary... rsync is kind of a pain here too.

[Betelgeuse]

[dufnutz]

Have you submitted this idea to buzilla with a link to this forum post? If you have I would like to read the bug.

[wizeman]

I didn't submit a bug because I thought it would be more appropriate to discuss here first.

But I submitted an email to the gentoo-dev mailing list, and I received some answers.
You may read the archive if you're interested

However, I'm having trouble responding to the mailing list, it seems my mails aren't getting through..

[wizeman]

Ok, here are some figures.
Remember, this is only to have an idea of the time it takes on my PC, it can vary a lot among systems!

My machine is an AMD Athlon 64, with 1 GB RAM, with a 120 GB SATA disk with the XFS filesystem.
The filesystem of /usr/portage, however, is Reiserfs.
Bandwidth usage was measured with jnettop.

Before each one of these tests, I did:

[wizeman]

If all goes well, we should have a working zsync portage mirror tomorrow for available for all gentoo users.

After that, I'll submit the zsync ebuilds to bugzilla and then I'll start to work on an easy-to-use emerge-zsync script for anyone to use

I'll keep you posted.

[dufnutz]

perhaps you should edit your first post to reflect the --sort option to be passed to mkisofs

[wizeman]

The mkisofs --sort option requires a list of the files and it's weights (it has to be precomputed).
I'm not sure if it's worth the trouble, especially if one always uses the same mirror to sync.
I think that may require more analysis.

Anyway, the zsync-0.3.2 ebuild (bug #86406) is submitted.

[cphipps]

[tuxp3]

I like the idea - but im not too familiar with how ISOs work (like the data layout)

can an ISO be mounted rw? or would it always be ro? just woundering

also how does rsync handle the iso? i mean how much "extra" information would be downloaded verses how much information would actually need to be downloaded. aka size of downloaded ebuilds vs. iso differences- also would it be possible to compress all this data onthefly?

just some thoughts after reading this thread.

i really like the idea tho - right now i have a seperate partition for portage just because of the sheer number of files

Thanks, Tux

[wizeman]

Well, ISOs must be mounted ro.

About how much more information it transfers (being an ISO), I don't really know.

But about compressed transfers.. it could be possible to store the ISO file uncompressed on the server, and then compress the transfers with the Apache mod_gzip, but perhaps only if mod_gzip compresses multi-part, pipelined ranged requests.

That's another thing we could try, if/when zsync supports it

[tuxp3]

see as much as i think ur idea rocks - i just see a few too many fallbacks of the ISO's being only able to be mounted ro - this is in a way "ineffiecnt" if we had another "loop" FS that was as light as iso (very similar) but r/w (maybe UDF or something) then i could see some really big advantages
forexample - the server would just append the new ebuild data to the end - and this would make the rsync even more effient - it just scans and then 'oh look more data that what i already have " and DL's the data - not as much searching around to figure out changes...

but then again im sure that has drawbacks as well /me wishes ram was cheaper and portage was smaller (RAM DISK lol)

but if u could get compression to work with zsync it would be freakin awesome - can ISO's be set to compress the data as its added to it upon creation? maybe that could be a work around.. but indeed it would hurt performance for clients when queuing the DB

Thanks, Tux

[kimchi_sg]

Someone make genone read this thread please... This sounds great, but if it is ever to make its way into portage, a portage dev must see this. And genone's the one portage dev who goes around the forums most often.

@wizeman: File a bug on https://bugs.gentoo.org too. Then they will have to give this a look.

[wizeman]

Just to let you know, today we've been making some tests with different storage formats and compression.

I used 2 portage trees with about a 24-hour difference to make some comparisons, trying mainly to reduce the necessary bandwidth to synchronize.

A compressed ISO file would take about 4.5M + 0.7M (for the .zsync) of bandwidth.

The best we've come so far, is to use an uncompressed squashfs (88M).
If we use gzip to compress the uncompressed squashfs on the server, it becomes 21M in size (with a 0.5M .zsync file, if we use 2048-byte blocks).

Then, the client synchronizes with zsync, using only (in this test) 2.5M + 0.5M of bandwidth.
So that's already a nice improvement - about 3 MB of bandwidth versus the 5.2 MB necessary for the ISO, and a much smaller file in the client
Of course if you consider the about 18-19 MB of bandwidth or so for emerge-webrsync, it looks even better

The problem is that not everyone has squashfs compiled in the kernel. However, karltk told me the latest gentoo kernel sources has it/will have it integrated.
But we don't want to force anyone to use a non-standard filesystem, so even if we use squashfs it will be optional, of course.

Meanwhile, karltk is setting up the server so you'll have to wait until he can be sure the scripts are working fine

BTW, I don't want to hassle the portage devs right now more than is necessary. I've already brought this subject up on the IRC channels, and I've sent a mail to the gentoo-dev mailing list, that should be enough for now

When people start using it more and we can be sure everything is working right, then we should discuss about including it in portage

[tuxp3]

i just thought of something extremely verstile : torrents - torrents are made to "fill in the missing data" and such like that and its all hash checks so u can be assured that ur ISO matchs what everyone else has - maybe a branch of exisiting torrent code and tracker would make this a vaable solution - the only thing is : will torrents be able to " make room" for new data without trashing too much data thats going to end up being redownloaded - i would think this is possible if we use very small piece sizes... but maybe this zsync is a better solutiion - but eitherway this could really help reduce the load on the rsync servers - because we copuld make it default (changable of course) to allocate some of the user's bandwidth anytime emerge is running and since most users do emerge -avuD right after emerge sync - this could make it very easy to 'repay' the leecher pool.. but this could go all wrong if no one shares.. but eitherway it would centralize the main portage DB a little bit more... the torrents them selves would have to be automagicly updated when the user starts the sync - but besides that the rsync servers could be turned into seeders.. cron them to fetch a new torrent every half hour..

just some thoughts as i make some more free space as i open up some torrents

Tuxp3

[ssvb]

One more way to reduce bandwidth is to use delta patches instead of rsync algorithm. So the server can create iso snapshots every day and create delta patches that convert one snapshot into another. The user will only have to download appropriate patch and reconstruct the most up to date portage image using just this patch without downloading anything else. One drawback here is that if the user does not sync for a long time, he may not find a proper patch for his outdated portage snapshot. Anyway, please try to use gzip compressed bdelta patch in your experiments just for comparison to see if this could theoretically save bandwidth and time when comparing with rsync and zsync.

[danhan]

Hi Community,

I've posted another idea for the "rsync problems" some threads ago.

>What' s against running some of the rsync mirror servers on tcp/443 instead of rsyncs default port.
>TCP/443 is allowed in most environments for "METHOD CONNECT" on the proxies.

https://forums.gentoo.org/viewtopic-t-102461-highlight-rsync+tcp+443.html

--daniel