| View previous topic :: View next topic |
| Author |
Message |
jjw
n00b
Joined: 20 Mar 2003
Posts: 59
Location: Austin, TX, US
|
 Posted: Wed Jun 18, 2003 4:56 pm Post subject: user uploads |
 |
|
Sorry for my late reply.
| BradB wrote: | | If you want that to get seriously hammered, submit a HOWTO in tips & tricks, and ask if it can go in the GWN Letter. |
Yes. I want to. I might wait until the next version.
| Death Valley Pete wrote: | | I've been following your progress in the forums (this thread and the other one) for about a month, and I'd like to say again that what you're doing here is incredible. When I suggested sunsite.dk I figured it got buried in the convertation between two people who clearly knew what they were doing, but apparently not. I'm glad I was able to contribute something even if it was something pretty insignificant. |
Hearing from people makes my work much more enjoyable! I'm very glad you suggested sunsite.dk. Uploading is so much easier than on sourceforge!
| Death Valley Pete wrote: | | Anway, I hope I can be forgiven for asking this again but, now that you've got a real non-SF mirror going is there anyway for regular users to submit patches? For example, I made a patch with deltup for gcc-3.2.3-3.3 (it's about 8 MB) that I'd be willing to spend an hour uploading for the cause. |
That's very generous of you. I'd love it if people were able to do this, but unfortunately the sunsite.dk ftp site doesn't allow anonymous access.
| Death Valley Pete wrote: | | Looking at the files list on the server it seems that the naming convention has changed but I won't ask any dumb questions about that until I've read the new manual. |
That particular piece of info is in my head and nowhere else! I have designed a system which allows multiple naming schemes. Currently the patch filenames look like stripextension(<previous package filename>).[fb]dtu. I will probably add another filename format for the cases where this is inadequate.
| Death Valley Pete wrote: | | I think we'll just have to wait and see what the man who knows what's going on thinks. |
Considering that we can't do it with sunsite, we'll probably get the resources for automatic patch making before this even beomes a possibility. I'll ask the sunsite guys if there's any way it could be done. I think it would be a big advantage. There are some problems though... Besides the technical issues involved, there is the problem of people submitting corrupt patches. If such a thing does come about, it would have to be disabled by default, because it would be a major nuisance for most people.
Thanks for your comments. Gimp users: You should have the first real benefit of this system when gimp-1.2.5 is unmasked. |
|
| Back to top |
|
 |
BradB
Apprentice
Joined: 18 Jun 2002
Posts: 190
Location: Christchurch NZ
|
| Posted: Wed Jun 18, 2003 7:58 pm Post subject: |
|
|
I did this last night & was going to post about the ACCEPT_KEYWORDS thing.
Also, is the /usr/portage/profiles...dtu (can't remember the rest) the test *.dtu file - is that the list of deltups that it can fetch?
Is it possible to set tell efetch to use another fetch command, ie
EFETCHCOMMAND=`prozilla ${URI}`
Nice work guys.
Brad |
|
| Back to top |
|
|
jjw
n00b
Joined: 20 Mar 2003
Posts: 59
Location: Austin, TX, US
|
| Posted: Thu Jun 19, 2003 3:18 am Post subject: |
|
|
| BradB wrote: | | Also, is the /usr/portage/profiles...dtu (can't remember the rest) the test *.dtu file - is that the list of deltups that it can fetch? |
Yes. dtu.list.aux. It refreshes the file every time you do an emerge sync. It allows many special features with a minimal amount of overhead.
| BradB wrote: | Is it possible to set tell efetch to use another fetch command, ie
EFETCHCOMMAND=`prozilla ${URI}` |
Not currently, but I've been planning to put this feature into the next version. |
|
| Back to top |
|
|
Death Valley Pete
n00b
Joined: 25 Mar 2003
Posts: 49
Location: The Inland Empire
|
| Posted: Thu Jun 19, 2003 3:55 am Post subject: |
|
|
I've got my first real-world success story now. Emerge -uf world invoked deltup this morning in updating gimp 1.2.3 to 1.2.4. It was the first time deltup's been used on my system except for me messing around with it. It cut the download time by about 45 minutes for me - which is about 40 minutes more than it takes to download, install, and configure deltup on a dialup modem. If that doesn't get people's attention then I don't know what will. When this gets into the GWN it'll be bigger than breakmygentoo.net last week.
Anway, keep us posted, especially if by some outrageous twist of fortune the folks at sunsite can work something out for us (though I'm not holding my breath).
_________________
<instert pithy statement here> |
|
| Back to top |
|
|
BradB
Apprentice
Joined: 18 Jun 2002
Posts: 190
Location: Christchurch NZ
|
| Posted: Thu Jun 19, 2003 8:21 pm Post subject: |
|
|
I'll try the gimp tonight - I looked through the DTU file the other night & couldn't see anything I wanted updated! 
I'd love to see this in portage official.
Brad |
|
| Back to top |
|
|
jjw
n00b
Joined: 20 Mar 2003
Posts: 59
Location: Austin, TX, US
|
| Posted: Fri Jun 20, 2003 5:14 am Post subject: Another version |
|
|
Hi guys,
I released 0.3.7_pre2, which checks several environment variables for customizing deltup. You can find out more by reading the GENTOO doc file in the tarball. I've added support for: user specified fetch programs, looking in several dirs for old packages, and placing the dtu.list.aux file in another location.
| BradB wrote: | | I looked through the DTU file the other night & couldn't see anything I wanted updated! |
Almost all the patches were just for testing purposes. I'm going to try to upload patches for every updated package that I download. I should able to provide you with patches for at least half of your packages. 
---JJW |
|
| Back to top |
|
|
Death Valley Pete
n00b
Joined: 25 Mar 2003
Posts: 49
Location: The Inland Empire
|
| Posted: Fri Jun 20, 2003 3:26 pm Post subject: |
|
|
BIG, HUGE RED FLAG!!!!!!
Now that I've got everybody's attention, there's a problem with deltup 0.3.7_pre2.
THE PROBLEM: With 0.3.7_pre2 it wants to download things that should go into /usr/portage/distfies into /usr/portage/profiles. This breaks portage: it keeps trying to fetch things over and over, putting them in the wrong place
THE FIX: I'm not sure this is entirely correct and I know that jjw will have something to say about this, but this is what worked for me: edit the file /usr/bin/efetch (nano, vi, whatever you want to use). Go to the VERY LAST LINE: the end of the line will say $PORTDIR/profiles - change that to read $PORTDIR/distfiles
Reverting to deltup 0.3.6.1 also works
When I synced, at least, 0.3.7_pre2 wasn't in the portage tree, I had to get the ebuild from sourceforge. Don't know if that'll change.
FWIW, this will probably be my last post around here for most of a month. I've got several obligations to fill (some of which are over 1000 miles away) and Gentoo will fall a long way down on my list of priorities. So if this was confusing just figure it out. 
_________________
<instert pithy statement here> |
|
| Back to top |
|
|
jjw
n00b
Joined: 20 Mar 2003
Posts: 59
Location: Austin, TX, US
|
| Posted: Fri Jun 20, 2003 6:05 pm Post subject: Big bad bug |
|
|
| Death Valley Pete wrote: | | THE PROBLEM: With 0.3.7_pre2 it wants to download things that should go into /usr/portage/distfies into /usr/portage/profiles. This breaks portage: it keeps trying to fetch things over and over, putting them in the wrong place |
You are indeed right. Thanks for mentioning this. I have fixed the problem and will release 0.3.7 final today after I do some more tests. |
|
| Back to top |
|
|
jjw
n00b
Joined: 20 Mar 2003
Posts: 59
Location: Austin, TX, US
|
| Posted: Thu Jun 26, 2003 6:20 am Post subject: New bdelta algorithm |
|
|
Hi folks,
I wanted to let you know that I released an alpha release of my new delta algorithm. This code is not stable yet, so don't use it for any practical purposes! I'm getting some unreal results. Even in this early stage of development I'm seeing 50% gain for some packages! You can download the sample code from the deltup website. Its name is "bdelta".
God bless,
---JJW |
|
| Back to top |
|
|
Death Valley Pete
n00b
Joined: 25 Mar 2003
Posts: 49
Location: The Inland Empire
|
| Posted: Thu Jul 24, 2003 6:00 am Post subject: |
|
|
Hi jjw,
I just noticed that Deltup 0.4.0 was released. From the changelog I'm assuming that this fixes the problems with bad patches that's been messing things up lately. Any news for all of us (at least, I'm pretty sure it's more than just me who's following this) on the status of the project? I noticed that there's a new release of bdelta on sourceforge - is it safe to try and if so how? What's the difference between bdelta and the old algorithm (if such a difference can be explained in any meaningful fashion to somebody with no knowledge of programming).
Anyway, good luck and keep us (me? ) posted.
_________________
<instert pithy statement here> |
|
| Back to top |
|
|
ferringb
Retired Dev
Joined: 03 Apr 2003
Posts: 357
|
| Posted: Thu Jul 24, 2003 6:20 am Post subject: |
|
|
| Death Valley Pete wrote: | | I just noticed that Deltup 0.4.0 was released. From the changelog I'm assuming that this fixes the problems with bad patches that's been messing things up lately. |
Curious, what problems were there? I haven't followed deltup (or this thread) all that closely as of late...
| Death Valley Pete wrote: | | is [bdelta] safe to try and if so how? |
Just download it and compile it- I've been testing it against my own implementation and haven't had any particular problems w/ it.
Of course, I didn't write it, so I'd pester jjw for his opinion...
The only 'dead-in-the-tracks' fault I know with it is getting it to compile on OSX due to the shipped gcc lacking the -shared option (note to jjw, might want to check into that)...
| Death Valley Pete wrote: | | What's the difference between bdelta and the old algorithm (if such a difference can be explained in any meaningful fashion to somebody with no knowledge of programming). |
Pretty easy- old deltup was using the ancillary xdelta program, new uses the included bdelta. Again, jjw could answer better (cause he wrote it), but bdelta from the looks of it is a greedy alg aproximation- basically it is more aware of the reference file compared to xdelta. Eg, it can find better matches for copies, hence produce a more optimal set of matches.
That and the format he uses compresses surprisingly well... |
|
| Back to top |
|
|
Death Valley Pete
n00b
Joined: 25 Mar 2003
Posts: 49
Location: The Inland Empire
|
| Posted: Thu Jul 24, 2003 4:44 pm Post subject: |
|
|
| ferringb wrote: | | Curious, what problems were there? I haven't followed deltup (or this thread) all that closely as of late... |
Actually neither have I until just this week. There seemed to be a problem with certain dtus (the only one I can specifically remember at the moment was gawk, but there were several others) producing incorrect MD5s, which naturally precipitated a need to download the whole file.
So does bdelta use the same dtus as xdelta?
Anyway, hopefully this thread will catch jjw's attention and he'll give us the scoop. I guess I'll have to start playing around with deltup again.
Edit: a couple minutes' experimentation suggests that yes, this does fix the problems that existed in 0.3.7.
_________________
<instert pithy statement here> |
|
| Back to top |
|
|
jjw
n00b
Joined: 20 Mar 2003
Posts: 59
Location: Austin, TX, US
|
| Posted: Thu Jul 24, 2003 9:20 pm Post subject: Deltup status |
|
|
| Death Valley Pete wrote: | | Any news for all of us (at least, I'm pretty sure it's more than just me who's following this) on the status of the project? |
I've written a GLEP. I hope it will start some wheels rolling. You can find it on http://deltup.sourceforge.net/glep.html
| Death Valley Pete wrote: | | I noticed that there's a new release of bdelta on sourceforge - is it safe to try and if so how? |
Deltup uses xdelta by default. To make a dtu based on bdelta you need to pass deltup the -j option. I made bdelta a separate package so I could release it without having to release deltup too. Bdelta will become a dependency of deltup just like xdelta is. I won't be making any bdelta patches until it becomes part of portage, otherwise users would have to manually install it.
| Death Valley Pete wrote: | | What's the difference between bdelta and the old algorithm (if such a difference can be explained in any meaningful fashion to somebody with no knowledge of programming). |
I think the biggest difference is that bdelta does multiple passes over the data. It uses only a fraction of the memory and has a library interface for anyone who wants to test their own delta format. The bdelta output format is quite good too and should have been thought up long ago.
| ferringb wrote: | | The only 'dead-in-the-tracks' fault I know with it is getting it to compile on OSX due to the shipped gcc lacking the -shared option (note to jjw, might want to check into that)... |
Thanks. I didn't know OSX gcc needed different flags.
| Death Valley Pete wrote: | | So does bdelta use the same dtus as xdelta? |
The dtu format is really just a wrapper around the (b/x)delta formats. Other formats could also be added easily, but the user installing the patch must have the necessary patch program installed.
| Death Valley Pete wrote: | | There seemed to be a problem with certain dtus (the only one I can specifically remember at the moment was gawk, but there were several others) producing incorrect MD5s, which naturally precipitated a need to download the whole file. |
Yes, it was a gzip timestamp problem. I had to break compatibility with older patches of gzipped packages, but now it's fixed permanently. |
|
| Back to top |
|
|
ferringb
Retired Dev
Joined: 03 Apr 2003
Posts: 357
|
| Posted: Fri Jul 25, 2003 2:08 pm Post subject: Re: Deltup status |
|
|
| jjw wrote: | | Thanks. I didn't know OSX gcc needed different flags. |
Bit more then just different flags... requires a fair tweaking of gcc flags for the lib's source, let alone the creation of the lib.
I'd suggest checking into libtool, that's what I've been using (alongside autoconf and automake).
| jjw wrote: | | The dtu format is really just a wrapper around the (b/x)delta formats. Other formats could also be added easily, but the user installing the patch must have the necessary patch program installed. |
Curious, you gotten around to creating your intermediate format you had mentioned in the past? I'd be curious how your alg implementation fairs format-agnostic.
| jjw wrote: | | Yes, it was a gzip timestamp problem. I had to break compatibility with older patches of gzipped packages, but now it's fixed permanently. |
? |
|
| Back to top |
|
|
ferringb
Retired Dev
Joined: 03 Apr 2003
Posts: 357
|
| Posted: Fri Jul 25, 2003 2:24 pm Post subject: Re: Deltup status |
|
|
| jjw wrote: | | The bdelta output format is quite good too and should have been thought up long ago. |
Why do you say that? Hunting through the format definition/code, it looks like a subset of how vcdiff is structured, w/ the exemption of the copy/add switching assumption.
Related to that, personally I usually find that doesn't hold- I usually end up w/ something on the order of 2 copies to 1 add, which could be worked around by inserting a zero length add.
Also, I'd be curious of your hash collision rate- adler32 starts to peter out a bit when the seed length gets smaller, which can be dealt w/ but is annoying. |
|
| Back to top |
|
|
jjw
n00b
Joined: 20 Mar 2003
Posts: 59
Location: Austin, TX, US
|
| Posted: Fri Jul 25, 2003 6:05 pm Post subject: Re: Deltup status |
|
|
| ferringb wrote: | | jjw wrote: | | The bdelta output format is quite good too and should have been thought up long ago. |
Why do you say that? Hunting through the format definition/code, it looks like a subset of how vcdiff is structured, w/ the exemption of the copy/add switching assumption. |
Why do formats have to have so much baggage? Usually baggage ends up to be garbage and in the end the simple formats win out. I'm not convinced that even a thorough implementation of vcdiff could always do better than raw data + bzip2. On the other hand, I'm sure a lot of thought was put into vcdiff, and I shouldn't judge it hastily. It would be a good experiment.
| ferringb wrote: | | Related to that, personally I usually find that doesn't hold- I usually end up w/ something on the order of 2 copies to 1 add, which could be worked around by inserting a zero length add. |
Instead of an "add" you can think of it as a relative seek before a match. The bdelta format is nothing more than a list of matches. It is quite logical to say "the next match starts 0 bytes from the end of the last match". An alternative is to have another list with the number of adjacent matches but I'm quite happy (for now anyway) with the compressor's ability to root out those things for itself.
| ferringb wrote: | | Also, I'd be curious of your hash collision rate- adler32 starts to peter out a bit when the seed length gets smaller, which can be dealt w/ but is annoying. |
Yes, adler32 isn't practical for small blocks. I invented my own checksum which is pretty strong and quite similar to adler, but because of memory limitations the hash reduces the strength to around 16-24 bits (I have a dynamic hash size based on the number of checksums), so the hash collision rate varies wildly depending on the size of the dataset/hashsize. |
|
| Back to top |
|
|
ferringb
Retired Dev
Joined: 03 Apr 2003
Posts: 357
|
| Posted: Fri Jul 25, 2003 6:29 pm Post subject: Re: Deltup status |
|
|
| jjw wrote: | | Why do formats have to have so much baggage? Usually baggage ends up to be garbage and in the end the simple formats win out. I'm not convinced that even a thorough implementation of vcdiff could always do better than raw data + bzip2. On the other hand, I'm sure a lot of thought was put into vcdiff, and I shouldn't judge it hastily. It would be a good experiment. |
Eh, to some degree I'd agree on vcdiff having some extra crap thrown in- the thing about it I've found is that if you want to devolve to just specifying a set of matches and adds, you can do it. It's an imposing format, but it's open enough that it encompasses a lot- whether copy caching, offset caching, or just a simple list of matches and adds.
| jjw wrote: | | Instead of an "add" you can think of it as a relative seek before a match. The bdelta format is nothing more than a list of matches. It is quite logical to say "the next match starts 0 bytes from the end of the last match". An alternative is to have another list with the number of adjacent matches but I'm quite happy (for now anyway) with the compressor's ability to root out those things for itself. |
I'm assuming for added data (eg non-matched data), you're specifying a match against the data in the patch? Your format file is a bit vague... Note also I'm talking the traditional copy/add methods, I'm assuming you're attempting something slightly different from that...
| jjw wrote: | | ferringb wrote: | | Also, I'd be curious of your hash collision rate- adler32 starts to peter out a bit when the seed length gets smaller, which can be dealt w/ but is annoying. |
Yes, adler32 isn't practical for small blocks. I invented my own checksum which is pretty strong and quite similar to adler, but because of memory limitations the hash reduces the strength to around 16-24 bits (I have a dynamic hash size based on the number of checksums), so the hash collision rate varies wildly depending on the size of the dataset/hashsize. |
2/3 bytes? Ouch. In my experience attempting to multiply the accumulator doesn't work incredibly well, regardless if it's by a prime or not.
I've been using a simple variation of adler32, basically using the char to look up a primes. Works surprisingly well, in my experience bad collisions drop pretty heavily for a 16 char seed, something like down to a third of what non-prime results in. I haven't hunted for a better array of primes (just did a quick randomization of it), but works fairly well. |
|
| Back to top |
|
|
Death Valley Pete
n00b
Joined: 25 Mar 2003
Posts: 49
Location: The Inland Empire
|
| Posted: Fri Jul 25, 2003 9:26 pm Post subject: |
|
|
Not to interrupt, but one trivial problem: when I emerge bdelta from the SF ebuild it didn't install libbdelta.so to /usr/lib. I've no idea why (I haven't taken the time to learn ebuilds yet) but I took care of it by just unpacking the source, running 'make' and manually copying the file over. It's no big deal though. I'm experimenting with it now. Keep up the good work.
_________________
<instert pithy statement here> |
|
| Back to top |
|
|
jjw
n00b
Joined: 20 Mar 2003
Posts: 59
Location: Austin, TX, US
|
| Posted: Fri Jul 25, 2003 9:26 pm Post subject: Re: Deltup status |
|
|
| ferringb wrote: | | jjw wrote: | | Instead of an "add" you can think of it as a relative seek before a match. The bdelta format is nothing more than a list of matches. It is quite logical to say "the next match starts 0 bytes from the end of the last match". An alternative is to have another list with the number of adjacent matches but I'm quite happy (for now anyway) with the compressor's ability to root out those things for itself. |
I'm assuming for added data (eg non-matched data), you're specifying a match against the data in the patch? Your format file is a bit vague... Note also I'm talking the traditional copy/add methods, I'm assuming you're attempting something slightly different from that... |
It's exactly the same as copy/add, but I'm looking at it from a different angle. Here's the same format from your perspective:
| Code: | for (number of matches) {
relative reference file location
number of bytes to add (done before copy)
number of bytes to copy
} |
| ferringb wrote: | | jjw wrote: | | ferringb wrote: | | Also, I'd be curious of your hash collision rate- adler32 starts to peter out a bit when the seed length gets smaller, which can be dealt w/ but is annoying. |
Yes, adler32 isn't practical for small blocks. I invented my own checksum which is pretty strong and quite similar to adler, but because of memory limitations the hash reduces the strength to around 16-24 bits (I have a dynamic hash size based on the number of checksums), so the hash collision rate varies wildly depending on the size of the dataset/hashsize. |
2/3 bytes? Ouch. In my experience attempting to multiply the accumulator doesn't work incredibly well, regardless if it's by a prime or not. |
The checksum isn't 2-3 bytes. The index to the hash lookup table is. According to the rsync paper rsync uses only 16bits. I think my method of multiplying the accumulator works excellently with small block (seed) sizes |
|
| Back to top |
|
|
ferringb
Retired Dev
Joined: 03 Apr 2003
Posts: 357
|
| Posted: Thu Aug 28, 2003 8:50 pm Post subject: Re: Deltup status |
|
|
I'll comment on the previous post whenever I get a chance/remember to, but in adding support for your format into my setup, noticed a couple of things-
First off, you're method for storing negative relative references is actually tied to the compiler/host- gcc uses two's complement for it's numbers, so as long as bdelta is compiled via gcc yoursafe. On a different compiler (one that doesn't enforce two's regardless of the host), you'll end up w/ a different number stored.
Got bored, so I modified your source to explicitly store the negative number in two's complement- also modified your source (as of 0.1.0) to enable variable intsize, although I personally would not store nummatche and size/size2 in intsize- it's basically a gurantee they will be the largest numbers, hence the intsize will never be smaller then the src/ver file size. Worth tweaking.
Either way, so follows a unified diff w/ an addition of 2 functions for reading/writing variable byte size, adjustment of the text file Format so that it actually makes sense, modification to fix two's complement issue, and a quick tweak to enable intsize- why didn't you originally enable intsize btw, it's a pretty straightforward thing...
| Code: |
diff -ur bdelta-0.1.0/Format bdelta-dev/Format
--- bdelta-0.1.0/Format Wed Aug 27 18:07:34 2003
+++ bdelta-dev/Format Wed Aug 27 18:11:28 2003
@@ -6,7 +6,7 @@
unsigned file 2 size
unsigned number of matches
for (number of matches) {
- unsigned match relative location 1 (this value can represent a negative)
- unsigned match relative location 2
- unsigned match size
+ 2's complement, copy offset relative to last offset.
+ unsigned pre-copy add len
+ unsigned copy len
}
diff -ur bdelta-0.1.0/bdelta.cpp bdelta-dev/bdelta.cpp
--- bdelta-0.1.0/bdelta.cpp Wed Aug 27 18:07:34 2003
+++ bdelta-dev/bdelta.cpp Wed Aug 27 18:08:40 2003
@@ -98,26 +98,40 @@
fwrite(magic, 1, 3, fout);
unsigned short version = 1;
write_word(fout, version);
- unsigned char intsize = 4;
- fwrite(&intsize, 1, 1, fout);
- write_dword(fout, size);
- write_dword(fout, size2);
- write_dword(fout, nummatches);
-
+ unsigned long max_num=0;
+ unsigned char intsize = 0;
+#define MAX(x,y) ((x) > (y) ? (x) : (y))
unsigned lastp1 = 0,
lastp2 = 0;
for (int i = 0; i < nummatches; ++i) {
unsigned p1, p2, num;
bdelta_getMatch(b, i, &p1, &p2, &num);
-// printf("%*x, %*x, %*x, %*x\n", 10, p1, 10, p2, 10, num, 10, p2-lastp2);
- copyloc1[i] = p1-lastp1;
- write_dword(fout, copyloc1[i]);
+ if(lastp1 > p1)
+ copyloc1[i] = p1 + (~lastp1) + 1;
+ else
+ copyloc1[i] = p1-lastp1;
copyloc2[i] = p2-lastp2;
- write_dword(fout, copyloc2[i]);
copynum[i] = num;
- write_dword(fout, copynum[i]);
lastp1=p1+num;
lastp2=p2+num;
+ }
+ max_num = MAX(nummatches, MAX(size, size2));
+ if(max_num <= 0x7f)
+ intsize=1;
+ else if(max_num <= 0x7fff)
+ intsize=2;
+ else if(max_num <= 0x7fffff)
+ intsize=3;
+ else
+ intsize=4;
+ fwrite(&intsize, 1, 1, fout);
+ write_bytes(fout, size, intsize);
+ write_bytes(fout, size2, intsize);
+ write_bytes(fout, nummatches, intsize);
+ for (int i = 0; i < nummatches; ++i) {
+ write_bytes(fout, copyloc1[i], intsize);
+ write_bytes(fout, copyloc2[i], intsize);
+ write_bytes(fout, copynum[i], intsize);
}
if (size2!=lastp2) {
copyloc1[nummatches]=0; copynum[nummatches]=0;
diff -ur bdelta-0.1.0/bpatch.cpp bdelta-dev/bpatch.cpp
--- bdelta-0.1.0/bpatch.cpp Wed Aug 27 18:07:34 2003
+++ bdelta-dev/bpatch.cpp Wed Aug 27 18:07:23 2003
@@ -57,23 +57,27 @@
}
char intsize;
fread(&intsize, 1, 1, patchfile);
- if (intsize!=4) {
- printf("unsupported file pointer size\n");
- return 1;
+ unsigned long or_mask = 0;
+ switch(intsize) {
+ case 1: or_mask |= 0xff00;
+ case 2: or_mask |= 0xff0000;
+ case 3: or_mask |= 0xff000000;
}
- unsigned size1 = read_dword(patchfile),
- size2 = read_dword(patchfile);
+ unsigned size1 = read_bytes(patchfile,intsize),
+ size2 = read_bytes(patchfile,intsize);
- unsigned nummatches = read_dword(patchfile);
+ unsigned nummatches = read_bytes(patchfile,intsize);
unsigned *copyloc1 = new unsigned[nummatches+1],
*copyloc2 = new unsigned[nummatches+1],
*copynum = new unsigned[nummatches+1];
for (int i = 0; i < nummatches; ++i) {
- copyloc1[i] = read_dword(patchfile);
- copyloc2[i] = read_dword(patchfile);
- copynum[i] = read_dword(patchfile);
+ copyloc1[i] = read_bytes(patchfile,intsize);
+ if(copyloc1[i] > (0x10 << ((intsize -1) * 8)) )
+ copyloc1[i] |= or_mask;
+ copyloc2[i] = read_bytes(patchfile,intsize);
+ copynum[i] = read_bytes(patchfile,intsize);
size2-=copyloc2[i]+copynum[i];
}
if (size2) {
diff -ur bdelta-0.1.0/file.h bdelta-dev/file.h
--- bdelta-0.1.0/file.h Wed Aug 27 18:07:34 2003
+++ bdelta-dev/file.h Wed Aug 27 18:07:23 2003
@@ -12,7 +12,20 @@
*
* Author: John Whitney <jjw@linuxmail.org>
*/
-
+
+unsigned long
+read_bytes(FILE *f, unsigned int len)
+{
+ unsigned char b;
+ unsigned long num=0;
+ unsigned int x=0;
+ for(; x < len; x++) {
+ fread(&b, 1, 1, f);
+ num |= (b << (8 * x));
+ }
+ return num;
+}
+
unsigned read_word(FILE *f) {
unsigned char b, b2;
fread(&b, 1, 1, f);
@@ -24,6 +37,16 @@
return (read_word(f)<<16)+low;
}
+void
+write_bytes(FILE *f, unsigned long num, unsigned int bytes)
+{
+ unsigned char b;
+ unsigned int x=0;
+ for(; x < bytes; x++) {
+ b = ((num >> (8 * x)) & 0xff);
+ fwrite(&b, 1, 1, f);
+ }
+}
void write_word(FILE *f, unsigned number) {
unsigned char b = number&255;
fwrite(&b, 1, 1, f);
|
In making the modifications, I also verified that they work... although again, I would remove size/size2/nummatches from being stored intsize, mainly since I doubt most offsets and copy/add len's will ever need as many bytes as storing the file size would need. |
|
| Back to top |
|
|
Death Valley Pete
n00b
Joined: 25 Mar 2003
Posts: 49
Location: The Inland Empire
|
| Posted: Thu Sep 04, 2003 3:35 pm Post subject: |
|
|
Hi jjw,
I've been noticing that the patch repository at sunsite hasn't been updated in three weeks or so. After downloading several full kernel sources in the -test series, a full gcc version (23 MB when a 1.5 MB patch would have worked), and now the latest binutils (downloading at this moment and with 25 minutes remaining), I'm remembering the bad old days before deltup existed.
I don't know your status at the moment, but I have a modest suggestion: if you advertise for a few (four to seven, maybe?) people to make and submit patches, everybody would win. I'd certainly volunteer. I'm on dialup now, but I go to the Uni. in a few weeks, so I could contribute something (and even if that weren't the case I'd be willing to help out). I don't (and I daresay nobody does) have the time to cover everything, but if you assigned different people to different packages (I'm envisioning maybe the openoffice series, the KDE series, the GNOME series, the compiler/binutils series, and the mozilla/misc. series), then this would be a lot easier for everybody. I remember you saying that sunsite doesn't allow anonymous uploads, but would they/you consent to <10 trusted maintainers?
Well, whatever comes of this, deltup is an awesome piece of software, and I appreciate the time you took to write it. Vaya con dios.
_________________
<instert pithy statement here> |
|
| Back to top |
|
|
ferringb
Retired Dev
Joined: 03 Apr 2003
Posts: 357
|
| Posted: Thu Sep 04, 2003 9:18 pm Post subject: valgrind and bdelta |
|
|
Something worth checking into/fixing, try running bdelta w/ valgrind (it's a memory debugger)- while bdelta doesn't appear to leak memory (aside from a possible 1800 bytes or so), it does seem to tick of valgrind in it's allocating/freeing.
Worth trying. |
|
| Back to top |
|
|
jjw
n00b
Joined: 20 Mar 2003
Posts: 59
Location: Austin, TX, US
|
| Posted: Mon Sep 08, 2003 4:26 am Post subject: Time is priceless |
|
|
Hi Guys,gentoo-supported machine containing all the distfiles/patchfiles and generate the patch right there. That way we wouldn't have to download any files and the dtu description file could be automatically generated. Let's bug seemant about it
It's good to hear from you again, Brian. I've actually had a new version of both
Sorry I didn't respond to your comments right away. I've been so busy I haven't had time even to update my own system, which is why I haven't been uploading the patches. I apologize. I like your suggestion for trusted patch makers DVP - it would be nice if someone else could fill in for me when I can't find the time. It would be X times better if we could ssh into a gentoo-supported machine containing all the distfiles/patchfiles and generate the patch right there. That way we wouldn't have to download any files and the dtu description file could be automatically generated. Let's bug seemant about it
It's good to hear from you again, Brian. I've actually had a new version of both bdelta and deltup for a while now, but I haven't tested them properly yet. I see you have a problem with the way I implemented two's-complement numbers. Notice how I declared the variables unsigned so they don't get interpreted by the compiler and will use simple overflow rules instead. Please let me know if I'm wrong about that. The only reason I didn't add support for different byte sizes is because I wanted to make the first release simple and bugfree. I'll try to get it into the next version. It would be good for doing deltas on a large number of small files.
Thanks for the support. |
|
| Back to top |
|
|
ferringb
Retired Dev
Joined: 03 Apr 2003
Posts: 357
|
| Posted: Mon Sep 08, 2003 9:18 pm Post subject: Re: Time is priceless |
|
|
| jjw wrote: | | I've actually had a new version of both |
Format change at all? You may have gathered I added support for your format into my setup. Curious, you actually looked at xdelta's 1.14 format? I've pegged down everything in the file (can do reconstruction), *except* for 4 bytes which look like they're the memory requirement for representing the patch in memory. That one has thrown me for a loop, since mainly now to get full write support, I need to figure out the exact size of the patch commands in memory for xdelta.
I had fired off an email to xdelta's author asking about it, but haven't had any responses yet. You attempted supporting xdelta format yet? Read isn't bad once you figure out how he stores his uints, likewise, write ain't bad exempting that weird 4 bytes...
| jjw wrote: | | I see you have a problem with the way I implemented two's-complement numbers. Notice how I declared the variables unsigned so they don't get interpreted by the compiler and will use simple overflow rules instead. Please let me know if I'm wrong about that. |
May be right, not really sure- I haven't tried compiling it on a different compiler. I know for signed int's, gcc is 2's complement always- for unsigned int, may still result in the same value. None the less, if you're going to use two's complement might as well make it explicit, and not leave the possibility of a something weird popping up. One thing that may be an issue is if as you say, it resorts to just using the host's overflow rules- rules which can differ system to system.
Either way, writing it explicitly makes it easier to identify how negative numbers are stored.
| jjw wrote: | | The only reason I didn't add support for different byte sizes is because I wanted to make the first release simple and bugfree. I'll try to get it into the next version. It would be good for doing deltas on a large number of small files. |
Well, like I mentioned, I'd personally limit intsize to offsets and lens, possibly a different intsize for each. If you have it so that the file1/file2 is intsize, you'll loose the benefit of intsize rather quickly... |
|
| Back to top |
|
|
Ssl
Apprentice
Joined: 21 Feb 2003
Posts: 178
Location: Serbia
|
| Posted: Sat Sep 20, 2003 9:32 pm Post subject: |
|
|
Hello John, Brian and all other supporters (like me:))
Just want to ask are there any plans to put gnome fdtus when gnome 2.4.1 arrive in next month or two. (2.4.0-->2.4.1).
What happend with announce of establishing one test gentoo diff repository with ssh access? Any good news ? Any plans?
Thank you for your work and answer.
Ssl |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Portage & Programming
