| View previous topic :: View next topic |
| Author |
Message |
uriahheep__
n00b
Joined: 03 Feb 2003
Posts: 15
|
 Posted: Sat May 03, 2003 8:48 pm Post subject: MD5 checking on fetched packages using emerge |
 |
|
I've been updating my system with portage in general and KDE in particular. The problem I've been having is when I do an "emerge -fu kde" that it fetches the packages fine, but when I go back later to do the "emerge -u kde" it tells me about halfway through that some of the packages that it fetched earlier don't have a matching MD5 sum.
Since I'm on a connection that isn't always available, it would be nice if there was some way of making emerge check the MD5 sum immediately after fetching a package & deleting/refetching if necessary. It sucks to be out with my computer somewhere far away from a net connection having it build away only to find that a package emerge downloaded earlier is bogus and the emerge can't continue... Grr!
Is there already a way to do this with emerge without it actually having to build the packages first? It seems logical to me that if you're doing a "fetch only" emerge, that it should check the MD5 sum immediately after fetching, instead of waiting to build it...
BTW, kudos to the Gentoo team for making this awesome distro!
Last edited by uriahheep__ on Tue May 06, 2003 6:47 pm; edited 1 time in total |
|
| Back to top |
|
 |
abhishek
Retired Dev
Joined: 28 Jun 2002
Posts: 393
Location: Los Angeles, CA
|
| Posted: Sat May 03, 2003 11:02 pm Post subject: |
|
|
| I don't belive there is an option for emerge to make it just check md5 sums, or fetch and check md5sums. However, if you do ebuild [path to ebuild] unpack this should fetch the sources if you don't have them, check their md5 sums, and unpack the files. Note path to ebuild needs to be the full path to the ebuild(for gaim-0.62 this would be /usr/portage/net-im/gaim/gaim-0.62.ebuild). Or you can always md5sum the file yourself and check to make sure it matches what is listed in the digest(thiswould be in the files/ firectory in the directory where the ebuild is stored(/usr/portage/net-im/gaim/files/digest-gaim-0.62 for the gaim-0.62 again). |
|
| Back to top |
|
|
uriahheep__
n00b
Joined: 03 Feb 2003
Posts: 15
|
| Posted: Sun May 04, 2003 8:44 pm Post subject: |
|
|
| Quote: | | However, if you do ebuild path to ebuild unpack this should fetch the sources if you don't have them, check their md5 sums, and unpack the files. |
This is fine if you just have a single package, but if you're updating (and emerge -u pulls 90+ packages) then it's a pain... Exactly the type of problem that computers are supposed to be good at! I don't necessarily want the sources unpacked, either--at least not until they're ready to be built by emerge.
I suppose there would be a way to write a script that takes the output of emerge -u and then somehow finds the ebuild script, pulls the MD5 sum out of it, computes the MD5 sum of the package listed in the output of emerge -u, compares them and deletes them if they don't match...
Then again, it would probably be easier to modify emerge to do this. Any ideas on where I should look to modify emerge to conform to the behavior I outlined above? If I'm successful with the modification, should I post a patch? |
|
| Back to top |
|
|
Genone
Retired Dev
Joined: 14 Mar 2003
Posts: 9532
Location: beyond the rim
|
| Posted: Mon May 05, 2003 5:55 pm Post subject: |
|
|
Hmm, a script for this would be possible but not so easy, cause you need the filenames, the package name and the portage directory (ok, that's not the hard part  ). Once you get those it's a trivial thing.
For modifying emerge, it's not that hard if you find the right spot. Searching for "fetchonly" should bring you there. Python is rather easy to learn (took me half an hour). When patching something in portage, the most time is spent searching the right functions in portage.py to do something.
If you create a patch for emerge / portage post it to https://bugs.gentoo.org, so the devs (particularly carpaski) will know about it and maybe import it into the official version. |
|
| Back to top |
|
|
uriahheep__
n00b
Joined: 03 Feb 2003
Posts: 15
|
| Posted: Tue May 06, 2003 6:26 pm Post subject: |
|
|
OK, I got a nice patch ready--I'll be posting it to bugs shortly. Just in case it doesn't make it there, here's the patch:
| Code: |
--- portage.py.old 2003-05-02 01:49:39.000000000 -0700
+++ portage.py 2003-05-06 11:11:56.000000000 -0700
@@ -1073,7 +1073,17 @@
else:
#we already have it downloaded, skip.
#if our file is bigger than the recorded size, digestcheck should catch it.
- fetched=2
+ # Unfortunately, for those who don't have a stable/continuous network
+ # connection, this is too late--so check it NOW, while we're connected!
+ mymd5=perform_md5(settings["DISTDIR"]+"/"+myfile)
+ if mymd5 != mydigests[myfile]["md5"]:
+ print "!!! Previously fetched file:",myfile,"MD5 FAILED! Refetching..."
+ os.unlink(settings["DISTDIR"]+"/"+myfile)
+ fetched=0
+ else:
+ print ">>> Previously fetched file:",myfile,"MD5 ;-)"
+ fetched=2
+ break #No need to keep looking for this file, we have it!
else:
#we don't have the digest file, but the file exists. Assume it is fully downloaded.
fetched=2
@@ -1112,8 +1122,19 @@
except:
pass
continue
- fetched=2
- break
+ # File is the correct size--check the MD5 sum for the fetched
+ # file NOW, for those users who don't have a stable/continuous
+ # net connection. This way we have a chance to try to download
+ # from another mirror...
+ mymd5=perform_md5(settings["DISTDIR"]+"/"+myfile)
+ if mymd5 != mydigests[myfile]["md5"]:
+ print "!!! Fetched file:",myfile,"MD5 FAILED! Removing corrupt distfile..."
+ os.unlink(settings["DISTDIR"]+"/"+myfile)
+ fetched=0
+ else:
+ print ">>>",myfile,"MD5 ;-)"
+ fetched=2
+ break
except (OSError,IOError),e:
fetched=0
else:
|
This is patched against 2.0.47-r10, BTW. |
|
| Back to top |
|
|
Genone
Retired Dev
Joined: 14 Mar 2003
Posts: 9532
Location: beyond the rim
|
| Posted: Tue May 06, 2003 9:00 pm Post subject: |
|
|
| Briefly looked over it, will check it out with portage-2.0.48_pre5 soon. |
|
| Back to top |
|
|
Genone
Retired Dev
Joined: 14 Mar 2003
Posts: 9532
Location: beyond the rim
|
| Posted: Tue May 06, 2003 10:33 pm Post subject: |
|
|
OK, applied good against portage-2.0.48_pre5, here is the modified patch. Only line numbers changed and space/tab conversion (of course this is lost after posting). If you go to bugzilla CC me if you like (genone@genone.de).
| Code: | --- portage.py.org 2003-05-07 00:16:52.000000000 +0200
+++ portage.py 2003-05-07 00:25:40.000000000 +0200
@@ -1121,7 +1121,17 @@
else:
#we already have it downloaded, skip.
#if our file is bigger than the recorded size, digestcheck should catch it.
- fetched=2
+ # Unfortunately, for those who don't have a stable/continuous network
+ # connection, this is too late--so check it NOW, while we're connected!
+ mymd5 = perform_md5(settings["DISTDIR"]+"/"+myfile)
+ if mymd5 != mydigests[myfile]["md5"]:
+ print "!!! Previously fetched file:",myfile,"MD5 FAILED! Refetching..."
+ os.unlink(settings["DISTDIR"]+"/"+myfile)
+ fetched=0
+ else:
+ print ">>> Previously fetched file:",myfile,"MD5 ;-)"
+ fetched=2
+ break #No need to keep looking for this file, we have it!
else:
#we don't have the digest file, but the file exists. Assume it is fully downloaded.
fetched=2
@@ -1160,8 +1170,18 @@
except:
pass
continue
- fetched=2
- break
+ # File is the correct size--check the MD5 sum for the fetched
+ # file NOW, for those users who don't have a stable/continuous
+ # net connection. This way we have a chance to try to download
+ # from another mirror...
+ mymd5=perform_md5(settings["DISTDIR"]+"/"+myfile)
+ if mymd5 != mydigests[myfile]["md5"]:
+ print "!!! Fetched file:",myfile,"MD5 FAILED! Removing corrupt distfile..."
+ os.unlink(settings["DISTDIR"]+"/"+myfile)
+ fetched=0
+ else:
+ print ">>>",myfile,"MD5 ;-)"
+ fetched=2
except (OSError,IOError),e:
fetched=0
else:
|
|
|
| Back to top |
|
|
uriahheep__
n00b
Joined: 03 Feb 2003
Posts: 15
|
| Posted: Wed May 07, 2003 3:36 pm Post subject: |
|
|
OK, you're in the loop.  |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Portage & Programming
