HOWTO: Download ISOs faster and error free with metalink

[twanj]

metalinks are an XML format listing mirrors, p2p locations, checksums, & other stuff in one file used by download managers.

downloads are usually quite fast, and if aria2 is used, they will be error free and repairable.

1. Install aria2
2. http://www.metalinker.org/samples.html#isos has a large number of ISOs, including Gentoo
3. Download the Metalink for the ISO you want with 'aria2c URLtoMetalink'

if you download the .metalink file, you can use it locally with 'aria2c -M metalink'

there's a patch for Bouncer if gentoo wanted to use this for ISO downloads. different sources can be given priority, and downloads can be filtered by location and other things.

here's an example of what one looks like, minus the chunk checksums used for repairs.

Code: Select all

<?xml version="1.0" encoding="utf-8"?>
<metalink version="3.0" generator="Metalink Editor version 0.4.1" xmlns="http://www.metalinker.org/">
  <publisher>
    <name>Gentoo</name>
    <url>http://www.gentoo.org/</url>
  </publisher>
  <version>2007.0</version>
  <files>
    <file name="livecd-i686-installer-2007.0.iso">
      <size>734308352</size>
      <os>Linux-x86</os>
      <verification>
        <hash type="md5">7a21c2f09edd51612fb2cd7984dd3577</hash>
        <hash type="sha1">d876060d2b322b9afffdcd81da2ec8433e489d93</hash>
      </verification>
      <resources>
        <url type="http" location="us" preference="100">http://distro.ibiblio.org/pub/linux/distributions/gentoo/releases/x86/2007.0/livecd/livecd-i686-installer-2007.0.iso</url>
        <url type="http" location="kr" preference="100">http://ftp.gentoo.or.kr/releases/x86/2007.0/livecd/livecd-i686-installer-2007.0.iso</url>
        <url type="http" location="tw" preference="100">http://ftp.isu.edu.tw/pub/Linux/Gentoo/releases/x86/2007.0/livecd/livecd-i686-installer-2007.0.iso</url>
        <url type="http" location="ro" preference="100">http://ftp.roedu.net/pub/mirrors/gentoo.org/releases/x86/2007.0/livecd/livecd-i686-installer-2007.0.iso</url>
        <url type="http" location="uk" preference="100">http://gentoo.blueyonder.co.uk/releases/x86/2007.0/livecd/livecd-i686-installer-2007.0.iso</url>
        <url type="http" location="jp" preference="100">http://gentoo.gg3.net/releases/x86/2007.0/livecd/livecd-i686-installer-2007.0.iso</url>
        <url type="http" location="ca" preference="100">http://gentoo.mirrored.ca/releases/x86/2007.0/livecd/livecd-i686-installer-2007.0.iso</url>
        <url type="http" location="us" preference="100">http://gentoo.mirrors.pair.com/releases/x86/2007.0/livecd/livecd-i686-installer-2007.0.iso</url>
        <url type="http" preference="100">http://gentoo.scphost.com/releases/x86/2007.0/livecd/livecd-i686-installer-2007.0.iso</url>
        <url type="http" location="de" preference="100">http://linux.rz.ruhr-uni-bochum.de/download/gentoo-mirror/releases/x86/2007.0/livecd/livecd-i686-installer-2007.0.iso</url>
        <url type="http" location="jp" preference="100">http://mirror.gentoo.gr.jp/releases/x86/2007.0/livecd/livecd-i686-installer-2007.0.iso</url>
        <url type="http" location="it" preference="100">http://www.die.unipd.it/pub/Linux/distributions/gentoo-sources/releases/x86/2007.0/livecd/livecd-i686-installer-2007.0.iso</url>
        <url type="http" location="us" preference="100">http://www.gtlib.gatech.edu/pub/gentoo/releases/x86/2007.0/livecd/livecd-i686-installer-2007.0.iso</url>
        <url type="http" location="br" preference="100">http://www.las.ic.unicamp.br/pub/gentoo/releases/x86/2007.0/livecd/livecd-i686-installer-2007.0.iso</url>
        <url type="http" location="uk" preference="100">http://www.mirrorservice.org/sites/www.ibiblio.org/gentoo/releases/x86/2007.0/livecd/livecd-i686-installer-2007.0.iso</url>
      </resources>
    </file>
  </files>
</metalink>

[twanj]

DownThemAll! 1.0 Beta now supports metalink downloads from mirrors and full file checksum verification.

[steveL]

Looks interesting. There's no DTD mentioned in your sample file. That means it's not valid XML aiui. Do you believe this file-format is necessary, or could you see it being done with say an ini-style format? (or anythng but XML ;)

[twanj]

A schema is at http://www.metalinker.org/schema/3.0/metalink.xsd & spec/other info at http://www.metalinker.org/

What do you have against XML?

[steveL]

A schema is at http://www.metalinker.org/schema/3.0/metalink.xsd & spec/other info at http://www.metalinker.org/

What do you have against XML?

http://modeemi.fi/~tuomov/b////archives ... T18_33_42/
http://modeemi.fi/~tuomov/b///archives/ ... T11_58_29/
..pretty much explain it. I like Gentoo GuideXML for instance as it is simple and clean, but usually XML files are way too verbose/ content-light for my liking. I blame MS for making it so bloated in their usage of it; it's ended up being something difficult for people to edit.

Most of all i like rst (as a format) and TeX for complex stuff. For configuration I prefer simple plaintext, although I haven't yet seen the one true format (can't find the url for that one but i don't really like it either)

[twanj]

Metalink is aimed at browsers and other programs that already support XML. XML seems to fit fine with what it's doing, it's been working fine for years, so I doubt it will change soon unless there were huge benefits for doing so.

[twanj]

About 20 apps now support downloading w/ metalinks now, including KGet in KDE 4.0, the DTA Firefox extension, aria2 (CLI), wxDownload Fast, GetRight, Net Transport, FDM, Orbit, Retriever, and Speed Download.

There are a bunch of tools for automating the creation of metalinks too.

[kernelOfTruth]

*subscribes*

thanks !
I need to try this the next time I download a livecd