[webkit-dev] Metalink: higher availability and self repairing downloads

[Anthony Bryan]

Hi,

This may not be the right mailing list as this would probably deal
more with Safari in general, but I figure it could easily get to
someone interested if I posted here. I couldn't find a better place,
but apologies if there is one. I know WebKit goes back to KDE's KHTML,
and there is Metalink support in KDE 4.0's KGet [1] (I don't know if
it's currently functional tho).

Metalink is an XML format that lists multiple ways
(FTP/HTTP/P2P/rsync) to retrieve files, along with checksums and other
metadata, to increase availability and guarantee integrity in
downloads. It is used by download programs such as download managers,
P2P & FTP programs, but was designed for web browsers.

Around 21 programs have Metalink support. Metalink is frequently used
for large downloads, such as Linux/BSD ISOs (600 megs to multiple
gigs). Projects/sites such as OpenOffice.org, cURL, openSUSE, DOFUS,
Eiffel Software, and others use it.

Some basic features the information in a metalink can be used for:
- Listing multiple mirrors for higher availability.
- Automatic error recovery and repair of corrupted downloads.
- More information about a download for users. [2]
- Add a file or multiple files to a download queue.
- Share a library of files between filesharing applications.
- Simplified download pages.

"Metalink solves the first problem — how to find the most speedy way
to download a file — by grouping different download protocols into one
protocol. This enables Metalink clients to automatically switch
between different mirror servers without explicit user instructions.
Not only multiple server information is described but also meta-data
(e.g., the location of the server, document tags and license)." [3]

An example .metalink:

<?xml version="1.0" encoding="utf-8"?>
<metalink version="3.0" generator="Metalink Editor version 1.0.0"
xmlns="http://www.metalinker.org/">
 <publisher>
   <name>Xi Software</name>
   <url>http://www.xi-soft.com/</url>
 </publisher>
 <identity>Net Transport</identity>
 <version>2.43</version>
 <copyright>2005-2007</copyright>
 <description>Net Transport (NetXfer) is a downloading tool.</description>
 <files>
   <file name="NXSetup_multi.zip">
     <size>2014232</size>
     <os>Windows-x86</os>
     <verification>
       <hash type="md5">18148ed6dc0fc6a48f345579f099d6bd</hash>
       <hash type="sha1">01586b2c0ec5d8e985138204404878f5ecbeef58</hash>
       <pieces type="sha1" length="262144">
         <hash piece="0">ce50319f58e846e3e0c66a9ada9015e601126864</hash>
         <hash piece="1">c9e73e336831a364cedcd889259b7330d48f90fd</hash>
         <hash piece="2">a7f5281c629ea63c791f320db3b0df85fdb01861</hash>
         <hash piece="3">e6cf5571185db0ec79b551222596462db445bdd6</hash>
         <hash piece="4">8982e850d2e3ea005119beea3cc05daca28de474</hash>
         <hash piece="5">a0e7a3a3a46f1ae5fb4eb8529b6fb8ee82fa43a8</hash>
         <hash piece="6">96540cfade1ce9d3846f41510fbed46965b29568</hash>
         <hash piece="7">87108128d4dde5ba62ffec1f1cc3fc292fc298ca</hash>
       </pieces>
     </verification>
     <resources>
       <url type="http"
preference="100">http://www.xi-soft.com/downloads/NXSetup_multi.zip</url>
       <url type="http" location="gb"
preference="100">http://nettransport.abangadek.com/NXSetup_multi.zip</url>
       <url type="http" location="hu"
preference="100">http://www.axitech.hu/nettransport/downloads/NXSetup_multi.zip</url>
       <url type="ftp" location="cn"
preference="100">ftp://nettransport:nettransport@202.107.248.221/NXSetup_multi.zip</url>
     </resources>
   </file>
 </files>
</metalink>

For more information: http://www.metalinker.org and
http://en.wikipedia.org/wiki/Metalink

Thanks,
--
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
 )) Easier, More Reliable, Self Healing Downloads

[1] http://websvn.kde.org/trunk/KDE/kdenetwork/kget/
[2] http://code.downthemall.net/maierman/metaselect4.png
[3] http://www.geospatialsemanticweb.com/2007/02/25/metalink-unifies-internet-downloads