HowTo: Migrate an Apt-Mirror-generated Ubuntu archive to another mirror source or merge a foreign Apt-Mirror archive into yours
So, you’ve gone and created your very own local Ubuntu mirror using Apt-Mirror, and you’ve come across a situation similar to:
- You’ve decided to change where you update your Apt-Mirror archive from (eg: you’ve changed ISP’s or feel that another source is more reliable than your current one to update from)
- You’re adding another large repository to your Apt-Mirror archive (such as the next version of Ubuntu) and don’t have the quota to download it, so you’re getting a friend to download it for you from their free server using Apt-Mirror (eg: iiNet and Internode customers can access their respective Ubuntu mirrors for free), so you need to be able to merge it with your own Apt-Mirror archive and have it update from your preferred source afterwards.
So how do you do this? Read on.
Migrating your Apt-Mirror archive to update from a new source
This one is really easy. Let’s say you are updating your Ubuntu mirror from Internode, but now want to get your updates from iiNet. To make this happen you need to change the following files:
- Your /etc/apt/mirror.list file needs to be updated to point to the new source, and
- the Apt-Mirror’s record of downloaded files needs to be updated so that it doesn’t waste time trying to re-download the entire mirror again not realising that it’s already got 99% of all the files already, because Apt-Mirror tracks the files it has downloaded by the source URL and filename, not just the filenames themselves.
So let’s go through this.
- Open a terminal load your /etc/apt/mirror.list file into your favourite text editor. In this case I will use the Nano text editor:
$ sudo nano /etc/apt/mirror.list - In your mirror.list file, The lines for updating Ubuntu 32 and 64-bit versions plus source code from Internode can look similar to this:
# Ubuntu 9.10 Karmic Koala 32-bitdeb-i386http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic main restricted universe multiversedeb-i386 http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic-updates main restricted universe multiversedeb-i386 http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic-backports main restricted universe multiversedeb-i386 http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic-security main restricted universe multiversedeb-i386 http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic-proposed main restricted universe multiverse# Ubuntu 9.10 Karmic Koala 64-bitdeb-amd64 http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic main restricted universe multiversedeb-amd64 http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic-updates main restricted universe multiversedeb-amd64 http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic-backports main restricted universe multiversedeb-amd64 http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic-security main restricted universe multiversedeb-amd64 http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic-proposed main restricted universe multiverse# Ubuntu 9.10 Karmic Koala Sourcedeb-src http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic main restricted universe multiversedeb-src http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic-updates main restricted universe multiversedeb-src http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic-backports main restricted universe multiversedeb-src http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic-security main restricted universe multiversedeb-src http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic-proposed main restricted universe multiverse - We need to change the Internode URL to the iiNet URL, so bring up Nano’s search and replace function by pressing CTRL+Backslash (“\”
). - Now type in the text to replace, in this case:
http://mirror.internode.on.net/pub/ubuntu/ubuntu - Press Enter and you’ll be prompted for the text to replace this with. In this case it’s:
http://ftp.iinet.net.au/pub/ubuntu/ - Press Enter and Nano will find the first occurrence of the Internode text string and highlight it for you. If the selection is correct, press “A” on the keyboard to automatically replace “all” occurrences.
- Once the update is done, manually go back and visually verify that all the entries were changed correctly.
- When you’re happy, save your changes by pressing CTRL+X, then “Y” and then Enter.
- Now we need to update the Apt-Mirror record of downloaded files. First, let’s take a backup of the index in case you stuff up. Type in:
$ sudo cp /var/spool/apt-mirror/var/ALL /var/spool/apt-mirror/var/ALL_Backup
NOTE: the filename “ALL” must be in uppercase - Now let’s bring up the original file into the Nano text editor.
$ sudo nano /var/spool/apt-mirror/var/ALL - Depending how large your index file is, there may be a brief delay while Nano opens it up. Once it appears, do the same search and replace as you did in steps 3-6 again. Note: If the editor comes up blank, then you have not opened up the index file – check your path spelling in Step 9 and try again.
- Save your changes by pressing CTRL+X, then “Y” and then Enter.
- Finally, we need to modify the Apt-Mirror’s cache of downloaded files so that its directory structure matches that of the new source. In the case of iiNet, you’ll notice it’s URL has one less ubuntu word in it compared to Internode’s URL, so we’ll need to move some directories to eliminate the extra ubuntu directory.
At the terminal, move the dists and pool directories of the mirrored files one directory back using the commands:$ sudo mv /var/spool/apt-mirror/mirror/mirror.internode.on.net/pub/ubuntu/ubuntu/dists /var/spool/apt-mirror/mirror/mirror.internode.on.net/pub/ubuntu
$ sudo mv /var/spool/apt-mirror/mirror/mirror.internode.on.net/pub/ubuntu/ubuntu/pool /var/spool/apt-mirror/mirror/mirror.internode.on.net/pub/ubuntu - Now rename the mirror.internode.on.net directory to become the name of the iiNet server:
$ sudo mv /var/spool/apt-mirror/mirror/mirror.internode.on.net /var/spool/apt-mirror/mirror/ftp.iinet.net.au - The directory structure now matches iiNet’s server and your ALL file is up to date, so now we can test your changes by launching Apt-Mirror. Launch it manually with:
$ apt-mirror - Watch the output. First Apt-Mirror will download all the repository indexes from the new location and will compare the files presented in those indexes to your local index of downloaded files (the modified ALL file). It will skip all files already listed as being present and will only download new files not listed in your local mirror. You should find Apt-Mirror advises only a small subset of data to download, perhaps only a few megabytes or no more than a gigabyte or two since your last update under the old setup. If you see that Apt-Mirror wants to download some 30GB or more, then you have made an error in changing the URL in the ALL index file or you incorrectly renamed the mirror directories. Press CTRL+C to stop Apt-Mirror, and go check your configuration from Step 10.
$ apt-mirror
Downloading 1080 index files using 5 threads...
Begin time: Wed Dec 9 15:59:23 2009
[5]... [4]... [3]... [2]... [1]... [0]...
End time: Wed Dec 9 16:00:45 2009
Proceed indexes: [SSSSSSSSSSPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPP]
1.7 GiB will be downloaded into archive.
Downloading 998 archive files using 5 threads...
Begin time: Wed Dec 9 16:02:31 2009
[5]... [4]... [3]... [2]... [1]... [0]...
End time: Wed Dec 9 16:54:15 2009
207.4 MiB in 256 files and 1 directories can be freed.
Run /var/spool/apt-mirror/var/clean.sh for this purpose.
$ - You’re done! Pat yourself on the back.
Inserting a foreign Apt-Mirror archive into your own archive
This one is slightly more involved, but is not difficult. In the case of a full Ubuntu Mirror, let’s say you were adding an Ubuntu Karmic mirror archive taken from iiNet’s mirror servers into your own local Apt-Mirror archive that featured only Intrepid and Jaunty, both of which you are updating from Internode’s mirror servers. There are some obstacles we need to overcome such as:
- Continuing to perform future updates for the Karmic repository from Internode rather than iiNet.
- The foreign iiNet Karmic archive contains lots of files that you already have in your own archive – files that are common between all releases of Ubuntu. How do you filter those ones out and only copy the new files?
- Finally, how do you update the Apt-Mirror index file with the potentially thousands of new entries from the foreign archive? How do you avoid duplicate lines potentially confusing Apt-Mirror?
Follow these steps:
- First ensure that you have the full copy of the foreign Apt-Mirror archive supplied on a suitable storage medium. Aside from the mirror directory itself (usually under /var/spool/apt-mirror/mirror), you must have a copy of its /var/spool/apt-mirror/var/ALL file. It does not matter if the foreign mirror is not completely up to date, as Apt-Mirror will catch up with what is missing when you run the next update.
- Let’s prepare your local Apt-Mirror installation for grabbing Ubuntu Karmic from our preferred source first. We need to load up the /etc/apt/mirror.list file into your favourite text editor and add the entries relevant to our new repository that we are mirroring. I will use the Nano text editor for this, but you can use any text editor you like:
$ sudo nano /etc/apt/mirror.list - Now we add the entries relevant to Ubuntu Karmic for Apt-Mirror to use. In this case, I am going to update Ubuntu Karmic from Internode and I will be grabbing both the 32-bit and 64-bit versions plus the source code (reflecting what is already included in the foreign archive on my storage medium, or Apt-Mirror will be doing a LOT of downloading the next time you run it), so I need to add the following entries:
# Ubuntu 9.10 Karmic Koala 32-bit
deb-i386 http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic main restricted universe multiverse
deb-i386 http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic-updates main restricted universe multiverse
deb-i386 http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic-backports main restricted universe multiverse
deb-i386 http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic-security main restricted universe multiverse
deb-i386 http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic-proposed main restricted universe multiverse
# Ubuntu 9.10 Karmic Koala 64-bit
deb-amd64 http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic main restricted universe multiverse
deb-amd64 http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic-updates main restricted universe multiverse
deb-amd64 http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic-backports main restricted universe multiverse
deb-amd64 http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic-security main restricted universe multiverse
deb-amd64 http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic-proposed main restricted universe multiverse
# Ubuntu 9.10 Karmic Koala Source
deb-src http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic main restricted universe multiverse
deb-src http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic-updates main restricted universe multiverse
deb-src http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic-backports main restricted universe multiverse
deb-src http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic-security main restricted universe multiverse
deb-src http://mirror.internode.on.net/pub/ubuntu/ubuntu karmic-proposed main restricted universe multiverse - Save your changes and exit the editor using CTRL+X, then “Y” and then Enter.
- Make a backup copy of the foreign mirror’s /var/spool/apt-mirror/var/ALL file, so you can revert to it if you make a mistake. Call the copy something like ALL_Backup.
- Now open the foreign mirror’s original /var/spool/apt-mirror/var/ALL file into your favourite text editor.
- Use your text editor’s search and replace function (in Nano, press CTRL + Backslash “\”) to replace the URL of each entry in the foreign mirror’s ALL file to the URL of the mirror you will be performing your future updates from. In the case of changing iiNet URLs to Internode URLs, you would replace any occurrence of the text string:
http://ftp.iinet.net.au/pub/ubuntu
…with…http://mirror.internode.on.net/pub/ubuntu/ubuntu - Once updated, save your changes and close your text editor.
- Now we need to merge the modified foreign ALL file into the ALL file from your local Apt-Mirror setup. First up, rename the modified foreign ALL file so we don’t confuse it. For this tutorial, I will assume your foreign mirror is supplied on an external USB hard-drive called “myhdd” and is simply a copy of the foreign system’s /var directory in its entirety. The following will rename the file from ALL to ALL_modified in a terminal:
$ mv /media/myhdd/var/spool/apt-mirror/var/ALL /media/myhdd/var/spool/apt-mirror/var/ALL_modified - Now concatenate the original ALL file and the modified foreign mirror’s ALL_modified file into one new file called ALL_new in your local Apt-Mirror’s var directory. Concatenating alone will result in duplicate lines and we need to sort the file so that any duplicate lines in both the local and foreign ALL files are brought together. We can sort the content of the concatenated files and remove duplicate lines in one hit with:$ sudo cat /var/spool/apt-mirror/var/ALL /media/myhdd/var/spool/apt-mirror/var/ALL_modified | sort | uniq > /var/spool/apt-mirror/var/ALL_new The cat part of the command simply joins the content of /var/spool/apt-mirror/var/ALL and /media/myhdd/var/spool/apt-mirror/var/ALL_modified into one big file, but before it’s written to a physical file, the concatenated data is “piped” using the pipe symbol “|” into the sort command, which sorts the concatenated data into alphabetical order which will group duplicate lines together. But before that resultant output is written anywhere, the sorted data is then piped again into the uniq command which automagically removes all duplicate lines, leaving one unique copy of each line. Finally, we direct the output from uniq using the “>” character into our physical destination file at /var/spool/apt-mirror/var/ALL_new at the end. The sudo command at the start is used simply because only the root and the apt-mirror users can actually write to the /var/spool/apt-mirror/var directory.
Alternatively, we can replace the “| sort | uniq” part with “| sort -u” which does the exact same thing, since the sort command does have it’s own “unique” functionality as well. I’ll leave it up to you which way you’d like to go. - Check your new /var/spool/apt-mirror/var/ALL_new file and you will find it now contains all your local and foreign mirror’s entries in alphabetical order and with no duplicate lines. If you’d like to see how this worked, re-work Step 10 without the sort and uniq commands or the pipe characters and see how it affects the output file. Try adding just the sort or just the uniq command too.
- Now rename your local mirror’s original ALL file because we’re about to replace it with the new one:
$ sudo mv /var/spool/apt-mirror/var/ALL /var/spool/apt-mirror/var/ALL_old - Now rename the new ALL_new file to take the place of the old one:
$ sudo mv /var/spool/apt-mirror/var/ALL_new /var/spool/apt-mirror/var/ALL - Right, that’s the index taken care of. We’re nearly done! Now we only have to merge the foreign mirror’s actual files into your local mirror. Once again, for the purposes of this tutorial I’m going to assume you have them stored on an external USB hard-drive called “myhdd” and is a copy of the foreign system’s entire /var directory, so the path to the foreign mirror’s files will be /media/myhdd/var/spool/apt-mirror/mirror – got that? Let’s change to that directory now in a terminal to save us having to type so much:
$ cd /media/myhdd/var/spool/apt-mirror/mirror - Now, the observant of you may have noticed that Apt-Mirror stores its mirrored files using a directory structure that follows the path of the URL the data is obtained from, so in the case of a mirror from iiNet, there is a directory here called ftp.iinet.net.au. You can see it by using the ls command to list the directory contents:
$ ls -l$
-rw-r--r-- 1 apt-mirror apt-mirror 198599 2009-12-09 10:19 access.log
-rw-r--r-- 1 apt-mirror apt-mirror 544373 2009-12-01 06:45 access.log.1
-rw-r--r-- 1 apt-mirror apt-mirror 1863467 2009-11-03 06:44 access.log.2
-rw-r--r-- 1 apt-mirror apt-mirror 1865334 2009-10-01 06:28 access.log.3
-rw-r--r-- 1 apt-mirror apt-mirror 18152891 2009-09-01 06:42 access.log.4
-rw-r--r-- 1 apt-mirror apt-mirror 6135 2009-12-09 06:46 error.log
-rw-r--r-- 1 apt-mirror apt-mirror 33898 2009-12-01 06:45 error.log.1
-rw-r--r-- 1 apt-mirror apt-mirror 124512 2009-11-03 06:44 error.log.2
-rw-r--r-- 1 apt-mirror apt-mirror 554851 2009-10-01 06:28 error.log.3
-rw-r--r-- 1 apt-mirror apt-mirror 831227 2009-09-01 06:42 error.log.4
drwxr-xr-x 3 apt-mirror apt-mirror 4096 2008-09-11 02:00 ftp.iinet.net.au - We need to modify the foreign directory names and structure to exactly match that of the URL path your local mirror updates from. Starting with the obvious, we need to rename the ftp.iinet.net.au directory to be mirror.internode.on.net with:
$ sudo mv ftp.iinet.net.au mirror.internode.on.net - Next we need to create an extra subdirectory called “ubuntu” because Internode’s URL path is mirror.internode.on.net/pub/ubuntu/ubuntu/ and iiNet’s path is ftp.iinet.net.au/pub/ubuntu/ only:
$ sudo mkdir mirror.internode.on.net/pub/ubuntu/ubuntu - Now we need to move the “dists” and “pool” directories under the first “ubuntu” directory to be under the second “ubuntu” directory:
$ sudo mv mirror.internode.on.net/pub/ubuntu/dists mirror.internode.on.net/pub/ubuntu/ubuntu
$ sudo mv mirror.internode.on.net/pub/ubuntu/pool mirror.internode.on.net/pub/ubuntu/ubuntu - With the directory structure and directory names all amended, we are now ready to merge the foreign mirror’s files into your local mirror. We will do this using RSync. This tool traditionally is used to make backups and is indeed used to keep the official worldwide Ubuntu mirrors up to date 1:1 with the master archive, but in our case we are using it to add the “missing” files in the local mirror with the files from the foreign mirror whilst skipping the files that are already present, which means instead of copying around about 60GB worth of data from the foreign mirror, we’ll only copy a percentage of that instead, saving us time and drive space:
$ sudo rsync -avz --progress /media/myhdd/var/spool/apt-mirror/mirror/mirror.internode.on.net /var/spool/apt-mirror/mirror/ - The “–progress” parameter allows you to see which file is being copied over. You may see a large number of directory names whizz past because those directories don’t have any files that are different between your current Ubuntu Intrepid and Jaunty mirror and the Karmic mirror you are merging. Unfortunately rsync does not provide an all-over progress. It only provides a progress of the file it is currently working on. This procress can take several hours to complete depending on how much data needs to be copied and the speed of your storage medium containing the foreign mirror (which if on a USB HDD can take a looooong time).
- Once RSync has finished, it will give a summary of what was copied. If you were to run the rsync command in Step 16 again, you will see it finish rather quickly because there is no data that has changed or is missing anymore.
- Now we just quickly ensure that all the merged foreign files belong to the Apt-Mirror user with:
$ sudo chown apt-mirror:apt-mirror -R /var/spool/apt-mirror - And now we are ready to try a manual update to see if it all worked. If you now execute the Apt-Mirror application manually, you should now see that it reads in the new repository entries you added into your /etc/apt/mirror.list file in Step 3 and will compare the files presented in those indexes to your local index of downloaded files (the newly modified ALL file). It will skip all files already present and will only download new files not present in your local mirror. You should find Apt-Mirror advises only a small subset of data to download, perhaps only a few megabytes or a gigabyte or two since your last update under the old setup and depending on how old the foreign archive was. If you see that Apt-Mirror wants to download about 30GB or more, then you have made an error in changing the URL in the ALL index file or the renaming of mirror directories. Press CTRL+C to stop Apt-Mirror, and go check your configuration from Step 5.
$ apt-mirror
Downloading 1080 index files using 5 threads...
Begin time: Wed Dec 9 15:59:23 2009
[5]... [4]... [3]... [2]... [1]... [0]...
End time: Wed Dec 9 16:00:45 2009
Proceed indexes: [SSSSSSSSSSPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPP]
1.7 GiB will be downloaded into archive.
Downloading 998 archive files using 5 threads...
Begin time: Wed Dec 9 16:02:31 2009
[5]... [4]... [3]... [2]... [1]... [0]...
End time: Wed Dec 9 16:54:15 2009
207.4 MiB in 256 files and 1 directories can be freed.
Run /var/spool/apt-mirror/var/clean.sh for this purpose.
$ - If all is good, then pat yourself on the back. You’ve successfully merged the foreign repository and it will now update from your preferred ISP’s mirror from now on.
Leave a Reply