HowTo: Encode a Blu-ray rip into a smaller format without losing quality

Those of you who archive or backup their Blu-ray movie media to hard-drive will already be aware that the average movie comes out at a good 25GB. Some of the bigger titles top out at around 40GB or more. This eats up an awful lot of disk space.

Blu-ray titles are already compressed down using the MPEG2 codec, and quality pundits will abhor the idea of re-compressing the title again for fear of losing image and audio quality. Certainly if you go down the Xvid route, you will definitely lose image quality, but as per my previous DVD HowTo, you can do excellent rips with virtually indistinguishable quality to the original using the x264 codec, and have a significantly smaller footprint to go with it.

The process of encoding a Blu-ray rip isn’t quite the same as doing a DVD, however, so here’s a quick guide on how to take your decrypted .m2ts file and finish up with a much smaller, but 99% perfect copy in a Matroska .mkv file.

Pre-requisites:

  • A pre-decoded Blu-ray movie file (.m2ts file).
  • Approximately the same amount of free disk space as the size of the movie file. Eg: If you have a 25GB movie file, then you should have another 25GB free space to work with. You can have less, since the final resulting file will be much smaller than the original movie anyway, but since this process can take a number of hours to complete, you don’t exactly want to discover you ran out of disk space and have to start over, do you?
  • A nice powerful CPU. I use a Intel quad-core Q9450 CPU at 2.66GHz. It takes my machine roughly 9-12 hours to process just one movie using four threads. A dual-core will take longer.
  • Time to let the PC do its work, eg: overnight.

This guide was written using Ubuntu 8.10 Intrepid Ibex 64-bit, but will work quite happily in 32-bit and should also work with most previous versions of Ubuntu.

  1. You will need some extra software installed if you haven’t already got it. Open a terminal and type in the following at the $ prompt:

    $ sudo apt-get install mencoder mplayer gpac x264 mkvtoolnix

    (Don’t worry if you’ve already got some of those apps installed, Ubuntu will skip over them if they already exist on your system)
  2. Create a new text file somewhere using your favourite text editor, eg:

    $ gedit ~/encodevideo.sh

    …will create a new text file called “encodevideo.sh” in the root of your Home directory using the GEdit text editor.
  3. Now copy and paste the following script into it:

    #! /bin/bash
    # =====================================================================
    # Blu-ray encoding script by HyRax February 2009 http://www.serenux.com
    # =====================================================================
    # Make sure the user has specified what to work on.
    if [ -z "$1" ]; then
    echo "\nBlu-ray movie encoding script\n-----------------------------"
    echo "Written by HyRax February 2009\nhttp://www.serenux.com"
    echo "\nUsage: $0 <m2ts file without extension>"
    echo "\nExample: If your movie file is called TheDarkKnight.m2ts then\nyour usage will be: $0 TheDarkKnight\n"
    exit
    fi

    # The crf=21 option controls encoding quality, and indirectly the final
    # filesize. The higher the value, the more compression and thus smaller
    # file size. Reduce the value and file size will go up. A value of 21
    # should produce a file of approximately 4GB in size for a typical movie.

    # Encode the video using x264, ignore the audio for now.
    mencoder $1.m2ts \
    -ovc x264 -x264encopts crf=21:frameref=3:bframes=3:b_pyramid=normal:direct_pred=auto:weight_b:partitions=all:8x8dct:me=umh:mixed_refs:trellis=1:nopsnr:nossim:subq=6:level_idc=41:threads=4 \
    -nosound \
    -of rawvideo \
    -o $1.x264

    # Dump the first original audio track (should be the English track) but
    # don't re-encode it. Ignore the video.
    mplayer $1.m2ts -dumpaudio -dumpfile $1.ac3

    # Copy the raw x264 encoded video into an MP4 container so we can set
    # the correct framerate (generally 23.976 - adjust it if MPlayer or
    # MEncoder report something different)
    MP4Box -add $1.x264 $1.mp4 -fps 23.976

    # Finally, merge everything into a single MKV file
    mkvmerge -o $1.mkv $1.mp4 --track-name 0:Eng $1.ac3

    # Remove the hash in front of the next command to have the script delete # your working files when encoding is complete.
    # rm $1.m2ts $1.x264 $1.ac3

    # Tell the user we're done.
    echo "All done! Your final movie file is called $1.mkv - enjoy!"

  4. Save and exit your text editor.
  5. Change directory to where you have your original .m2ts file, eg:

    $ cd ~/Videos/BDRips/MyMovie
  6. Let’s say your movie file is called TheIsland.m2ts. To begin encoding it, you will use the following command:

    $ sh ~/encodevideo.sh TheIsland

    (Notice that we don’t specify the file extension – the script assumes .m2ts on the end already)
  7. Hit enter and the encoding process will begin. The script does the following in order:
    1. Extract the video component only and encode it in a single pass to a raw x264 video file called TheIsland.x264 (in this example).
    2. When that has finished, go back and extract the first audio track only and save it to a file called TheIsland.ac3 without any re-encoding.
    3. When that has finished, take the raw x264 video data and put it into an MPEG4 container so we can fix the framerate at 23.976 frames per second, typical of most Blu-ray movies. If you don’t do this step, then the video will play faster than the audio in the final product.
    4. Finally we create a new Matroska container called TheIsland.mkv and in it we place the framerate-adjusted MPEG4 video track and the unmodified audio track, producing our final product..
  8. When the script has finished some 9-12 hours later, you will have a file called TheIsland.mkv ready for playback in Totem, MPlayer, VLC, or whatever your favourite player is. You will notice that the filesize is significantly smaller than the original but when you play it back, the image quality will look pretty much 100% identical to the original. About the only compression artefacts you may notice is around the edges of some text titles such as opening credits, but you’d really have to look hard to spot them.
  9. We’ve finished with the working files now, so you can delete the .m2ts, .x264 and .ac3 files now to reclaim the disk space and enjoy your new .mkv file.

Enjoy! Smilie: :)

NOTES:

  • While not covered by this HowTo (I’ll add to this later), you can optionally extract other audio tracks from the .m2ts file such as other languages or the commentary track, and add them to the .mkv file at any time. You can also download subtitles and add them to the .mkv file at a later date also, all without needing to re-encode the entire video again because all you’re doing is adding tracks to the existing Matroska container. see the man pages for the mkvmerge utility for more information.
  • A high-definition x264-encoded movie takes a fairly reasonable amount of CPU power to decode. Since Ubuntu Intrepid and previous versions do not make any direct use of on-board video decoding hardware such as that found on NVidia and ATi based video cards, you may find certain busy scenes in your movie may stutter or even skip altogether during playback – this does NOT mean you are missing data from your movie file – it’s just that your PC is having trouble decoding AND displaying the movie all at once because the CPU is doing everything, especially those with slower CPU’s. Indeed you will notice that your CPU usage will probably be quite high in the 80-90% region. Ubuntu Jaunty will bring forth with it VDPAU support for NVidia-based video cards (restricted NVidia driver v180 and above). Using a patched version of MPlayer with VDPAU support, all the decoding work is passed completely to the video card, freeing up your CPU considerably to concentrate on other tasks, and dropping utilisation rates down to about 3%. This makes busy movies that much more watchable. If you can’t wait for Jaunty, there are some backported VDPAU modified versions of mencoder, mplayer and even MythTV available here, however you try them at your own risk. If you are an ATi video card user, you’re out in the dark for the moment. Go buy yourself an NVidia card – they are much better supported under Ubuntu than ATi are.
  • If you are using only a dual-core processor, you should modify the threads=4 section of the mencoder line in the script to read either threads=2 or threads=3. If you are using a single-core processor, change this to threads=1 or threads=2 depending on how your CPU performs (and while you’re at it, seriously consider an upgrade!).
  • If you wish to modify the quality of the encoded video to make the resulting file larger or smaller, re-edit the encodevideo.sh script and change the crf=21 value in the mencoder line to a different value. There is no definitive filesize that the resulting encode will have. This value simply adjusts the quality of the encode. With some trial and error, I have found that the average 1080p movie encode ends up roughly 8GB in size when using a value of 21 and provides excellent image quality. If you make the value larger, this will apply greater compression and will reduce the final file size at the expense of losing some image quality. If you reduce the value, then your final file size will increase, however your image quality will also go up. As a guide, animated movies such as those made by Pixar and Dreamworks and visually dark movies such as Underworld, compress very well. In one instance, the final product was only 3.5GB in size. Visually complex movies such as Transformers blew out to 12GB in size using the same encoding script, so you can see that there is no exact science to this. There are mencoder options to specifically set bitrate and target filesize, but I chose to ignore those options for this HowTo as I’m a bit of a quality freak, not a size freak. I store all my movies on a MythTV server at home and backup to an external drive as I’m not a fan of doubling up my movie purchases on the shelf with a second disc containing the compressed version of the same movie!

HowTo: Install OpenOffice.org 3.0.1 in Ubuntu Intrepid 8.10 right NOW rather than wait for Ubuntu Jaunty 9.04

OpenOffice.org 3.0.x has been out for awhile now, but unfortunately did not get released in time to be included with Ubuntu Intrepid 8.10 back in October 2008, so Intrepid shipped with OpenOffice.org 2.4.1 instead. Bummer.

OpenOffice.org 3.0.x will be included in Ubuntu Jaunty 9.04 in April 2009, but as some people have noticed, there are some useful features in OpenOffice.org 3.0.x (such as much improved Word doc importing) that can make waiting another four months seem like a bloody long time to upgrade. You want 3.0 now, not in four months!

So for the impatient among you, here’s the most painless and easiest way to upgrade your OpenOffice.org to 3.0.1 without having to deal with downloading individual packages or TAR archives from the OpenOffice.org website, or manually having to satisfy the extra dependencies that OpenOffice.org 3.0.1 requires.

  1. First up, open a terminal.
  2. Now type in:

    $ sudo gedit /etc/apt/sources.list.d/openoffice.list

    …and your text editor will appear with a blank new file to type into.
  3. Copy & paste or type in the following line into the editor:

    deb http://ppa.launchpad.net/openoffice-pkgs/ubuntu intrepid main

    Adding the repository using GEdit
  4. Save your file and close the text editor.
  5. Now type in at the terminal:

    $ sudo apt-get update
  6. You should observe that a warning message regarding NO_PUBKEY will appear after the update has completed as follows:

    W: GPG error: http://ppa.launchpad.net intrepid Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 60D11217247D1CFF

    This is because you don’t yet have the public key for this OpenOffice.org repository to be able to authenticate anything from it, so we need to add it with the following command at the $ prompt:

    $ gpg --keyserver keyserver.ubuntu.com --recv 60D11217247D1CFF
    gpg: requesting key 247D1CFF from hkp server keyserver.ubuntu.com
    gpg: key 247D1CFF: public key "Launchpad PPA for OpenOffice.org Scribblers" imported
    gpg: Total number processed: 1
    gpg:               imported: 1  (RSA: 1)
    $


    Right, that downloaded the key from the Ubuntu keyserver to your PC. Now we need to tell Apt to use that downloaded key to authenticate with. Do do this, type in:

    $ gpg --export --armor 60D11217247D1CFF | sudo apt-key add -
    OK
    $

  7. If you now run Step 5’s command again (sudo apt-get update), you will no longer have the NO_PUBKEY warning at the end of it.
  8. Anyway, once Apt has finished updating itself with a new list of packages, within a few seconds, your Update Manager will pop up the red arrow in the system tray advising that there are new updates to download.

    .Software Updates are Available
  9. When you open Update Manager, you will find that they are all upgrades to OpenOffice.org 3.0.x including the extra dependencies required! Sweet!

    OpenOffice.org Updates ready to install
  10. Hit the “Install Updates” button. If you are presented with a dialog box warning you that some of the packages cannot be authenticated, then you have not got the Public Key from the Ubuntu Keyserver to authenticate the new OpenOffice.org packages. This is because this OpenOffice.org 3.0.1 repository is not official nor permanent and was only setup to satisfy the impatient ones out there (ie: you!). It is not critical that the packages be authenticated, but that is up to you to decide. If you’d rather not install unauthenticated packages, then just click the Cancel button and stop following these instructions. If you followed Step 6 properly, then you will not get this warning message because the public key you downloaded in Step 6 will authenticate the packages properly and the download of updates will commence immediately as per Step 11.

    .Warning about Not Authenticated Packages
  11. Hit the Apply button and allow the system to download and install as normal.
  12. When completed, start up OpenOffice.org as normal and you will see that the splash loader and the Help->About page show that you now have OpenOffice.org 3.0.1!

    .About OpenOffice.org
  13. Pat yourself on the back. You’re done.

How is this possible anyway? Well, seems that demand for OpenOffice.org 3.0.x has been so great that the nice guys at OpenOffice.org Scribblers decided to put together a quick little dedicated repository for it on the Launchpad site, so all kudos and credit go to them. The repo only works for Ubuntu Intrepid, so for users of earlier versions of Ubuntu, you will still need to download the .debs and install them manually from the OpenOffice.org website.

Credit goes to stmok from Overclockers Australia for advising of the public key solution used in Step 6 – thanks!

HowTo: Rip a Blu-ray movie using an LG GGC-H20L Blu-ray drive with Ubuntu

The Blu-ray disc format has brought with it the ability to easily provide the next generation of High-Definition 1080p movie content. There’s just one problem – Ubuntu and Linux in general has no official support for Blu-ray, and its encryption scheme is vastly different to that of DVD – it’s not just a simple case of installing a library like the libdvdcss2 library for decrypting DVD’s – the protection is done both at a software and hardware level.

This article discusses how I used my recently purchased LG GGC-H20L Blu-ray ROM drive to successfully read and watch movies using Ubuntu Intrepid.

DISCLAIMER: This article describes decrypting BD titles using an Intel or AMD based PC with Ubuntu Linux. While you can use a PlayStation3’s BD drive to read and decrypt a title using known decryption keys using the PS3 version of Ubuntu, at this time of writing you cannot use Ubuntu installed on a PlayStation3 console to identify unknown decryption keys of a given BD title because the application used to derive those keys from the disc is not available for the PPC processor used by the PS3. You must use a consumer BD-ROM drive on an Intel or AMD based PC instead.

Hang on, you say – if there is no support for playback of Blu-ray movies on Linux, then why buy a Blu-ray drive if you can’t watch the movies? Well, I might not be able to watch them directly, but I can certainly rip the little buggers and watch a file version of it instead. But wait again, you say, if there’s no official Blu-ray support, and you can’t watch the discs directly, then how on earth do you rip them?? I’m glad you asked, and even if you didn’t ask, I’m about to tell you anyway. Smilie: :)

First up, a little Blu-ray 101.

Blu-ray movies feature Digital Rights Management (DRM). Like DVD’s before it, most Blu-ray movies are encrypted. This is to stop those naughty pirate types from making illegal backups of the movie and giving or selling it to their mates. In the case of DVD’s, however, there was only one decryption key which was eventually discovered and from then on allowed all DVD’s to be easily decrypted using a simple library (called libdvdcss2). The movie industry as a whole were not impressed by this and insisted that the next format be more difficult (preferably impossible) to decrypt, so Blu-rays (and HD-DVD’s, but I’ll concentrate on Blu-ray for this article) each have a different decryption key now. But to complicate this further, this key is kept hidden from you by an authentication mechanism to ensure that there isn’t a repeat of the De-CSS scandal that brought the DVD encryption scheme undone.

Each and every player out there, hardware or software, has a unique player authentication key which is passed to the Blu-ray optical drive, essentially like giving your passport to Customs at the country border, to validate whether or not you are legally authorised to  playback a movie. If the drive has not got a blacklisted record of your authentication key, and the key is accepted as a generally valid key that has been paid for, THEN the drive will give up the movie’s decryption key and movie playback can commence.

Blacklist?? What blacklist? Well, like DVD players, it’s not difficult to pick up a Blu-ray player’s authentication key that is used to prompt the drive for the disc’s decryption key, after all, it has to be held in memory somewhere. Once an industry authority discovers that an authentication key has been compromised, it is added to a blacklist so that it will not work anymore. This is why hardware Blu-ray players need to be firmware-upgradable, and why software players need to be upgraded to the next version periodically with patches, etc, so that new, non-blacklisted authentication keys can be provided.

OK, that sounds all well and good, but if the Blu-ray drive itself is doing the blacklisting, how exactly does it know when a given authentication key is no longer valid? Simple – its blacklist will get updated with the next latest-release movie you buy.

Say what?

Every Blu-ray movie you buy has a little file on the disc under the “AACS” folder called “ContentRevocation.lst”. This file contains a complete list of blacklisted authentication keys for the drive to update itself with, and – get this – you can’t stop the drive loading it. Well, actually to be more accurate this file is simply a copy of the same list that is actually hidden in a non-tamperable, non-user-readable area elsewhere on the disc for the drive to read, but basically the instant you stick that movie disc in, the hidden version of this file is read and the drive automatically updates its blacklist right away with any new blacklist data, even before the disc icon appears on your desktop. Sneaky, huh?

So the next time your legal (or more specifically, pirated) copy of PowerDVD or whatever tries to playback a movie, all of a sudden you’ll see an error message instead saying that your player’s authentication key has been revoked – thus the movie is now unplayable. What’s worse is that you won’t be able to watch any of your older discs that worked previously either! It’s this exact reason that many people have called for the Blu-ray (and HD-DVD) formats to be boycotted.

But not all is lost. Remember, this is an encryption technology created by Man, and therefore can be broken by Man with a bit of help from the Open Source Community at large. Smilie: :) With a little help from a few external resources including the Ubuntu Community Documentation and the Doom9 forums, I discovered a plethora of projects by various people, from dumping discs to breaking the encryption and authentication.

On the surface are general applications to dump the movie, decrypted, to a file. One of the best projects is DumpHD which is a Java app that provides a nice easy to use GUI that can rip a movie with a minimum of fuss (see detail about how to use it later in this article), however it requires that you already know what the decryption key for a given movie is which you can only obtain if you have authenticated with the drive. If you have the decryption key already, however, then authentication is not necessary and you can rip the movie right away without a problem, so this project is heavily supported by people posting up various decryption keys for all sorts of movie releases. The problem with this approach, however, is that different countries usually get different releases of the same movie, so for example a release of “Batman Begins” in Australia might have a completely different decryption key to the release of “Batman Begins” in America or Europe. This is not always the case of course, a good example being the Ewan McGregor movie “The Island” – the Australian release is actually the UK release, right down to the age-rating and film-office classification markings on the disc itself – only the box bears any Australian ratings markings!

The UK/AU release of “The Island” on Blu-Ray

So, how do you find out the decryption key of your locally purchased movie then? You don’t want to keep buying a commercial player, especially one that doesn’t run under Ubuntu, just to get a valid authentication key. There’s got to be a better way! Well, there is! How about we just bypass the authentication procedure altogether? How? Again, through another great contribution on the Doom9 forums.

The LG GGC-H20L drive is but one of many Blu-ray/HD-DVD drives which have had their firmware reverse-engineered. Firmware is largely just a computer program that operates the drive. Since the firmware is upgradable to fix bugs and add new features to the drive, it means the program can be altered by a third party. A Doom9 contributor has provided modified firmware for various popular drives, including the GGC-H20L that effectively allow the drive to ignore the authentication procedure no matter what player authentication key is provided, blacklisted or not, thus making the drive give up the decryption key for the Blu-ray movie currently inserted every time!

EDIT 10th Jan 2012: It appears the firmware is no longer available from the Doom9 forums as it has not been updated in quite some time. Here is a mirror of the v1.03 modified firmware to suit the LG GGC-H20L only.

In the case of the LG GGC-H20L that I use, the firmware is provided as a Windows Executable file. There are three ways to run this, either via a native Windows installation on your PC, a virtualised Windows installation on your PC, or via the Wine compatibility layer. I successfully upgraded my drive using the Wine option as follows:

WARNING: The following information can damage or even brick your LG Blu-ray drive if not followed correctly, or if you have a power failure during firmware update. You proceed at your own risk and I will not be held responsible for any damage incurred by your drive, or for loss of hair being torn out, by following these instructions.

  1. You need a normal Wine installation. If you’ve never installed it before, then type in:

    $ sudo apt-get install wine

    …and this will install the Ubuntu-repository version of Wine.
  2. Since we’ll be modifying a physical device, only root can do that, so we will need to use sudo to execute the firmware upgrade, however Wine will not work as root until we change permissions of your Wine configuration in your Home directory, so type in the following to make root the owner of your Wine configuration:

    $ sudo chown -R root:root ~/.wine
  3. Now execute the downloaded firmware file:

    $ sudo wine GGC-H20L_1.03_VolumeID_Patch.exe
  4. The Upgrade GUI will appear. Click on the button to commence update and let it do its thing. After a minute or two, it will tell you that the drive firmware has been successfully updated. At this point, you will need to reboot your PC, but before you do, remember to change the owner of your Wine installation back to yourself. If your login was “jbloggs”, then you’d type in:

    $ sudo chown -R jbloggs:jbloggs ~/.wine

Upon returning from restart, every time you query the drive for the inserted disc’s decryption key, the drive will now happily give it to you without question. Nice.

So how do we query the drive for that decryption key anyway? A simple tool to do this is aacskeys (version 0.4.0c at this time of writing, but check further along the thread for newer releases) written by another Doom9 member, which queries the drive and tells you its Volume Unique Key and its Disc ID, which you can then copy and paste into DumpHD’s keys config file and happily begin dumping your movie.

$ ./aacskeys /media/cdrom aacskeys 0.3.6 by arnezami, KenD00 Volume Unique Key:              5D9BCD44522B6940F8705400DA612ED9 Unit Key File Hash (Disc ID):   837487B4D6F614D5B4D5F566387B41C2D284F393 $ 

If your drive’s firmware was not patched, instead of seeing the Volume Unique Key and Disc ID, you would get this error message instead: “The given Host Certficate / Private Key has been revoked by your drive.”.

We now need to take the output data and copy it to DumpHD’s “keydb.cfg” file. Each key is placed on its own line in the following format:

DISC ID = Movie Title | D | YYYY-MM-DD | V | VOLUME UNIQUE KEY

Thus in the example above, we would enter:

837487B4D6F614D5B4D5F566387B41C2D284F393 = The Island | D | 2007-03-22 | V | 5D9BCD44522B6940F8705400DA612ED9 

Now technically it appears that the date is largly irelevant, and most people just use 0000-00-00 instead of a real date (which is supposed to be the file date of the “Unit_Key_RO.inf” file in the “AACS” folder of the disc). I have tested this and I can’t see any difference in how DumpHD handles the disc.

Once you have finished editing the keydb.cfg file, save it.

NOTE: DumpHD (below) can now use AACSKeys directly, saving you having to edit to the keydb.cfg file manually, because DumpHD now does all the work for you. See this article for more information.

Since DumpHD is a Java application, you will need Java installed to run it. If you haven’t already got it installed, you can install it with the following command:

sudo apt-get install java-common

…or more realistically you should install it as part of the ubuntu-restricted-extras package which also installs a number of other useful packages:

sudo apt-get install ubuntu-restricted-extras

Once that is done, launch DumpHD by simply running the “dumphd.sh” script by either double-clicking on it and select “Run” when prompted, or in a terminal, change to where you extracted the DumpHD program and type in:

$ sh ./dumphd.sh

When loaded, you will be presented with the following interface:

The DumpHD user interface

At the top-right of the window, there is the Source Browse button. Click on it and a new window will appear. In that window, type in or select “/media/cdrom” and then click OK. After a few seconds, the disc should be identified and you will see the window change as follows:

Identifying the BD-ROM content

Now all you need to do is click on the Destination Browse button, specify a place to store the decrypted movie and then click the Dump button to start the whole process! Once finished, you will have every video title found on the disc dumped in its originally encoded (video-wise) format, but without DRM. You can then use MPlayer to play these files directly. Generally the movie itself is the largest file, so in the case of my example, it’s the “00000.m2ts” file. I can play it simply with:

$ mplayer -fs 00000.m2ts

The -fs parameter plays the movie full-screen. I can toggle audio tracks using the hash (#) key (as for some reason, English is generally not the first audio track on the disc).

Remember that Blu-ray movies are very large. In my example, “The Island” is a good 21GB! It’s now up to you to decide whether or not you want to provide storage for your rips of this size, or whether or not you want to compress them down to save space. In my case, I was able to compress the movie down to about 4GB with negligible quality loss at a bitrate of 1200 using the x264 codec. I’ll probably increase this bitrate and allow the filesize to go to 8GB so I can maintain as near-perfect image quality to the original Blu-ray as possible. I personally choose to preserve the audio tracks as-is without down-converting them – they really don’t eat up that much space – a few hundred megabytes only.

Happy ripping!

Woah! After I’ve ripped the movie, it plays but it’s all corrupted! What’s going on?

The movie you have ripped is likely protected using BD+ protection. This is where some or much of the movie is deliberately corrupted to annoy you. I have written a guide on how to deal with this and correct the corruption here.

HowTo: Create and manage SSH tunnels using Gnome SSH Tunnel Manager

The Secure SHell, or SSH, is a simple but effective tool for encrypted remote login to a Linux-based workstation or server. Not only does it give you secure command-line access to the remote PC, but it can also be used to securely redirect data to be routed or processed on the remote PC’s LAN such as HTTP traffic or perhaps another protocol such as Jabber that is blocked on your local Internet connection. This process is called tunneling.

Creating an SSH tunnel is easy. Say you wanted to forward the local port 2100 to a web server inside your remote network on IP 192.168.0.99. The public hostname of the SSH server you are using to connect to is myserver.com. You can issue the command:

$ ssh jbloggs@myserver.com -L 2100:192.168.0.99:80 -N

…this will redirect local port 2100 to port 80 on the server with LAN IP 192.168.0.99 on the remote network, via the Internet-accessible SSH server on myserver.com, logging into it as the user jbloggs. Everytime you now access http://localhost:2100 with your web browser, your traffic will be redirected to the remote web server, using an encrypted tunnel over the Internet.

Now this is great, but that command line looks a bit convoluted. Isn’t there a more intuitive way to create an SSH tunnel?

There sure is – a neat little Gnome GUI app called the Gnome SSH Tunnel Manager or gSTM for short.

Here’s a diagram to illustrate what we are trying to do.

The Internal LAN web server cannot be accessed from the Internet, only local LAN users. For the workstation on the other side of the Internet to be able to access the LAN web server at 192.168.0.99, we need to tunnel into the SSH server and then pass through it to get to the Internal LAN web server.

First up, we need to install gSTM. It exists in the Ubuntu repository, so all you have to do is enter:

$ sudo apt-get install gstm

Once installed, you will find it in the Applications->Internet->gSTM menu. When you fire it up, you will be presented with the following window (in this case I’ve already setup three tunnels – your list will be empty):

GTSM Interface

The list shows all the tunnels you have created to various locations. Each tunnel can have multiple ports directed over it. The green light next to the tunnel name indicates an active tunnel. A red light indicates an inactive tunnel.

  1. Creating a new tunnel is simplicity itself – just click on the Add button. A new window appears.
Creating a new tunnel
  1. In this window, we specify an arbitrary name for the tunnel, then the SSH server we’re connecting to, eg: myserver.com in the previous example (or its public IP address), the login name used to access that SSH server, eg: jbloggs, and the port that the remote SSH server is listening to (which by default is 22, but you may have changed it to a non-standard port number above 1000 for greater security).
  2. These are the only values you need to fill in to establish the basic tunnel. Next, we need to specify which ports we are going to redirect into that tunnel. Click on the “Add” button to add a new port.
  1. First we specify where the port we’re redirecting is coming from. “Local” means that we’re redirecting a port from the PC you are working on, to the remote PC. In the above example, we are going to redirect an arbitrary port number 32456 to port 80 on the PC 192.168.0.99 on the remote network’s internal LAN (via the remote SSH server). Click OK and then click OK on the previous window to save your Tunnel settings. It now appears in the list of available tunnels.
  2. Simply click on the Start button and you will be prompted for the password for the account you specified for the tunnel to access the remote SSH server with. Type it in and hit Enter.
  1. Once the green light appears next to your tunnel name in the list, you are ready to go. Open your web browser and type in http://localhost:32456 and you will suddenly see the web page of the internal machine on the remote network.

That’s pretty cool, but let’s try something even cooler. Stop your tunnel by clicking on the Stop button, then bring up the Properties for it again.

Let’s say you are at an office or school where the local Internet web access is filtered. There is a website that you’d like to visit, but it is blocked by the local organisation’s filter. Assuming outbound SSH access is allowed, you can effectively bypass the filter by routing all your web traffic via your remote SSH server whose web access is unfiltered. Here’s a diagram to illustrate what we’re doing:

An SSH tunnel in action

The workstation tries to access the blocked website (indicated by the red line) but fails. By using an SSH tunnel (the green line), we can access the blocked website via the remote SSH server.

  1. To do this, add a redirection but change “Local” to be “Dynamic”, and specify an arbitrary port number, eg: 9100. You will notice that you cannot specify a To host or To port. Click OK.
  1. Now open up your web browser and bring up the Proxy settings. In Firefox, this is located by going to the Edit->Preferences menu, then click on the Advanced icon, then the Network tab, then finally click on the Settings button. You are presented with the following window:
  1. Change the radio selection at the top of the window to Manual proxy configuration and then in the SOCKS Host box ONLY, specify “localhost” and port “9100”.
  1. Now click OK and then close the Firefox preferences window.
  2. Start your SSH tunnel again by clicking on the Start button.
  3. Now try surfing to your blocked website. Instead of it being filtered, you will now find it magically appears because your web traffic is being proxied to your SSH tunnel and retrieved through your unfiltered remote Internet connection!
  4. When you are done, don’t forget to shutdown your SSH tunnel and change your web browser’s proxy settings back to whatever they were previously.

Happy tunneling! Smilie: :)

HowTo: Rip a DVD video title into an x264 and Ogg encoded MKV video file

Many people like myself jumped aboard the revolution that was the DVD ten years ago (has it already been that long??) and collected a vast library of discs that now take up space on several shelves in the corner of your lounge room. In this day and age of the PVR and DVR, even I myself find it frustrating to go to the shelf, find the movie I want to watch, take the disc out, make sure it’s free of fingerprints, stick it in the drive, skip all the blasted “mandatory” ads and trailers before you can actually get to the movie itself. At least with YouTube and downloaded AVI and MPEG files, you can simply double-click and watch what you want, when you want, on demand, 24/7 – no mess, no fuss.

So here I present a guide on how to rip your DVD collection into convenient, tidy x264-encoded MKV files. You may find that you can store your entire collection of DVD’s onto a single external hard-drive to carry with you, and will certainly serve as a useful backup the day that some inconsiderate soul scratches or steals your DVD’s! This HowTo is based on The Smorgasbord HowTo, but with modifications to bring it up to date with current implementations of x264 and MEncoder.

Purpose:

To rip a DVD video disc title’s components (video, audio, subtitles, etc) into a single Matroska (.mkv) file.

Why a Matroska file instead of an MPG or AVI file?

Matroska is an open-standards container format that is rapidly gaining support. It is an envelope for which there can be many audio, video and subtitles streams, allowing the user to store a complete movie or CD in a single file. Matroska offers many benefits including:

  • Fast seeking in the file
  • High error recovery
  • Chapter entries
  • Selectable subtitle streams
  • Selectable audio streams
  • Modularly Extendable
  • Streamable over internet (HTTP and RTP audio & video streams)
  • Menus (like DVDs)

Pre-requisites:

To rip a DVD title, you will need:

  • A DVD video disc with a title or two to rip from (yes, sounds obvious doesn’t it?).
  • Any garden-variety DVD-ROM drive to read the disc with.
  • A reasonably powerful PC – any Core 2 Duo or better will suffice. You can use a slower machine such as a Pentium 4, but the ripping and encoding time can be a difference of several hours.
  • Suitable storage space to store your ripped video plus some working space – 8GB all up is ideal for a single title.
  • An Ubuntu Intrepid 8.10 installation (which is what this guide is based on).
  • Some patience (unless you’ve got a multi-core monolith that can work out Pi to the sixth billionth place in less than a few nanoseconds, in which case you’re laughing).
  1. We’ll need to install some extra software tools. Start by opening a terminal and enter the following:

    $ sudo apt-get install mplayer mencoder normalize-audio vorbis-tools mkvtoolnix gpac x264 libdvdcss2
  2. Once all that is installed, insert the DVD you wish to rip from. Now generally most DVD’s have the main feature as title number “1”, but these days we are increasingly seeing a number of trailers or advertisements starting with this title, so to find out what title is what, type in the following:

    $ mplayer dvd://1
  3. This will attempt to play the first title on the disc in a new window. You will notice in the console output that it will also list some other useful information such as how many titles are on the disc, etc. If the title being played is not the feature you’re after, simply close the playback window and change the command to use dvd://2 or dvd://3 etc until you find the title you want. An example of this output is:

    $ mplayer dvd://1
    MPlayer 1.0rc2-4.3.2 (C) 2000-2007 MPlayer Team
    CPU: Intel(R) Core(TM)2 Quad  CPU   Q9450  @ 2.66GHz (Family: 6, Model: 23, Stepping: 7)
    CPUflags:  MMX: 1 MMX2: 1 3DNow: 0 3DNow2: 0 SSE: 1 SSE2: 1
    Compiled with runtime CPU detection.
    Playing dvd://1

    There are 3 titles on this DVD.
    There are 20 chapters in this DVD title.
    There are 1 angles in this DVD title.
    audio stream: 0 format: ac3 (5.1) language: en aid: 128.
    audio stream: 1 format: ac3 (5.1) language: es aid: 129.
    audio stream: 2 format: ac3 (stereo) language: en aid: 130.
    number of audio channels on disk: 3.
    subtitle ( sid ): 1 language: en subtitle ( sid ): 3 language: en
    subtitle ( sid ): 5 language: es subtitle ( sid ): 7 language: sv
    subtitle ( sid ): 9 language: no subtitle ( sid ): 11 language: da
    subtitle ( sid ): 13 language: fi subtitle ( sid ): 14 language: es
    subtitle ( sid ): 15 language: sv subtitle ( sid ): 16 language: no
    subtitle ( sid ): 17 language: da subtitle ( sid ): 18 language: fi
    subtitle ( sid ): 20 language: es number of subtitles on disk: 13

    …etc…
  4. In the above example, there are three titles, 20 chapters, three audio tracks and 13 subtitles for various languages. If I wanted to see what Title 3 was about, I would pass the argument dvd://3 to the mplayer command.
  5. Now let’s start ripping that title. First up, create yourself a working directory:

    $ mkdir mydvdrip
    $ cd mydvdrip

  6. Assuming Title 1 was the correct title, type in the following:

    $ mplayer dvd://1 -v -dumpstream -dumpfile mydvdrip.vob

    This tells MPlayer to dump the title data to a physical file called “mydvdrip.vob” instead of displaying it to a window. The .vob file contains both the video and audio tracks of the selected title.
  7. Once that is done, you can optionally rip the DVD subtitles too using:

    $ mencoder dvd://1 -oac copy -ovc frameno -o /dev/null -slang en -vobsubout mydvdrip

    …which will result in the English subtitles (the “en” in the -slang argument)  files mydvdrip.idx and mydvdrip.sub being created.
  8. Now, the audio can be kept in its original AC3 surround format, but can take up a lot of extra space. If you’re not fussed about proper full surround sound, you can convert the audio to plain stereo, but to do that means we need to convert the audio into PCM format as follows (if you want to preserve the original AC3 surround audio, then skip to step 11 now):

    $ mplayer mydvdrip.vob -ao pcm:file=audio1.wav -vc dummy -aid 128 -vo null

    The “-aid 128″ part refers to the audio track to rip. In this case, it’s the first audio track. If I wanted to rip the second audio track, I would use “-aid 129″ instead. See the MPlayer console output when you were playing the disc for the other aid parameters to access the other audio tracks.
  9. Once the audio is ripped and converted, we need to normalise it so it’s not too loud or too soft with:

    $ normalize-audio audio1.wav
  10. …and then finally convert it into the final audio format we want to put into the final ripped product, in this case Ogg Vorbis which is a much cleaner-sounding and open format than MP3:

    $ oggenc -q5 audio1.wav

    The -q5 parameter encodes at 160 kbits/s which should be very clear and crisp compared to the original audio. Change this parameter to alter the compression if you want to save space. Skip to step 12 now.
  11. If you didn’t want to convert the audio to stereo, and preserve the original surround audio, you can rip the AC3 audio with:

    $ mplayer mydvdrip.vob -aid 128 -dumpaudio -dumpfile mydvdrip.ac3
  12. We have the audio and subtitles finished, so all we need now is the video itself. As you have probably noticed, many DVD titles generally have a border of black around them to some degree. We don’t want this in our rip and can save some space by omitting it, but we need to get the cropping bounds. Thankfully MPlayer does a pretty good job of detecting this by itself. Type in:

    $ mplayer mydvdrip.vob -vf cropdetect
  13. Allow the title to play for a bit, or skip forward a bit. You will notice some data output to the terminal. This is what MPlayer believes is the optimal cropping bounds for this title where it says -vf crop=688:448:18:64 (your output will vary). Stop playback and copy the series of numbers after the equals sign to the clipboard.
  14. Now we need to setup two-pass encoding. Two passes are made for quality reasons. The first pass makes note of details about the video, such as slow and fast moving scenes, which allow MEncoder to vary the bitrate of the resulting video accordingly, providing great image quality whilst maintaining a small file size. Encoding requires two executions of MEncoder one after the other, so let’s create a shell script to do it. Create an empty script with:

    $ gedit encodevideo.sh
  15. …and an empty text editor will appear. Copy & paste in the following:

    # Encoding pass #1
    mencoder -v mydvdrip.vob -vf pullup,softskip,crop=688:448:18:64 \
    -ovc x264 -x264encopts subq=4:bframes=3:b_pyramid:weight_b:\
    turbo=1:pass=1:psnr:bitrate=1000:threads=auto -oac copy \
    -of rawvideo -o mydvdrip.264

    # Encoding pass #2
    mencoder -v mydvdrip.vob -vf pullup,softskip,crop=688:448:18:64 \
    -ovc x264 -x264encopts subq=6:partitions=all:me=umh:frameref=5:\
    bframes=3:b_pyramid:weight_b:pass=2:psnr:bitrate=1000:threads=auto \
    -oac copy -of rawvideo -o mydvdrip.264

    (Remember to substitute the “crop=” values from your own crop output)

    I won’t go through describing what all the options above do suffice that they will produce a very high quality rip of your DVD title using a modest amount of disc space (generally no higher than about 700MB for a single DVD movie), and it will make full use of any multi-CPU or multi-core processing power at your disposal (a Quad-core system will encode video roughly 3 times faster than a single core system). If the resulting file size is still too big, you can make it smaller by changing the bitrate=1000 parameter to something slightly smaller, eg: 900, but you will begin to sacrifice image quality.
  16. Save the file and quit your text editor.
  17. We’re now about to begin ripping and encoding the video data. Depending on your system, this part could take anywhere between 60 minutes to several hours. In this example, my 3.2GHz Quad-Core Q9450 based system took 32 minutes to perform just the first encoding pass at 87fps, with the second pass taking 52 minutes to complete at 54fps. Execute the script with:

    $ sh ./encodevideo.sh
  18. …and go order a pizza, or wait. The video from the .vob file will be extracted, encoded using the x264 codec and the resulting raw data placed into a file called mydvdrip.264 at the end.
  19. Once the script has finished, you should now have the following files in your working directory (except the .ac3 file if you chose not to rip it):

    -rw-r--r-- 1 hyrax hyrax   91783421 2008-12-21 19:29 audio1.ogg
    -rw-r--r-- 1 hyrax hyrax 1058132012 2008-12-21 19:26 audio1.wav
    -rw-r--r-- 1 hyrax hyrax   13936736 2008-12-21 23:02 divx2pass.log
    -rw-r--r-- 1 hyrax hyrax  688634256 2008-12-21 23:49 mydvdrip.264
    -rw-r--r-- 1 hyrax hyrax  264532992 2008-12-21 19:31 mydvdrip.ac3
    -rw-r--r-- 1 hyrax hyrax      38423 2008-12-21 19:23 mydvdrip.idx
    -rw-r--r-- 1 hyrax hyrax    3426304 2008-12-21 19:23 mydvdrip.sub
    -rw-r--r-- 1 hyrax hyrax 4473358336 2008-12-21 19:16 mydvdrip.vob
    -rw-r--r-- 1 hyrax hyrax        482 2008-12-21 22:35 encodevideo.sh


    In the above example, my original DVD title rip was some 4.4GB in size. The resulting compressed video file using x264 is only a paltry 688MB – notably smaller and the visual quality on playback looks identical to the original DVD data. The original AC3 audio consumed 264MB, but the Ogg compressed version only consumes 91MB. Put the x264 video data and converted Ogg audio data together and we have a rip that will consume a total of 779MB all up (give or take a few MB for the container that will hold them both).
  20. Now we need to recombine the audio, video and subtitles into one Matroska container file. First up, the video. We need to convert the video from raw x264 data into MP4 format before we add it to the MKV container. To do this, type in:

    $ MP4Box -add mydvdrip.264 mydvdrip.mp4
  21. Now we’re ready to build the final MKV file. Type in:

    $ mkvmerge -o mydvdrip.mkv *.mp4 *.ogg *.idx

    (replace the *.ogg with *.ac3 is you want to use the original audio instead, or include them both so they are selectable on playback! Beware this will blow out your final file size, however)
  22. When the muxing is complete, your rip is complete. You can now play it in your favourite video player such as Totem or MPlayer.

— o O o —

To recap the whole process:

  1. mplayer dvd://1 -v -dumpstream -dumpfile mydvdrip.vob
  2. mencoder dvd://1 -oac copy -ovc frameno -o /dev/null -slang en -vobsubout mydvdrip
  3. mplayer mydvdrip.vob -ao pcm:file=audio1.wav -vc dummy -aid 128 -vo null
  4. normalize-audio audio1.wav
  5. oggenc -q5 audio1.wav
  6. mplayer mydvdrip.vob -vf cropdetect
  7. mencoder -v mydvdrip.vob -vf pullup,softskip,crop=688:448:18:64 \ -ovc x264 -x264encopts subq=4:bframes=3:b_pyramid:weight_b:\ turbo=1:pass=1:psnr:bitrate=1000:threads=auto -oac copy \ -of rawvideo -o mydvdrip.264
  8. mencoder -v mydvdrip.vob -vf pullup,softskip,crop=688:448:18:64 \ -ovc x264 -x264encopts subq=6:partitions=all:me=umh:frameref=5:\ bframes=3:b_pyramid:weight_b:pass=2:psnr:bitrate=1000:threads=auto \ -oac copy -of rawvideo -o mydvdrip.264
  9. MP4Box -add mydvdrip.264 mydvdrip.mp4
  10. mkvmerge -o mydvdrip.mkv *.mp4 *.ogg *.idx

Or, to preserve the original AC3 surround audio track without conversion, replace steps 3 to 5 with:

mplayer mydvdrip.vob -aid 128 -dumpaudio -dumpfile mydvdrip.ac3

…and replace step 10 with:

mkvmerge -v -o mydvdrip.mkv *.mp4 *.ac3 *.idx

That’s it. Happy ripping!

HowTo: Utilise the RAM disk

Those of you who are Amiga veterans from the 80’s and early 90’s will be familiar with the oh-so-useful RAM: disk. A virtual device that treated your unused system RAM as a disk device that you could extract temporary files to and make a general mess of without worrying about cleaning it up later, because when you rebooted, the content of the RAM disk would be cleared. The amount of data you could put in it was simply limited to how much physical RAM you had. The RAM disk was always 100% full because it was dynamic in nature. If you had 10MB of data in there, then the RAM disk consumed 10MB of system RAM. If you only had 2MB of data in there, then it only consumed 2MB of system RAM – the RAM disk never impacted on the rest of system memory in a prejudiced fashion.

Microsoft DOS and Windows have tried to replicate this feature with limited success, and the RAM disk was always a fixed size and pre-allocated from the rest of the system as well, which made it somewhat useless. If you had a 512MB RAM system and allocated 50MB for the RAM disk, then you only had 462MB left for the rest of the system, and this also meant that you could not put more than 50MB of data into the RAM disk.

Ubuntu Linux (and pretty much every other distro) has its own RAM disk feature as well and is just as flexible as the Amiga’s RAM disk in its use. Here’s a guide on how to use it.

Using a RAM disk in Ubuntu is simplicity itself – it’s already setup for you!

The RAM disk is located at /dev/shm on your system. Anything placed in this directory is in the RAM disk and will be lost when you reboot the system. The RAM disk is already utilised for temporary files from PulseAudio, so if you list the content of the RAM disk in a terminal with:

$ ls /dev/shm

You might get one or more temporary files that look similar to “pulse-shm-2343031262″.

Let’s make the RAM disk more accessible – how about an icon for it on the Gnome desktop?

  1. Start by doing a single right-mouse-click  anywhere on the Gnome desktop. A menu appears.
  1. Choose “Create Launcher” from this menu. A “Create Launcher” window will appear.
  1. Change the “Type” option from Application to Location.
  2. In the “Name” field, type in My RAM disk (or whatever you prefer).
  3. In the “Location” field, type in file:///dev/shm/
  4. And finally in the “Comment” field, type in a suitable description such as “Accesses the RAM disk for temporary data”.
  1. Click OK and a new icon will appear on the desktop entitled “My RAM disk”.
  1. When you double-click on it, a Nautilus window will open showing the content of the RAM disk.

Try copying some data to the RAM disk now and then reboot your PC. When you re-open the RAM disk, you will find the data you copied is now gone because it was never physically saved to your hard-drive.

Great stuff, but that Launcher icon looks a bit boring. Let’s give it some meaning. On my system, I use an icon of a male sheep, or (you guessed it) a Ram! Male sheep = Ram = RAM disk, geddit? Oh never mind… You can use this visual gag yourself by replacing the icon of the launcher with an image file.

  1. First up, download the sheep icon from here and place it somewhere appropriate on your system where it can be re-retrieved from, eg: /home/jbloggs/Pictures/RamSheep.png because it needs to be reloaded as your Launcher icon whenever you reboot. Leave this Nautilus window showing pictures folder open as we’ll need it to drag and drop from later.
  2. Now do a single-right-click of your RAM disk icon on your desktop and choose “Properties” from the menu. A new window appears.
  1. The properties window shows the current icon for your Launcher.
  1. Now drag the sheep icon from where you downloaded it to on top of the default icon of the launcher and release the mouse button to drop it. This will replace it with the picture of the sheep.
  1. Click close to close the Properties window.
  1. If the icon is too small for you, you can make it bigger by resizing it. Do a single-right-click on the  icon and choose “Stretch icon” from the menu. Four anchors will appear on the icon for you to pull on.
  1. Grab one of the anchors with your mouse and drag it until the icon is at the desired size.
It’s a RAM Disk, geddit?
  1. Click anywhere outside of the icon to set the icon size and remove the anchors.
  2. You’re done!

HowTo: Eliminate the Green Bar on video playback in Totem (and other media players)

Some people, including myself, have experienced an annoying issue whereby some videos you playback, particularly those encoded with the Xvid codec, have an apparent corruption of video represented by a thick Green Bar(TM) somewhere on the image. Generally the colours are askew as well and sometimes sections of the image are blurred. You might assume that the original video is corrupt in some way, but this is not the case as the video will typically run fine through media players on other platforms.

The Green Bar(TM) and general video corruption looks similar to this:

The dreaded Green Bar of DeathTM

The issue is that the XVideo extension of X Windows (aka Xv) is having trouble dealing with the decoded video data when using your video card to scale the video to fit the window. There is a fix, however, at a minor cost of some CPU time – instead of getting your video card to do the scaling, get the CPU to do it instead.

  1. Press ALT+F2 to bring up the Run Application dialog (or just open a terminal) and type in gstreamer-properties and hit Enter.
  1. You will be presented with the GStreamer properties app. Click on the Video tab and you will see the following:
  1. Change the Default Output Plugin from Autodetect to X Window System (No Xv) and then close the application.
  1. Now re-run your video file through Totem and the Green Bar(TM) should now be gone. Scaling of the video to fit the window is now being done by the CPU. The impact should be negligible. Should you experience playback slowdown with other videos that never had the Green Bar(TM) issue previously, you should go back into GStreamer’s properties and re-enable Xv as required.
  1. Retrieve popcorn and enjoy movie.

Subtitle Downloader

I’m a big fan of simple applications that fill a big void or annoyance in life, and one of those annoyances is finding suitable subtitle files (.srt files) for your movie rips. To fill this void is a simple application called SubDownloader that does its best to crawl the ‘net for suitable subtitle files for whatever movie you select in the file browser.

Download the application’s .deb file from the GetDeb website and install it with:

$ sudo dpkg -i subdownloader_2.0.9-1~getdeb1_all.deb

(or if you’re afraid of the command line, just double-click it within a window to launch the GUI version of the Package Installer instead – either way will take care of the QT4 and Python dependencies for you)

…and that’s it. now just fire it up from the Applications->Sound & Video menu, browse to your movie folder and let the app find the subtitles for you. Sweet! Smilie: :)

You don’t have to search by your collection, though – you can do a regular title search as well.

The selected subtitles are then downloaded into the same directory as your source video, allowing you to execute them in Totem or other subtitle-supporting player right away.

Simple, yet effective. This really should be in the Ubuntu repositories.