Archives February 2009

I was asked to rebuild a friend’s old Medion laptop – you know, those ones Aldi used to sell for peanuts. It’s an AMD Turion64 based machine with NVidia GeForce Go 6100 gfx and SigmaTel STAC 9200 “High Definition” audio and a 1280×800 display. I have to admit it’s not a bad little machine.

Anyway, the default installation of Ubuntu Intrepid picks up everything except the Fn keys, wireless LAN adapter and the audio. The wireless LAN adapter has proven to be a bit of a challenge, so if I suss it out, I’ll write up about it later. Fn keys I’m not really fussed about (and neither is the laptop’s owner), but we needed the audio.

The SigmaTel STAC 9200 has a dotted history of frustration among Linux users in general, but thankfully it was fixed in Kernel 2.6.27-7 and Alsa 1.0.15. Unfortunately a vanilla installation of Ubuntu it still is unable to autodetect the hardware 100% properly (it will find it and name it, but all you get is silence), so you have to make one tiny manual adjustment before sound will work properly.

1. Open up a terminal and type the following at the $ prompt:
   
   $ sudo gedit /etc/modprobe.d/alsa-base
   
2. This opens up the file in the GEdit text editor. Scroll to the end of the file, hit Enter to make a new line and add the following:
   
   options snd-hda-intel model=gateway
   
3. Save your changes and close the editor.
   
4. Now reboot. When Ubuntu comes back, you will have working audio!

Note: Reader Mouhcine has suggested that if “model=gateway” does not work for you in step 2, try using “model=gateway-m4″ which may give you more success in Ubuntu Jaunty.

Whilst rebuilding a friend’s Medion laptop, in my attempt to get the Wireless LAN adapter working, I set about trying to use ndiswrapper and the Windows drivers. Unfortunately upon rebooting, the system failed to boot, always locking up when the boot process tried to load the Windows driver. It was so bad that I wasn’t even able to boot to a recovery prompt because it still attempts to load the hardware drivers before dropping you into a root shell.

The solution was simple – get rid of ndiswrapper and that will prevent the offending Windows driver loading which I can then delete afterwards, but how do you do this when you can’t even boot to a terminal?

With the assistance of an Ubuntu LiveCD (on USB stick in this case), I was able to remove ndiswrapper without needing to do a complete re-install of the system. Here’s how to do it.

1. Shutdown your system, insert your Ubuntu LiveCD and turn on your system.
   
2. Allow the system to boot to the LiveCD desktop and then open a terminal.
   
3. Determine what your system’s root partition is called by getting a list of available drives and partitions with:
   
   $ sudo fdisk -l
   
4. In my case, this Medion lappy had Windows installed and a couple of other partitions too. The root Linux partition ultimately resided on sda8, so I need to mount this somewhere. First I need a mountpoint:
   
   $ mkdir /dev/shm/medion
   
   …this creates a directory in the RAM disk called “medion” (but you can call it whatever you want).
   
5. Now mount the partition to the mountpoint with:
   
   $ sudo mount /dev/sda8 /dev/shm/medion
   
6. Check that we can access the mounted drive with:
   
   $ ls -l /dev/shm/medion
   
7. If it contains the root filesystem of your lappy’s Ubuntu install, then you’ve done well so far. Now we need to change the system’s root filesystem from the LiveCD over to the hard-drive’s root filesystem so we can work on it. Type in:
   
   $ sudo chroot /dev/shm/medion
   
8. This will temporarily make the current session’s root filesystem the one that is on your hard-drive, as though we’d actually booted from it. From here, it’s now a simple case of removing the ndiswrapper application that was causing all my problems (notice that we are at a root prompt designated by the hash symbol):
   
   # apt-get remove ndiswrapper-common
   
9. A few warning messages popped up advising that it couldn’t find the log to write to, but that’s OK – the software still gets removed, and thus will prevent the Windows driver from killing the boot process. Once Apt had finished doing its thing, just type in:
   
   # exit
   
   …and the root filesystem will revert back to the LiveCD’s root filesystem.
   
10. Shutdown and reboot as normal without the LiveCD, and viola – the system boots without hanging this time!

As you can see, the ability to change root filesystem can be very useful in trouble-shooting a non-booting installation but also has other uses too, one of the most common being creating custom LiveCD’s by modifying the squashfs filesystem on the LiveCD and burning to a new disc – thus allowing you to add or remove components from the default Ubuntu LiveCD environment.

Those of you who archive or backup their Blu-ray movie media to hard-drive will already be aware that the average movie comes out at a good 25GB. Some of the bigger titles top out at around 40GB or more. This eats up an awful lot of disk space.

Blu-ray titles are already compressed down using the MPEG2 codec, and quality pundits will abhor the idea of re-compressing the title again for fear of losing image and audio quality. Certainly if you go down the Xvid route, you will definitely lose image quality, but as per my previous DVD HowTo, you can do excellent rips with virtually indistinguishable quality to the original using the x264 codec, and have a significantly smaller footprint to go with it.

The process of encoding a Blu-ray rip isn’t quite the same as doing a DVD, however, so here’s a quick guide on how to take your decrypted .m2ts file and finish up with a much smaller, but 99% perfect copy in a Matroska .mkv file.

Pre-requisites:

• A pre-decoded Blu-ray movie file (.m2ts file).
• Approximately the same amount of free disk space as the size of the movie file. Eg: If you have a 25GB movie file, then you should have another 25GB free space to work with. You can have less, since the final resulting file will be much smaller than the original movie anyway, but since this process can take a number of hours to complete, you don’t exactly want to discover you ran out of disk space and have to start over, do you?
• A nice powerful CPU. I use a Intel quad-core Q9450 CPU at 2.66GHz. It takes my machine roughly 9-12 hours to process just one movie using four threads. A dual-core will take longer.
• Time to let the PC do its work, eg: overnight.

This guide was written using Ubuntu 8.10 Intrepid Ibex 64-bit, but will work quite happily in 32-bit and should also work with most previous versions of Ubuntu.

1. You will need some extra software installed if you haven’t already got it. Open a terminal and type in the following at the $ prompt:
   
   $ sudo apt-get install mencoder mplayer gpac x264 mkvtoolnix
   
   (Don’t worry if you’ve already got some of those apps installed, Ubuntu will skip over them if they already exist on your system)
   
2. Create a new text file somewhere using your favourite text editor, eg:
   
   $ gedit ~/encodevideo.sh
   
   …will create a new text file called “encodevideo.sh” in the root of your Home directory using the GEdit text editor.
   
3. Now copy and paste the following script into it:
   
   #! /bin/bash
# =====================================================================
# Blu-ray encoding script by HyRax February 2009 http://www.serenux.com
# =====================================================================
# Make sure the user has specified what to work on.
if [ -z "$1" ]; then
echo "\nBlu-ray movie encoding script\n-----------------------------"
echo "Written by HyRax February 2009\nhttp://www.serenux.com"
echo "\nUsage: $0 <m2ts file without extension>"
echo "\nExample: If your movie file is called TheDarkKnight.m2ts then\nyour usage will be: $0 TheDarkKnight\n"
exit
fi

# The crf=21 option controls encoding quality, and indirectly the final
# filesize. The higher the value, the more compression and thus smaller
# file size. Reduce the value and file size will go up. A value of 21
# should produce a file of approximately 4GB in size for a typical movie.

# Encode the video using x264, ignore the audio for now.
mencoder $1.m2ts \
-ovc x264 -x264encopts crf=21:frameref=3:bframes=3:b_pyramid=normal:direct_pred=auto:weight_b:partitions=all:8x8dct:me=umh:mixed_refs:trellis=1:nopsnr:nossim:subq=6:level_idc=41:threads=4 \
-nosound \
-of rawvideo \
-o $1.x264

# Dump the first original audio track (should be the English track) but
# don't re-encode it. Ignore the video.
mplayer $1.m2ts -dumpaudio -dumpfile $1.ac3

# Copy the raw x264 encoded video into an MP4 container so we can set
# the correct framerate (generally 23.976 - adjust it if MPlayer or
# MEncoder report something different)
MP4Box -add $1.x264 $1.mp4 -fps 23.976

# Finally, merge everything into a single MKV file
mkvmerge -o $1.mkv $1.mp4 --track-name 0:Eng $1.ac3

# Remove the hash in front of the next command to have the script delete # your working files when encoding is complete.
# rm $1.m2ts $1.x264 $1.ac3

# Tell the user we're done.
echo "All done! Your final movie file is called $1.mkv - enjoy!"
   
4. Save and exit your text editor.
   
5. Change directory to where you have your original .m2ts file, eg:
   
   $ cd ~/Videos/BDRips/MyMovie
   
6. Let’s say your movie file is called TheIsland.m2ts. To begin encoding it, you will use the following command:
   
   $ sh ~/encodevideo.sh TheIsland
   
   (Notice that we don’t specify the file extension – the script assumes .m2ts on the end already)
   
7. Hit enter and the encoding process will begin. The script does the following in order:
   1. Extract the video component only and encode it in a single pass to a raw x264 video file called TheIsland.x264 (in this example).
      
   2. When that has finished, go back and extract the first audio track only and save it to a file called TheIsland.ac3 without any re-encoding.
      
   3. When that has finished, take the raw x264 video data and put it into an MPEG4 container so we can fix the framerate at 23.976 frames per second, typical of most Blu-ray movies. If you don’t do this step, then the video will play faster than the audio in the final product.
      
   4. Finally we create a new Matroska container called TheIsland.mkv and in it we place the framerate-adjusted MPEG4 video track and the unmodified audio track, producing our final product..
      
8. When the script has finished some 9-12 hours later, you will have a file called TheIsland.mkv ready for playback in Totem, MPlayer, VLC, or whatever your favourite player is. You will notice that the filesize is significantly smaller than the original but when you play it back, the image quality will look pretty much 100% identical to the original. About the only compression artefacts you may notice is around the edges of some text titles such as opening credits, but you’d really have to look hard to spot them.
   
9. We’ve finished with the working files now, so you can delete the .m2ts, .x264 and .ac3 files now to reclaim the disk space and enjoy your new .mkv file.

Enjoy! 

NOTES:

• While not covered by this HowTo (I’ll add to this later), you can optionally extract other audio tracks from the .m2ts file such as other languages or the commentary track, and add them to the .mkv file at any time. You can also download subtitles and add them to the .mkv file at a later date also, all without needing to re-encode the entire video again because all you’re doing is adding tracks to the existing Matroska container. see the man pages for the mkvmerge utility for more information.

• A high-definition x264-encoded movie takes a fairly reasonable amount of CPU power to decode. Since Ubuntu Intrepid and previous versions do not make any direct use of on-board video decoding hardware such as that found on NVidia and ATi based video cards, you may find certain busy scenes in your movie may stutter or even skip altogether during playback – this does NOT mean you are missing data from your movie file – it’s just that your PC is having trouble decoding AND displaying the movie all at once because the CPU is doing everything, especially those with slower CPU’s. Indeed you will notice that your CPU usage will probably be quite high in the 80-90% region. Ubuntu Jaunty will bring forth with it VDPAU support for NVidia-based video cards (restricted NVidia driver v180 and above). Using a patched version of MPlayer with VDPAU support, all the decoding work is passed completely to the video card, freeing up your CPU considerably to concentrate on other tasks, and dropping utilisation rates down to about 3%. This makes busy movies that much more watchable. If you can’t wait for Jaunty, there are some backported VDPAU modified versions of mencoder, mplayer and even MythTV available here, however you try them at your own risk. If you are an ATi video card user, you’re out in the dark for the moment. Go buy yourself an NVidia card – they are much better supported under Ubuntu than ATi are.
  
• If you are using only a dual-core processor, you should modify the threads=4 section of the mencoder line in the script to read either threads=2 or threads=3. If you are using a single-core processor, change this to threads=1 or threads=2 depending on how your CPU performs (and while you’re at it, seriously consider an upgrade!).
  
• If you wish to modify the quality of the encoded video to make the resulting file larger or smaller, re-edit the encodevideo.sh script and change the crf=21 value in the mencoder line to a different value. There is no definitive filesize that the resulting encode will have. This value simply adjusts the quality of the encode. With some trial and error, I have found that the average 1080p movie encode ends up roughly 8GB in size when using a value of 21 and provides excellent image quality. If you make the value larger, this will apply greater compression and will reduce the final file size at the expense of losing some image quality. If you reduce the value, then your final file size will increase, however your image quality will also go up. As a guide, animated movies such as those made by Pixar and Dreamworks and visually dark movies such as Underworld, compress very well. In one instance, the final product was only 3.5GB in size. Visually complex movies such as Transformers blew out to 12GB in size using the same encoding script, so you can see that there is no exact science to this. There are mencoder options to specifically set bitrate and target filesize, but I chose to ignore those options for this HowTo as I’m a bit of a quality freak, not a size freak. I store all my movies on a MythTV server at home and backup to an external drive as I’m not a fan of doubling up my movie purchases on the shelf with a second disc containing the compressed version of the same movie!

My site’s been offline for most of the last 24 hours due to an unexpected power failure caused by a constant week and a half of rain shorting underground mains power lines, and taking half of my street with it. I have to admit I’ve been a bit complacent and have never acquired a UPS for the server that serves this blog!

But the power’s back now, and meantime I’ve gone and bought a shiny new UPS to keep the server running should there be any future power issues. Sorry the inconvenience, folks!

Some of you may be aware that Unix systems and its variants including Linux, Apple OS X, etc store the time in seconds using a 32-bit integer, with the 1st January 1970 as the epoch.

Tomorrow morning at 10:31 and 30 seconds, eastern Australians (except Queensland who will get it at 09:31 and 30 seconds) will experience 1234567890 Unix time. Where will you be on this momentous occasion?

If you’d like to check your own local time, simply drop into a terminal and paste this line of perl in:

$ perl -e 'print scalar localtime(1234567890),"\n";'
Sat Feb 14 10:31:30 2009
$ 

After that, we only have the 32-bit Unix Millennium Bug to look forward to in 2038!