scomar

Encrypted storage on an external eSATA or USB drive with Kubuntu 10.04

It is very important to keep regular backups of your data. Photos, emails, documents, code.. your HDD could fail for a myriad reasons right _now_ and if you don’t have backups then it’s gone for good.

However what if your backup HDD gets lost or stolen? I personally would hate the idea that anyone who got their hands on my backup drive could see all my photos, read my resume, emails, receipts etc. Not only is it creepy, but it could also leave you open to ID theft.

The solution to both those problems is to do regular backups on an encrypted drive. This post will show you how to set up an encrypted eSATA drive which is hot-pluggable – just like your USB thumb drive.

NOTE: you can use a USB flash or hard drive, just skip step 6 which is only necessary for eSATA drives.

First a warning: this will irreversibly wipe all existing data from the drive you use, so make sure you don’t have anything on there that you want to keep.

Step 0:

Make sure cryptsetup is installed. It’s probably already installed on *buntu 10.04, but you can make sure it is by running:

sudo aptitude install cryptsetup

Step 1:

Now you have to identify the disk /dev/ entry which refers to your device:

sudo fdisk -l

You should see a list with an entry similar to this:

...

Disk /dev/sdd: 1500.3 GB, 1500301910016 bytes
81 heads, 63 sectors/track, 574226 cylinders
Units = cylinders of 5103 * 512 = 2612736 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x79177b3a

Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1      574226  1465137607+  83  Linux

...

My backup disk is easy to identify because it is the only 1.5TB drive in the list! If you have multiple similar drives and can’t tell which is which, you can get the /dev/ entry from dmesg. Just unplug the device, then plug it back in and run:

dmesg | tail

You should see something similar to this:

[ 5471.702033] ata5.00: configured for UDMA/133
[ 5471.702039] ata5: EH complete
[ 5471.702127] scsi 4:0:0:0: Direct-Access     ATA      WDC WD15EARS-00Z 80.0 PQ: 0 ANSI: 5
[ 5471.702277] sd 4:0:0:0: Attached scsi generic sg2 type 0
[ 5471.702309] sd 4:0:0:0: [sdd] 2930277168 512-byte logical blocks: (1.50 TB/1.36 TiB)
[ 5471.702368] sd 4:0:0:0: [sdd] Write Protect is off
[ 5471.702372] sd 4:0:0:0: [sdd] Mode Sense: 00 3a 00 00
[ 5471.702396] sd 4:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 5471.702515]  sdd: sdd1
[ 5472.182586] sd 4:0:0:0: [sdd] Attached SCSI disk

As you can see above, my device is /dev/sdd.

Step 2: Partition the device

Now we want to create a single partition on the device. If there are any partitions, we want to wipe them!

sudo fdisk /dev/sdd

Then delete any existing partitions and create a new one like so:

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').

Command (m for help): c
DOS Compatibility flag is not set

Command (m for help): u
Changing display/entry units to sectors

Command (m for help): p

Disk /dev/sdd: 1500.3 GB, 1500301910016 bytes
81 heads, 63 sectors/track, 574226 cylinders, total 2930277168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x79177b3a

Device Boot      Start         End      Blocks   Id  System
/dev/sdd1              63  2930275277  1465137607+  83  Linux

Command (m for help): d
Selected partition 1

Command (m for help): p

Disk /dev/sdd: 1500.3 GB, 1500301910016 bytes
81 heads, 63 sectors/track, 574226 cylinders, total 2930277168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x79177b3a

Device Boot      Start         End      Blocks   Id  System

Command (m for help): n
Command action
e   extended
p   primary partition (1-4)
p
Partition number (1-4): 1
First sector (2048-2930277167, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-2930277167, default 2930277167):
Using default value 2930277167

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Step 2.5 (optional, not recommended on a large drive): Select your level of paranoia

With my backups, I’m after a good level of protection. However I’m not trying to keep out the FBI. If you are, you may want to write random data to your partition before creating the encrypted volume. This may help protect your encrypted data from a ‘Bit-pattern based integral attack’ (research paper here).

You can fill the partition with zeros, test patterns, pseudo-random data or truly random data. The commands below are copied verbatim from here, and are only included for completeness. I have no idea how effective they are at making your data harder to crack, and I’m sure they will take a long time to complete on a 1.5TB drive like mine so I didn’t bother.

NOTE: If you know more than me about this and believe that skipping this step is a mistake, please let me know in the comments!

zeros:
sudo dd if=/dev/zero of=/dev/sdbX bs=4K
test patterns:
sudo badblocks -vfw /dev/sdbX [block-size-of-your-device]
pseudo-random data:
sudo dd if=/dev/urandom of=/dev/[your device] bs=4K
random data:
sudo dd if=/dev/random of=/dev/[your device] bs=4K

Step 3: encrypt the partition

These are my recommended encryption settings. Please read man cryptsetup, and cryptsetup --help for more information.

sudo cryptsetup luksFormat /dev/sdd1 -c "aes-cbc-essiv:sha256" -s 256 -h sha256

It will then ask you to set a passphrase.

WARNING!
========
This will overwrite data on /dev/sdd1 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter LUKS passphrase:
Verify passphrase:

Step 4: name the partitions

For example I have called my encrypted partition ‘SecureBackup’.

sudo cryptsetup luksOpen /dev/sdd1 SecureBackup

It will ask for the passphrase you just set:

Enter passphrase for /dev/sdd1:
Key slot 0 unlocked.

Step 5: create and label the filesystem

I have matched the filesystem label and the partition name, and have chosen ext4 as my filesystem:

sudo mkfs.ext4 /dev/mapper/SecureBackup -L SecureBackup

It may take some time to create the filesystem, but it will show its progress:

mke2fs 1.41.11 (14-Mar-2010)
Filesystem label=SecureBackup
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
91578368 inodes, 366284133 blocks
18314206 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
11179 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 29 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Step 6 (Only for eSATA devices! Not necessary for USB devices.):

NOTE: This step is likely to change in future releases of *buntu, as HAL is being retired in favour of DeviceKit / udev, however the timetable of this change is currently unclear.

USB devices will automatically ask for a password & give the option to mount when plugged in, however eSATA devices will not. To get your eSATA device to mount easily like a USB stick, you need the serial that HAL uses to refer to the device:

lshal | grep 'storage.serial ='

You should see something like this:

storage.serial = 'WDC_WD15EARS-00Z5B1_WD-WXXXXXXXXXXX'  (string)
storage.serial = 'STXXXXXXXXXXXXXXXXXX'  (string)
storage.serial = 'INTEL_SSDSA2M080G2GN_XXXXXXXXXXXXXXXXXX'  (string)
storage.serial = '0xXXXXXXXX'  (string)

If you can’t tell which device is yours, unplug it and then re-run the lshal command to see which one disappears. Mine is the first device in the list.

Then add the device information to HAL preferences so that it knows the device is removeable and hotpluggable:

sudoedit /etc/hal/fdi/policy/preferences.fdi

Add the following inside the <deviceinfo> tag in this file, changing the serial string to match your device:

<device>
    <!-- SafeBackup eSATA -->
    <match key="storage.serial" string="WDC_WD15EARS-00Z5B1_WD-WXXXXXXXXXXX">
        <merge key="storage.is_external" type="bool">true</merge>
    </match>
    <match key="storage.is_external" bool="true">
        <merge key="storage.removable" type="bool">true</merge>
        <merge key="storage.hotpluggable" type="bool">true</merge>
        <merge key="volume.ignore" type="bool">false</merge>
    </match>
    <match key="@block.storage_device:storage.is_external" bool="true">
        <merge key="volume.ignore" type="bool">false</merge>
    </match>
</device>

Step 7: permissions

Your backup storage is now almost ready to use. Just unplug and plug it back in, and it should pop up just like a regular USB device except it will ask for a password. Enter the password and mount the encrypted filesystem.

However, the file system is owned by root, not your user:

To fix this, you need to recursively chown the mounted filesystem (replace ‘username’ with your actual user name, and the path to your device if it is different):

sudo chown -R username:username /media/SecureBackup/

Conclusion:

You now have a secure backup location! Now you just have to remember to make regular backups. I recommend luckyBackup, a friendly GUI frontend to the powerful rsync.

References / Further Reading:

eSATA automounting instructions adapted from here.
Encrypted removable storage instructions adapted from here.
10.04 cryptsetup man page.

Lectopia rtsp stream capture IV

Another version of the script, this one attempts to get the address of the media from the address of the ‘Copyright’ window. It may work better or be easier to use than the script in Lectopia III. YMMV

getlect2.sh:

# extract the fid from the URL
FID=`echo "$1" | sed 's/^.*fid=\(.*\)\&cnt.*/\1/'`

# extract the base from the URL
BASEURL=`echo "$1" | sed 's/\(.*\)\/cast.*/\1/'`

# get the playlist file address
PLAYLIST=`wget -q -O - "$BASEURL/servepointer.lasso?FormatID=$FID" | grep rtsp | sed 's/.*rtsp:\/\/\(.*\)".*/\1/'`

# get the media file from the playlist file
MEDIA=`wget -q -O - $PLAYLIST | grep rtsp: | sed 's/.*src="\(.*\)".*/\1/' | sed 's/\r//'`

# get the date of the media file (to save file as)
DATE=`echo $MEDIA | sed 's/.*iLectures.\{10\}\(.\{10\}\).*/\1/'`

# actually rip the stream
mencoder $MEDIA -oac mp3lame -lameopts cbr:br=56 -ovc x264 -x264encopts bitrate=100 -rtsp-stream-over-tcp -o $DATE.avi

Steps to use:

• Select the highest quality quicktime stream from the dropdown menu in the Lectopia Recordings List and click the Open button.
• Copy the address of this window. If the address bar is hidden in your browser, you can get the address from the Page Info box. (shortcut in some browsers is Ctrl-I)
• run ./getlect2.sh ‘http://…’ (make sure you have quotes around the address as shown)

Lectopia rstp stream capture III

Quick update in response to a comment on my previous lectopia stream ripping saving script. Apparently they changed the playlist format slightly which was enough to break my script.
Anyway, this should work for the new format and the old format. (not entirely certain about the old format, since it’s no longer available)

#!/usr/bin/env bash

# remove the protocol from the front of the URL if if necessary
INPUT=`echo $1 | sed 's/.*:\/\///'`

# Snip the URL from the playlist file
URL=`wget -q -O - $INPUT | grep rtsp: | sed 's/.*src="\(.*\)".*/\1/' | sed 's/\r//'`

# Store the DATE from the file
DATE=`echo $URL | sed 's/.*iLectures.\{10\}\(.\{10\}\).*/\1/'`

# re-encode the file data, using the DATE as the name
mencoder $URL -oac mp3lame -lameopts cbr:br=56 -ovc x264 -x264encopts bitrate=100 -rtsp-stream-over-tcp -o $DATE.avi

Quick notes on usage:

• Copy the script above to a file eg. ‘getlect.sh’. Make the file executable with ‘chmod +x getlect.sh’.
• On the Lectopia Recordings List, select the highest quality Quicktime stream from the dropdown and click the open button.
• Click the ‘continue’ button after ignoring the copyright warning
• I have the the gecko-mediaplayer plugin for firefox installed under (k)ubuntu, and it gives a black screen here. You can right-click this black screen and select ‘copy location’ to get the playlist URL (it should start with “rstp://…”). If you have a different video plugin, you might have to inspect the frame source or something to get the stream address.
• Run the script ‘./getlect.sh rstp://…’

Note that it will take as long to ‘rip’ the stream as it does to watch it, because the server sends it at a set rate. However if you have a good internet connection you can save several in parallel by running the script in a few different terminal windows (I’ve had 4 going in parallel before).

Any issues with the script, or if you find it useful, let me know.

Simple, high-quality DVD ripping on Linux (Ubuntu 9.10)

Workflow: Rip DVD to HDD using libdvdcss2 & vobcopy, encode rip to compressed video file using handbrake and x264.

Handbrake can rip DVDs by itself, but only by reading from the DVD drive directly, which has several disadvantages (slower, cannot enqueue more encodes than dvd drives, dvd drive is in use for entire rip & encode).

I am assuming that you want the original AC3 audio. See the handbrake guide for other options.

DVD ripping on linux is now very simple all you need for high-quality rips are four tools:

• libdvdcss2 – library to decrypt commercial dvds.
• vobcopy – decrypt (using libdvdcss2) and dump the dvd to disc.
• handbrake – control all aspects of encoding.
• x264 – video encoder.

Part 0: Get the tools

Use the most recent version of x264 you can for best results, since improvements are regularly being made. I used rvm’s Personal PPA repository (author of the excellent SMplayer), at the time of writing since the ubuntu repositories had a very old version of x264. If you can find a more recent version in a trustworthy PPA or don’t mind compiling x264, use that.

Install libdvdcss2 if you haven’t already. Install handbrake from the handbrake PPA. vobcopy is in the ubuntu universe repository.

Part 1: Rip

#go to directory you are storing DVD rips
cd ~/Videos
# copy DVD to HDD (make sure DVD is mounted or vobcopy won't find it)
vobcopy -m

Part 1 can be done multiple times to rip multiple DVDs to your HDD if you want to set up several to encode overnight or something.

Part 2: Encode

Open your rip in handbrake (enable ‘Open VIDEO_TS folder’ option in the ‘Open’ dialog).

Select encoding options. I usually load the ‘High Profile’ preset, then check the tabs:

Video

• Set the quality you want, I use the default 61%. 60-70% is generally “transparent” for most people with a DVD source video (source), but do some tests to see what you like!

Audio

• Remove all audio tracks but the AC3 (pass-thru) track.

Subtitles

• Add if you want.

H.264

• Leave settings alone; these have been set by the ‘High Profile’ preset to the ‘best’ defaults. You can select other Presets to see the different settings in this tab, but this also resets the options in the other tabs to default – you have to go back and change them again.

Chapters

• Default is to have chapters, deselect if you don’t want.

Add to queue if you want to set up another encode, or just ‘Start encoding’. handbrake (ghb) is multithreaded and should max out all cores when encoding.

---

edited 15/3: use vobcopy instead of dvdbackup & dvdauthor. Also use constant quality over fixed size.

Lectopia rtsp stream capture II

A new semester, and some more hacking on the ‘dumpstream’ script. Now it is a little smarter because it can read the playlist file given by lectopia, and extract the rtsp stream URL from it:

#!/usr/bin/env bash

# Snip the URL from the playlist file
URL=`wget -q -O - $1 | grep rtsp | sed 's/.*src="\(.*\)".*/\1/'`

# Store the DATE from the file
DATE=`echo $URL | sed 's/.*iLectures.\{10\}\(.\{10\}\).*/\1/'`

# re-encode the file data, using the DATE as the name
`mencoder $URL -oac mp3lame -lameopts cbr:br=56 -ovc x264 -x264encopts bitrate=100 -rtsp-stream-over-tcp -o $DATE.avi`

Lectopia rtsp stream capture

Lectopia is used by my university to publish recordings of lectures. Many of the lecturers make downloadable copies of the lectures available. Unfortunately, some lecturers only make streams of their lectures available. This sucks because skipping back and forth in a lecture is extremely slow, and unlike youtube or similar, if you want to skip back and re-watch a section of the lecture, that section has to be downloaded again – no caching. The streams are not capturable using standard utils – it isn’t possible to rip the stream directly using either vlc -demux=dump or mplayer -dumpstream.

eg. mplayer -dumpstream gives:

unsupported RTSP server. Server type is 'QTSS/6.0.2 (Build/526.2; Platform/MacOSX; Release/Mac OS X Server; )'.
...
Cannot dump this stream - no file descriptor available.

Instead, if you want to make a copy of the stream you must re-encode it. Fortunately, this is easy with mencoder. Below is a script I wrote, which takes a command line argument of the stream address and saves the file using the first 10 characters of the stream name – which is usually YYMMDDTTTT (year, month, day and time).

#!/usr/bin/env bash
DATE=`echo $1 | sed 's/.*iLectures.\{10\}\(.\{10\}\).*/\1/'`
`mencoder $1 -oac mp3lame -lameopts cbr:br=56 -ovc x264 -x264encopts bitrate=100 -rtsp-stream-over-tcp -o $DATE.avi`

The default video bitrate of 100 kbits/second is fine for “slides+voiceover” lecture recordings and results in ~1MB/minute filesize (with the audio bitrate of 56). You may want to use a higher video bitrate if you’ve got more action on-screen. I set up 6 instances of at the same time and yanked

Hopefully someone will find this useful.

Links like this…

When I first started using Vim, I watched a fascinating talk given by the author, Bram Moolenaar, titled “7 Habits For Effective Text Editing 2.0” (click the link to see the presentation. It’s 80 minutes long so maybe grab a coffee first ). One of the ideas he talks about, and something which seems to be a pervasive philosophy of Vim, is the “Three basic steps”:

1. Detect inefficiency
2. Find a quicker way
3. Make it a habit

I think this is an excellent philosophy for computing in general – why do something repetitive yourself, when the computer can do it much more quickly and accurately? An obvious (and trivial) example is “search & replace”, which saves you from having to scroll through a file to find all the occurences of a string, then manually type in its replacement.

Since watching that talk, and as I got more familiar with Vim, I started “detecting” such inefficiencies everywhere; often in apps which are not nearly as flexible, which usually makes these inefficiencies more frustrating because they are harder to solve. I had one of these moments while browsing stackoverflow today.

Stackoverflow is a great resource, with lots of good information, however there is so much discussion that you could never read it all. That’s why every now and then I like to browse the Hottest Questions This Week, which does a pretty good job of distilling out the good stuff.

Now, I like to browse all those articles in full because it’s difficult to tell from the question whether or not it’s going to have one of those “gem” answers. So I started middle clicking all the links down the page to open them in new tabs as I normally do, when I suddenly realised how inefficient I was being. If only there was some kind of macro that could open all these links for me…

Well as it turns out, there’s an extension for that! It’s called Links Like This and does exactly what it sounds like it should. Just right click on a link, select “Open Links Like This…”, and all the similar links on the page are highlighted, with the option to open them all presented in a very slick interface. I’m not sure what heuristics it uses to determine which links are alike, but it does exactly what I want it to do: open all the “Hottest Questions This Week” in new tabs, saving me a full minute of my life and 48 mouse clicks. It even differentiates between visited links and unvisited links, so if I view the page twice in one week, it can open only those links that I haven’t visited! Awesome .

This illustrates to me a couple of things. Firstly, for repetitive computing tasks there is always an easier way. I wonder where else am I doing something manually which is more suited to some clever code?
And secondly, this is the power of open source software. Firefox’s value lies in it’s openness and the ease of writing extensions. There is no other browser which can compete with firefox’s features, because whenever someone sees an inefficiency or wants to add a feature, mozilla has made it simple for them to code a solution and share it with the world. I doubt very much that any other browser (especially a proprietary one like IE or Opera) would add support for a feature like this.

That’s also why I think KDE’s plasma is so important and exciting: it’s doing for the desktop just what firefox is doing for the web. Plasma widgets are really just “extensions” for your desktop, and you can code them in Javascript (just like a firefox extension), as well as Python, C, C++ and Ruby. Anyway, enough of a braindump. Back to work.

git: simple!

I’ve been working on a project at home, using git for source control, and needed to make the project available from anywhere, so that I could work on it wherever I needed to. I have an ssh account at the university, so I decided I’d try setting up a git repo I could access through that. I was expecting it to be complicated and difficult, but in fact it was incredibly simple.

Google found these instructions, which I based my setup off; most of what’s below is quoted from that article, but I’ll put it here anyway for reference.

# login to remote server
ssh -l myUsername REMOTE_SERVER

# once logged in
mkdir /path/to/example.git
cd /path/to/example.git
git init

Note: the link above recommends

git --bare init

in the last line. using the –bare switch will create a repo without a working tree, which is great if you don’t need to work on the code on the machine where the repository is housed. However I need to work on the code on that machine too when I’m at uni.

Then, on my home machine I just run:

git remote add origin myUsername@REMOTE_SERVER:/path/to/example.git
git push origin master

The first command adds a the remote repository we just created and names it ‘origin’. The second command then pushes your ‘master’ branch to the remote repo.

And that’s it! You can now push/pull/clone from this repo to your heart’s content. For example after I work on the code at uni then want to work on it at home with those changes I just do:

git pull origin master

Simple!

vim + konsole

I’ve recently been teaching myself vim, and it’s amazing how much faster coding is already! Just moving around a document is so much more efficient when you don’t have to resort to a mouse.

I’ve been using the non-gui version of vim in konsole, partly because I can’t “cheat” with the mouse this way, and partly because it’s so much cooler having a “native” kde-style vim that fits in with font styles of the rest of my system automatically.

Update: I should have guessed it was possible given vim’s flexibility, but enabling mouse support is as simple as:

set mouse=a

Still, I’m leaving it disabled for learning purposes.

Anyway, I’ve spent some time learning how vim and konsole can work together, so I thought I would document it somewhere.

First, how do you get cool gvim colour schemes working in konsole? Get CSApprox, a very clever little vim script that will translate a colour scheme designed for gvim, and approximate it in the 256 colours available to you in the konsole.

For vim + konsole, you also have to tell vim that it can use 256 colours (by default it only thinks it can use 8, thus colours won’t work properly). So add to your ~/.vimrc:

set t_Co=256

Then I found an awesome theme called darkspectrum, and now my vim experience really rocks.

Another advantage of konsole over gvim is that fullscreen vim is only a <ctrl> + <shift> + <f11> away. Sweet!