Archiving data files to DVD

IBM has been using Lotus Notes as its email platform for the last 14 years or so, and during those years I’ve seen the size of emails increasing dramatically, mostly due to the ease with which people can attach large (and often unnecessary) attachments. IBM meanwhile would like us to minimise the size of our mail databases to minimise the cost of storing all this email across the corporation. However, at the same time, keeping many of these emails so they can be referred back to is very useful. Consequently I’ve been archiving all my email from my mail database to my local mail archive database. Unfortunately, the larger that database has become, the slower it becomes for certain classes of operation, such as opening or searching it. The actual file that represents that database also starts to become very unwieldy for the operating system; my email archive database was nearing 6GB in size.

I recently decided to do something about it, and split that single large mail archive database into a series of smaller archives, each covering a two year period, limiting the size of each to a GB or less.

I currently back these up to my NAS, but I’d like to have extra copies somewhere less accident-prone than spinning media. A couple of sets of DVD’s, preferably stored “offsite” from where my NAS and laptop are kept (ie, at work, not at home) seems like a good extra insurance policy. Which leads me to the question of how to burn some very large files to DVD under Linux.

It turns out that Ubuntu comes with dvd+rw-tools already installed, which provides command line utilities to do everything that is required.

dvd+rw-format -force /dev/dvdrw Formats the DVD-RW, allowing it to be used. Doing this too often can seriously shorten the life of your DVD-RW media, and it isn’t necessary to format before each reuse – just overwrite the old contents with new data.

growisofs -Z /dev/dvdrw -R -J /etc/*.conf Initialises and writes an initial set of files to the DVD-RW. If there is already data on your DVD-RW, this will overwrite it.

growisofs -M /dev/dvdrw -R -J /var/log/*.log Adds some more files to the DVD-RW. On a DVD-R I believe this will create another session, and write the data into that.

Note that with DVD-R media you skip the formatting step, then use the same growisofs commands as with DVD-RW media, EXCEPT it appears that you can’t easily add files to an existing DVD-R without creating a multisession DVD-R, which looks like it could be a heap of pain. I figure it’s probably best to just save all the required files in a single write.

For maximum compatibility, close the DVD-RW off with growisofs -M /dev/dvdrw=/dev/zero once all the files have been written to it, and just for completeness, it’s possible to get detailed technical information on the contents of any optical disc, by using the command dvd+rw-mediainfo /dev/dvdrw

Note that in all cases, these commands must be run by a “normal” (non-root, non-sudo) user, and the DVD media must be unmounted at the time.

Mobile broadband on Karmic Koala Alpha 6

Today I decided to try out the latest code drop for the next release of Ubuntu. This will be released as Ubuntu 9.10 next month, under the codename Karmic Koala. There are some rough edges, but so far I am extremely impressed. This release has moved to some very new technologies, and yet even at this early stage almost everything is basically working.

Under the current version of Ubuntu getting my USB mobile broadband modem (a ZTE MF627) working was quite awkward. It required quite a lot of configuration work, and the installation of some additional programs. Whereas under this newer version it is almost (but not quite) working out of the box. One simple configuration file needs to be added, called /etc/udev/rules.d/ZTE.rules, containing the single line:

SYSFS{idVendor}=="19d2", SYSFS{idProduct}=="2000", RUN+="/usr/bin/eject %k", OPTIONS+="last_rule"

Once the system has been rebooted (or udev restarted), inserting the USB modem causes the udev system to automatically detect it, and switch it to the correct mode, at which point Network Manager recognises the broadband modem becoming available, and allows you to configure it with your provider details. From then on you can connect by simply selecting the mobile broadband connection in Network Manager.

All very simple. As it should be.

Hampshire Linux User Group meeting follow-up

The non-working Vodafone SIM problem has been resolved. It turns out that Vodafone don’t simply have contract mobile broadband and PAYG mobile broadband, they also have “Top Up and Go” mobile broadband. Which seems to be some sort of PAYG mobile broadband with some port blocking, implemented to stop you using VoIP or P2P services.

Anyway, that’s the SIM that I have.

Knowing that, a simple google on the settings of the various offerings shows that the APN, userid and password are different from the normal PAYG offering. Changing my connection properties in Network Manager to use the correct settings resulted in an immediate connection to the internet @ 3.6Mbps, though admittedly I could see the Vodafone mast from where I was connecting.

My next problem is to work out how to top up my credit as I use the service. Vodafone appear to have set this up so everything is done through the software that they supply to make their dongle work on Windows. Which of course, I don’t (and can’t) use, as I’m running under Linux. I was really hoping that I’d be able to check my credit & top it up via the Vodafone website, but it’s looking like it’s pretty difficult to register the SIM on their website unless you’re running their desktop software, which requires Windows. And if you’re not running Windows there is very limited support, which is rather frustrating.

I’ll keep working on it for a while longer, but it’s starting to look like I might have to install Windows and their dashboard software on a spare partition just to get the SIM registered on their website, which is not exactly ideal.

Update: I finally managed to work out how to register the SIM on the Vodafone website so that I can check my credit balance, and top up the credit on the SIM using my credit card. The website is quite large and complex, and there seems to be more focus on advertising and upselling of products to their customers than on the usability of the core function of managing existing products, which didn’t help much. Still, I got there in the end.

As a Linux user, wanting to register a “TopUp and Go” SIM without resorting to Windows, this is the process I suggest you follow:

• Register for “My Account” on the Vodafone website. This simply gives you an account on their website, but doesn’t associate any products with your account.
• Now put your data SIM into an ordinary (Vodafone) mobile phone.
• Enter *#100# and press send. The network will now tell you what mobile phone number is associated with your data SIM.
• Login to your account and register a “Pay As You Use” phone. The website will ask you to enter the phone number of the phone. Enter the phone number you obtained in the previous step.
• The website will now send you an SMS message, containing a secret code. You can read this on your mobile phone.
• Enter the secret code back into the website (proving you own the SIM you are registering), and the website will ask you for the remaining information it needs to associate the SIM with your account. It automatically works out that you have a “TopUp and Go” data SIM rather than a standard PAYG mobile phone SIM.
• You can now see your credit limit by logging into your account.
• You can also associate a credit card (or your bank details) with your “My Account”, and use it to top up your SIM as required.
• Remember to put your data SIM back in the USB dongle, and your voice SIM back in your mobile phone.

Recovering recordings from dead Pace Twinview PVR

A couple of years ago the kids discovered the benefits of our PVR, and suddenly (a) there was no space on the PVR, and (b) we could never find anything we recorded amongst the zillions of recordings of CBeebies, CBBC and CITV. And then we inherited an old Pace Twinview PVR from my father-in-law, who had traded up to a Humax 9200. So I set it up for the kids on “their” TV. This meant that they could record whatever they wanted, without filling up the PVR in the lounge. And for a fair time, life was good.

And then the Twinview started playing up, and eventually died. And suddenly I have three kids who want me to recover all the recordings that they’ve been making.

No problem I think – those recordings are probably just recorded directly off air as transport streams (.ts files) which I can easily transform (using ffmpeg) either into something like H264 video and MP3 in an mp4 container, which they can then watch on the PS3, or an MPEG2 file which I can then make into a standard DVD. So I say not to worry, I’ll fix it for them.

Which was hasty. Very hasty. And possibly a big mistake.

So I extract the hard drive from the PVR. Good news, it’s a simple 20GB PATA laptop drive which will fit nicely into my Thinkpad ultrabay. So I boot up Ubuntu, and do a quick “fdisk -l /dev/sdb” to discover that there are three partitions, all of which are unrecognised by fdisk. They are however flagged with partition identifier 0xE0, which after a bit of Googling turns out to be a completely proprietary filesystem, designed by ST microelectronics, called ST AVFS – presumably the ST Microelectronics Audio Visual FileSystem.

So currently I’m struggling to find a way to get at the data. It’s not possible to just mount the partitions, but it turns out that there has been some work done on some linux command line tools (TwinRIP) to extract the data from those partitions. However, the tools are at least two years out of date, supplied in binary form only, and no longer run under any of my Ubuntu installs (lots of problems with missing libraries). Now, it transpires that there is a GUI program to do much the same under Windows (TDU), which looks to be more recently updated, and (plus point) can directly produce MPEG output files. It will be interesting to see if that runs under Windows 7 RC, which is the only Windows install I currently have on my Thinkpad!

Update: And the results are in. The TDU program doesn’t work any better than the Linux one. And Windows 7 doesn’t want to talk (reliably) to the hard drive, whether mounted using my Thinkpad ultrabay, or a USB caddy. Worse, the author of TDU & TwinRIP has not published any source code, so there’s no chance of my doing anything geeky at this point, so I’ve declared defeat, and told the kids that their old recordings are now officially lost

The only upside is that this gives me another free 20GB PATA laptop drive to play with, if only I can think of a use for something so small.

Tuesday spent “tinkering”

Yesterday I decided it was time to work out exactly what is going on with my home server. This was one of those little jobs that I’d decided wouldn’t require much physical effort, and that could be picked up and put down again as required when I felt tired. In short, ideal for my current situation.

The main problem is that I have three large SATA drives in the server, that I want to run as a RAID 5 array, but the motherboard I am using (in common with almost all other mini-itx motherboards) only supports two SATA drives. Originally I tried to work around this by using a PATA to SATA converter, but found that it resulted in some odd errors and what appeared to be data loss. So I bought a PCI card with a pair of SATA ports, based around the SiL3124 chipset, to provide me with support for a total of four SATA drives. However, any drive connected to that PCI card started showing errors whenever the I/O rate went above some arbitrary level, making it completely unsuitable for inclusion in a RAID array.

So yesterday I replaced the SiL3124 PCI card with another based on the VIA VT6421, which provides 3 x SATA ports and an IDE port, and since I had the server in pieces, took the opportunity to add an IDE to Compact Flash adapter, along with an 8GB Compact Flash card.

My intention was to be able to install the operating system onto the Compact Flash card, and then use the 3 x SATA drives purely as a RAID 5 array for data storage. By installing the operating system onto the memory card (with no moving parts that wear out) I would minimise the chance of any failures, making the system more reliable. The RAID 5 array would provide redundancy for my data storage, allowing a drive to fail without impacting my data.

However, in the tradition of plans laid by mice and men, when I got everything updated, the drive connected to the new PCI card still exhibits strange I/O errors. And swapping the drives around still leads to errors only from the drive connected to the PCI card. Which leaves me with the frustrating conclusion that it’s probably something to do with the implementation of the PCI bus on the motherboard – which is not what I was hoping for.

This somewhat Frankensteinian server has worked well for the last couple of years, but I’m starting to think that given how much of my home now depends on it working smoothly, it may be sensible to actually buy a new server designed specifically for the task, rather than wasting precious time chasing down obscure hardware issues in my DIY creation. At the moment, this one looks just about perfect (apart from the price … gulp …) would allow me to reuse my existing drives, and can probably even be made to boot from my compact flash card still. Decisions, decisions!

Wireless broadband internet – fixed

An exchange of emails with the friend who has lent me his 3G USB modem revealed that there is no PIN number on the SIM. However, apparently these things can be a little awkward, and error messages can be obscure, especially when there is no 3G or GPRS signal.

Reassured that I’m not about to lock myself out of the network by getting the SIM PIN number wrong too many times in a row, I did a little debugging. The modem may be recognised as a USB device, but it most definitely is not recognised as a modem. A quick check on my kernel configuration parameters reveals that I’ve not included a couple of crucial drivers.

15 minutes recompiling and reinstalling my kernel, insert the USB stick, and Network Manager cuts in, and establishes a 3G connection to the Internet. It really was as simple as that.

So expect updates from my hospital bed in due course

Wireless broadband internet

As I’ve already mentioned, I’m hoping to be able to take my Aspire One into hospital with me. Assuming that I’m well enough, then from my perspective, it will be the ideal distraction from boredom. I’ve got 20GB of music on it, and I’ve just been given another 10GB of plays and drama etc that have been recorded from the radio. I’ve even got a digital TV/radio tuner, which ought to let me watch a bit of live TV if I want.

However, what would be really cool would be to be able to continue updating this blog from my hospital bed. And its now looking like that might be possible, as I’ve just been loaned a Vodafone USB 3G modem for the time that I’m going to be in hospital – thanks Andy!

Unfortunately, it doesn’t work with the custom kernel that I’m currently running to help improve the performance of the One. So it looks like I will need to spend some time tweaking the configuration of my custom kernel again. In addition, when I switched back to the “normal” kernel, the modem still doesn’t work, failing this time at the request for the SIM pin.

So, I guess I need to resolve the kernel compilation, and find out either what the pin code is for the SIM card, or what I’m doing wrong…

Internet TV

A colleague pointed me at get_iplayer the other day, and I spent a couple of hours this weekend playing with it. Without a doubt it is the easiest way I have come across for getting hold of content from the various UK “play it again” services for TV and Radio.

BBC TV programme downloads are in H264 video and AAC audio, and although they are not high definition – more like good VHS quality – the convenience is spectacular. I sucked down a couple of programmes from the BBC; an episode of Being Human, and an episode of Doctor Who, which took about 15 minutes each (on my current 2Mbit connection). Both were downloaded as Quicktime .mov files because the feed that get_iplayer uses is designed for the iPhone.

I then converted them to .mp4 containers (using the command ffmpeg -i input.mov -vcodec copy -acodec copy output.mp4) and stored them on my NAS. From there, they were streamed (using the Mediatomb DNLA server) to my PS3 over my wireless network, which was able to play them back onto my flatscreen TV. And it worked beautifully.

Except for the stuttering. And the occasional unexpected pause. Turns out that despite my PS3 seeing 85% WiFi signal strength, it can’t consistently transfer more than about 0.5Mbit/s over my home wireless network. Which is not enough for video streaming. A quick check with a long Cat5e cable shows no signs of stuttering or pausing, so this is clearly a problem with the wireless drivers in the PS3. So it looks like I will need to hurry along my plan to drop some more ethernet lines to the back of the TV area of the lounge. Except of course, that isn’t going to happen for a few months now

However, notwithstanding the networking issues, the potential here is great. I can write some webpages on my NAS that I can access from my PS3, on the flatscreen TV in the lounge. That will run get_iplayer and return a list of available programmes, which I can select to download and have stored on my NAS, where they will then be converted to .mp4 format which I can stream to the PS3, so I can watch them. When my fast connection comes online (hopefully tomorrow!) that should take no more than 2 or 3 minutes for an hour-long TV programme.

Internet TV … almost on demand.

Living on the bleeding edge

As detailed in an earlier post, my Acer Aspire One has been doing a sterling job, replacing my larger, more powerful, but MUCH less portable Thinkpad.

And it occurred to me that the One might well be the ideal machine to take into hospital with me so I can listen to my entire music collection. And then I thought how it would be nice to also be able to listen to the radio. Or watch TV. And I remembered that Hauppauge make really good USB digital TV tuners, which can also receive radio. It was too much for a geek to resist, so a day later I was the proud owner of a Hauppauge Nova-T USB2 stick.

Only to discover that the highly optimised pre-compiled custom kernel that I was running (to keep the performance high) on the One lacked any support for TV tuners. Ooops. So all I need to do is produce a new kernel thats as efficient as my current one, but includes support for the TV tuner.

The last time I tried to compile a linux kernel was back in the days of SUSE 7, and I failed. Big time. So big time that I managed to also destroy the machine I was trying to compile the kernel on. I’ve avoided compiling my own kernels ever since. But hey, time heals all wounds … I’m smarter now, and much more linux-savvy. How hard can it be?

And the answer is, it’s a complete breeze. I plugged in all my hardware, and ran autokernconf, which worked out what kernel options I needed for my current hardware. I then merged it with the kernel config from my pre-compiled kernel, and sorted and uniq’ed it. A little bit of tweaking, and some judicious use of “meld” to check my new config against the original, and then I then downloaded the latest 2.6.28-7 kernel source, imported my kernel config, brought it up to date with newer config options, and ran the compile. All in my own home directory, without the need for root, or messing about in /usr/src. And the output was a pair of .debs that could be easily installed on any machine with next to no effort.

The physical process is about 10 minutes for the minimal kernel I created. Of course, working out exactly which options to select/deselect can take longer. A lot longer. But overall, I am deeply impressed with the effort that has been put into making compiling the kernel both a simple, and more importantly, safe job, with little to no risk of damaging the existing installation.

Oh yes, and now my TV tuner works just fine under Kaffeine

Smaller is better?

Work provide me with a laptop for business, changing them every three years or so. I’ve had a succession of IBM Thinkpads, which I have to say are truly the Rolls-Royce of laptops. Relatively compact and portable, stuffed with features and the kind of build quality that allows them to survive years of abuse (mine have travelled all over the world with me). My experiences with them have been exemplary … until the last few years, when IBM sold the Thinkpad business off to Lenovo. The last couple of machines have not been so great. My top of the range T40p needed 3 motherboards in as many years. My current T60p has already had 2 motherboards, a fan and a heatsink assembly in two years. And now the screen (a rather nice 15″ 1600×1200 job) is showing lines of dead pixels that probably indicate either the LVDS connector is failing, or I need yet another new motherboard. In addition Thinkpads seem to have been steadily growing. My last two machines have not fitted into my “old faithful” laptop bag, requiring a bigger heavier bag to carry them in.

So recently I found myself wondering why I needed a machine with 2 x 2.16GHz processors, 2GB of ram, and a 1600×1200 pixel 15″ screen. Fundamentally I do the same stuff on this machine that I did on my old Thinkpad 600 some 10 years ago. I do email. I browse the web. I use IM. I follow some (real NNTP!) newsgroups. I create documents and I read documents. I do presentations. I occasionally print stuff. So why do I now need the above monster specification, when I used to get by on a single 300Mhz processor, with 384MB of ram, and a 1024×768 13″ screen? And the more I thought about it, the less I could understand it. All I’ve done is adopt a much bigger and heavier machine, that runs a lot hotter, to do the same stuff. The bigger machine isn’t actually allowing me to do anything faster … I still think at the same speed. In fact, in one area my performance has actually decreased – I actually walk a lot slower with it because it weighs so much.

So at Christmas I decided to have my own personal “grand challenge” – to see how small a machine I could get away with to do my job. So I went out and bought my first ever laptop – an Acer Aspire One “netbook”. I got the one with 512MB of ram, and a 120GB hard drive. Cost me 200GBP here in the UK (I know, they’re a LOT cheaper in the USA – we get screwed on computer stuff here).

A day playing with the preloaded Linpus Lite linux (aka hacked Fedora 6) convinced me that while it was set up to be a foolproof computing appliance for the computer-illiterate, I needed something fuller-functioned. So after a little bit of fiddling with gparted, I now have the Aspire set up to triple boot any of (a) the original Linpus Lite, (b) Windows XP SP3 and (c) Ubuntu 8.10. Ubuntu will be my OS of choice, with Windows there purely for the odd work application that I have to use that isn’t supported under Linux. I kept Linpus purely to compare my performance tuning of Ubuntu to the stock preload.

To Ubuntu I added Thunderbird, Lotus Notes and Sametime (our corporate-standard groupware) and some VPN software. I then added OpenProj, Freemind and Dia, and upgraded the OpenOffice install to v3.0. And to my astonishment, in the last month I’ve needed nothing else.

There have been some teething problems though.

The first problem I experienced was that on Linux, Lotus Notes is implemented as an eclipse plugin, so unfortunately by the time it’s loaded there’s very little left of the 512MB of ram. I gave up after a couple of days, and sprang for another 1GB of ram, at another 13 GBP. Installing it required the complete disassembly of the Aspire … but in actual fact only took about 20 minutes with a set of jewellers screwdrivers. This maxes-out the memory on the Aspire One (at 1.5GB), but I now seem to have memory to burn, even when I have all my applications open at once.

The second problem is that as a long-term thinkpad user, I’m used to having a trackpoint (the little red “nipple” embedded in the keyboard) for moving the cursor. Compared to the control that that provides, a trackpad is a hopelessly painful experience. Fortunately I already had a Logitech bluetooth mouse (a V470), but unfortunately the Aspire One doesn’t come with internal bluetooth. So I had to add an external USB bluetooth adaptor. I imported a USB Bluetooth 2.0 EDR micro-stick from Hong Kong for 6 USD which resolved that problem, and is actually small enough to be left permanently attached, though longer term I’ll probably solder the internals of one to the motherboard somewhere.

Other than that, it’s proving a remarkably solid little performer. The keyboard is a little cramped, but just about big enough to almost-touch-type on. The screen is painfully small, but with multiple workspaces and compiz-fusion doing its thing, you can work around that. And I have external monitors at home and at the office, so it’s only really a problem while travelling anyway. If I have a lot of stuff going on then some applications can be a little sluggish, but even then it’s not problematic … just noticeable.

Boot times for standard Ubuntu were very slow compared to the preloaded Linpus lite, but a custom kernel tailored to the Aspire One has got that down to about 25 seconds, which is faster than standard Ubuntu boots on my T60p. Still not as fast as Linpus though – which is up in about 15 seconds. Battery life on the standard 3 cell battery is only about 2:20; a high capacity 6 cell 3rd party battery should take that to about 7 hours but for my working patterns that’s unnecessary, and would start to add to the weight and size again.

Which leads us to the question of do I declare the challenge beaten or not? I’d justified buying the Aspire One on the basis that if I couldn’t live with it, I’d simply give it to one of my daughters, as I figured it would be fine as a first machine for school-work etc. Well, at the moment I’m not prepared to give it up. It does everything that I need – just, and no more. In fact, these Aspire Ones would be great for my daughters, and the only thing holding me back from going out and buying them a pair is the faint hope that we may see some netbooks built around NVidias ION platform this year.

Replacing the Intel 945GSE chipset (which seems to be based on 3-4 year old designs) with the NVidia GeForce9400 chipset ought to result in a stupendous graphical performance hike for no increase in power consumption. The only question is, will anyone do it? The problem is that it makes low-end netbooks compete with mid-range laptops (problematic for the manufacturers), and Intel is sure to price the Atom/945 combination in such a way that its almost as cheap to take the pair as an Atom on its own – which would stuff any chance that Nvidia have. Will be interesting to see how it works out.