That means I’ll get to have the summer off work again. Which would be grand, where it not for the fact that I’m very unlikely to be in a condition to enjoy it. So, at a guess, most of July in hospital, then recovering at home through August/September and hopefully returning to work sometime in October. Thank heavens for a very understanding employer.

I must admit that I’m dreading the whole thing, not helped by the fact that having done this once before, I now know exactly what I’m going to have to go through. Still, who said life would be easy, eh?

But having been using it for some 18 years now, I’m fairly comfortable with it. The only issue I have is that as I’ve been using it for so long, my mail archives contain a huge amount of useful information from old projects that I’ve worked on. I also have other information related to those projects stored elsewhere on my laptop harddrive, and pulling all that information together and searching it coherently isn’t a trivial problem. However, in recent years desktop search engines have begun to provide a really nice solution to this.

The problem here is that Lotus Notes is based on a series of binary databases which form the backbone of its ability to efficiently replicate documents between clients and servers. Desktop search engines generally don’t understand those databases, and hence do not work with Lotus Notes. So searching my laptop becomes a somewhat tedious process, involving the Lotus Notes client search feature, and manually correlating with a desktop search engine of some type. It works, but it’s just not as good as it could be.

What I really want, what I really really want (as the Spice Girls would sing) is a desktop search engine that can understand and integrate my Lotus Notes content. And that’s what this post is all about.

Since I run Linux I have a choice of open source desktop search engines such as Tracker or Beagle (now deceased). But my current preference is for Recoll, which I find to be very usable. And then, last year, I discovered that a colleague had written and published a filter, to enable Recoll to index documents inside Lotus Notes databases. So I had to get it working on my system!

Unfortunately, it turned out that my early attempts to get the filter working on my Ubuntu (now Mint) system completely failed. He was a RedHat user, and there are quite a lot of packaging differences between a Debianesque Lotus Notes install, and a RedHat one, especially inside IBM where we use our own internal versions of Java too. So the rest of this post is essentially a decription of how I hacked his elegant code to pieces to make it work on my system. It’s particularly relevant to members of the IBM community who use the IBM “OCDC” extensions to Linux as their production workstation. I’m going to structure it into a description of how Recoll and the Notes filter work, then a description of how I chose to implement the indexing (to minimise wasteful re-indexing), and hence what files go where, and some links to allow people to download the various files that I know to work on my system.

At a very simplistic level, Recoll works by scanning your computer filesystem, and for each file it encounters, it works out what it is (plain text, HTML, Microsoft Word, etc) and then either indexes it (if it’s a format that it natively understands) using the Xapian framework, or passing it to a helper application or filter which returns a version of the file in a format that Recoll does understand, and so can index. In the case of container formats like zip files, Recoll extracts all the contents, and processes each of those extracted files in turn. This means Recoll can process documents to an arbitrary level of “nesting”, comfortably indexing a Word file inside a zip file inside a RAR archive for example. Once all your files are indexed, you can search the index with arbitrary queries. If you get any hits, Recoll will help to invoke an appropriate application to allow you to view the original file. The helper applications are already existing external applications like unRTF or PDFtotext that carry out conversions from formats that Recoll will commonly encounter, while filters are Python applications that enable Recoll to cope with specialist formats, such as Lotus Notes databases.

So, the way the Lotus Notes filter works, is that:

1. Recoll encounters a Lotus Notes database, something.nsf
2. To work out what to do with it, Recoll looks up the file type in its mimemap configuration file, and determines what “mimetype” to associate with that file
3. It then looks up what action to take for that mimetype in the mimeconf configuration file, which tells it to invoke the rcllnotes filter
4. It then invokes rcllnotes, passing it the URI to something.nsf
5. rcllnotes then extracts all the documents (and their attachments) from the Notes database, passing them back to Recoll for indexing
6. It does this by invoking a Java application, rcllnotes.jar, that must be run under the same JVM as Lotus Notes
7. This Java application uses Lotus Notes’ Java APIs to access each document in the database in turn
8. These are then either flattened into HTML output (using an XLST stylesheet) which Recoll can consume directly, or in the case of attachments, output as a document needing further processing; Recoll can tell which is which from the mimetype of the output. Included in the flattened HTML are a couple of metadata tags, one marking this HTML document as descended from a Lotus Notes database, and the other containing the complete Lotus Notes URI for the original document. This latter information can be used by the Lotus Notes client to directly access the document – which is crucial later in the search process
9. Recoll then indexes the documents it receives, saving enough information to allow Recoll to use rcllnotes again to retrieve just the relevant document from within the Notes database.
10. So, when a search results in a Notes document, Recoll can use the saved information (the URI of the database and the Notes UNID of the document?) and the rcllnotes filter to obtain either the flattened HTML version of the document, or a copy of an attachment. Recoll then uses the documents mimetype to determine how to display it. In the case of an attachment, Recoll simply opens it with the appropriate application. In the case of the HTML, Recoll combines the expected “text/html” with the information in the metadata tag that describes this HTML as being derived from a Lotus Notes document. This produces a mimetype of “text/html|notesdoc”, which it then looks up in the mimeview configuration file, which causes it to use the rclOpenNotesClient script. That reads the Notes URI from the other HTML metadata field in the flattened HTML file, and then invokes the Lotus Notes client with it, causing the actual document of interest to be opened in Lotus Notes.

One of the problems with using Recoll with Lotus Notes databases is that it’s not possible to index just the few changed documents in a Notes database; you have to reindex an entire database worth of documents. Unfortunately there are usually a lot of documents in a Notes database, and the process of indexing a single document actually seems relatively slow, so it’s important to minimise how often you need to reindex a Notes database.

To achieve this, I make use of a feature of Recoll where it is possible to search multiple indexes in parallel. This allows me to partition my system into different types of data, creating separate indexes for each, but then searching against them all. To help with this, I made the decision to index only Notes databases associated with my email (either my current email database, or it’s archives) and a well-known (to me) subset of my filesystem data. Since my email archives are partitioned into separate databases, each holding about two years of content, I can easily partition the data I need to index into three categories: static Lotus Notes databases that never change (the old archives), dynamic Lotus Notes databases that change more frequently (my email database and its current archive), and other selected filesystem data.

I then create three separate indexes, one for each of those categories:

1. The static Notes databases amount to about 5.5GB and takes about 2.5 hours 8GB and takes a little under 4 hours to index on my X201 laptop; however, since this is truely static, I only need to index it once.
2. The dynamic Notes databases amount to about 4GB and take about 2 hours 1.5GB and take about 40 minutes to index; I reindex this once a week. This is a bigger job than it should be because I’ve been remiss and need to carve a big chunk of my current archive off into another “old” static one.
3. Finally, the filesystem data runs to about another 20GB or so, and I expect this to change most frequently, but be the least expensive to reindex. Consequently I use “real time indexing” on this index; that means the whole 20GB is indexed once, and then inotify is used to determine whenever a file has been changed and trigger a reindex of just that file, immediately. That process runs in the background and is generally unnoticable.

So, how to duplicate this setup on your system?

First you will need to install Recoll. Use sudo apt-get install recoll to achieve that. Then you need to add the Lotus Notes filter to Recoll. Normally you’d download the filter from here, and follow the instructions in the README. However, as I noted at the beginning, it won’t work “out the box” under IBM’s OCDC environment. So instead, you can download the version that I have modified.

Unpack that into a temporary directory. Now copy the files in RecollNotes/Filter (rcllnotes, rcllnotes.jar and rclOpenNotesClient) to the Recoll filter directory (normally /usr/share/recoll/filters), and ensure that they are executable (sudo chmod +x rcllnotes etc). You should also copy a Lotus Notes graphic into the Recoll images directory where it can be used in the search results; sudo cp /opt/ibm/lotus/notes/notes_48.png /usr/share/recoll/images/lotus-notes.png.

Now copy the main configuration file for the Notes filter to your home directory. It’s called RecollNotes/Configurations/.rcllnotes and once you have copied it to your home directory, you need to edit it, and add your Lotus Notes password in the appropriate line. Note that this is by default a “hidden” file, so won’t show up in Nautilus or normal “ls” commands. Use “ls -a” if necessary!

Next you need to set up and configure the three actual indexes. The installation of Recoll should have created a ~/.recoll/ configuration directory. Now create two more, such as ~/.recoll-static/ and ~/.recoll-dynamic/. Appropriately copy the configuration files from the subfolders of RecollNotes/Configurations/, into your three Recoll configuration folders. Now edit the recoll.conf files in ~/.recoll-static/ and ~/.recoll-dynamic/, updating the names of the Notes Databases that you wish to index. Now manually index these Notes databases by running the commands recollindex -c ~/.recoll-static -z and recollindex -c ~/.recoll-dynamic -z.

At this point it should be possible to start recoll against either of those indexes (recoll -c ~/.recoll-static for example) and run searches within databases in that index. I leave it as an exercise for the interested reader to work out how to automate the reindexing with CRON jobs.

Next we wish to set up the indexing for the ~/.recoll/ configuration. This is the filesystem data that will run with a real-time indexer. So start by opening up the Recoll GUI. You will be asked if you want to start indexing immediately. I suggest that you select real-time indexing at startup, and let it start the indexing. Then immediately STOP the indexing process from the File menu. Now copy the file RecollNotes/recoll_index_on_ac to your personal scripts directory (mine is ~/.scripts), ensure it is executable, and then edit the file ~/.config/autostart/recollindex.desktop, changing the line that says Exec=recollindex -w 60 -m to Exec=~/.scripts/recoll_index_on_ac (or as appropriate). This script will in future be started instead of the normal indexer, and will ensure that indexing only runs when your laptop is on AC power, hopefully increasing your battery life. You can now start it manually with the command nohup ~/.scripts/recoll_index_on_ac &, but in future it will be started automatically whenever you login.

While your filesystem index is building, you can configure Recoll to use all three indexes at once. Start the Recoll GUI, and navigate to Preferences -> External Index dialog. Select “Add Index”, and navigate into the ~/.recoll-static/ and ~/.recoll-dynamic/ directories, selecting the xapiandb directory in each. Make sure each is selected. Searches done from the GUI will now use the default index (from the filesystem data) and the additional indexes from the two Lotus Notes configurations.

There is one final configuration worth carrying out, and that is to customise the presentation of the search results. If you look in the file in RecollNotes/reoll-result-list-customisation you will find some instructions to make the search results easier to read and use. Feel free to adopt them or not, as you wish.

Update: To answer the first question (by text message no less!), my indexes use up about 2.5GB of space, so no, it’s not insignificant, but I figure disk really is cheap these days.

Update: Corrected command to copy Notes icon to Recoll image directory to match configuration files, and a couple of the pathnames where I had introduced some typos.

Update: Added the main .rcllnotes configuration file to my archive of files, and updated the installation instructions to discuss installing it, and the need to add your Lotus Notes password to the file.

In a lot of ways the ileostomy looks a lot more attractive at this point in time; it’s a known quantity (since I already have one) so I know I can cope with it and continue to work. However, if I’m brutally honest, I’m just hanging on in there, rather than actually enjoying life. It’s too limiting, and has too many awkward side-effects to be easy to live with, especially long term.

The colostomy on the other hand, is an unknown quantity, and as with all these things the medicial community have an infuriating tendency to give you a very rose-tinted vision of what life could be like with one. In fact, depending on how well the colostomy turns out I could end up with anything from a much improved quality of life, to (whisper it!) a complete nightmare. This seems to be largely determined by how much colon the patient has left, so the fact that my cancer was so low in my colon helps here; my surgeon feels that I’m likely to have a “good” result from a colostomy. But of course, he cannot guarantee it.

But perhaps the deciding factor for me was based on a comment from one of the nurses I was chatting with, who brought up the issue of my life much further into the future – something I’d not really considered up to that point. Ultimately a patient with a colostomy that can no longer actively manage it themselves can still live (largely) independantly in their own home with a small amount of nursing daycare. A similar patient with an ileostomy would have to be moved into fulltime care in some form of institution. I’m certainly not planning to end up unable to manage, but I’m now much more aware of the uncertainty that the future can hold, so I’d rather hope for the best, and plan for the worst.

So on balance I’m going to take the colostomy. And (as the same nurse said) if the absolute worst were to happen then I could always convert a colostomy into an ileostomy, albeit with another huge operation. But choosing the ileostomy now would leave me with no other options at all.

And I’m convinced that having a Plan B is always a good idea.

On the surface, this looked like a really good deal for almost everyone – O2 and BE Unlimited had been steadily losing customers for a long time, because Telefonica wasn’t making the investments to allow them to keep up with technologies such as FTTC. Sky was already offering some of those technologies (particularly FTTC) and has a history of investing to create premium solutions for their customers. And Sky would gain the additional customers to jump them past Virgin, moving from the third largest ISP in the UK to the second largest, just behind BT.

However, there was a fly in the ointment; BE Unlimited in particular started life as the niche-market provider-of-choice for the demanding techie. As such they offered some less common features – static IP addresses (either individually or in blocks), full Reverse DNS resolution, Annex M ADSL2+ with user-configurable noise margin, multiple bonded connections … the types of features that are usually reserved for very expensive business class connections. And Sky, as a mass-market ISP, did not (and to my knowledge still do not) offer any of those kinds of features.

What the announcement did was galvanise many of the BE Unlimited customers to start looking for alternatives, in case the features that they depended on would go away when their service was transferred to Sky. In my case my requirements were fairly simple; at least 1 static IP address with configurable Reverse DNS, no port-blocking, no usage limits, and the ability to get at least 12Mb down and 2Mb up. And ideally I didn’t want to pay any more than I already was.

Frankly, I was expecting it to be a real challenge, but to my surprise the second ISP I called was able to meet all my needs. Admittedly the nice lady in the call-centre had never heard of Reverse DNS before, but she took down all my requirements, and put me on hold while she talked to their technical support team. Several long minutes later, and we were in business; their offer was 60Mbps down, 20Mbps up on an unlimited FTTC connection, with one static IP address with reverse-DNS, and no ports blocked. Even better, it was £2 a month less than I currently paid, provided I also transferred my phone service to them.

And so I became one of what appears to be the many who have left BE Unlimited rather than be moved to Sky. Which is rather sad really – they were an exceptional ISP who got bought by a large organisation (Telefonica) and then starved of the cash they needed to continue to grow. The end result is that they won’t exist soon.

The one thing that I have “lost” in this process of changing ISP’s is the “anonymous caller rejection” feature on my phone line. BE provided that for £2 a month, whereas Plusnet wanted £4. Which is a lot for a feature that in practice, I found rather limited anyway.

Which brings me to the point of this post. Rather than paying for the Anonymous Call Reject, I’m going to use a modem to monitor the callerid of inbound phone calls, and then decide on a call by call basis what to do with the call. I can either let the call continue (and ring the phones) or I can pick-up & immediately hang-up the line (using the modem) before the phones even ring. That should enable me to cut off all the callers with witheld numbers, just as the £4 a month ISP option would. However, I can also reject calls from “unavailable” numbers, which are usually cold callers using VoIP systems. And I can also easily implement a system that will reject all calls outside of certain hours (say 8am to 6pm), unless the caller is on a whitelist of known contacts. Which is a lot better than the ISP anonymous caller rejection can achieve.

To do this I’m using an old USR 5668B serial-attached modem that correctly understands the UK’s completely proprietary callerid system. However, this modem is no longer available, and getting a clear answer on what modems can actually cope with UK callerid is, er, troublesome. However, I have discovered reports that the currently available USR 5637 USB fax modem works well with UK callerid. See here for ordering information (no affiliation).

To access the modem I’m using a simple self-written script, written in Perl, using an external library called Device::Modem, which gives me a nice high level abstracted interface to the modem, making the programming much easier. My code is based on some example code provided with Device::Modem for reading the callerid, but in my case, is now logging the inbound calls into a file so I can better investigage who to put on a future whitelist.

Update: code attached for interest.

#!/usr/bin/perl
#
# Capture Callerid information for incoming calls, using the modem attached to /dev/ttyS0
#
#
use Device::Modem;

# Assuming we dont want the file open all the time, so open, append and close the log
open (FH, ">>/home/richard/callerid/phone.log") || die "calleridmonitor.pl: Couldn't open file: $!\n";
print FH "$call_date $call_time $call_number\n";
close FH ;
}
# Repeat forever
}

Without going into all the gory details, the good news is that the tests show that I am still (so far as we can tell) free of cancer. In March it will have been 4 years since my operation. One more year to go, and I will be discharged from most treatment. News that’s definitely worthy of celebration!

On the other hand, the bad news is that all the scans show the same thing. The healing of the abscess (more correctly, sinus) in my pelvis has slowed to a halt. Fundamentally the problem is that the radiotherapy that I elected to take back in 2009 to improve my chances of getting to where I am now, in 2013, has diminished my capacity for healing to the point where the abscess and the join in my colon is never going to mend.

Unfortunately this leaves me in a situation that cannot continue; my kidney function is being impaired by the inflammation in my pelvis, and that will continue to get worse unless we intervene more dramatically.

So I’m out of options. The surgeons need to “take down” the original repair in my bowel, and replace it with something that works so I can get on with as normal a life as possible. My options are either a pemanent colostomy or a permanent ileostomy. We discussed (briefly) attempting to remake the join even lower in my pelvis, but my surgeon feels that with the amount of surgery I’ve already had, even if the operation “worked”, it would be unlikely that my bowel would function adequately anyway. There would be a fairly high chance I would end up incontinent – so that’s easily ruled out.

So for now, I’m coming to terms with the results. Over the next few weeks I need to make the decision over the colostomy or ileostomy, and then decide on when to actually schedule the operation. The surgeon tells me that the operation is larger than the first operation I had, and that I should expect 2-3 weeks in hospital, and 3 months off work recovering. Which is all rather frustrating, if not actually somewhat depressing.

So, still winning the war, but I’ve just lost a major battle. Darn it.

At the current time there are very few places offering this treatment. There are about 10 in the USA, and a few in Europe. There are none in the UK – the NHS sends what few patients can justify this treatment abroad to Switzerland and the USA. That’s at least partly because the systems are very expensive to install, and then can’t be heavily used (which would justify the high build cost) because each case requires a long and complex planning phase to ensure that the beam of protons only goes exactly where it should.

What our researchers are doing is automating a lot of the difficult manual work involved in planning the treatments, reducing the planning time for each treatment from several days to a few minutes. Ultimately this will probably result in a (very small) clinical improvement in the effectiveness of the treatment for each patient, but by making it possible to use the machine much more effectively it will enable their wider adoption, helping to bring this new radiotherapy treatment to a much larger number of patients.

There was a positive output from my various tests; namely that there are some signs of improvement in the output from the Gastrografin test. That’s not to say that the abscess has healed – nothing so dramatic – but that there are signs of improvement. Which means we keep on fighting.

On the other hand, the results from the MAG3 Renogram were far less positive. The function of my left kidney is continuing to decline. The split of filtration between my kidneys is now about 70/30, rather than the normal 50/50. So my left kidney is down to 3/7th’s function (about 43%). This is still more than enough that were I to lose my right kidney in some catastrophic accident, I could live on the left without dialysis, so it’s important to try to preserve that remaining function.

And so we’ve spent the last couple of weeks planning what to do about the kidney problem, culminating in my coming into hospital today to have the urologists give me a serious going over. Under a general anaesthetic, thank goodness. The high level plan was to do a more thorough endoscopic investigation, including some work with contrast agents and x-rays to see if the problem was a stricture in the ureter, or constriction of the ureter due to external pressure. My urologist also decided to draft in some assistance from a urologist who specialises in endoscopic treatments.

And so today I checked back into hospital.

After a morning of pre-op tests (including a game of “chase the vein” to draw some bloods) and catching up with all the people who I’ve got to know so well here, I got taken down to theatre first thing this afternoon. Only to discover that despite my surgical team booking the necessary radiological kit well in advance, someone had pinched it for a different operation, much to the frustration and embarrassment of my team. It did make me smile to see that even in a hospital the same human games go on.

Once we’d rounded up all the necessary kit, we did the usual dance with Fentanol and Propafol, and I blinked and woke up in recovery again. I love that trick!

And the conclusion is that the narrowing of the ureter is confined to the lowest point of the ureter, nearest to the bladder (and my abscess) and is caused by external pressure. Apparently there was considerable discussion about the merits of trying to effect a cure by going ahead and dilating the ureter with a balloon catheter, but the final view was that the potential advantages were significantly outweighed by the risks of introducing infection into my urinary system. So for now the urologists have chosen to restent that ureter again which ought to preserve the function I have while we continue to deal with the abscess. Rather sensibly they want to ensure that the stent really is maintaining my kidney function adequately, so they’ll schedule another MAG3 renogram for me in a few weeks time.

Clearly the hope here is that once the abscess is dealt with, the pressure on the ureter will go away, and (ideally) no further intervention will be necessary. And if intervention is required then at least there will be no infection floating around to add any complications.

And as before, the ultimate plan is that if we can’t resolve the abscess then they’ll just sort out everything at once in one massive operation to end all operations. Fun.

I’m now waiting for all those diagnostic tests to be interpreted, and feedback to go back to my surgeon, who will coordinate that and pass it all back to me along with whatever recommendations he thinks most appropriate.

The radiographers running the MAG3 renogram were first up. And very close-mouthed too. They wouldn’t tell me anything, or show me any of the pictures they were taking, which was a bit frustrating. Especially as the whole process took 4 hours. Mind you, a good proportion of that was them trying (and largely failing) to cannulate me, which was quite amusing when viewed in retrospect. Not quite so much fun at the time when I’ve got two of them working on both my arms at once, vying to see who can find a usable vein first. Now I know what a pin-cushion feels like!

Obviously, I know that my left kidney will be reduced function compared to my right. It won’t have improved since the last test. But the key question is, is it now stable, even though they have removed the stent? If it is, then all is good. If not then the urologist is going to want to take some actions fairly quickly to try to rectify the situation. And if he can’t solve it, my surgeon will want to start thinking much more seriously about moving me along to the colostomy. So I’m really just hoping for signs of stability.

The gastrografin test was as unpleasant as usual; having said that, it is somewhat mitigated by the delightful staff who administer it. The radiographer and consultant radiologist are lovely, and genuinely make the process a lot easier to bear. The consultant radiologist tells me that the abscess looks very well defined, but couldn’t say (immediately) whether the cavity is healing up or not. He’ll need to compare the scans with the last set to be able to give me that information. I expect that will come via my surgeon in a couple of weeks time. One thing that dealing with the medical profession teaches you; patience.

The MRI scan was much longer than normal. This was because they wanted a much more detailed image of the ureters and kidneys than previously, as well as the whole of my pelvis as before. In the end I was in the machine for about 45 minutes. I always used to think of myself as being quite claustrophobic, but actually once in the machine I found it quite relaxing. It’s a chance to turn off the outside world and actually think about some of the problems that I’m working on at the moment. I’m not sure I came to any successful conclusions, but at least it distracted me from the confined space I found myself in!

One thing of note, in addition to all the standard programmes that I’ve been through before, they also ran a special programme on the machine where it monitors your breathing, and then automatically and repeatedly scans you at the same point in your breathing cycle, eliminating the movement caused by your breathing. Very impressive stuff.

So now I just have to wait to see what the tests have shown, and what the surgeon and his team of experts think is the right next step.

Having been through this once before, I knew what I was in for; it’s not fun, as it’s carried out with only a light local anaesthetic, but it is straightforward, so I was back on my feet and home again by late lunchtime. I’ll feel a bit rough for the next couple of days, but should be back to normal by Monday or Tuesday.

Once my urological system settles down (2-3 weeks) we’ll run a series of non-invasive radiological tests on my kidneys that will reveal any signs of blockage or restriction in the ureter. This tends to show as enlargement of the associated kidney, but I expect they’ll also run some differential drainage tests too, to measure exactly how well its all working. More “gunk” to drink!

If all is OK then I will be able to put the whole stent thing behind me. If not then they’ll figure out what to do next – there are some fairly non-invasive options available that may help alleviate the problem.

But if there are problems it’s also likely to precipitate a conversation with my surgeon about the advantages of pro-actively undertaking the colostomy operation.

Speaking of whom, I have the first of my monthly consultations with my surgeon on Monday morning, where he will check if our plan to “just give me some time” for my abscess to heal is bearing any fruit. More news on that in due course I expect, though I’m not convinced it will be terribly positive.

It’s fairly clear that I’m entering the end-game now.

More importantly it means that just doing the normal day to day family things, like chatting to your kids about their day in school can rapidly become an extortionately expensive luxury that you feel ought to be kept to the brief minimum. Especially when the company that you’re travelling for thinks that it shouldn’t pay for those phone calls – which particularly irks me.

And that got me thinking – I actually have all the facilities I need to enable me to call home to my family for nothing. My company expects me to need an Internet connection in whatever hotel I stay in, and fully funds it. I carry a laptop and an Android Smartphone. In combination with the rather sophisticated phone that I have at home, I can talk to my family for as long as I want for no additional costs, using a technology called VoIP, based on an open standard called SIP.

My home phone is a Siemens Gigaset S685 IP DECT system, with 5 handsets. It’s what the marketing world likes to term “pro-sumer”, by which they really mean it’s got lots of features that most consumers will never use, but it’s not really got enough features for true commercial use. They also mean it’s a bit expensive for a consumer product.

But in this case, we’re talking about a DECT phone that connects to both my home phone line and my home broadband, and can have up to 7 logical phone lines connected to it – the physical “POTS” line, and 6 VoIP connections over the internet. The base unit can support separate conversations on up to 3 lines in parallel, with as many handsets connected to each of those lines as required. Each handset can act as a speaker-phone, or have either a headset or bluetooth earpiece attached to. It can even do call routing, where it looks at the number you dial, and decides which line to place your call on. In short, it’s absolutely packed with features.

The key to the free calls home is of course, the VoIP lines, because as of version 2.3, Android can also make VoIP calls. The trick is simply getting the two to talk to one another, while they are in different countries.

So first you need to find a SIP provider for your home phone and your Smartphone. The best way that I’ve found to do this is to set up a SIP-based virtual PBX. You create a free account with them (your PBX) and then add users to your PBX. Each user is given their own SIP credentials so they can logon, and you can associate an extension number with each of those users, allowing them to easily exchange calls – which is exactly what I need, as (unlike my Android smartphone) the handsets on my old Gigaset cannot call SIP URI’s directly.

The first provider I came across that allows this is OnSip, in the USA. My experience with them has been good, but there are many others out there too. Unfortunately it’s not perfect – for me, there are a couple of quirks with their service. Firstly, they don’t seem to expect anyone outside the USA to be using their service, so I cannot enter my address correctly. Which I suspect is technically a breach of their T&C’s. And secondly, it means that all the call tones you’ll hear when using this (ringing, engaged, unobtainable etc) will be American rather than British. I can live with that, but if you choose to go down this route too, DO NOT ENABLE THE E911 OPTION. You could theoretically call out US emergency services to whatever pseudo-fictitious address you have registered with them, which would not be good.

To make it work:

1. Register with your chosen free SIP PBX provider. I’ll assume OnSip, who will ask you to register a userid, then validate it by email, before letting you set up your first PBX.
2. 

   Registering for OnSip PBX

   When you come to define that PBX, you’ll see a screen similar to this one, asking for initial configuration information. You can call the domain name (which is essentially the Internet name of the PBX) anything you like. Incidentally, changing that country field seems to do nothing. It doesn’t even change the format of the information they collect for your address or (real) phone number.
3. 

   Creating a new user

   Having now got a SIP PBX available, we can add some users to it. Each user is roughly equivalent to a telephone on a normal PBX, but in this case the user accesses the PBX by way of some SIP credentials. The users can be called via those credentials, or (if the caller is another user of this PBX) the extension number that is associated with that user. This happens irrespective of the device that the user is using those credentials on, or its location.
4. 

   List of virtual PBX users

   After entering a user for the Gigaset (my house phone) and one for my smartphone, I have a PBX with two users associated with it. I’ve obscured some critical pieces of information to maintain some privacy, but fundamentally the OnSip system gives me a set of SIP credentials for each “user” of the system (bottom right on the image), and associates an extension with them too.
5. Next we need to get the Gigaset to register with the SIP PBX. To do this, logon to the Gigaset web interface, select the “Settings” tab, and then navigate to Telephony, Connections.
   

   
   
   Gigaset Settings

   Now define a new provider by clicking on one of the “Edit” buttons as seen in the first of these screenshots. This will bring up the basic settings page seen in the second screenshot. Into this screen enter the data found on the OnSip user configuration screen under the Phone Configuration heading. Copy the OnSip username field into the Gigaset username field, the OnSip Auth Username field into the Gigaset Authentication Username field, the OnSip SIP Password into the Gigaset Authentication Password field, and then click the “Show Advanced Settings” button, which will extend the form with the additional fields seen in the third screenshot. Now copy the OnSip Proxy/Domain field into the four Gigaset fields: Domain, Proxy Server Address, Registrar Server, and Outbound Proxy. When you save the settings the Gigaset will attempt to register with the OnSip PBX. Sometimes this takes a few seconds. You may need to refresh the browser to see the status change to Registered.
6. Now we need to make the Android Smartphone register to the OnSip PBX too. To do this, bring up the Android Settings by pressing “Menu” from the home screen, and tap “Settings”.
   

   
   
   Android SIP setup

   Navigate to the Call Settings, and tap it to reveal the screen in the first screenshot.
   Tap on “Use Internet Calling” and set it to “Ask for each call”. Then tap on Accounts to bring up the Internet Calling (SIP) Accounts screen where we can enter SIP accounts associated with the phone. See the second screenshot.
   Now add a new account for the OnSip PBX by tapping “New Account”; this will bring up a screen like the one shown in the third screenshot, into which you need to enter your credentials.
   The content of each of the fields (from the OnSip phone configuration information) should be obvious by now. When you save the account you will want to check that the registration information is correct. The easiest way to do this is to enable the “Receive incoming calls” option (back on the first screenshot again), which will force Android to register all it’s accounts with their associated providers. If you get an error then either the provider is down (possible) or the settings are incorrect (more likely). Leaving this option enabled forces Android to keep those connections active, which runs all the radios, and eats the battery, but allows incoming VoIP calls to your Smartphone (say from the Gigaset). In my experience it’s too power-hungry to use in this mode, other than very sparingly. Fortunately you can make outgoing calls from the Smartphone without enabling it.
7. 

   Android Internet Calling enabled Contact

   Finally you need to define a contact in the Smartphone contacts to use with Internet Calling. As all my contacts are normally managed by Lotus Traveler for me, which has no concept of Internet Calling, I defined a new local-only contact that is not synched with any of my Accounts (ie, Google or Lotus Traveler) and used that. Enter the name of the contact as normal, then scroll all the way to the bottom of the contact, where you will find a “More” section. Expand that, and continue to scroll to the bottom, where you will find a field for “Internet call”; into that simply enter either the OnSip SIP URI of your Gigaset, or it’s OnSip extension number.

Note that this really only works when connected to a reasonably good quality WiFi network. Poor quality networks seem to give quite variable results. Sometimes they still work, other times one end or the other may experience audio problems and/or dropped calls. It seems to work just fine through both the IBM and my home firewalls, even at the same time. I’ve not checked the actual audio codecs in use, but sound quality is subjectively better than a normal cellular call. Neither Android or the Gigaset seem to do silemce suppression (ie injection of white-noise when no-one is speaking) so the pauses in a conversation are totally silent, which can be slightly disconcerting.

Normally if you want to dial the Smartphone from the Gigaset you would need to indicate to the phone that it should send the call over the appropriate VoIP provider. This quickly becomes a pain, but it’s easy to set up a simple dial plan on the Gigaset so calls that start “700″ (remember I made my extensions be 7000 and 7001?) go out over the OnSip VoIP connection automatically, which makes the solution really easy to use (ie family-friendly) from the Gigaset.

Finally, there is a really interesting technology called iNum available. Sadly it’s not (as far as I can tell) implemented by any of the major telecoms ISPs in the UK, that when combined with SRV records in special DNS records, would allow some really cool automatic rerouting of calls over different networks to different endpoints, depending on context. In theory the network could understand whether I was at home or out with my mobile, and route the call appropriately. It could also do smart things like look at the inbound number and the time, and route business calls to voicemail out of office hours, but still let personal calls through as normal.

If only it were implemented.