Ronald Duncan's Blog

Our Software Industry needs to step up to the crease and tell the world that we can write code

ronaldduncan — Wed, 30 Nov 2011 23:10:12 +0000

8 years ago I had a very embarrassing meeting. I laughed at a major global company that had invested over $ 600 million in a shared system. It took 5 years for the companies representative to talk to me again.

I did not actually laugh initially. I did something much worse.

They explained what they had done, and I kept trying to understand how they could have taken so long and spent so much money, and they explained again and again, as I looked at them in complete disbelief.

I think they may have initially thought that I was impressed or astounded. They certainly thought that their system was amazing, a first of its kind, the best of the best etc. etc., and they had been jumping up and down shouting from the roof tops about its brilliance.

Eventually, I said “is that it, is that really all it does, its trivial” I was shocked. It was about 3 months effort for a competent developer on a decent platform. We had done something much much better 5 years before, and regarded it as nothing to boast about.

My shock showed. They were affronted and angry. I laughed from embarrassment, and was promptly shown the door.

It has taken a very long time to rebuild the relationship.

So it would be very nice, if other people could point out that the UK software industry does create great things, and that when someone from another culture says theirs is “Amazing” and we say that we have created something that is barely adequate. Our barely adequate may only be 100 or 1,000x better than their Amazing world beater.

Caching is king or how to make a fast web app

ronaldduncan — Wed, 30 Nov 2011 22:35:30 +0000

Your developers have created a brilliant new application, and you are new starting to get some traffic, but it is collapsing under the load, and you are having to throw hardware at the problem. A solution is at hand in the form of caching.

Varnish is the first thing to give your app some polish, it is relatively slow as a reverse proxy compared to lighttp or nginx, but a massive improvement over dotNet, Java or php. However these all use naïve caching where content is cached for a time out period and then checked again after the time out, so the load will soon over take you and you will need to move to the next level.

Intelligent caching has a number of levels so lets move to the top end the @UK Platform Generator Cache. The key differences are as follows:-

Compressed content
No time to live, but updating as content changes
Html Block level compression and de-duplication

The result is that the @UK Platform Generator Cache can handle millions of pages per gig of memory and eliminates the issue of having to balance between out of date content and system load since it never needs to go back to the database/middle ware to check if the cache is stale, because the cache is kept up to date in real time.

The result is that @UK can provide customers with free infrastructure since even high volume customers do not produce a measurable load on the system. A typical customer using Php, java or dotNet for their web layer will see both web and database servers collapse before the @UK system starts to register a measurable load.

This means that @UK offers a migration service where it will provide an expanded infrastructure to the customer and some tuning to their application as part of an upgrade onto the @UK Platform, so that they can continue their expansion and then take the brakes off once they are on Platform Generator.

Understanding the cloud

ronaldduncan — Wed, 30 Nov 2011 22:09:02 +0000

First it has been around a lot longer than you think and in a lot of different formats. The modern cloud came about from the requirements of online companies to solve the problems of providing 100% uptime and security on cheap hardware. Mainframes solved these problems a long time ago, but were not cheap.

The modern cloud was independently created by a number of ecommerce and software as a service companies whose revenues and reputations depended on providing 100% Security, and Availability to their customers. In the case of the ecommerce companies both @UK and Amazon derived 100% of their income from online sales, and had carried out pioneering work into low cost clustering, which is part of the basis for the modern cloud technology.

Amazon had virtually unlimited cash for their operations and consistently lost significant sums of money as they built the business. This meant that they threw hardware at the problem and needed tools and systems to manage a vast infrastructure.

@UK was created in 1999 by a leading software house @Software PLC, and missed out on the dot com boom for funding. Thus, @UK developed its cloud between 1999 and 2006 with minimal financial resources. This changed with the @UK IPO in 2006, however the philosophy of high quality software as the solution rather than hardware remained.

Cloud Application Stacks

The result is that @UK has developed a cloud application stack that is at least 100x more efficient than well know web software stacks such as

Linux Apache PHP and MySQL
Oracle Java and
Microsoft IIS, DotNet and SQL server

For most real world applications the level of change is not measurable since they collapse under load before a measurable change in load is recorded for the @UK system.

Clustered Storage and Virtualisation

The other parts of the modern cloud are clustered storage, and virtualisation. The first large scale deployments of clustered storage were by Google who created their own file system to cope with their requirement for large amounts of inexpensive storage. Cern (the creators of the web) also contributed to this effort with low cost hardware designs for their storage requirements.

Virtualisation has been around since the mainframe, however the x86 chip is difficult to virtualise and there are a limited number of virtualisation solutions for the older chips and they were often very inefficient. The new VT enabled chips mean that commodity x86 hardware now provides reasonably efficient virtual hosts.

It is the combination of low cost virtual hosts, clustered storage and the experience of managing clustered application stacks that created the modern cloud. However the problems that they were trying to solve of providing secure available applications have been around for a much longer time.

Cloud Delivers Security and Availability

This is why for most major cloud players the current concerns about cloud security and availability are things that they normally started to address over 10 years ago and solved a long long time ago.

The Government Needs to Answer back

ronaldduncan — Mon, 28 Nov 2011 21:50:47 +0000

One of our more noisy US colleagues recently stated that the UK Government needs to do more in the cloud. Given our experience the UK Government is doing lots in the cloud, they are just not doing it with our US colleague.
So I thought it would be helpful to provide a few examples of what the UK Government is doing in the cloud with my company @UK PLC a single UK cloud provider.
The major projects so far this year are as follows:-

February this year the National Audit office used our cloud services for an in depth analysis of NHS procurement (£ 500 million savings identified)
August go live of the worlds first nationwide B2B card based marketplace for all UK Universities and Colleges
September launch of nationwide schools marketplace with all items carbon neutral voluntarily offset by the suppliers as part of a drive for savings and sustainability(another world first).
September start of data collection for the NHS carbon footprint project another world first looking at the carbon footprint of all the items purchased by the NHS.

Last year Richard Benyon the Minister for the Natural Environment launched our Green Marketplace and we passed the £ 100 billion of spend Analysed. We have users from every single major public sector body on our system, which we would suggest is a reasonable level of engagement in the cloud for a single supplier sample, and that our Government is doing interesting and innovative things.

New email system is brilliant

ronaldduncan — Mon, 04 Oct 2010 22:46:06 +0000

@UK PLC’s new cloud email system is brilliant.

I am in the midst of parallel running our new cloud email system vs Microsoft Exchange, and we have just been sending out 100′s of thousands of emails as part of our new green marketplace launch. The new cloud email system has been fast and responsive through out, where as exchange has frozen the computers that are connected to my exchange account.

I wanted to make sure that customers were not upset by our sending out emails, and so had all the emails send out with delivery and read status turned on to feedback to my own email address.

The result was hundreds of thousands of mails hitting my account at the same time, which was a great test of both systems. Sue and my pc’s ground to a halt with outlook connected to exchange, but my PC was fine when just connected to our Cloud Email 4 business system.

In terms of managing the email, it was quick and straightforward with a reasonable imap client. I have been experimenting with Mutt, which is brilliant if you like vi. However, the great thing about imap is that virtually all email clients from Outlook to thunderbird support imap and work fine.

I was able to move tens of thousands of emails around quickly from folder to folder, and I was delighted with the speed and responsiveness. It was remarkable how much faster it was than exchange, especially as exchange is sitting on our local network and has about 30 accounts and our cloud email 4 business is in a remote data centre and has tens of thousands of domains a huge number of clients.

We are going to try and switch over from Exchange to cloud email 4 business as quickly as we can.

Duncan Conjecture

ronaldduncan — Sat, 28 Aug 2010 09:05:28 +0000

Last year I had an insight into how matter escaped from the black hole at the start of the Universe.

It seemed brilliant to me!

So I did a bit of research into the current state of the art, and checked my membership of the Institute of Physics.

I am now a lapsed Chartered Physist.

I forgot to pay my dues back in 2003 or so, and given that it has taken me a year to have the time to think about my insight again. I am very unlikely to either have time to go through the hoops to re-establish my status as a Charted Physicist or do the required research to validate my insight.

So here is my insight.

The known facts are that the universe started from a “Big Bang”, and that when large amounts of matter comes together you have a black hole, and that http://en.wikipedia.org/wiki/Hawking_radiation decreases as the mass of the black hole increases.

The implication is that some thing different must have happened for the universe to exist and our current matter to have escaped from the black hole at the origin of the universe.

The other key observation is that we live an an asymmetric universe. i.e. Our universe is made up of protons, neutrons and electrons as opposed to antiparticles.

http://en.wikipedia.org/wiki/Big_Bang#Baryon_asymmetry and http://en.wikipedia.org/wiki/Baryon_asymmetry

The final element is that as energy levels increase larger mass particles become stable and the predominant form of matter.

My conjecture is as follows;

At very high temperatures there exists a very massive particle(escape particle) that is able to tunnel out of a high energy black hole. We know that the initial black hole must have had very high mass, surface gravity and temperature. Normally the surface gravity would limit the radiation temperature, and the question is how evaporation occurred given this limit..

The conjecture is that big bang occurs when the black hole reaches an internal temperature and energy that allows the escape particles to exit by quantum tunnelling. Theory predicts that for small black holes the evaporation is very quick generating a gama ray burst at the end.

If there is a particle with a high enough energy that it is more likely to escape than not, there could have been a very quick phase of evaporation of the original black hole. Until the internal pressure and temperature reduced to a level where escape particles where no longer produced.

Followed by a series of energy releases as the escape particle broke down into lower energy particles until we reached observable particles like bosons, quarks and baryons (protons, neutrons etc)

The assumption is that quantum tunnelling produced an escape particle and an anti escape particle, and an imbalance could account for the observed Baryon Asymmetry.

Because the evaporation would be a form of black body radiation it would be consistent with the observed cosmic back ground radiation.

Proof of the conjecture, would be discovery of the remnant blackhole, since as the mass and energy decreased at some point it would become a normal if very large black hole and stop emitting.

This might be observable as a void, combined with gravitation lensing of objects on the other side, since it would have swept up the matter that did not escape a sufficient distance. during the 377,000 years current theory predicts from the initial singularity to the start of the cosmic microwave background radiation.

Bye Bye Gas Guzzlers – revisited

ronaldduncan — Sun, 11 Jul 2010 10:07:56 +0000

Back in 2006 the RSA set up a personal carbon trading scheme for fellows, and I joined in.
The result was Bye Bye Gas Guzzlers

Author: RonaldDuncan

Posted: Oct 5 2006 10:03AM Comments: 2439

We are a two car household currently with a 3ltr Subaru Outback and a 4ltr Jaguar XK8, but switching to a pair of Toyota Prius Hybrids.

We get as good a driving experience, quieter and much smoother and 1/3 of the fuel consumption, and if we lived in town we could have got a pair of fully electric cars and made even larger reductions.

For the financially minded the Prius is depreciating very slowly and has an 8 year warranty on its hybrid drive train.

For the technical it has both battery and petrol engine and recharges the battery when you brake thus reusing the energy and saving significant amounts.

It was a popular post, and the switch to our Prius saved us a lot of money as well as reducing our carbon footprint. However, there was much, much more that we could do:-
The key areas were not as individuals but through our companies and organisations. In my case I had a really look at my company @UK PLC and how it could help the environment, whilst I am the largest shareholder in @UK PLC, and I started the company, in October 2006 I was in a particularly powerless position within the company.

We floated in December 2005, and things had exploded in September 2006 with our half year results. My wife Lyn was chief executive and fighting to keep on the board, and I was trying to be an invisible technical director that just kept everything running amidst the mess.

Lyn was ousted after a very stressful battle, our sales and marketing directors were fired and a new chief exec brought in by our non execs. The new chief exec had no chance because the market had dried up, however it took until October 2008 for enough cash to drain away, and we could start running the company again, and focusing on the future.

@UK PLC is a very low carbon, knowledge based company, so whilst we recycle and try to minimise our impact on the environment a reduction in our companies carbon footprint is not going to have much effect in the overall scheme of things.

However, our solutions make a significant effect on our customer carbon footprint, and primary effect being reduction in the use of paper, and making processes more efficient. You would think that it is easy to find out the carbon footprint of paper, and so we could quantify our impact. It was not easy, and it was clear that the information that was passed around the various databases, went back to some fairly rough studies of a rural paper mill that had published its results in an unscientific manner. This was a disappointment.

It was very clear that there was a real need for a good database of environmental data at product level, and a method for accurately calculating product level environmental footprints. e.g. Paper uses up a lot of water as well as energy, and cutting down trees, and can produce large amounts of pollution, especially in the developing world.

So, I am delighted that we now have a method to provide environmental data in a comprehensive format with transparent levels of accuracy to provide a complete picture. This will allow us to get a complete picture of our customers environmental impact, and the effect of the benefits from our solutions.

There are 3 parts to the solution

Our SpendInsight spend analysis that provides a detailed item level analysis in less than 1 week
Our e2class database of over 3.6 million commonly purchased items and their environmental impact
Our new GreenInsight that combines SpendInsight detailed data with e2class environmental data to provide a detailed picture of an organisations environmental impact

This allows us to accurately benchmark our customers, and measure the change from our projects.

It will have a much bigger effect than a family swapping out a couple of gas guzzlers.

Eternity

ronaldduncan — Wed, 27 Jan 2010 23:24:01 +0000

In his novel The Picturegoers, David Lodge gives a graphic description of infinity as it applies to time: “Think of a ball of steel as large as the world, and a fly alighting on it once every million years. When the ball of steel is rubbed away by the friction, eternity will not even have begun.”

It give a nice feeling of what infinity means.

However from a physics point of view there are a couple of little issues.

If the ball of steel is surrounded by air, you have an atmosphere with a lot more friction than your fly landing, so it gets rubbed away a lot quicker.

If it is not surrounded by air, the fly will not be able to fly and will hit the sphere with an almighty splat. You could count the millions of years by the splat marks. The sphere might get melted and a bit of debris knocked off by each splat, but the gravity of the sphere is very high so none of the debris will escape and you will end up with a steel sphere covered in desiccated fly. The fly will dry up and freeze in the vacuum of space before it hits the sphere.

And who is going to shunt the poor fly out every million years into space to fall onto the sphere with a silent crunch.

Someone suggested using this analogy, to seriously contemplate eternity. I think I understand eternity, and infinity, and using this analogy got me thinking about a sphere the size of the world, and how a fly could land on it. At which point the analogy broke down.

Another analogy would be to use human history assuming that we are now at 2,400,000 years where 1 hour is 100,000 years. Then recorded history is a minute or two. You then expand that up to the history of the earth, which is much longer, and you have a period of time that if it were a 24hour atomic clock powered by the energy of the sun, then eternity is much longer than the life of the clock and this history of the earth is much less than the first second of the sun powered clock.

Unix on Jamie’s Laptop

ronaldduncan — Sat, 14 Nov 2009 11:27:20 +0000

I wanted to install unix on Jamie’s new laptop for him. It is a wireless enabled notebook from Novatech with an AMD x64 processor.

I kept notes as I went along, unfortunately the windows box I was putting the notes on restarted, so the following is from memory.

The distros I tried were as follows:-

freebsd 7.2 and 8 RC2,
opensolaris 09 06
centos 5.4,
debian 5.03
fedora 11
gentoo 20091029
knopix v6
openSUSE 11.2 RC2
PC-BSD 7.1.1
Ubuntu 9.04 and 9.10

The quick summary is that only a few of the distro’s installed at all, and almost all were unable to deal with the network card. This is because the card driver is only available in the linux 9. There is some info on the hardware and a how to at http://forum.novatech.co.uk/showthread.php?t=15068, but if it is your only machine you need to get networking set up before you can browse and find things.

The really bad

Centos – install crashed can not remember why now
Fedora – install crashed can not remember why now
PC-BSD – install got to the point after configuring disks and the could not mount the CD, that it had used to boot off

Installed, but did not have network card driver

Freebsd 7.2 – I was able to get an IP address that I could ping from another machine so probably me, could not get dhcp to work or to get it to look out at the world to download files
opensolaris
debian
gentoo
knopix
openSUSE
Ubuntu 9.04

Fully worked from install

Free BSD 8 RC2
Ubuntu 9.10

So the quick review of the various distros, starting with what worked and going downwards.

Working Distros

Ubuntu 9.10

This is definitely the easiest to install and use and just worked straight off, with a quick painless install. There was a quick download of the wireless drive and make; make install, reboot and both wired and wireless networking were working perfectly.

This is what Jamie’s laptop is using.

Free BSD 8 RC2

Personally, I like the ugly text based BSD installs, because they work. However they are a very long way behind Ubuntu if you are not a complete techi.

FreeBSD installed fine, DHCP worked with the wired network, and I was able to install everything from ports, apart from Open Office, which I downloaded.

Issues

I love ports, but compiling and installing X11, Gnome2, etc takes a long time.
Open office installed fine using the compiled version
Java is a real pain – because of sun licencing

Java on Free BSD

I installed wget, because the Gnome2 browser was a bit iffy about where it tried to save files. Which was a problem for the various things that wanted me to accept licence terms around “very evil SUN” java. I had to download somethings onto the windows box and scp them over.

I also needed to edit the make files and filelist since things had changed since the port was created, and I was not able to find the tzupdate file that was in the port.

By this time I had found the Novatech howto for the wireless side of things, and decided that Ubuntu would be much, much easier for Jamie.

Did not Support network card

Free bsd 7.2, Gentoo, Debian

Card was required to work for rest of install so did not get much further, all fairly similar text based working installers

Knopix, Open Suse, Ubuntu 9.04

Nice graphical working installers, and OS installed fine. Just did not have network support built into kernal.

OpenSolaris

First time I have tried openSolaris, and it was really good. It has a nice set of hardware diagnostics, that worked. Unfortunately it did not support the hardware, and I did not try compiling the drivers.

Network Manager issues

Goes mad if network card not supported same issue across all linux distros that did not support the card in the kernel.

Summary

Ubuntu, has a more up to date kernel at the point when the laptop was updated, and this made it the only distro to install out of the box. Free BSD also had kernal support for the system, however there was significantly more work to make it work and it was not appropriate as a starting form of Unix.

Text File formats – ASCII Delimited Text – Not CSV or TAB delimited text

ronaldduncan — Sat, 31 Oct 2009 15:09:03 +0000

Unfortunately a quick google search on “ASCII Delimited Text” shows that IBM and Oracle failed to read the ASCII specification and both define ASCII Delimited Text as a CSV format. ASCII Delimited Text should use the record separators defined as ASCII 28-31.

The most common formats are CSV (Comma Separated Values) and tab delimited text. Tab delimited text breaks when ever you have either a field with a tab or a new line in it, and CSV breaks depending on the implementation on Quotes, Commas and lines. Sadly Quotes, Commas and Tab characters are very common in text, and this makes the formats extremely bad for exporting and importing data. There are some other formats such as pipe (|) delimited text, and whilst better in that | is less frequently used they still suffer from being printable characters that are entered into text, and worst of all people, when they look at a file format and see the delimiter, think that it is a good idea to break things up with in fields using the same delimiter as the file format.

The most anoying thing about the whole problem is that it was solved by design in the ASCII character set.

If you use ASCII 31 as your field separator instead of comma or tab, and ASCII 30 as your record separator instead of new line. Then you have a text file format that is trivial to write out and read in, with no restrictions on the text in fields or the need to try and escape characters.

It is even part of the design of the file encoding system. The ASCII standard calls these fields

31 Unit Separator
30 Record Separator

And ASCII has two more levels with Group and File Separators

29 Group Separator
28 File Separator

See http://en.wikipedia.org/wiki/Unit_separator and
http://en.wikipedia.org/wiki/Delimiter#ASCII_Delimited_Text

In summary ASCII Delimited Text is using the last 4 control characters (28-31) for their purpose as field and record delimiters and not using CSV (Comma Separated Values)