On Friday, Charlie and I attended a hack day with a difference.  The ScraperWiki Hack and Hackers Hack Day was designed to bring designers, developers (hackers) and journalists (hacks) together to see what they could do.  While other hack days may have involved journalists in a media capacity, this event involved journalists from the beginning.

After an introduction to how the day would work, we all split out into groups and started to work on our projects.  I worked with Charlie, two other journalists (Ian and Annamarie) and another developer (Edmund) to investigate new election candidates in the safest seats in the country.  We mashed together data on constituencies, candidates and then did some good old fashioned journalism to fill in the blanks

Other teams worked away on other projects, only interrupted by the food and drink provided.  Finally, at the end of the day we sat down to present our work – eight projects in total.  It was amazing to see what different directions the different teams had gone in.

A few of the projects used data to begin investigating a story.  One group looked at the transport data released by data.gov.uk and mapped it onto constituencies; another group looked at awards paid out by Enterprise Ireland.  Richard Pope looked at the register of interests for London Assembly members and plotted how often and where they were taken out to lunch by third parties.  The largest group produced “They Write For You”, a mashup that demonstrates how many MPs write articles in newspapers.

Other projects were more focused towards building tools such as a system that allowed you to classify the output of data.gov.uk data (e.g. Excel, CSV, PDF, linked data etc.).  A last minute entry also wrote a scraper for UK bank holiday dates (a much needed tool for developers!).  A final fun project scraped all of the horoscopes for the last year and analysed the most used words for each star sign – producing some fantastic visualisations.

The day aimed to explore how hacks and hackers could / would work together.  I think the quality of the projects created show that it is a good match.  Journalists have an eye for a story that us developers might not have.  While we may enjoy throwing tools together and mashing up data, we don’t necessarily know what makes a good story.  Similarly, journalists don’t necessarily know or understand what developers do and how we do it.

Lessons were learned on both sides and show the growing importance of data based journalism.  As an experiment, I think it was a success and certainly should be repeated again.

If it is – I will be first on the list to attend!

Thanks must go to Channel 4/4IP for providing the venue for the day, Richard Pope and everyone from ScraperWiki for organising the event and helping us get working.

The past 36 hours haven’t been the best for those travelling with Eurostar. Five trains were stranded for hours in the Channel Tunnel without power, light, food, drink, heating or information. A further train became stuck near Ebbsfleet yesterday evening.

The focus today is on the cause of the incident and the safety procedures. However the issue that concerns me is how public service companies like Eurostar handle communication in such emergencies. Poor communication can lead to frustration, anger and even panic. I’ve experienced this in much less drastic circumstances when commuting to and from Cardiff by train.

Today the media are reporting that Eurostar passengers were given no information about what was going on. People at stations had similar experiences. Even the public and media had no idea of the unfolding events until late on Saturday morning.

It seems a standard practice for UK transport companies to ignore incidents, keep quiet, disappear and leave customers to their own devices. It doesn’t take much to keep people informed – even if there isn’t any ‘useful’ information to give them at the time. Even a simple “The track ahead is flooded; we are awaiting an engineer to confirm we can proceed” provides re-assurance that something is happening.

There are three distinct audiences in any incident, each with different information needs. Those directly involved (i.e. train passengers) who need to know when they will reach their destination. Those affected through waiting to travel or waiting to meet passengers, and everyone else (for example the media and general public).

Open communications to all three groups is key, but how can it be done effectively? Which channels should be used? Well, every possible channel actually. If you are preparing a message it doesn’t take much more effort to push this out on all available options

By example, recently I’ve read reports that Eurostar didn’t start updating Twitter until 14 hours after the incident began. Some have suggested that Twitters low usage justifies it being left until last. But for passengers on trains with phones and computers, this may be the only way that information can be easily received.  It only takes one person with access to disseminate the details.

Communication (or the lack of) is a big problem when it comes to any form of crisis. Yet the prevalence of new social media communication actually makes communication easier than ever. Businesses of all kind need to recognise this and plan to make best use of it. I hope that Eurostar and those providing public services learn the communication lessons of this weekend.

On a final note – Eurostar’s social media company behind (We Are Social) has come in for what I believe is unjustified criticism. They actually did a great job given difficult circumstances and went well beyond the requirements of any contractor. The poor communication lies squarely with Eurostar. You can read about “We Are Social’s” involvement on their blog and a interesting response from Grapevine Consulting about the criticism levelled on We Are Social.

Karoo disconnects file sharers
Hull’s only internet provider – Karoo – has been disconnecting people accused of file sharing without the right to appeal. Although they’ve now changed their policy, as a monopoly provider, they have a greater responsibility to provide connections to all residents of hull.
Source: BBC News

Government departments are sticking with IE6
Aside from the fact that web developers (myself included) hate IE6 because of its complete lack of standards support, there are big concerns over its security. IE6 is over 8 years old and has been superseded by not 1, but 2 versions. Despite this, a number of Government departments/agencies (including MoD, DCSF and others) have no plans to upgrade to newer browsers.
Source: Kable

London – Swansea to be electrified
The train line between London and Swansea is to be electrified at last. Fantastic news for South Wales and about time it happened.
Source: BBC News

Publicising product recalls
Consumer Focus Labs has launched its first project (and several major revisions since). You can now sign up to receive notifications by email and RSS for product recalls from the EU.
Source: CF Labs Blog 1, CF Labs Blog 2

Firstly I need to make a few disclaimers. Obviously the views expressed here are my own and not that of my employer – Consumer Focus (if you want the organisation’s view, you can see it here). I’ve also not read every paragraph in the 240 page report – some parts nearly sent me to sleep, so I skipped past them to some of the more interesting sections.

Overall, I can’t help but feel a little bit disappointed with the report. To me it doesn’t seem to go far enough – although a guaranteed 2Mbps broadband connection to every home in the UK makes a good headline, it’s poor compared to other countries such as South Korea (which is rolling out 1Gbps connections). It also skirts around some issues such as copyright fair use and leaves a lot to chance.

The report outlines 2 phases of broadband development in the UK. The first is the Universal Service Commitment (USC) which aims to provide a base to the whole country; the second is the Next Generation Broadband Network (NGB) which will provide for the future.

Universal Service Commitment (USC)

Digital Britain outlines its intention to provide every home in the UK with a minimum of 2 Mbps broadband by 2012. This will be part funded by the production of the NGB networks, with the rest funded through the underspend in the Digital Switchover budget. Initial thoughts are that this is great – everyone in the UK will have broadband of a ‘satisfactory’ quality. However, things soon start to fall apart.

Countries like South Korea are currently achieving 50 Mbps as a minimum, with some places getting as high as a 1 Gbps connection. When put in this context, 2 Mbps is exceptionally poor. Also, when you delve into a bit more detail, it turns out that this speed is actually optimistic.

Some homes will not be able to get broadband though traditional means, and so the Government is going to rely on mobile networks and satellite to provide the USC in these areas. The report suggests that 2 Mbps is unlikely for these houses, and connections of between 1 and 1.5 Mbps is more accurate. So actually, a 2 Mbps USC isn’t really correct.

Next Generation Broadband (NGB)

This is the next phase in broadband development and aims to provide broadband speeds of around 50 Mbps by using technologies such as Fibre to the Cabinet (FTTC) and Fibre to the Home (FTTH). Digital Britain expects that market investment will fund this deployment and anticipates that it will reach approximately two thirds of the population.

Assuming BT and the mobile networks do reach this target, that leaves about a third of the UK back on the not quite 2 Mbps USC broadband. This quite obviously is not acceptable, so a fund is being set up to provide NGB to this final third (called The Final Third Project).

The Final Third will be funded by a 50p per line per month levy on all fixed lines from next year. Contrary to some reports, this includes residential copper lines (like your BT phone line), business copper, ISDN and cable telephony lines (like those provided by Virgin Media). You will not have to pay this levy for mobile phones as they already pay license fees for network upgrades.

After all of this, it is expected that 90% of the population will be covered by NGB, leaving a poor 10% with USC speeds.

Illegal file sharing

This is of course a big bone of contention among many people. The interim report issued in January suggested the creation of a Digital Rights Agency which would be responsible for tracking illegal file sharers. This final version has moved away from this proposal and aims to stick with an industry lead effort, although supervised by Ofcom the industry regulator.

The report outlines a commitment to reduce illegal file sharing by 70-80%. This is a huge commitment, so they aren’t just leaving it to chance. If the industry effort of writing letters to account holders does not reduce numbers significantly within 12 months, Ofcom will be able to force ISPs to introduce ‘technical measures’. These might include blocking of sites, protocols, ports, bandwidth capping (speed and data transfer), bandwidth shaping and content identification and filtering.

However, for this to happen, Ofcom will need to ask Parliament to legislate.

Copyright fair use

Unfortunately this important issue gets all of a one paragraph mention in the document which makes no commitment to resolve the situation. As a result, it remains illegal to copy music you already own to your mp3 player.

Conclusion

As you can see, it is a bit disappointing. The USC isn’t quite the 2 Mbps they promised, NGB relies on the market to do the right thing, no clear direction on illegal file sharing and no movement at all on fair use.

I’ve not gone though everything in this post; there are a lot of other planned actions including the digital radio switchover, tax relief for video game makers, and the digital public service programme. Maybe next time!

Image courtesy of Media Guardian and Wordle.

Twitter is going crazy with (i reckon) an average of 700 tweets a minute regarding the problem which appears to be affecting the whole world.

More coverage at TechCrunch and Twitter Search.

Update: All seems to be fine now – problem lasted around 20 minutes

Following his election in November, his team continued their use of aspects of social media to keep people informed during their transition to government. Visitors to their Change.gov site were encouraged to submit their ideas and join in the discussion.

As he was being sworn in at the official ceremony, the new White House website was launched, featuring the new, clean look we have come to expect, simple navigation and the all important blog. This website is perhaps the best website from a government I’ve ever seen. I can only hope that this trend continues not only in the US, but here at home in the UK.

Throughout this week, social media has been making the headlines – and one site in particular has been getting a lot of attention. It is of course, Twitter – the addictive micro-blogging service which allows you to post 140 character updates about what you are doing, and things that interest you. The site has finally hit the mainstream, being discussed on radio, in newspapers and on TV. Various figures show that UK visitors to Twitter have skyrocketed in the last few months.

Not only are we now starting to see a proliferation of celebrities on the service, but also MPs, and members of local government. A fantastic site called Tweetminster has been set up to allow people to locate members of parliament and prospective parliamentary candidates (PPCs) who are using the service.

I think it’s brilliant to see government start to use social media. I feel that it is important for government and the people that are supposed to represent us connect with the country, listen to us, and respond to our ideas. I believe that the use of services like Facebook and Twitter will help reinvigorate politics and get people involved in how their country is run.

I hope that this trend continues, and if it does, 2009 could turn out to be the year government finally ‘gets’ the internet. In the meantime, I’m going to do all I can to help it along.

During the day, there were five sessions, each with four talks on different subjects, from building a scalable web application and server virtualisation, to ASP.NET MVC and how to develop .NET on Linux.

Unfortunately, due to the limited amount of time, it wasn’t possible for me to go to all of the talks; but luckily, each one was filmed for us to watch online later.  The day was really worth the early start to get to Reading!  I came away with lots of useful information that we can utilise to make the products we produce better, and a very good insight into the way the platform is going to progress over the next 12 months.

The organisers did a fantastic job getting everything sorted for the event and making sure it all went so smoothly.  Microsoft was also great for hosting the event, providing tonnes of free food and drink and looking after us all day.

This was the first technology event I’ve been able to go to – but definitely won’t be the last.  I’ve already bought my ticket for Future of Web Apps (FOWA) taking place in Dublin next March.  I’d also recommend that anyone interested in .NET development look out for tickets for next year’s developer day.

According to Michael Arrington, owner of TechCrunch, they are working with a supply chain management company who believe it is possible and can have a few prototypes ready in a short period of time (within a month).

I’ve been considering buying a UMPC for quite a while as I no longer have a need for a fully fledged laptop.  I do however still want a portable device, so I can sit down in the evening in front of the TV, browse the web and reply to emails.  If TechCrunch pull this off – and I really hope they do – I will definitely be investing in a unit.

Something like this is right in Apple’s territory – they already have the thin MacBook Air and the fantastic touch screen interface of the iPhone, but no tablet PC.  If TechCrunch get this device out quickly, Apple will have been caught asleep at the wheel.

What is the government thinking? The BBC and The Times are reporting that the Home Office are planning to put together a national database of all phone and email communications. It is another of the schemes proposed to combat terrorism and improve national security – but like all the others, it is just a disaster waiting to happen.

I have a number of problems about introducing a database such as this from the point of view of both a service provider, and an email user. In my opinion, this is a step too far (but then again, so are some of the other schemes).

From a service provider’s point of view, I do not see how it is going to be possible. I operate 2 mail servers (3 if you include the one I use for testing). The company I work for operates 4, and my internet provider has loads more. If you think that a small operation like mine is using 2 servers, how many mail servers must be operating in the UK? This is a hell of a lot of data! How are we expected to pass this information along to this national database?

If it is going to be compulsory to provide this data – and I suspect it is or there wouldn’t be any point having the database in the first place – then how is it going to be policed? How would the government know that I am running a mail server at home and not putting the data into the database?

Any tech person with half a brain knows how to set up a mail server, so this could become the standard way for those acting against the national security of the UK to communicate (if it isn’t already). When you consider that hundreds of thousands of computers are sending out SPAM emails without their owners realising, how would the government check that these weren’t being used for terrorists to communicate? On the other hand, what would stop them setting up emails servers in another country? For that matter, there isn’t anything stopping me from moving my servers to another country to avoid having to comply with the law.

Another more worrying thought is that the government could attempt to ban the use of mail servers other than those that are registered with them and provide them with the data. Again though, it would be impossible to police. However this would cause significant financial hardship on a lot of small businesses that operate their own systems. These companies would either have to move their servers out of the country, or pay whatever is required to comply with the law.

From a more personal point of view, why do they need the information? They already force phone companies to keep records of my calls for 12 months; they already keep details of my tax payments, capture me on CCTV countless times a day and even store records of where I’ve driven. Now they want my emails as well?

Bearing in mind how well the government has looked after my data in the past, I’m not sure I trust them with this amount of data. All it takes is for someone to burn off a DVD of the data and sell it to the highest bidder. If systems like Phorm become more widespread, how long will it be before the websites I visit become the next part of this database? It certainly makes sense that it would be next on the list.

Luckily, the Information Commissioner has suggested that this is a step too far. We can only hope that this idea is stopped before it gets much further.

Edit: I suppose I should mention that under current legislation, email providers are supposed to keep this information already and hand it over when the police come knocking on the door.  However, these plans would involve all the data being held in one central place rather than with the provider.

Over the last few entries, I’ve looked at some of the websites used by county councils to provide details of the 2008 local elections. Some of these websites (mainly Monmouthshire County Council) were just plain irritating, whereas others were really easy to use, but didn’t provide enough information (Swansea). Cardiff’s site was somewhere in between.

Overall though, none of these sites provided the level of detail and usability required for the age of the Internet. It is becoming less and less acceptable for councils to avoid providing this information online.

It is, however, encouraging to note that the 3 councils I have covered actually provided live information online. A number of councils I looked at during election night did not provide information at all (for example Neath Port Talbot and Torfaen. What is even worse – and completely unacceptable – is Powys County Council, who still did not have results of their elections available 4 days after the election took place!

The next set of council elections in Wales take place in 2012, I sincerely hope that by then, every council gets its act together and produces live systems that not only comply with the UK’s accessibility laws, but are also easy to use and provide the level of detail expected.

Will this happen? I don’t know – we will have to wait and see.