The use of cookies on websites has become an increasingly hot topic over the past few months. A new EU law – specifically The Privacy and Electronic Communications (EC Directive) (Amendment) Regulations 2011 (PECR) – requires that websites ask for consent before setting cookies that aren’t “strictly necessary” for the operation of the website. We’ve put together this post to give you an overview of how we are approaching this law on the beta for the Single Domain.

“The law was meant to protect the privacy of people using the Internet. To accomplish this, the EU made over 90% of websites illegal”
- Oliver Emberton, silktide.

Back in May I wrote about the new EU law on cookies that came into effect this year and will be enforced from May 2012.

On Monday, Matt Jukes pointed me to a free ebook (written by Oliver Emberton from silktide) about the law and what it means for websites within the EU.

I think this book is an essential read for anyone who develops or commissions websites. It should also be required reading for those who developed the law.

The more I look into this law, the more stupid & ridiculous it becomes. While I support what it is trying to do, the implementation is probably the worst possible solution. To use a common phrase, it is like using a sledgehammer to crack a nut. However, it’s actually more like using all the sledgehammers in the world plus a few Acme style anvils at the same time to try.

It is worth pointing out that the law doesn’t just effectively ban cookies. It also covers flash cookies, the HTML5 local storage facilities, java storage functions, silverlight and basically anything that stores data at the client end.

If this law is fully enforced, I think it could be the end of any form of innovation in European web development.

I’ve been a regular user of trains for nearly 10 years.  When I was in college, I had to use the train to get to classes.  Before I moved to Cardiff in December, I commuted to work by train every day.  Now, I travel by train to meetings in London on a pretty regular basis.  I like to think that I’m a bit of an expert at train travel.

One of the things that always intrigued me is the rail performance figures that are published every few weeks. Somewhere in a secluded corner of a station will be a poster, sometimes titled “Passenger Charter Figures”, that is supposed to tell you how many trains run ‘on time’.

Normally (at least every time I’ve ever looked), the train company seems to have achieved its targets.  I always thought this was quite strange given the number of times my train was delayed or didn’t show up at all.

It turns out that there are numerous loopholes and tricks* they can use to make their figures look good.  For example, on time doesn’t mean on time, it actually means the train arrives at its destination no more than 5 minutes late (10 minutes for intercity travel).  Note the use of ‘destination’.  The train could depart 30 minutes late and drop me off at my stop 25 minutes late; but as long as it reaches its destination less than 5 minutes late (which for my journey could be nearly 5 hours later), the train is classed as ‘on time’.  Timetables have lots of padding built in which helps with this.

Luckily, last year I got to work on a joint project with Passenger Focus (the rail passenger watchdog) on ways of making these figures more useful.  We produced a basic HTML prototype to show how more detailed performance figures could be used by the public to hold train companies to account.

Unfortunately, even though much more detailed performance information is collected (the time a train passes each monitoring point on the network is recorded by Network Rail), it isn’t published, let alone open for re-use.  It’s generally considered ‘commercially sensitive’ – which is ridiculous.  Even if we wanted to use the, in my view, crap existing figures we wouldn’t be able to – it isn’t open.

We demoed our prototype and our ideas to a few people including the Office for Rail Regulation (ORR) – the public body that sets the terms of rail company operator licences – in the hope that they would either force the rail companies (or Network Rail) to open up the data.

We are, of course, not the only ones who have been trying to get this data opened up. Many other individuals and groups have been campaigning to the plethora organisations involved. Some of them have been campaigning for a lot longer than we have.

Last week the government announced a huge number of public datasets to open up.  Amongst these, two important elements stand out:

• Office of Rail Regulation to increase the amount of data published relating to service performance and complaints by May 2012
• Rail timetable information to be published weekly by National Rail from December 2011.

Obviously, everyone involved in the various campaigns have managed to get the issue to register on the radar of ORR!  Publishing this data is a huge & welcome step forward.  Everyone involved deserves a pat on the back.

But, we’re not there yet. We now need to make sure that the published performance data is detailed enough to be useful to the public.  Then, once it’s published, we need to make sure there are tools available that use it, so that the public can finally start to hold their train companies to account.

* the train companies would say that their figures are independently audited and there are no tricks and loopholes.

Photo Credit: AndrewHA (Flickr CC)

Edit: added a few links.

Essentially, this law blocks the use of cookies until a user has been asked for (and has given) informed consent. Francis Davey, the fantastic tech legal expert has written an in-depth blog about the changes.

The more I look into it, the more I think it will be an uphill struggle to fully implement this law. So many services use cookies such as embedded Vimeo videos, Facebook, Twitter and even WordPress. The law could make services like Google Analytics completely useless if the have to ask every time they are used. Of course Google is a US company and could potentially argue that EU law doesn’t apply (which would leave EU website owners with bit a problem).

If fully implemented, this law could significantly change the way websites work in Europe, and not necessarily for the better. However, at the same time, it could vastly increase the privacy of website users.

So, given that the law comes into force this week and there are so many problems challenges, what can website owners do? Well, the Information Commissioner (ICO) has published some useful guidance (pdf) which you should read. Luckily, they aren’t expecting full compliance just yet!

In the public sector, we’ve been given some temporary guidance on some small steps we can take. This includes auditing our use of cookies and producing a list of what they are, what they are used for and how long they last. This list should be published on the site along with the website privacy policy.

I produced a first version for work’s Recalled Products website yesterday and I’m working through the others over the next few days. If you want to copy / reuse / alter my example, go right ahead. I’d also appreciate any comments on how this could be improved.

Hopefully over the next few months, the ICO & government will put together some more comprehensive guidance on how website owners can move towards full compliance. For now though, start producing these lists of cookies!

December 2011

In early December we launched a directory of post offices owned by and operated for communities across the UK.  Consumer Focus commissioned the Plunkett Foundation and Social Enterprise Coalition to compile the data which CF Labs then presented through this site.

The site allows users to search for community post offices in their area, find out where they are located and see what services they provide.  The site also provides some statistics on core activities and the data is exposed in JSON format.

A few days later we launched a service to publish ratings of how energy companies handle complaints.  We’d worked closely with our colleagues in the Energy team at Consumer Focus to design and develop a meaningful way of showing this data.  We produced a number of prototypes which were refined through several stages to a shortlist of 3 possible options.  These were then independently tested by TNS-BMRB in September which helped us to pick the final option.

The system ranks each of the ‘Big Six’ energy suppliers using stars – 0 for the worst performing companies, 5 for the best.  The site publishes a snapshot of the data (the last quarter) and also a more detailed historical view.  All the data is also published using JSON and CSV formats.

On Friday, Charlie and I attended a hack day with a difference. The ScraperWiki Hack and Hackers Hack Day was designed to bring designers, developers (hackers) and journalists (hacks) together to see what they could do. While other hack days may have involved journalists in a media capacity, this event involved journalists from the beginning.

After an introduction to how the day would work, we all split out into groups and started to work on our projects. I worked with Charlie, two other journalists (Ian and Annamarie) and another developer (Edmund) to investigate new election candidates in the safest seats in the country. We mashed together data on constituencies, candidates and then did some good old fashioned journalism to fill in the blanks

Other teams worked away on other projects, only interrupted by the food and drink provided. Finally, at the end of the day we sat down to present our work – eight projects in total. It was amazing to see what different directions the different teams had gone in.

A few of the projects used data to begin investigating a story. One group looked at the transport data released by data.gov.uk and mapped it onto constituencies; another group looked at awards paid out by Enterprise Ireland. Richard Pope looked at the register of interests for London Assembly members and plotted how often and where they were taken out to lunch by third parties. The largest group produced “They Write For You“, a mashup that demonstrates how many MPs write articles in newspapers.

Other projects were more focused towards building tools such as a system that allowed you to classify the output of data.gov.uk data (e.g. Excel, CSV, PDF, linked data etc.). A last minute entry also wrote a scraper for UK bank holiday dates (a much needed tool for developers!). A final fun project scraped all of the horoscopes for the last year and analysed the most used words for each star sign -producing some fantastic visualisations.

The day aimed to explore how hacks and hackers could / would work together. I think the quality of the projects created show that it is a good match. Journalists have an eye for a story that us developers might not have. While we may enjoy throwing tools together and mashing up data, we don’t necessarily know what makes a good story. Similarly, journalists don’t necessarily know or understand what developers do and how we do it.

Lessons were learned on both sides and show the growing importance of data based journalism. As an experiment, I think it was a success and certainly should be repeated again.

If it is – I will be first on the list to attend!

Thanks must go to Channel 4/4IP for providing the venue for the day, Richard Pope and everyone from ScraperWiki for organising the event and helping us get working.

Sometimes I come away from the office and struggle to see what we have actually achieved. When you are stuck into day to day work, it is quite easy to lose sight of the bigger picture.

This isn’t helped by the fact that part of our team’s aim does not have a tangible outcome. We spend a sizable portion of our energy on promoting our open principles within the organisation and converting people to our way of thinking. It isn’t an easy task, as I’m sure other people in this field will confirm but I’m starting to feel like we are making some headway.

The first report (of which I hope there will be many) has just gone online, in a format allowing the public to comment on each part of the document. The Consumer Focus website is now powered using the open source WordPress Content Management System (joining the likes of bis.gov.uk and number10.gov.uk). Other developments are also in the pipeline for the next few months and all have stemmed from us ‘spreading the gospel’.

This sort of work is going on across Government on many different levels, but recently there seems to have been a bit of a breakthrough. There is lots of things happening at the moment which you might find interesting. I’m going to briefly run through them.

Data.gov.uk

This is the UK version of the USA’s Data.gov and has been operating in a restricted beta for a few months. Government departments have been posting raw datasets for developers to use for free in their open projects. From sometime in January, the site will go live with over 1100 datasets from central government.

Ordnance Survey, NHS Data, Office of National Statistics Data

Work is underway to open up more datasets and make them freely available. These include health data from NHS Choices, mapping data from Ordnance Survey (OS) and statistics from the Office of National Statistics. Consultations are planned for early in the new year with aims to open them up by April 2010. There also seems to be some movement on publishing data of how the money allocated in the 2008 Pre-Budget report has been spent (possibly by spring, with further data released in the summer).

More information about the data releases can be found in the Government’s recent strategy ‘Putting the Frontline First: Smarter Government‘ and on the Government’s Digital Engagement blog.

Crime Maps

The various Police forces around the UK have released crime statistics using their new mapping service. I’ve had a bit of a play around and it looks pretty good. You can see lots of information about your local area and download the data in CSV format.

More information about the Crime Maps can be found on the Directgov website.

Post Code Data

A few months ago I wrote about the issue of post codes after ErnestMarples.com was forced to shut down by the Royal Mail. Since then, a campaign has been ongoing to set the post code free. Well, it looks like someone might have listened. As part of the opening up of OS data above, a consultation will be taking place shortly on the opening up of post code datasets, which could see the data released by April 2010.

More information about the release of post code data can be found on the Ernest Marples blog.

Digital Economy Bill

The Digital Economy Bill has recently been introduced to Parliament and is due to go to committee stage shortly. This bill contains a large chunk of the actions and recommendations from the Digital Britain Report earlier in the year (with a number of strange and scary additions). There is a lot of talk about this at the moment and the effect it will have on the digital life and innovation in the UK. You can follow the progress of the bill on the Parliament.UK website, or on twitter using the #debill hashtag. If you want more information on the bill and how it might affect us, there are a large number of blogs and websites dedicated to the issue.

Directgov

This brings me to one of the last bits for this entry, and I make no apologies for the shameless self-promotion here.

Direct.gov.uk is the UK Government’s official website for the general public. It aims to collect all the public facing information into one place and make it easy to find. Unfortunately, its usefulness and ability to achieve this aims has been called into question. Personally, I’ve never liked the website and can never find information I want from the site (in fact, I tried several times to find information relating to bits in this post but gave up).

Consumer Focus has recently published a report on the website that analyses how well it delivers on its promise. As I alluded to earlier, this report is available online in fully commentable format. We are encouraging everyone and anyone to send us their thoughts on Directgov, what the problems are and what can be improved. Similarly, if you think it is doing something right, we want to know that too. You can find out more about the report at http://directgov.consumerfocuslabs.org.

I believe there will be a series of workshops planned for early in the new year to identify ways of engaging people in digital public service development. You can sign up on the site to be kept informed (and I’ll no doubt be publishing the information here as well).

Looking back at everything that has been achieved and everything that has happened in the last nine months, I’m actually quite excited at what we can do by this time next year (election permitting of course!). I have my fingers crossed 2010 will continue making leaps towards open government.

In Govilon, we’ve tried to make this process easier. We’ve launched a “Village Information Pack” (VIP) – a small A5 folder which is delivered when someone new moves into the village. The pack contains a list of community groups and services offered in nearby villages (post offices, libraries, leisure centres etc). It lists bus times, school term dates and opening times of the shop & pubs. We also allow local businesses to put in a leaflet about their services.

The initial version of the pack was delivered to every house in the village, and updated ones are delivered by volunteers whenever we hear about someone moving in. So far it has been well received.

Recently, the Government began the process of opening up its datasets in a similar way to Data.Gov in America. The Newspaper Club has taken some of this data and produced a prototype of an information pack for people moving into a new area. It gathers information about local services, environmental information and crime statistics from the Government’s datasets and presents them in a small newspaper. They’ve also gone a step further and included maps, travel times to certain locations, NHS information and council details.

This is a fantastic idea and makes really good use of data which was not previously available. It shows what can be done when you make public data available for free. Something like this would really complement the village information pack we’ve created for Govilon.

You can find out more information and some pictures of the prototype on the Newspaper Club website.

The issue surrounds a small piece of text – the Post Code. It sits at the bottom of every address (similar to the Zip code in the USA). It is the holy grail of data in the UK. It maps each place in the country to a physical location. The Royal Mail uses it to assist with routing mail from A to B, developers use it to verify addresses and locate other services nearby. Unfortunately, access to this database costs thousands (approximately £3800 per website per year).

In the majority of cases, small not-for-profit websites or online experiments just can’t afford to pay out for this license. I know from experience – we don’t have a postcode lookup on CllrTweeps for exactly this reason. I can’t afford to pay out the best part of a sixth of my salary (before tax) each year for a postcode licence for a personal project.

Enter ErnestMarples.com – a website that provided access to the postcode database to not for-profit websites for free. A developer could submit a Post Code to the API and get the appropriate longitude and latitude back in return. Although the legality of the site was questionable, it was used by a large number of sites similar to CllrTweeps. These included PlanningAlerts.com, JobCentreProPlus, TheStraightChoice, HealthWhere and so on.

Planning Alerts used it to good effect. It allowed you to sign up for email alerts of planning applications that were submitted near you. How did it work this out? By mapping your Post Code to a physical location. The site was so successful that other sites have cropped up that disseminate its data through other communication channels.

Unfortunately yesterday, the Royal Mail threatened legal action against ErnestMarples and the site was taken down. As of yesterday, all the above sites have gone offline too. Sites of real value to people in the UK have been taken down because of the Royal Mail’s actions.

Now some people will argue that they shouldn’t have been using a dubious source for their data. But as I’ve explained before, they don’t really have much choice. It is a case of using a dodgy dubious source, or not building the site at all. None of these sites were run on a for-profit basis. None of them had a spare £4000 lying around to buy the license.

The Royal Mail claims that they were losing money because of the site (although they have yet to answer enquiries about how much they’ve actually lost). I’m sceptical about this – none of these sites could have afforded to buy the data anyway, so how they could have lost money I do not know!

Lets not forget that the database was created using public money through taxes paid for by the people of the UK. This is public data that is of benefit to the whole of the UK.

By closing down this site and refusing to provide a not-for-profit licence for their Post Code database, the Royal Mail is stifling innovation in the UK.

Last week I attended the Future of Web Apps conference in London. I was amazed at how behind we are in the UK compared to the USA when it comes to innovation. I’m now convinced it is partly due to the failure of organisations like the Royal Mail and Ordnance Survey to modernise. They are acting in the same way as the music industry and trying to stop innovation rather than cultivate a mutually beneficial environment.

I’m sorry to see PlanningAlerts and The Straight Choice die. But maybe it will be the kick up the backside that the UK needs to get its house in order.

Until the Royal Mail releases its Post Code database with a free (or very very cheap) not-for-profit license, we will never have a Digital Britain.

What is the government thinking? The BBC and The Times are reporting that the Home Office are planning to put together a national database of all phone and email communications. It is another of the schemes proposed to combat terrorism and improve national security – but like all the others, it is just a disaster waiting to happen.

I have a number of problems about introducing a database such as this from the point of view of both a service provider, and an email user. In my opinion, this is a step too far (but then again, so are some of the other schemes).

From a service provider’s point of view, I do not see how it is going to be possible. I operate 2 mail servers (3 if you include the one I use for testing). The company I work for operates 4, and my internet provider has loads more. If you think that a small operation like mine is using 2 servers, how many mail servers must be operating in the UK? This is a hell of a lot of data! How are we expected to pass this information along to this national database?

If it is going to be compulsory to provide this data – and I suspect it is or there wouldn’t be any point having the database in the first place – then how is it going to be policed? How would the government know that I am running a mail server at home and not putting the data into the database?

Any tech person with half a brain knows how to set up a mail server, so this could become the standard way for those acting against the national security of the UK to communicate (if it isn’t already). When you consider that hundreds of thousands of computers are sending out SPAM emails without their owners realising, how would the government check that these weren’t being used for terrorists to communicate? On the other hand, what would stop them setting up emails servers in another country? For that matter, there isn’t anything stopping me from moving my servers to another country to avoid having to comply with the law.

Another more worrying thought is that the government could attempt to ban the use of mail servers other than those that are registered with them and provide them with the data. Again though, it would be impossible to police. However this would cause significant financial hardship on a lot of small businesses that operate their own systems. These companies would either have to move their servers out of the country, or pay whatever is required to comply with the law.

From a more personal point of view, why do they need the information? They already force phone companies to keep records of my calls for 12 months; they already keep details of my tax payments, capture me on CCTV countless times a day and even store records of where I’ve driven. Now they want my emails as well?

Bearing in mind how well the government has looked after my data in the past, I’m not sure I trust them with this amount of data. All it takes is for someone to burn off a DVD of the data and sell it to the highest bidder. If systems like Phorm become more widespread, how long will it be before the websites I visit become the next part of this database? It certainly makes sense that it would be next on the list.

Luckily, the Information Commissioner has suggested that this is a step too far. We can only hope that this idea is stopped before it gets much further.

Edit: I suppose I should mention that under current legislation, email providers are supposed to keep this information already and hand it over when the police come knocking on the door.  However, these plans would involve all the data being held in one central place rather than with the provider.