Yesterday, Richard Pope and I launched WhatTheyClaimed.com, a site aimed at digitising and collating all of the data from MPs expenses.

The website is based upon a system I built a few weeks ago to monitor our own expenses at Consumer Focus Labs . The site was designed to match the processes at Consumer Focus, but when Richard and I realised that MPs expenses were being published, we realised we could make use of the same codebase with a few minor changes.  I need to note at this point that the website was a personal project and not supported by Consumer Focus.

We rushed through these changes on Wednesday afternoon, sorted out some hosting and put the site up. Our initial plan was to get a few interested people to help us convert the data and start loading the information into the site bit by bit.

When the data was published yesterday morning, I realised how much of a mammoth task it was going to be. After a quick discussion, we decided to set up a generic username and password for the admin system so that lots of people could help convert the data.

Once MySociety posted the site on Twitter, everything went crazy. It very quickly became clear that the basic back end was a data entry nightmare and things needed to change. Some changes were quickly scoped out and implemented in between untimely distractions such as this and real work. We removed a number of fields from the forms including location since any location details had been completely redacted from the PDFs. I hacked some of the code to make things work and broke the site a few times in the process.

By 5.30 when I left the office to head home, I was absolutely shattered. I decided it was best to take a step back and take stock before making any further changes. I’d like to thank everyone for their messages of support through the day, and also everyone who sent in useful feedback. I haven’t been able to implement all of them yet, but hopefully a few more will be added over the next few days. I’d also like to thank the Guardian and Financial Times for mentioning the work we’ve done.

So, what lessons can be learned from this exercise?

Firstly and most importantly, plan! We threw the site together assuming the overall format of the data would be similar to the expenses claims made at Consumer Focus. In fact, it was much more complicated, and much more fragmented, with some (almost all!) crucial details missing. We also should have put much more effort into the data entry part of the site. While a basic system might be ok for internal use among a small number of people, it really isn’t appropriate for a crowd sourcing site- particularly when so many things can go wrong.

It is also worth considering that speed isn’t everything. We launched the site at 8.30am on Thursday. The Guardian waited until 3.30pm to launch their site and it was still successful. In hindsight we probably could have waited for a while and got things right first off.

Where next for WhatTheyClaimed.com?

I’m going to try and make some more changes over the next few days, in the hope of making things work a bit better. But in the long term, I don’t have the time to commit to the site. So if anyone is interested in taking over the project and giving it the attention it deserves (and needs!), then get in touch – whattheyclaimed@dafyddvaughan.co.uk. I’ve already had messages from a few people interested in working on the site.

Consumer Focus Labs will be publishing its expenses using the original system shortly – if you are interested in that then keep an eye on our website for the announcement. We’ll also be releasing the code for anyone to use to produce their own expenses websites. We hope that MPs will make good use of it to publish their expenses in the future.

In case you somehow missed the media frenzy, the final version of the Digital Britain report was published yesterday. After reading some of the media reports, I decided to read as much of the detail of the report as I could to see what was actually proposed.

Firstly I need to make a few disclaimers. Obviously the views expressed here are my own and not that of my employer – Consumer Focus (if you want the organisation’s view, you can see it here). I’ve also not read every paragraph in the 240 page report – some parts nearly sent me to sleep, so I skipped past them to some of the more interesting sections.

Overall, I can’t help but feel a little bit disappointed with the report. To me it doesn’t seem to go far enough - although a guaranteed 2Mbps broadband connection to every home in the UK makes a good headline, it’s poor compared to other countries such as South Korea (which is rolling out 1Gbps connections). It also skirts around some issues such as copyright fair use and leaves a lot to chance.

The report outlines 2 phases of broadband development in the UK. The first is the Universal Service Commitment (USC) which aims to provide a base to the whole country; the second is the Next Generation Broadband Network (NGB) which will provide for the future.

Universal Service Commitment (USC)

Digital Britain outlines its intention to provide every home in the UK with a minimum of 2 Mbps broadband by 2012. This will be part funded by the production of the NGB networks, with the rest funded through the underspend in the Digital Switchover budget. Initial thoughts are that this is great – everyone in the UK will have broadband of a ‘satisfactory’ quality. However, things soon start to fall apart.

Countries like South Korea are currently achieving 50 Mbps as a minimum, with some places getting as high as a 1 Gbps connection. When put in this context, 2 Mbps is exceptionally poor. Also, when you delve into a bit more detail, it turns out that this speed is actually optimistic.

Some homes will not be able to get broadband though traditional means, and so the Government is going to rely on mobile networks and satellite to provide the USC in these areas. The report suggests that 2 Mbps is unlikely for these houses, and connections of between 1 and 1.5 Mbps is more accurate. So actually, a 2 Mbps USC isn’t really correct.

Next Generation Broadband (NGB)

This is the next phase in broadband development and aims to provide broadband speeds of around 50 Mbps by using technologies such as Fibre to the Cabinet (FTTC) and Fibre to the Home (FTTH). Digital Britain expects that market investment will fund this deployment and anticipates that it will reach approximately two thirds of the population.

Assuming BT and the mobile networks do reach this target, that leaves about a third of the UK back on the not quite 2 Mbps USC broadband. This quite obviously is not acceptable, so a fund is being set up to provide NGB to this final third (called The Final Third Project).

The Final Third will be funded by a 50p per line per month levy on all fixed lines from next year. Contrary to some reports, this includes residential copper lines (like your BT phone line), business copper, ISDN and cable telephony lines (like those provided by Virgin Media). You will not have to pay this levy for mobile phones as they already pay license fees for network upgrades.

After all of this, it is expected that 90% of the population will be covered by NGB, leaving a poor 10% with USC speeds.

Illegal file sharing

This is of course a big bone of contention among many people. The interim report issued in January suggested the creation of a Digital Rights Agency which would be responsible for tracking illegal file sharers. This final version has moved away from this proposal and aims to stick with an industry lead effort, although supervised by Ofcom the industry regulator.

The report outlines a commitment to reduce illegal file sharing by 70-80%. This is a huge commitment, so they aren’t just leaving it to chance. If the industry effort of writing letters to account holders does not reduce numbers significantly within 12 months, Ofcom will be able to force ISPs to introduce ‘technical measures’. These might include blocking of sites, protocols, ports, bandwidth capping (speed and data transfer), bandwidth shaping and content identification and filtering.

However, for this to happen, Ofcom will need to ask Parliament to legislate.

Copyright fair use

Unfortunately this important issue gets all of a one paragraph mention in the document which makes no commitment to resolve the situation. As a result, it remains illegal to copy music you already own to your mp3 player.

Conclusion

As you can see, it is a bit disappointing. The USC isn’t quite the 2 Mbps they promised, NGB relies on the market to do the right thing, no clear direction on illegal file sharing and no movement at all on fair use.

I’ve not gone though everything in this post; there are a lot of other planned actions including the digital radio switchover, tax relief for video game makers, and the digital public service programme. Maybe next time!

Image courtesy of Media Guardian and Wordle.

I’m sorry to say that today I’m slightly embarrassed for the UK. Not only are we being badly affected by the recession, have a crumbling ruling party in Government and have a Parliament full of people milking the system for all its worth, but we’ve just sent two members of the racist British National Party to the European Parliament.

Counting started yesterday after Thursday’s vote, and stretched into the early hours of the morning. Although we still have incomplete results (as I write this we are still waiting for Northern Ireland and Scotland), we have a pretty good picture of what has happened. Unfortunately, two parts of the North – Yorkshire and Humberside, and the North West have voted for the BNP, who now have 2 seats and for the first time, a proper platform and funding structure to support them in the future.

It is a dark day for the UK.

I’ve been tweeting the result announcements throughout the count and it was quite comforting to see the response of some of the other people on Twitter. A lot of people are disappointed that this has happened. How did it happen though? Why did they get in?

I received this message from someone I know last night while the results were coming through:

“No one gives a f about politics! Stop boring everyone!”

It is thoughts like this that has allowed the BNP to get into such a prominent position. While I understand that not everyone is as interested in politics as me, some people obviously do not understand how important it is. Politicians are responsible for making decisions that affect everything we do. People should care about who gets in and who doesn’t.

Part of the reason the BNP got in is not necessarily because more people voted for them (in fact, their percentage share only increased by 1.4%, which in reality is not that much), but because people didn’t vote for any of the other parties. In fact, with a turnout of just 35%, a lot of people obviously just didn’t vote at all.

I have a pretty strict viewpoint on not voting – if you don’t vote, you have no right to complain about the state of the country. So to everyone who didn’t vote in this election, you are as much to blame as the people who actually voted for the BNP.

Of course blame must party lie with the politicians as well - if they hadn’t been milking the system, we might not have been in this situation.

For web developers and designers the question of who owns your portfolio is an important one.  It is a question that is not as clear cut as you might think.  It seems that depending on your circumstances, you may not have any legal right to display the work that you produce.

The general opinion is split into two areas – freelancers and contractors/employees.  If you are a freelancer, the answer seems pretty simple. Unless you have signed an agreement with a client that hands over all rights to them, you are the copyright owner, and so you can use it in your portfolio.  If you are a contractor or employee, things are much more complicated.  This isn’t helped by the UK’s outdated copyright laws, recently declared to be the worst in the world by Consumer Focus – the new public organisation responsible for campaigning for consumer rights.

If you are a contractor, you should check the terms and conditions of your contract. If there is a clause handing all copyright to your employer, then there is nothing you can do.  If on the other hand, there is no condition, you might be covered by fair use (this is a part of copyright law that allows certain actions that would normally be regarded as infringement).

For employees, the picture is even bleaker. Anything you create during work hours belongs to your employer.  This even covers things like photographs taken during work time using your own camera!  In these situations, fair use might still come into play.

In the past, when the printed portfolio was king, this issue wasn’t that important. After all, who would know that you are using work you produced while in employment in your portfolio?  Now that it is essential for a developer / designer to have a digital portfolio open to everyone, this is issue has come up time and time again.

I would argue that your portfolio is covered by copyright fair use as you are not selling on the work, and not causing the copyright owner financial harm.  Additionally, you may be providing your employer with free publicity.  If you do go down this route, it is important to credit your employer.  Also remember that claiming full credit for something you didn’t do or only had a part hand in comes under a different issue – plagiarism (you can see my thoughts on this here).  You could also argue that as a website is in the public domain, there is nothing wrong with a screenshot of it – sites display screenshots of other sites all the time - it is accepted practice.

Recently I’ve been talking to lots of designers about this topic. It seems that lots of web companies don’t mind their employees showing work on their portfolio. Some even seem to go as far as to encourage their employees to do it because of the extra publicity it provides.  There are, however, a minority that see things differently.

Of course, I’m not a lawyer, so you shouldn’t take anything I’ve said as legal advice. However there are plenty of places on the internet that do offer this advice and these are well worth a read (1, 2, 3, 4).

If you are an employee and leave to take on another job or become a freelancer, you need to seriously consider the legality of your portfolio. Also remember that what may have been OK while you worked for the company, may not be once you leave.

What do you think? Should a designer be allowed to display a copy of the work he/she has done in their digital portfolio? Have you had problems with your online portfolio?  Let me know – I’d be interested to gather a better picture about this issue.

What’s been going on recently? – well quite a bit really. I’m now settled into my new job at Consumer Focus Labs. We’ve spent a few weeks getting our systems set up (not as easy a task as we’d have liked), our development environment in place and courting ideas from anyone and everyone. We’re now pretty much ready to go and should begin to work on our first project next week.

Aside from that, I managed to push out a new version of CllrTweeps. The new version runs from a proper database, enabling us to categorize councillors by party as well as councillors. It also gives us quite a good base from which to start adding new features, so keep an eye out for more announcements over the next month or two.

As part of the development of CllrTweeps 2, I’ve spent quite a lot of time looking at the websites of local authorities. The difference between some of them is staggering. Some councils have obviously put quite a bit of thought into the designs of their website and the services they offer online. This has made me think about doing a series of blog entries looking at council websites - possibly something similar to my local elections series last year.

For now though, I’ll leave you with an example of a great set of online services from Lichfield District Council. Their web team deserve a pat on the back for the good work they’ve been doing. Also, don’t forget to submit any ideas you have for APIs or online apps you’d like to have for business data at the CF Labs website.

I’ve been a part of the team at VSI for the best part of nine years, originally as a summer job, but more recently as a full time developer. During this time I’ve seen significant changes in the way the internet works, from new technologies and new standards in design, to new ways of communicating and ways of thinking. The internet landscape is completely different now compared to when I first started out in the industry. For the first time, we are starting to see businesses and government use the internet in completely new and innovative ways. Groups are being set up to take the vast quantity of data available and turn it into something useful. From services such as FixMyStreet.com by MySociety to the more recent School Closures website from Directgov, the internet is starting a new phase of open development.
For me personally, this means new challenges and horizons. Just over a week ago, I said goodbye to VSI-thinking for the last time.

Change is good

I’m happy to announce that I am now a part of one of these groups helping to set the vast quantities of data free – Consumer Focus Labs. Consumer Focus is a new organisation set up by the government to campaign for a fair deal for consumers. The organisation has established a small team of developers fronted by Richard Pope (one of the people behind My Society and Rewired State). We will be building online tools in order to pursue justice for consumers in new and exciting ways.

We’re currently collecting ideas for the first batch of tools we will build. Work should start on these in the next few weeks once we are all settled in. If you have any ideas or questions, take a look at our website and get in touch.

As always, as our work progresses I’ll be posting updates here on my blog, as well as on the Consumer Focus Labs website.

Earlier today at the Future of Web Apps conference in Dublin, Robin Christopherson from UK charity AbilityNet gave a talk on website accessibility.  I’ve seen quite a few different talks and sessions on website accessibility and understand how important this topic is.  This talk however really hit home - Robin is blind and gave the whole presentation & demonstration while using a screen reader.

I understand how screen readers work and have tried out some myself, but seeing someone use one in a real situation was really humbling.  I can’t even begin to imagine how difficult it is to use a large part of the internet for someone who suffers from a disability that affects their ability to use a computer.

Robin showed off number of web apps that worked well with screen readers such as FixMyStreet.com, a service that allows you to report problems to your local council.  Another site that worked well was Google Maps (in particular the ability to get directions to a specific location).

What really surprised me however was how poor the accessibility was on new Downing Street website.  When the size of the text on the site is scaled up, the design just breaks completely.  For a public service website, this is quite disturbing, and really needs to be addressed ASAP by their web team.

I wonder how many other Government / public service sites do not consider these issues?

A quick update this week!

Last weekend I was pleased to launch CllrTweeps.  The new site, inspired by TweetMinster, aims to list every local councillor in the UK who is on Twitter.  After our first week online, we have already tracked down and verified the first 100 councillors using the service.

The first version of the site is run from a very basic Wordpress template.  As great as Wordpress is, it isn’t really cut out for this kind of site.  Luckily, we’re already working hard on version 2, which will bring a more organised directory, cleaner look, and provide us with a good basis to add in more feature such as post code search and mapping. You can keep track of the developments on the CllrTweeps blog or here in my new CllrTweeps category.

A big thanks must go out to James Cousins (twitter.com/jamescousins) who has put in pretty much all the leg work for the launch of the site.  Thanks also goes out to Liz Azyan for helping to identify the councillors and Hywel Vaughan who is working with us to create the branding for version 2.

While we’re busy working on the next version, if you know of a councillor on Twitter please point them in our direction!

Unless you’ve been in a cave (or outside of the UK) for the last few days, you’ll know that we’ve been experiencing the fun of a ‘severe snow event’. The exact definition of a severe snow event remains unanswered, and I’ll leave the debate over whether what we’ve been experiencing is severe to another time. What is important, however, is that a large number of schools have been closed because of the snow. The BBC reported earlier that over 500 schools were closed today in Wales alone.

This brings up an interesting questions – how do schools inform parents and students that they are going to be closed? Traditionally, schools have informed local radio stations which have spread the news. This however comes with a number of problems – most importantly, not everyone can receive the local radio station. For example, one member of my family goes to Hereford Sixth Form College, which informs BBC Hereford & Worcester, and Wyvern FM of closures. However, here in Abergavenny, we receive neither of these stations.

The college could phone all parents – but with over 1700 students, that would be nearly impossible. Parents ringing the college would freeze up the phone lines. Maybe the schools could use the internet to pass this message? Hereford Sixth Form College does just that – an emergency message has been displayed on the homepage since Monday morning detailing the current status of the college.

What if there was one central resource for information on all schools? One website address for parents to remember - a website which was updated by each school to show their status. Tom Watson, MP for West Bromwich suggested this yesterday morning. He purchased schoolclosures.org.uk and challenged DirectGov (the UK’s government gateway) to produce the site ready for the next day (today).

DirectGov must have pulled out all the stops, because just 28 hours later, schoolclosures.org.uk was released as a beta to show how the system might work. This is fantastic work from a government department - not something you will usually hear from me. Being able to take an idea to a working system in just under 28 hours is no mean feat – but is something the government needs to get used to. Well done to everyone who worked on this!

Of course, no one system is going to solve the problem. But with all systems, schools, and government departments working as one, there is a real chance to make an event like this easier on parents and pupils.

Well, I never thought i’d see the day, but Google search has completely fallen over. No matter what you search for, Google returns a message saying that “This site may harm your computer”. Google will not let you visit the page unless you manually type the web address into your browser. It is even labelling its own site as potentially harmful.

Twitter is going crazy with (i reckon) an average of 700 tweets a minute regarding the problem which appears to be affecting the whole world.

More coverage at TechCrunch and Twitter Search.

Update: All seems to be fine now - problem lasted around 20 minutes