Numbers in the news

I’ve left it a little late to blog today. Still blog I must if I am to avoid a terrible forfeit from Dan Slee. So here are some numbers that caught my eye in the news.

10

We face 10 years of stagnant wage growth and living standards according to The IFS analysis of the OBR forecasts.

30

The conviction and sentencing of a far right extremist for the politically motivated murder of an MP appeared on page 30 of the Daily Mail apparently. Which makes you wonder about their news values. And what they think of the values of their readers.

59bn

The chancellor can expect to have to borrow £122bn more than previously thought according to the independent OBR. £59bn of that predicted extra borrowing is directly related to Brexit. But it’ll probably be OK. Because the Italians really want us to buy their sparkling wine.

1 9 8

Article 1, Section 9, Clause 8 of the Constitution of the United States of America reads:

No Title of Nobility shall be granted by the United States: And no Person holding any Office of Profit or Trust under them, shall, without the Consent of the Congress, accept of any present, Emolument, Office, or Title, of any kind whatever, from any King, Prince, or foreign State.

A clause Donald Trump is about to become very familiar with.

Now we are all currency trading experts

"Facsimile de un billete de banco de cinco libras esterlinas" by Fondo Antiguo de la Biblioteca de la Universidad de Sevilla https://flic.kr/p/7fhv5k
“Facsimile de un billete de banco de cinco libras esterlinas” by Fondo Antiguo de la Biblioteca de la Universidad de Sevilla https://flic.kr/p/7fhv5k

I was minding my own business on Facebook the other day when one of my friends made a crack about her trip to Spain being more expensive because of Brexit. One of her friends, with the anger of the frustrated Brexiter, explained that the currency was only returning to its long term average.

That doesn’t seem right.

I thought. But maybe it is. So I resolved to see whether it really was.

The exchange rate

I kind of know what the exchange rate is. It’s the amount of another currency I can buy for a pound (for I am British). People tell me the pound has fallen which means, effectively, I can buy less of another currency for my pound.

There is a market in currency (the Foreign Exchange or FOREX market) and when people on the news (or on Facebook) talk about the pound weakening they are talking about the what a pound can buy in the FOREX market.

The value of the pound

There are, somewhat inconveniently, many currencies and the amount of each currency that the pound will buy varies. So often we talk about “the pound falling against the Euro” (£1 will buy fewer Euros) or “strengthening against the Dollar” (£1 will buy more dollars.

Thankfully for those of us who do not wish to become foreign exchange traders, the Bank of England provides a handy summary of the value of the pound against all currencies. (Currencies from countries we do more trading with count more in this index).

This is available in a handy file going back to January 1980. In this dataset the exchange rate in January 2005 was set as 100 so if the exchange rate is higher it will be bigger than 100, if lower it will be lower.

It looks like this

Which certainly does make it look like the pound is at a very low level. In fact it seems to be at a similar level to 1993 which was when I graduated and I rather remember being a pretty tough time economically (not for me the impact of the demographic time bomb). More recently the pound was at a similar level in 2008. That was definitely a bad year.

So if the pound is returning to a historic norm, it is a norm that coincided with struggling economies.

Does a weak pound matter?

Broadly speaking a weak pound is good for people (or businesses) selling things from the UK to people (or businesses abroad). So it makes the UK a cheaper tourist destination, helps exports and encourages foreigners to invest in the country. It increases the cost of importing things, makes a UK holiday abroad more expensive and discourages UK companies investing abroad. There’s a nice guide on economicshelp.org.

Overall (that guide suggests) prices will increase with a lower pound.

That’s one consequence of the Brexit vote: we are all experts on currency trading now.

 

 

Adventures in (less serious) data land

I’ve been wrangling data for a project.

And creating interactive data visualisations using tableau. The results are OK but, you know, a bit serious.

But it doesn’t have to be that way.

(These are all interactive but this page is a bit thin for the true majesty to work, so I’m afraid you’ll have to click through to play with the data. Do come back though.)

1. Magic

Skyler Johnson shows us every spell muttered, flicked, and yelled in the Harry Potter book universe, visualised at occurrence position.
Skyler Johnson shows us every spell muttered, flicked, and yelled in the Harry Potter book universe, visualised at occurrence position.

2. Lyrics

Chris Love all of the song lyrics to the Top 100 songs
Chris Love all of the song lyrics to the Top 100 songs

3. Culture

David Newman visualised the Simpsons. Nuffsed.
David Newman visualised the Simpsons. Nuffsed.

 

4. Music

Will Jones and Eric Shiarla look at  history of music #1s in the United Kingdom
Will Jones and Eric Shiarla look at history of music #1s in the United Kingdom

5. Movies

 Ryan Sleeper asks do movies, like fine wines, get better with age?
Ryan Sleeper asks do movies, like fine wines, get better with age?

6. Chess

Joe Mako recreates one of the earliest recorded chess matches in Rome
Joe Mako recreates one of the earliest recorded chess matches in Rome

7. More music

Dan Lane visualised all the songs which have reached million-seller status in the UK
Dan Lane visualised all the songs which have reached million-seller status in the UK

Handle with care

handle with care by Hash Milhan https://flic.kr/p/DdPcU
handle with care by Hash Milhan https://flic.kr/p/DdPcU

It seems like everybody is trying to work out who voted Trump. In the same way that loads of people where trying to work out who voted in favour of Brexit. There is a sense in which maybe if we can work out who voted that way we can work out why they voted that way.

The Economist has done a nice piece of analysis looking at how (poor) health is a good predictor of a swing to Trump in areas that were already republican. Interestingly (to me) this is based on not a single measure but a whole basket of public health measures and is a (slightly). This is actually a slightly better predictor of the swing to Trump in republican areas than looking at the numbers of non-college educated white people (though those two factors are often related).

This still doesn’t really answer the question of why they voted this way. A Trump administration is not likely to improve public health one iota. The Economist suggests that unhappy, unhealthy people were voting for change. Maybe that is true, I have no reason to suppose it isn’t.

But democracy, certainly the US presidential election and democracy in the UK is a practice of choosing from a short list. We do not know, on the basis of the vote, why people made those selections. We have to infer and theorise. And most of those theories are politically loaded. Did people really vote to leave the EU because they wanted to end immigration or was it to free themselves from the EU legal system. Did they, in fact, not vote to be poorer?

No-one knows.

And those who say they do have an angle.

Facts are hard, to get and to confront. So much easier to guess, or speculate or state with confidence.

Density

For much of my working life I have been based in extremely sparsely parts of the UK. No, not the Highlands, but as close as you can get in England and Wales. Shropshire, Powys and now Herefordshire. Very sparse areas require very different policy approaches to, well, most other parts of the country. It’s hard to get across quite how different these areas are and, I feel, when it has been my task to I have largely failed.

So I’ve tried playing about with some dataviz. Let’s see how it works. The average population density across England and Wales is a rather low 383 people per km2. To demonstrate this here are 383 people in a square.

density

That’s nice but no-one really lives in an average location.

In Birmingham people live at an average density of 4,100 people per km2. At the same scale as above that looks something like.

birmingham

And finally to Herefordshire. In Herefordshire we have an average population density of 86 people per km2. Which, in our virtual square kilometre, looks like this.

herefordshire

Which I think makes it look quite different.

But that’s just me.

How much data do you need?

I’ve been playing about with data pertaining to Herefordshire as part of a project for The Bulmer Foundation.

There is a lot of data out there. At a rough guess hundreds of datasets of public data that pertain to Herefordshire. And I’m just imagining data that you could put into a nice simple table. When you factor in mapping data and unstructured data the levels become very large indeed. And Herefordshire isn’t even a place you would typically describe as being well served in terms of data.

But this dataset brought me up short.

It’s a simple time series (well simple-ish) showing one number per year. That number is a measure of the affordability of housing in the county.

It’s a figure widely used which looks at the ratio of wages to house prices.

To measure wages. Now lots of people in Herefordshire each money and they earn wildly different wages. You need to simplify that spread of wages. You might normally pick the average wage, which is a widely understood way to summarise data. In this case they pick the lower quartile wage which is less widely understood but actually pretty straightforward. Average is the wage where 1/2 the wages earned are above that level and 1/2 are below. Lower quartile is the wage where 1/4 of the wages in the county are below that level and 3/4 are above that level. It’s a good measure of the sort of wages people on lower incomes earn.

To measure house prices. The approach to house prices is similar. There are lots of houses bought and sold and houses come in all sorts of different sizes. We look at the lower quartile price. The price where 1/4 of houses sold cost below that price and 3/4 cost more than that price.

So this is a measure of the ratio between house prices at the lower end to wages at the lower end. If house prices go up, the ratio goes up. If house prices go down, the ratio goes down. If wages go up the ratio goes down. If wages go down, the ratio goes up.

Or to put it another way smaller is better (assuming you think people should be able to afford houses).

And the affordability ration is not getting smaller.

There is a lot more data available about housing.

But really, how much data do you need?

The source data and many other interesting facts can be found on the LGA Inform website.

Open source intelligence and ethics

Spy by Leonardo Veras
Spy by Leonardo Veras used under CC-BY 2.0 https://flic.kr/p/oEcacy

Evanna Hu has written a guest post on the Responsible Data Forum about Responsible Data Concerns with Open Source Intelligence. I basically agree with everything in that post, which you might think makes this post a bit superfluous but I’ve got to write something every day in November. Cos Dan Slee challenged me to.

Open Source Intelligence is intelligence based on publicly available information. Unless you work with open source intelligence (as I do for humanitarian purposes) I think it is hard to get your head around the detail and sophistication of data that it is often possible to derive from public sources.

I’ve written before of the privacy concerns that I think need to be more properly addressed by public bodies. What Evanna Hu’s post highlights is how much further this debate should go in organisations.

There is the legal framework (which is really what I was talking about in that post). I was really struck by Evanna’s statement

As with many responsible data concerns, legal compliance is just one part of a much bigger picture, and it often forms the lowest bar rather than the best practice we should strive for.

And the legal concerns I raised are really fairly specific and limited. An ethical framework would go much further. I’d be really interested to hear from any public bodies that have done work on this.

As a society, we need to go much further. We don’t really have cultural norms for the use of publicly available data by anyone. In many circumstances it may not be state actors that we should be most concerned about. We need to encourage, as well as legal frameworks, a global set of standards that we can hold all organisations to.

But maybe we can start in the UK. Is there any organisation interested in leading?

An open source Brexit?

Open
Open Sign by Andy Wright used under CC BY 2.0 https://flic.kr/p/okaQ

David Allen Green has always been worth following on Twitter (if that’s your sort of thing). He’s been particularly interesting on the Brexit mess/process/glory (delete according to your inclination). He wrote a lengthy series of Tweets today, essentially critiquing the government’s failure to introduce a Bill to give it the power to trigger Article 50 (don’t argue with me, read what he said). One of these Tweets jumped out at me.

Now that is interesting. What would an open and collaborative Brexit process look like?

Well.

The UK Government already has an Open Policy Making toolkit. Brexit clearly is going to require new trade, immigration, foreign, data protection and environmental policy (that’s just off the top of my head, smarter people can probably think of many more). So at the very least the development of these new policies could be undertaken in an open and collaborative way.

The government is already committed to publishing Open Data. So the data (read evidence) upon which policy ideas are based should already be public and available to use. It would be really helpful if this could be brought together in one place. So that we can all see what the evidence is (and then argue about what it means).

Open modelling. The government is going to have to make estimates about the impact of different aspects of policy and legislative changes on the UK as a whole, on particular parts of the UK (in terms of geography and in terms of sectors of the economy) and on the rest of the EU (and probably on individual member states like the Irish Republic). Those assessments could usefully be made public (along with the models that underpin them). This would help everyone understand the trade-offs that are being made and (perhaps of more practical use) allow academics, companies and other stakeholder groups the opportunity to suggest improvements. They’d also help businesses (and communities) make sensible contingency plans.

None of this is hard or expensive to do. In fact there are loads of really useful tools to facilitate this. Why not maintain Brexit on Git?

The Government is arguing that it can’t reveal its hand in the negotiations. The idea of Brexit as poker is seductive but superficial. This is a negotiation between nation states not a high stakes card game. In fact useful negotiations can be undertaken in open environments. In fact having a shared picture of the facts and the implications of decisions would be likely to lead to better, more solid and long lasting agreements.

In fact we could use this process to as the start of a new, open, relationship with the EU and the rest of the world.

And it would go a long way to improve the level of trust between communities and people within the United Kingdom.

 [updated 21:44 on 7 November to remove a stray apostrophe and add a link]

Models

Brompheniramine Model by Iain George under CC BY 2.0 https://flic.kr/p/5Abyh9
Brompheniramine Model by Iain George under CC BY 2.0 https://flic.kr/p/5Abyh9

I am (when I have access to the Internet) somewhat glued to the FiveThirtyEight coverage of the US presidential election (their elections podcast is also awesome). At the heart of their coverage is “The Model”. They take data from published opinion polls, feed this into their model and then publish the likelihood of Trump or Clinton winning the race. The model allows them to understand and, to a certain extent smooth, the inherent biases and uncertainties in different polls with different techniques. They also factor in none polling factors (like the state of the economy).

So what they have is a simplified version of the United States. It’s based on some theories about how polling data relates to actual turnout validated against data from previous electoral races.
We are poised to find out how good a model it is.
Models are useful, possibly vital, in terms of understanding the world. We all have models in our head. In order to be able to predict with any likelihood the impact our actions will have on others, we need a model of how they are likely to behave. Luckily our brains handle all this for us in the background so we don’t have to be conscious about it.
If you ever imagine that an action you undertake will have a particular consequence you have developed a model.
I’m pretty confident that when I write a news release it will be picked up and used by a journalist. This is, of course, because of my great skill and experience. It’s also because I have a model of how newsrooms work, of the values of my target journalist, of the sort of writing style that will appeal to them.
This model was originally handed to me in my professional training and I have improved and refined it based on the experience of 20 years of communications roles. It’s a pretty good model. It’s not foolproof but most of the time its good enough.
We tend not to be very explicit about our models and that can be a problem. If we don’t tell others the basis of our models they find it hard to assess how confident they can be of our predictions. It’s also harder for others to understand how new data mesh with our existing model (and reinforce it or suggest it should be changed). And they can’t challenge the fundamental assumptions underpinning our model. Which is something we should welcome.
This may not matter that much in terms of getting news articles in local papers. But it really matters when we talk about the impact that changes to the funding of social care (for example), or housing allocation policy, or educational selection at age 11.
We have subconscious models for all these aspects of social policy. They are probably poor fits for reality. If we really want to improve services for the public we need to state our models, show how they fit the evidence and be ready to change them in the light of new data.

Cool mapping things for real people

I was at Housing Camp Cymru last weekend. It was great.

I pitched a bring and share session on mapping apps. Because housing is so much about physical place putting stuff on maps feels like an obvious thing to do. I often find that, unless they really love maps, many people don’t realise how easy it has become to do cool mapping things without spending money or getting a degree in geography.

(This is not to denigrate GIS specialists because there is even more really cool stuff if you actually know what you are doing).

So here’s a quick run down of some of the stuff we covered.

Open, shared and closed data

Perhaps inevitably  (given that I’m part of the ODI-Cardiff team) we talked about the question of being able to use data.

Closed data is data you can’t use, shared data is data that you may be able to use but there will be restrictions on what you can do with it, open data is data anyone can access, use and share.

This video explains this rather better.

Google Maps

Chances are you have used Google Maps to do something like work out how to get to a new location. It’s got very powerful mapping and routing tools. It’s also really (really) easy to create simple custom maps.

(Not the most dramatic map I’ll confess).

To get started with this you just need to create a map on Google Maps.

Fusion Tables

To put more complex and rich data on Google Maps there is a nice tool in the Google Drive suite called Fusion Tables.

It does two jobs really well:

  • it “fuses” tables (links two spreadsheets together based on a column of data that they share)
  • it takes columns of geographical information and puts them on a Google map

It’s smart enough to process some sorts of data so if you have postcodes in a column Fusion Tables will turn those into points on a map for you. It’s also capable of handling polygons (shapes: that you might use to show borders or boundaries).

I used Google Fusion Tables to create this clickable map of changes in crime rate in different parts of Herefordshire.

More information in the excitingly titled Investigating crime rates at small geographies.

OpenStreetMap

While the Google tools are very useful there are limits to what you can do with Google Maps. Not least because the Google Mapping data belongs to them and you can access the underlying data nor can you use Google Maps for any purpose.

There is a source of mapping data that you can do all this with: OpenStreetMap.

OpenStreetMap started as an open data alternative to Ordnance Survey and is one of the most incredible datasets in existence. It is “Wikipedia for maps” anyone can edit it and add data to it.

There is so much to say about OpenStreetMap that it would require 100 more blogposts. To get some sense of the possibilities have a look at Mappa Mercia and the Missing Maps project.

There are so many projects built around the OpenStreetMap community. Check out Mapillary and Walking Papers to get a sense.

With OpenStreetMap data you can do anything with your map, create web tools, or print out a massive noticeboard without paying anyone or asking permission.

Ordnance Survey Open Data

The UK’s mapping agency is releasing more and more of its data under and open licence. You can download all sorts of different files from OS Open Data.

 

Carto

A nice alternative to Google Fusion Tables which allows you to use a range of different mapping data sources is Carto (previously CartoDB). Carto is actually more powerful than Fusion Tables and it’s on my list of things to learn more about.

It’s a freemium product but the free stuff is very good.

Ushahidi

Developed in Nairobi to map post-election conflict Ushahidi has found uses in disaster response, monitoring buildings at risk and hundreds of other situations where citizens want to report and monitor things happening in their locality. Ushahidi 3 is just out

You can experiment freely at crowdmap.com.

QGIS

Want to try full-fat Geographical processing? You can do that for free too with the open source QGIS. This is a professional scale Geographical Information System. Don’t expect to become a satellite processing expert in a few minutes. But QGIS is powerful, free and has a healthy community of advice and plugins. So if you like maps you’ll love QGIS.

Other random things we mentioned

Crime data can be downloaded from police.uk.

LIDAR (a laser version of RADAR) data is avaiable for much of England and Wales. This is geeky but clever people have started doing fun things with it like writing a script to turn your estate into a minecraft world.