Open Data in Calderdale: Interview with Open Innovations

Interview with Simon Greaves, Lead Business Intelligence Officer at Calderdale Council and Paul Connell, Founder of Open Innovations and Grace Durkan, Events and Communications Co-ordinator at Open Innovations. Calderdale is one of Open Innovations founding sponsors and this interview talks about the benefits of this collaboration and showcases our innovative approach to Open Data.

Calderdale Council and Open Innovations logos

Paul: Why are you part of the sponsor group at Open Innovations?

Simon: We have been a sponsor for 5 or 6 years now. Open Innovations helped us to get up and running with open data. To begin with, we looked at the good work you had done with Leeds City Council through Data Mill North (using the same DataPress platform) and tried to learn from this and make it work for us. Getting ideas and support from Open Innovations along the way has really helped.

Paul: Did being part of the sponsoring group help?

Simon: Yes. We took a couple of years to work out which direction we wanted to go in but basically just started publishing data and building from there. We wanted to make publishing open data part of our daily work as a business intelligence team, so it became business as usual rather than a separate job. This has worked well and allowed us to make a lot of progress in both the number and quality of datasets published, and also really important that processes were in place to ensure they were well maintained. Then we tried to align with some of the projects you ran on things like planet data and air quality and replicate what had worked well on Data Mill North.

Paul: And joining in and making it easy to join in, that’s what we try to do. Does that work?

Simon: Yes, putting on events on the right subjects at the right time means we can easily get involved, whether that’s in person or picking up the resources after the event. We can use that to try and publish data in the right areas.

Paul: What we talk about a lot is timing, giving the sponsors resources and work that is available when internally or externally, people start to ask if you have anything on a particular subject. Having something ready, rather than going looking when people start asking is another big thing, we try to support our sponsors with.

Simon: That really helps on things around climate emergency for example. You’ve run a few projects and events on this which link very well with the data we’ve published and developments like our air quality dashboard. We have a very proactive climate emergency team in Calderdale who are getting involved too and I think attended a workshop that Giles Dring (Head of Delivery, Open Innovations) was running a few weeks ago so it all links together quite nicely. This doesn’t happen on everything but there are those nice examples that make it worthwhile.

Paul: And then we also have the Open Data Collaboration Group meetups hosted by Open Innovations, which you helped to initiate with Leeds City Council on questions and problems that you were interested in.

Simon: It’s a regular meet up with sponsors and local organisations who want to talk about open data, share good practices, ask for help with problems, see where we can work together to make use of high-value datasets and learn from each other. It’s all very informal so you can join in as much as you want or just listen and learn! And that’s the thing with a lot of the sponsorship or approach to open data in general, there are no hard and fast rules about what you should be doing. It can be through high profile innovation events and projects like Leeds or the more operational and practical approach we’ve taken to just publishing data.

Paul: And how does that work? Because we help to get the data published, then we build some tools and then from Dataworks’ perspective, how does that work internally at Calderdale?

Simon: We’d look at the datasets that were being discussed at events like Levelling Up or Flood Hack and then we would try and get them published in advance. We’d speak to services and show some of the outputs that can be produced if the relevant data is published. For example, we’d say look at what Yorkshire Water and Open Innovations are developing on the Diversity Dashboard, if you publish the data in this format then your data can be included, and you will have a useful comparison tool to use. We’d help get the data in the right format and publish it for them, but it wouldn’t involve a lot of work for anyone.

Paul: It starts off with input and then it is conversations, we haven’t managed the project, if you like, we’ve all been working collectively. But we haven’t had any conference calls or project management.

Simon: The diversity data standard and prototype dashboard was discussed a few times at the collaboration group so it was useful for all involved to shape the project and be consulted along the way. It’s a great example of collaboration with Open Data.

Paul: I think it’s great. And it’s one that we want to do more of where we agree on a standard.

Simon: The Spending Explorer is another one from a few years ago. We had to publish it anyway as a part of transparency code but Stuart Lowe (Data Projects at Open Innovations) built a tool that pulled these datasets together (think it took a bit of work to standardise!) and visualise it so it was much easier for people to interrogate and make sense of. I think there are similar tools for the business rates and brownfield sites datasets.

Paul: Definitely. We do some experiments, but not all of them work and that’s fine. But then that allows you to pick up the ones that are more useful. And then that allows you to focus on the ones that are useful to you. And then when you say ‘Yeah, we were using that, we can spend a bit more resource on doing it’. That’s basically what the sponsorship is for – it pays for us to exist and work on stuff that is relevant to you.

Simon: We also make good use of some of the other tools Stuart developed such as the CSV checker and CSV to GeoJSON tool (check out the Open Innovations toolbox). These tools help us publish files in different formats without needing specialist software. The Data Mapper is great too, we publish data in the right format, it automatically feeds the Data Mapper and provides a good, quick visualisation, which we then link to on Data Works. It doesn’t take a lot of work but then there is real value added to the dataset.

Paul: So you publish your data, we give you a tool to publish it in a usable format, like GeoJSON. That then means that we can easily create a map of that data, which you can add to your published Data Works. And then people see the value of publishing the data.

Simon: Yes, it’s as easy as that. I think it’s also mutually beneficial for us and Stuart too – we make use of these small tools and then we test them for him with real datasets and spot things that don’t quite work because we’re using it a lot and we can make suggestions for improvements. I think he appreciates that; it really is collaboration and innovation in action.

Paul: I think that’s great. It suits us because when we’re working on projects that might help, we just share that straight across because we’re doing it in the open. The big thing for me is that we’ve managed to fund a team to do work rather than projects. Because if we have, to say ‘Oh, we need to fund a project, which is the Data Mapper’ we would have had to try and cost it, whereas what we’re saying is, ‘If we build this, and you can use it, that then generates value for you in a different place, which makes it easy for you to join in and sponsor us.’

Simon: That’s exactly it, if you said ‘we want to build this Data Mapper but it is going to cost’ we would have thought it was a nice idea but would have found it difficult to fund, but if this is a by-product of the sponsorship, something like this every year, as well as access to the network and events and projects then it’s easier to see the value.

Paul: Then because we’re working with you, which means we have got a direct line into people that would help us out on a project that we’re working on – there’s goodwill both ways. We get value out of the relationship as well. I think that the whole idea of trying to fund a team to work on projects that are collectively beneficial is where we’ve managed to get through with you.

Paul: If you were going to talk about a dataset that you are publishing, run us through the process of the thinking at Calderdale, how have you got it set up?

Simon: We work with our Information Governance (IG) team to identify frequent FOI’s (Freedom of Information) requests and then we work with the service to get the dataset behind it published on Calderdale Data Works. Next time we get that FOI, the service or IG team can just signpost to Data Works and it saves everyone time, including the requestor who eventually stop asking and self-serves the data. We may also see that Leeds has published a dataset that is interesting or potentially high value, like footfall data for example. And now we’ve got some good weekly footfall data from the Woolshops shopping centre in Halifax which is fascinating to look at the changes over the last two years and the impact of the pandemic.

Paul: That timing thing is really important.

Simon: You have to keep selling the advantages of open data. Sometimes you get a good response, sometimes it’ll take a while if there is a change in personnel or system. But we keep giving the same message and more people are seeing the benefit and more services are proactively coming to us and asking to publish data which is great. We’re focused more on quality rather than quantity in terms of datasets now – at first we wanted to just get things published but now we’re almost at 300 we can be more selective about what may add value.

Paul: They get the value of the data then. It-s beneficial to them, isn’t it? People have worked out they’re not giving data to someone else to use. The main user of the data is themselves.

Simon: Yes, there is a lot of internal use of Calderdale Data Works.

Paul: You should get some data on that – we sort of made up this number which is 80% of the value of data publishing is to the organisation itself. We just guessed 80 because it is definitely much more than 50% and probably a bit less than 90. Most of the value of publishing is to the organisation itself.

Simon: We’ve also created several dashboards, many of which are available publicly on Data Works. They bring a lot of high-value datasets to life such as anti-poverty, air quality, performance, budgets, deprivation, libraries, highways and traffic and with a few more in the pipeline. We’ve been using Qlik for a few years and we’re now migrating over to Power BI.

Paul: So you’re here at Open Innovations today working on that with the team moving into Power BI. And how easy is it?

Simon: Everyone is really enjoying learning and using Power BI and some of the other tools within Microsoft 365. It feels like we’re making good progress, migrating existing dashboards but also improving them at the same time with the benefit of a few years doing this. We get a lot of good feedback about public dashboards and Data Works in general, so it is nice to know we’re on the right track and that others are using our good practice to follow in this area.

Paul: And if anyone wants any advice about how to publish data as a council, you’d be willing to help them out?

Simon: Yes of course but the best thing to do is just start publishing some data and build from there. And come along to the Open Data Collaboration Group to find out more, ask questions and see what other people are doing.

Paul: And that’s our call to action – If you want to come and help, just email us, get involved and join the collaboration group.

Published originally on Open Innovations: Wednesday 20 April 2022