Notes — On Impact Data
Getting better at impact measurement and studying data ethics.
I’ve completely lost track of time and refuse to believe we’re in November. On reflection, though, I like to think that the year went in a blink of an eye because it was fun (I even went to Ukraine and have a photo to prove it). Today seems to be the first quiet day in some time and an opportunity to publish Note 2, which mainly covers musings on impact data and ethics.
The Value of Impact Data
Some of the challenges around getting high-quality data
In my conversations with other founders and investors, the words ‘data’ and ‘reporting’ always prompt a mix of reactions. You get everything from quiet horror and dread to existential deliberations on the Platonic ideal of a reporting form. For context, investors typically request quarterly data from their portfolio companies, which is mostly a mix of commercial and [in the world of impact investment] impact metrics.
Commercial and financial metrics tend to be easier to collect as there is a broad consensus on terminology and use cases. There might be some discussion on what figures to include but for the most part, these questions can be resolved with a quick email. Impact metrics and the process of impact measurement in general, however, tends to be a more gradual process for founders and investors alike. It usually takes time to improve because:
a) The value of collecting and improving this data is not obvious.
b) There’s often a disconnect between theory and practice. People have heard of the impact frameworks but are not sure how to apply them to their own businesses.
c) It can feel hard! I keep hearing everyone talk about how challenging it is to measure impact and while that’s true if you are measuring the impact of a ten thousand person organisation, it can be fairly straightforward if you’re an early-stage startup.
For various reasons, knowing what to measure, when and finding ways to make good use of the data is not always easy. The lack of practical resources is a shame because, when used well, impact data can be a valuable asset with many commercial use cases. In fact, founders (who I’ve spoken to) who have embraced rigorous impact measurement practices see them as a core part of their operations, helping them validate assumptions, develop products, sell their proposition, and stand out against competitors.
With this in mind, I’m working on guidance that I hope will show the value of collecting high-quality impact data and ways to make better use of it. There will be frameworks, case studies, resources and boundless optimism about the future of impact measurement.
Questions I’m asking — Have any impact investors publicly laid out their assessment criteria for investing? Are there any good ‘best practice’ guides on data collection, management and communication that I should know?
Learning data ethics
On a slightly different note, just as I began to dive into the world of impact data, I have also started on the Open Data Institute’s Data Ethics Professional Certification. It’s been great. There is an equal emphasis on ‘data’ and ‘ethics’, and the educators do a great job stretching one’s understanding of how both interact.
Try this Moral Machine exercise, for example. The site calls itself:
…a platform for gathering a human perspective on moral decisions made by machine intelligence, such as self-driving cars.
The site shows your ethical dilemmas, and you have to pick what you deem to be a more acceptable outcome. You can then see how your answers compare to others who have done this test.
Forty million submitted decisions later, and we can see how moral values range not only by person but also by country. For example, when choosing to save one group in a collision, some prefer the young, while others the older group. Most choose humans over animals. This ‘Self-driving car dilemmas reveal that moral choices are not universal’ article in Nature covers this topic in more detail.
Throughout the course, we had a go at some exploratory data analysis. I refreshed my knowledge of basic statistics and learnt concepts like ‘overfitting’ that have changed how I look at machine-learning models. It’s hardly news, but one of my more significant takeaways was the realisation of how much more rich discussions of tech ethics can be if one knows their data, as complex or limited as it may be.
📚 Noteworthy Finds
Apps, sites and tools that I find along the way
Impact Beacon
An online tool by Penn State, Nasdaq and City Spark
A tool that helps to quantify and articulate a company’s impact in a few simple steps. Finally! It won’t cover all use cases but looks like a great starting point.
A visual introduction into machine learning
An interactive illustration by R2D3
I love anything that breaks down complex concepts into digestible parts — the visual does that to machine learning. Some additional tools are on the main page.
ODI Data Ethics Canvas
A tool by the Open Data Institute
This is probably the least original suggestion of this post as many people would have seen this already but thought it’s still worth sharing. It’s a great tool for methodically assessing data ethics and should be a must in the arsenal of any person working with tech.
