Over the past number of years I’ve worked with data of all sorts for a variety of reasons and I have some thoughts about it. It’s important to note that these thoughts aren’t generally informed by a fathomless depth in any individual area. My perspective is broader than that―inclusive of more terrain and wider purpose―so adjust your expectations accordingly.
Keep in mind also that this is a perspective from working with startups and non-profits. In all cases these entities were interested in doing what I would broadly describe as ethically “good”. That is important to me, but so too is it important that data strive to be honest. There’s no shortage of stories or graphs demonstrating that data can be collected or presented in intentionally or unintentionally misleading ways.
Who are you?
I’m Kyle. I’m just a guy who has written grants for federal & state dollars and setup the systems and done the reporting and analysis to maintain those grants. I’ve penned letters of intent and applications for anonymous, public, private, and invite-only grants and set up systems to track and report on those grants as well. I’ve turned data into grants, grants into data, and everything in-between.
At the time of writing a system I set up is collecting and analyzing data for a non-profit member organization under a state grant; there are also several states using a system I created to collect and analyze data for a national program.
I value privacy, don’t want to waste people’s time, and find that data has a communicative value that goes far beyond the requirements of submitting a report to a funder. Data can help improve your organization’s internal functions and processes, it can define how partners, staff, and the general public perceive your organization. Data can also drive change, secure funding, or be the red herring that wastes time, money, and sanity.
What follows are some broad categories and general advice from what I’ve learned.
What to collect.
“An error in the beginning is an error indeed.” – Quotation I originally heard from the mouth of Matthew Phelps (I do not recall to whom he credited it).
“…since the least initial deviation from the truth is multiplied later a thousandfold.” – Aristotle, De Caelo
“Because a small error in the beginning grows enormous at the end…” – Thomas Aquinas, On Being and Essence
One of the worst mistakes that can be made is to track the wrong thing. This can happen for a variety of reasons, but most often is the result of misaligned incentives or the relative ease of allowing the status quo to continue on unexamined forever. It can also cause burnout―adding additional friction to every person on staff for unclear (or no) benefit―causing distraction and dilution of the actual product or service you intended to provide in the first place.
Most small organizations aren’t employing entire departments dedicated to tracking “all the things, all the time” and implementing comprehensive quality assurance and continuous quality improvement on an ongoing basis. Most smaller organizations are having trouble staying afloat and as part of that struggle needing to lean on the fact that they can be nimble and quick to survive and adapt. They need to do this with what to track too.
Put bluntly: you should always track what you’re aiming to impact.
An example: On organization I worked for once had hired an “SEO expert” to manage a $1MM+ annual online ad-spend. This person was also broadly in control of a mostly static website that the ads directed the small percentage of ad-clickers to, which in turn compelled them to reach out to the organization by phone (or form) to a sales rep who would try to close the deal.
What did this person report on primarily? Click performance of ads (as a percentage of how many people saw the ad to how many clicked… and presumably landed on the website). The majority of their reports were focused on increasing the CTR (click-through-rate) of their ads.
The marketing person who controlled the call center mostly talked about the “quality” of leads that came to the call center in the form of phone calls (and the occasional form submission).
Are these the right things to focus on? Is it worth $10k/month to make extra people click an ad? (Maybe, but only if they become a customer, right?) How should we categorize lead “quality”? (Does failure to become a customer mean you were a “bad” lead?)
All of these numbers are great, in theory, but what are we trying to do? The answer is simple: serve people who need our help.
Even if we assume those numbers are worth tracking, are we sure they’re the right ones? What about the landing page for the ads? Does that convert well? Have we A/B tested? And the calls: have we listened to the calls? What does a “bad” lead sound like? And what if we saved $10k/month and just left the ads unchanged?
What happened? To start, we had a contest. This same consultant had been developing a new $20k+ website that was overtime and the cost kept going up. Myself and a co-worker launched a competitor website for $50 over a weekend, showed the CEO, and we ran an A/B test on his ads: half the clicks would come to our site and half to his. We’d track how many “calls” each website resulted in (and from there how many callers became customers). Our site won on all counts, especially in conversions from clicks to calls (performing more than twice as well).
From there we learned that the SEO expert’s careful ad curation, that had been billed as an onerous daily activity, was actually a lower level staffer logging in once per week (Thursday at 10am) and not the daily and hourly curation he had sold to the boss (a quick download of logins from Google Analytics, the right data revealed this). I graphed a chart. This would have been less of a problem had they not lied… but they’d lied. We saved $120k+ per year.
Eventually we’d improve each part of the sales funnel until we had concluded that it was the calls… so we listened to a lot of calls, and then assigned the manager to listen to a sample week of calls. It turned out that a lot of “bad” leads were actually great leads if we didn’t assume that a lot of leads were bad.
I’m glossing over a lot of things here (heatmaps, call tracking, CRM software launches, etc.), but the result is that we started helping more people for lower cost per call and acquisition… which is what the company existed to do in the first place. We just had to change the focus of our reports to track something different that we wanted to improve, instead of focusing on the numbers we’d always used.
How to collect.
“In thy boldness over-rash
Madly thou thy foot didst dash
’Gainst high Justice’ altar stair.” – Sophocles, Antigone“non nobis solum nati sumus ortusque nostri partem patria vindicat, partem amici” (“Not for us alone are we born; our country, our friends, have a share in us”; De Officiis, 1:22) – Marcus Tullius Cicero
“Saving children―one spreadsheet at a time.” – Abby Patterson
Your organization is made up of people. Most of those people are there to do the thing that your organization is meant to do. They are not there, primarily, to help you make a chart.
Tracking things is never “free” because it has a human cost in time, effort, frustration, attention, etc. How lucky you (we) are to trouble ourselves with information and data instead of the actual things that the organization exists to do. It is not that the task of collecting and tracking data is unimportant (as it obviously can be), it is that it is rarely the primary thing the organization exists to do.
In small organizations you are likely one of a very small group of people who want this information, so I implore you to take seriously the fact that you are adding burdensome tasks to the people who make your organization run. Do not add tasks lightly. If there is a task that would take you an hour but save 50 people 10 minutes each a week: spend the hour. If the data is important, it’s time well spent.
Separately, if there is good reason to burden your people you must share the reason. If you can’t explain the reason then it’s not a good enough reason to burden them. You should be able to say: we’ll be able to secure more funding, or serve more people, or hire more people, or something. Share why this matters. If you’re being honest, and your reason is good, your people will not find it as burdensome because they are part of the benefit you’re creating. Their efforts would mean something.
One example: I was once dropped into a medical billing department with a co-worker and told, in general, to figure out “what was going on” because we hadn’t been getting paid and the billing professionals were talking in code―and trust was depleting rapidly.
Many process improvements and discoveries resulted from this―including a more specific and informed distaste for American “healthcare”―but one key change that involved burdening a large number of staff needed to be made due to an error in the original setup of the Electronic Healthcare Record (EHR) system they used.
We had to setup an entirely separate system to track some specific data components, retrain key staff, and modify policies and procedures to ensure certain information would be accessible, accurate, timely, and secure for reporting.
We had conversations with on-the-ground staff, managers, site-directors, and the board. Everyone already knew about payment delays and denials, so the importance was high. We explained the issue at a high level, re-trained key staff, and even though the effort was onerous, it resulted in more payments, fewer denials, and the organizations ability to continue doing the work again.
If we had not communicated with them, or if the effort wasn’t meaningful, the resistance would have been substantial (and warranted). Instead, we were able to be nimble and work as an cohesive organization toward a single goal.
What (and how) to share.
“Often the most effective way to describe, explore, and summarize a set of numbers―even a very large set―is to look at pictures of those numbers. Furthermore, of all methods for analyzing and communicating statistical information, well-designed data graphics are usually the simplest and at the same time the most powerful.” – Edward R. Tufte, The Visual Display of Quantitative Information
“Lying graphics cheapen the graphical art everywhere. Since the lies often show up in news reports, millions of images are printed. When a chart on television lies, it lies tens of millions of times over…” – Edward R. Tufte, The Visual Display of Quantitative Information
The information you choose to share (and how you choose to share it) will define your organization to whatever audience you’re sharing it with―whether board members, donors, staff, partners, or the general public. So choose wisely what you share.
One of my favorite examples is this simple chart in a hand drawn annual report I made in late 2024. It simply depicts an outcome we wanted in a simple bar graph that’s disguised as hand drawn materials we actually distributed that year. The number goes up (obviously), depicting the number of materials we distributed during our statewide Child Abuse Prevention (CAP) Month efforts in April.
This should be a simple number to collect, but it was the product of several complex systems that didn’t exist before I set them up. Before I came this data wasn’t tracked at all. We didn’t really know spend, certainly not on individual materials, and because this data didn’t exist, I couldn’t add years prior to 2021 (even though they were almost certainly smaller). Because we focused on distributing more materials we were able to setup a complex system that captured this information (along with much more) and that allowed this data to be easy to collected and made certain that it was comparable year-to-year.
But the fact that I’m showing “Materials Distributed” is also a choice. There’s only so much room on the page. I could have showed the increase in the number of partners receiving materials, or participating in CAP month, or amount spent, donated, or saved. I could have show a map of where those materials were shipped, or any other number of data I had to collect in the course of managing orders and shipping them across the state. I chose to highlight materials because that was a focus area we wanted to grow (and we were provably growing).
(Also, I’d already used a state map the page before to show where conference attendees came from.)
As to how you should share the information, unless your audience is exclusively people with graduate degrees who still love reading journal articles for fun, you’re going to have to limit both the amount of data you share and the complexity of that data―even data in visual form.
It is not uncommon for me to try to talk people out of using a stacked bar graph. What is a stacked bar graph? Well, it’s a bar that literally “stacks” different parts of whole (kind of like a vertical pie chart) into a single bar. In most cases I think that the general public will struggle to interpret a chart with lots of stacked bar graphs over multiple years. It’s too much information in a small space. Efficient, but confusing.
Below is an example where because of the audience (funders interested in the cause) the topic (literally how a single dollar is spent on substance abuse from a study), and the number of bars (two) I created the below graphic from a much more boring table of numbers (read more here).
Making changes; Accepting Problems
One of the best uses of data is just explaining otherwise confusing phenomena. How much time does something take? How much money does something cost? What are we actually trying to achieve? Is this actually a problem? What price should we set?
Everything is ultimately a decision, but data can inform that decision with objective facts. It may be painful to know that “we are losing money on every sale,” but it’s also information you can use to decide if this is worthwhile as a loss leader. You may be unhappy to hear that your partners don’t care about that $40k/year marketing material you’ve been producing for them for the past 3 years, but it’s truly great to learn that you can reallocate that much money to something they’d love. Your manager, boss, or board of directors may not want to believe that the fundraiser they’d planned can never recoup the expenses unless they get more efficient and do 52 of these events each year, but it’s important to be clear-eyed about fundraisers needing to actually raise funds.
Each of the above examples are real examples of data projects I undertook to answer questions at various organizations. In some cases I believe the organizations made bad decisions and ignored the information. In other cases they reallocated focus and funding and delighted the people they serve. In either case, at least they knew.
One last thing on public goals: I like creating large dashboards with as close to real-time information as possible to drive change through constant focus. If there’s a metric you want to improve you need to make it visible: data makes things visible. There’s a reason sales people ring big bells and gala’s have temperature-shaped bar charts tracking giving towards a very visible goal: no one can be confused what the objective is if it’s in front of their face.
New Habits
Not every metric needs to be tracked forever. Habits do form and organizational change can happen. If the tracking is onerous and you’ve seen massive change, find the next problem to tackle. It’s okay to not capture every piece of data that is possible. The goal isn’t data, the goal doing better.
Semi-closing
I wrote this up in a single evening and haven’t really spent the time I’d normally spend to map it out or think about it fully. This is the result of a fun conversation I had this afternoon over coffee that spurred a recognition that I don’t much talk about the data side of what I’ve done over the past decade plus. I’ll likely return to this topic later with more sections and examples, but for now I’ll leave this as a start of a syntax for my philosophy of data in smaller orgs focused on rapid improvement and growth. I mean what I’ve said and the lessons were hard won. If you have anything to add do share.
Just my inflation-adjusted two cents.
“Yeah well, that’s just, ya know, like, your opinion, man.” – The Dude, The Big Lebowski