Learning Center Growth Part 2: Running Experiments
Updated: Jan 26, 2021
Last post, I wrote about a simple system for tracking your center’s impact revolving around your Key Performance Indicator, or KPI. This system forms the foundation of consistent growth. This week, I will discuss how to consider and run experiments that will help drive your KPI up over time.
Experimenting for Growth
What ideas do you have for growing your KPI?
Let’s say that your chosen KPI is total session count over a given period. You are now collecting and reviewing this KPI weekly, storing the results in a growing spreadsheet. The next question you should begin asking yourself is: what can I do to drive this number up? Your weekly decision making should be centered on improving this KPI, and I bet you have a few ideas already.
Begin a brainstorming session. An obvious place to start at is in marketing. What are new ways to get the word out about your center? Perhaps you already advertise in the once-a-semester university newsletter, but not in the individual department newsletters. Write that idea down in a spreadsheet. On top of that, tutors could present in lectures, and faculty might post about you in their LMS. Write them down to. Here’s my spreadsheet so far.
But wait - marketing isn’t the only thing that matters. For total session count, high quality session lead to returning students, and thus an increase in your KPI! So how can we improve session quality? Maybe sharing students’ feedback with our tutors will help them improve. I also heard of another center who matches new tutors with a tutor-mentor for 3 meetings during their first semester. Maybe that improves quality and retention, too. So here is my updated spreadsheet:
Now that we have some ideas, it’s time to go try them all, right!? Well, not so fast.
Running experiments and collecting data
If you made efforts on all of the above at once, your KPI would likely improve, but you wouldn’t know why. Just as important as short-term growth are the learnings that can help you over the coming years. You’ll need to run organized, small-scale, data-drive experiments on each idea to compare their effectiveness and efficiency. Here are our next steps:
Prioritize our ideas: Over time, you might have 20 or more ideas untested. So take a guess at which ideas are the most exciting. We’ll add a 1-3 priority rating column and sort high to low using Filters.
Consider how “testable” each idea is: Ideally, for each idea you’ll be able to “try it” a few times and see if you: successfully recruit students, if a tutor performs better, or something like that. Keep your KPI in mind.
Design a concrete, small-scale test that will allow one idea to be compared to another. The goal is to quantitatively decide where to invest your resources in future semesters.
(NOTE: it’s hard to design a perfectly clean experiment, with full statistical significance. Just do your best to generate some numbers that feel like a strong indicator.)
Our spreadsheet with a few tests mapped out:
Now plan these out in detail with your staff and execute them one by one. A few tips:
Link tracking is important for measuring the traffic from a given channel (for example, faculty posts on Canvas in which students click a link) - you can use free online services like Linkly or perhaps work with your IT to add tracking.
Don’t rush through this planning step. Test the systems, like link tracking, you will use before you “launch”.
Once you have run the experiment and gotten some data in, spend the time to wrap things up tidily and summarize takeaways. You want a number that captures the efficiency of the channel: eg. “tutor presentations generated 3 new students per hour spent by staff”.
Each experiment you run will teach you new things. You’ll understand your students better, and you’ll become a better manager. The ultimate result, however, is when you have proven that a certain tactic or investment is highly efficient. You’ll start to gain certainty in your decision making process because you have data to back it up. Your KPI will rise, the center will improve, and more students will be helped each semester.
Some experiments don’t work out, and you’ll have to be OK with that loss. Those will feel like a waste of time. But I promise, if you stick with them and keep adjusting your system to work for you, they start paying big dividends. 5 years from now, 10 years from now, the learning center will still be seeing the benefits of these experiments.