(7) Getting Experimental

Impact Hypothesis: Getting experimental

Let's face it. No matter how much research, analysis and strategy you do, you don't know what's going to have an impact. But you can try things and test their effectiveness. If you want to work on societal or systemic change, everything you do is an experiment. From the simple agenda you use to run a volunteer meeting to the in-depth plan you present to a policy-change coalition you've been working with for six years - all of it is experimental. Above all else - what you believe to be an effective method is an assumption until you prove that it has had a good outcome of some kind. Even your organisational mission statement, or the 10 year strategy everyone references when they're designing a new program, is fundamentally in question every day. You do not need to re-evaluate it everyday, but it is also not a law of physics - you can change it if you are awake to what you are testing, and review the big picture at some key points in the year.

The work you've done to fill out the different sections of the Impact Canvas may feel very spread out, very broad. It is a jigsaw puzzle snapshot of your assumptions, the story you are telling yourself right now about what you're aiming for, why, for whom, with whom, and who you are in that landscape.

Articulating a clear Theory of Change (how you believe change happens and thus how you are trying to intervene) is simple from here. Let's pull everything together and figure out how to run experiments that test your thinking. You need to iterate on your Theory of Change based on what you learn - so let's work out what you need to learn and how you can learn it.

Exercise: FILL IN THE BLANKS

[clearly named problem] is the problem which breaks my heart. It is caused by [underlying problems] which I identify as systemic root causes. By doing [named activities] with or through [approaches and strategy including working with partners], we intend to have [numerically measurable] short term impacts. The anticipated ripple effects of [descriptive explanation of activities] will have ongoing impacts like [further description].

Worksheets: Start measuring now

GREAT WORK!

Now what? You need to work out how to try it in the cheapest, fastest way possible.

Why? You do not have much time or money or person power, so you can't waste time doing things that aren't working.

Write some numbers you could test! There's nothing holding you back from making a guess at the likely measurable outcomes. You want these to help you understand whether your assumptions are right or need to change.

• What are the core/strategic questions you're trying to answer with these experiments?

• What information will you need to gather to learn that?

Types of impacts you can measure:

• Primary impact you can measure in numbers easily. "We engaged with 25 people through the workshop who gave 5 hours each in volunteer time the following month as a result"

• Secondary impacts & Ripple effects you can measure using stories and anecdotal evidence. "3 months after the workshop we invited 6 of the participants to a lunch and they reported X changes to their behaviour and Y hours saved in the workplace as estimates of ongoing effects of the project. One of the participants said her mother had also changed their behaviour and was teaching the new skills in her workplace too."

• When you have more information you can make guesses about how many of your primary impacts lead to secondary impacts. "In the 90 day followup survey that was sent to all 25 participants, 24 said they had used the tools from the workshop in their daily life and professional context, 20 of them said other people had noticed a difference in their skills in this area and 10 of them said their colleagues wanted to do the workshop after seeing what they had learned. If one out of 5 who came to the catch-up lunch also said their mother has passed on the new capabilities to her professional team we can say that when we train 25 people approximately 20 organisations and families gain new skills and culture."

Run an experiment

Background	Raw Results
Hypothesis	Key Lessons
Testing method	Next Experiment

Draw up a table like the one above.

Based on what you're thinking so far, you should already know your Background and Hypothesis. But if you're finding Hypothesis hard to articulate that's understandable. You have to write the hypothesis as something you could prove right or prove wrong. Here's an example: "People will call their mother when they receive an online newsletter with the suggestion to call their mother." Now, you can test that directly.

If your hypothesis is a little big and vague, try sharpening it down to something specific which represents your bigger ideas in a small way. "People are more likely to talk in a group if the people in that group know who they are.'' How can that be sharpened? Every individual will contribute to a discussion if they have introduced themselves before the discussion begins."

Ok great, now how to test that? Use the Testing Method box to explain how you will practically test that hypothesis.

The second half of the table can be completed once you have run your test.