Let Evaluations Be Fun, Be Life

Posted by Marc Maxson On May - 1 - 2012

Marc Maxson

Think about the most fun you’ve had doing charity work. What was it that really appealed to you? Was it the smiling faces of kids playing a sport or painting a mural? Maybe it was the moment you realized someone’s life would be forever changed by the small token of love that a program enabled one person to give another.

Do you know what those moments have in common?

First, they are significant on an emotional, social, or metaphyiscal level—and so no traditional evaluation is well-suited to quantify them.

Second, these moments belong to those whose lives have changed. Your impact, as the person who helped make it happen, should not be the focus (unless you enjoy being self-centered and alone in the world).

So why do we continue to act as if “quantitative” surveys about our own “impact” are smart?

My decade working as a neuroscientist with actual “quantitative” data enables me to confidently dispel this notion once and for all. Here me out:

  • Social change is social. That means it depends on people. Lots of them. People lie, especially in surveys, and often with the best intentions. Self-reports from people are not quantitative.
  • So that’s why we have statistics, right? Inferential statistics depend on random sampling, and sampling is almost never random given the reasonable time and cost constraints placed on nonprofits.
  • Even more alarming, statistics has no really solid way of telling if the sampling was done randomly.
  • If random sampling is a problem, then results will not be reproducible over time and in different places. That’s why a lot of high-paid people interpret them and argue over methodology. But I think that’s a distraction from the core problem—which is our obsession with extrapolating from brief and tiny samples of life to broad and timeless descriptions of social change and impact.
  • If you want quantitative data about people and social change, it’s probably more practical to transform our evaluation tools into a regular part of daily life—like Facebook or Google—so that we’re constantly looking at tens of thousands of bits of knowledge instead of just a few hundred.

That is why GlobalGiving has been running a massive experiment around storytelling in East Africa since 2010. A standard evaluation would never work for us. It would be both too costly and too limited to cover our 530 partner organizations in Kenya and Uganda who typically receive about $6500 in a year through our online fundraising website.

So how did we merge our “evaluation” with life’s activities?

We worked with our partners to find and train thousands of young people. We targeted the poor, typically girls—the kinds of people denied opportunities in their community—and gave them a job as a scribe.

For 15 cents a story, they would go out and interview people each month and collect two stories from each about two different “community efforts.” Anything that a person or organization tried to do to help the community was a valid story.

Since this was an open-ended, continuous, and broad survey, we learned a lot more about every kind of social problem. Certain themes emerged again and again, such as child abuse, problems paying school fees, and runaway street children. And with 40,000 stories and counting, when a story theme emerges, it really does—in the sense that 25 stories are anecdotes, but 2,500 stories are data.

This is not a random sample, but we could easily collect a million stories by 2016. Because we are not pushing people away from their social circles and into some artificially controlled “random sampling” environment, we get more stories more often. This allows us to aim for a large and diverse sample instead of a random one.

In the long run, we will be able to do what standard evaluations cannot—start to see if trends around, say, child abuse, rape, and hungry homes are the same over time and across many locations. But even more important, tens of thousands of Kenyans and Ugands are finally getting to have their say.

These stories are all online, and GlobalGiving is only the steward; the stories belong to the storytellers and we work to ensure their voices are heard by all the organizations working in their community.

This has been difficult, requiring us to develop new tools for visualization, real-time SMS feedback & polling, and define a strategy for hosting regular community meetings across the region.

Being a steward of stories, and a servant to the community, is the smart way to view our place in the world. Instead of asking, “are we making an impact?” we should have been asking, “are things changing for the better?”

Were the goals of the community attained?” I’m convinced that defining “impact” and our own attribution to it is a misguided dream spawned from the minds of the powerful and self-centered. And I am not alone.

The next time a funder asks you for proof that your work was the reason the goal was achieved, ask them for proof that the dollars they gave you were the crucial input in you achieving your goal. Sounds absurd when you flip it around, doesn’t it?

The world is full of storytellers. With more vision, we should be able to build the means to capture life in greater depth, so that we will know if things are improving. If we’re doing our best, things will, as singer Mark Erelli explains:

I wonder sometimes what I will pass on

And how much can one voice do with just a song

Sometimes injustice and indifference

Are the only things I see.

 

But I refuse to let my hope

Become the latest casualty.

 

So I’ll sing of love and truth

And try to practice all I preach

And if I can’t change the world

I’ll change the world within my reach.

 

If we show them all the tools

They might leave this place in

A little better shape than me and you

After all, we are only passing through.

6 Responses to “Let Evaluations Be Fun, Be Life”

  1. Sherri Emily says:

    What an inspiring story. This is fantastic what you are doing for others. I truly enjoyed the read and the awesome poem at the end. God speed.

  2. Pam Korza says:

    It’s great to learn about the work of GlobalGiving and your “massive experiment around storytelling!” First, the notion of being “a steward of stories” should immediately set many skeptical arts practitioners at ease about evaluation. Second, the question of scale often comes up along side that companion question about how you can really generalize from single projects or organizations, as Mark Stern and Susan Seifert also point out via the work of the Social Impact of the Arts Project. (Check out Mark’s exceptional posts in this Salon!) I’m intrigued by the scale of GlobalGiving’s efforts and that the collection process is so literally on the ground. I’m curious how the thousands of stories over time are analyzed in order to turn them into data that can reveal trends. While many evaluators advise to focus, focus, focus, yours is an emergent approach which seems generative in its open-endedness. Thanks for this provocative post!

    • Marc Maxson says:

      Hi Pam,

      Thanks for your comments. The analysis that turns stories into data and the emergence of stories into themes is definitely the hardest unresolved part of the project. We use SenseMaker(R) from cognitive-edge.com along with several tools we’ve built. Some feed data into Gephi – a free, open source networks visualizer, but the lion’s share of work lies ahead. Since we run a website for thousands of orgs, I’d like to develop features that provide customized analysis to each organization, using algorithms to match their existing project reports with stories and showing them maps of themes, locations, and partner orgs in a big web.

      I agree – emergence is much more appealing to me than focus. Our survey (currently draft #14) aims to be a universal form for every kind of community organizing effort you can imagine, which is the opposite of programmatic evaluations.

  3. Marc Maxson says:

    The next post in this series will give examples from the arts of where good simple evaluation design and good data visualization have made a huge difference in the artistic design.

    One more point for Pam – the Instavaluation (personalized to each org) delivers three kinds of feedback to orgs:

    1. What are your core strengths? (alignment with your community’s needs)
    2. What are your missed opportunities? (what community wants but nobody is doing)
    3. What are your network opportunities (good local orgs with mission alignment that you should collaborate with).

    I think that’s a good start to giving everyone free universal evaluation-style feedback. And it costs pennies per org.

  4. Tim Mikulski says:

    Just an FYI that Marc’s next post goes live at 10 am ET tomorrow.

  5. [...] the final days of my work in Chajul, reading a recent article on ARTSblog has helped me reframe my work here. When speaking about stewardship and methods to [...]

Leave a Reply

ARTSblog holds week-long Blog Salons, a series of posts by guest bloggers, that focus on an overarching theme within a core area of Americans for the Arts' work. Here are links to the most recent Salons:

Arts Education

Early Arts Education

Common Core Standards

Quality, Engagement & Partnerships

Emerging Leaders

Taking Communities to the Next Level

New Methods & Models

Public Art

Best Practices

Evaluation

Arts Marketing

Audience Engagement

Winning Audiences

Powered by Community

Animating Democracy

Arts & the Military

Scaling Up Programs & Projects

Social Impact & Evaluation

Humor & Social Change

Private Sector Initatives

Arts & Business Partnerships

Business Models in the Arts

Local Arts Agencies

Cultural Districts

Economic Development

Trends, Collaborations & Audiences

Art in Rural Communities

Alec Baldwin and Nigel Lythgoe talk about the state of the arts in America at Arts Advocacy Day 2012. The acclaimed actor and famed producer discuss arts education and what inspires them.