Is Your Arts Programming Usable?

Posted by Ian David Moss On October - 5 - 2011

Ian David Moss

At Fractured Atlas, we’re in the process of rolling out a few new technology products that have been in the pipeline for the past year or so. One of these is Artful.ly, which is the hosted version of the ATHENA open-source ticketing and CRM platform that was released earlier this year.

Another is a calendar and rental engine add-on to our performing arts space databases in New York City and the San Francisco Bay Area that will allow visitors to the site to reserve and pay for space directly online.

For both of these resources, we felt it was important to get feedback from actual users before proceeding with a full launch. So we engaged in a round of what’s called usability testing.

Usability testing differs from focus groups in that it involves the observation of participants as they actually use the product. So, rather than have people sit around a room and talk about (for example) how they might react to a new feature or what challenges they face in their daily work, you have people sitting in front of a computer and trying to navigate a website’s capabilities while staff members look over their shoulders and take notes.

Users are given concrete, specific assignments like “exchange Bob’s Friday night tickets for Wednesday night’s show,” and the degree to which they were successful, the time it took, and how they felt about it are all recorded as part of the testing process.

I was thinking recently about how the idea of usability testing could be applied to a marketing or programming context. After all, the purpose certainly translates: don’t you want to know how people really interact with what your organization offers? Especially if they’re not already extremely familiar with what you do?

They key feature of usability testing that makes it different from most other kinds of feedback-gathering methods is that it is based on direct observation rather than self-reporting. As applied to artistic programming, for example, it would be more like Drew McManus’ Take a Friend to the Orchestra series than Clayton Lord’s intrinsic impact initiative.

Because of its labor-intensive design and need for complicity from the study subject, it doesn’t generally allow for a large sample size, but the richly detailed information that comes out of it can yield tremendous insights.

Here are a few other scenarios that come to mind where usability testing (or something similar) could be relevant:

  • Can your administrative assistant’s grandmother find tickets to your latest show, without having been provided with your website or any other information about it?
  • How long does it take a visitor to your museum to find the gift shop and purchase a mug for her friend’s birthday?
  • Is a newcomer to your organization able to follow the plot of your opera’s performance of Rigoletto? How much, and what, does he remember about it afterwards?

8 Responses to “Is Your Arts Programming Usable?”

  1. Ian, I love the idea of re-contextualizing “usability testing” as a feedback mechanism for arts audiences. As with online usability testing, the devil will be in the design, but the concept is well worth pursuing.

  2. Drew McManus says:

    Great topic Ian and one often overlooked area for usability testing (on and offline application) is straightforward guerilla testing; meaning, grab a bunch of current users and have them use or do whatever it is you’re working on. It helps sidestep many of the devilish details David is justifiably bringing up plus it produces much better “real” results than lab testing. Then there’s the double bonus that it is less expensive and it forces those of us in the business to get some face time with our patrons. It’s rarely a problem to get people to participate, who doesn’t love to spout off opinions.

    Even more interesting is passive guerilla testing; which would be more useful for non-online based testing. So if you have a new store layout, simply sit back and observe user behavior and make adjustments as necessary. Sure, there are terrific algorithms used for crowd management etc. But show me an arts group who has money to burn on that and I’ll show you a group I want to send a proposal.

    In order to plug some of the holes in guerilla testing, you have to revisit the issue from time to time to make sure the desired results are consistently reproduced; if not, then you may have missed something in the original analysis or conditions have changed. Either way, time for more testing.

    Lastly, a bear trap to avoid if you’re testing an online component is don’t use your own laptops, tablets, or smartphone. Instead, let subjects use their own device so you can get a better sense of how different variations of the same interface can impact your design. In fact, whenever I do client demos for anything tech based service, I have them use their machine. Whether they realize it or not, it builds trust becasue they are using their own, familiar system (plus it means I don’t have to haul my laptop around as much).

  3. Street Art says:

    nice article. it help me a lot. Try to do it my self.
    Thanks for sharing…

  4. Ian – this is something so often overlooked in arts organizations. Because we in the arts are so comfortable with how things work, we sometimes get that insider “you’re just supposed to know” attitude towards things, forgetting that newcomers don’t have the knowledge base we all have. What you are suggesting is essentially taking the “secret shopper” approach so often used in competitive analysis and applying it internally. And Drew’s point about guerilla testing is a good one. Doesn’t have to be expensive and perfect to be useful.

  5. Marcus says:

    What measures do researchers who use usability testing take to ensure their observations are objective. You mentioned some factors but I am not totally convinced that an opinion would skew the data. Thoughts?

    detroitdance.tumblr.com

    • Hi Marcus,
      There’s really a wide range in the degree to which usability testers try to make their data objective. The “guerrilla” testing that Drew mentions above prioritizes convenience and cost over scientific exactitude. By contrast, old-school usability testing is very tightly controlled and more closely resembles a true experiment, but often at great cost.

      A famous paper by Jakob Nielsen suggested that most obvious problems with a product are detected by just a handful of users, giving support to the more casual approach (which is also endorsed by usability expert Steve Krug).

      http://en.wikipedia.org/wiki/Usability_testing#How_many_users_to_test.3F
      http://sensible.com/

  6. [...] Is Your Arts Programming Usable? considers the concept of usability testing taken outside of its usual tech- or product-specific milieu. Here’s an excerpt: At Fractured Atlas, we’re in the process of rolling out a few new technology products that have been in the pipeline for the past year or so. One of these is Artful.ly, which is the hosted version of the ATHENA open-source ticketing and CRM platform that was released earlier this year. Another is a calendar and rental engine add-on to our performing arts space databases in New York City and the San Francisco Bay Area that will allow visitors to the site to reserve and pay for space directly online. [...]

  7. [...] in that it involves the observation of participants as they actually use the product,” said Ian David Moss, a development consultant who works in the Arts. “They key feature of [...]

Leave a Reply

ARTSblog holds week-long Blog Salons, a series of posts by guest bloggers, that focus on an overarching theme within a core area of Americans for the Arts' work. Here are links to the most recent Salons:

Arts Education

Early Arts Education

Common Core Standards

Quality, Engagement & Partnerships

Emerging Leaders

Taking Communities to the Next Level

New Methods & Models

Public Art

Best Practices

Evaluation

Arts Marketing

Audience Engagement

Winning Audiences

Powered by Community

Animating Democracy

Arts & the Military

Scaling Up Programs & Projects

Social Impact & Evaluation

Humor & Social Change

Private Sector Initatives

Arts & Business Partnerships

Business Models in the Arts

Local Arts Agencies

Cultural Districts

Economic Development

Trends, Collaborations & Audiences

Art in Rural Communities

Alec Baldwin and Nigel Lythgoe talk about the state of the arts in America at Arts Advocacy Day 2012. The acclaimed actor and famed producer discuss arts education and what inspires them.

Recent Comments