#bcni13 measuring the impact of journalism recap

Below is a round-up of materials from BarCamp News Innovation Philly 2013 discussions about measuring the impact of journalism. Thanks to Gino Canella and the Center for Public Interest Journalism for the videos! Also, check out Jim MacMillan’s roundup of posts from #bcni13.

Erika Owen’s post:

So you want to measure impact: BarCamp 2013

CPIJ summary video of the session Brian Abelson, Erika and I led:

Full session video:

Temple University journalism chair Andrew Mendelson’s “Beyond metrics: Thinking more broadly about journalism’s impact”  talk:


Full session

I live-tweeted the session and Storified some key points he shared:

#bcni13: Andrew Mendelson’s "Beyond metrics" talk

Temple University journalism chair Andrew Mendelson presented a talk called "Beyond Metrics: Thinking more broadly about the measurement of journalism impact (or building journalism’s theory of change)."

Storified by Greg Linch· Mon, May 06 2013 21:16:35

This session is being led by @andrewmendelson. #bcni13Greg Linch
He’s interested in public interest, investigative, accountability journalism. "Theory of change." — @andrewmendelson #bcni13Greg Linch
Two ideas: Public interest journalism goal to create informed & engaged citizens and "journalism as a curriculum" @andrewmendelson #bcni13Greg Linch
Characteristics: stories themed thematically, focus on problems + solutions, address civic values — @andrewmendelson #bcni13Greg Linch
Also: incl mobilizing info (how to get involved), incl opportunities for engmt and interaction, diverse sourcing — @andrewmendelson #bcni13Greg Linch
"We need to think of multidimensional aspects of change," @andrewmendelson citing my use of the term in bit.ly/GLimpact thx! #bcni13Greg Linch
Affective level (individ) shorter term: usability measures, attitudes to topic, civic agenda shift, attitude to org @andrewmendelson #bcni13Greg Linch
Cognitive level (individ) shorter term: issue knowledge lvl (free & cued recall), mobilizing knowledge, issue frame @andrewmendelson #bcni13Greg Linch
Conative/behaviorial level (individ): engmt, issue disc + research, mobilizing + voting intention, behav chg intnt @andrewmendelson #bcni13Greg Linch
Affective (individ) longer term: +/- public mood, cynicism, poltcl self-efficacy, social capital, credblty, divrsty @andrewmendelson #bcni13Greg Linch
>@andrewmendelson calls for A/B testing on news content… couldnt agree more but wonder if editors are on board.brian abelson
Cognitive level (individ) longer term: knowledge retention, cognitive complexity abt socl issues, knwldg abt commty @andrewmendelson #bcni13Greg Linch
Conative/behavioral level (individ) longer term: cmty groups activity lvl, readership, willing to support org 1/2 @andrewmendelson #bcni13Greg Linch
Conative/behavioral level (individ) longer term: netwrk lvl/diversity, voting, willing to disc, behavior change 2/2 @andrewmendelson #bcni13Greg Linch
Showing impact: A/B testing, matched sets btw cmtys, compr betw read vs not, comp attn lvls, content/attndnce links @andrewmendelson #bcni13Greg Linch
Societal level: issue-related behavior + policy change, civic activity, voting levels, more robust news ecosystem @andrewmendelson #bcni13Greg Linch
How do people wrestle with community issues and what are they’re looking for? asks @andrewmendelson #bcni13Greg Linch
You want to get beyond just knowledge Qs to the nature of knowledge Qs, says @andrewmendelson re NYT health quiz example. #bcni13Greg Linch

Join us at #bcni13 to work on measuring journalism by impact

It’s April so it must be time for BCNI! Brian Abelson, Erika Owens and I will lead a session this Saturday at BarCamp News Innovation Philly 2013 (see schedule) about implementing ways measure impact in journalism.

Last year I led a more general discussion about impact, which Andrew Spittle expertly documented in his notes. Springboarding from that, Wendy Warren and I did a follow-up #BCNI12 session to dive into specific, qualitative possible news metrics for success.

The thinking and conversations about impact have grown and evolved in the past year, such as with a conversation about impact at #ONA12. With this progress, we’re going to take a much more hands-on approach at the unconference this time, as Erika wrote in her preview post:

It’s pretty easy for the conversations around impact to go in circles, but the aim is for this discussion to be more focused. Greg suggested an “impact-a-thon” format where folks share case studies, discuss them in small groups, and then report back on the findings.

Do you have any case studies of stories you’ve worked on? Analytics and/or anecdotes to share of impact of the stories? Think back to those awards application cover letters, what did you describe for why your story should be honored?

The session, rather than theory, will focus on implementation. How have organizations monitored impact, and what are some specific examples of things they can do for future stories? We look forward to an insightful, action-oriented discussion, and want your help in designing the session. Please email the Google Group, share your thoughts, and join us Saturday morning.

Brian will share his experiences thus far exploring impact measurement as an Open News fellow at The New York Times:

In preparation for the discussion, I’m hoping to get your thoughts and concerns about impact measurement (read up here) – what have we done right so far and what’s missing from the conversation?

In that line Brian links to an extremely thorough list of posts, articles, studies, etc. relating to journalistic and media impact. Be sure to read up beforehand, especially ProPublica’s white paper, “Non-profit journalism – Issues around impact.”

The conversations and the company should be fantastic, so please join us on Saturday — and register for BCNI before tickets are gone!

ONA12 session pitch: How can we better measure journalism?

UPDATE: The proposal has been accepted! Session page: Have an impact. Then measure it.

I’ve proposed a session for the Online News Association 2012 conference in San Francisco on:

Better ways to measure journalism

If you’re interested, please up-vote/comment soon! Voting closes Thursday, April 12.

It would begin with an overview of interesting metrics in other fields, how we can learn from those and how we can possibly apply them to journalism. After that groundwork is laid, there would be a moderated discussion among the participants of other examples and how we could best implement new metrics.

For more background on the idea, check out:

Quantifying impact: A better metric for measuring journalism

Carnival of Journalism: Responses to “How can we better measure journalism?”


Carnival of Journalism: Responses to “How can we better measure journalism?”

This month’s Carnival of Journalism topic, “What’s the best way — or ways — to measure journalism and how?” grew out of a post I wrote in February, Quantifying impact: A better metric for measuring journalism.

With this question, I wanted to broaden the possibile metrics beyond just impact. Higlights from the discussion are below. Enjoy!

Sheree Martin offered four important questions we must first consider:

  • What is journalism?
  • What is impact?
  • How do we measure?
  • Who is measuring?

Kathy E. Gill asked about similar fundamentals, “What matters? What is the role of journalism, our purpose, our challenge?” and raised a good question of balance:

how does “but it matters!” coexist in an environment where assessment is measured in large part by short-term page views and click-throughs?

Denise Cheng noted an opportunity to measure “inert engagement,” described as “the engagement that doesn’t want to come out of hiding as big steps like shares and comments.” She said:

measuring impact by measuring engagement are manifold. First and foremost, the modus operandi on my patch of the Internet is that journalism’s highest ideal is to equip its readership with information from which they can take judicious action

Specifically, she recommended defining metrics before you embark, measuring topical importance to the audience and acts accordingly, and that the sturdier your metrics become over time, the more of a road map you have.

Jonathan Groves looked at the issue on a more elemental level:

The root of journalism is truth, and the time-tested method that journalists have to uncover that truth is verification. If we want to measure journalism, it must begin here.

Addressing the quantitative vs. qualitative measurement distinction, he said:

Measurement assumes quantification, and some ideas — such as verification — are better evaluated qualitatively. Creating a measure requires including some attributes and excluding others; inevitably, such measures are always imperfect approximations, especially when it comes to complex concepts.

Clarisa Morales Roberts described a possible formula framed by the effects of journalism:

If we want to measure the impact of media and online journalism, we need to consider action. Action is what defines Effective Media (EM), and Effective Media can be measured by the Action that is a direct result of Quality Dialogue that is Shared

So, if we want to consider Impact by measuring Action, that measurement has to be proportional.

Michael Rosenblum emphasized the importance of finding niche instead of mass audiences:

As more and more content begins to fill the blogosphere and cyberspace and the cloud and wherever else ‘it’ all is, the competition for the Holy Grail of mass audience becomes ever more intense, and as such, the content itself becomes ever more amorphous.

Yet where is the ‘real’ value?

The web gives us access to discrete groups with specific interests. Our goal should be ‘narrowing the field’, not expanding it. Creating affinity groups with a common interest and common goals, and then, making it possible for those people to achieve those goal – whether its contributing to a new project – as in Kickstarter, or going on a golfing trip to St. Andrews.

Steve Outing looked at how social media is gaining an edge in the impact realm:

When I look at the question, I can’t help but get sidetracked into thinking how social media (i.e., “the crowd” utilizing digital social tools like Twitter, Facebook, and Change.org, among others, to amplify their voices) in a growing number of cases is having more impact than the traditional news media can achieve themselves — or is driving the mainstream news media to pay attention to stories that their editors fail to recognize as important.

Carnival ringmaster David Cohn also proposed an alternative approach :

I want to measure a different kind of impact. The impact of the dollars we spend in pursuit of journalism and its meaningful impact.

What we don’t appreciate is the strength of the little guy. What they don’t have in “impact” they do have in efficiency.

Steve Fox challenged the assumption of measuring something like impact:

Perhaps we all need to remember that the true “impact of journalism” rests with the impact we have on people’s lives. Have we given readers/viewers an amazing piece of writing or video that makes them appreciate parts of their life more? Have we created an “Oh, wow” moment for readers/viewer? Have we expanded someone’s universe? Isn’t that why we got into this business? Isn’t that what journalism has always done?

Perhaps the real question should be: “Why are we spending so much time measuring the “impact” of journalism?” Because, it really isn’t quantifiable now, is it?

What would be your ideal measure for journalism?

Quantifying impact: A better metric for measuring journalism

Before Isaac Newton, words like mass and force were general descriptors, as James Gleick writes in The Information:

“the new discipline of physics could not proceed until Isaac Newton appropriated words that were ancient and vague—force, mass, motion, and even time—and gave them new meanings. Newton made these terms into quantities, suitable for use in mathematical formulas.”

The term information was similarly amorphous until Claude Shannon, while working at Bell Labs, quantified the concept in bits.

* * *

The journalism goals and business goals for news organizations are out of sync.

Pageviews. Unique visitors. Time on site.

Some journalism might be best quantified partly or wholly by one or more of those ways, but we need to explore deeper beyond these fairly simplistic metrics.

We know how these terms are defined, but what do they really mean? What do they help us achieve?

In creating a theory of information and quantifying information in bits, Shannon aimed to remove meaning. “Shannon had utterly abstracted the message from its physical details,” Gleick says.

For journalism, the goal should be to add more meaning to the information we use to measure our work. Granted, our current metrics aren’t meaningless. We use them because they do have meaning: views, comments, shares, etc. each has a meaning and can be measured based on that one-dimensional measure. The quantities of metrics increase because the works of journalism they describe are meaningful. Or, put another way, impactful.

So, what if we measured journalism by its impact?

Continue reading Quantifying impact: A better metric for measuring journalism