Making Reputation Measurable, Usable in Emerging Media Ecosystem

In an era where we have nearly unlimited amounts of information, one of the key issues is how to separate the good from the bad, the reliable from the unreliable, the trustworthy from the untrustworthy, the useful from the irrelevant. Unless we get this right, the emerging diverse media ecosystem won’t work well, if at all.

I’ve long believed that we’ll need to find ways to combine popularity — a valuable metric in itself — with reputation. This sounds easier than it is, because reputation is an enormously complex problem. But whoever gets this right is going to be a huge winner in the marketplace.

What do we mean by reputation? In this context, we mean many things. If someone points to a news article, for example, we have to consider reputation at many levels. Among them:

• What “media outlet” — traditional, blog, whatever — is behind the article? If it’s the Economist, the reputation starts at a high level. If it’s Joe’s Blog, and I have no idea who Joe is or what he’s (if the poster is a he) has been doing for the past few years, the reputation starts lower, much lower.
• What is the reputation of the writer/video-maker/etc.? I give a generally high rating to New York Times reporters, but I can name a few who’ve wrecked their credibility with me over the past few years. This can vary even within organizations.
• How about the sources of the information cited in the article or broadcast or whatever? When the Times quotes unnamed sources who have clear axes to grind, I actively disbelieve what the Times is reporting. When it quotes a person I believe to be generally trustworthy, I put it in a different place on my credibility scale. Too bad newspapers don’t use footnotes; and way too bad they are so reluctant to link on their websites to more directly relevant source material. Bloggers don’t have this problem.
• Then there’s the reputation of the person recommending that I pay attention to the report. If David Weinberger suggests that I read something, I have much more reason to trust that it’ll at least be interesting, because I trust David so much, and this trust goes exponentially higher when he’s recommending something about which I know he has domain expertise.
• Other reputations of interest in this sphere could include the collective reputation of the readers or followers of the publication or person. The readers of the Economist know a lot about a lot of things the magazine covers, and the fact that they pay the high subscription price tells me I should give the publication more of my trust.

Measuring reputation is another rub. It’s incredibly hard, and currently the tools for measuring are at best crude.

In a world of Web APIs and other emerging tools, however, there are glimmerings of hope. I’ve been begging people at eBay for years — to no avail — to make people’s reputations as buyers and sellers portable. By that I mean let people create a badge of some kind, with some real data behind it, and let them post that badge on their own work and make the data available in a granular way.

Your eBay reputation is not an exact proxy for your general trustworthiness, as a person or as an information creator. For one thing, we know that people are constantly gaming eBay’s system. For another, how you behave in buying and selling goods online doesn’t say how you’ll behave in other situations. But at the very least it’s a useful thing to know.

Your Karma at Slashdot are another useful metric. So are the individual users’ contributions in the collaborative filtering at Digg and Reddit. Useful, but clearly not sufficient by themselves to let you make big decisions about someone’s overall integrity.

But combine a bunch of reputation systems and you’re getting somewhere — and a world of APIs and interactive data suggest at least the possibility of finding a way to blend various measures into something that is more useful than what we have. At least I hope so.