Making Reputation Measurable, Usable in Emerging Media Ecosystem

In an era where we have nearly unlimited amounts of information, one of the key issues is how to separate the good from the bad, the reliable from the unreliable, the trustworthy from the untrustworthy, the useful from the irrelevant. Unless we get this right, the emerging diverse media ecosystem won’t work well, if at all.

I’ve long believed that we’ll need to find ways to combine popularity — a valuable metric in itself — with reputation. This sounds easier than it is, because reputation is an enormously complex problem. But whoever gets this right is going to be a huge winner in the marketplace.

What do we mean by reputation? In this context, we mean many things. If someone points to a news article, for example, we have to consider reputation at many levels. Among them:

  • What “media outlet” — traditional, blog, whatever — is behind the article? If it’s the Economist, the reputation starts at a high level. If it’s Joe’s Blog, and I have no idea who Joe is or what he’s (if the poster is a he) has been doing for the past few years, the reputation starts lower, much lower.
  • What is the reputation of the writer/video-maker/etc.? I give a generally high rating to New York Times reporters, but I can name a few who’ve wrecked their credibility with me over the past few years. This can vary even within organizations.
  • How about the sources of the information cited in the article or broadcast or whatever? When the Times quotes unnamed sources who have clear axes to grind, I actively disbelieve what the Times is reporting. When it quotes a person I believe to be generally trustworthy, I put it in a different place on my credibility scale. Too bad newspapers don’t use footnotes; and way too bad they are so reluctant to link on their websites to more directly relevant source material. Bloggers don’t have this problem.
  • Then there’s the reputation of the person recommending that I pay attention to the report. If David Weinberger suggests that I read something, I have much more reason to trust that it’ll at least be interesting, because I trust David so much, and this trust goes exponentially higher when he’s recommending something about which I know he has domain expertise.
  • Other reputations of interest in this sphere could include the collective reputation of the readers or followers of the publication or person. The readers of the Economist know a lot about a lot of things the magazine covers, and the fact that they pay the high subscription price tells me I should give the publication more of my trust.

Measuring reputation is another rub. It’s incredibly hard, and currently the tools for measuring are at best crude.

In a world of Web APIs and other emerging tools, however, there are glimmerings of hope. I’ve been begging people at eBay for years — to no avail — to make people’s reputations as buyers and sellers portable. By that I mean let people create a badge of some kind, with some real data behind it, and let them post that badge on their own work and make the data available in a granular way.

Your eBay reputation is not an exact proxy for your general trustworthiness, as a person or as an information creator. For one thing, we know that people are constantly gaming eBay’s system. For another, how you behave in buying and selling goods online doesn’t say how you’ll behave in other situations. But at the very least it’s a useful thing to know.

Your Karma at Slashdot are another useful metric. So are the individual users’ contributions in the collaborative filtering at Digg and Reddit. Useful, but clearly not sufficient by themselves to let you make big decisions about someone’s overall integrity.

But combine a bunch of reputation systems and you’re getting somewhere — and a world of APIs and interactive data suggest at least the possibility of finding a way to blend various measures into something that is more useful than what we have. At least I hope so.

5 thoughts on “Making Reputation Measurable, Usable in Emerging Media Ecosystem”

  1. seems to me that one reason that these measures are difficult to combine is that they actually are measuring different things. Doctorow’s “whoofie” is challenging for a social environment for some of the same reasons that a single currency (pace Douglas Rushkoff) doesn’t always make sense.

    My e-bay reputation (should I have one) might — at best — represent my respectability and reliability as an e-bay seller/buyer, but that’s very different from my academic reputation (perhaps citation index would make sense?), which in turn is very different from my reputation as a cool-hunter (perhaps my twitter followers graph? some kind of Slashdot/Digg index?).

    to “combine” these measures seems like an attempt to boil these many dimensions to a single dimension — and I’m not sure this is even what we would *want* in a social domain, let alone whether it’s possible.

    I’ll go out and assert a position that’s maybe a little stronger than it deserves to be: we don’t *want* 1-dimensional metrics for these questions. Some measures may be better for finding (say) a romantic match, others might be better for identifying interesting articles I should read, and yet a third might be good for identifying bargains I should snap up.

  2. Jeremy’s points above are definitely valid, but I still think it would be possible to develop some kind of aggregated reputation tool. It might not be possible to condense it down to a single value, but you could at least get an overview (both machine and human readable) of that person’s reputation through all the communities where they interact.

    If anyone’s interested in participating in the development of such a tool, let me know.

  3. Jeremy said: “… we don’t *want* 1-dimensional metrics for these questions. Some measures may be better for finding (say) a romantic match, others might be better for identifying interesting articles I should read, and yet a third might be good for identifying bargains I should snap up.”

    So, let me, as a reader or shopper or dater determine how much weight to give the reputations from different systems. Give me a default aggregated reputation and then let me tinker.

  4. Jeremy, I wouldn’t suggest boiling down everything to some single number — a FICO-like (god forbid) reputation score. I’m envisioning something more along the lines of Ari’s user-configurable blending system. 

    Ari, the blending of various reputations should indeed be up to the user. This makes it fuzzier, of course, but more useful if we can get it right.

    Ben, I’d love to see a project like this get started.

  5. A man is as good as his word and a man’s value shows through his actions. Would I honour anyone for that? Sometimes I hate too succesfull people. Primitive jealousy I guess.
    Reputation is situational and relative. Like a local currency without a standard. Reputations are based on opinions based on experience. Opinions (about) someone are difficult to measure. Today we estimate them by gossiping about people. Behind their back, to be honest. Literally that is. Only if their are statistically enough people to judge me, my actions, being and everything, the measurement would be trustworthy enough for practical purposes. I  fear most people will be like me; just not interesting enough to even be qualified for gossip.

Comments are closed.