Consumer Reports is a publication that works hard to get things right. But its February 2007 issue ran a dramatically wrong review of children’s car seats—due to poor testing methods—and seriously jeopardized the trust it had won from its readers.
The organization’s recognition of the problem was the best demonstration I’ve seen of a) owning up to one’s mistakes, b) figuring out what went wrong, c) explaining what happened and d) putting into place policies to prevent such messes in the future.
And it was all done in a public way, with a systematic transparency that’s exceedingly rare in journalism.
Soon after the article, which reported that many car seats failed the magazine’s tests, came under challenge, it became clear that the tests themselves were flawed. The response from the magazine to its readers and the world was quick: It issued a retraction.
I subscribe to the CR online site. I got an email, and a friend who gets the paper version got the same letter via postal mail, from Jim Guest, president of Consumers Union, the title’s parent. He apologized, sincerely. He explained what he knew so far about the error, apparently caused by an outside lab’s tests. He announced a further investigation. And he promised extraordinary efforts not to let it occur again.
In March 2007, the very next issue, CR posted a detailed report (which also ran online) titled “How our car seat tests went wrong.” The “series of misjudgments” described in the piece is remarkable. It was especially worrisome given the publication’s record. I don’t rely on CR for everything I buy, but I’ve learned to trust its overall judgment on relatively uncomplicated consumer goods such as kitchen appliances, where I’m unlikely to spend much time on my own extra research. Were I the parent of small children, I might well have included car seats in that category.
The report explained everything about the tests in clear and unsparing language. It included justifiably angry comments from a car seat manufacturer and from outside critics. It was self-criticism of the sort one almost never sees from a journalistic organization, blogger or other media creator of any kind.
CR also posted a story called “Learning from our mistake,” a description of what it would do to avoid similar catastrophes in the future. Among other things, it announced that the publication planned to bring outside experts into the process when creating complicated testing procedures (and already does that to a degree), to fix the way it works with outside labs and to look much harder “when our findings are unusual.”
The last of those should have been second nature to the journalists and scientists at CR. After all, it’s famous for telling readers that when something seems too good to be true, it probably isn’t. In this case—with all those car seats failing the test—perhaps it was too bad to be true.
The magazine might consider opening its testing procedures in other ways. For example, it could create videos of the tests as they’re being conducted and post them online. Bring in the designated experts, by all means, but maybe some readers who are experts in their own way might spot something useful, such as an omission in the testing procedure or a valuable way to improve it.