We all know the many benefits of user-generated data, including ratings and recommendations. Underlying all of it is a general belief in “truth in numbers” - if enough people say it is so, then it must be so. But what happens when the crowd returns a result that is obviously and intuitively wrong? A small but perfect example of this occurred recently when TripAdvisor published its list of “Americas Top Cities for Pizza.”
I’ll spare you the suspense: the top-rated city for pizza in the United States is (drumroll, please) ... San Diego, California. Are you already checking flights to San Diego or are you shaking your head in disbelief? I am guessing the latter, and that’s exactly my point.
Something went wrong in this tally by TripAdvisor. Most likely the underlying methodology wasn’t sound. TripAdvisor merely rolled up ratings for individual pizzerias and restaurants around the country and ranked them by city. There’s a little problem with that: context. What TripAdvisor converted to a national ranking were individual ratings made in the context of specific geographies. That’s why you see comments for top-rated San Diego along the lines of “it’s just as good as New York.” These people clearly didn’t see themselves voting in a national poll.
Also odd is that TripAdvisor decided to go ahead and publicize the results of this analysis. Clearly, TripAdvisor saw the potential for buzz with its surprising findings. Yet the flip side of this is the creation of a credibility issue: if TripAdvisor thinks the nation’s best pizza is in San Diego, how can I trust the rest of its information?
A few lessons for the rest of us can be found here. First Big Data is only valuable if properly analyzed. Second, evaluating data supplied by a large and disparate crowd can be tricky. Third, always balance the potential for buzz in the marketplace against reputation risk. Say enough dumb things online and you will hurt your credibility.
TripAdvisor will no doubt say in its defense that it is simply reporting what its users are saying. But if your user base is saying something dumb, it’s probably not in your interest to amplify it. And in fact, the TripAdvisor user base isn’t dumb; their individually smart comments were aggregated in such a way that they yielded a dumb result. But most people aren’t going to dig that deep. They’ll simply say that’s one less reason to rely on TripAdvisor.