There is endless innovation and variety in what I call the “best guides” segment of the market. These are guides, print and online, that help consumers find and select the best of something – from hotels to restaurants to contractors to consumer electronics.

It’s a huge market segment. Consider such vast scale businesses as Yelp and TripAdvisor that largely focus on restaurants and hotels respectively, though you can find on their platforms crowd-sourced reviews for just about anything. If you have huge ambitions, it’s pretty well established that the fastest way to develop massive amounts of content is via crowd-sourcing. If you build it, the crowd will comment on it and rate it for you. The downside to crowdsourcing has always been lack of control. Too many people posting ratings and reviews have malign motives. More fundamentally, two people can honestly have diametrically opposed opinions about a restaurant, for example, and it’s close to impossible to reflect this in an overall rating. Crowdsourcing depends on sheer volume for accurate reviews to drown out inaccurate and biased reviews. It works, more or less.

Knowing the limitations of crowdsourcing, there have been many who have tried to refine the concept. Angie’s List was an early pioneer, aggregating reviews but only fromits members and only forits members, what we call a “closed pool” model in our Business Information Framework. The concept worked, but was difficult to scale, in large part because members didn’t want to pay for ongoing memberships. Angie’s List has since shifted to a lead generation model. I also wrote in 2016 about a company called BestPickReports.com, that’s building its business both online and with expensive print guides mailed to consumers. It appears to be a long-term play, and is backed, somewhat surprisingly, by EBSCO.

 We all know about Consumer Reports. For decades, Consumer Reports was the first place to check before making a major consumer purchase such as a car or a dishwasher. Consumer Reports did all the testing and rating itself, and understood that its reputation was everything, so much so that it prohibited manufacturers from citing its reviews, and the owner was a non-profit organization. In the days of print, Consumer Reports did very well selling subscriptions to its print magazine. It wasn’t an ideal way to distribute information (how do you buy that new car in March when the new car reviews didn’t come out until the May issue?) but it worked for a long time because there weren’t a lot of options. That’s why Consumer Reports had some struggles when it moved online because consumers didn’t want an online subscription as much as they wanted to be able to buy just dishwasher reviews and only when they were in the market for dishwashers. Consumer Reports continues to flourish, the result of momentum, its pristine reputation and quality reviews, but it’s quite possible its business model will come under increasing pressure for the same reason as Angie’s List: selling a continuous information service to consumers who don’t continuously need information is just plain hard.

Finally, let’s look at the original arbiters of what’s good, better and best: newspapers and magazines. 

Hearst Magazines has a review site called BestProducts.com. While the name might imply product testing, the site recommendations appear to be closer to the traditional “editor’s picks.” There is heavy use of the phrase “what welike,” and the site overall seems to be much more about informed personal preferences of the writer – more taste-making than research. Indeed, aside from a great (and arguably misleading) domain name, these are product recommendations that would not look out of place as print magazine articles from ten or twenty years ago. Online forced a change in business model, however. Hearst links to vendors of all the items it recommends, hoping to profit from online referral fees.

 The New York Times blends a few models together through its Wirecutter.com site, a business it acquired in 2016. Wirecutter offers much more than the personal opinion model of Hearst, but less than the rigorous product testing of Consumer Reports. It walks a middle ground, doing real product research, but not actual product testing. In terms of business model, Wirecutter follows Hearst, generating revenue from product referral fees.

 Depending on product referral fees is a risky business because of “leakage.” Simply put, it’s too easy to take your recommendation but not click your link. When that happens, the business generates no revenue. The only real solution to the leakage problem is sheer traffic volume, something both Hearst and the New York Timesalready have and can easily leverage. The New York Times, for example, is increasingly citing Wirecutter in its own news stories, albeit with full disclosure of its ownership.

 There is no single best model, and here’s a rundown of the tradeoffs. Crowdsourcing works, and it is cheap, but the quality of the content is uneven. Closed pool crowdsourcing yields a huge step-up in quality, but it’s a tough model to execute. 

 You can generate your own reviews to guarantee the quality, but you have to fight the trend towards unbundling. Consumers will pay for reviews and recommendations, but only the ones they want when they want them. It’s tough to generate adequate revenue on that basis.

 Online referral fees are an inherently dicey business because it’s too hard to mask the name of the manufacturer, and there are far too many sellers, all a click away. You can make it work if you have gobs of traffic, and this is even a better business if you can leverage your existing traffic and not start from scratch.

 If I was trying to build a “best guide” site, I’d select Wirecutter as my starting point. It has the benefit of offering true product research without the huge testing costs incurred by Consumer Reports. It totally controls both the research process and resulting recommendations. It can leverage the brand and traffic of its parent, the New York Times. What would I change? First, I’d see if I could sell recommendations on an a la carte basis. Buying a dishwasher? Then buy our dishwasher reviews. I might also be able to generate some additional revenue from national retailers or manufacturers who could offer special deals along with the recommendations, though I would need to be careful to make it clear nobody had paid for a preferential rating. I’d ditch the referral fee model because it’s catnip for free-riders. Finally, I’d wrap the New York Timesbrand more aggressively around Wirecutter to reinforce the quality of the recommendations. I understand why the New York Times is moving cautiously here, but at some point, if you want to be in this business, you need to be in this business. You can’t hold it at arms-length. 

If you’re considering getting into the business, leverage your strengths, choose the right content and business model, and plan for the worst and hope for the best, because as of yet, there is no clear pathway to success in this huge and tantalizing area.