The long, tortured quest to make Google unbiased
Next week, Sundar Pichai will try to reassure Congress that Googleâ€™s search engine isnâ€™t rigged. The Google CEO is testifying before the House Judiciary Committee on Tuesday, answering questions about â€œpotential bias and the need for greater transparencyâ€� in Googleâ€™s business practices. Itâ€™s Republican lawmakersâ€™ latest move in a series of hearings over Silicon Valley political bias. â€œGoogle has created some of the most powerful and impressive technology applications,â€� wrote House Majority Leader Kevin McCarthy in the announcement. â€œUnfortunately, recent reports suggest Google might not be wielding its vast power impartially. Its business practices may have been affected by political bias.â€�
We donâ€™t know exactly what questions will arise during Pichaiâ€™s testimony. But this summer, President Donald Trump caused a brief uproar by claiming (without evidence) that Google suppressed positive news about him. Reports indicated Trump might even direct regulators to investigate Google and other platforms for bias. But that proposal hadnâ€™t come from one of Silicon Valleyâ€™s many ideological enemies â€” it was supposedly promoted by recommendations site Yelp, which has spent years protesting what it calls unfair demotion of its search results.
That investigation never came to pass. But it highlighted a major underpinning of the current anti-Google backlash: a decade-long fight over how search engines, which have become many peopleâ€™s primary gateway to the internet, should treat the websites they list.
More than any other infrastructure, search engines reshape the web in profound and often invisible ways. Itâ€™s a potentially frightening power, particularly when 90 percent of the market belongs to a single company. So itâ€™s understandable to ask Google to be impartial â€” but can a search engine, whose goal is ranking pages, ever be meaningfully neutral? If it can, should a government be in charge of regulating it? And if it canâ€™t, what recourse do sites have if Google decides to remake the web without them in it?
Websites have been fighting Google over search rankings almost since the company was founded â€” in 2002, a site called Search King sued Google over a suddenly demoted PageRank score. Courts dismissed that suit, and a handful of similar lawsuits didnâ€™t fare any better. But in the late â€˜00s, as Googleâ€™s power grew, one complaint started to gain traction. It came from Shivaun and Adam Raff, the creators of a price-comparison service called Foundem.
Foundem was whatâ€™s known as a vertical search engine: one that helps users sort through specific types of information, like the cost of an airplane ticket or television. Like many niche search services, Foundem got the vast majority of its traffic from product searches on Google and other engines. But soon after launch, Google tanked Foundemâ€™s ranking, pushing results far past the first page and slowing usage to a trickle. Meanwhile, results for Googleâ€™s price-comparison service (originally called Froogle, now Google Shopping) started routinely topping search lists.
Google said that Foundem was demoted because most of its content included links to other pages, indicating that the site offered little original content. But the Raffs took their case to the European Unionâ€™s antitrust watchdogs, arguing that Google had deliberately blacklisted them. And in 2017, the EU agreed with them â€” hitting Google with a massive $2.7 billion fine.
The EU censured Google because of its dominant position online. â€œWhen you get as big as Google, you become so powerful that the market bends around you,â€� EU antitrust watchdog Margrethe Vestager told The New York Times. But the Raffs donâ€™t see this as simply an antitrust case. In the late â€˜00s, they helped popularize the controversial concept of â€œsearch neutralityâ€� â€” which turned Googleâ€™s support for network neutrality, a well-known and popular idea, against it.
â€œSearch engines obviously have to sort their results by some kind of criteria,â€� Shivaun Raff tells The Verge. But Raff believes that search neutrality is a natural extension of net neutrality, which forbids internet service providers from artificially speeding up, slowing down, or blocking certain websites and services. â€œSearch neutrality is saying, apply your best guess at relevance,â€� she says, even for smaller search engines. â€�Donâ€™t artificially promote your own services in the search results and demote or exclude competing services.â€�
The Raffs didnâ€™t invent the idea of search neutrality. Search King made similar arguments in its lawsuit, and a 2008 Cornell Law Review article by professors Oren Bracha and Frank Pasquale laid out a more comprehensive theory. They argued that search engines were â€œessential bottlenecks of an emerging economic order,â€� and they proposed rules against overtly manipulating results â€” complete with a court that could examine search algorithms for evidence of it, without revealing companiesâ€™ secrets to spammers.
But the Raffs linked search and net neutrality just as the American Federal Communications Commission introduced its first official net neutrality rules, supported by Google and opposed by internet service providers like AT&T and Verizon. They registered the site SearchNeutrality.org, which defines â€œsearch neutralityâ€� using Googleâ€™s own net neutrality information page. In 2009, Adam Raff published an editorial in The New York Times. Search engines â€œare now as essential a component of its infrastructure as the physical network itself,â€� he wrote. â€œWill [Google] embrace search neutrality as the logical extension to net neutrality that truly protects equal access to the internet?â€�
Internet service providers and other net neutrality opponents latched onto that argument, but in reverse: they warned that if net neutrality rules were passed, regulators would inevitably extend them to the point of absurdity. As one think tank put it, if the FCC decided that all data should be equal, it might also declare that â€œall search results shall appear first!â€�
Google, unsurprisingly, dismissed the idea. â€œIt is hard to imagine what â€˜neutral searchâ€™ would even look like,â€� it wrote in a 2010 letter to the FCC. Google argued that internet service providers were a â€œfailed marketâ€� that offered very little choice to users. By contrast, any internet user could easily navigate to a different search engine â€” even if most of them didnâ€™t do so.
Search neutrality was also less than popular among internet policy enthusiasts, even those who supported net neutrality. Techdirt founder Mike Masnick called the whole concept an â€œabsurdâ€� term â€œcreated by companies eager to bog Google down in bad regulation,â€� and he devoted several posts to deconstructing it: â€œGoogle might stop violating â€˜search neutralityâ€™ if anybody knew what that actually meant,â€� he quipped in one headline.
If an ISP violates net neutrality by throttling a streaming video service, you can use the video quality as a concrete reference point â€” and if the company stops throttling, it doesnâ€™t really affect other sites. Search engines, by contrast, are subjective and zero-sum. It can be hard to judge whether one website is â€œbetterâ€� or more relevant than another, and if you improve one personâ€™s site ranking, youâ€™re implicitly down-ranking someone else.
Some search results are partly guesswork: if a user searches for â€œapple,â€� Google has to decide whether they want the tech company, the music label, or the literal fruit. And thatâ€™s not even getting into personalized search results, which factor in things like earlier searches and location â€” and can help potentially shape which perspectives users see.
Law professor James Grimmelmann, who wrote a lengthy critique of search neutrality in 2010, also pointed out that people expect Google to down-rank lots of spammy, low-quality websites. Deciding whatâ€™s â€œspammyâ€� is a value judgment that will lock some website operators out in the cold, even when it improves the overall experience for users. â€œSystematically favoring certain types of content over others isnâ€™t a defect for a search engine â€” itâ€™s the point,â€� he wrote. â€œSearch inevitably requires some form of editorial control.â€�
Shivaun Raff believes that some of search neutralityâ€™s critics were simply attacking a straw man. â€œIt isnâ€™t saying that you have to rank the results in alphabetical order,â€� she says. â€œâ€˜You know, we canâ€™t have everyone ranking firstâ€™ â€” no, you canâ€™t! Thatâ€™s a really stupid version of the definition.â€� The Raffs argued that if Google was serving Froogle results, it clearly hadnâ€™t just made a value judgment against price-comparison search engines. In fact, Google had internally pursued an â€œaggressiveâ€� attempt to put Froogle results at the top of the page, despite apparently having problems with the service. More generally, it was switching over to a â€œUniversal Searchâ€� system, which blended links from other Google services (like Google Books or Googleâ€™s local business directory) into the â€œorganicâ€� results.
The EU decision over Foundem proved that regulators can punish a search engine for deliberately manipulating results â€” even if itâ€™s using the language of antitrust. Yelp renewed a similar EU complaint against Google in May, raising the possibility of another fine. And last month, the EU reportedly began asking other niche search engines if theyâ€™d been demoted by Google.
American regulators havenâ€™t gone nearly that far, though. The Federal Trade Commission investigated Google for potential bias in its search engine, but it never made a formal case against Google. While some FTC employees disagreed with that choice, even they didnâ€™t recommend an anti-competitive bias complaint â€” they focused on Google copying search results and striking exclusivity deals. An internal report said that Google had harmed â€œmanyâ€� competing search services with its ranking changes, but that Google could credibly argue it was improving usersâ€™ search results.
Recently, antitrust reformers have pushed the FTC to take harm against competitors more seriously, even when it seems to help consumers. And people are looking at Google more critically in general. Thereâ€™s been serious talk of limiting many Google services, not just regulating its search engine â€” including discussion of breaking up the company.
And in 2015, former FTC advisor Tim Wu â€” who had previously defended Google from search bias complaints â€” changed his mind, saying Google was actually offering â€œdegraded and intentionally worseâ€� service by favoring its results. Yelp had shown Wu a browser plug-in called â€œFocus on the User,â€� which altered the special Google-based recommendations (known as the OneBox) for local business searches. It changed the OneBox results, which were linked to Google+ and Google Places pages, to the results from Googleâ€™s organic search engine, which were normally pushed further down the page.
In a study that Wu helped author, participants searched for local businesses either with or without the plug-in. People using the plug-in clicked through to results at higher rates â€” suggesting that those results were more relevant, and therefore better. By using Googleâ€™s own results as a baseline for â€œunbiasedâ€� search, and directly measuring user behavior, the plug-in helped critics make a concrete argument against Googleâ€™s search practices.
The term â€œsearch neutrality,â€� however, hasnâ€™t gotten much traction as a distinct legal principle. â€œI do think that antitrust is up to the task. I donâ€™t think we need new laws or regulation,â€� says Luther Lowe, public policy head at Yelp. At this point, he says he avoids the term â€œsearch neutralityâ€� when he talks to tech policy experts. â€œThere was this debate ten years ago, and Google sort of strangled it in the crib.â€� Instead, Yelp has promoted â€œFocus on the Userâ€� with an advocacy website describing the potential effects on users if Google can smother competition.
But the idea of neutrality â€” not just competition or relevant results â€” has flourished in the conservative backlash against Silicon Valley. A 2018 documentary called The Creepy Line, for example, supported a radical version of search neutrality. It used complaints like Yelpâ€™s and Foundemâ€™s to argue that if Google had manipulated its results for commercial purposes, it probably did the same thing for political reasons, including a supposed attempt to rig the 2016 presidential election.
Like search neutralityâ€™s harshest critics might have predicted, The Creepy Line could describe examples of bias, but struggled to explain what ideologically unbiased search results would look like. When I spoke to one of its subjects, longtime Google critic Dr. Robert Epstein, he suggested making Google deliberately devote â€œequal timeâ€� to positive and negative results involving political candidates â€” effectively, Google shall make all viewpoints appear first.
Raff says search neutrality could theoretically apply to politics; if Google manually demoted the conservative news outlet Breitbart for political reasons alone, she explains, that would violate the principle. (â€œIâ€™m reasonably confident that no one at Google is doing that,â€� she says.) But it would be â€œabsolutely acceptableâ€� to down-rank it for being untrustworthy or otherwise low-quality, as long as the search algorithm applied those rules consistently to all sites.
According to leaked discussions, Google employees have weighed the possibility of making â€œopinion blogsâ€� including Breitbart less prominent than more traditional media outlets. But these ideas appear to have been shot down, and Google has resisted publicly changing politically charged search results, even when theyâ€™re downright factually unhelpful. In 2016, it initially refused to intervene when the first result for â€œdid the Holocaust happenâ€� was a page called â€œTop 10 reasons why the Holocaust didnâ€™t happen,â€� only reversing course after extensive public shaming.
An automated search system doesnâ€™t necessarily produce neutral-looking answers, and itâ€™s trained on distinctly biased human input. In 2006, for instance, Amazonâ€™s search system asked users who searched for â€œabortionâ€� if theyâ€™d meant to type â€œadoption.â€� According to Amazon, the algorithm had noticed that many people seeking abortion resources often looked up adoption, too, while lots of other people were solely interested in adoption. Amazon manually removed the suggestion, overriding a political statement that was obvious to a human, but meaningless to a machine.
Yelp has used the newfound interest in political bias to promote its antitrust complaints. But despite his pointed criticism of Google, Lowe thinks that Trumpâ€™s tweets â€” and many similar claims â€” are nonsense. â€œPolitical bias is not a thing, I donâ€™t think,â€� on Google, he says. â€œI donâ€™t buy that.â€�
Today, Grimmelmann thinks the entire debate has evolved for the better. â€œI think the discussion split in a way thatâ€™s helpful,â€� he says. Businessesâ€™ complaints can be covered under antitrust rules, where â€œyou can make a good argument against Google without having to directly attack questions of neutrality.â€�
Meanwhile, the growing importance of Facebook, Twitter, and other social networks has broadened the philosophical debate. â€œI donâ€™t think the concerns about search neutrality have gone away. Theyâ€™ve just merged with similar related concerns people have about multiple online platforms.â€�
Facebook and Twitter arenâ€™t primarily search engines, and their biggest controversies involve banning users. They tend to absorb serious competitors who might complain about anti-competitive behavior, instead of attacking them. But theyâ€™re criticized in ways that echo older debates. Lawmakers have repeatedly grilled internet companies about whether theyâ€™re â€œneutral platformsâ€� â€” although the term is difficult to define, let alone enforce, on a sprawling web platform.
The legal arguments are muddy. But theyâ€™re gesturing at issues that have only gotten more important. A few companies have become gateways to the internet, and becoming less visible â€” let alone being locked out â€” can have painful consequences. (Facebookâ€™s ever-changing algorithm has built and buried several media operations, most recently the digital news outlet Mic.) Even if a platform canâ€™t be neutral, scholars and policymakers will have to decide whether a non-neutral algorithm should push its users away from time-wasting or actively harmful media â€” and then decide what that means.
There are lots of bad examples of bias, and a lot of bad legal arguments for eliminating it, says Grimmelmann. â€œBut at least itâ€™s about questions of, who controls the public square? What voices should be heard in a democracy? How do you sample among competing voices? Those are deep, hard normative questions,â€� he says. â€œAnd at least sometimes, the public debate actually engages with them.â€�