Skip to main content

The future of online search

John Batelle: "Search is one of the hardest computer science problems in the world."


Which search engine do you use the most?
Ask Jeeves
or View Results




Technology (general)

LONDON, England (CNN) -- If anyone can guess what is in store for Web searching, it is Wired magazine co-founder John Batelle, who has spent most of his career as a technology journalist trying to find the answer.

He has written a book about the rise of online search and spoke to CNN about his observations and predictions for Web search.

Here is an edited version of the full interview from his recent appearance on Spark.

CNN: Do you remember when you first realized how big Google could become?

JB: It was late 2001 and I was looking for a breakout. I wanted to know if there anything happening that was interesting and that would have significant implications? I stumbled upon Google. Three years earlier, I'd put the company on the cover of my magazine, "The Industry Standard." We hadn't realized the implications back then. In 2001, I started researching and became convinced that the story of search would lead to a broader understanding of the impact of the Internet technology, on our culture, on our business and on our government.

It's interesting that the largest businesses on the Internet -- whether it's Amazon, Ebay, Google or Yahoo -- are all search driven. The two largest customers of Google, in term of buying the advertising that you see on the side of the page or the top, are Ebay and Amazon, because you go online looking to buy something, right? Twenty five percent of the time people are looking for a commercial transaction when they go online. And it's extremely important to those businesses that they are found. And the way you are found is to list well in the result to the search.

Search is becoming a navigational interface. The way we use to navigate this vast knowledge, space that exists out there is much larger than anything we could jump up on the computer. This is the computer connected to the rest of the world. We need to find a way to understand how to make a sense of all that, and searches become our way.

CNN: Google isn't the only search business, but its name is synonymous with search. How has it done this?

JB: It's certainly not the only one. There were these companies, apart from Google, that were doing the same thing essentially. But the timing wasn't right, the technology wasn't right. The moment Google broke out, there were a number of things that happened. One of them was the bubble actually blew up -- pieces were all over the ground. But the public, the audience, us, we didn't stop using the Internet. People stopped making [it] on the Internet, lot of people lost a lot of money in the stock market, but the rest of us kept using the Internet. The portals, the Yahoos, were not worried about search, they were worried about holding you on their sites. They didn't want you to find something and go over to it. They want you to stay in one place and watch their ads. It turned out that their ads had very little to do with what you might be interested in.

Google's model, which is how they broke out, was that when you put your intention into that box, it would reorganize the page around your intention. If you put the word "minivan" in there, the page would reorganize the advertisements with regards to minivans. Whether there's cars or whatever would be right next to the results about minivans that Google served up. This was a very efficient and productive way of organizing and advertising in Google, who have made $6 billion in revenue this year.

CNN: Is the Google success story based on good fortune or good technology?

JB: A mixture of both. They were definitely the best at their jobs at the time. It's arguable whether they are now. But the fact is that they've got a brand, which is an absolute world beater. People are so devoted to Google because they had that moment when someone would tell them about it, usually a friend, and they went there and they put in something they were looking for -- and they found it. Or they didn't know what they were looking for but they had a vague idea. And the idea that you can use natural language, conversational words to navigate your way around something was completely new in the late 1990s and early into the new century. It was a new way of discovering things and at that moment we had tens of millions of new people coming on to the Internet and saying, "What's out here?" They were using search to find something, and Google was the answer. They paid attention to the right thing at the right time, and they had great technology.

CNN: The founders of Google, Larry Page and Sergey Brin, were undoubtedly tech-savvy, but were they also business-minded or did they have to become business minded to make their idea work?

JB: I think they were business-minded, but I don't think that they are typical businessmen. We've got to remember these were two extremely bright graduate students in computer science and mathematics at Stanford University, one of the lead universities in the United States. So we are not talking about people who had dreams of being great businessmen. On the other hand, they chose Stanford for a reason. Stanford is the place that [produces] some of the greatest names in the technology business. And this is why in the mid-'90s when they came, they had every knowledge that Internet companies would literally be coming out of Stanford every six months, and being funded and going on to glory, so they knew this was happening.

But they are not business people. They made quite a few statements saying they are actually very anti-corporate. They pride themselves on doing things differently, and it turns out they were right and the way they did things and the things they paid attention to were extremely profitable and have made incredible business.

The big test for them really has not happened yet. Getting from where they started to where they are now, they are in front of a very big wave with just the right product, and just the right business model. Where they have to go now is a territory that would require extraordinary business acumen, because they are competing with Bill Gates now, they are competing with Yahoo, they are competing with News Corp. They are in a world where they are going to have to cooperate and partner with world governments.

This is not, "Let's build a better mousetrap and watch it take off." This is a case of, "We've got a $6 billion business -- we are trying to get it to be a $60 billion business. And we have to elbow a lot of other people without getting them so upset." This is a different test. And whether or not these two founders are up to it is an open question.

CNN: What would others have to do to be the next Google?

JB: First, you have to create an innovation that makes people say, "I've got to use this, this is better than that." That is extremely hard. Search is one of the hardest computer science problems in the world, because basically we are trying to create artificial intelligence so that we can speak with our computer, they can understand us and deliver what we are looking for. That is equivalent to turning your computer into a very intelligent research librarian, which of course is the holy grail of computer science, to create artificial intelligence. So it's not easy, you know. And to make a leap beyond Google and create a better mousetrap requires computer science that hasn't been invented yet.

Secondly, you have to do search well. It's pretty easy to do it on a limited scale based on the technology we currently have. But if you want to do that as well as Google, you have to have hundreds of thousands of machines, and you have to have thousands of engineers and keeping that up in running in real time is very expensive. You can't just start a search engine and all of a sudden everyone goes to it, because what would happen is the computers you are running off would blow up. You have to have a scale infrastructure, which is quite expensive so there is that as well that gets in the way. The third thing is funding the next big thing.

CNN: What is the next big thing on the Web?

JB: The idea to create a semantic Web where everything is described not by one researcher and his team but rather by all of us as we root about the Web. I might say, "This is a picture of a seaside with a sunset," but someone else comes and says, "No this is a picture of a beach in Thailand," and another person comes along and says, "This is a picture of a place I like to go diving." And over time, this one object, and every object in the world gets thusly tagged, gets enough intelligence around it that it can be found no matter how you might ask for it, the brittleness problem is solved.

The idea is that we might get to the point where everything in the world of value is in the index correctly, is on the Internet and some way represented, whether it's your car, your child or whether it's a media object like a page or an audio file or whatever, or in this case a picture. And then you create these vast semantic attachments to everything and that becomes the seedbed for the next generation of search to crawl and make sense of.

That's a long way off but we are starting to see any number of applications that are making this possible right now where people are starting to tag things and create engines based on those taggings and we are just seeing the beginnings of it. Whether it will be the next great breakthrough in search remains to be seen but it is a promising development.

CNN: People are getting nervous because, what you're looking for can be monitored. Do you think people are viewing it with a degree of suspicion?

JB: I think this is quite possibly the largest roadblock both to Google and anyone else involved in this space, which is that if it can be known, it more likely will be known. What we know now is everything that we do can be known, every footstep that we take on the Internet, using a search engine or once we've used a search engine and where we've gone -- wherever we've gone and whatever we do there -- can be recorded and it can be recorded by one central recorder. This is in the case of some of the tools now that Google and others give you, called the tool bar, which watches everywhere you go. And why they do that is that so they know everywhere you've been on the Internet so they can give you better search. "Oh, I see, he's been to this site before, so perhaps I should make that site higher in the results." That's called personalized search.

It seems like a service to you, right? But all the information about where you've been doesn't live with you, nor do you control it nor can you control who has access to it because of course the government or Google itself -- or it doesn't necessarily have to be Google, it could be Microsoft or Yahoo or anyone else -- could do whatever they like with it without telling you. In fact, it's part of their privacy policy that they will not sell this, they will not give it to third parties unless the laws of that country require them to.

Now we've seen the laws of various countries and the law of China is very different from the law of Britain and very different from the law of the United States. In fact, the law of the United States is a lot more like China. It turns out that after 9/11 we've passed a law called the Patriot Act which gives the government right to riffle through all of our stuff on line without our knowledge, which frankly as a journalist terrifies me. Not that I don't think we should have tools to combat terrorism -- I do. I just think we need a little sunlight.

We are creating these vast records of everything we do. It's very difficult to pull those records together and make sense of them but that's exactly what Google's job is. That's why they are in business. Are they doing the work of a potentially corrupt or abusive government? Yes, you can't deny that they are. The question is will they make a stand against it at some point and/or will they make it transparent to me so I can see what they know about me and I can edit it or decide that I wish to become anonymous.

CNN: Google has apparently set itself up as an anti-corporate company, but in a way, they are the biggest corporate company in the world. How does this tension work in reality?

JB: It's a huge tension when their goal is to provide you the best services, the most useful, efficient, productive, amazing, astonishing services. I believe they really want to do that. I've met so many people at Google and they really are well intentioned. But in order to do that they have to build the exact same kind of service that those with ill intentions might want to take advantage of. That's a tension that they are going to have to navigate.

CNN: Is there anything on the Web that you haven't been able to find?

JB: Absolutely. Microsoft did a study about a year and a half ago that claimed that only 50 percent of any search actually completes what people are looking for. That is quite a high percentage. There's lots of things I haven't found on the Web I've been looking for, so when that happens I pick up the phone and call a person and then they just tell me one thing and I go back and do the search and there it is. Whether it's just you know the name of an obscure regulation, or whatever it is, or perhaps I just don't know how to ask the right question. I think the hardest thing in the world is how to ask the right question, it is not how to find the answer it is how to ask the right question.

-- John Battelle's book, "The Search: How Google and Its Rivals Rewrote the Rules of Business and Transformed Our Culture" (2005), is published by Nicholas Brealey Publishing in the UK and Portfolio in the U.S.

Story Tools
Subscribe to Time for $1.99 cover
Top Stories
Get up-to-the minute news from CNN gives you the latest stories and video from the around the world, with in-depth coverage of U.S. news, politics, entertainment, health, crime, tech and more.
Top Stories
Get up-to-the minute news from CNN gives you the latest stories and video from the around the world, with in-depth coverage of U.S. news, politics, entertainment, health, crime, tech and more.

© 2007 Cable News Network.
A Time Warner Company. All Rights Reserved.
Terms under which this service is provided to you.
Read our privacy guidelines. Contact us. Site Map.
Offsite Icon External sites open in new window; not endorsed by
Pipeline Icon Pay service with live and archived video. Learn more
Radio News Icon Download audio news  |  RSS Feed Add RSS headlines