Web Masters Episode #70: Erik Sleberg


MetaCrawler - Wikiwand

Erik Selberg:

Really, the first web search engines would download the web and then find them. That was it. There was no SPAM. There was very little unsaved content. You know, for example, “Oh, was there a whole lot of porn, back in the day?” Well, occasionally, people would find stuff that grad students had, kind of, hidden away. But, by and large, we know. The biggest problem with early-day web search engines was occasionally, people would find things. Like, at U-dub, we had a website that provided all the census data, which usually filled up people’s hard drives, if they did that.

Another one is somebody, a grad student, made a game that’s called Hunt the Wumpus, where you move from room, to room, to room, and if you hear a wumpus, you’d try to shoot an arrow into the right room. And, otherwise, if you walk into the room with the wumpus, you die. So the web spiders would play Hunt the Wumpus. They really don’t know what’s a real page versus a game. So those would be, kind of, the humorous problems you’d have with the early days. But the model of Download the Web, a decently-powered, but still like desktop-class computer, was what we did.

Aaron Dinin:

Yes. Web search has come a long way from the early days when search engine spiders would get stuck playing Hunt the Wumpus. In fact, these days, most people don’t give a search much thought. We go to Google, type what we’re looking for. And, before we’re even done typing, Google seems to have figured out, as if by magic, exactly what we wanted to search for. That wasn’t always the case. There was a lot of trial and error and iteration in the search industry before Google claimed its throne. And the man you just heard talking was one of the early experimenters. His name is Erik Selberg. And he created Meta Crawler, the early web’s most popular meta search engine. Are you ready to hear the story? Let’s get dialed in.

[INTRO]

Aaron Dinin:

Welcome to Web Masters, the podcast that explores entrepreneurship by talking with some of the internet’s most successful and impactful innovators. My name is Aaron Dinin. I’m a serial entrepreneur. I teach Entrepreneurship at Duke University. And I, like all of you, often use web search engines, well, not really search engines, basically just Google. But, like some of you, I also remember a time before Google, when search wasn’t nearly as streamlined.

In those early days, we’d all bounce around between different search engines, as we tried to find things online. And I’ll never forget the moment when I was looking over my big brother’s shoulder, as he opened up a search engine, I hadn’t seen before. It was called MetaCrawler. “What’s this?” I asked him. And he nonchalantly replied, “What? MetaCrawler? Oh, it’s a search engine that searches other search engines.” That’s right, a search engine that searches other search engines, otherwise known as a meta search engine. It’s a concept that’s totally unnecessary in the post-Google takeover of the world. But it was a godsend on the early web. In this episode, we’re going to hear all about it. But first, we’re going to hear about the company helping make this podcast possible.

Web Masters is being brought to you, thanks to the generous support of [Latona’s 00:03:19]. Latona’s is a boutique mergers and acquisitions broker. They specialize in helping people buy and sell cashflow positive internet businesses and digital assets. And what qualifies as an internet business? Basically any type of web-based company, SAS apps, Amazon FBAs, e-commerce stores, content websites, and even domain portfolios. Basically, if you’ve got a business that’s generating revenue, using the internet, you’ve got yourself an internet business. And if you’re interested in selling that business, you’ve got a great partner in Latona’s. The expert team at Latona’s specializes in helping internet entrepreneurs sell their companies. Oh, and they also help people buy companies. In fact, if you’d like to see a listing of all the companies they’re helping sell right now, you can find it on their website. latonas.com. That’s L-A-T-O-N-A-S.com.

These, days, search is a core part of the web. But that’s only because the web is enormous with so many pages, nobody would ever be able to find anything without a good search engine. In contrast, the early web wasn’t like that. It was small and navigable, without search. But, as the web grew, finding stuff became harder. That problem is what interested people like this episode’s guest, Erik Selberg, who, at the time, was a graduate student at the University of Washington. And, even though search engines seem like an obvious solution to the growing problem of finding content online, that wasn’t actually the case. Remember that the early internet wasn’t just the web. Not that it is today. But, before the worldwide web existed, content on the internet was much more spread out across different protocols and networks and services.

Erik Selberg:

One of the main things that the web did is it took all of these kind of, some legit, but a number of also-ran protocols. So just wiped them out and said, “Okay, here’s how to find something. Here’s a standard thing,” that URL, which was really awesome. It’s like, “Here’s a standard way to identify a protocol and a host name and a location to find something. And here’s a way to have documents where you can have lots of URLs to just go and find things. And then here’s you know, HDP,” which was kind of the dominant protocol that came out with that. “Here’s the way to get it.”

So, suddenly, you could share things much more easily. You could find things much more easily. Every university kind of had their own system. They all went away. They all got replaced by the web. And so suddenly, because we’re all on a kind of a standard infrastructure of HTP, things like search engines become really easy and kind of the right thing to do.

Aaron Dinin:

So how did people find stuff back in the pre and early web days?

Erik Selberg:

I mean, at the time, here was what my advisor, Oren, was working on.

Aaron Dinin:

The, Oren Erik references here is, Oren Etzioni, who is Erik’s PhD Advisor, at the University of Washington, and eventually, his partner in MetaCrawler.

Erik Selberg:

He was a big planning guy. And planning is a robotics concept, where you have a task, and you’re trying to figure out, from your current state to the end state, what steps you have to do to achieve it. A common thing pre-surf, pre-internet is, “Oh, I really want to find a paper by So-and-So.” Okay, this is very academic. Today, with a search, and you’re done.

Well, back then, this was a doctorate thesis, by one of my peers, which would, you say, “Well, let’s assume that the person who wrote this paper is an academic. Okay. Then, let’s use this thing called fingering”… Kind of sounds sort of inappropriate, nowadays. But you could finger a email address to see if the person was at an institution. And Finger would say, “Yep. This email is a legit email.” So it would say, “Well, try fingering at U-dub, at MIT in Chicago. Oh, we got a hit at MIT. Great. Let’s see if this person has an FTP repository. Oh, look. We see something that says FTP colon machine name. So let’s list that FTP directory, and see if there anything that looks like papers. Oh, look. We find something that’s a .ps file. So, that must be a paper. So let’s grab that.”

So, for pre-search, it was actually a whole lot of software robotics planning, to try and actually get the paper. Whereas, nowadays, we’re like, “Well, simply have a spider download everything, have a link to it, dump it into index. And you’re done.” Sounds so simple nowadays. But, really, that’s where people were because everything was so fragmented. I mean, it could’ve been an FTP site. It could’ve been a waste site. Again, it could’ve been a gopher. The person, maybe, had information, and it was known as their profile, which you get by fingering. Maybe it’s in some other place, et cetera.

So, really, you have all these tools which are trying to deal with all of these different services. And they’re all created by grad students, too, just for fun, because we all don’t have anything to do. And you’re trying to unify them.

Aaron Dinin:

Okay. So the web comes along, creates a standard protocol for content. And how does that lead to web search, as opposed to that robotics model you were just describing?

Erik Selberg:

The web was couple thousand pages, and primarily it was .edu sites. It’s like, who has a .com site? Well, that’d be DEC. So the tech houses, back in the day, are like Deck and HP. There aren’t any businesses online yet, because there isn’t any customers, aside from grad students, and what not. So it was really a lot of .edu sites and some government sites. Then things started to grow. Okay. Well, once it starts to grow, you can’t remember all these sites in your head.

The first thing people did was they said, “Hey, let’s make some directories.” And, the most famous one that lasted was Yahoo, which came out of Stanford, which was, effectively, more of a wiki-style directory, where they tried to have various taxonomies. And you’d just, kind of, find stuff via Yahoo!. And it was kind of that critical mass. And it was easy to use, and it worked. So that kind of stuck.

The problem is, very quickly, the browse model also didn’t scale. So a number of people who’d been working on various search technologies, which is not so much web search, but computer search. “Let me find files on my computer system.” They said, “Hey, let’s just apply the same technology to the web.” Quite literally, this is how we thought. And, at the time, it worked. “Let’s just download the web to our computer,” meaning our desktop computer, “And index it, and then provide a searchable interface to that.” And this seems crazy nowadays. But, back when the web was order of magnitude like, you know, a million pages, this was very doable. And, in fact, that was what the first search engines were.

Aaron Dinin:

And that, obviously, wasn’t scalable. What were some of the problems those early search engines ran into?

Erik Selberg:

The model of download the web to a decently-powered desktop class computer was what we did. Now, the number of problems with this, as the web was really on an exponential growth rate, as opposed to all of our desktop computers, which were the very best of 1991, but not growing nearly as fast. They kind of overwhelmed things. The other thing, which we all didn’t realize back in the day, is this thing called operational excellence.

So, when you’re a grad student, you put something up there, and it kind of works. And, if it crashes or goes down or whatever, you’re a grad student. Who cares? You’re not going to… Hey, this isn’t your job. You’re busy, just kind of playing around to try and see, “Well, gee, is this interesting enough where I could write a paper about it?” You’re not actually trying to provide something that’s a fundamental utility to the world.

The other thing, which we all didn’t realize, back in the day, is this thing called operational excellence. So when you’re a grad student, you put something up there and it kind of works. And if it crashes or it goes down or whatever, you’re a grad student who cares, right. You’re not getting paid. This isn’t your job. You’re busy. Just kind of playing around to try and see. Well, gee, is this interesting enough where I can write a paper about it. You’re not actually trying to provide something that’s a fundamental utility to the world.

So, a number of the early search engines… You know, there was WebCrawler, out of University of Washington. There was Lycos, out of Carnegie Mellon. There was Inktomi, out of Berkeley. These were all grad student ware. Things would go down all the time. WebCrawler in particular, the Achilles heel was Brian initially used his desktop. But that became overwhelming so then he used his office mate’s desktop to do all the work. The office mate kind of got cranky about it, and had to do his work. So they eventually had to try and figure out funding and all that stuff.

So, grad students, we were all learning, the hard way, how to keep our services up because search engines became very quickly a critical part of the web to find anything.

Aaron Dinin:

Erik is pointing to an interesting phenomenon, here, that’s easy to overlook with everything else that was changed by the growing popularity of the internet in the worldwide web. Specifically, as the web became more popular, people began accessing websites 24 hours a day, seven days a week. The organizations operating websites, whether those were grad students or companies or churches had to transition from serving people during quote/unquote “normal business hours” to supporting people 24/7. That requires lots of additional resources.

Erik Selberg:

That was like a new thing for all of us. We were like, “Oh, wow. We have to keep this running seven by 24? But we’re grad students.” Right? That’s… And even companies were like, “What? We have to keep this running seven by 24, and people care? Really? We can’t just take it down, just randomly to do something?” “Nope. Now, of course, you have to do this.” But, back in the day, this was a new thing for us. We never really thought through what actual operations meant.

Aaron Dinin:

Of course the eventual result of search becoming a critical part of the fast-growing consumer internet was that businesses would eventually come along to replace graduate students, as the operators of search engines.

Erik Selberg:

Companies start to emerge. There was Infoseek, one of the early ones, of course, Alta Vista had a deck which started off as a showcase for the biggest deck computers you could buy. Google’s also [inaudible 00:13:29] ware, until it turned into a company.

Aaron Dinin:

You can hear all about one of those companies building the first truly consumer grade internet search engines in the very first episode of Webmasters, where we got to talk with Louis Monier, who created AltaVista inside of DEC, D-E-C Digital Equipment Corporation. You can also learn about Steve Kirsch building the first search engine that began as a business, rather than an academic experiment, when he launched Infoseek. That’s Webmasters Episode Number 58.

But before consumer-grade business-focused search engines existed, with their more powerful servers and much, much, much bigger budgets, especially compared to what grad students had, Erik developed a particularly innovative way overcome the general unreliability of early search engines.

Erik Selberg:

MetaCrawler was my invention and my project for my doctorate thesis at the University of Washington. MetaCrawler was created, actually, as a secondary offering, which is I didn’t actually create my own search engine. What I created was something that could query all the other search engines and then merge the results. And the main purpose of MetaCrawler was, at any given time, two or three of all these search engines are down. This is back in the day, when the 404 page was the most common page on the internet. Things would just be down, or things would move and you’d click on this link, and you wouldn’t get anything. So, MetaCrawler was this layer on top of search engines to make them more reliable. That was it. Right? And it turned out just doing that was a hard enough project that uncovered lots of interesting things on its own.

So, that’s when we started to do what does parallel search look like? How do you combine different algorithms? How do you merge results? How do you do different query types like, “How do you phrase search in the meta engine when they all don’t support it natively?” Back in the day, phrases? What’s that? I mean, everything was primarily an and search. So this word, and that word, and that word. Again, it looks very much like computer search. So we just used the tools that we had available, as opposed to doing anything fancy.

That was really kind of the genesis of where MetaCrawler came. And, again, it’s trying to fill that need of, everything’s being run by grad students. We don’t know what we’re doing. So here’s some automation on top of that to make it a little bit easier to use.

Aaron Dinin:

So you weren’t so much solving the search problem. You were solving the problem of search engines being unreliable. And I’m guessing, judging by MetaCrawler’s success, people really needed that problem solved on the early web.

Erik Selberg:

Yeah. Exactly. Back in the day, when the web is growing, it’s the get another search engine with the really convenient URL of metacrawler.cs.washington.edu. But people would use it. They’d bookmark it and go… We had to get approval, from a number of people at U-dub, to get metacrawler.com. That was like, “Oh my God. You can’t get a .com address at a .edu.” I mean, bureaucracy. So the people would know that URL. They would find it. And a lot of my early testers were from Australia and New Zealand, which I thought was fascinating. I know this because, back in the day, the mailto tag was introduced. So you could click on a link and it’d open up a window to mail. And this was when things were safe. You didn’t have SPAM, and bots, and people seeing your information.

So, the MetaCrawler page, oh, by Erik Selberg and Oren Etzioni. Both of those were mailto tags. And, since I was listed first, people could send me feedback, which they did. And a lot of it was great. I would put up a new verse with MetaCrawler, with mostly work. But I’d go home for the day, around five, 6:00 p.m. Pacific Time, which is right when a specific, particularly Australia and New Zealand, is getting up. And they would discover what I’d screwed up, and then they would probably tell me what I screwed up. This was the early days of learning about customer feedback and really understanding what it means to be operationally sound.

Back in the day, early precursors of operator selective protocols, I was in a 14-person office. So, at the time, MetaCrawler was now on four deck boxes. You could kind of call in, because usually somebody would be in the office, except at like three in the morning. But, even then, we’re grad students. “Hey, can you please push the number three box reset button?” So my protocol was, if it got stuck, which it did, just reboot the computer. Just kind of reset. It would reboot it. MetaCrawler would come back up. We’re good to go. Really basic stuff. But it worked.

Aaron Dinin:

As you can probably imagine, asking your fellow grad students back in the office to reboot the servers wasn’t a viable long-term strategy for a web service that was exponentially increasing in popularity. As Erik quickly outgrew his meager grad student resources, he had to start thinking about commercializing what he’d built. However, at the time, commercializing web software built inside universities was a new phenomenon, and came with its own series of unique challenges.

Erik Selberg:

All the universities around the country don’t understand this web thing. And all of them do have this organization that does deal with commercializing stuff. However, it’s geared for two things. Number one, it’s geared for books. So copyright stuff. So you publish a book. Lots of professors do this. Students do this. They understand that process.

Number two, it’s geared for royalties, typically medical. So you create a new protein, you create a new drug. So get every research hospital, put stuff out there. And university technology transfer groups are really good at going, “Okay. We will help you go through the patent process,” because that’s what matters, to patent something. And then you sell royalties. They don’t know what to do with software. They’re like, “Is that trademark? Is that copyright? patent?” And we’re like, “No, it’s software. You sell it.”

Some people understood selling software. Like, again, you used to be able to buy a copy of Microsoft Word. You go to the store. You buy Microsoft Word. You have it. Nobody understands what a service is. Like, you just run it. “Okay. So what gets transferred to a company?” Nobody knows. And so my predecessor, [Brian 00:19:50], he got out lucky. He sold WebCrawler to AOL for a million bucks. And he got away with it because he was like, “Hey, I did this thing on my own time, not part of my doctorate work. Some company wants to buy it. Can I sell it?” They’re like, “Is it [inaudible 00:20:05]?” “No, it’s software.” “So, it’s copyright?” “Well, no. It changes.” “Who wants to buy it, AOL? Who are they? Yeah, okay, this does not seem worth our time. Let’s sign off. You’re good. You enjoy. Good for you. Get some pizza money.”

Price tag comes out, a million bucks. They’re like, “Okay, we screwed that one up.” I was not so lucky. So now, all the universities understand this game, and that there’s a lot of money in software and services. And so, again, a couple of the early folks were able to kind of make out pretty well. Most of the rest of us, the universities came in and said, “Hey, we’ll help transfer your software to some company. And we’ll take a big cut out of that.” It’s the way things ended up working.

Aaron Dinin:

And so, you did eventually bring it out of the university. Right? Though you didn’t make as much money as you, maybe, would’ve liked. Is that what I’m hearing?

Erik Selberg:

I mean, I got a nice house out of it. But a number of us suddenly got paper rich, and then paper poor. So very few of us went paper rich to still rich. A number of folks didn’t. We had no clue what we were doing. We’re all experiencing something, and we don’t know what it is. And we think it’s permanent forever. And the answer is, no. I mean, people seem to forget the first .com bust of 2000. It was real. I was there. I had good times.

Aaron Dinin:

Yeah. That’s, kind of what I’ve heard. But, before the proverbial wheels came off of, well, the entire internet, what was it like going from a graduate student project to building a tech company?

Erik Selberg:

For MetaCrawler, we followed a pretty common path, which is a lot of venture capitalists started trolling the computer science building, looking for stuff to invest in. So it’s happening at Washington, at Stanford, at Berkeley, at CMU, all the places, because the VCs were like, “Oh wow, this internet thing is going to be huge. We’re seeing stuff come out of university. Let’s get in early.” And so they were just throwing money at anything. And so what happened with crawler is we put together MetaCrawler and actually three other technologies that other grad students had created. There was a shopping engine. There was a homepage finder. And I forget the fourth. The homepage finder and the fourth one never made it out of U-dub. It’s like, “Okay, we’re licensed.” But nothing ever happened with it. The shopping engine turned into a kind of shopping link on the second page of the Excite@Home commerce engine, because Excite@Home eventually bought the company we created.

And then MetaCrawler was still [inaudible 00:22:45] site, doing stuff. Company that we founded, called NetBot, we were trying to figure out how to monetize. We had customers. But we didn’t know how to make money because, if you were a primary search engine, you just put ads. And that was straightforward. But we’re like, “Well, are we putting up all of their ads?” Having a page with 12 ads, just doesn’t seem to make a whole lot of sense. And these were back, when they were all like the one-inch by four-inch banner ads. So it, kind of, swarmed the page.

Aaron Dinin:

And, judging by the fact that MetaCrawler didn’t become the size of Google, I’m guessing that didn’t work out very well. Right?

Erik Selberg:

So we were really struggling on how to make money. And eventually we decided to know how to do this. And another company called GoToNet, which is kind of was a portal. They had an idea how to make money. So they ended up licensing the technology from the startup. And then, when I graduated, both I and my advisor went over to this company for a year or so, to kind of really understand how they could figure out how to make money. We had no idea. That’s really kind of how that happened.

But one thing I did do, I kind of recognized this, is I didn’t actually join the startup as a founder. I wanted to finish my doctorate. So it’s like, “Okay, I’ll be there for tech transfer, and I’ll take a bunch of shares. But I’m going to go back and finish. I don’t really know if this has enough legs.” And I also realized I was turning into peer operations. It’s like, “Yeah, I built a thing. It works. It’s great. I want to work on the next thing.” But, nope, I’m still stuck just fixing bugs and making it work a little bit better. And I’m like, “I don’t want to spend my life doing that.” I definitely wanted to finish, which I did a couple years later. And that was probably a good call, on my part, given where things went.

Aaron Dinin:

Are you surprised by that outcome? Are you surprised MetaCrawler didn’t succeed as a business?

Erik Selberg:

I didn’t think MetaCrawler would last. It actually lasted a lot longer than I thought it would. But once there started to be consolidation in the industry, meaning we went from like 10 search engines down to eventually three and it’s actually, now we’re down to one. It’s like, there’s really no need for a MetaCrawler. It’s like the operational excellence problem, solved. Relevance problem, pretty solved. So MetaCrawler’s not really providing anything new. So that value goes away, as the core search engines figured out.

Also, there’s not a whole lot of extra money to be made because you have to pay the core search engines for their service. Then you have to make money on top of it. And this is a razor-thin margin business. So I was like, “Yeah, between the money and the utility, it’s not going to last.” So that was kind of one of the reasons why I’m like, “Yeah, I need to let this one go. It’s great for now, but it’s not the long-term thing.”

I think so many people kind of stuck with their baby until it’s too late. The hard thing we have to kind of understand for a piece of technology, is this a five-year piece of technology, 10-year? Or is this like a 50-year piece of technology? Most technology’s in the five-year camp. And so you don’t want to bank on this for life, if it’s really going to become obsolete and supplanted in five years.

Aaron Dinin:

What you just heard Erik talking about is a critical entrepreneurial lesson. Hit the rewind button and listen to it again. Most new technologies are what we might call gap-filler technologies. They’re short-term solutions to problems or, more accurately, I’d say they’re solving problems in inefficient ways. A lot of the entrepreneurial solutions people develop ultimately fall into this category, including MetaCrawler. Basically Erik looked at the search engine problem and said, “I can solve this by building a meta search engine that aggregates results from across all the problematic search engines.” Sure, that was one way of solving the problem of unreliable search engines. But there were other ways that were ultimately better you know, like the strategy another grad student named Larry Page, along with his buddy, Sergey Brin, were developing a bit further south of Erik.

Erik Selberg:

Speaking of Larry, he and I hung out at the web conference, dub-dub-dub 97 in Santa Clara, which was miserable because they had problems with the budget. So all of our lunch food was like spoiled with bad mayonnaise, and whatnot. So Larry was just going around complaining about everything. And he was just unhappy. And I’m like, “Okay, so you’re going around saying you’ve got something much better than the lab in Stanford. Well, why don’t you do something with it and showed off to the world, punk? I mean, all these people presenting papers, they’ve done the work. That’s why they’re on stage and you’re in the audience with me, man. So why don’t you go do something?” And it turned out he did. So…

Aaron Dinin:

Yeah, I’d say he did. He definitely showed you. So can you talk about Google, a bit, and maybe the modern search industry? Did you ever think search would become as lucrative an industry as it is?

Erik Selberg:

I think we all knew it would be profitable. I don’t think we had ideas that it was going to be where Google is today, profitable. I mean, they’ve basically cornered the market. So I mean, at the time, we were trying to figure out how to survive with display ads, which were not terribly profitable. And it was really two things, turned it into the money press. One was GoTo, which Yahoo Acquired. And GoTo was basically a hundred percent paid listing thing. So they said, “Hey, we’re going to create a search engine where people want to pay for being there.” Now they didn’t really have any guidelines around what to do.

So it was basically the more you pay the higher you show up in search results, when there’s a hit. So Yahoo picked that up and said, “Wow. This is great, and this is how we’re going to make an ads business that adds relevance and sends sponsored content, as opposed to these display ads,” which everybody… They weren’t clicking on… These were all the people who still hadn’t created an ad blocker to get rid of them. So, day of the dawn of the ad blocking technology.

And then, what Google did, is they said, “You know what? We love the technology of having paid-for links. But we’re actually going to get rid of links if nobody’s clicking on them.” Because, it turns out if you bid for a link… “You say, “I’m going to just bid $20 to be the top result.” If nobody clicks on you, you actually don’t get charged. So what Google says, “Hey, we’re going to make people optimize for our revenue. So there has to be a click. We don’t care what you bid. If nobody’s clicking on it, you’re wasting our time.” And so that was the genesis of the ads model, where it compelled advertisers to make relevant ads that get clicked on, which means they’re going to use their budget. And Google’s going to rake in all that money. And that was the genius of it.

And then, here we are today, where Google Search is still bankrolling the entire alphabet empire, and doing quite well. I don’t think I would’ve said Google’s going to be one of the top 10 companies in the world, back in the day. But, Search was going to be lucrative. Yeah.”

Aaron Dinin:

And what do you think about there basically being only one player in the search industry? Do you think that’s a good thing or a bad thing?

Erik Selberg:

It’s effectively one with a challenger that’s got deep pockets, which is Microsoft. And that’s reasonably okay. A lot of markets end up look like that, where there is the dominant player. And then there is somebody who’s trying to keep the dominant player honest. You look at Intel and AMD, same general dynamic. So, generally, I’m okay with that. And the other thing which has happened is… And this has been true for about a decade, now. In the search world, both teams are really working hard to get like a 0.02% improvement in relevance. I mean, it’s really minor. I mean, they’re still working. They’re still publishing papers, and they have to. But you’re not seeing the leaps and bounds improvements you did during the early days of the web.

The places where search for monetization have been changing, and where the interests are, have been changing because search has effectively turned into a solved problem. I’m not trying to make light of it. You know. And everything’s solved. But, for what most people need, it kind of works. And so this is why you see a lot of the investment that both Google and Microsoft are making are not in core searches much, but in different niches or things like the Cloud, things like mobile, et cetera.

Aaron Dinin:

So has Google basically taken the entire web search opportunity, unless of course you’re a company with a pocketbook, the size of Microsoft’s? In other words, is there more innovation left in web search, or is Google going to continue to own it forever?

Erik Selberg:

I imagine that’s going to continue. If we were in a different country, I might have a radically different answer. If we’re in Germany, say, I might say, “Yeah, they’re both crappy. If I’m in Kazakhstan, I’m like, “Oh, we have nothing.” So I do think there is still opportunity in kind of the non-first-tier markets from the U.S. point of view.

When I was at Microsoft, MSN Search, the time now being, worldwide was number five. Google was number one. AltaVista was number two, Yahoo!, number three. And number four was NAVER. Most people had never heard of NAVER. That’s because it’s the dominant, by like 98%, or something like that, search engine/browser in South Korea. So, if you’re in South Korea, you’ve got NAVER. But they’re nowhere else. So, just all of South Korea was enough to make us number five worldwide. You’ll have things like that, where a local player can dominate the market because they catered to the local interests. So that’s still something other places the world have.

Aaron Dinin:

There, you have it from one of the earliest pioneers of web search. Maybe don’t bother picking a fight with Google for web search dominance, at least not in a large market. But you might have a chance going niche. Hey, maybe you could create a podcast search engine that always writes Web Masters, number one in every category. Just you know, throwing that out as a suggestion. But if you don’t have time for that right now, there are other things you can do to help us out. Start by subscribing to Web Masters on your podcasting app of choice, so you’re sure to get the next episode as soon as it’s released. While you’re there, be sure to leave a nice review. And don’t forget to share this episode with a friend, or stranger, or really anyone you think might like it.

If you have any thoughts or feedback about the episode, you can find us on Twitter. We’re @webmasterspod.

I’m on Twitter too @aarondinin. That’s A-A-R-O-N-D-I-N-I-N. I also write lots of articles about business startups and entrepreneurship. You can find them on my website. It’s aarondinin.com.

A quick thanks to this episode’s guest, Erik Selberg, for taking the time to share his story and the story of MetaCrawler.

I also want to thank our sound engineer, Ryan Higgs, and our sponsor, Latona’s. Remember if you’re interested in buying, or selling, an internet business, you should start by checking out latonas.com.

And if you’re interested in more great stories about web entrepreneurs, and the things they’ve built that have shaped our lives, then stick around for the next episode of Web Masters. Until then, it’s time for me to sign off.