(Wednesday, November 23)

Google Base and the loss of need for Web Spidering

So I keep up with a decent amount of search engine optimization techniques and one of the best places on the web to do that is WebMasterWorld.com. The site is owned (?) and ran by Brett Tabke, who is a large icon in the SEO realm already. A few days ago Brett decided that due to the ever increasing traffic from rogue spiders crawling his forums en mass (and thus increasing his already large bandwidth) costs for the site, he decided to block all spiders from crawling the forums. When I say all spiders, that also includes Google/Yahoo/MSN and other large search engine spiders. While I am sure this WILL cut down on the bandwidth in one fell swoop, it will also result in the site being dropped from all the search engines on anything it ranked for using those pages that were previously being crawled. I personally did not understand why he would be giving up all those spots, I thought what if he had another method to submit his content to the other major search engines without having them crawl the content. Then it hit me. He does. At least for Google he does. He could break the content up into bite sized chunks and submit it to Google Base. Now, I am completely speculating here (which also includes the Google Base submission as speculation) but if he did this would bring up 2 interesting and scary ideas: 1.) If this were to be given to Google Base and indexed, this for one gives G the monopoly on his data. This, in my opinion, is a VERY bad thing. Yes it might be nice for his users (what if Google designs a certain page in Base for his subscribers to utilize for searching his site only?) This does help his user base find the information they need faster, but this lends to a scary idea. What if webmasters package up their data and send it to Google (via Google Base)? This eliminates the need for Google to crawl the site, but it also gives Google a monopoly on the search ability of the end user on that data. I, from a search engine user/consumer perspective, do not like that idea. As soon as we eliminate the ability for other search engines to compete then we all lose. Even if you don't utilize Y! or MSN, their inherent value of just existing in the same market is a major bonus to Search Engine users anywhere. It MAKES Google innovate. 2.) If he were to submit this information specifically via Google Base, this might also have the effect of Webmasters and other SEO individuals having to go to Google base to search Webmasterworld (thus increasing the visibility of the site to webmasters hopefully drawing their data into the fold). This goes back to #1. Also, there has been a lot of talk about Google and if they are going to give people (application developers from other companies) the ability to crawl Google base for their own means. This is a tricky situation for Google, if they collect all this data (submitted by users not crawled) and they lay claim to it (and don't let people crawl it) then they will be perceived as evil and to a point hypocritical because they built a business of crawling, indexing, organizing other people's data) I am eager to see how they play the 'Is-GoogleBase-Crawlable' card and what they do with it. I have not posted about Google Base because personally I don't understand it. It is not that I don't understand it from a 'Why should I give Google my core data (sans the pretty navigation of my site)?' it is that I don't understand exactly where all this data is going to end up. I do think that Google has very good results compared to other search engines, but they are not ANYWHERE NEAR what I would call "good" results, all search engines still need TONS of work to really answer any non-subjective question I give them. In my opinion, Google (and all other SEs) still can't organize the information that is presented in the semi-stateful presentation method we call the web, why should I think that they can do it if I give them large chunks of it in xml? I have no clue what Brett Tabke is going to do for organization and Search Engine presence for Webmasterworld.com, but I am actually really eager to see what happens. WMW is a huge site with a great following. Can it live (and continue to increase its membership base) without a mass amount of pages on G? I certainly hope so, for not just WMW users , but for anyone who uses a search engine. I also hope that Google Base (or anything like it from other Search Engines) is not going to be the run around to site spidering. I completely understand the bandwidth issues and his reasons, I am eager to see how it all pans out. It really is a brave thing to do and really is going to be neat to see what the ramifications are from it.