How Google Closed Source Wanted To Change The Destiny Of Humanity By Changing Your DNA Information During 2004

Change In the DNA 9: The latest on update Austin (Google’s January update)

Posted on January 31, 2004 by rankforsales | 4 comments

 

Some of the websites that haven’t been hit too hard in

Google’s Florida update (November 2003) got hit real hard on or around January 23. Google’s latest update is called Austin, and they are beginning to ‘sound’ like elections!

Depending on the industry you happen to be in, you could have been hit less, or harder- it depends on a whole number of factors and not one situation is usually the same. One of our clients that, up to recently had their site optimized by another SEO firm was completely devastated to realize his site was gone from the face of the earth.

Things such as FFA’s (Free for All) link farms, invisible text (!) and stuffed meta tags that are 20 screens long, filled with useless spam will have your site penalized, or even banned faster than you can blink an eye.
Google’s new search algorithm is getting smarter and better. Part of it is due to the Hilltop Algorithm, which I wrote about at the beginning of January. In combination with the Page Rank™ algorithm that Google has been using since 1998, these two algo’s work in tandem in what is supposed to be producing better, more relevant search results.

This last statement remains to be tested, as in some cases I have found that some of the results produced are not always relevant. But Google is continuously tweaking and ‘fine-tuning’ its technology to make it better and the next few weeks should see more improvements.

To read the complete article on the Rank for $ales website, click here.

Posted by Search Engine News Blog and by Tech Blog.

Source: http://www.searchenginejournal.com/the-latest-on-update-austin-googles-january-update/237/

Google Update Austin: Google Update Florida Again

January 2004

"Wow Google just destroyed my rankings"

A common line I have been told over and over again. Ever since the middle of November Google has been up to some real trickery...
While this update appears to be similar to update Florida, there has not been anywhere near the same amount of coverage. I have read the posts located at the Highrankings Forums, Cre8asite Forums, SearchGuild, and IHelpYou Forums. I have not read the posts at WebmasterWorld since I am not a subscriber. Like the Florida update it is clear that some spam has risen to the top (Google Search: Michigan Data Recovery courtesy MakeMeTop) and some results have improved.
Most SEOs would recommend not making large changes to your site while the search engine is still in a state of flux.

Search Results

Many sites that were not beaten down by the Florida update have been dropped during this update. Some of the common sites which have dropped are

• those who relied heavily on "on the page" SEO
• those who exchanged links with off topic sites to rank well
• location specific sites

Please note that the update is still occurring and many changes were occurring long after the Florida update was done. Right now it appears that the current winners from this update are

• Amazon
• large shopping sites
• directories
• deep inner pages within sites which may have a strong site rank

What is Google Trying to do?

Contrary to popular belief the CIA does not control the Google Dance.
People like me can make a living manipulating Google's search results. Google does not like this. Essentially Google wants the most relevant (not the most optimized) sites at the top of the search results.
Google may actually periodically re weight factors only for the sake of making it harder to figure out what their algorithm is. Most of the core powerful websites will remain unaffected by the changing algorithms. Those site most likely to rise or fall are those gaming the search engine. Some innocent sites may get lost in the shuffle, but that is not what Google is concerned with. Google wants to provide the most relevant search results to its users.
If Google can make it hard to optimize your site for Google then it will require you to waste resources figuring out the latest algorithm change or conform to what Google desires. Google wants information to appear in the free listings. Google would like commercial sites to use their ads.

What is with the big shakeups?

Any time you introduce a new algorithm which is based on a relational database the results will be a bit shaky off the start. With each update the results will fall more in line with the goals of the new algorithm.
Off the start many of the results may not make sense, but the end goal is that most will. You and I both notice the shakeups more than a normal searcher since we track our specific websites.

Google coping with problems

Many search engine algorithms become less efficient as the size of the database grows. To fight off spamming and to improve functionality of their search product Google is trying to aid PageRank by using other algorithmic features. Instead of being primarily powered by just PageRank they are most likely trying to introduce techniques such as LocalRank, stemming, and latent symantic indexing.

What is PageRank?

PageRank is an overall view of the popularity of a specific page against the entire internet. In general it is an approximation of the odds a random web surfer would come across a page.
The problem with PageRank is that since its base is against the entire internet it is very easy for a small sector of the internet to bond together to manipulate results.

What is LocalRank?

LocalRank is a concept similar to PageRank, but based on a local segment of the web. Teoma reranks the relevancy of search results on the fly (after the search) based upon their local interconnectivity. When Google bought Kaltix it purchased a similar technology.

What is Stemming?

Stemming is the conversion of a word to its base form in an attempt to better understand it. Typically the exact match of the the searched version should have a slightly higher weighting, but the other versions of the word also play into account in search results.

What is Latent Symantic Indexing?

Latent symantic indexing is the organization of search results through conceptually understand each page. Based upon the natural patterns in language, latent symantic indexing can mathematically approximate a good understanding of the purpose of a document.

Why does Google test new technologies on its largest database first?

The reason the change is occurring on the main site first is that it has the most content...meaning that when you look for patterns in a relational database, the larger your initial data set, the better the results you will be able to extract.
After the results are extracted it then easy to apply the results of the research to the other search locations. Some of the short term sloppy results may actually be necessary for Google to compare data to fine tune its algorithm. It may even compare data retrieved from search results.

What Did Google Do?

I think off the start Google implemented a keyword filter on some phrases. The setpoint level is based upon many dynamic factors and could even be determined by the search results.
During the Florida Update I noticed that I was able to get some sites listed (which were filtered out) just by splitting up the words and lowering the keyword density. I have noticed that this update seems to be even less lenient.
I do not believe the change is just a filter, I only believe they use a strong filter in combination with other technologies. After they start to collect data the filter becomes more and more lenient. To see what the search results would be like without the "filter" use allintext: before your search. This will show results similar to the pre Austin dance. I would not grieve about the results though as they will again be changing as the database settles out.
I believe as Google collects data from the search results, whatever type of filter they are using will slowly be lifted and replaced with data from the search results.

What Can I do to Help My Site Out?

If you are a purely commercial website Google is trying to push you to use their ad system. If you still do not want to use the ad system it is going to require accumulating inbound links from quality websites.
A good place to start is by registering your site with some of the top web directories. In addition you will want to look for directories specific to your field.

Where else can I get links?

You can get links from anywhere, but it is appearing that topic related links are becoming more important. The best way to get links is to have content that other people like enough that they just want to link to it.
I hate to be the person to say focus on the content, but as search engines get more advanced it will be harder to manipulate them. Pick a topic that is tight enough that you can be the best at it, and pick a topic you are interested it...as it will be far easier and more enjoyable to create.
If you can get people to want to link to you because you have valuable information, especially if they link without you even asking...then you have a great competitive advantage against most of your competitors. If you need any help with promoting your website feel free to give me a call or shoot me an email.
- by Aaron Wall

We Value Privacy

This article may be syndicated in whole are part. Simply provide a link back to the original article or http://www.search-marketing.info. Please note that I do not usually update articles over time and the date last modified on article pages is usually referring to a navigational change.

Source: http://www.search-marketing.info/newsletter/articles/austin-florida.htm

Change In the DNA 10:

If you’re a forum regular you know by now that Google’s results are changing again. I hope your rankings improve.

Brandy Exposed

This latest update, dubbed Brandy, (we thought it should be called Brittany…) was accurately predicted last week by the guys at Stepforth.

While this update has not yet fully rolled out to all the datacenters, you can search the new Google here: http://64.233.161.105/.

What difference do you notice?
So what’s different now? Andy Beal said that it “looks like they have tweaked things to include a lot more “authority” sites that were previously not included.” This speaks to the Stepforth announcement that Google had increased the number of links that they recognize.

Daniel Brandt from over at Google-Watch.org said that he’s seeing “lots of Austin type results in many categories.” About the change in results he said “It’s too early [to tell what the difference is], or not a very impressive update.”

“In this case the change is not as significant as Florida or Austin. I don’t think it will be as sticky,” he concluded.

GoogleGuy, a Google spokesperson who posts at WebMasterWorld, verified the update and gave a rough timeline for its roll out, “I just talked to somebody else at Google. Sounds like 64.x.x.x is indeed the wave of the future. They did say that it may roll out over several days instead of being done over the weekend though.”

In answer to a poster’s question on whether the Brandy update was a move back to pre-Florida results GoogleGuy said, “we’ve definitely been working to incorporate new signals of quality and improve the way that we rank pages, so the results at the 64.x.x.x data center are not a rollback or pre-Florida results–it’s several steps forward based on new ways of gauging quality and relevance.”
Visit this Google datacenter: http://64.233.161.105/ and post any differences you notice.
Garrett French is the editor of iEntry’s eBusiness channel. You can talk to him directly at WebProWorld, the eBusiness Community Forum.

Source: http://www.webpronews.com/googles-brandy-update-exposed-2004-02

The Google update of the 17th-20th February 2004 (nicknamed ‘Brandy’ by WebmasterWorld) resulted in major changes in the results the search engine returns.
The ‘Brandy’ update seems to have incorporated some pre-’Florida’ results (another major update that occurred at the end of 2003), mixed with numerous new factors. Google stores its index on a number of data centers around the world. Since ‘Florida’, some of the old data centers were taken offline, and pundits believe that Google has kept the old SERPs (Search Engine Results Pages) in a preserved state for the last few months.
Indeed, Google brought these data centres back at the same time that Yahoo! broke from Google, in favour of its new Inktomi-based results. Consequently, I don’t think this is the last of the major changes we’ll see in Google, but it does seem that Google is getting closer to what it aims to achieve.

Five Changes

Brin, one of the founders of Google recently said,
Google has made five significant changes to its algorithmic formulas in the last two weeks.
(Associated Press (AP), Feb 17th 2004)
While we can only guess at what those changes were, the following are probably a good bet.

1. Increase in Index SizeGoogle’s spider, Googlebot, has had a busy few weeks — at the time of the update, Google announced that it had massively increased the size of its index.
   This move was probably made to ensure Google made headlines at the same time as Yahoo! (for example, in this report in the BBC News, Feb 18th 2004). However, in order to increase the index size, Google may have had to re-include some of the pre-Florida results that had previously been dropped.
2. Latent Semantic Indexing (LSI)This is a very significant new technology that Google has always been interested in, and the incorporation of LSI has been on the cards for some time. If you are an insomniac, then Yu et al.’s paper is quite helpful in explaining the concept, but, in short, LSI is about using close semantic matches to put your page into the correct topical context.
   It’s all about synonyms. LSI may see Google effectively remove all instances of the search keyword when analysing your page, in favour of a close analysis of other words. For example, consider the search term ‘travel insurance’. LSI-based algorithms will look for words and links that pertain to related topics, such as skiing, holidays, medical, backpacking, and airports.
3. Links and Anchor TextLinks have always been the essence of Google, but the engine is steadily altering its focus. The importance of Page Rank (PR), Google’s unique ranking system, is being steadily downgraded in favour of the nature, quality, and quantity of inbound and outbound link anchor text. If PR is downgraded, and the wording of inbound links is boosted, this may explain, to a large degree, the position in which many sites currently find themselves.
   For example, most people will link to a site’s homepage. In the past, due to internal linking structures, PR was spread and other pages benefited. Now, it is more important for Webmasters to attract links that point directly to the relevant pages of their sites using anchor text that’s relevant to the specific pages.
   Furthermore, Google seems to be using outbound links to determine how useful and authoritative a site is. For example, directories that are doing well are those that direct link to the sites, rather than use dynamic URLs.
4. NeighbourhoodsNow, more than ever, has the question of who’s linking to your site become critical. Links must be from related topic sites (the higher the PR the better); those links are seen to define your ‘neighbourhood’.
   If we again consider the example of travel insurance, big insurance companies might buy links on holiday-related sites in order to boost their ranking. These businesses will actively invest in gaining targeted inbound links from a broad mix of sites. Consequently, their neighbourhoods appear tightly focused to Google.
5. Downgrading of Traditional Tag-Based OptimisationClever use of the title, h1, h2, bold, and italics tags, and CSS, is no longer as important to a site’s ranking as it once was. It is very interesting to listen to Sergey (co-founder of Google) talk about this, because he’s the one usually quoted about the ways in which people manipulate his index. Google has taken big steps to downgrade standard SEO techniques in favour of LSI and linking, which are far less manipulable by the masses.

The Impact of Brandy

These changes make for sober reading if you’re a Webmaster — to optimize your site successfully for Google has become a lot more difficult. Nevertheless, there are a number of practical steps that can be taken to promote your ranking in the short and long term.

1. SynonymsAs LSI appears to be so significant, it is important to start looking carefully at the information architecture of each major section of your site, and to increase the use of related words. It is also important to re-examine the title tags to include this concept; good title tags have synonyms and avoid repetition of the key phrase.
2. Outbound LinksLink to authority sites on your subject. In the travel insurance example, these authority sites could include places like the State department, major skiing directories, etc. Not only will this help with LSI, it also allows Google to define the neighbourhood more easily. Furthermore, you could engage in link swaps with other companies so that you gain the benefit of an on-topic, LSI-friendly link.
3. Inbound Links and Link to Us PagesBased on what we have just said, sites need to formulate a link development strategy. A budget needs to be set aside to buy links and develop mini-sites. Look to set up links with university sites (.edu or ac.uk), as these seem to be valuable given Google’s informational bias.
   Each section of a site should have its own link-to-us page. For example, HotScripts, the major computer script directory, has a great link-to-us page.
   By providing people with creatives and cut-and-paste HTML, you can vastly improve your chances of attracting reciprocal links to your site. You’ll need to have a separate page for each section, to maximise on-topic inbound links.
4. Mini-SitesIt is important to develop separate mini-sites (also known as satellite sites) for each key subject of your Website. This is a useful tactic that improves your chances of appearing in the SERPs for your keywords. Furthermore, as the last three Google updates have shaken things up so much, having more than one site reduces the likelihood that your business will be disrupted by the engine’s updates. However, Google is likely to view satellite sites as spam, so you must take some steps to reduce the chances of your being blacklisted on this basis.
   First, make it as hard as possible to for Google to detect host affiliation between your main site and its mini-sites. Google may define sites to be owned by the same person if the first 3 octets of the sites’ IP addresses are the same (e.g. 123.123.123.xxx). Therefore, if you’re going to run mini-sites, put them on different Web hosts.
   Secondly, use different domain names for your mini-sites, rather than sub-domains of your main site. In the past, Google has not penalised sub-domains, but the early results from the Brandy update show a considerable reduction in the presence of sub-domains in the SERPs.
   Finally, be very careful with the linking strategy you use between mini-sites — Google will look at the linking structure very critically. Don’t plaster each of your sites with links to the others, and don’t reciprocate links between the sites.
   Mini-sites make it easier to create on-topic neighbourhoods and experiment with LSI techniques. Creating a large network can be a means to boost your main site’s rank, but make sure you’re well aware of the risks involved with creating these mini-sites before you embark.

Use Brandy to your Advantage!

Google optimisation is now a lot harder than it used to be. However, the index is still manipulable. Success involves hard work, and potentially the expenditure of funds to develop a good mini-site network and buy links on relevant pages.

Source: http://www.sitepoint.com/brandy-google-update/

Joe Rogan's Twitter: https://twitter.com/joerogan
Rogan's Forums: http://forums.joerogan.net/
Tom Campbell's Forums: http://www.my-big-toe.com/forums/

My FREE Newsletter: http://redpillphilosophy.com
Forums: http://redpill.boards.net
Cool Shirts: http://redpillphilosophy.com/store
Support this Message: http://redpillphilosophy.com/invest
My Bitcoin Address: 1NtSZ7SrVRm9z6KWURNiNPpNbgmoo774sK
Twitter: https://twitter.com/chriswritingnow

Exposed: Exclusive How Google Closed Source Monopoly Wanted To Become The World Wide Web And To Control The Entire Population Of The Planet - Do You Want A Machine To Control Your Thoughts?

Exclusive: How Google’s Algorithm Rules the Web

• By Steven LevyEmail Author
• February 22, 2010 | 
• 12:00 pm | 
• Categories: Wired March 2010

When it comes to finding stuff, there's Google — and there's everyone else.
Photo: Mauricio Alejo

Want to know how Google is about to change your life? Stop by the Ouagadougou conference room on a Thursday morning. It is here, at the Mountain View, California, headquarters of the world’s most powerful Internet company, that a room filled with three dozen engineers, product managers, and executives figure out how to make their search engine even smarter. This year, Google will introduce 550 or so improvements to its fabled algorithm, and each will be determined at a gathering just like this one. The decisions made at the weekly Search Quality Launch Meeting will wind up affecting the results you get when you use Google’s search engine to look for anything — “Samsung SF-755p printer,” “Ed Hardy MySpace layouts,” or maybe even “capital Burkina Faso,” which just happens to share its name with this conference room. Udi Manber, Google’s head of search since 2006, leads the proceedings. One by one, potential modifications are introduced, along with the results of months of testing in various countries and multiple languages. A screen displays side-by-side results of sample queries before and after the change. Following one example — a search for “guitar center wah-wah” — Manber cries out, “I did that search!”

You might think that after a solid decade of search-market dominance, Google could relax. After all, it holds a commanding 65 percent market share and is still the only company whose name is synonymous with the verb search. But just as Google isn’t ready to rest on its laurels, its competitors aren’t ready to concede defeat. For years, the Silicon Valley monolith has used its mysterious, seemingly omniscient algorithm to, as its mission statement puts it, “organize the world’s information.” But over the past five years, a slew of companies have challenged Google’s central premise: that a single search engine, through technological wizardry and constant refinement, can satisfy any possible query. Facebook launched an early attack with its implication that some people would rather get information from their friends than from an anonymous formula. Twitter’s ability to parse its constant stream of updates introduced the concept of real-time search, a way of tapping into the latest chatter and conversation as it unfolds. Yelp helps people find restaurants, dry cleaners, and babysitters by crowdsourcing the ratings. None of these upstarts individually presents much of a threat, but together they hint at a wide-open, messier future of search — one that isn’t dominated by a single engine but rather incorporates a grab bag of services.

Still, the biggest threat to Google can be found 850 miles to the north: Bing. Microsoft’s revamped and rebranded search engine — with a name that evokes discovery, a famous crooner, or Tony Soprano’s strip joint — launched last June to surprisingly upbeat reviews. (The Wall Street Journal called it “more inviting than Google.”) The new look, along with a $100 million ad campaign, helped boost Microsoft’s share of the US search market from 8 percent to about 11 — a number that will more than double once regulators approve a deal to make Bing the search provider for Yahoo.

Team Bing has been focusing on unique instances where Google’s algorithms don’t always satisfy. For example, while Google does a great job of searching the public Web, it doesn’t have real-time access to the byzantine and constantly changing array of flight schedules and fares. So Microsoft purchased Farecast — a Web site that tracks airline fares over time and uses the data to predict when ticket prices will rise or fall — and incorporated its findings into Bing’s results. Microsoft made similar acquisitions in the health, reference, and shopping sectors, areas where it felt Google’s algorithm fell short.

Even the Bingers confess that, when it comes to the simple task of taking a search term and returning relevant results, Google is still miles ahead. But they also think that if they can come up with a few areas where Bing excels, people will get used to tapping a different search engine for some kinds of queries. “The algorithm is extremely important in search, but it’s not the only thing,” says Brian MacDonald, Microsoft’s VP of core search. “You buy a car for reasons beyond just the engine.”

Google’s response can be summed up in four words: mike siwek lawyer mi.

Amit Singhal types that koan into his company’s search box. Singhal, a gentle man in his forties, is a Google Fellow, an honorific bestowed upon him four years ago to reward his rewrite of the search engine in 2001. He jabs the Enter key. In a time span best measured in a hummingbird’s wing-flaps, a page of links appears. The top result connects to a listing for an attorney named Michael Siwek in Grand Rapids, Michigan. It’s a fairly innocuous search — the kind that Google’s servers handle billions of times a day — but it is deceptively complicated. Type those same words into Bing, for instance, and the first result is a page about the NFL draft that includes safety Lawyer Milloy. Several pages into the results, there’s no direct referral to Siwek.

The comparison demonstrates the power, even intelligence, of Google’s algorithm, honed over countless iterations. It possesses the seemingly magical ability to interpret searchers’ requests — no matter how awkward or misspelled. Google refers to that ability as search quality, and for years the company has closely guarded the process by which it delivers such accurate results. But now I am sitting with Singhal in the search giant’s Building 43, where the core search team works, because Google has offered to give me an unprecedented look at just how it attains search quality. The subtext is clear: You may think the algorithm is little more than an engine, but wait until you get under the hood and see what this baby can really do.

Key Advances in
Google Search

Google’s search algorithm is a work in progress — constantly tweaked and refined to return higher-quality results. Here are some of the most significant additions and adaptations since the dawn of PageRank. — Steven Levy

Backrub
[September 1997]

This search engine, which had run on Stanford’s servers for almost two years, is renamed Google. Its breakthrough innovation: ranking searches based on the number and quality of incoming links.

New algorithm
[August 2001]

The search algorithm is completely revamped to incorporate additional ranking criteria more easily.

Local connectivity analysis
[February 2003]

Google’s first patent is granted for this feature, which gives more weight to links from authoritative sites.

Fritz
[Summer 2003]

This initiative allows Google to update its index constantly, instead of in big batches.

Personalized results
[June 2005]

Users can choose to let Google mine their own search behavior to provide individualized results.

Bigdaddy
[December 2005]

Engine update allows for more-comprehensive Web crawling.

Universal search
[May 2007]

Building on Image Search, Google News, and Book Search, the new Universal Search allows users to get links to any medium on the same results page.

Real-Time Search
[December 2009]

Displays results from Twitter and blogs as they are published.

The story of Google’s algorithm begins with PageRank, the system invented in 1997 by cofounder Larry Page while he was a grad student at Stanford. Page’s now legendary insight was to rate pages based on the number and importance of links that pointed to them — to use the collective intelligence of the Web itself to determine which sites were most relevant. It was a simple and powerful concept, and — as Google quickly became the most successful search engine on the Web — Page and cofounder Sergey Brin credited PageRank as their company’s fundamental innovation.

But that wasn’t the whole story. “People hold on to PageRank because it’s recognizable,” Manber says. “But there were many other things that improved the relevancy.” These involve the exploitation of certain signals, contextual clues that help the search engine rank the millions of possible results to any query, ensuring that the most useful ones float to the top.

Web search is a multipart process. First, Google crawls the Web to collect the contents of every accessible site. This data is broken down into an index (organized by word, just like the index of a textbook), a way of finding any page based on its content. Every time a user types a query, the index is combed for relevant pages, returning a list that commonly numbers in the hundreds of thousands, or millions. The trickiest part, though, is the ranking process — determining which of those pages belong at the top of the list.
That’s where the contextual signals come in. All search engines incorporate them, but none has added as many or made use of them as skillfully as Google has. PageRank itself is a signal, an attribute of a Web page (in this case, its importance relative to the rest of the Web) that can be used to help determine relevance. Some of the signals now seem obvious. Early on, Google’s algorithm gave special consideration to the title on a Web page — clearly an important signal for determining relevance. Another key technique exploited anchor text, the words that make up the actual hyperlink connecting one page to another. As a result, “when you did a search, the right page would come up, even if the page didn’t include the actual words you were searching for,” says Scott Hassan, an early Google architect who worked with Page and Brin at Stanford. “That was pretty cool.” Later signals included attributes like freshness (for certain queries, pages created more recently may be more valuable than older ones) and location (Google knows the rough geographic coordinates of searchers and favors local results). The search engine currently uses more than 200 signals to help rank its results.
Google’s engineers have discovered that some of the most important signals can come from Google itself. PageRank has been celebrated as instituting a measure of populism into search engines: the democracy of millions of people deciding what to link to on the Web. But Singhal notes that the engineers in Building 43 are exploiting another democracy — the hundreds of millions who search on Google. The data people generate when they search — what results they click on, what words they replace in the query when they’re unsatisfied, how their queries match with their physical locations — turns out to be an invaluable resource in discovering new signals and improving the relevance of results. The most direct example of this process is what Google calls personalized search — a feature that uses someone’s search history and location as signals to determine what kind of results they’ll find useful.¹ But more generally, Google has used its huge mass of collected data to bolster its algorithm with an amazingly deep knowledge base that helps interpret the complex intent of cryptic queries.

Take, for instance, the way Google’s engine learns which words are synonyms. “We discovered a nifty thing very early on,” Singhal says. “People change words in their queries. So someone would say, ‘pictures of dogs,’ and then they’d say, ‘pictures of puppies.’ So that told us that maybe ‘dogs’ and ‘puppies’ were interchangeable. We also learned that when you boil water, it’s hot water. We were relearning semantics from humans, and that was a great advance.”

But there were obstacles. Google’s synonym system understood that a dog was similar to a puppy and that boiling water was hot. But it also concluded that a hot dog was the same as a boiling puppy. The problem was fixed in late 2002 by a breakthrough based on philosopher Ludwig Wittgenstein’s theories about how words are defined by context. As Google crawled and archived billions of documents and Web pages, it analyzed what words were close to each other. “Hot dog” would be found in searches that also contained “bread” and “mustard” and “baseball games” — not poached pooches. That helped the algorithm understand what “hot dog” — and millions of other terms — meant. “Today, if you type ‘Gandhi bio,’ we know that bio means biography,” Singhal says. “And if you type ‘bio warfare,’ it means biological.”
Throughout its history, Google has devised ways of adding more signals, all without disrupting its users’ core experience. Every couple of years there’s a major change in the system — sort of equivalent to a new version of Windows — that’s a big deal in Mountain View but not discussed publicly. “Our job is to basically change the engines on a plane that is flying at 1,000 kilometers an hour, 30,000 feet above Earth,” Singhal says. In 2001, to accommodate the rapid growth of the Web, Singhal essentially revised Page and Brin’s original algorithm completely, enabling the system to incorporate new signals quickly. (One of the first signals on the new system distinguished between commercial and noncommercial pages, providing better results for searchers who want to shop.) That same year, an engineer named Krishna Bharat, figuring that links from recognized authorities should carry more weight, devised a powerful signal that confers extra credibility to references from experts’ sites. (It would become Google’s first patent.) The most recent major change, codenamed Caffeine, revamped the entire indexing system to make it even easier for engineers to add signals.

Google is famously creative at encouraging these breakthroughs; every year, it holds an internal demo fair called CSI — Crazy Search Ideas — in an attempt to spark offbeat but productive approaches. But for the most part, the improvement process is a relentless slog, grinding through bad results to determine what isn’t working. One unsuccessful search became a legend: Sometime in 2001, Singhal learned of poor results when people typed the name “audrey fino” into the search box. Google kept returning Italian sites praising Audrey Hepburn. (Fino means fine in Italian.) “We realized that this is actually a person’s name,” Singhal says. “But we didn’t have the smarts in the system.”

The Audrey Fino failure led Singhal on a multiyear quest to improve the way the system deals with names — which account for 8 percent of all searches. To crack it, he had to master the black art of “bi-gram breakage” — that is, separating multiple words into discrete units. For instance, “new york” represents two words that go together (a bi-gram). But so would the three words in “new york times,” which clearly indicate a different kind of search. And everything changes when the query is “new york times square.” Humans can make these distinctions instantly, but Google does not have a Brazil-like back room with hundreds of thousands of cubicle jockeys. It relies on algorithms.

Voila — when a hot dog is not a boiling puppy.
Photo: Mauricio Alejo

The Mike Siwek query illustrates how Google accomplishes this. When Singhal types in a command to expose a layer of code underneath each search result, it’s clear which signals determine the selection of the top links: a bi-gram connection to figure it’s a name; a synonym; a geographic location. “Deconstruct this query from an engineer’s point of view,” Singhal explains. “We say, ‘Aha! We can break this here!’ We figure that lawyer is not a last name and Siwek is not a middle name. And by the way, lawyer is not a town in Michigan. A lawyer is an attorney.”

This is the hard-won realization from inside the Google search engine, culled from the data generated by billions of searches: a rock is a rock. It’s also a stone, and it could be a boulder. Spell it “rokc” and it’s still a rock. But put “little” in front of it and it’s the capital of Arkansas. Which is not an ark. Unless Noah is around. “The holy grail of search is to understand what the user wants,” Singhal says. “Then you are not matching words; you are actually trying to match meaning.”

And Google keeps improving. Recently, search engineer Maureen Heymans discovered a problem with “Cindy Louise Greenslade.” The algorithm figured out that it should look for a person — in this case a psychologist in Garden Grove, California — but it failed to place Greenslade’s homepage in the top 10 results. Heymans found that, in essence, Google had downgraded the relevance of her homepage because Greenslade used only her middle initial, not her full middle name as in the query. “We needed to be smarter than that,” Heymans says. So she added a signal that looks for middle initials. Now Greenslade’s homepage is the fifth result.

At any moment, dozens of these changes are going through a well-oiled testing process. Google employs hundreds of people around the world to sit at their home computer and judge results for various queries, marking whether the tweaks return better or worse results than before. But Google also has a larger army of testers — its billions of users, virtually all of whom are unwittingly participating in its constant quality experiments. Every time engineers want to test a tweak, they run the new algorithm on a tiny percentage of random users, letting the rest of the site’s searchers serve as a massive control group. There are so many changes to measure that Google has discarded the traditional scientific nostrum that only one experiment should be conducted at a time. “On most Google queries, you’re actually in multiple control or experimental groups simultaneously,” says search quality engineer Patrick Riley. Then he corrects himself. “Essentially,” he says, “all the queries are involved in some test.” In other words, just about every time you search on Google, you’re a lab rat.

This flexibility — the ability to add signals, tweak the underlying code, and instantly test the results — is why Googlers say they can withstand any competition from Bing or Twitter or Facebook. Indeed, in the last six months Google has made more than 200 improvements, some of which seem to mimic — even outdo — the offerings of its competitors. (Google says this is just a coincidence and points out that it has been adding features routinely for years.) One is real-time search, eagerly awaited since Page opined some months ago that Google should be scanning the entire Web every second. When someone queries a subject of current interest, among the 10 blue links Google now puts a “latest results” box: a scrolling set of just-produced posts from news sources, blogs, or tweets. Once again, Google uses signals to ensure that only the most relevant tweets find their way into the real-time stream. “We look at what’s retweeted, how many people follow the person, and whether the tweet is organic or a bot,” Singhal says. “We know how to do this, because we’ve been doing it for a decade.”

Along with real-time search, Google has introduced other new features, including a service called Goggles, which treats images captured by users’ phones as search queries. It’s all part of the company’s relentless march toward search becoming an always-on, ubiquitous presence. With a camera and voice recognition, a smartphone becomes eyes and ears. If the right signals are found, anything can be query fodder.
Google’s massive computing power and bandwidth give the company an undeniable edge. Some observers say it’s an advantage that essentially prohibits startups from trying to compete. But Manber says it’s not infrastructure alone that makes Google the leader: “The very, very, very key ingredient in all of this is that we hired the right people.”

By all standards, Qi Lu qualifies as one of those people. “I have the highest regard for him,” says Manber, who worked with the 48-year-old computer scientist at Yahoo. But Lu joined Microsoft early last year to lead the Bing team. When asked about his mission, Lu, a diminutive man dressed in jeans and a Bing T-shirt, pauses, then softly recites a measured reply: “It’s extremely important to keep in mind that this is a long-term journey.” He has the same I’m-not-going-away look in his eye that Uma Thurman has in Kill Bill.
Indeed, the company that won last decade’s browser war has a best-served-cold approach to search, an eerie certainty that at some point, people are going to want more than what Google’s algorithm can provide. “If we don’t have a paradigm shift, it’s going to be very, very difficult to compete with the current winners,” says Harry Shum, Microsoft’s head of core search development. “But our view is that there will be a paradigm shift.”

Still, even if there is such a shift, Google’s algorithms will probably be able to incorporate that, too. That’s why Google is such a fearsome competitor; it has built a machine nimble enough to absorb almost any approach that threatens it — all while returning high-quality results that its competitors can’t match. Anyone can come up with a new way to buy plane tickets. But only Google knows how to find Mike Siwek.
Senior writer Steven Levy (steven_levy@wired.com) wrote about Twitter in issue 17.11.

1. Correction appended [February 25] Google’s personalized search uses someone’s search history and location to determine what kind of results they’ll find useful. It does not require them to opt-in or be logged in, as previously reported.

Exclusive: How Google’s Algorithm Rules the Web

• By Steven LevyEmail Author
• February 22, 2010 | 
• 12:00 pm | 
• Categories: Wired March 2010

Key Advances in
Google Search

Google’s search algorithm is a work in progress — constantly tweaked and refined to return higher-quality results. Here are some of the most significant additions and adaptations since the dawn of PageRank. — Steven Levy

Backrub
[September 1997]

This search engine, which had run on Stanford’s servers for almost two years, is renamed Google. Its breakthrough innovation: ranking searches based on the number and quality of incoming links.

New algorithm
[August 2001]

The search algorithm is completely revamped to incorporate additional ranking criteria more easily.

Local connectivity analysis
[February 2003]

Google’s first patent is granted for this feature, which gives more weight to links from authoritative sites.

Fritz
[Summer 2003]

This initiative allows Google to update its index constantly, instead of in big batches.

Personalized results
[June 2005]

Users can choose to let Google mine their own search behavior to provide individualized results.

Bigdaddy
[December 2005]

Engine update allows for more-comprehensive Web crawling.

Universal search
[May 2007]

Building on Image Search, Google News, and Book Search, the new Universal Search allows users to get links to any medium on the same results page.

Real-Time Search
[December 2009]

Displays results from Twitter and blogs as they are published.

The story of Google’s algorithm begins with PageRank, the system invented in 1997 by cofounder Larry Page while he was a grad student at Stanford. Page’s now legendary insight was to rate pages based on the number and importance of links that pointed to them — to use the collective intelligence of the Web itself to determine which sites were most relevant. It was a simple and powerful concept, and — as Google quickly became the most successful search engine on the Web — Page and cofounder Sergey Brin credited PageRank as their company’s fundamental innovation.

But that wasn’t the whole story. “People hold on to PageRank because it’s recognizable,” Manber says. “But there were many other things that improved the relevancy.” These involve the exploitation of certain signals, contextual clues that help the search engine rank the millions of possible results to any query, ensuring that the most useful ones float to the top.

Web search is a multipart process. First, Google crawls the Web to collect the contents of every accessible site. This data is broken down into an index (organized by word, just like the index of a textbook), a way of finding any page based on its content. Every time a user types a query, the index is combed for relevant pages, returning a list that commonly numbers in the hundreds of thousands, or millions. The trickiest part, though, is the ranking process — determining which of those pages belong at the top of the list.
That’s where the contextual signals come in. All search engines incorporate them, but none has added as many or made use of them as skillfully as Google has. PageRank itself is a signal, an attribute of a Web page (in this case, its importance relative to the rest of the Web) that can be used to help determine relevance. Some of the signals now seem obvious. Early on, Google’s algorithm gave special consideration to the title on a Web page — clearly an important signal for determining relevance. Another key technique exploited anchor text, the words that make up the actual hyperlink connecting one page to another. As a result, “when you did a search, the right page would come up, even if the page didn’t include the actual words you were searching for,” says Scott Hassan, an early Google architect who worked with Page and Brin at Stanford. “That was pretty cool.” Later signals included attributes like freshness (for certain queries, pages created more recently may be more valuable than older ones) and location (Google knows the rough geographic coordinates of searchers and favors local results). The search engine currently uses more than 200 signals to help rank its results.

Google’s engineers have discovered that some of the most important signals can come from Google itself. PageRank has been celebrated as instituting a measure of populism into search engines: the democracy of millions of people deciding what to link to on the Web. But Singhal notes that the engineers in Building 43 are exploiting another democracy — the hundreds of millions who search on Google. The data people generate when they search — what results they click on, what words they replace in the query when they’re unsatisfied, how their queries match with their physical locations — turns out to be an invaluable resource in discovering new signals and improving the relevance of results. The most direct example of this process is what Google calls personalized search — a feature that uses someone’s search history and location as signals to determine what kind of results they’ll find useful.¹ But more generally, Google has used its huge mass of collected data to bolster its algorithm with an amazingly deep knowledge base that helps interpret the complex intent of cryptic queries.

Take, for instance, the way Google’s engine learns which words are synonyms. “We discovered a nifty thing very early on,” Singhal says. “People change words in their queries. So someone would say, ‘pictures of dogs,’ and then they’d say, ‘pictures of puppies.’ So that told us that maybe ‘dogs’ and ‘puppies’ were interchangeable. We also learned that when you boil water, it’s hot water. We were relearning semantics from humans, and that was a great advance.”

But there were obstacles. Google’s synonym system understood that a dog was similar to a puppy and that boiling water was hot. But it also concluded that a hot dog was the same as a boiling puppy. The problem was fixed in late 2002 by a breakthrough based on philosopher Ludwig Wittgenstein’s theories about how words are defined by context. As Google crawled and archived billions of documents and Web pages, it analyzed what words were close to each other. “Hot dog” would be found in searches that also contained “bread” and “mustard” and “baseball games” — not poached pooches. That helped the algorithm understand what “hot dog” — and millions of other terms — meant. “Today, if you type ‘Gandhi bio,’ we know that bio means biography,” Singhal says. “And if you type ‘bio warfare,’ it means biological.”
Throughout its history, Google has devised ways of adding more signals, all without disrupting its users’ core experience. Every couple of years there’s a major change in the system — sort of equivalent to a new version of Windows — that’s a big deal in Mountain View but not discussed publicly. “Our job is to basically change the engines on a plane that is flying at 1,000 kilometers an hour, 30,000 feet above Earth,” Singhal says. In 2001, to accommodate the rapid growth of the Web, Singhal essentially revised Page and Brin’s original algorithm completely, enabling the system to incorporate new signals quickly. (One of the first signals on the new system distinguished between commercial and noncommercial pages, providing better results for searchers who want to shop.) That same year, an engineer named Krishna Bharat, figuring that links from recognized authorities should carry more weight, devised a powerful signal that confers extra credibility to references from experts’ sites. (It would become Google’s first patent.) The most recent major change, codenamed Caffeine, revamped the entire indexing system to make it even easier for engineers to add signals.

Google is famously creative at encouraging these breakthroughs; every year, it holds an internal demo fair called CSI — Crazy Search Ideas — in an attempt to spark offbeat but productive approaches. But for the most part, the improvement process is a relentless slog, grinding through bad results to determine what isn’t working. One unsuccessful search became a legend: Sometime in 2001, Singhal learned of poor results when people typed the name “audrey fino” into the search box. Google kept returning Italian sites praising Audrey Hepburn. (Fino means fine in Italian.) “We realized that this is actually a person’s name,” Singhal says. “But we didn’t have the smarts in the system.”

The Audrey Fino failure led Singhal on a multiyear quest to improve the way the system deals with names — which account for 8 percent of all searches. To crack it, he had to master the black art of “bi-gram breakage” — that is, separating multiple words into discrete units. For instance, “new york” represents two words that go together (a bi-gram). But so would the three words in “new york times,” which clearly indicate a different kind of search. And everything changes when the query is “new york times square.” Humans can make these distinctions instantly, but Google does not have a Brazil-like back room with hundreds of thousands of cubicle jockeys. It relies on algorithms.

The Mike Siwek query illustrates how Google accomplishes this. When Singhal types in a command to expose a layer of code underneath each search result, it’s clear which signals determine the selection of the top links: a bi-gram connection to figure it’s a name; a synonym; a geographic location. “Deconstruct this query from an engineer’s point of view,” Singhal explains. “We say, ‘Aha! We can break this here!’ We figure that lawyer is not a last name and Siwek is not a middle name. And by the way, lawyer is not a town in Michigan. A lawyer is an attorney.”

This is the hard-won realization from inside the Google search engine, culled from the data generated by billions of searches: a rock is a rock. It’s also a stone, and it could be a boulder. Spell it “rokc” and it’s still a rock. But put “little” in front of it and it’s the capital of Arkansas. Which is not an ark. Unless Noah is around. “The holy grail of search is to understand what the user wants,” Singhal says. “Then you are not matching words; you are actually trying to match meaning.”

And Google keeps improving. Recently, search engineer Maureen Heymans discovered a problem with “Cindy Louise Greenslade.” The algorithm figured out that it should look for a person — in this case a psychologist in Garden Grove, California — but it failed to place Greenslade’s homepage in the top 10 results. Heymans found that, in essence, Google had downgraded the relevance of her homepage because Greenslade used only her middle initial, not her full middle name as in the query. “We needed to be smarter than that,” Heymans says. So she added a signal that looks for middle initials. Now Greenslade’s homepage is the fifth result.

At any moment, dozens of these changes are going through a well-oiled testing process. Google employs hundreds of people around the world to sit at their home computer and judge results for various queries, marking whether the tweaks return better or worse results than before. But Google also has a larger army of testers — its billions of users, virtually all of whom are unwittingly participating in its constant quality experiments. Every time engineers want to test a tweak, they run the new algorithm on a tiny percentage of random users, letting the rest of the site’s searchers serve as a massive control group. There are so many changes to measure that Google has discarded the traditional scientific nostrum that only one experiment should be conducted at a time. “On most Google queries, you’re actually in multiple control or experimental groups simultaneously,” says search quality engineer Patrick Riley. Then he corrects himself. “Essentially,” he says, “all the queries are involved in some test.” In other words, just about every time you search on Google, you’re a lab rat.

This flexibility — the ability to add signals, tweak the underlying code, and instantly test the results — is why Googlers say they can withstand any competition from Bing or Twitter or Facebook. Indeed, in the last six months Google has made more than 200 improvements, some of which seem to mimic — even outdo — the offerings of its competitors. (Google says this is just a coincidence and points out that it has been adding features routinely for years.) One is real-time search, eagerly awaited since Page opined some months ago that Google should be scanning the entire Web every second. When someone queries a subject of current interest, among the 10 blue links Google now puts a “latest results” box: a scrolling set of just-produced posts from news sources, blogs, or tweets. Once again, Google uses signals to ensure that only the most relevant tweets find their way into the real-time stream. “We look at what’s retweeted, how many people follow the person, and whether the tweet is organic or a bot,” Singhal says. “We know how to do this, because we’ve been doing it for a decade.”

Along with real-time search, Google has introduced other new features, including a service called Goggles, which treats images captured by users’ phones as search queries. It’s all part of the company’s relentless march toward search becoming an always-on, ubiquitous presence. With a camera and voice recognition, a smartphone becomes eyes and ears. If the right signals are found, anything can be query fodder.
Google’s massive computing power and bandwidth give the company an undeniable edge. Some observers say it’s an advantage that essentially prohibits startups from trying to compete. But Manber says it’s not infrastructure alone that makes Google the leader: “The very, very, very key ingredient in all of this is that we hired the right people.”

By all standards, Qi Lu qualifies as one of those people. “I have the highest regard for him,” says Manber, who worked with the 48-year-old computer scientist at Yahoo. But Lu joined Microsoft early last year to lead the Bing team. When asked about his mission, Lu, a diminutive man dressed in jeans and a Bing T-shirt, pauses, then softly recites a measured reply: “It’s extremely important to keep in mind that this is a long-term journey.” He has the same I’m-not-going-away look in his eye that Uma Thurman has in Kill Bill.
Indeed, the company that won last decade’s browser war has a best-served-cold approach to search, an eerie certainty that at some point, people are going to want more than what Google’s algorithm can provide. “If we don’t have a paradigm shift, it’s going to be very, very difficult to compete with the current winners,” says Harry Shum, Microsoft’s head of core search development. “But our view is that there will be a paradigm shift.”

Still, even if there is such a shift, Google’s algorithms will probably be able to incorporate that, too. That’s why Google is such a fearsome competitor; it has built a machine nimble enough to absorb almost any approach that threatens it — all while returning high-quality results that its competitors can’t match. Anyone can come up with a new way to buy plane tickets. But only Google knows how to find Mike Siwek.
Senior writer Steven Levy (steven_levy@wired.com) wrote about Twitter in issue 17.11.

 Source: http://www.wired.com/magazine/2010/02/ff_google_algorithm

Vince Vaughn and Owen Wilson star as ex-salesmen who defy the odds by talking their way into a coveted internship at Google.

How Google Closed Source Wanted To Change The Destiny Of Humanity By Changing Your DNA Information During 2003

Change In The DNA 3: What Happened To My Site On Google?

Danny Sullivan, December 6, 2003 0 Comments

As I explained earlier this week, a significant change to Google's ranking algorithm has caused some web sites to lose top positions for some search terms. The outcry from affected site owners has been unprecedented, in my opinion. In this article, I'll take a Q&A-style approach to examine many of the issues and questions that have arisen from the change. Also be sure to see other articles about the recent changes on the Florida Google Dance Resources page.

• My page no longer comes up tops at Google for a particular search term. Why not?
• Why does/did running the filter test bring my site back into the top results?
• Has Google done this to force people to buy ads?
• Is there REALLY no connection with ads and free listings at Google?
• Does Google have a "dictionary" of "money terms" it uses to decide when to filter out some web sites?
• How can Google be allowed to hurt my business in this way?
• I heard Google's dropping pages that show signs of search engine optimization. Do I need to deoptimize my web pages?
• Does the filter test indicate that I've spammed Google?
• Does this mean Google no longer uses the PageRank algorithm?
• I thought the Google Dance was over, that the massive monthly update of pages had been replaced by a consistent crawl?
• If we remove our shopping cart, could that help us get back on Google, even though we'd be booted off Froogle?
• Can you get on a soapbox about all these Google changes?

Please note that the longer version of this article for Search Engine Watch members also cover these questions:

• Does Google favor some large sites like Amazon because of partnerships it has?
• Does being in AdSense help sites rank better?
• Can I sue Google for being dropped?

Q. My page no longer comes up tops at Google for a particular search term. Why not?

Google, like all search engines, uses a system called an algorithm to rank the web pages it knows about. All search engines make periodic changes to their ranking algorithms in an effort to improve the results they show searchers. These changes can cause pages to rise or fall in rank. Small changes may produce little ranking differences, while large changes may have a dramatic impact.

Google made a change to its algorithm at the end of last month. This fact is obvious to any educated search observer, plus Google itself confirms it. The change has caused many people to report that some of their pages fell in ranking. These pages no longer please Google's algorithm as much as in the past.

If your page has suddenly dropped after being top ranked for a relatively long period of time (at least two or three months), then it's likely that your page is one of those no longer pleasing the new Google algorithm. Running what's called the filter test may help confirm this for you, at least in the short term.

Keep in mind that while many pages dropped in rank, many pages also consequently rose. However, those who dropped are more likely to complain about this in public forums that those who've benefited from the move. That's one reason why you may hear that "everyone" has lost ranking. In reality, for any page that's been dropped, another page has gained. In fact, WebmasterWorld is even featuring a thread with some comments from those who feel the change has helped them.

Q. Why does/did running the filter test bring my site back into the top results?

My belief is that Google, for the first time, has been using two significantly different algorithms at the same time. The "new" system has been used for many queries since the change, but some queries were still handled by the "old" system. More and more queries now appear to be processed by the new system, suggesting that the old algorithm is being phased out entirely.

Why would Google run two different systems? My ideas are covered more in the Speculation On Google Changes article for Search Engine Watch members. The short answer is that I think the new system requires much more processing power than the old one. If so, then Google probably applied it initially to "easy" queries, such as those that didn't involve the exclusion or "subtraction" of terms

Why are more and more "hard" queries now going through the new system? It could be that Google was testing out the new system on easier queries and then planned to slowly unleash it on everything.
Alternatively, Google may have intended to run two algorithms all along but is being forced to abandon that plan because of the furor as site owners who've lost rankings use the filter test to see what they consider to be "old" Google.

Since it was discovered, the filter test has been used by hundreds, if not thousands of webmasters. These queries are processor intensive. They also have created an embarrassing situation Google has never faced before, where anyone can compare what looks to be "old" versus "new" results to show how the old results are better. Sometimes the new results might be better, of course -- but it's the mistakes in relevancy that get the most attention. They can be used as proof that new Google is worse than old Google.

As a result, Google may have ultimately decided that it needs to bring all queries into the new system -- if only to plug a "hole" it may have never anticipated opening into how it works internally.

Google won't confirm if it has been using two algorithms simultaneously. I can only tell you I've spoken with them at length about the recent changes, and that they've reviewed the article your reading now.

Whether you choose to believe my speculation or instead the idea that Google has employed some type of "filter" almost makes no difference. The end result is the same. For some queries, there are now dramatic difference from what "old" Google was showing.

Q. Has Google done this to force people to buy ads?

Some feel Google has dropped their sites to make them buy ads. In the short term, purchasing ads will be the only way they can be found. For some, it may even be the only long-term solution. In either case, it means more money for Google.

However, there's also plenty of evidence of people who, despite being advertisers, lost their "free" top rankings. There are also people who've never run ads that continue to rank well. This makes it difficult for anyone to conclusively say that this change was ad driven.

Google completely denies charges it's trying to boost ad sales. The company says the algorithm change was done as part of its continual efforts to improve results. Google has always said that there is no connection between paying for an ad and getting listed in its "free" results.

In my view, there are far easier ways that Google could boost ad revenue uptake without doing sneaky, behind-the-scene actions -- which is why I tend to believe this is not why the change happened.

For instance, Google could make the first five links on a page -- rather than the first two links -- be paid ads for certain queries. They might also make this happen for terms determined to be commercial in orientation and offer up a defense that they've determined the commercial intent of the query is strong enough to justify this.

Q. Is there REALLY no connection with ads and free listings at Google?

In terms of boosting rankings, yes, I believe this doesn't happen at Google. Neither does Andrew Goodman, in his recent article about the Google changes. Other serious observers of search engines I know also doubt this, though certainly not all. Those in the "I believe" camp feel Google would simply risk too much in the long-term for any short-term gains it might get.

In terms of listing support, buying ads may be helpful. Some who spend a lot on paid listings at Google have reported success in getting their ad reps to pass along problems about their entirely separate free listings to Google's engineering department for investigation.

To some degree, this is like a backdoor for fast support. Those who aren't spending with Google's AdWords program have no such speedy solution to getting an answer back. Google has continually rejected suggestions that it should offer a "listing support" or paid inclusion program, saying it fears this might be seen as establishing a link between payment and its free results. For a deeper exploration of this, see my article for Search Engine Watch members from last year, Coping With Listing Problems At Google.

For the record, Google flatly denies that those who are advertising get more access. The company says it takes feedback from many sources, and every report is assessed for how it might have an impact on search quality.

Indeed, it's important to note that Google does provide another backdoor that plenty of non-advertisers have made use of. This is the WebmasterWorld.com forum site, where public and private messages to "GoogleGuy," a Google employee monitoring discussions, have been acted upon.

Google also turns out to various search engine conferences, such as the Search Engine Strategies show produced by Search Engine Watch that begins in Chicago on Tuesday. Google provides assistance to those with questions at these type of conferences, as well.

Google also offers a front door in the form of email addresses it publishes. Yes, expect you'll likely get a canned response to many queries. However, people do get some more personal investigation, as well.
It's also crucial to make the HUGE distinction between listing support and rank boosting. Investigating why a page may not be listed at all (rather than ranking well) is an appropriate activity for Google or any search engine. Boosting the rank of a particular page in return for payment, and not disclosing this, is not acceptable.

Q. Does Google have a "dictionary" of "money terms" it uses to decide when to filter out some web sites?

This theory has emerged as people have run the filter test and discovered that for some queries, Google will show many more changes than for others. The Scroogle hit list provides a long look at examples like this. It reflects 24 hours worth of queries various people have tried at Scroogle to see if they've declined in the new ranking algorithm. Terms that had many changes are at the top of the list.

For example, earlier this week the Scroogle hit list showed that the top 99 of 100 results in a search for christmas present idea at Google were different under the new algorithm compared to the old. That's not entirely accurate, as explained more in my previous article. But overall, it's close enough. For that query, things have radically changed. The same was true for terms such as diet pill and poker gambling, both of which could be considered highly commercial in nature.

That's where the idea of there being "money terms" comes out of. Sites aiming to rank well for these terms may be expecting to make money. Some believe Google has thus decided to filter out some of these sites -- particularly the ones showing an intent to optimize their pages for Google and which are not major commercial entities -- and force them into buying ads.

It's a compelling theory. However, there are also commercial terms that showed little change, such as christmas time, books, sharp ringtones and games. The hit list is also compiled by those who are checking their own terms. As you might expect, that means it will be heavily skewed toward commercial queries. If a bunch of librarians entered a mass of non-commercial terms, there might have been some dramatic changes seen for that class of queries, as well.

In fact, a search for 1 2 3 4 5 6 7 8 was on the Scroogle hit list, someone obviously trying to test what happens with non-commercial searches. It came up with a score of 36 dropped pages. That's high enough to make you think that phrase might be in the "money list" dictionary, yet nothing about it appears commercial in nature.

There's no doubt the new algorithm does seem to have impacted many commercial queries very hard, in terms of the amount of change that's been seen. However, this seems more a consequence of how the new algorithm works rather than it coming into play only for certain terms. In other words, new criteria on how much links should count, whether to count particular links, when to count anchor text more (text in a hyperlink) and even what's considered spam probably have more impact on commercially-oriented queries.
It is possible that Google is also making use of its AdWords data. It wouldn't be difficult to examine what terms attract a lot of earning and use that data to make a list or even to feed the new algorithm.
For its part, Google won't confirm whether it is using some type of list or not.

In the end, whether there's a predefined list of terms or this is something happening just as a consequence of the new algorithm is moot. The final result is the same -- many sites that did well in the past are no longer ranking so highly, leaving many feeling as if they've been targeted.

Q. How can Google be allowed to hurt my business in this way?

There's no end of people complaining how they're losing business because Google is no longer sending them traffic for free. The painful lesson to be learned is that it's foolish to assume that any search engine will deliver traffic for free.

Back before we had paid listings, one of my top search engine optimization tips was not to depend solely on search engines. They have always been fickle creatures. Today's cries about Google and lost traffic are certainly the worst I've ever heard. But I can remember similar complaints being made about other major search engines in the past, when algorithm changes have happened. WebmasterWorld.com even has a good thread going where people are sharing past memories of this.

We do have paid listings today, of course. That means you can now depend on search engines solely for traffic -- but only if you are prepared to buy ads.

As for free listings, these are the search engine world's equivalent of PR. No newspaper is forced to run favorable stories constantly about particular businesses. It runs the stories it decides to run, with the angles it determines to be appropriate. Free listings at search engines are the same. The search engines can, will and have in the past ranked sites by whatever criteria they determine to be best. That includes all of the major search engines, not just Google.

To me, the main reason Google's changes are so painful is because of the huge amount of reach it has. Google provides results to three of the four most popular search sites on the web: Google, AOL and Yahoo. No other search engine has ever had this much range, in the past. Go back in time, and if you were dropped by AltaVista, you might still continue to get plenty of free traffic from other major search engines such as Excite or Infoseek. No one player powered so many important other search engines, nor were typical web sites potentially left so vulnerable to losing traffic.

The good news for those who've seen drops on Google is that its reach is about to be curtailed. By the middle of January, it will be Yahoo-owned Inktomi results that are the main "free" listings used by MSN. Sometime early in next year, if not earlier, I'd also expect Yahoo to finally stop using Google for its free results and instead switch over to Inktomi listings.

When these changes happen, Google will suddenly be reduced from having about three quarters of the search pie to instead controlling about half. That means a drop on Google won't hurt as much.
Inktomi will have most of the other half of that pie. Perhaps that will be better for some who were recently dropped in ranking at Google. However, it's possible they'll find problems with Inktomi, as well.
In the past, I've heard people complain that paid inclusion content with Inktomi gets boosted or that crawling seems curtailed to force them into paid inclusion programs. Those complaints have diminished primarily because Inktomi's importance has diminished. Indeed, when Inktomi changed its algorithm in October, there were some negative impacts on site owners that surfaced. However, those concerns were hardly a ripple compared to the tidal wave of concern over Google. Once Inktomi's importance returns, so will likely a focus on any perceived injustices by Inktomi.

Q. I heard Google's dropping pages that show signs of search engine optimization. Do I need to deoptimize my web pages?

If you absolutely know you are doing something that's on the edge of spam -- invisible text, hidden links or other things that Google specifically warns about -- yes, I would change these.
Aside from that, I'd be careful about altering stuff that you honestly believe is what Google and other search engines want. In particular, I would continue to do these main things:

• Have a good, descriptive HTML title tag that reflects the two or three key search phrases you want your page to be found for.
• Have good, descriptive body copy that make use of the phrases you want to be found for in an appropriate manner.
• Seek out links from other web sites that are appropriate to you in content

Should you start removing H1 text around copy? Drop comment tags that are loaded with keywords? Cease doing other specific things you've heard might help with search engines. If you put these there only because you thought it helped with search engines, then perhaps. It wasn't natural to do this, and Google potentially could seek such indicators to determine you have an overly optimized page.

I almost hesitate to write the above. That's because I'm fearful many people will assume that some innocent things they may have done are hurting them on Google. I really don't feel that many people have dropped because Google is suddenly penalizing them. Instead, I think it's more a case that Google has done a major reweighing of factors it uses, in particular how it analyzes link text. In fact, that's exactly what Google says. Most changes people are seeing are due to new ranking factors, not because someone has suddenly been seen to spam the service, the company tells me.

Should you start asking sites to delink to you, or to drop the terms you want to be found for from the anchor text of those links? Some have suggested this. If these sites have naturally linked to you, I wouldn't bother. Links to you shouldn't hurt. In fact, the biggest reason for a lot of these changes is likely that links are simply being counted in an entirely new way -- and some links just may not count for as much.

Should you not link out to people? Linking out is fine in my view and should only hurt you if you are linking to perhaps "bad" sites such as porn content. Do that, and you could be associated with that content.
It's also a good time for me to repeat my three golden rules of link building:

1. Get links from web pages that are read by the audience you want.
2. Buy links if visitors that come solely from the links will justify the cost.
3. Link to sites because you want your visitors to know about them.

None of these rules involve linking for purely search engine reasons -- and so doing them should keep you on the right path in terms of getting appropriate links, I feel.

Q. Does the filter test indicate that I've spammed Google?

No, just because your site no longer ranks so highly on Google does not necessarily mean that you've spammed Google. Instead, it most likely means that some of the many factors Google uses to rank web pages have been adjusted -- and you no longer do so well with these. In other words, you haven't done anything wrong. It's simply that the scoring criteria has changed.

Think about it like a test. Let's say that in this test, people were judged best primarily on how they answered a written question, but multiple choice and verbal portions of the test also counted. Now the criteria has changed. The verbal portion counts for more, and you might be weaker in this area. That means someone stronger might do better in the test. You aren't doing worse because of any attempt to "cheat" but simply because the criteria is different.

Q. Does this mean Google no longer uses the PageRank algorithm?

Google never used the PageRank algorithm to rank web pages. PageRank is simply a component of that overall algorithm, a system Google uses to measure how important a page is based on links to it. It has always -- ALWAYS -- been the case that the context of links to the page was also considered, as well as the content on the page itself.

Unfortunately, some writing about Google have called its system of ranking PageRank, and Google itself sometimes makes this mistake, as seen in its webmaster's information page:

The method by which we find pages and rank them as search results is determined by the PageRank technology developed by our founders, Larry Page and Sergey Brin.
In reality, the page describing Google's technology more accurately puts PageRank at the "heart" of the overall system, rather than giving the system that overall name.

By the way, PageRank has never been the factor that beats all others. It's has been and continues to be the case that a page with low PageRank might get ranked higher than another page. Search for books, and if you have the PageRank meter switched on in the Google Toolbar, you'll see how the third-ranked Online Books Page with a PageRank of 8 comes above O'Reilly, even though O'Reilly has a PageRank of 9. That's just one quick example, but I've seen others exactly like this in the past, and you can see plenty first-hand by checking yourself.

Q. I thought the Google Dance was over, that the massive monthly update of pages had been replaced by a consistent crawl?

To some degree, the Google Dance had diminished. Historically, the Google Dance has been the time every month when Google updated its web servers with new web pages. That naturally produced changes in the rankings and so was closely monitored. Sometimes, an algorithm change would also be pushed out. That could produce a much more chaotic dance.

Since June, life has been mercifully quiet on the dance front. Google has been moving to refresh more of its database on a constant basis, rather than once per month. That's resulted in small changes spread out over time.

Google says that continual updates are still happening. The dance came back not because of a return to updating all of its servers at once but rather because of pushing out a new ranking system.

Q. If we remove our shopping cart, could that help us get back on Google, even though we'd be booted off Froogle?

This question coincidentally came in just after I saw Google implement Froogle links in it search results for the first time. Talk about timing!

No, removing your shopping cart really shouldn't have an impact on your regular Google web page rankings. Lots of sites have shopping carts. It's perfectly normal to have them.

As you also note, having an online shopping service means you have data to feed Google's shopping search engine Froogle. And Froogle's now hit Google in a big way. If Froogle has matches to a query, then Froogle links may be shown above web page matches at Google.

It happens similar to the way you may get news headlines. Search for iraq, and you'll see headlines appear above the regular web listings next to the word "News." If you search for a product, then you may see similar links appear listing product information from Froogle, next to the words "Product Search."

Google unveiled the new feature late Friday, and it's to be rolled out over this weekend, the company tells me. A formal announcement is planned for next week, and Search Engine Watch will bring you more about this.

In the meantime, anyone who's been dropped by Google in its regular web search results should seize upon Froogle as a potential free solution to getting back in. Froogle accepts product feeds for free -- see its Information For Merchants page for more. And since Froogle listing are now integrated into Google's pages, it means you can perhaps regain visibility this way.
For more about Froogle, see these past articles from Search Engine Watch:

• Online Shopping with Google's Froogle
• Getting Listed In Google's "Froogle" Shopping Search Engine (for Search Engine Watch members)

Q. Can you get on a soapbox about all these Google changes?

Sure. Let me start with something one of my readers emailed:
I truly believe that Google has done us wrong. We worked hard to play by the rules, and Google shot us in the back of the head.

That comment is typical of many you see in the forums. Many people are mystified as to why they are suddenly no longer deemed good enough by Google, especially if they had been doing well for a long period of time and feel they played by the "rules."

Yes, free listings aren't guaranteed. Yes, search engines can do what they want. Yes, it's foolish for anyone to have built a business around getting what are essentially free business phone calls via Google.

None of that helps the people feeling lost about what to do next. Many have been dropped but may see sites similar to theirs still making it in. That suggests there's a hope of being listed, if they only understood what to do. So what should they do? Or what shouldn't they be doing?

My advice is unchanged -- do the basic, simple things that have historically helped with search engines. Have good titles. Have good content. Build good links. Don't try to highly-engineer pages that you think will please a search engine's algorithm. Focus instead on building the best site you can for your visitors, offering content that goes beyond just selling but which also offers information, and I feel you should succeed.

Want some more advice along these lines? Brett Tabke has an excellent short guide of steps to take for ranking better with Google, though I think the tips are valid for any search engine. Note that when GoogleGuy was recently asked in a WebmasterWorld members discussion what people should do to get back in Google's good graces, he pointed people at these tips.

I Did That -- And Look At How It Hasn't Helped!
Unfortunately, some believe they've followed these type of tips already. Indeed, one of the nice things about Google's growth over the past three years is that it has rewarded webmasters who have good content. As they've learned this, we've seen a real shift away from people feeling they need to do what's often dubbed "black hat" techniques such as targeted doorway pages, multiple mirror sites and cloaking.

That's why it's so alarming to see the sudden reversal. Some people who believe they've been "white hat" now feel Google's abandoned them. Perhaps some have not been as white hat as they thought, but plenty are. Many good web sites have lost positions on Google, and now their owners may think they need to turn to aggressive tactics. This thread at WebmasterWorld is only one of several that show comments along these lines.

Maybe the aggressive techniques will work, and maybe not. By my concern is really reserved for the mom-and-pop style operations that often have no real idea what "aggressive" means. To them, aggressive means that they think they need to place H1 tags around everything, or that every ALT tag should be filled with keywords, or that they should use the useless meta revisit tag because somewhere, somehow, they heard this was what you need to do.

More Openness From Google
One thing that would help is for Google to open up more. It has a new ranking system, obviously. It should be trumpeting this fact and outlining generally what some of these new mystery "signals" are that it is using to help determine page quality and context.

Google can provide some additional details about how it is ranking pages in a way that wouldn't give away trade secrets to competitors nor necessarily give some site owners a better ability to manipulate its listings. Doing so would make the company look less secretive. It might also help explain some of the logic about why sites have been dropped. That would help readers like this:

What really concerns me right now is that there doesn't appear to be any rhyme or reason as to why some sites have a good ranking and what we could do to improve our rankings.

Maybe Google has decided that it makes more sense to provide informational pages on certain topics, because otherwise its listings look the same as ads (see the honeymoon case study for an example of this).
If so, that's fine. It can defend this as helping users, ensuring they have a variety of results. But at least the declaration that it is doing so will let site owners understand that they may need to create compelling informational content, not sales literature. They may also realize that they simply are not going to get back free listings, for some terms. With that understanding, they can move on to ads or other non-search promotional efforts.

Searchers Want To Know, Too
Google doesn't just need to explain what's going on to help webmasters and marketers. Most important, some of Google's searchers want to know how it works behind the scenes.

Google has set itself up almost as a Consumer Reports of web pages, effectively evaluating pages on behalf of its searchers. But Consumer Reports publishes its testing criteria, so that readers can be informed about how decisions are made. It's essential that Google -- that any search engine -- be forthcoming in the same manner.

To its credit, Google has given out much information. There's a huge amount published for webmasters, and even more is shared through forums and conferences. But if Google is now doing things beyond on-the-page text analysis and link analysis that it has publicly discussed, it needs to share this so searchers themselves can be more informed about how decisions are reached.

Right now, some of these searchers are reading news reports that a search for miserable failure brings up US president George W. Bush's biography as the top result. They'll want to understand why. Is Google calling Bush a miserable failure? Is this an example of Google's "honest and objective way to find high-quality websites with information relevant to your search," as its technology page describes?

The answer to both question is no. Google Bombing has made that biography come up first, and those doing the bombing have no "objective" intentions behind it. They think Bush is a failure, and they are using Google as a means to broadcast that view.

Does this mean Google is a miserable failure as a search engine? No. Ideally, Google should have caught such an overt attempt to influence its rankings, and it's notable that this got past even its new ranking system. However, Google is not perfect, nor will it ever be. Fortunately, searchers seeing a listing like that can understand why it came up if they understand a bit about how link analysis works. That helps them better evaluate the information they've received.

Now go search for christmas at Google. I bet plenty of searchers are wondering why, like my colleague Gary Price of ResourceShelf who reported this to me, Marylaine Block's web site is ranked sixth for christmas out of 36 million possible web pages?

Block's not sure herself. Links may have something to do with it, but so might some of these new "signals" about page quality and content of which Google cannot speak. Since Google's not talking, we can't understand -- and crucially -- forgive when it makes mistakes.

Marketer Reality Check
Having dumped on Google, it's also important that webmasters and marketers understand that Google is never going to outline exactly how it works. No popular search engine will ever do this, because the volume of successful spam that would result would bring the search engine to its knees.
Marketers also have to recognize that Google and other search engines will continue altering their ranking systems, just as they always have done -- and that listings will change, sometimes dramatically, as a result.
Whether Google and the others discuss openly how they work or not, people eventually discover new ways to be successful with spam. That has to be fought.

More important, the nature of search keeps changing. Links were a useful "signal" to use and one that gave the relevancy of web crawling a new lease on life several years ago. Now linking is different. Blogs link in a way that didn't exist when Google launched. Reciprocal linking and link selling is much more sophisticated and often designed to take Google and search engines into account. These are just two reasons why the methods of analyzing links has to change.

It's also a certain fact that the most popular and lucrative real estate on a search engine is not going to continue to use web crawling as its first source of data. It simply makes more sense to go with specialized data sources when these are available. Web search's destiny is to be backfill for when these other forms of data fail to find matches.

Free traffic from web listings will inevitably decline as search engines make use of specialized data sources through invisible tabs. It won't go away entirely, and there's always going to be a need to understand "search engine PR" to influence free results. But smart marketers will realize that they need to look beyond web search to stay ahead.

If Google dropped you, Froogle just got a promotion as a new way to get back in. So, too, will other opportunities come up. The downside is, unlike Google -- or even Froogle -- they'll likely cost money. Smart businesses will realize they need to budget for this, just as they budget for advertising and to obtain leads in the real world. It's the rare and exceptional company that can get by on PR alone -- even the UK's popular Pizza Express chain had to diversify into advertising.

---

Become an Expert Digital Marketer at SES New York
March 25-28, 2013: With dozens of sessions on Search, Social, Local and Mobile, you'll leave SES with everything and everyone you need to know. Hurry, early bird rates expire February 21. Register today!

Source: http://searchenginewatch.com/article/2066309/What-Happened-To-My-Site-On-Google

Change In The DNA 4: Google - Update "Cassandra" is here

1. Brendon Scott Senior SEO at Weboptimiser
   11 April 2003 11:57am
   Well, Google started updating late on the 10th April (UK time), and you can see the beginnings of the new database at www2 and www3.google.com. Look slike multiple links from the same site are being hit this time round, so we may see some cleaner results this time, as the "Damn the crosslinking spam filters! Full speed ahead!" crowd might take a real knock this month.
   
   As promised, Google are including previously banned sites which have mended their wicked ways, and are requesting that anyone else who wishes to be reconsidered sends an e-mail to webmaster@google.com, with a subject of "Reinclusion request", or similar, along with a confessional ("Receive my confession, O most loving and gracious GoogleGuy...") saying what you did, and how you've cleaned up. If you are going to do it, do it quickly, before the next deep crawl starts for maximum chance of reinstatement
   
   I personally would be a bit economical with the truth here; these requests are one of Googles best sources of information on how people spam them and exploit the more glaring weaknesses in their algorithm, and it wouldn't do for them to be able to find and fix ALL of them ;)
   
   Also, GoogleGuy dropped a BIG HINT :
   He is actively requesting spam reports on hidden text, and hidden links. If you are using these techniques, I would SERIOUSLY look at ways to remove/replace them, BEFORE the deep crawl comes around in a few days time (DeepBot comes from the 216.* block, don't confuse it with the FreshBot out of the 64.* block. Both report themselves as GoogleBot, but its Ms. DeepBot who you need to court most)
   
   So, there we have it, Google update "Cassandra", April 2003. Luck to you all
   
   Source: http://econsultancy.com/us/forums/other-topics/google-update-cassandra-is-here
   
   Change5:
   
   

   WebGuerrilla msg:160169

   12:38 am on May 16, 2003 (gmt 0)

   Earlier this week, albert [webmasterworld.com] started a thread called “Understanding Dominc [webmasterworld.com].” In that thread, he compiled a list of all of GoogleGuy’s recent comments concerning update dominic. The excerpts were a big help for many of our members. However, the following discussion quickly ended up much like the various threads that albert originally had to dig through in order to compile his list. Trying to wade through all the various update threads has clearly caused some frustration [webmasterworld.com] for many of our members. So in order to try and alleviate some of that frustration, I thought we might try a new version of albert’s original post. This version includes all of albert’s original excerpts. It will also be updated periodically. However, we are going to keep this thread “read only.” Anyone interested in following GoogleGuy’s contributions to the current update discussions can bookmark or flag this thread. If/when we post updates, this thread will appear on the active list. If you have any questions or comments relating to anything in this thread, I would greatly appreciate it if you post them in one of our current update threads. Update dominic – Part 9 [webmasterworld.com] Or Serious Google update analysis thread [webmasterworld.com] WG SJ started testing a new index with a sightly different build of backlinks. [webmasterworld.com...] GoogleGuy msg #44 May 5 rfgdxm1, every index has to pass a really stringent battery of tests before it becomes visible. SEOs might notice a slightly different build of backlinks, but things like that could be balanced by factors that improve search quality more in other areas. The other thing to bear in mind is that it's easy to re-sync something like backlinks or spam snapshots once you're convinced that an algorithm or method is an improvement. SJ results will show up at other data centers soon [webmasterworld.com...] GoogleGuy msg #107 May 5 Critter, it wouldn't surprise me to see SJ results start to show up at other data centers soon. Test of new method with a known base of backlinks, bringing in more up-to-date backlink and spam info later on [webmasterworld.com...] GoogleGuy msg #134 May 5 Traveler, good question. From the first few posts of that 500+ thread, several people mentioned that they have some very new results in SJ. It's natural that we would test new methods by using a known base of backlinks, but that shouldn't be discouraging to people--backlinks are the sort of data that Google could bring back in over a relatively short time frame. And the same thing goes for known snapshots of spam--that can be brought in fairly quickly as well. SEOs notice whether a backlink comes from two months ago or one month ago, but typical users would care more about fresher pages. SJ index is not old [webmasterworld.com...] GoogleGuy msg #146 May 5 Critter, the SJ index isn't an older index. You can verify that by doing a topical query such as SARS. The results are more fresh in SJ than they are in our regular index. About backlinks from forums [webmasterworld.com...] GoogleGuy msg #160 May 5 Critter, if it's the site listed in your profile, it looks like you only have 5-6 domains that link to your site. A few of those are forum links that might not have made it into the base of backlinks. Getting links from places like the Open Directory Project would help, for example. About guestbook links [webmasterworld.com...] GoogleGuy msg 165 May 5 Much more likely that those guestbook links just aren't given weight now, rfgdxm1. Backlinks and spam snapshots will be added later [webmasterworld.com...] GoogleGuy msg 178 May 5 mcavic, I think I did say that newer backlinks and spam snapshots would be pending to be applied over time. Or at least I tried to. :) SJ data will show up at other data centers first, new data / filters after that [webmasterworld.com...] GoogleGuy msg #298 May 5 albert, what you said, except I wouldn't be surprised to see SJ show up at other data centers first, and then to start applying the newer data/filters after that. Less Backlinks for all sites [webmasterworld.com...] GoogleGuy msg #108 May 6 [i]Don't be alarmed if the number of reported backlinks goes down. That's actually to be expected in the update. Most of it affects all sites uniformly, so it comes out in the wash as being equal. The better way to measure it is how your rankings/traffic change. SJ results will show up at other data centers GoogleGuy msg #43 May 6 What rfgdxm1 said. I think you'll see SJ results appear at more data centers over time. Nothing new. [webmasterworld.com...] GoogleGuy msg #11 May 8 I'm still hanging around. There's not that much new info to convey, but I'm here. A few more backlinks were added [webmasterworld.com...] GoogleGuy msg #197 May 10 I think we added a few more backlinks in yesterday. I'm assuming people have read HitProf's thread on backlinks too? rfgdxm1, sorry to hear that you don't like the SJ index. I also checked your ingredient theory in your spam report. People had suggested that a long time ago at the GooglePlex, but that's not the primary addition for SJ. It was only a minor update (so far) [webmasterworld.com...] GoogleGuy msg 203 May 10 Twas a minor update in backlinks, MyWifeSays. I still expect SJ results to be seen at more data centers first. sj/fi index will shift to other data centers. After more backlinks and other data [webmasterworld.com...] GoogleGuy msg #52 May 14 webdev, I think almost all of these questions have been answered several hundred threads ago. As late as this morning, I posted saying that I expected sj/fi index data to make its way to other data centers and to various sites. Once that data appears more broadly, we'll gradually be pulling in more backlinks and applying other data. sj/fi data centers have been approved [webmasterworld.com...] GoogleGuy msg #59 May 14 steve128, the sj/fi data centers have been tested and approved. What I said several hundred posts ago was that you can expect an index like that to show up at more (and possibly all) data centers in the future. More pages and backlinks to be added [webmasterworld.com...] GoogleGuy msg #73 May 14 Ltribe, I expect more pages and backlinks to be brought in with time. Help spread the word ... [webmasterworld.com...] GoogleGuy msg#86 May 14 Maybe I'll just set things up to auto-post every 50 posts or so. :) Please help spread the word so people know what to expect and don't worry too much. SJ and/or FI - and other data centers [webmasterworld.com...] GoogleGuy msg #377 May 14 I wouldn't draw huge distinctions between sj and fi. When I say "I expect the sj index to spread to other data centers," that could be sj or fi. SJ and FI similar, but emphasizing different things like topicality - same with different data centers [webmasterworld.com...] Google Guy in this thread msg #16 May 14 There's a lot of backlinks on the web. :) It will take some time to bring them all in. To clarify a couple things: - it helps to think of sj and fi as similar. It's better conceptually to think of them as cut from the same cloth. - chiyo, every data center has different machine characteristics. So similar/identical indices might look slightly different at different data centers. This goes back to the point above. Don't think of it as if we build a different index with a different theme for different customers. Our partners get the same scoring/data that we use. That said, the global index that we build can emphasize different things more, such as topicality or more diverse file types. So sj/fi are different in several ways. I would expect that difference to spread to other data centers. Then things will resume moving forward. Hope that helps, GoogleGuy Different nature of SJ and FI is expected to spread to more data centers [webmasterworld.com...] Google Guy in this thread msg #28 May 14 trillianjedi, I'm saying that sj and fi are of a different nature than previous indices, and that I expect that different nature to spread to more data centers (ex, anyone? :) I've been using "sj" to denote this different nature, but I appreciate the chance to clarify. Measuring the size/usefulness of an index [webmasterworld.com...] GoogleGuy msg 4 May 14 There are lots of different ways to measure the size/usefulness of an index. Nice job to Allergic for noticing something that most people usually don't. :) Timeframe to bring in backlinks - and: should we watch sj/fi [webmasterworld.com...] GoogleGuy msg 43 May 14 x_m, you could always do that yourself by just not querying sj/fi. :) trillianjedi, just to clarify on the other point, I mentioned that backlinks could be brought in on a relatively short timeframe, but remember that we are talking about terabytes of data here--the web is a big chunk of data. I think I also replied to someone else at one point that bringing in those backlinks wasn't the sort of thing that could happen in a day or two. Hope that sheds more light on things

   
   

   WebGuerrilla msg:160170	9:45 pm on May 17, 2003 (gmt 0)
   	Backlink reduction due to improved estimation of link counts [webmasterworld.com ] GoogleGuy msg #7 May 9 HitProf, I really like your insights. I think our newer systems do a much better job of estimating link counts. -dc entered the game [webmasterworld.com ] GoogleGuy msg #2 May 17 Good deal. Sounds like things are on schedule. Recent indexing changes - Google gives no guarantee for static search results [webmasterworld.com ] GoogleGuy msg #3 May 17 Hollywood, we never promise that we're going to return the same static set of results for any length of time for any query. The only thing we promise is that we're going to try to return the best results that we can. I would not base your business deals on the assumption that Google search results will remain static for long periods of time. That's just not something that we promise. Bringing in backlinks is no overnight thing [webmasterworld.com ] GoogleGuy msg #2 May 17 Bringing in backlinks and other data will be a gradual process over time. It won't be an overnight thing once all the data centers have the sj-type index. Once that index is everywhere, I'm looking forward to us bringing in fresher data--but I want to set expectations that it will take some time. Timeframe is more than days, less than months. - Calm down, and remember last september [webmasterworld.com ] GoogleGuy msg #16 May 17 I would say more than days, less than months. That's just my personal take. A lot of people are paying attention to every microdetail at this point. I would say that stepping back a level of detail would give better insight and less stress. Suppose you're on a long bus trip. If you scrutinize the road for every bump, pebble, or sharp turn, you're going to be more stressed than you need to be. If people are newer to WebmasterWorld, I'd recommend going back to last September, when we improved our scoring. If you re-read those threads, you'll see the same sorts of reactions that you see now. People claimed the sky was falling. You saw 5-10 people making pretty alarmist claims as loudly as possible. People suggested that scoring changes were a secret attempt to boost AdWords sales. The imminent destruction of Google was predicted several times. A few personal attacks on GoogleGuy took place. :) We got through that, and re-reading those threads will give you a different perspective on these more recent threads. Take that, plus the fact that I've said 5-6 (9-10?) times that we'll be bringing in better filters and fresher data, and most people should feel better, I hope. We still use the scoring improvements from last September, but you don't hear people worrying about it now. I'm looking forward to doing a similar post for our next major improvement, whenever that comes: "You know, if you go back to May, you'll see several webmasters were worried about that change too. Would you believe that there were over 4000+ posts from people who were anxious about that change? ..." Anyway, each webmaster is free to do whatever they want, of course. I do think that people would feel less stressed if they took a step (or two) back though. I think you've seen that somewhat; many of the senior members of WebmasterWorld haven't been doing tons of posts in the dominic discussions. Spam showing up right now will be short-lived [webmasterworld.com ] GoogleGuy msg #24 May 17 tigger, sometimes weekends are easier for making it over here. :) Hey nutsandbolts, let's try something new. If you have feedback about this index, do a spam report with your nickname, mention webmasterworld, but also include "dominic" in the comments. If people have general feedback or specific examples about this index, that's the best way to get the examples to us. I'm not worried about spammers that were banned and are back for a brief time--that will be short-lived. But if you have comments about searches that seem better/worse (preferably not just searches for your own site), send a spam report with "dominic" somewhere in the comments. We'll read what people have to say when they can give specific searches. [edited by: Marcia at 3:39 pm (utc) on May 19, 2003] [edit reason] Modified nesting for formatting. [/edit]
   WebGuerrilla msg:160171	3:35 pm on May 19, 2003 (gmt 0)
   	Look at the overall picture. - Do you show well at sj/cw/fi and the like? - Spam will disappear again [webmasterworld.com ] GoogleGuy msg 39 May 17 Kirby, I'm not worried about spammers that have been handled before and appear to be back. Most of those will be gone again. But I do want to hear spam reports like "this type of search seems to work better or worse with the new system." worker, if I were playing Yoda like Alphawolf suggests, I'd say: worry less about PR on the toolbar and more about rankings. And less about rankings for high-profile phrases and more about overall rankings. And less about rankings and more about traffic. And less about traffic and more about conversions. Maybe that doesn't sound much like Yoda though. I think the shorter answer would be that if you're showing well at datacenters like sj/cw/fi, I wouldn't worry much about what the PR says. I just wanted to say thanks to the other posters who are taking a step back and looking at the overall picture. Please help to remind people that the longer-term view will keep folks less stressed and more productive. :) About micro-level views, larger landscapes, and a post showing deep understanding [webmasterworld.com ] GoogleGuy msg #54 May 17 teeceo, when bringing up a new system, you want to work from a known base of data. I fully expect that after that, we'll be working to bring in newer sites. And your anchortext, rfgdxm1. ;) FWIW, BigDave has (in my experience reading his posts--I don't know who he is) a deep understanding of Google's workings and perspective. I would give his comments as much consideration as you would mine. mrdch, I understand why webmasters may be anxious--mainly my post was meant to bring out a few points that people may have missed. WebmasterWorld is great for the micro-level view ("I dropped from #4 to #16, but only in cw!"), but not always for the larger landscape. On-page factors: Well optimized individual pages bring traffic from different important key phrases [webmasterworld.com ] GoogleGuy msg #78 May 17 (referring to msg #71) annej, I really wish every webmaster would do the log analysis that you just did. :) Still a lot of work left to do [webmasterworld.com ] GoogleGuy msg #109 May 18 Stefan, there was a lot of work at Google behind these changes. We're trying the make the transition as gentle as possible, but there's still a lot of work left to do. Specific feedback about improved or worsened searches welcome [webmasterworld.com ] Google Guy msg #181 May 18 (Remark: quote shortened) This is just my take, but I don't think it helps anybody to have folks call other people dancing monkeys, or post claiming "B£*LLSH!T" in all caps, or virtually jump out of windows. If people have constructive comments for this index, I gave a method back in msg #24 of giving us specific searches or types of searches that you consider good or bad. I just checked, and I don't mind telling you that so far it's a single digit number of reports. I'm guessing that number will go up at least some after this reminder post :) but if you have specific searches or suggestions to pass on, that's probably the best way to get them to us. Current status of update [webmasterworld.com ] GoogleGuy msg #205 May 18 (Remark: read accurately, also posts referred to) Sure, Anon27 and deanril. I think the plan will be 1) deploy the new index/system across all data centers 2) begin pulling in more data (i.e. newer backlinks, pages, and spam updates) 3) once new data is into the system, begin pulling in new algorithms that have been waiting in the wings I believe the current status is that we're around step 1.5 or thereabouts; something like 7 or so data centers have the new index/system. I expect the current pace of switching data centers to continue about as it has been. I would expect step two to occur over roughly the same timeframe as a typical index cycle (thus the "more than weeks, less than months" comment). Step 3 is longer-term and ongoing, but I'm really excited about what we'll be able to do to improve quality across the board. Hope that helps, GoogleGuy (Staza, that should help with your questions: yes, and I believe so to #1 and #2.) [edited by: Marcia at 3:44 pm (utc) on May 19, 2003] [edit reason] Formatting of nested elements. [/edit]
   WebGuerrilla msg:160172	8:31 pm on May 21, 2003 (gmt 0)
   	About hidden text [webmasterworld.com ] GoogleGuy msg #22 May 19 Thanks for the feedback. If people want to mention specific searches/sites, they can do a spam report and put "dominic" in the comments somewhere--and hopefully mention your nickname. Just today, I've fielded questions from two sites that had hidden text and were taken out for 30 days. One site owner knew that they had had hidden text and one didn't. If you had hidden text on your pages and you think that might be the problem, you'll want to double-check that all the hidden text is gone. The first set of hidden-text penalties will be expiring over the next week or two. Sounds like that probably isn't an issue, but it's something to bear in mind. Don't gun for #1 at "that major keyword" [webmasterworld.com ] GoogleGuy msg #36 May 20 paynt, feel free to use that. I think I'm paraphrasing something Danny Sullivan would say when I talk like that. :) Just the idea that paying more attention to users, what they're finding and what they want, rather than gunning for #1 on that "major keyword" is one thing that demonstrates a wise SEO. :) TLD / language, give specific comments to Google [webmasterworld.com ] GoogleGuy msg #31 May 20 Hey Napoleon, I think I've already commented on our SJ index at length, and given people some idea of what to expect. I believe the change for non-U.S. IP addresses is geared to allow users to separate language from TLD, and I believe that it was tested for several weeks on one TLD, with us watching for feedback. Given that it's a UI change on the home page, the amount of negative feedback has been very low--a good sign. I know that the UI team is interested in the best way to present international users with a choice of which TLD to use and which language to use. If you write to help@google.com with some subject line such as "UI for TLD/language," I'm sure it will reach open ears on the UI team--they want feedback on how to make things smoother for everyone. I'll be curious to read suggestions for how to make that UI better myself. As far as things like expired domains, changes such as SJ are exactly the sort of thing that will allow us to pull in better algorithms that can take full advantage of that data. Anyway, I hope that addresses some of your questions. We're always looking for feedback on what we can do better. In another thread, I gave a way to give specific feedback to Google about the SJ index: do a spam report with "dominic" in the comments. I've since reminded people about that method a couple times. Yet the number of reports via that form has been less than even the number of posts on this thread. After 3000+ posts about this index on WebmasterWorld, we're sitting at around 30 concrete suggestions about what's bad/good about the SJ index. By that measure, SJ is definitely an improvement over past indices. More changes to come [webmasterworld.com ] GoogleGuy msg #42 May 20 re msg #37 (shortened) NovaW, I would certainly expect to more changes like the ones you mentioned, which is a good thing. links=PR0 like red herring [webmasterworld.com ] GoogleGuy msg #45 May 20 Napoleon, the links = PR0 sounds like a red herring, but I'll be happy to root around and read it. :) Keep on doing what's right [webmasterworld.com ] GoogleGuy msg #11 May 20 Oh well. People knock on Google sometimes. If you just keep doing what you know is right, things seem to work out just fine. :) About link pages [webmasterworld.com ] GoogleGuy msg 158 May 20 Sorry, that's what I was trying to say earlier. :) Links pages don't get PR0, and we index them. Funny one about link farms [webmasterworld.com ] GoogleGuy msg 161 May 20 They're mostly located in the Midwest, I think. Lots of irrigation and sunshine to raise those little links. The links need time to mature, or else they aren't as good. Once the links are harvested, they're shipped throughout the world to different companies that make HTML editors, and those companies can feed the links right in. If you drive across the U.S., you can still see link farmers from time to time. Links will be 'organic' [webmasterworld.com ] GoogleGuy msg #170 May 20 walthamstow, just remember that you want all-organic links. :)
   WebGuerrilla msg:160173	9:28 pm on May 23, 2003 (gmt 0)
   	Phase 2 Begins Status: index is at all data centers. Subtle spam filter comes soon. Fresh data coming in next days. Timeframe: more than weeks, less than months. [webmasterworld.com ] GoogleGuy msg #53 May 23 Some fresh data might be incorporated next-day, but I wouldn't expect everything freshbot found yesterday to make it in a day. There's a lot of data that needs to be fetched and cross-checked--I would expect the full data to show up more on the timeframe of what you would expect from a crawl/index. Step 1 is done (index is at all data centers). I know that one subtle spam filter is going in soon, but Napoleon, I would start counting on the weeks-but-not-months comment beginning from the time that the index switchover finished. Again, just trying to give webmasters iinformation so they have the right expectations: more than weeks, less than months. But I'm glad that people have noticed that freshie has been crawling deeply in addition to normal freshie duties. Timeframe again. And changes expected for country language / TLD / redirection next week. [webmasterworld.com ] GoogleGuy msg #58 May 23 I think we're on the same wavelength, Napoleon. P.S. About the questions you raised about country language/tld/redirection, I think we've got some changes scheduled soon (next week) that should bring it in line with what many users expect again. Just wanted to let you know that that was coming too. Traditional update(s) expected for a little while longer. After, things will be more gradual. [webmasterworld.com ] GoogleGuy msg #65 May 23 crobb305, some things will be filtering in sooner (I know a few more autospam filters will happen earlier), and I wouldn't be surprised to see fluctuations in backlinks and pages, but I would hold on to the idea of an update that brings in more data for a little while longer. In time, I do think things will be more gradual. However, we're still in the transition period for this system, so I wouldn't be surprised to see a traditional update for a little while longer. Hope that helps.
   WebGuerrilla msg:160174	10:42 pm on May 23, 2003 (gmt 0)
   	At least another traditional update expected [webmasterworld.com ] GoogleGuy msg #81 May 23 Sure, reneewood. I would expect at least another update of the form where the crawl/index cycle finishes and then data centers are updated in the traditional dance. Next update: what will/should be brought in. Freshbot looks like Deepbot. [webmasterworld.com ] GoogleGuy msg #88 May 23 dnbjason, I've learned never to make promises, but I would say the odds are good. :) rfgdxm1, more anchor text, backlinks, spam filters, etc. should be brought in by the next update. I think someone else remarked that freshbot is looking more like deepbot these days. Definitely expect newer data affecting ranking [webmasterworld.com ] GoogleGuy msg #91 May 23 I would definitely expect newer data that we bring in to affect rankings.
   WebGuerrilla msg:160175	10:30 pm on May 25, 2003 (gmt 0)
   	About toolbar, status of index, and backlinks [webmasterworld.com ] GoogleGuy msg #43 May 23 MurphyDog and notsleepy, I wouldn't worry. I've seen at least a couple grey bars for well-known sites, so I'm asking around about that. I wouldn't worry about the toolbar display during this transition. shaoye, the index includes many new pages and new backlinks, but also includes an older snapshot of backlinks for now as well. What exactly should happen with next update [webmasterworld.com ] GoogleGuy msg #102 May 23 Let me clarify. The next update should bring in more backlinks, data, etc. Those dang prepositions. :) Fluctuations before next update [webmasterworld.com ] GoogleGuy msg #109 May 23 rfgdxm1, I wouldn't go as far as saying that SERPs aren't going to change at all until the next update. Some amount of fluctuations might occur before then (e.g. a couple of small spam filters, or some fresher data). Older data for cross-verification and worst case backup [webmasterworld.com ] GoogleGuy msg #113 May 23 TheAutarch, that data can be used for cross-verification. It's also there as a worst-case back-up, but I don't think we'll need to use it. Some new data such as spam filters come before next index, larger set of data comes with next index [webmasterworld.com ] GoogleGuy msg 120 May 23 jojojo, some data will be brought in, such as spam filters. A larger set of data will be brought in with the next index.
   Brett_Tabke msg:160176	8:05 pm on Jun 10, 2003 (gmt 0)
   	Thanks for the killer post WebG.

   Source: http://www.webmasterworld.com/forum3/13088-1-30.htm?highlight=dominic#msg160172
   
   
   Change In The DNA 6: Google Update Esmeralda
   
   By kpaul in Op-Ed
   Tue Jun 24, 2003 at 07:34:57 AM EST
   Tags: Internet (all tags)
   
   
   
   

   Google is dancing to a different tune this month. Some think the beat is from war drums playing in anticipation of a Microsoft search product. Following the naming convention started at WebmasterWorld (Google updates are given names like hurricanes), the Esmeralda Google Update is currently winding down.
   For those of you who aren't regular Google-watchers, the Google dance (update) refers to a time roughly once a month when Google updates their index of the web with info from the last month's crawl of websites. During the update, the results Google displays differ in various servers in the Google cluster, hence the term Google is dancing.

   Google has been undergoing a lot of big changes during the last couple months, though, and things have been quite odd to say the least. A lot of people seem to think Google is moving toward a rolling update instead of a re-ranking of the entire web once a month. The weirdness started in May. The monthly update (Dominic) was late - very late. The Deepcrawler, which spidered sites deeply once a month for the big update, hadn't been seen in the logs of anyone who watched for it. Very strange. Forums across the 'net experienced a flurry of activity as people tried to figure out what was going on at the Googleplex. To add to the mysteriousness, Freshbot began to act a little more like the Deepcrawler. Traditionally, Freshbot came from a different IP range and added pages to the index immediately as opposed to once a month like Deepbot did. Freshbot's purpose was to add 'freshness' to the search results in between monthly updates. Did the late monthly update and disappearance of Deepbot signal the birth of a new bot? Names like deepfreshBot and FredBot were posted around the 'net as people pondered the new activity in their logs. GoogleGuy (a Google employee who posts anonymously at WebmasterWorld - yes, it's been verified) confirmed that a big change was in the works and that people should be patient as Google evolved. He said the process would take "more than weeks, less than months." Cryptic, but to be honest, I'm happy to get any first-hand, 'unofficial' info from them. During Dominic, SERPs (search engine result pages) were all over the place. Some pointed back to the now infamous September '02 update that was talked about in Wired and on that other site. The doomsayers always appear during an update, though, predicting the downfall of the 'obviously-evil-because-they-don't-list-my-site' Google. The conspiracy theorists come out in droves as well. One theory that was offered to the world was that Google had run out of unique id's because they used four bytes instead of five or more when designing their database infrastructure. Googleguy mentioned that a Google employee fell out of his chair laughing when he heard that. Beyond the anger and conspiracies, though, some took GoogleGuy's messages and tried to make sense of what was going on. (This whole marketing thing Google has going on with GoogleGuy may be a story in itself.) Anyway, although no one from Google has officially confirmed it, they're apparently moving away from a once-a-month crawl/update cycle to a more continuous updating process. As the web grows, this is becoming a necessity to have the freshest results possible. They're also perfecting (as much as they can) automated spam detectors to help clean up the SERPs. For Google to stay on top (as they edge closer to an eventual IPO), these are worthy tasks. They need to innovate. They need to stay two steps ahead of their closest competitors, even if it means a little collateral damage in the SERPs for a little while. At this time, in my humble opinion, the only SE that comes close to rivaling Google is FAST's AllTheWeb. Yahoo is most likely going to dump Google as a provider of their search engine results at some point this year, though. And Microsoft is also showing an interest in the search engine market. With two giants like that (not to mention FAST becoming a giant in its own right), it's becoming more and more important for Google to maintain their lead. As I said in the intro, the Esmeralda update appears to be settling down somewhat and should stabilize sometime in the next few days. From there, SEOs (search engine optimizers) will again look at how Google ranks the web and how best to optimize for it. The bigger question, though, is what it means for the Internet at large? Personally, I think it's an effort by Google to maintain their dominance in the search engine game. And Google has always been a good neighbor in cyberspace so this is not necessarily a bad thing. Sure, some may snort and chuckle when Microsoft and search engines are mentioned in the same sentence, but they have a *lot* of money. In our society this means they don't necessarily need the best product to have the most market share. Just take a look at Internet Explorer and its dominance over Netscape and other browsers. The thing is, if Google can remember what they were like when they were still the little guys, they have a fighting chance at fending off Microsoft and the others. This latest major change to their algorithm and crawling methods is, I think, a move by Google to sprint ahead of the competition before it's too late.

   Source: https://www.kuro5hin.org/story/2003/6/23/201523/090
   
   

Matt Cutts: Gadgets, Google, and SEO

Change In The DNA 7: Explaining algorithm updates and data refreshes

by Matt Cutts on December 23, 2006

in Glossary/Definitions,Google/SEO

A thread on WMW started Dec. 20th asking whether there was an update, so I’m taking a break from wrapping presents for an ultra-quick answer: no, there wasn’t.

To answer in more detail, let’s review the definitions. You may want to review this post or re-watch this video (session #8 from my videos). I’ll try to summarize the gist in very few words though:

Algorithm update: Typically yields changes in the search results on the larger end of the spectrum. Algorithms can change at any time, but noticeable changes tend to be less frequent.

Data refresh: When data is refreshed within an existing algorithm. Changes are typically toward the less-impactful end of the spectrum, and are often so small that people don’t even notice. One of the smallest types of data refreshes is an:

Index update: When new indexing data is pushed out to data centers. From the summer of 2000 to the summer of 2003, index updates tended to happen about once a month. The resulting changes were called the Google Dance. The Google Dance occurred over the course of 6-8 days because each data center in turn had to be taken out of rotation and loaded with an entirely new web index, and that took time. In the summer of 2003 (the Google Dance called “Update Fritz”), Google switched to an index that was incrementally updated every day (or faster). Instead of a monolithic monthly event, the Google would refresh some of its index pretty much every day, which generated much smaller day-to-day changes that some people called everflux.

Over the years, Google’s indexing has been streamlined, to the point where most regular people don’t even notice the index updating. As a result, the terms “everflux,” “Google Dance,” and “index update” are hardly ever used anymore (or they’re used incorrectly ). Instead, most SEOs talk about algorithm updates or data updates/refreshes. Most data refreshes are index updates, although occasionally a data refresh will happen outside of the day-to-day index updates. For example, updated backlinks and PageRanks are made visible every 3-4 months.

Okay, here’s a pop quiz to see if you’ve been paying attention:
Q: True or false: an index update is a type of data refresh.

A: Of course an index update is a type of data refresh! Pay attention, I just said that 2-3 paragraphs ago. Don’t get hung up on “update” vs. “refresh” since they’re basically the same thing. There’s algorithms, and the data that the algorithms work on. A large part of changing data is our index being updated.

I know for a fact that there haven’t been any major algorithm updates to our scoring in the last few days, and I believe the only data refreshes have been normal (index updates). So what are the people on WMW talking about? Here’s my best MEGO guess. Go re-watch this video. Listen to the part about “data refreshes on June 27th, July 27th, and August 17th 2006.” Somewhere on the web (can’t remember where, and it’s Christmas weekend and after midnight, so I’m not super-motivated to hunt down where I said it) in the last few months, I said to expect those (roughly monthly) updates to become more of a daily thing. That data refresh became more frequent (roughly daily instead of every 3-4 weeks or so) well over a month ago. My best guess is that any changes people are seeing are because that particular data is being refreshed more frequently.

Source: http://www.mattcutts.com/blog/explaining-algorithm-updates-and-data-refreshes/

Change In The DNA 8: Search Engine Size Wars & Google's Supplemental Results

Danny Sullivan, September 2, 2003 14 Comments

A longer, more detailed version of this article is available to Search Engine Watch members. Click here to learn more about becoming a member

Ah, summer. Time to play on the beach, head out on vacation and if you're a search engine, announce to the world that you've got the largest index.

Around this time last year, AllTheWeb kicked off a round of "who's biggest" by claiming the largest index size. Now it's happened again, when AllTheWeb said last month that its index had increased to 3.2 billion documents, toppling the leader, Google.

Google took only days to respond, quietly but deliberately notching up the number of web pages listed on its home page that it claims to index. Like the McDonald's signs of old that were gradually increased to show how many customers had been served, Google went from 3.1 billion to 3.3 billion web pages indexed.
Yawn.

Actually, not yawn. Instead, I'm filled with Andrew Goodman-style rage (and that's a compliment to Andrew) that the search engine size wars may erupt once again. In terms of documents indexed, Google and AllTheWeb are now essentially tied for biggest -- and hey, so is Inktomi. So what? Knowing this still gives you no idea which is actually better in terms of relevancy.

Size figures have long been used as a surrogate for the missing relevancy figures that the search engine industry as a whole has failed to provide. Size figures are also a bad surrogate, because more pages in no way guarantees better results.

How Big Is Your Haystack?

There's a haystack analogy I often use to explain this, the idea that size doesn't equal relevancy. If you want to find a needle in the haystack, then you need to search through the entire haystack, right? And if the web is a haystack, then a search engine that looks through only part of it may miss the portion with the needle!
That sounds convincing, the reality is more like this. The web is a haystack, and even if a search engine has every straw, you'll never find the needle if the haystack is dumped over your head. That's what happens when the focus is solely on size, with relevancy ranking a secondary concern. A search engine with good relevancy is like a person equipped with a powerful magnet -- you'll find the needle without digging through the entire haystack because it will be pulled to the surface.

Google's Supplemental Index

I especially hate when the periodic size wars erupt because examining the latest claims takes time away from other more important things to write about. In fact, it was a great relief to have my associate editor Chris Sherman cover this story initially in SearchDay last week (Google to Overture: Mine's Bigger). But I'm returning to it because of a twist in the current game: Google's new "supplemental results."

What are supplemental results? At the same time Google posted new size figures, it also unveiled a new, separate index of pages that it will query if it fails to find good matches within its main web index. For obscure or unusual queries, you may see some results appear from this index. They'll be flagged as "Supplemental Result" next to the URL and date that Google shows for the listing.

Google's How To Interpret Your Search Results page illustrates this, but how about some real-life examples you can try? Here are some provided by Google to show when supplemental results might kick in:

• "St. Andrews United Methodist Church" Homewood, IL
• "nalanda residential junior college" alumni
• "illegal access error" jdk 1.2b4
• supercilious supernovas

Two Web Page Indexes Not Better Than One

Using a supplemental index may be new for Google, but it's old to the search engine industry. Inktomi did the same thing in the past, rolling out what became known as the small "Best Of The Web" and larger "Rest Of The Web" indexes in June 2000.

It was a terrible, terrible system. Horrible. As a search expert, you never seemed to know which of Inktomi's partners was hitting all of its information or only the popular Best Of The Web index. As for consumers, well, forget it -- they had no clue.

It also doesn't sound reassuring to say, "we'll check the good stuff first, then the other stuff only if we need to." What if some good stuff for whatever reason is in the second index? That's a fear some searchers had in the past -- and it will remain with Google's revival of this system.

Why not simply expand the existing Google index, rather than go to a two tier approach?

"The supplemental is simply a new Google experiment. As you know we're always trying new and different ways to provide high quality search results," said Google spokesperson Nate Tyler.

OK, it's new, it's experimental -- but Google also says there are currently no plans to eventually integrate it into the main index.

Deconstructing The Size Hot Dog

Much as I hate to, yeah, let's talk about what's in the numbers that are quoted. The figures you hear are self-reported, unaudited and don't come with a list of ingredients about what's inside them. Consider the hot dog metaphor. It looks like it's full of meat, but if you analyze it, it could be there's a lot of water and filler making it appear plump.

Let's deconstruct Google's figure, since it has the biggest self-reported number, at the moment. The Google home page now reports "searching 3,307,998,701 web pages." What's inside that hot dog?

First, "web pages" actually includes some things that aren't web pages, such as Word documents, PDF files and even text documents. It would be more accurate to say "3.3 billion documents indexed" or "3.3 billion text documents indexed," because that's what we're really talking about.

Next, not all of those 3.3 billion documents have actually been indexed. There are some documents that Google has never actually indexed. It may list these in search results based on links it has seen to the documents. The links give Google some very rough idea of what a page may be about.

For example, try a search for pontneddfechan, a little village in South Wales where my mother-in-law lives. You should see in the top results a listing simply titled "www.estateangels.co.uk/place/40900/Pontneddfechan" That's a partially indexed page, as Google calls it. It would be fairer to say it's an unindexed page, since in reality, it hasn't actually been indexed.

What chunk of the 3.3 billion has really been indexed? Google's checking on that for me. They don't always provide an answer to this particular question, however. Last time I got a figure was in June 2002. Then, 75 percent of the 2 billion pages Google listed as "searching" on its home page had actually been indexed. If that percentage holds true today, then the number of documents Google actually has indexed might be closer to 2.5 billion, rather than the 3.3 billion claimed.

But wait! The supplemental index has yet to be counted. Sorry, we can't count it, as Google isn't saying how big it is. Certainly it adds to Google's overall figure, but how much is a mystery.

Let's mix in some more complications. For HTML documents, Google only indexes the first 101K that it reads. Given this, some long documents may not be totally indexed -- so do they count as "whole" documents in the overall figure? FYI, Google says only a small minority of documents are over this size.

Auditing Sizes

OK, we've raised a lot of questions about what's in Google's size figure. There are even more we could ask -- and the same questions should be directed at the other search engines, as well. AllTheWeb's 3.2 billion figure may include some pages only known by seeing links and might include some duplicates, for example. But instead of asking questions, why not just test or audit the figures ourselves?
That's exactly what Greg Notess of Search Engine Showdown is especially known for. You can expect Greg will probably take a swing at these figures in the near future -- and we'll certainly report on his findings. The last test was done in December. His test involves searching for single word queries, then examining each result that appears -- a time-consuming task. But it's a necessary one, since the counts from search engines have often not been trustworthy.

Grow, But Be Relevant, Too

I'm certainly not against index sizes growing. I do find self-reported figures to also be useful, at least as a means of figuring out who is approximately near each other. Maybe Google is slightly larger than AllTheWeb or maybe AllTheWeb just squeaks past Google -- the more important point is that both are without a doubt well above a small service like Gigablast, which has only 200 million pages indexed.

However, that's not to say that a little service like Gigablast isn't relevant. It may very well be, for certain queries. Indeed, Google gained many converts back when it launched with a much smaller index than the established major players. It was Google's greater relevancy -- the ability to find the needle in the haystack, rather than bury you in straw -- that was the important factor. And so if the latest size wars should continue, look beyond the numbers listed at the bottom of the various search engine home pages consider instead the key question. Is the search engine finding what you want?

By the way, the baby of the current major search engine line-up Teoma did some growing up last month. The service moved from 500 million to 1.5 billion documents indexed.

Paul Gardi, vice president of search for Ask Jeeves, which owns Teoma, wants to grow even more. He adds that Teoma is focused mainly on English language content at the moment -- so the perceived smaller size of Teoma may not be an issue for English speakers. Subtract non-English language pages from Teoma's competitors, and the size differences may be much less.

"Comparatively speaking, I would argue that we are very close to Google's size in English," Gardi said.

Source: http://searchenginewatch.com/article/2067049/Search-Engine-Size-Wars-Googles-Supplemental-Results