After releasing a preview of the Bing sitemap plugin a few months ago, Bing is ready to officially launch the Bing XML Sitemap Plugin 1.0 to the public.

This sitemap plugin allows webmasters to easily create XML sitemap files for their websites that can be submitted to Bing Webmaster Tools or other search engines. It works on both Internet Information Services (IIS) for Windows Server and Apache HTTP Server.

What has changed between the beta version released a few months ago and the release today?

• Automatically ping the search engines after each Sitemap update (currently ping Bing and Google)
• An improved computation algorithm for the value that is calculated – Multi-machine merge memory improvements and various other improvements
• Custom installation and data directory at installation time

The plugin can be downloaded for over here for free.

For more details, see the Bing Webmaster Blog.

When you begin to get into multilingual and multiregional SEO, you know that you have taken a step forward as an SEO expert. Why? Well, because you are probably dealing with a large, complex site that demands the expertise of someone who knows what they are talking about. If you are dealing in multilingual or multinational SEO, then you are managing a complex website strategy that serves multiple locations and languages — not just one.

Lucky for you, most everything you need to know about multilingual and multiregional SEO is listed in this post. So, even if you are a first-timer, you now have a cheat sheet that can guide you down the right path. Please keep in mind that multilingual and multiregional SEO are always changing — so, make sure to verify your strategy with the most up-to-date materials before making any drastic decisions.

What Are Multilingual SEO & Multiregional SEO?

Multilingual SEO is the practice of offering optimized website content in a variety of languages. Multiregional SEO is the practice of creating optimized website content that is tailored specifically to multiple geographic regions.

These two strategies often have overlap, which is why we are covering them both in this post. They also tie in with other aspects of online marketing, such as conversion rate optimization, pay per click and content strategy.

First, Ask Yourself: Where Are You Targeting?

The first thing you need to do is determine what regions you are targeting. Next, you need to decide which languages you are going to make available to those regions. This is critical, because it allows you to lay out the entire project.

Domain & URL Structure

After you determine the language and regions, you need to choose your domain and/or URL structure strategy.

Google’s Official Webmaster Central blog states, “It’s difficult to determine geotargeting on a page by page basis, so it makes sense to consider using a URL structure that makes it easy to segment parts of the website for geotargeting.”

I strongly agree with this — if you can map out the right location structure first, the rest of the project will be much easier. But, before we start to map website architecture, let’s take a look at the pros and cons of each URL or domain option. (Note: The information below is taken directly from Google.)

ccTLDs (e.g., example.de, example.fr)

Pros (+)

• clear geotargeting
• server location is irrelevant
• easy separation of sites
• legal requirements (sometimes)

Cons (-)

• expensive
• potential availability issues
• more infrastructure
• ccTLD requirements (sometimes)

Subdomains With gTLDs (e.g., de.site.com, fr.site.com, etc.)

Pros (+)

• easy to set up
• can use Webmaster Tools geotargeting
• allows different server locations
• easy separation of sites

Cons (-)

• users might not recognize geotargeting from the URL alone (is “de” the language or the country?)

Subdirectories With gTLDs (e.g., site.com/de/, site.com/fr/, etc.)

Pros (+)

• easy to set up
• can use Webmaster Tools geotargeting
• low maintenance (same host)

Cons (-)

• users might not recognize geotargeting from the URL alone
• single server location
• separation of sites is more difficult/less clear

URL parameters (e.g., site.com?loc=de, ?country=france, etc.)

Pros (+)

• none (not recommended)

Cons (-)

• segmentation based on the URL is difficult
• users might not recognize geotargeting from the URL alone
• geotargeting in Webmaster Tools is not possible

We’ve now outlined the pros and cons of the most popular methods for geotargeting in a URL or domain; but, which is the best option for you?

The answer to this question always depends on the goals of the website and who you need to target through SEO and other integrated marketing methods to reach those goals. Top-level domains are a strong indicator to Google, so they can be a good option. However, subdomains and subdirectories have their advantages, as well. We’ll explore those in a moment.

Important Information About Top Level Domains

Some domains are generic and others are location-specific. According to Google, these are the generic top-level domains (gTLDs) which do not specific a country. Google treats the following as gTLDs that can be geotargeted in Webmaster Tools.

Generic Top Level Domains (gTLDs)

.aero

.biz

.cat

.com

.coop

.edu

.gov

.info

.int

.jobs

.net

.mil

.mobi

.museum

.name

.net

.org

.pro

.tel

.travel

Regional Top-Level Domains (rTLDs)

According to Google, these domains are associated with a region, but they treat them as top level as well (much like .com or .org).

.eu

.asia

Generic Country Code Top Level Domains (gccTLDs)

Here is a list of the ccTLDs. Keep in mind that Google is always changing this list. Also, this list is specific to the Google search engine.

.as

.bz

.cc

.cd

.co

.dj

.fm

.la

.me

.ms

.nu

.sc

.sr

.tv

.tk

.ws

An Argument For Subdomains Or Directories

In my opinion, your best option for a large website is going to be either a directory or subdomain structure. From an SEO and analytics tracking perspective, things can get very messy when you have multiple top-level domains. I recently worked on a large client that had a mix of these strategies and it was pretty tricky.

According to Google:

If your time and resources are limited, consider buying one non-country-specific domain, which hosts all the different versions of your website. In this case, we recommend either of these two options:

• Put the content of every language in a different subdomain. For our example, you would have en.example.com, de.example.com, and es.example.com.
• Put the content of every language in a different subdirectory. This is easier to handle when updating and maintaining your site. For our example, you would have example.com/en/, example.com/de/, and example.com/es/.

A directory structure would be my preferred choice, in most cases. It is very clean, and I like how the directories add to the overall authority of the entire site. This is basically the case with subdomains, as well; but, let’s be honest: subdomains are more “separate” than a directory structure as far as content segmentation is concerned.

ccTLD & Webmaster Tools Geotargeting

It has to be noted that the use of a ccTLD is generally a very strong signal for users and search engines; so, if you really want to target a particular region, there is nothing wrong with using a ccTLD. (Though, as mentioned above, you should be prepared for the fact that issues can arise with tracking, SEO and branding if you are using multiple ccTLDs — this is why many multilingual/multinational SEOs prefer directories or subdomains.)

However, it is not necessary to use a ccTLD in order to geotarget a particular country. Google Webmaster Tools allows for manual geotargeting of gTLDs (such as .com and .net) with the Set Geographic Target tool. To set up geotargeting in Webmaster Tools, follow these steps:

1. On the Webmaster Tools Home page, click the site you want
2. Under Site configuration, click Settings
3. In the Geographic target section, select the option you want

If you want to ensure that your site is not associated with any country or region, select Unlisted.

Google points out that it generally does not make sense to set a geographic target if the same pages on your site target more than a single country. This is because your geotargeting settings might limit the reach of your content.

Click here to view the embedded video.

You can learn more about geotargeting here.

Is Server Location A Factor?

Many people think that server location is a big deal for this multilingual or multiregional search, but the degree to which this is true has recently changed due to the wide adoption of new technologies.

According to Google, “Server location (through the IP address of the server) is frequently near your users. However, some websites use distributed content delivery networks (CDNs) or are hosted in a country with better webserver infrastructure; so, we try not to rely on the server location alone.”

This is something which has evolved over years; you can see an older video here when things were much different. I really wish Google would take these old videos down, as they throw people off.

Specifying Regional Landing Pages With The Rel=”Alternate” Hreflang Annotation

Now that we’ve discussed URL structure, server location, and geotargeting in Webmaster Tools, it is time to talk about hreflang markup. The rel=”alternate” hreflang=”x” annotation is used to help Google identify which URLs should be served to which visitors based on language and geographic location. This is useful when you have multiple versions of the same content that has been translated or otherwise adjusted to target users in a specific region.

According to Google, there are 3 ways that we can implement this markup:

• HTML link element in header. In the HTML <head> section of http://example.com/us, add a link element pointing to the Spanish version of that webpage at http://example.com/us-es, like this: <link rel=”alternate” hreflang=”es” href=” http://example.com/us-es” />
• HTTP header. If you publish non-HTML files (like PDFs), you can use an HTTP header to indicate a different language version of a URL: Link: < http://example.com/us-es>; rel=”alternate”; hreflang=”es”
• Sitemap. Instead of using markup, you can submit language version information in a Sitemap.

Out of the options above, most go with the HTML link element, the sitemap or both of those items.

It is important to note that there are specific hreflang supported values for language and region. For example:

• de: German content, independent of region
• en-GB: English content, for GB users
• de-ES: German content, for users in Spain

For more information on supported values, visit the Google Webmaster support article on the topic.

The X-Default Hreflang Attribute Value

So, what happens when someone visits your site from a country that you don’t have a landing page for? Perhaps you’d want them to land on a generic home page, or a page where they can select their country or language.

Thanks to a new bit of markup from Google called “x-default,” you can now specify a default page for users outside your target regions. When specifying your region-specific landing pages with hreflang, you would use the value “x-default” to indicate which page is the default in situations where you don’t have a region-specific landing page. Thus, you might have a cluster of HTML link tags that look like this:

<link rel=”alternate” href=”http://example.com/en-gb” hreflang=”en-gb” />

<link rel=”alternate” href=”http://example.com/en-us” hreflang=”en-us” />

<link rel=”alternate” href=”http://example.com/en-au” hreflang=”en-au” />

<link rel=”alternate” href=”http://example.com/” hreflang=”x-default” />

Above, http://example.com would be the default page for users outside of Great Britain, the United States or Australia.

Quick Note On Base On-Page Elements

When it comes to the meta information on these pages, it is generally a good idea to vary them based on language and region. Make sure to have a template, or unique content, depending on how large your website is, and consider altering the following elements:

• Title
• Description
• Keywords
• H1
• H2
• Copy
• Images
• Products
• Internal Linking

User IP & Use Agent Detection

User agent detection is the process of detecting the device a person is using and delivering content based on the best practices for that device. IP location detection is the practice of detecting the location of a user and delivering content based on what is more relevant for that IP location. If you implement this correctly, there is a good chance you will lower bounce rates, increase conversions and show the user what they are looking for more quickly.

This is a very common practice for mobile optimization (as we often detect location and device), but it is also important for multilingual and multiregional SEO. Google supports both HTTP redirection and JavaScript redirects. You can watch a video on this topic here.

Click here to view the embedded video.

Below are some notes from Google on the topic.

Using HTTP redirection

HTTP redirection is a commonly used to redirect clients to device-specific URLs. Usually, the redirection is done based on the user-agent in the HTTP request headers. It is important to keep the redirection consistent with the alternate URL specified in the page’s link rel=”alternate” tag or in the Sitemap.

For this purpose, it does not matter if the server redirects with an HTTP 301 or a 302 status code.

JavaScript Redirects

If HTTP redirection is difficult to implement, you can use JavaScript to redirect users to the URLs pointed to by the link rel="alternate" tag. If you choose to use this technique, please be aware of the latency caused by the client side of redirection due to the need to first download the page, then parse and execute the JavaScript before triggering the redirect.

There are many approaches to implementing a JavaScript-based redirect. For example, you can use JavaScript to execute the media queries your site already uses in the link annotations on the page using the matchMedia() JavaScript function.

Outside of the detection and redirects, you also want to make sure your website has good internal linking and navigation so that the user can find the pages that match their demographic.

Rel Author & Rel Publisher Considerations

It is interesting; now that rel author and rel publisher can easily be implemented on any website, people have to stop and think about whether they should implement it. This is a huge topic, so for the purposes of this post let’s just consider a few elements from the mutilingual and regional perceptive.

Rel Author

If you implement rel author across an entire multilingual or multiregional website, you run the risk of positioning a figurehead for your brand in the search space which may not relate to everyone you’re targeting. If you are considering implementing rel author, think about whether or not it makes sense to have one person’s presence associated with all of your results. If it does not, consider if it makes sense to break the rel author implementation up by region, language, or on an article-by-article basis.

As rel author becomes more advanced, we could see regional sales managers attached to certain location segments of the website. There are some problems with this, however. For example, generally, you need authoritative Google+ profiles — and even when you have them, it takes some time before they show up in the SERPs. Rel author can make sense for the right multilingual or multiregional website, but there needs to be a clear strategy in place.

Rel Publisher

Rel publisher is an easier sell than rel author, although I do think there is room for both on the right multilingual or multinational website. Rel publisher causes a very nice Google+ page along with updates to be displayed in search results — and, you can segment your updates based on circles. So, as long as you have a good language and region circle strategy in place, the rest of the rel publisher strategy should be fine.

Bing Multilingual & Multiregional SEO

When it comes to Bing, they love the language meta tag. But, they also pay attention to HTTP headers, top-level domain and reverse IP lookup.

Bing Language Meta Tag

Use the “content-language” meta tag to embed a document location in the <head> section of your documents:

<meta http-equiv=”content-language” content=”en-us”>

The “content” attribute is comprised of a 2-letter ISO 639 language code, followed by a dash and the appropriate ISO 3166 geography code. For example:

• de-at: German, Austria
• de-de: German, Germany
• en-us: English, United States
• es-ar: Spanish, Argentina

Alternatively, embed the document location in either the <html> or the <title> element using the same format:

• <html lang=”en-us”>
• <title lang=”en-us”>

HTTP Headers

For more information on setting HTTP response headers please refer to:

• In IIS7: http://technet.microsoft.com/en-us/library/cc753133(WS.10).aspx
• In IIS6: http://technet.microsoft.com/en-us/library/cc732442.aspx
• In Apache 2.2: http://httpd.apache.org/docs/2.2/mod/mod_headers.html

Top Level Domain

According to Bing, “Out of the top level domain categories distinguished by the IANA, only the country code top-level domains (or ccTLDs) influence the document location. For an overview of the currently assigned ccTLDs, please visit IANA’s website at: http://www.iana.org/domains/root/db/.”

Reverse IP Lookup

As we mentioned, for each document added to Bing, the search engine does a reverse IP lookup to determine the documents location; so, this is a small factor in Bing indexing.

A Note On Duplicate Content

One issue that comes up a lot in SEO is duplicate content. Basically, if you have duplicate content on your website or share content that is housed on another site, it forces Google to pick the “winner,” if you will, only ranking that page. When it comes to multilingual and multiregional websites, this can become an issue — you will often have multiple versions of the same content for different regions and languages.

The good news is, if you implement the rel=”alternate” hreflang link element and x-default hreflang annotation correctly, duplicate content should not be an issue. In the past, SEOs would use rel=”canonical”, block pages with robots.txt, etc. But today, the alternate/x-default is the best option.

Summary

Mutilingual and multiregional SEO is always changing. I expect this to be an expanding topic as we move into the future of this crazy Internet world. One thing that I love about how things are evolving is that if you pay attention to new strategies for search, you are going to get other amazing ideas to improve your online business.

For example, if you are taking the steps to build out a better multinational and multilingual strategy for SEO, you are probably working that same strategy into your other online performance channels. The key is making everything work together and pay off.

Other Great Resources On This Topic

• http://searchengineland.com/11-considerations-for-international-seo-117798
• http://googlewebmastercentral.blogspot.com/2010/03/working-with-multi-regional-websites.html
• http://googlewebmastercentral.blogspot.com/2008/08/how-to-start-multilingual-site.html
• http://support.google.com/webmasters/bin/answer.py?hl=en&answer=62399
• http://googlewebmastercentral.blogspot.com/2009/12/region-tags-in-google-search-results.html
• http://googlewebmastercentral.blogspot.com/2008/04/where-in-world-is-your-site.html
• http://support.google.com/webmasters/bin/answer.py?hl=en&answer=189077
• http://support.google.com/webmasters/bin/answer.py?hl=en&answer=182192&topic=2370587&ctx=topic
• http://support.google.com/webmasters/bin/answer.py?hl=en&answer=2620865&topic=2370587&ctx=topic

An online survey examining SEO assumptions by SurveyMonkey found that the No. 1 reason users would block a website from their search results was if the website contained too many ads.

If given an option to remove a website from future search results, 68 percent of the respondents said they would block a website because it had too many ads, while 60 percent claimed they would block a site because of poor quality content. Respondents were more forgiving of typos and grammar mistakes, with 26 percent claiming to block a website with typos and only 23 percent blocking a website containing bad grammar.

When respondents were asked what they would do if a Google search result did not contain the expected information, 34 percent of respondents said they would return to the search results page and try a different result; 25 percent said they would return to Google to enter a new search.

The online survey was conducted using SurveyMonkey’s Audience tool, surveying 615 respondents from a random selection of males and females age 18 to 60. The participants were located across the country with annual household incomes ranging from $25,000 to $150,000+ and varying education levels.

Other key findings from the survey demonstrated most users are not bothered by how recently an online article was published – trusting content published two years ago the same as content published ten days ago – but were less likely to trust an article with a number of typos and obvious grammar mistakes. The survey also concluded that users preferred Google search results over Bing.

In a recent study by SurveyMonkey examining SEO assumptions, respondents were given two search result pages, one with a page header labeled “Google” and the other with a page header labeled “Bing,” and asked which page of results they preferred. Even when the page header labels were swapped, more users preferred the Google search results.

Of 641 survey respondents, 379 participants received a survey asking which of two search result pages they preferred. One page of results for the term “file taxes” included true Google results and the other page included true Bing results. The Google page was chosen by majority of the respondents.

A second survey was given to 262 participants. Using the same search term, respondents had to choose between a Google search results page and a Bing search result page. In this survey the SERP headers were swapped with Google results listed as Bing results and Bing results listed as Google results. Of the respondents who received the swapped search result pages, a larger percentage of respondents still chose Google results, even though they were actually Bing search results.

Not only did the survey find that users are biased toward Google, but they are influenced by a site’s brand as well. In a separate question, participants who received the first survey were asked to rate the believability of an article on CNN.com, while participants who received the second survey were asked to rate the believability of an article on a generic website, InternetNews.com. Respondents were more likely to rate the CNN.com article as believable.

Another key finding of the survey revealed that the No. 1 reason users would block a website from future searches if given the option was if the website contained too many ads.

The study, conducted using SurveyMonkey’s Audience tool, asked participants a number of questions based on general SEO assumptions. The audience was a random selection of males and females, age 18 to 60 from across the country with annual household incomes ranging from $25,000 to $150,000+ and varying education levels.

Bing has announced support for HTML5 pushState as a way to implement AJAX on a site in a way that enables Bing to crawl and index the URLs and content. As Google has supported this implementation since early 2012, site owners finally have an AJAX option that can be crawled and indexed by both major search engines in the United States. (The ease of implementing is another story altogether.)

Bing tells me that while they still support the #! version of crawlable AJAX originally launched by Google, they’re finding it’s not implemented correctly much of the time, and they strongly recommend pushState instead.

Why AJAX Can Be Difficult To Crawl & Index

One common use of AJAX is to make the website experience faster for a visitor, but this implementation can have drawbacks for SEO. Imagine, for instance, a page with several tabs of content.

A web developer could implement this one of several ways.

• A separate URL for each tab – with this implementation, when the visitor clicks a tab, a new request is made to the server for a completely new page. The URL structure might look something like:
  • http://www.example.com/my-web-page?tab=one
  • http://www.example.com/my-web-page?tab=two
  • http://www.example.com/my-web-page?tab=three
• CSS for each tab – with this implementation, the server returns the contents of all tabs with the first page request. When the visitor clicks a tab, the CSS rules cause the browser to hide the contents associated with one tab and show the contents associated with clicked tab. Only one URL is associated with the page, like this:
  • http://www.example.com/my-web-page
• AJAX rending of each tab – with this implementation, when the visitor clicks a tab, only the changing portion of the page is replaced. The URL structure might look something like:
  • http://www.example.com/my-web-page#tab=one
  • http://www.example.com/my-web-page#tab=two
  • http://www.example.com/my-web-page#tab=three

As with most things, pros and cons exist for each option. A separate URL for each tab is easy to share and bookmark and is easy for search engines to crawl and index (they can extract all of the content from each page and have a separate URL to associate with each), but reloading the entire contents of each page can be slow.

CSS for each tab is also easy for search engines to crawl and index, and in some cases, the combined page may rank higher than the same content broken into three pages (due to consolidated incoming links and relevancy signals). But the request for all of that content at once can be the slowest of all to render, and users can’t share or bookmark the page with a secondary tab as active.

AJAX rendering is fastest and enables easy sharing and bookmarking. But, search engines have historically had the hardest time with this implementation. Search engines have trouble extracting content from AJAX/JavaScript calls (although Google has been getting better at it). And # in a URL started as a way to link to content within a page, and so search engines tend to ignore everything in a URL past the #.

Crawlable AJAX

In 2009, Google put together a way to make AJAX crawlable. With this method, a the webpage would use #! rather than #, like this:

• http://www.example.com/my-web-page#!tab=one

For a normal user agent, such as a browser, the # would trigger the AJAX portion of the page, just as it would in a normal AJAX implementation. However, a search engine user agent such as Google would see the #! section of the URL and then request a special version of the page (replacing #! with ?_escaped_fragment_=). In response, the server would return a static version of the page with the contents normally rendered through JavaScript. The benefits of this implementation were that search engines could associate a separate URL with each set of content; and even better, could extract all of that content.

In 2011, Bing began supporting this implementation and included a checkbox in their webmaster tools so site owners could let them know it was being used on a site (they’ve since removed the checkbox, as they’ve gotten better at detecting and crawling it).

As with the other implementations, this has its drawbacks as well, not least of which is the complicated implementation. Bing’s latest blog post notes:

“Developers had to rely on overly-complicated protocols such as “crawlable AJAX”, which uses the #! (“Hash bang”) signature.  This was meant to make AJAX development easier in regards to SEO, but it actually complicated things both for search engines as well as webmasters trying to implement, maintain, and debug their AJAX-driven web pages and applications.

With pushState, we can fully omit the complexity of transforming between “pretty” AJAX URLs and “ugly” static URLs. Search Engines will crawl and index the same URL used by you customers. We are back to business as usual for SEO, including pretty SEO well-understood URLs schema http://domain/path/file?name=value_parameters. This helps you focus on the usual SEO activities (links, page content, etc.) without having to do worry about complex page transformations.”

I asked Bing’s Fabrice Canel, who wrote the post, if Bing still supports the #! version of AJAX URLs, and he told me:

 ”We are still supporting the #! crawlable AJAX method but as I said, we do not recommend it at all and we really prefer pushState which is far easier for webmasters and web developers to adopt and maintain.”

HTML5 pushState

With HTML5 pushState, pages can take advantage of the best of both worlds: URLs without # (so search engines can easily index them) and dynamically rendered content for only the change portion of the page (to make things as speedy as possible).

With pushState, URLs look like the first example (a separate URL for each tab), but operate like the third example (AJAX rendering of each tab and the resulting URLs look as follows:

• http://www.example.com/my-web-page?tab=one
• http://www.example.com/my-web-page?tab=two
• http://www.example.com/my-web-page?tab=three

There are other, more complex ways of getting to this same result, such as Hijax, but pushState can be much easier.

Google has been supportive of HTML5 since the beginning (Google’s Maile Ohye in particular began recommending it in conferences since early 2012) and recently published a video in support of HTML5 pushState.

Click here to view the embedded video.

In the video, Google’s Matt Cutts noted:

“A correctly implemented site that uses pushState typically doesn’t need any extra support in order for us to be able to crawl it. We do support both [pushState and #!] and we do handle both standards… but [pushState] is something that I would encourage you to look into and it can be quite helpful in making sure things can be crawlable.”

And now Bing has announced support as well.

Of course, HTML5 has its drawbacks as well, notably that not all older browsers support it, and can require significantly engineering resources to implement (as you’re replacing the current site’s HTML implementation).

If your site uses AJAX-based URLs (either the # versions noted above, or versions that don’t change at all when  content changes) and subsequently, the site is not fully crawled and indexed and you’re looking for solutions, HTML5 pushState is definitely worth looking into.

If your site uses the crawlable #! URLs and isn’t having any trouble being indexed, then you can leave things as they are for now. Both Google and Bing continue to support this implementation.

If you’re considering adding AJAX to your site, make sure you think through the implementation carefully and take into account what content search engines can extract and whether the URLs are indexable.

Related:

• Bing Now Supports Google’s Crawlable AJAX Standard
• Google Can Now Execute AJAX and JavaScript for Indexing
• How to Best Take Advantage of Google’s Crawlable AJAX

One of the highlights from the SMX West show a week ago was the session named “The Search Police: Matt & Duane’s Excellent Search Engine Adventure.”

The panel featured Google’s Matt Cutts and Bing’s Duane Forrester as Bill & Ted with Danny Sullivan playing Rufus. Matt (Bill) and Duane (Ted) both give presentations and then answer questions from the attendees.

Here is the full video:

There were a lot of nuggets that came out of this video, from the next Penguin and Panda updates to link networks being discredited and penalized to a discussion of whether the PageRank indicator in the Google toolbar should go.

Bing announced, on its  Bing Webmaster Blog, a new feature that helps webmasters create two types of XML Sitemaps.

Bing created a server-side application that works on both IIS and Apache servers that helps you create a comprehensive Sitemap and, also, a recently updated Sitemap. The binaries can be downloaded at the Microsoft download site.

How do the two Sitemap files differ? The first is a comprehensive Sitemap of all URLs seen in server traffic, and the second is a Sitemap dedicated to store URLs that have changed recently.

Duane Forrester explained:

Having both comprehensive and delta Sitemaps provides you with significant benefits, as you will always have a full, up-to-date list of all URLs on your website that search engines can use for deep crawl, as well as a concise Sitemap of URLs that were modified recently, which search engine crawlers can prioritize. This can help in keeping bot traffic bandwidth down. In addition, the Sitemap Plugin automatically adds values to your Sitemap, and generates values to the Sitemap based on how popular your URLs are.

Search engines continue to go old school, encouraging people to “tag” content in a way that makes it feel like it’s 1999 all over again. Today, Bing makes another push in that direction, making content you tag with “Bing Tags” more visibile.

Honestly, I feel like this tagging crazy is simply Google and Bing falling down on the job. It used to be that we praised search engines for moving away from the tagging that was popular with them in the mid-to-late 1990s. That’s because they were getting smart enough to know that every word on a page was effectively a tag rather than forcing human beings to waste their time categorizing things.

Still, Google — long the lone-standout against tagging — tossed in the towel with the meta news keywords tag last year. As for Bing, it renamed Bing Linked Pages into Bing Tags last December.

That tagging of content was only visible to you. Now, tags will supposedly be public, if you allow:

The whole Bing Tag system really makes my head hurt. The idea is that you — or others — can tag pages to appear associated with you in Bing’s search results. By “you,” Bing means associated with a special “tagged” area at the top of its listings, visible to those who have a Facebook account that’s connected to Bing.

Up Close With Bing Tags

I think. Like I said, it makes my head hurt trying to figure this out.

For example, here’s a search on Bing for Search Engine Land news editor Barry Schwartz, showing what the public sees if they aren’t signed-in to Facebook:

Barry — our Barry, not the famous psychologist Barry — has pages about him that show up midway down the page, as the arrows show. Now here’s what I see when I am signed-in:

See how there’s a new section at the top, saying, “Barry Schwartz, tag more pages to your friend.” This is the product of Bing Tags in action. If someone is tagged, then a section like this should appear for them or those they are friends with on Facebook, when they are searching at Bing when it is connected to Facebook.

Want to add pages to tag to your friend? It seems kind of crazy. I mean, if they’re my friend, I probably already know the pages they have on the Web that are important. But maybe you want to help ensure other friends they have know about these on the chance they’re searching for their friend on Bing. If so, click on “tag more pages” link, and you get something like this:

That shows me what’s already tagged to Barry, and I have the option to use the “Find pages to tag” button, which brings up some suggested pages (you can also search for anything when in this mode to tag to a particular person). Found a page you want to tag to someone? Then you use the button next to the page:

Notice that warning at the bottom, “Friends will see your tags on Facebook.” Yes, whenever you tag a page on Bing like this, all your Facebook friends potentially see this in their stream.

Why do they need to see this? My assumption is because Bing would like you to help spread the word about Bing to your Facebook friends. I’d rather this wasn’t a requirement.

You’ve Been Tagged! Do You Approve?

Once you’re done, then that tagged page may show up assuming the person you tagged approves it. And if people tag you? When you go to the Bing Tags area when signed-in, you can see what’s been tagged to you and decide if you want to approve it.

In the end, I don’t really see the usefulness here. As I said, I think your friends already know what pages you’re likely associated with. And tagging only seems to work to show these pages to them, not to the general public. If there was a way to tag pages to better ensure people who don’t know you got better visibility of your content, that would be useful.

Does The Public Really See Them?

That, by the way, is what today’s Bing post suggests is the case — that these tags really are going to be visible to the general public now, if you allow that. But so far, I don’t see that this is actually happening. If it’s still rolling out, and it really works, maybe Bing Tags will deserve a second look.

Postscript: I’m getting further clarification from Bing, but apparently the tags associated with your name on Bing should show to the general public depending on whether you have have Facebook page that shows up high in Bing’s results AND if you’ve have enabled them to be seen by the public. If you don’t enable public sharing, they won’t appear.

To simulate this, if you don’t see it, try searching for someone’s name plus the word “facebook,” as shown in the example below:

See how I appear at the top under the “Danny Sullivan’s Tags” heading? This is what shows for me when I’m completely logged out of Bing and Facebook, i.e. — a public view.

I’ll update further when I get more clarity from Bing.

Postscript 2: Bing has now sent me that further clarification:

The placement of tags is based on a variety of factors including the ranking of a person’s profile page, e.g. a Facebook page.  To start, the ranking of public tags is on the conservative side, so the tagged pages don’t displace results the searcher is likely seeking.

In the case of a query like “Danny Sullivan” there are already a lot of high value search results, so tags don’t get as much visibility.

For a name query where the intended result is more ambiguous, the grouping and placement of tags can be more prominent.

We will continue to experiment with ranking of tags based on engagement data.  As previously, tags will be ranked higher if the searcher has a Facebook connection with the tagged person.

 

The US Senate Committee on Commerce, Science & Transportation has written to the major US search engines of Google, Yahoo and Bing asking them to examine how some moving services are using “bait-and-switch” tactics on customers who find them through search results.

Moving Ripoffs

Each search engine received the same letter, covering how a committee investigation found that consumers are doing searches for topics relating to moving services and encountering companies with names that seem similar to large, reputable brands.

Consumers pay what they believe to be a moving deposit but which instead is a brokerage fee. The actual moving company that does the work demands further payment, sometimes for thousands of dollars more than the original estimate.

And Winning Search Results

From the letter each search engine received (they can be found here):

Frequently, Internet moving brokers identified in the investigation, which received high numbers of consumer complaints, ranked highly in the search results. Based upon evidence obtained through the investigation, it appears that some of these companies may be “gaming the system” in order to boost their search rankings.

These companies appear to be using paid links to inflate their popularity. For example, one company had tens of thousands of external links to its web sites and, upon closer review, these links proved to be largely irrelevant. They included abandoned blogs, link directories for unrelated topics, and college student groups and organizaiotns, such as the Cornell Gymanstics Club.

The Committee’s Search Spam Report

The investigation took place over the past year, concluding in a report (PDF file) that was summarized in a committee press release last week.

I’m still going through the report myself and may do a follow-up story on some of the findings, but it’s fascinating reading so far. It might even be considered the most official search engine spam report I’ve ever seen. It focuses in particular on two carriers and two brokers:

• Able Moving (carrier)
• Best Price Moving & Storage (carrier)
• Nationwide Relocation Services (broker)
• Budget Van Lines (broker)

Nationwide Relocation Services (and owner Aldo DiSorbo) especially get called out, in part because DiSorbo is said to run many brokerage services using different names, ranging from American Van Lines to Patriot Van Lines. From the report:

Websites operated by or on behalf of the DiSorbo Broker Companies use questionable website and linking strategies that appear to be intended to enhance the companies’ search engine rankings.

For example, Mr. DiSorbo operates movingcost.com, a company that purports to provide consumers information about “the most qualified and professional relocation experts in the industry.” The operators of this website have attempted to increase its popularity by embedding text that includes hyperlinks to www.movingcost.com in tens of thousands of other websites.

In many instances, these linked websites are college newspapers and student organizations – like the Cornell Gymnastics Club and the Yale Democrats – or they are irrelevant link directories such as “Muscle-Body Links.” While these links have little or no relevance to the content of the websites in which they are embedded, they help increase movingcost.com’s popularity with search engines, making it more likely that consumers will find the page during an Internet search.

Once at the website, www.movingcost.com appears to be a legitimate tool for consumers to locate reputable moving companies. The homepage includes links to social media and a section on “Moving Tips.” Upon closer review, however, the site is little more than a tool for DiSorbo Brokers Companies to attract Internet customers.

The site’s “Featured Movers” – Moving Squad, MBM Moving Systems, American Van Lines, and Nationwide Relocation Services – are all companies owned by Mr. DiSorbo or his family members.

“Penguin” Hasn’t Bitten These Listings

The study began in October 2011 and lasted several months. The timing is important, because Google’s Penguin Update launched in late April was a change Google introduced to its search engine designed precisely to prevent sites using some of the spam tactics described in the Senate report from ranking well.

If the report was done largely before Penguin hit, then it might reflect a worse situation than after the upgrade spam filters went into place. But it seems Penguin hasn’t solved the problem. A committee spokesperson cited these examples of the problem still happening:

• “Mover in Surfside” brings up American Van Lines in the top results of all three search engines
• “Raleigh movers” has American Van Lines in the first page of results on Google and the second page for Yahoo and Bing
• “Mover in Tuscon” has Nationwide Relocation ranking in the second page of results on all three search engines

Here’s an example of the first, the top results from Google for mover in surfside. American Van Lines actually shows up twice in the listings there, once under its the americanvanlines.com domain and once under the moveinsurfside.com domain that it also appears to operate:

On Bing, the same search brings up American Van Lines in the first and second results for its moveinsurfside.com domain:

A Reasonable, Measured Request From DC

Remarkably, the letter doesn’t make any type of strange, outrageous demands of the search engines. Indeed, after having watched so much come out of Washington DC and government in general that seems ignorant of how search engines work, the report and lettter both reflect that some real homework has been done. Nor does it single out Google for what’s clearly an industry-wide problem.

The letter, signed by committee chair Senator John D. Rockefeller IV (D-WV), asks simply that the search engine companies take a closer look at the issue:

Because I know that your company devotes significant time and resources to improving the quality of your users’ searches, I am sharing the results of my Committee’s investigation with you and asking you to review them.

Internet search is a powerful tool for consumers. It helps them learn more about products and servies they are interested in purchasing, and it helps them find the best price and value when they decide to buy.

Unfortunately, the Committee’s investigation shows that a number of moving companies are using Internet-based commerce to take advantage of consumers.

A committee spokesperson I corresponded with about the report echoed the same:

We did the investigation into the moving companies and along the way, realized that the Internet brokers are a major component of the scam.

After the hearing, where Senator Rockefeller released the report, he figured the next best step is to let Google, Bing, and Yahoo know that we have evidence that suggests these shady companies are taking advantage of the algorithms. Then let them look into it and determine whether the companies are really gaming the system to boost their rankings.

It’s a refreshing, reasonable approach.

I wouldn’t be surprised to see Google in particular take snap action to correct this particular problem, as it did in November 2010, after a New York Times article focused on how merchants with bad reviews could still rank well. Within days, Google introduced a change intended to solve that issue. As an aside, the particular merchant that was featured was recently sentenced to four years in federal prison.

I’ll be checking more with Google, as well as Bing, about their responses to the letter. Yahoo is largely dependent on whatever Bing does, since Yahoo’s results come from Bing.

Of course, there are more scams that plague search results beyond these. The squeaky wheel of this particular problem will likely get fixed quickly. However, the letter will probably give Google more ammunition for pressing ahead with harsher Penguin Updates.

Some hit by it have argued that Google is trying to somehow wipe out businesses; here’s a US Senate committee actually encouraging that to happen, at least businesses that are getting ahead in search listings despite bad business practices.

Postscript (Sept 26): I now have a statement from Bing:

We appreciate the information passed along by the Committee. We work aggressively to detect low quality results and re-evaluate low quality links. We are continuing looking for ways to improve our search results to provide consumers with the highest quality and most relevant results.

And one from Google:

We make more than 500 improvements to our search algorithms every year to make them more useful, including a significant update this past April to combat practices like link schemes. We’re always looking for ways to make it harder for scammers to trick consumers, so we appreciate the specifics the Committee provided. Senator Rockefeller’s concerns point out how important it is that search engines continue to have the ability to constantly and quickly improve our results for our users.

Google also told me:

• The Penguin Update significantly impacted the Google search rankings of movingcost.com, the main site mentioned in the report, reducing its search traffic by about 75%
• The movingcost.com site was also caught in a crackdown on spam link networks in March 2012
• Google will also investigate more deeply based on the report’s findings of link buying, link spam, and other violations of Google’s quality guidelines — and demotions wil be seen in “very short order”

Related Articles

• The Penguin Update: Google’s Webspam Algorithm Gets Official Name
• Dear Congress: It’s Not OK Not To Know How Search Engines Work, Either
• The Emanuel Update: Google Will Penalize Sites Repeatedly Accused Of Copyright Infringement
• Google: Now Likely Using Online Merchant Reviews As Ranking Signal
• DecorMyEyes Merchant Vitaly Borker Sentenced To Four Years In Federal Prison

SERPs.com has quietly launched a SERPs Volatility Index, which aims to work like the MozCast tool to track search results fluctuations and help webmasters know if there was an algorithm update at Google and Bing.

While MozCast only tracks Google, SERPs.com tracks both Google and Bing. SERPs.com also shows a 30-day and 90-day volatility charge. It works a lot like the MozCast solution in that it tracks the top keywords for 1,000 popular websites on Google and Bing, adding up to tens of thousands of keywords being tracked. The tool is also updated daily.

Here are static screen captures of the SERPs Volatility Index as of today:

By the looks of it, SERPs.com and MozCast share the same interpretation of the Google fluctuations in the search results, for the most part.

Both the tools do show a change at Google over the weekend, which is what we reported this morning.

You can give the tool a try yourself at serps.com/tools/volatility.