SEO For Dummies, 6th Edition (2016)

Part III. Adding Your Site to the Indexes and Directories

Visit www.dummies.com/extras/seo for great Dummies content online.

In this part …

Mapping your site

Registering with search engines

Appearing in specialized registries

Selling in retail marketplaces

Visit www.dummies.com/extras/seo for great Dummies content online

Chapter 13. Getting Your Pages into the Search Engines

In This Chapter

Submitting your pages to the search engines

Creating, submitting, and pinging sitemaps

Submitting to secondary search engines

Using registration services and software

You built your Web pages. Now, how do you get them into the search systems? That’s what this chapter and Chapter 14 explain. In this chapter, I talk about how to get your pages into search engines, and in Chapter 14, I explain how to list your site with search directories.

Search engines can find out about and index your site in essentially three ways:

· Links pointing to your site

· Simple link-submission pages

· Sitemap submissions

The most important of these is the first — links pointing to your site lead the search engine to it. The second is worthless. And the third is very important and comes with a number of ancillary benefits. In this chapter, I tell you about these strategies one by one.

Linking Your Site for Inclusion

In Chapters 16 through 18, I talk about the importance of — and how to get — links pointing from other sites to yours. The more links you have pointing at your site, the more important the search engines will think your pages are.

Links are also important because they provide a way for search engines to find your site. Without links, you might get your site into search engine indexes, but chances are, you won’t. And being included in the most important index — Google’s — is much quicker through linking than through registration. I’ve had sites picked up and indexed by Google within two or three days of having put up a link on another site. If you simply submit your URL to Google (the next method I look at), you may have to wait weeks for it to index your site (if it ever does).

But one or two links sometimes aren’t enough for some search engines. They often are for Google: If it sees a link to your site on another site that it’s already indexing, it will probably visit your site and index it, too. But other systems are far choosier and may not visit your site even if they do see a link to it. So the more links you have pointing to your site, the better. Eventually, search engines will get sick of tripping over links to your site and come see what the fuss is all about.

It’s all about links. The search engines believe that they can maintain the quality of their indexes by focusing on Web pages that have links pointing to them. If nobody points to your site, the reasoning goes, why would the engines want to index it? If nobody else thinks your site is important, why should the search engines? They would rather find your pages themselves than have you submit the pages.

Simple Link Submissions to the Major Systems

The top two search engines — Google and Bing — provide a very simple way for you to tell them where your site is. Submit your site for free on these pages:

· Google: https://www.google.com/webmasters/tools/submit-url

· Bing: http://www.bing.com/toolbox/submit-site-url

Google actually provides a similar method, from within the Google Webmaster account (which I talk about in the next section). The Webmaster account has a Fetch as Googlebot tool (under the Crawl menu). You can use this tool to have Google pull a page and show you the code it has crawled. You can then click the Submit to Index button to submit that page to the Google index. Still, it doesn’t guarantee that Google will index the page you submit, though it might be worth trying if you have some particular pages on your site that you can’t seem to get indexed.

How about Ask.com? It offers no way to submit your Web site. It relies on its searchbots to find pages through links.

However, now that I’ve given you these links for submitting your site’s pages, let me say one more thing: Don’t bother.

Using these submission pages is unnecessary and may not even work. As Google says, “We don’t add all submitted URLs to our index, and we cannot make predictions or guarantees about when or if submitted URLs will appear in our index.” Bing doesn’t bother saying anything beyond “Submit Your Site to Bing.”

I don’t bother submitting URLs to search engines using these simple URL submission pages. But I want you to be aware of them because these submissions are one of the biggest scams in the history of SEO (a business that is rife with scams). Hundreds of submission services convinced many thousands of Web site owners to pay to have their sites submitted to the search engines, often repeatedly. It was a totally pointless waste of money.

You don’t need to use basic URL submission. You need links to your site, and you should submit a sitemap, which I cover next.

There is one context in which “submitting” a site to the search engines does make sense, though: in the case of businesses submitting their information to the local-search systems. I cover this subject in Chapter 12.

Submitting an XML Sitemap

I do recommend one form of site submission. It isn’t a replacement for pointing links to your site, but it can, in some cases, improve indexing of your site. I’m talking about creating XML sitemaps, submitting them to Google and Bing, and making it easy for other search engines, such as Ask.com, to find the sitemap on their own.

In 2005, Google introduced a new XML-sitemap submission system and was soon followed by Yahoo! and Bing (and now, of course, Yahoo! and Bing have partnered, so they act as one system). In Chapter 7, I recommend creating an HTML sitemap page on your site and linking to it from the homepage. The sitemap is a page that visitors can see and that any search engine can use to find its way through your site.

But an XML sitemap is different: It’s a special file placed on your site that contains an index to help search engines find their way to your pages. You create and place the file and then let the search engine know where it is. This hidden file (visitors never see it) is in a format designed for search engines. I think it’s worth your time to create this file because doing so is not a huge task and may help, particularly if your site is large. Also, after you’ve submitted a sitemap and “verified” your submission, the search engines give you lots of interesting information about your site.

Sitemap, schmitemap

Don’t think that creating an XML sitemap is a magic trick guaranteed to get your site indexed. I’ve seen, for example, Yahoo! (in the pre-Yahoo!/Bing partnership days) pick up the sitemap file, put it into the search results, come by and pick up 3 pages from the index, ignore the other 20, and leave a whole bunch of old, nonexistent pages in the index. I’ve seen Bing apparently pay little attention to a sitemap. On the other hand, I’ve also seen Yahoo! pick up hundreds of thousands of pages on a large site soon after finding multiple XML sitemaps. The sitemap may help, but you have no guarantee whether or when the search engines will use it. So don’t think of the sitemap as an alternative to the hard work of creating links to your site.

In Google’s words, “Using sitemaps to inform and direct our crawlers, we hope to expand our coverage of the Web and speed up the discovery and addition of pages to our index.” If I can help Google by getting it to index my site more quickly, well, that’s fine by me.

What does this sitemap file look like? Take a look at Figure 13-1.

Figure 13-1: A small XML sitemap.

Don’t worry: XML sitemaps are easy to create; I show you how in the next section.

These sitemaps are typically named sitemap.xml (though different names can be used) and placed into the root directory of the Web site — in the same directory as the home page.

Creating your sitemap

What if you own a small site, though, and have limited technical skills? No problem: Plenty of free and low-cost sitemap-creation tools are available. Firstly, many CMS systems can automatically create XML sitemaps; if you use WordPress to create your site, for example, you can find a number of plugins that, among other SEO functions, create sitemaps for you. You can find “extensions” and “modules” that create sitemaps for Joomla and Drupal systems, too, and probably many, if not most, CMS systems these days. Even simple site builders, such as GoDaddy’s Website Builder, have functions to automatically create XML sitemaps.

If your site does not have some kind of built-in sitemap module, there are other options. If you’re the proud owner of a large, sophisticated, database-driven site, it’s probably a job for your programmers; they should create or find a script that automatically builds the XML sitemap.

You can also find a variety of scripts and tools that will create XML sitemaps for you; just search for create xml sitemaps. Note that many of the tools call themselves Google sitemap creators, because Google was the first search engine to use sitemaps. All you need, however, is the basic Google XML sitemap format for all the other search engines, so if it creates a “Google Sitemap” it will work fine.

Some of these programs run on your computer, some require installing on your Web server, and some are services run from another Web site. You can find a large list of these sitemap generators here:

code.google.com/p/sitemap-generators/wiki/SitemapGenerators

Google itself created a Sitemap Generator program (https://code.google.com/p/googlesitemapgenerator), which you can install on your Web server. It’s no longer maintained, but still available (it’s a Python script, so if you don’t know what that means and don’t have a geek who does, consider creating the file another way).

My favorite sitemap generator for small sites is XML-Sitemaps.com. You simply enter your domain name into a Web page, and it spiders your site, creating the sitemap as it goes, up to 500 pages. If your site is bigger, you can get the service to install a script on your Web server for $20 (assuming that your server can run PHP scripts). They also have a service for a monthly fee in which they automatically create a sitemap and periodically update it, and they host the sitemap on their servers; all you need to do is point to the sitemap. (I’ll explain how you tell the search engines where the sitemap is in a moment. The sitemap itself does not have to hosted on your site.)

Sitemap files can be as large as 10MB and contain as many as 50,000 URLs. If you need to exceed either limit, you must create multiple sitemap files and a sitemap index file that refers to the individual sitemap files. Your XML Sitemaps can also be compressed in the .gz file format so that they transfer more quickly to the search engine. For specifics of the sitemap protocol —which you generally don’t need to know — see www.sitemaps.org.

Submitting your sitemaps

You can tell search engines about your sitemaps in three different ways:

· Submit a sitemap through the search engine’s Webmaster account.

· Include a line in the robots.txt file telling a search engine where the file is.

· Ping the search engines.

The last method is pretty much optional. You should definitely submit your sitemap to Google and Bing and use a robots.txt file.

Submitting using the Webmaster account

You should set up an account on both the major systems (Google and Bing; Ask.com doesn’t currently have Webmaster accounts, and Yahoo!’s data comes from Bing, so Yahoo! no longer has a Webmaster system) and submit your Web site’s XML sitemap. Then review the various tools that are available. Here’s where you can find the Webmaster areas and sitemap-submission pages:

· Google: www.google.com/webmasters

· Bing: www.bing.com/toolbox/webmaster

You don’t need a separate account on each system for each Web site you own — you can submit multiple sites through each account, though the number of Web sites that you can manage through each account is limited (Bing limits the number of sites to 125 sitemaps; for Google, it’s 1,000 … enough for most people!).

However, in some cases, Web site owners don’t want search engines to know that their sites are associated; for instance, if a site owner has three sites, all of which rank on the first page for important keywords, he may not want it to be obvious that all sites are owned by the same person. In such a case, of course, the owner would set up separate accounts for submitting each sitemap.

Here’s the basic process you use at all these services:

1. Set up an account.

In each case, you have to set up a password-protected account.

2. Add the URL of your Web site.

3. Tell the search engine where the sitemap is.

You provide the URL that points to your sitemap; the search engine checks to see whether it can find the file.

4. Verify your site.

You can verify your site in several ways:

· You may use a special meta tag that you add to your home page (something like <meta name="google-site-verification" content="1bkH4kUFdeKgcMhxOj8e06-7faFDZqAnS2jvGUDITRg" />). See Figure 13-2.

· You can choose to get a verification file, a small text file, that’s stored on your server. For instance, Google provides you with an .html file that you can download and place in your site. The file contains a single line of text, something like this: google-site-verification: googlec4b99698d01b26f2.html.

· You can choose to add a record — a snippet of information — to your domain name’s DNS information. Don’t try this unless you’re sure you know what you are doing.

· Google Webmaster also allows you to verify your site by associating it with your Google Analytics account (see Chapter 24).

· Google Webmaster also allows you to verify by associating with your Google Tag Manager account (a system used for managing JavaScript tags).

5. Tell the search engine that you have used one of these methods, and ask it to “verify” the file.

The search engine then checks to see whether the file, meta tag, or DNS record is present (or Google checks to see whether your site contains the correct Google Analytics code). If the search engine finds what it’s expecting, it assumes that you must own or have control over the specified site, and thus is willing to provide you with more information about the site.

Figure 13-2: Verifying ownership of a Web site in Google Webmaster Tools.

Each system is different, so I can’t go into detail about how each works (though I look at Google Webmaster account in detail later in this chapter). Spend a little time digging around and you’ll soon figure it out. Remember, the basic process is to add your site URL, add your sitemap URL, and then verify or authenticate.

Using the robots.txt file

You need a robots.txt file in the root directory of your Web site, with the following line inside it:

Sitemap: http://www.yourdomain.com/sitemap.xml

The URL should point to your sitemap, of course. The URL tells the search engines that don’t provide a Webmaster account — such as Ask.com — where your sitemap is, so you’ll want to do this even though you’re submitting your site through the Google and Bing Webmaster account.

Pinging search engines

Pinging a search engine means sending a message to the search engine telling it where a sitemap is, and telling it that the sitemap has changed. At the time of writing, you can ping Google and Bing; Ask.com took down its ping service some time ago.

You can see this process in action for yourself. Create a sitemap and then change the following URL to show the path to your sitemap:

http://www.google.com/ping?sitemap=http://www.yourdomain.com/sitemap.xml

Copy and paste this URL into a browser and press Enter, and you receive this response from Google:

Sitemap Notification Received

Your Sitemap has been successfully added to our list of Sitemaps to crawl. If this is the first time you are notifying Google about this Sitemap, please add it via http://www.google.com/webmasters/sitemaps so you can track its status. Please note that we do not add all submitted URLs to our index, and we cannot make any predictions or guarantees about when or if they will appear.

These are the sitemap-submission URLS; put the full URL to your sitemap, including http://, after the = sign:

http://www.google.com/ping?sitemap=
http://www.google.com/webmasters/sitemaps/ping?sitemap=
http://www.bing.com/webmaster/ping.aspx?sitemap=

You can manually ping sites each time the search engine is updated. If you have a programmer build your sitemaps automatically by pulling data from a database, the programmer should add a ping function to ping the search engines each time the sitemap is updated. And, some sitemap-creation programs that you can buy have built-in ping functions; the version of the XML-Sitemaps.com program that you install on your server can automatically ping for you.

Are sitemaps bad? I’ve heard the argument that sitemaps are bad because you might rely on the sitemap rather than create links to your site. This bizarre argument is similar to the one that airbags are bad because you might rely on them rather than driving safely. Sitemaps aregood. But they’re not a substitute for links pointing to your site. You still need links, and many of them.

Using Webmaster tools, too

As the previous section notes, the two top search engines — Google and Bing — provide Webmaster accounts through which you can submit your sitemap and provide various tools related to your sitemap and your Web site. You need to verify that you own or manage the site (through the authentication or verification process I mention earlier), then submit a sitemap. You’ll be provided with some very useful information and tools to influence how your site appears in the search results.

Right now, Google has the best tools and statistics associated with its Webmaster account (see Figure 13-2). Table 13-1 explains some of these tools.

Table 13-1 Google’s Webmaster Tools

Figure 13-3: You can see how often Google crawls your site and how quickly pages download.

Google also has some important settings hidden away in the Settings menu; the little cog button on the top-right of the page:

This is pretty nifty stuff, eh? Sitemaps are great for big sites; they can really help with indexing. But even if you have a small site, submitting and verifying a sitemap is a great way to get some very useful information about your site. (Actually, strictly speaking, you can create a Webmaster account for your site, verify the site, and then access this information even if you haven’t submitted a sitemap. However, if you’re going to all that trouble, you might as well spend a few minutes with a tool such as XML-Sitemaps.com and submit the sitemap as well.)

Submitting to Secondary Systems

You can also submit your site information to smaller systems with perhaps a few hundred million pages in their indexes — and sometimes far fewer. The disadvantage to these systems is that they’re seldom used, compared to the big systems discussed earlier in this chapter. Many search engine professionals ignore the smaller systems altogether. On the other hand, if your site is ranked in these systems, you have much less competition because they’re so small.

These secondary systems may be worth submitting your site to:

· ExactSeek: www.exactseek.com/add.html

· Gigablast: www.gigablast.com/addurl

You can find more, including regional sites, listed on the following pages:

· www.searchenginewatch.com/article/2067248/Guides-To-Search-Engines

· http://dmoz.org/Computers/Internet/Searching/Search_Engines

Some smaller search engines encourage you to pay for a submission. Don’t. Unless you know for sure otherwise, you can safely assume that the amount of traffic you’re likely to get probably isn’t worth the payment.

If you plan to submit your site to many search engines, you may want to use a programmable keyboard or a text-replacement utility or macro program, such as ShortKeys (www.shortkeys.com), which can make entering repetitive data much quicker:

· Programmable keyboard: You can assign a string of text — a URL, an e-mail address, and so on — to a single key. Then all you need to do, for instance, is press F11 to enter your e-mail address. The old Gateway AnyKey keyboard, which you can sometimes find on eBay, is great (if it’s still functioning); I’ve also used the Corsair K95 with some success.

· Text-replacement utility: Replace a short string of text with something longer. To enter your e-mail address, for instance, you might type just em.

A few sites require that you submit your site with a username and password. Most sites require at least an e-mail address; some also require that you respond to a confirmation e-mail before they add your site to the list. Don’t use your regular e-mail address! Use a throwaway address because you’ll receive a lot of spam.

Is it really worth submitting to these secondary search engines? As I mention in Chapter 1, somewhere around 95 percent of all search results are provided by the major search engines, so submitting to the secondary search systems may not be worth the trouble. I think it’s one of those “Well, I’ve got the time, so I might as well do it” kind of things.

Using Registration Services and Software Programs

You can also submit your pages to hundreds of search engines by using a submission service or program. These services used to be very popular, though they are definitely dying out. Search for search engine submission and you’ll find both Web-based services and software.

You may find free services, but some are outdated, running automatically for a number of years now and not having kept up with changes on the search engine scene.

Many free services are definitely on the lite side. They’re provided by companies that hope you’ll spring for a fee-based full service.

Some submission services also combine search engine registrations with directory registrations, sometimes along with link packages. (I discuss directory registrations in Chapter 14.) By providing links pointing to your site, they can guarantee that your site will be picked up by the major search engines; it’s not the submissions to the search engines that are doing the work — it’s the links!

Note also that some submission services increase their submission counts by including all services that are fed by the systems they submit to. Some submission services inflate their numbers by including search engines that don’t even exist any more. In fact, I feel that some (many?) submission services are little more than scams, charging in some cases very high monthly fees for what’s really a service with relatively few benefits.

A few submission software programs are available as well, such as Submit Wolf (www.trellian.com/swolf). The big advantage of these software programs is that you pay only once rather than pay every time you use a service.

You actually have a couple of reasons to use these automated tools. You may get a little traffic out of these small search engines, but don’t bank on getting much. In many cases, though, the systems being submitted to are not really search engines — they’re search directories, and being listed in a directory may mean that major search engines pick up a link pointing to your site. I talk more about submitting your site to search directories in Chapter 14, and you discover the power of linking in Chapters 16 through 18.