Disappearing Spiders
Reasons
why a spider may not index all or part of your site
Return to page
1.
Make sure your also read the
bottom of this page!
An algorithm of Natural Discovery.
If your site is large or may be a little overgrown
with less ranked pages, an algorithm may consider those pages not
listed on the engines WOULD BE found naturally if a surfer worked
through your site once a 'hidden
query' or 'home
page' was returned.
What to do: If your rubbishy page is listed, and your best isn't,
swap but make sure you keep the same URL name.
Server Down Time, Hops and Skips Missing
Your web host server, national line rental,
oversea cable etc may prevent your site from being spidered by the
regional datacenters. There may be one hop missing, and bang, your
missed.
Poor spider ability
Check the many different elements that make a page
suitable for spider bots, robots etc at
oy-oy. There are some particularly handy tools and datacenter
co-location tools.
You share an IP address
If you used a shared IP address, it it less and
less likely that your pages will be recorded. Those shared IP's are
heavily taken and abused by spammers. If you can, purchase separate
web hosting from a reputable company AND... Ask if they service any
poor sites. Ask lots of pre sales questions and research them
thoroughly.
Choose a respectable and well noted hosting
company. Some add pages inside your domain pointing back to
themselves and offer a free submission to the SE's (of those pages).
BEWARE: These web hosting companies change your web
pages when a search engine spider requests them. The web server will
return a different web page than the web page that is returned when
a normal web surfer visits your web pages (this is cloaking a web
page, domain or folder so you must check out the potential web
hosting company on a forum search) (type the name 'company forum' to
find if any other webmasters have comments) (also, go to the Google
Cache page of your web - it should be the same as your designed, not
someone else's).
For yourself, never attempt a cloaking practice.
Timing
For what ever reason, their database of upcoming
scans may be booked, overloaded, awaiting lag periods, etc.
If after 3 months, no indexation, re submit.
Poorly Submitted URL
If you submit your domain on a whim, expect the
same result. Delve into the categories on offer by each SE. For
Google, add the category where 'suggestions' are allowed. Be
accurate, your site's traffic will be better ranked if you do.
Check to see if your site can be indexed.
On this page, we show some free spider analyzers
you can use to test your site or page. Use them, they may show
errors that you were not aware of. If you find a better search
engine simulator, contact us and we'll add it to the list.
Check to see if the spider can see your links
If you have an unusual script controlling your
hypertext links and the SE's cant see them, you will suffer from low
page count. Your page may appear as a one page site if your
navigation instantly blocks a spider moving past the first
millisecond.
Is your site content OK, or just all fluff designed
for Advertising?
A page that uses a system of drawing in sponsored
results just for revenue purposes wont last long in rankings. There
is a definite backlash building on those non content sites.
Make site with real content, valuable to the human
surfer and the bots will soon recognize your pages.
Does your site have spam elements, inadvertently
or unknown.
Go through all your pages and look for things that
maybe at first your thought were a good idea, and are now hurting
your web pages. Most webmasters as they build sites forget about
little snippets, e.g. A white font that was once on black, is now on
#FFFFFF. Its considered a spam element even if unintentional.
If you have any hidden text on your pages. Remove
it immediately. This is definitely penalized.
Have you purchased an older domain?
If you are transferring an older domain to new
content, you should contact the SE directly and let them know of the
change in direction. They might never answer, but somewhere down the
line, it will be actioned.
Strongly Recommended
If you are intending to register a new domain
OR have registered a New domain. Check to see if it was
once live, and is now expired. It may have a bad history!
BE VERY CAREFUL ABOUT THE NAME YOU CHOOSE - REGISTER ANY NEW DOMAIN
WITH ABSOLUTE CARE &
PRIOR KNOWLEDGE.
Go immediately to
WayBack Machine
and look at any domain name and it's history.
http://web.archive.org/collections/web.html
Also, ask the domain registry reseller for a history of the domain.
Ask lots of pre sales questions when deciding to start a new site.
We use this free web based service on any domain we optimize
for our clients PRIOR to any optimization or web page design work.
In most cases, the reason your site has not yet been
indexed is...
Time. Time to get in the cue and get a result. In the
meantime, you can work on better optimizing your pages for your
audience, build a better navigation structure, read all our articles
and from others posted on the web. Every 0.001 improvement you make
now will reward you for years to come.
The biggest thing you should concentrate on
is More Online Page Time, To Do this, create a site that
gives your readers more information about your site or interest.
