You Can Participate in Crawling the Web! From Your PC!
by Mike Banks Valentine

Oh, for the simplicity of the good old days of 1995! Where
all a web site owner had to do was put important keywords
about her web site in the metatags of the HTML code of the
front page of her site with a clear description and title
to rank well in the search engines and make it easy for web
searchers to find her.

What made it become so difficult to gain and maintain a bit
of internet visibility? It got crowded on the web. Crowds 
made it more difficult to get noticed - so instead of simply
including proper metatags, web site owners had to begin to
make themselves stand out to get noticed. Many did that by
resorting to trickery, like keyword stuffing and link farms.

Since then, it has become increasingly difficult to get a
web site indexed and ranked well at the major search engines.
Strategies have evolved that seem to work, then mergers and
partnerships change and so do all of the rules. 

It never ends and the confusion grows with each passing day.
It seemed that paid inclusion and pay-per-click were the
only ways to get your site listed anywhere besides Google
lately, let alone ranked well. 

LookSmart has come up with a brilliant idea to grow their
own index that, if it catches on, will offer huge benefits
to both LookSmart and to the participants in the program.
That seems to be the elusive win-win situation that all but
Google have been looking for. It's called distributed web
crawling and gives web site owners a hand in growing the
Looksmart index as well as the ability to  include their
own site in the results!

This win-win situation requires participants to download the
"Grub" distributed crawler client to their PC and allow it
to run when the computer is otherwise idle. The software to
run a LookSmart Grub web crawler is available at:

It is based on open source software and is made available
to all to distribute and even modify! Since installing
"Grubby" on my own PC three days ago, I've personally 
contributed over 100,000 URL's to the LookSmart database!
It's fascinating to watch and the GUI is both entertaining
and attractive. It can run invisibly when minimized in the
system tray or you can sit and watch as the URL's fly by
during a crawl from within two available views on your PC

I love the concept and plan to continue helping LookSmart
grow their database. I've also taken advantage of the option
to have "Grubby" crawl both my own sites and those of clients
by including a small text file with a digital ID in the root
directory of each server I have access to.

There are no drawbacks here, just benefits. I help "Grubby"
to crawl the web for LookSmart and Grubby helps me to get my
own sites listed there by agreeing to crawl them regularly.
No payment involved other than the donated bandwidth and the
PC computing power that would be running a silly screen saver
otherwise. They even provide a cool screensaver of their own
that simulates the crawling process with colorful orbs that
represent the sites you are crawling to help weave the web!

When you visit the front page at the URL above, you see the
latest statistics on your contribution to crawling the web.
They've even introduced a measure of competition into the
program by showing top contributors and ranking them. Sunday
the stats at the top of the page read, "323 clients running 
- crawling 24,355,058 URLs in the last 24 hours." It will
be interesting and very telling to see if that number grows
significantly or very soon because that will be some measure
of the success of the program over time.

This process and the software all seem to harken back to
friendlier times on the web where community mattered more
than profit. This could even threaten the mighty Google if
it spreads, since we all love to participate in things that
interest us and LookSmart has a really interesting little
Grub helping to grow the web from my home office and yours.

Mike Banks Valentine is a Search Engine Optimization
specialist practicing ethical small business SEO

Search Engine Placement, Optimization, Marketing