RubberStamped.org

Google Custom Search - Business Edition Review

2007-12-05T14:07:00.000+13:00

Emmanuel Evita from Google invited me to review the Google Custom Search Engine, Business Edition, and kindly provided me with access to the service for a case study.

I will use this software to enhance the usability and utility of my web directory, Rubberstamped.org. Hopefully this case study prove useful to others who are considering implementing Google's hosted search service.

So What Is It?

The Google Custom Search Engine allows you to create your own search engine. Sign up for the hosted service, select a few options, and you’re done. The Business Edition has additional features, including customization of search results available through an XML API, reporting features that give insight into visitor behaviors, and the option to turn off ads and Google branding.

What Problem Does The Business Edition Of Google Custom Search Engine Solve?

In-site search can be a frustrating, both from the perspective of a user and webmaster.

If the search result relevancy isn't high, the user will go elsewhere. Search scripts and databases can be cumbersome and create administrative overhead.

The Google software provides you with site search that works. You get to use Google’s leading edge search technology, so the result sets are, as you’d expect, impressive. The service is hosted, so there is no administration required once it is setup.

You can also incorporate the result sets of other sites into the search engine, which is a feature I intend to use heavily. The service is very easy to implement, and the prices are competitive.

The Project & Goal

I run the directory Rubberstamped.org. We accept listing requests from webmasters and, following hand-review, we place sites in a category. Our aim is to create a quality, spam-free directory in which all results have been hand reviewed and categorized by editors.

The downside of this model is the lack of relevancy. Directory indexes can be shallow, which means the result sets can be weak when compared to a crawler-based engine.

To solve this problem, I intend to replace the existing directory search function with Google Business Search and incorporate result sets from other hand-picked human edited directories and verticals to use as backfill.

This approach will serve two purposes.

Firstly, we still get to keep a degree of control on spam, because we hand-select sources which operate a similar editorial policy to Rubberstamped.org.

Secondly, we increase our utility to the end user. If the utility to the end user is increased, we stand to gain more users, which should, in turn, convince people to list with us.

Google have placed a high degree of emphasis on human reviewed content. We will run a search service that includes hand-reviewed resources to provide a meta-directory search service for those users who prefer directories. Hopefully this study demonstrates how webmasters can compliment Google search by creating second-tier hybrid search services.

A description of the Rubberstamped meta directory search service, powered by Google Business Search, can be found here.

How To Set-Up Google Custom Business Search

The setup procedure is very easy.

Go to http://www.google.com/enterprise/csbe/index.html
Click the Sign Up button on the top right hand side
Give your search engine a name, provide a description, and specify sites to include in the search.

For those of you running a search engine on one site, your list will obviously consist of just one site. You can add to the list at any time.

And that’s it. No really! Compare this procedure with implementing a search engine software script.

You can then sign up for different account levels, depending on your support and functionality requirements. Custom Search Business Edition starts at $100 a year for searching up to 5,000 pages, and extends to $500/year for up to 50,000 pages. Larger volumes of pages are supported through Google's enterprise sales group.

Core Sites

I’ve chosen these sites for inclusion based on their depth, freshness, and adherence to the hand-reviewed philosophy. We will be adding quality sites on a regular basis. Note: none of these sites are affiliated with Rubberstamped.org. Their results sets are incorporated via the Google search index.

The core sites, besides Rubberstamped.org, include:

Mahalo.com is a search service beta launched in May 2007 by Jason Calacanis. The aim of Mahalo is to track and build hand-crafted result sets for many popular search terms. Mahalo's directory employs human editors to review websites and write search engine results pages that include text listings, as well as other media, such as photos and video. Their emphasis on hand-review is similar to ours, so Mahalo is a good fit for inclusion.

BestofTheWeb.org is a directory established in 1994. They have around 172,000 pages listed in Google, and these consist of hand-reviewed results. The BOTW model is very similar to our own.

Yahoo Directory requires little introduction. Huge directory based around hand reviewed listings, which should help to provide needed depth.

Business.com is a well established business search engine and directory.

We have used a filter option so users can choose to search just Rubberstamped, or include backfill from other sites. Give it a go. The setup process took a few minutes, then a few hours spent reading the FAQs to learn the finer points. We'll be looking to delve into the API in order to further hone our result sets, and will make this a topic of another article.

Google & The Semantic Web

2006-12-20T14:57:00.001+13:00

Good post on Googles' initiatives in the semantic web space.

Google already uses largely automated techniques to identify and deal with Web spam, email spam in gmail, click fraud, etc. We won’t begin by using completely automated techniques to process and make decisions based on data found on the Semantic Web and will be able to develop partly automated systems to decide what data can and should be trusted and by how much.

The notion of trusted data is important, and Google has such mechanisms in place, such as Page Rank. Part of the picture is human categorisation.

What Is The Semantic Web?

2006-11-13T14:11:00.000+13:00

People have been using this phrase for years, although it hasn't always been clear what is meant by "semantic web".

In What Is The Semantic Web, Nova Spivack does a great job of explaining the theory.

Summary:

The Semantic Web is not separate from the existing Web.....It simply adds new metadata to the existing Web. It merges right into the existing HTML Web just like XML does, except this new metadata is in RDF....

Text Links

2006-03-21T13:20:00.000+12:00

Excellent blog from on linking strategies from SEW Conference speaker Debra Mastaler.

Excellent Directory List

2006-03-09T11:03:00.000+13:00

The Strongest List provides an excellent, regularly updated list of directories.

Free Directories

2005-03-13T00:27:00.000+13:00

I've often been asked why we don't provide a free submission service.

As much as we'd like to, the model won't scale. The modest review fee we do charge keeps out the chancers, and ensures serious webmasters are provided with fast response times. We'll treat you well, and our users will benefit from fresh listings featuring the latest content.

For those people looking for free directory listings, try Dmoz, Zeal, and Wow. Or search Google.

Moving Beyond Outmoded Directory Models

2005-01-31T21:06:00.000+13:00

We're turning round listing requests in far less than 48 hours. Usually within the hour, so we're exceeding the goals we've set for ourselves. Thanks for all the kind emails about our response time!

People often ask how we can hope to be comprehensive if we don't accept free listings. The thing is, we do list sites for free. Works like this - for every paid listing request we accept, we seek out and list a non-commercial listing in one of the less commercial categories. The end result should be a good balance of commercial and non-commercial interests, thus mimicking a real world environment.

The politics of search

2005-01-22T10:14:00.000+13:00

As pretty much expected when politics come into play, DMOZ pulled the listing of RubberStamped. Apparently, not enough listings, although when we asked how many listings were required, we didn't get a reply. Que Sera Sera.

When I posted the "No Sleep 'Til Dmoz" thread, the objective wasn't really to get listed. It was to illustrate the slow turn-around time. Suffice to say, if Rich hadn't stepped in, my submission would have rotted along with the hundreds of others that no doubt sit in the queue. DMOZ should be turning around all submissions in far less than eight days. If the submissions are too numerous to deal with, then surely policy and procedures need to change to address that problem? If policies cannot be applied consistently, then what use are they?

The judgement of "quality" or "completedness" of this directory is a red herring. The real issue is that there are many directories on the web that do qualify to be listed but aren't because the queue has been neglected. And because the category isn't fresh, it isn't particularly useful. Will we see the category carefully maintained and kept up to date from now on? I doubt it. It will likely be left dormant like so many others. DMOZ has long since lost sight of it's original vision.

I don't begrudge DMOZ as a whole. They're overwhelmed with submissions and subject to the political problems that come with "free" and volunteerism on that scale. What I don't see is any will or effective strategy to solve those problems.

Those are the real issues that need addressing :)

DMOZ come through. Sort of.

2005-01-21T10:44:00.000+13:00

Here's how not to do marketing.

There are two main features to the Rubberstamped directory: turn-around time and fresh content. To illustrate this, we decided to show the problem with DMOZ i.e. slow turn around time and lack of fresh content. We submitted to DMOZ, then planned to count the days to acceptance. Of course, this count would go into the hundreds before acceptance. It was highly likely we would never be accepted.

We were wrong.

What happened was Mr DMOZ founder himself, Rich Skrenta, got in touch, personally, and mentioned that we had been added to DMOZ. Days to DMOZ: Erm...eight. And a personal note from the founder.

Heh.

How embarrassing. The least we could do was approve Mr Skrenta's listing request in RubberStamped :) Rich has also been so good as to agree to an interview, and talk about what he's up to these days. We'll be publishing this on searchengineblog.com as soon as we're done chatting. That may take some time as Rich is a very interesting chap.

So, DMOZ:1 RubberStamped: 0

Own goal.

Mining the deep web

2005-01-18T14:58:00.000+13:00

It's interesting trying to find that data beyond the reach of the search engines, or the great sites that, for one reason or another, have been buried in the search results.

Here's a few tools I've been using to look way beneath the surface:

Specialized vertical search engines
Deep Query Manager
Gary Price's compilation of links to deep search interfaces

More to come. Started a deep search category.

Ping-o-matic

2005-01-13T22:31:00.000+13:00

I'm going to add a few of my favourite blogging tools over the next few days.

One of the cool tools we have listed in our Blogging Tools category is Ping-o-Matic. This tool enables bloggers to ping the leading blog aggregator sites using one simple form. The bots then know when to come and crawl your fresh content.

The search engines need something like this.

Two day deadline

2005-01-13T22:09:00.000+13:00

It's now day two of our 'No Sleep 'till DMOZ' experiment. Unsurprisingly, we are not listed in DMOZ, nor had any communication with regards to our submission. Here's the cat we submitted to: Computers: Internet: Searching: Directories

To be fair, we didn't expect a response within two days. DMOZ make no assurances as to when a site will be processed, if at all.

However, day two is significant - Rubberstamped turns around all requests within this time period. We feel that webmasters should be given a high level of service from directories, which in turn keeps our directory fresh and up-to-date, which in turn benefits users. Everyone wins.

Stay tuned...

No sleep 'till DMOZ

2005-01-12T13:00:00.000+13:00

One of the reasons for starting this directory was that we felt that DMOZ can often be slow to add sites, and that for a directory to remain fresh and up-to-date, then it has to add relevant sites quickly.

Is this statement unfair?

Let's try a little experiment - we'll submit this site to DMOZ and count the days until acceptance.

Step One: Search dmoz for an appropriate category. Received error message "The Open Directory search is temporarily unavailable". No problem, we'll browse the categories.

Step Two: Found appropriate category: Computers: Internet: Searching: Directories. Browsed the existing sites listed in that category and it appears that our site is a good fit.

Step Three: Read submission guidelines. We appear to qualify.

Step Four: Complete the Add URL form. Here's what we submitted: Rubberstamped - A searchable web directory of sites categorised by topic. Features a mix of editorial and webmaster submissions.

Step Five: Read the terms, and the bit about granting Netscape Communications non-exclusive rights to our submission. All looks ok. Submit.

Step Six: According to the confirmation page, our submission was received. There's no detail about when consideration will take place, but we shall see.

The count begins...

Internet Trends

2005-01-11T17:46:00.000+13:00

I enjoy watching trends. Here's a list of top ten trend sites that I regularly keep an eye on.

Why DMOZ started

2005-01-02T14:25:00.000+13:00

Interesting to note the reasons DMOZ (then GnuHoo) was started:

"...a San Francisco programmer named Rich Skrenta was getting frustrated with Yahoo: It's links were rotted all to hell, and it was slow to add the sites he submitted. "Clearly they were neglecting the maintenance of their directory"

and

"I think that GnuHoo will have to become some kind of commercial enterprise if it's going to fulfill its mission."

It's true that frustration exits today, only not with Yahoo. It's also interesting to note Skrenta thought a commercial model was required.

Tsunami

2004-12-31T16:37:00.000+13:00

It's a cliche to say "words cannot describe". But it's true. The horror and sense of loss is unspeakable. We urge everyone to give what they can. Tonight, up to five million people lack basic supplies to stay alive.

Categories and Ontology

2004-12-31T13:48:00.000+13:00

Took some good advice from some good people, and have decided to reign in the empty categories a bit. Having too many empty categories wasn't a good look at this early stage. Instead, the ontology will roll out gradually.

I've hired another editor to to help build a unique dataset of authoritative websites. We don't have a free-for-all submission queue, rather we seek out internet sites and add them to the directory by hand. Quality, not quantity. We also welcome listing requests from webmasters for a review fee of $25.

The volume of webmaster submissions is also ramping up (strange for this time of year, eh), and we're seeing very high quality overall - we've only needed to reject a handful thus far.

This reaffirms my opinion that free submission policies result in mostly junk. Professional webmasters who value their domains know the value of a quality directory listing, and the value of their time. They don't want their listings buried under a pile of fly-by-night sites. Our review fee and editorial policy ensures there's a barrier to entry, and this has the benefit of filtering out most of the deadwood.

In a few weeks time, we'll have the start of something unique and valuable. Stay tuned, and happy new year :)

Living life in public

2004-12-28T15:25:00.000+13:00

When it comes to launching websites, there are two schools of thought:

a) plan extensively, test, retest, wait until everything is ready. Launch.
b) throw it out there, see what happens

I favour the later school. I'll tell you why:

1) Feedback from the community is invaluable: if your website is cast in stone before it is launched, it's already dead. The web is an interactive space, and users shape websites' every bit as much as the webmaster does. Well, good ones, anyway. Always listen. Seek feedback. Change tact. Iterate.

2) I wouldn't have met up with Edward Lewis: Edward is a killer coder. He told me where I was going wrong, and showed me what I could do better. If I hadn't put this out there, Edward wouldn't have seen me. I'm glad he did.

3) What doesn't kill you, makes you stronger: I've tied together some good ideas (if I do say so myself). I've planned. I aim to fill a gap in the market that has been overlooked. So far, I've got stuff right. I got some stuff wrong. So I've stripped out the stuff that was wrong, and pushed more of the stuff that was right. This directory has been made stronger.

The danger: is that people will judge you once. Then overlook you.
The benefits: outweigh that risk enormously. I've got a growing list of fresh new angles I hadn't even considered before, and that haven't been covered in the directory space up until now.

Things are taking shape faster than I ever thought possible. Roll on 2005.

Topic Centers

2004-12-28T10:23:00.000+13:00

We're populating categories with "Topic Center" sites, also known as authority sites.

We identify and select these sites using a methodology that includes both machine and human analysis. These listings are not the result of webmaster submission or paid placement. These are free listings sought out by our editorial staff.

Works like this - for every paid listing request we process, we seek out and add a less commercial Topic Center site. We also seek out non-profit and educational resources. This policy helps us achieve a balance in editorial between commercial and non-commercial interests.

Over time, authorities can change. So too will the authority sites we list. We feel that this approach will give us a unique, interesting and fresh data set. Some directories have become tired, outdated and stale because the existing data isn't reviewed and churned on a regular basis.

Unlike some directories, we are not focused on delivering quantity. Instead, we're taking a reductive approach based on the notion that communities of interest have a limited number of authorities at the center, around which the entire community orients. We're looking to identify and categorize these authorities.

We do not feel that the traditional, passive directory model of waiting for submissions from webmasters will allow us to accomplish our goals. Instead, we have adopted a model that will enable our editors to pro-actively identify and categorize authorities and, more importantly, keep these listings up to date.

In addition, our URL queue is open to webmasters who feel that their site will compliment our Topic Center listings, and those submissions are subject to a review fee in order to help fund our work.

This policy will also serve to ensure our editors time is used in the most efficient and productive manner possible, as free submission policies can often result in significant administration over-head and low-quality listing requests.

Rubberstamped.org does not carry advertising, and accepted listing requests should not be construed as such.

The value of feedback

2004-12-26T10:24:00.000+13:00

We've been seeking feedback from the webmaster community. The value of this process has been enormous, and reaffirms my faith in the ClueTrain manifesto point 18: "Companies that don't realize their markets are now networked person-to-person, getting smarter as a result and deeply joined in conversation are missing their best opportunity".

What we've gained from the process:

We've seen problem areas in our approach. We're addressing them.
We've seen technical things we could be doing a whole lot better. We're doing them.
Our copy needed tightening up. Onto it.
We've seen spelling mistakes and omissions. Correcting.
We've seen whole new opportunities in the market we hadn't even considered previously. We're very happy about that!

We will be incorporating feedback that we received, and will be introducing fresh initiatives that will change people's perceptions about directories, and the value they can bring to the web.

Thanks all. Back early January.

Authorities

2004-12-24T18:17:00.000+13:00

If you look at communities of interest, anywhere, not just on the web, there will be a few authorities around which a given community orbits at a given time. e.g. a charismatic leader.

In the 60's, some communities oriented around Indian spiritualist leaders. The thinking of the time, and a widespread interest in eastern philosophy, made those people highly relevant to certain western communities.

Then, times changed.

New authorities, based on a number of factors, were chosen. People moved on. What seemed important in the past, was no longer.

You can see it on the web, too. Certain sites become the center of attention for a community, for a time, then something changes and people move on. I think it would be fair to say that the search engine marketing world has Danny Sullivan at the center. Danny is the main authority around which the community orients.

I'm interested in identifying those authorities. And keeping track of them. Looking at communities and seeing around which point they orient. Pick out those sites and add them to the directory, and not rely on passive submission to make that happen.

That would be valuable, would it not?

Rumours and gossip

2004-12-23T09:59:00.000+13:00

There's a great thread over at cre8asite [cough] started by me about the thinking behind the directory.

I'm a great believer in a book called the Cluetrain Manifesto, which states that you should converse openly with your customers. This helps you build a better service. Helps you build effective relationships. Google get this (GoogleGuy). Microsoft get this (SearchChamps). Yahoo get this (YahooBlog).

A few little birds tell me that some, behind closed doors, have a few questions and are debating this directory. Great! Head on over to cre8asite, start a thread on their own site, get in contact with me, whatever. Glad to have a chat! Help me shape the service the way you want it. I'm here for the little guy - the webmaster. My door will always be open to you.

Let me be clear: We're building a bigger, better Open Directory. A semantic web adds value to the world. We've got a business model that will help make this happen.

That's our goal.

Add Url to Directory

2004-12-22T15:40:00.000+13:00

Fairly self explainatory (hopefully). One question I've been getting is in regard to the top level categories. "Why can't I add a URL to these"?

These cats are locked for adding URL's because the topics are quite general. Conceivably, many sites could fall under the category, which would quickly become a mess. In order to keep the directory well ordered, please add a url to to a sub cat.

New Directory Beta Launches

2004-12-21T15:25:00.000+13:00

Another directory? So what? The net is flooded with them.

Yes, that's true. However very, very few look after both the needs of the user, webmaster and the needs of the search engines.

That's where Rubberstamped comes in. We're building a credible directory that categorizes sites based on human review. The webmaster gains a credible listing and doesn't have to wait weeks or months for approval, like they do in many free-to-list directories. The major search engines can crawl and use our data. The users get unique, fresh content with emphasis placed on the user experience.

We realize that time is valuable, so we won't waste yours. We offer a two day turnaround on Add URL requests. Often a lot quicker :)

Let's face it - we're not competing with the major search engines. End users aren't going to switch to general directories en-masse to do their searching - that's what Google, Yahoo, MSN and the other major engines are for. Instead, we're building a credible, third-party link citation service for search engines, and other machine based retrieval systems, to utilize. We're building a directory based on usability principles to better serve the end user. We're doing our bit for the semantic web. We're identifying hubs and authorities within web communities using our own mix of machine and human selection methodology. All sites are reviewed by humans, the link describes *exactly* what a user can do on the target site, and the site has been verified as being intended for end users.

We're going to be advertising heavily in the new year. We're going to build the best service of its' kind.

Stay tuned...

RubberStamped will launch soon

2004-12-06T15:28:00.000+13:00