It’s the end of the year and we here at 10e20 have a tradition of sharing our top stories in search and social media. This year is no exception. Without further ado, here are the top 50 stories in search.
50. Twitter microblogging tool goes mainstream (November/December)
49. Alibaba goes IPO (10/29)
48. Server downtime plagues Yahoo Stores on Cyber Monday (11/26)
47. Bodog online casino domain name frozen (8/28)
46. Porn.com sells for $9 million (5/15)
45. Jason Calacanis launches Mahalo human powered search engine (5/30)
44. Google starts to publish AP news in new partnership (8/31)
43. Digg partners with Microsoft for ads (7/25)
42. MySpace and Facebook study discovers different social class divisions (6/25)
41. Microsoft comes out with Silverlight to compete with Flash (4/15) and then launches Silverlight-powered search engine, Tafiti (8/20)
40. Google acquires Zingku mobile service provider (9/27)
39. Bloglines comes out with AJAX beta (8/27)
38. Google Checkout crashes eBay party, eBay pulls Google ads (6/13)
37. Yahoo acquires MyBlogLog (1/8)
36. Google acquires Postini (7/9)
35. Social image site Flickr gets its 2 billionth upload (11/13)
34. Google founders Sergey Brin gets married (May), and Larry Page follows (December)
33. Fox serves YouTube a subpoena after unaired episodes of 24 show up on the social video network (1/26)
32. Yahoo! Mash social network launched (9/17)
31. Business.com is sold for $345 million (7/27)
30. Yahoo appoints click fraud czar (3/22)
29. Yahoo announces Pipes (2/8)
28. Conferences, conferences, and conferences
27. Wikipedia adds nofollow to all its links (1/22) prompting much controversy to “reduce Wikipedia’s PR to zero” (1/23)
26. Google announces Android and joins the Open Handset Alliance (11/25)
25. MySpace hoax leads to suicide (11/16)
24. Digg drops its top 100 users list from the site (2/2)
23. Gmail is now open to everyone (2/14) and gives out 6GB of storage (12/25)
22. Digg’s HD-DVD revolt shows that the community owns (5/5)
21. Ask Eraser launches (12/11)
20. Google announces Knol, the Wikipedia killer (12/14)
19. Vanessa Fox leaves Google for Zillow (6/17), and then gets joins Ignition Partners and Search Engine Land (11/14)
18. Google acquires Feedburner (6/1) and later gives Pro options for free (7/2)
17. Microsoft invests $240 million in Facebook (10/24)
16. Threadwatch closes down (5/25)
15. eBay acquires StumbleUpon for $75 million (4/18)
14. Microsoft comes out with Webmaster Portal (11/7)
13. Yahoo’s Terry Semel resigns as CEO (6/18)
12. Google OpenSocial launches (10/30)
11. Facebook Beacon creates serious privacy concerns (11/23)
10. Yahoo! Panama becomes a reality in the US (2/5)
9. The paid link debate continues (4/13) and sites lose PageRank for selling links (10/25).
8. Michael Gray creates a controversy in his SES San Jose presentation when he declares “Google is not the government” (8/20)
7. Yahoo introduces Search Assist (10/2)
6. Facebook rolls out applications with F8 initiative (5/24)
5. Google goes over $700 a share (11/1)
4. Ask 3D is launched by Ask.com (7/5)
3. Microsoft acquires aQuantive for $6 billion (5/18)
2. Google acquires DoubleClick for $3.1 billion (4/13), a move that is harshly criticized by Microsoft (4/16). The acquisition is reviewed (5/29) and then approved (12/20)
And the biggest story of the year? Well, think blended search, because that’s what it is:
The #1 story of the year: Google comes out with Universal Search (5/16)
Don’t forget to subscribe to the 10e20 RSS Feed!
Bookmark this post:
robots.txt: People Don’t Always Want Search Engines to Crawl Their Content
May 18, 2007 by Tamar Weinberg | SEO, Search Engines
What goes through your mind when you read about the silly lawsuits against Google accessing portions of your website? What do you think when you visit the Internet Wayback Machine and find hundreds of pages of your site in its full form (almost)? Most of you wonder what is going on in the minds of these clueless people. Don’t they understand how the web works?
That’s right, folks. In case you’re not in the know, the web works in a certain way. A brand new site generally does not get indexed in search engines for a period of months. Over time, the search spiders find your site and your interlinked pages begin getting crawled. Eventually, someone will search for something and your website will hopefully come up.
Not everyone is happy with these search results, and oddly, some people just don’t want to be found. In fact, on websites that are relatively large, the spiders crawl so many pages at once that people have defined the silliest robots.txt files. For example, check the hilton.com robots.txt file (which I learned about during the robots.txt Summit at Search Engine Strategies last month). Note their first two lines.
# Daytime instructions for search engines
# Do not visit Hilton.com during the day!
Dear hilton.com webmaster: Search engine spiders have no understanding of English. Look at the picture that accompanies this blog post. The spiders run through a site, grabbing pieces of data to spin into a web — the World Wide Web — and then it moves on. It does not pay attention to your personalized messages (but on behalf of the spider, thanks for the attention). Spiders do not understand anything. Spiders are robots.
If you want more information about robots.txt, you can visit one of the premier sites for the Robots Exclusion Standard. Understanding the implementation is important. If you have personal information that you don’t want the search engines to find, you can block it out.
Simply use the following:
User-agent: *
Disallow: /mysecretdirectory
This code blocks all search engines from accessing content in “mysecretdirectory.” This is also helpful if you have concerns about duplicate content. The typical example is if you have printer-friendly versions of pages on your site, but you don’t want to be penalized by Google for having these pages available on your website. You could create a robots.txt file with the following code:
User-agent: Googlebot
Disallow: /printer-friendly
You’d obviously be replacing the /printer-friendly directory with the directory your printer-friendly documents reside upon.
There are additional applications of robots.txt. Some search engines, such as Google, will now let you specify the path of your sitemap in the robots.txt file as such:
Sitemap: http://www.mysite.com/sitemap.xml
You can also be selective and block off certain search engines, including the Internet Wayback Machine, as discussed earlier. This way, old versions of your site will no longer be accessible.
User-agent: ia_archiver
Disallow: /
There are a variety of other bots out there that crawl your site on a regular basis. It’s not just about Google, MSN, Yahoo, or Ask. You can get an idea of what works and what doesn’t by experimenting. Note that if you plan on blocking content, it takes time for it to drop out of the search results if these pages were indexed already.
Fortunately, in my post about the Google Webmaster Central tool, I mentioned that you can change the crawl rate of the Google spider. Unfortunately, you can’t be any more specific and invite the spider only during late night hours. But you can set the speed for the spider to crawl your page at a slower rate so that it doesn’t negatively impact your web server or website performance.
Two helpful tools on robots.txt are the Robots.txt Generator and the Robots.txt Builder Tool. You are also encouraged to read the additional helpful documentation at the Google Webmaster Help Center.
Before you get mad at the search engines and start frivolous lawsuits, realize that it is your responsibility too to prevent the searche engines from accessing your web page, if that’s your desire.
Bookmark this post:
Good advice does not necessarily have a single application. Solid advice can be “recycled” and used for a variety of situations. In January, I wrote how advice for startups can be applied to marketing a website. A recent post I read applied advice in human resources to SEO. I want to take that post and go the next step: into the world of social media.
This post highlights four main points, all of which have applications just about anywhere when you are looking to become active in the social media world and to have the sites that people care to share.
Lesson 1: Attract Attention
The key to being discovered is having something that attracts attention. Whether you write a list post, a how-to, or you gather a collection of viral photos, the key is about writing something that captivates your audience, whoever they may be. The title needs to grab the eye. Graphics can even further help the cause. Do something different. An old post by Robert Scoble about blogger Kathy Sierra resounds in my mind almost every time I write a blog post.
For social media sites like Digg, you’re limited to a description that is short and sweet, but you need to say something that attracts the attention of other users of the service so that your story gets promoted. Look at successful stories and see what it takes. Don’t simply reiterate the title in the description. Be descriptive enough without losing the reader.
Lesson 2: Know Your Keywords
Keywords in social media are important. Just like selling yourself in HR and your site in SEO, you need to sell your linkbait by using those prominent keywords that will ultimately be linked to when your viral campaign kicks off. Without at least some mention of the keywords in the obvious areas, you may not attract the right audience. Use the right keywords, and for any social media submission, study the power words and use them as needed.
Lesson 3: It’s Important to be Popular
You have no idea how important this is in social media. Having a lot of friends is critical. If you’re a nobody and are perceived as such, you won’t get far. You can’t create a website and expect traffic to come without any marketing or promotional efforts. The quote, “If you build it, they will come” is not true. Become well-known. With millions of users on the Internet, you’re not a big deal unless you put effort into self-promotion. That’s how you become popular. Be active in your community. Befriend people with similar interests and they’ll reciprocate.
Lesson 4: Stand Out in the Crowd
Beyond being like everyone else, you need to do more. What do typical users of social media networks do? Not much. In order to be a success, you need to study the network — and this will take time — to know what stands out and lets you be noticed by others. The popularity will follow, but they do go hand in hand.
There’s the difference between what’s good and what’s great. Be outstanding. Work on content that people notice. Use those keywords to draw the attention and your popularity will follow.
Bookmark this post:
Tips and Tricks: An Introduction to Google’s Webmaster Central
May 4, 2007 by Tamar Weinberg | GoogleAny webmaster should be aware of the tools available to them. This is especially true for Google’s Webmaster Central, a collection of tools that allow you to better understand your website and give you more information about it.
One of the first things you will notice when you go to Google Webmaster Central is that you can check if your site is being indexed by Google. The site status wizard allows you to see when your site was last crawled and if it’s in Google’s index.
You might be told that you have potential problems and that you should submit a sitemap. It is recommended that you do so, even though you can now declare the path of your sitemap through your robots.txt file.
If this is your first time at Google Webmaster Central, you need to verify that you are in fact the owner of the site. You can do so with a meta tag or you can upload an HTML file.
Once you are approved, Google will give you more information about your site, such as any HTTP errors it may have, 404 pages discovered, URLs that cannot be followed, URLs restricted by robots.txt, URLs that timed out and unreachable URLs.
You can also see the crawl rate of your site to determine how Google is accessing these pages:
And you can set up a preferred domain (e.g. www.10e20.com or 10e20.com).
Other useful features include the Enhanced Image Search and you need to specifically opt-in to have your images searched to improve search quality. More information on this is at the Webmaster Help Center.
A relatively new tool is the URL removal in Webmaster Central; you can remove content from the Google index easily with this functionality. If you wanted to remove any URLs, you can specify whether you want to remove individual URLs, a specific directory, or even your entire site.
Note that Google wants user feedback, so on the upper left hand corner, you can rate the effectiveness of the tool.
In the Statistics portion of the Webmaster Central, you can get an idea of your PageRank of individual pages and Google will also provide you with data of pages with the highest PageRank.
You are also able see which search terms are generating results from your site and the average top position. You can also see which of these queries was actually clicked. I assure you that you’ll find some interesting surprises here. 
One of the newer and cool things about Google Webmaster Central is that you can see a page analysis of external links to your site and the anchor text used. You can download CSV files of this data as well.
You can also see what pages have been linked to the most:
And finally, you can submit your sitemap and track its progress:
These are really the basics of the Google Webmaster Central. There’s a bit more if you dig deeper. All in all, this is a pretty comprehensive tool that is constantly being worked on. It gives webmasters a good deal of information about sites so that webmasters can get an understanding of the search engine’s understanding of your website. I encourage all webmasters to use it.
The Cluetrain Manifesto and What it Means for Your Business
Apr 26, 2007 by Tamar Weinberg | Blogging, Social Networks
On the Internet, information gets outdated pretty darn quickly. Months pass and old blog owners consider pruning old posts that seem outdated. However, there’s one piece of information that has remained true and current for over a span of nearly a decade. The message comes from the Cluetrain Manifesto, which was written in 1999. The Cluetrain Manifesto was written well before its time; it is a precursor to social media of today. The Cluetrain Manifesto’s message is simple yet extremely powerful: markets converse with each other and the Internet facilitates that communication.
Here’s the opening paragraph to their key 95 theses:
A powerful global conversation has begun. Through the Internet, people are discovering and inventing new ways to share relevant knowledge with blinding speed. As a direct result, markets are getting smarter—and getting smarter faster than most companies.
Eight years later, the Internet is laden with social networks and corporate blogging is considered a must. Social networking is mainstream. Businesses are talking to consumers in ways previously perceived to be unimaginable.
And still, more “traditional” businesses are treading carefully, afraid to take a plunge that could substantially improve relations for their businesses. But the Cluetrain Manifesto’s point is so important. Customers talk and the audience listens. Don’t believe me? One of the most pivotal examples is Consumerist. A recent blog post describes a man who wrote about the lousy maintenance performed when a Comcast technician installed wiring in his house. Consumerist, a very well known blog, picked up the story and Comcast fixed the problem.
On the other side, from a business perspective, Peter Da Vanzo writes about how opening communication through blogs has changed his life. He writes about the perks: job offers, conference speaking engagements, business opportunities, insider information, and of course, a bunch of friends, among other benefits from sharing your breadth of knowledge with your audience.
This could be you. This should be you.
If it isn’t, what are you waiting for?
Bookmark this post:
At SES NY, there was a surprising amount of people who I spoke to who were not embracing social media as much as they could, which I feel is a huge mistake. Rand Fishkin presented some very compelling reasons to pursue social media. From his examples, articles that had good content and subsequently were made popular on a variety of social sites got a burst of traffic, and as articles hit the front page on a regular basis, he saw an increase in rankings for particular search terms. Some numbers that were thrown out: Digg is a site with 2,000,000 million daily visitors. A Digg, once it hits the main page, can bring anywhere from 10,000 to 30,000+ visitors and over 1,000 links after three weeks.
It is no surprise, then, that an earlier survey by JupiterResearch reported that 48% of Internet Marketers intend to take the plunge into social media this year. More important is a very recent study performed by iProspect that tried to understand the behavior of social network users. Jordan has a great summary of the findings, all of which are important. Among these findings are that:
- 33% of Internet users have made purchases based on social networks
- 25% of Internet users visit the popular social networking sites at least once a month
What social networks were reviewed? MySpace, Facebook, YouTube, iVillage, Yahoo! Answers, TripAdvisor, del.icio.us, LinkedIn, Craigslist, and Amazon. All of these sites have user generated content and community elements, though one might wonder if some of these are truly the same kind of “social networks” that we are familiar with.
Another recent survey that I covered on the use of web 2.0 sites generated different results (which is likely due to demographics) and found that a whopping 43% of employees at large companies (>1000 employees) engage in social networks. That’s a huge number! This number is likely to grow — not diminish — as more and more people become more comfortable with online social interaction.
We are in the second quarter of 2007. I’d imagine only a small fraction of those 48% companies surveyed by JupiterResearch have started seriously using social media. It’s not too late to get involved, and you should, sooner rather than later. Leveraged properly (read our past coverage on being a power Digg user, being a power StumbleUpon user, and being a power Reddit user), social media can do wonderful things for your website, your business, your product, and your message.
Bookmark this post:
APIs: An Easy Way to Make Novel Websites Familiar and Fun
Apr 12, 2007 by Tamar Weinberg | Web DevelopmentOkay, you might not have had that two billion dollar idea, but that doesn’t mean you can’t partake in its features. APIs (Application Program Interfaces) allow programmers to access publicly available services from widely known services, which can make your “web 1.0″ (sorry about the moniker, folks) site more welcoming to users looking for something new.
The concept and utilization of APIs is simple. Many well-known sites share some portions of their site code so that you can experience similar functionality on your website without having to navigate to the original site. One of the most well-known APIs available is the Google Maps API, which allows you to embed Google Maps on your web page with just a little bit of code. Ever track a package on a website and see a Google Maps page showing your package’s progress as it travels from shipping facility to shipping facility? Didn’t it look cool to see Google Maps on that third-party website? That’s what you can do by unleashing the power of APIs.
APIs are available for just about any well-known online community or site today. Take a look at what’s available at your fingertips:
- YouTube
- Google APIs (includes Google Maps, AJAX Search, Google Calendar, Google Checkout, and others)
- del.icio.us
- Flickr
- Yahoo! Answers
- MediaWiki, which includes Wikipedia
- Netvibes
These are just some samples of APIs that you can reference for integration into your website application.
What does this mean for me? Let’s take a look at an example in one of our very own client websites, JohnnyJetAways.com. We combined Google Maps with the client’s vacation home system data, and the information can be plotted on a very familiar map, provided by Google, as seen in the screenshot below:
You don’t necessarily need a whole website overhaul if you already have an established site, so APIs make it much simpler and more fun for you and your users.
Bookmark this post:
Fresh content is a very important asset for the growth and popularity of your site. There are many reasons why this is important.
First and foremost, having fresh content benefits your link juice. The more you write, the more opportunity you have to be linked to. I wouldn’t say to take the Twitter approach, however. When writing an article, write something of substance. Write something longer than that — but not too long so as not to neglect your readers. If your content actually requires great length, break it up into small pieces and link to each other in subsequent articles, making sure you link appropriately to refer them to a previous related article. Create a series of related posts. This works in blogging as well. Michael Gray recently did a series on local search. If you have not seen his vast interview coverage, I urge you to check it out to get an idea of how he did it.
Fresh content also gives you crawling visibility. Search engines will completely neglect you if they see a website that hasn’t been touched in ages. Having content updated regularly means that you’ll have the spiders visiting you more often, and that gets your content on the search engines up sooner than a site that doesn’t update as often. If you just update an image, the spiders might come back. Content does matter.
If you write for a very specific vertical, it is important to have the content written by you, the expert, rather than someone else. Ultimately, you are the closest person to the service you provide or the product that you sell. You would be doing yourself a great disservice if you shifted that responsibility to someone less knowledgeable. Furthermore, as I learned in the Eric Ward linkbuilding webcast:
There is no outside party that has the passion to help you as much as you want to help yourself.
With more carefully crafted content, you get more link juice, and this in turn results in more visibility in the search engines. When people search, they are actively looking for someone to fulfill a basic need. If you fulfill that need, you’ve won.
Bookmark this post:
Can Twitter Serve as Highly Targeted Marketing Tool?
Mar 27, 2007 by Tamar Weinberg | Social NetworksMany of you may know of the recent surge in popularity of the Twitter service. Quite simply, the goal of Twitter is to answer a question: “What are you doing?” Last month (as I began to use the service more aggressively since I signed up in December) I realized that not everyone is answering that question anymore. Twitter is becoming more of a microblogging tool, a news announcement tool, a networking tool, and as of late, a marketing tool.
That’s right. Twitter can be used for marketing.
Currently, the hot “bot” (literally) update comes from Woot.com, an online store that provides daily deals. As you can see on Woot’s Twitter page, the site does not feature anything but hot deals on the products it is selling for the day.
For marketers who use Twitter, this is a great way to reach out to a different demographic. As you may be able to see from the screenshot, Woot currently has 1302 followers. Those 1302 followers, who made the choice to follow Woot’s announcements, receive highly targeted and desired traffic about products that Woot is selling.
Is this the only reach of Twitter? Apparently not. Today, TechCrunch announced that Twitter will be introducing web services, which will should broaden the Twitter audience.
Interestingly enough, for those who are not able to commit to blogging, particularly due to time restraints, using Twitter would be an interesting exercise for companies to try communicating their own announcements. Some blogs already provide headlines on the Twitter site. But if you didn’t have a blog, you could limit your announcements to 140 characters (that’s a Twitter restriction) and share what you think would best serve the community that follows you. This would, as previously stated, save time and give you the opportunity to have a “blogging” voice in “microblogging” terms.
If you’re looking to get on the Twitter bandwagon, tell your family and friends and let them sign up so that they can follow your business. The site itself makes broadcasting a lot easier with a very targeted audience.
Bookmark this post:
If there’s one request that I always cringe at, it’s to disable a site because a customer decides that it is in his best financial interest to “close shop.” Later, if we own the domain name and get notifications about its upcoming expiration date, the customer may already be long gone and is not reachable with any contact information we may have for them.
I can’t say this has happened a lot, but it isn’t unrealistic either. If there’s one thing you should keep when you decide to shut down your business, it’s your domain name (and let’s hope you grabbed your website files too). But why? Believe it or not, it’s an important element for ranking in search engine optimization.
Consider the idea that you are creating a brand new website. The domain name is unheard of and therefore is unranked. You search for your domain name in Google and find nothing. Weeks later, your brand new site is still nowhere to be found. Time passes, and your site may start appearing in the search results, but the progression is going very slowly. Google is crawling your site with caution. They don’t know you well and are determining if you can be trusted.
An element of trust is important for ranking well on search engines. Trust can be conveyed through linking strategies, like when your site is linked to from a site that Google already considers to be an authority, but age, too, plays an important role.
Why should it matter? Simple. Considering recent studies showing the spammy nature of brand new websites, how is Google supposed to know that your site is more legitimate than the other spam site (including on their own domain)? It doesn’t. Google is not human. Trust comes with time. Spam sites don’t last for a long time and certainly won’t be linked to from authority sites.
The problem is that if you lose your domain name, it’s likely gone forever. If it expires, it goes into a pool and it will get snagged pretty quickly if the domain name looks like a worthwhile investment on the part of people who actually make it their business to buy domain names that could make them rich. If you’ve made a name for yourself and have to shut down without keeping the domain name under your ownership, but later, you realize that you’ve made a regrettable decision, you might have a hard time getting your “brand” name back (especially if there are no associated trademarks). You’ll end up likely having to build a brand new name — and you’ll have to concern yourself with regaining that trust once again. If you don’t own the old domain anymore, you can’t set up 301 redirects to retain any rankings you might have. You’re back to the beginning.
Depending on the TLD (top level domain) you choose, your domain renewal shouldn’t be more than $15 per year in the worst case scenario. Even if you’re not sure you’ll ever use the domain name again, it’s a small price to pay for the time and money that you will have to reinvest in building your credibility in the search engines again.
Bookmark this post:
