Archive for the ‘Search engines’ Category

I apologize for a computer-geek-speak post. Skip it and bear with me, if it leaves you clueless.

When I updated my site a couple of months ago, little did I know that I was making it invisible to Google’s indexer – Googlebot.

One of the improvements that I wrote into the site was that it would identify the prefered language of the browser and select that as the default language.

In the headers passed to the server in the request there isa header called accept-language and it tells the server which language I prefer. My browser passes the value en-us,he;q=0.5 which means I prefer US English, after that I will accept Hebrew and the q=0.5 means that my preference is a 50% one and that I will also accept anything else. From reading the documentation it would appear that if the header said q=1would mean “send me nothing other than these languages”.

Anyway, my code does the following pseudo-code:

If you have a language cookie, then use that value

Otherwise, if the first prefered language (Request.UserLanguages[0]) is “he” then use Hebrew

Otherwise use English

The problem with my code is that the accept-language header is not mandatory and the standard I linked to above states that if no accept-language header is provided then it is assumed that all languages are equal. Googlebot knows this standard and does not provide an accept-language header in its request. This is very logical if you think about it. Google wants to index everything and they don’t what language it is in.

My code was returning an unexplained 500 to Googlebot and got me totally wiped out of search results. What was happening is that when I referenced Request.UserLanguages[0] I was getting an exception because the Request.UserLanguages collection was uninitialized. I have fixed this and put the whole section in a try-catch. I suppose I could have just checked for Request.UserLanguages.length == 0 but I decided to play super safe.

BTW, the way I debugged this is worth a mention as well. If you search for ServerVariables in Google you get a whole lot of pages that demonstrate a dump of the server variables of your request. The version of this page shows the server variables of Googlebot’s request. Thus, by comparing the page in the Google cache with the one I see I could see what the environment differences were between Googlebot’s request and mine. I then used this spoofing tool from “Smart IT consulting” to see when Googlebot could see the page.

Incidentally, before I got to this solution I found a simple “hack” for Mozilla (more a configuration than a hack) that allows you to change the user agent reported by the browser. I tried this and made my browser pretend to be Googlebot. However, this didn’t help because I still had an accept-language header in my request.

Read Full Post »

I just got listed on the Artists Blog Search. This is a useful site which uses a Google custom search to search listed artist blogs. If your blog isn’t there then you can drop them an email and get yourself listed. Sure can’t do any harm.

Read Full Post »

Since I started my blog, I’m getting a lot of hits on my site from Google searches. The links in my blog page seem to have done wonders for my site’s rating and the blog itself doesn’t too bad either:

Some searches to do for fun:

  • definition of real art (blog comes out #1)
  • real art studio (blog #5)
  • Paintings by Israeli Artists (rafistern.com comes in at #3)
  • israeli artist painter (rafistern.com#8)
  • jerusalem fine paintings (rafistern.com#4)
  • שדה פרגים (rafistern.com #4 after Van Gogh)
  • בניאס (rafistern.com #6)
  • Nahlaot (rafistern.com #8)
  • חומות ירושלים (rafistern.com #3)

The downer is that if you change the searches even a little bit, then the results come out differently. Shows just how fickle Google can be.

For example while “Paintings by Israeli Artists” comes in at #3, “Paintings Israeli Artists” comes in at #11 on the second page and they say that they are ignoring the “by”.

Worse, while “jerusalem fine paintings” does really well at #4, “jerusalem fine art” is just nowhere to be seen. My guess is that the latter is what most people search for.

Oh well, more work still to do…

Read Full Post »