Comments on: Bing is an Improvement over Live, but Still Not Google Quality: Evaluating Bing With Mechanical Turk

By: gino

gino — Sat, 27 Feb 2010 00:17:08 +0000

I’d like to know if someone has already carried out a blind test on search engine results.

It could be very interesting to ask users which results are considered best without knowing the name of the search engine that produced that particular SERP. This could avoid the risk that users’ opinions could be influenced by brand related ‘noise’.

Obviously the results could be compared only on a semantic basis, but I think that the statistical reliability could be significantly better.

By: Jessica

Jessica — Mon, 21 Sep 2009 13:35:29 +0000

Bing is currently dropping sites and pages like theres no tomorrow (just like MSN always has done)i’ve noticed this with several sites.. they do come back though in the majority of cases.
Bing is a joke.. the all new search engine, yet behaves exactly the same as MSN and returns the same results.

By: Santiago

Santiago — Sun, 09 Aug 2009 18:29:06 +0000

Hi, I´ve contacted you by email some days ago and did not have any feedback. My email address is in the email box

By: bset

bset — Mon, 22 Jun 2009 16:02:34 +0000

Similar to the experiment using 100 random queries, here is another example of the same type, where users can plugin their queries and select the most relevant search engine themselves. bset.royans.net Google and Yahoo both seem to be much better than Bing, though Google is a leader by a long margin. Whats also interesting is that, it looks like different search engines might be better for different types of content ( or could be based on location, language)

By: LisaStratus

LisaStratus — Sun, 14 Jun 2009 13:00:39 +0000

Very much a prompt reply :)

By: rourbboob

rourbboob — Thu, 11 Jun 2009 22:58:13 +0000

visit us!
newsbox.cc
newsbox.us
nbstatus.wordpress.com
NOW!

By: David

David — Thu, 11 Jun 2009 13:18:36 +0000

@Offbeatmammal: The problem with Blind Search is that the search engine used to retrieve each column of results in clearly marked in the page’s source code. Anyone who can use “View Source” can check which column is which and vote accordingly.

By: Vaibhav

Vaibhav — Thu, 11 Jun 2009 12:03:58 +0000

Hi Anil,

I had to write a post on this topic after reading yours. Here’s a link: http://blog.gadodia.net/bing-vs-google-no-fancy-analytics-pure-personal-experience/

By: Michael S.

Michael S. — Thu, 11 Jun 2009 08:58:44 +0000

Do any of the results vary much by location, or previous search history? Google’s results seem to vary significantly by location, for example, which is frequently annoying, but I guess it must affect the results for the better if the search engines are doing it.

By: Vaibhav

Vaibhav — Thu, 11 Jun 2009 07:11:26 +0000

Let me put it this way, even though I switched my default search engine to Bing, I consistently have to go back to Google to get what I want (after searching on Bing first).

I would love for Bing to be better, for the simple reason that even Google doesn’t do the best job in search.

But Bing is just not competing, IMO – look at the difference between these two queries:
http://tinyurl.com/l52fkt and http://tinyurl.com/ltbyl9

It looks to me that Google understands user intent better

By: brendano

brendano — Wed, 10 Jun 2009 19:08:44 +0000

err, i don’t mean differences, i mean the comparison responses (of course)

By: brendano

brendano — Wed, 10 Jun 2009 17:19:45 +0000

Xianhang: that’s what the null hypothesis tests are for. The first graph (bing vs google) has about p=0.04, which means that if people were guessing randomly, there’s less than a 4% chance we would have seen the sort of results we saw. 5% is the customary threshold for “statistical significance.” The second graph, as we stated, could have been due to chance (it’s p-value was higher).

That’s why we say we think google is slightly better than bing, but it’s a little bit of a wash whether bing is better than live.

Panos: i think it was a one-sample t-test of the differences, but i’m not sure

By: Panos Ipeirotis

Panos Ipeirotis — Wed, 10 Jun 2009 17:14:39 +0000

Lukas, what test did you run to confirm the statistical significance?

By: David

David — Wed, 10 Jun 2009 16:13:26 +0000

Kragen – tinfoil hat much?

By: Offbeatmammal

Offbeatmammal — Wed, 10 Jun 2009 15:12:40 +0000

For a quick and simple way to “taste test” the difference yourself check out the amazing Blind Search – http://blindsearch.fejus.com/

Adds Yahoo to the mix and removes the logos so you don’t know