Which of these Amazon homepage designs is better?
It turns out it depends what we mean by “better.” My gut would say the one on the right, because the design is cleaner, simpler. Amazon put it to an A/B test and it turns out the one on the left performed better. Orders dropped significantly when Amazon tested the simpler design.
Greg Linden points to a deck of PDF slides by Amazon employees. It’s a couple years old, but has some more good examples of lessons they learned with hard data to back it up.
For a good overview of A/B testing, something most of us can do, check out Jakob Nielsen’s Alertbox.