I’ve ranted on before about trial & error economics before and it’s good to see the wider-world start to realise the power in testing. Noone doubts this but the unique flexible, adaptable and changeable nature of the ‘Net means doing lots of tests really does make a difference. Recently we embarked on a rigorous T&E program, 101 tests in 101 days and the outcome was, unsurprisingly, predictable. 90% of the changes generated a 10% improvement and the big 10 ideas generated the vast bulk of improvement - in fact just 2 changes generated over 80% of the improvement in conversion rates.
Over the 101 days we saw a 52% increase in conversion from visitor to data on the site in question. Surprisingly (for us and the agencies involved) nobody predicted which of the changes would be “killers” which reinforces the concept that you have to test and test and test and test even if the results are to establish what doesn’t work.
Testing on the web is simple to do but many people make basic errors in statistical understanding or linking unrelated events and successes. It can be quite dangerous to make any assumptions about the correlation between the tests and the results unless you have a complete understanding of the data and the factors that could and do affect the test.
Bryan Eisenberg is running a series of webinars that, if knowing Bryan and Future Inc, will be pretty interesting. I’ll be watching closely.