Multivariate Testing Explained and Simplified

MVT Explained

Multivariate Testing (MVT) is a method of website optimisation that allows you to identify how different elements within a webpage effect your users’ behaviour.

Through MVT, changes can be made to some or all of the elements on a webpage and incredibly valuable information can be collected to show you the effect those changes are having on your users.

The variations of the changed elements are combined to create many different versions of the webpage and traffic is split between each of those different versions. By tracking relevant statistics it is then possible to identify the specific combination of elements that result in the best possible conversion.

Version Element Element
Version 1 A A
Version 2 A B
Version 3 B A
Version 4 B B

We can work out the amount of versions our test will run by doing some simple maths. If we were to choose two elements on a webpage to be tested, with each element having two variations the results would be four versions of the webpage (2 x 2), as shown in the table above.

What’s the Difference Between Multivariate Testing and A/B Testing?

Understanding the difference between MVT and A/B Testing can be a little confusing as they are similar concepts but there are some important differences in relation to both the implementation and the resulting data.

Say for example we want to test a new homepage design on our website, we will be changing the navigation style, the main banner across the top of the site and the mini basket. We ask two of our designers to go away and produce one wireframe each for the new homepage:

A/B Test OneA A/B Test OneB

Using A/B Testing we can prove that out of versions A and B of our homepage, version B had the greater click through rate and was therefore the winner.

The main limitation of this test and of A/B Testing itself is that we cannot determine why version B was the winner.

Was it the navigation style? The new banner? Or was it the change to the mini basket? MVT Testing can give you that information and therefore the data that is collected is much more valuable.

Using the MVT method for the above example would have resulted in testing of the following versions:

A/B Test One
A/B Test Two
A/B Test Three
A/B Test Four
A/B Test Five
A/B Test Six
A/B Test Seven
A/B Test Eight

This would have given us not just a winning version but also enough information to tell us exactly why the version won.

Is A/B Testing Dead?

It’s important to remember that A/B Testing is still a useful method for testing your website and D&W often still do use A/B Testing when we are helping a client test and optimise their website.

The main advantage of A/B Testing is the speed at which the test can reach completion.

An MVT test would usually have many, many more versions to be tested, this means that the traffic has to be split out across many more versions and it therefore takes longer to get enough traffic through the test campaign to start showing significant results.

A/B Testing can be useful if, for example, we need to run a very quick test that will reap results in a matter of hours, or perhaps when we are making a very radical change to a webpage. One example could be that we want to test out two entirely different designs for our homepage and wish to A/B Test the two versions to quickly find a winning version which we can then refine via MVT Testing.


To find out more about testing, from the ROI to the risks, download a free copy of our new CMO Guide to Multivariate Testing – written in partnership with iStrategy, the leading Global Digital Marketing Conference.

By 2018, 42.4% of world’s population will be plugged into the internet, with around 3.6 bn people able to access it at least once a month.
RT : What is a ? It attaches rich photos, videos and media experiences to Tweets. http:/…