Overlay and Non-overlay traffic in SiteSpect

SiteSpect is a great tool which offers server side testing. Benefits of server side testing include: not having a flicker effect and no cookie noise in data. 

When starting out with SiteSpect one of the hardest things is set up up tests and choosing between running Overlay or Non-Overlay traffic. After this post, you'll know what to do. 

Running tests Non-overlay

It's sounds perfect. Tests won't interfere with each other. But… your visitors will also be stuck in one test and won't be available for other tests. Plus, maybe you want to measure interaction effects (more about that at Overlay tests).

We were running A/B tests for a client with 1+ million visitors. All set up Non-Overlay. But if we ran too many of those tests on the same page one experiment would ‘eat up’ the visitors, and the newer experiment wouldn't get any because visitors where already bucketed in Non-overlay. 

When to run non-overlay tests in SiteSpect?

• To prevent technical interferences between tests (sometimes a test will work, but not in combination with the variant of another test)
• For special occasions

Running tests Overlay

This is the ‘normal way’ of testing. It means that visitors in Experiment 1 variant B can also be in Experiment 2 variant B at the same time. 

In this post on Speero Lukas Vermeer (ex-Booking.com) and Georgi Georgiev (data expert in CRO) suggest that we should always run tests with an overlay approach. Why? Well, if there is any interaction effect, you can measure and prove it. Plus you wánt to measure interaction effects because the tests you are running are heading to the direction you've chosen for the Product to develop towards. 

Additionally running tests Overlay means you won't lose any valuable time not testing and waiting for enough traffic to start a next experiment.

When and how to run overlay tests in SiteSpect?

• As default (you can filter out interaction effects). 
• QA en check if the tests work when visitors are in multiple tests at the same time

Read more in this additional guide on overlay and non-overlay traffic from SiteSpect. 

Need help setting up experiments and running a program to its full potential? Reach out to us.