2023 Most Common CRO Beginner Mistakes

CRO mistakes I've recently seen in agencies and startups.

Let me start by saying: trying something is always better than not doing anything. Some 'CRO purists' like to shame mistakes, but this post is here to help you (god knows how many mistakes I've made...) 🌱

❌ No pre-test analysis
➤ … you will not know if you can run a test with the traffic on your pages and how long it should run. Use the Speero calculator, or other bandwidth calcs.

❌  Not doing A/A tests
➤ Yes, A/A testing takes time. But the damage you can cause when you don’t run A/A tests is bigger than postponing a few tests. There are many things that can go wrong with a faulty technical or data setup, which you can catch early with A/A-testing. Imagine you report on conversion rate with an event that fires twice, or you only have returning visitors in your tests. This can cause irreversible trust issues with the stakeholders or clients, and they might even want to turn away from Experimentation.

❌ Always putting the tests live with an uneven split (5% in variant and 95% in control) because of 'the risk' and increasing this during the testing period.
➤ … leads to various problems like SRM-alert, not reaching required sample size, or finding the same result later (it will just take you longer to conclude your variant is not working). 

Uneven splits are not wrong, it's just more advanced. You need to be good in calculating sample size and checking for SRM-alert.

❌ Preferring redirect (split url) testing v.s. injecting code (in case of client side testing)
➤ … using a different page as a variant often leads to a more noticeable flicker effect for the user (+ the user might see variant-b.html) in the url bar, and don't get me started on data tracking issues.

❌ Loading the script of the testing tool late
➤ … Preferably the script of your (client side) A/B-testing tool is loaded as one of the first things to reduce the flicker effect of the variant. Some advice: get the code for the tool out of GTM and place it in header. Use the sync code. If this reduces your Page Speed then async is a good second choice. Or… start testing server side :).