Running an A/B test that actually means something
How sample size, significance and the Welch z-test work in Idukki. The pitfalls (peeking, novelty effect) and how we guard against them.
A/B testing without statistics is just guessing twice. Idukki runs a proper Welch z-test on every experiment and tells you when you can ship.
Sample size first
When you create an experiment, we compute the per-arm sample size required to detect a given relative lift at α=0.05 and 80% power. Run for at least that many sessions per arm before declaring a winner.
No peeking
Calling a winner the moment p < 0.05 inflates your false-positive rate. The platform gates the Declare winner button until you’ve hit the planned sample.
Novelty effect
For at least the first three days, returning visitors react to anything new. Run for at least one full purchase cycle (typically 14 days for DTC) to avoid being fooled.
Still stuck?
Email us at support@idukki.io or open the in-app chat. We answer within one working day.
Talk to a human
