- A/B Test
- A controlled experiment that compares two versions of a page or element by splitting live traffic between them to see which drives more conversions.
- AI Optimization
- Using AI to generate, deploy, and continuously adapt website experiences automatically, instead of running each experiment manually through design, dev, and analysis cycles.
- Confidence Interval
- The range your true conversion rate is most likely to sit inside, based on what your test has seen so far. Less data → wider interval → less certainty about the real number.
- Confidence Level
- The probability that your test result is real, not a fluke. A 95% confidence level means you accept a 5% chance you're chasing noise.
- Control
- The original, unchanged version in an experiment. It is the baseline that every variant is measured against.
- Conversion Rate
- The percentage of visitors who complete a desired action (purchase, sign-up, click) out of everyone who saw the page.
- Multi-Armed Bandit
- A multi-armed bandit is an algorithm that dynamically allocates traffic to the best-performing option in real-time, balancing data collection with immediate performance.
- Null Hypothesis
- The starting assumption in any statistical test: that nothing interesting is happening. In A/B testing, it means assuming your variant performs identically to your control, any observed lift is just random noise until proven otherwise.
- P-value
- The probability of observing a result at least as extreme as yours if there were truly no difference between variants. Lower means stronger evidence.
- Sample Size
- The number of visitors required in each variant to detect a meaningful difference reliably, calculated before the test starts.
- Statistical Power
- The probability that a test correctly detects a real effect when one genuinely exists, typically targeted at 90%.
- Statistical Significance
- The confidence that the difference between variants is real and not down to random chance, conventionally declared when the p-value drops below 0.05.
- Type 1 & Type 2 Errors
- The two ways an experiment can mislead you. Type 1 (false positive): you declare a winner that doesn't actually beat the control. Type 2 (false negative): you miss a winner that's genuinely better.
- Uplift (Lift)
- The relative change in conversion rate of a variant compared to the control, usually expressed as a percentage.
- Variant
- Any modified version of the page being tested against the control, labelled B, C, and so on.
- Website Personalization
- Showing different visitors different versions of your website based on who they are, where they came from, or what they're trying to do, so each person sees the experience most likely to convert them.
