Shopify Rollouts vs A/B Testing Apps: Can You Test a Whole Product Page Template?

Shopify now has native A/B testing. Shopify Rollouts, found under Markets > Rollouts in your admin, can split traffic between your live theme and a changed version, including a reworked product page template, and report which converts better. That raises two questions merchants keep asking: is Rollouts good enough for A/B testing, or do you still need an app? And how do you A/B test a whole Shopify product page template, not just a headline?

This guide answers both, checked against Shopify's help center and changelog and each app's own documentation as of November 2026.

Shopify Rollouts vs A/B testing apps at a glance

What Shopify Rollouts is

Shopify defines a rollout as "a scheduled set of changes" to your main theme, checkout and accounts pages, discounts, product catalogs, or a combination. There are three types:

• Launch. Applies changes permanently, either all at once or gradually to a percentage of visitors.
• Event. Applies changes for a set period, then rolls them back. Built for things like a seasonal sale.
• Experiment. The A/B test. It compares a control (the resources in their current state) with a treatment (your changes). New Experiments default to a 50/50 split and an end date 90 days after the start, and you can change both.

Rollouts arrived in stages. Shopify's Winter '26 Edition announced it as a way to "schedule theme changes and A/B test." The changelog then lists:

• March 31, 2026: scheduling and A/B testing of theme changes, with market targeting for Advanced and Plus plans.
• June 5, 2026: A/B tests between two entirely different themes or checkout setups, plus mutually exclusive experiments so several can run at once.
• September 22, 2026: a redesigned setup that shows reach and split, and flags conflicts earlier.
• October 2, 2026: discount tests.
• October 23, 2026: catalog tests, covering prices and availability per market.

Which plans include it

Per Shopify's requirements page, rollouts are available on the Basic plan or higher, and Experiments need the Grow plan or higher. On Shopify's pricing page in November 2026, Grow costs $79 per month billed yearly or $105 billed monthly. Experiments carry no separate fee.

What Rollouts can and can't test

A rollout can hold one theme change (edit or replace your main theme), one checkout and accounts configuration, one or more catalog changes, and one discount change.

It can:

• Test anything you change in the theme editor inside the rollout: sections, blocks and settings on any template, including a product template. Areas you don't touch stay in sync with your published theme.
• Test a whole replacement theme against your current one.
• Test checkout and customer account configurations, discounts and catalogs.

It can't (from Shopify's requirements and setup pages):

• Change Liquid templates. In Shopify's words, "You can't change Liquid templates as part of a rollout."
• Run on vintage themes.
• Target "a customer segment, a customer tag, or a device." Visitors are grouped at random.
• Test headless or custom storefronts.
• Change which template a product uses. Template assignment isn't one of the change types.

How traffic is split

Shopify's analytics page explains assignment in detail:

• Each visitor is put in a group "before the page loads, so the page doesn't flicker between versions."
• The group lives in an essential cookie. Returning visitors stay in their group until it expires or is cleared. A new browser or device gets a new group.
• Visitors get a group even without tracking consent, but only those who consent are counted in results.
• Your theme and apps can't read a visitor's group, so you can't pass it to GA4 or another analytics tool.

What it reports

For a theme Experiment, the help center lists four metrics: conversion rate over time, bounce rate, reached checkout rate and add to cart rate. Gross sales, average order value and orders appear for Launches, not Experiments. You can't customize the metrics or filter results by device or country, and results update a few minutes after each visit with bot traffic excluded.

Shopify doesn't say which statistical method Experiments use. None of the six Rollouts help pages mention significance, confidence or probability. The only note is that Launches and Experiments use "a slightly different algorithm," so the same change can show different numbers in each.

Is Rollouts good enough, or do you still need an app?

Rollouts is enough when:

• You're on Grow or higher and want to test a theme editor change, a redesigned template or a new theme against your current one.
• Conversion rate and add to cart rate are the decision metrics.
• You want zero extra scripts and no flicker.
• You want to test checkout configurations, discounts or catalogs, which most theme-testing apps don't cover.

Use an app when:

• You need results by device, traffic source or new versus returning visitors.
• You want to show the test only to some visitors, such as mobile shoppers or paid social traffic.
• Revenue per visitor or average order value is your decision metric.
• You need code-level variants without maintaining a second theme, price tests, or a documented statistics engine you can explain to your team.

One caution when reading critiques of Rollouts: Shoplift, a competitor, published a March 2026 post arguing that Rollouts lacked concurrent testing and whole-theme tests. Shopify added both in June 2026. The points about segmentation and reporting still match Shopify's own documentation.

How to A/B test a whole Shopify product page template

There are three working approaches. All of them start the same way: build the variant as a template in the theme editor. Our page template guide covers creating, assigning and previewing templates.

Option 1: an Experiment in Shopify Rollouts

1. Go to Markets > Rollouts and click Create rollout. Choose Experiment.
2. Click Add changes, then save the rollout (Shopify requires a save before you can edit your main theme).
3. Choose Edit theme, open the product template you want to test, and rework it: reorder sections, add reviews or comparison blocks, change the buy box. Click Save.
4. Set the traffic split and dates, review any conflicts, and start.

Every product assigned to that template shows the treatment to visitors in the treatment group, on the same URL. If your change needs code, duplicate your theme, make the change there, and run an Experiment that replaces the main theme instead. Don't publish a different theme mid-test: Shopify warns that doing so removes the theme edits in unended rollouts.

Option 2: a template test app

Shoplift, Intelligems and Visually all run template tests: you pick the live template as the control and a duplicate you've edited as the variant.

• Shoplift: a template test shows the variant "on every product, collection, or page assigned to that template," and any template in the live theme can be tested.
• Intelligems: template tests support 2 to 5 groups. The control must be the template currently live, and all templates must be in the live theme.
• Visually: template tests use Shopify's alternate template system to route visitors to the variant. They don't work on the homepage, because Shopify doesn't support alternate homepage templates.

The advantages over Rollouts are targeting (test only on mobile, or only for paid traffic), revenue metrics and per-segment reports. For a closer comparison of the main options, see Shogun vs Intelligems vs Shoplift.

Option 3: a split URL test with ?view=

Shopify renders an alternate template when you add ?view= and the template's suffix to a URL:

/products/linen-shirt              → product.json (original)
/products/linen-shirt?view=test    → product.test.json (variant)

A split URL tool such as Convert then sends half of visitors to the ?view= version. Convert's setup guide uses a regex rule so every product URL maps to its ?view= twin, and excludes URLs that already contain view= to "prevent redirect loops." Because this is a redirect, follow Google's testing guidance: put rel="canonical" pointing to the original on the variant, use a 302 rather than a 301 redirect, and remove the test as soon as it ends.

Sample size and test duration

Most Shopify tests end too early. Evan Miller, whose sample size calculator is a standard reference, gives a rule of thumb: n = 16 × σ² / δ², where σ² = p × (1 − p) for a conversion rate and δ is the smallest absolute change you want to detect. That corresponds to the conventional 5% significance and 80% power.

For a product page converting at 2.5% (see our conversion rate benchmarks for typical ranges):

Divide the total by the daily sessions that reach the template to get your duration, then round up to whole weeks. Intelligems recommends at least 300 orders per test group and at least 7 days. If the numbers come out at several months, test a bigger change, because small tweaks need traffic most stores don't have.

Pitfalls that ruin Shopify A/B tests

• Peeking. Checking results and stopping the moment they look significant inflates false positives. In Evan Miller's worked example, a change that does nothing shows as significant 26.1% of the time if you check after every visitor and stop at the first significant reading. His advice: "Decide on a sample size in advance and wait until the experiment is over." Bayesian dashboards don't remove the temptation; Shoplift, for instance, requires its probability to stay above a threshold for multiple days before it calls a result.
• Testing during sales. Black Friday shoppers, discount hunters and gift buyers behave differently from your usual traffic, and a winner found in late November may lose in January. Use a Rollouts Event to schedule the sale and start new tests after it ends.
• Novelty effect. Returning visitors may click on something just because it's new, and the effect fades. Microsoft researchers describe novelty as interest "that tends to diminish over time." Run full weeks and check whether the lift shrinks as the test goes on.
• Flicker. Client-side tools can show the original page for a moment before swapping in the variant. Chrome's developer guidance notes that anti-flicker snippets, which hide the page until the script runs, delay First Contentful Paint and Largest Contentful Paint. Rollouts assigns groups before the page loads. For any app, load the variant on a slow mobile connection before you launch. Our site speed and SEO guide covers this in more depth.
• Editing mid-test. Shopify warns that editing the treatment or control while an Experiment is active can affect results. Freeze both.
• Cookie and consent gaps. In Rollouts, visitors who switch devices or clear cookies count as new, and only visitors who consent to tracking are counted.

A/B testing vs segment personalization

An A/B test asks one question: which version works best for everyone? You split the same audience at random, wait for enough data, ship the winner to all visitors, and end the test. Rollouts and every app above are built for that.

That average can hide opposite results. A product page that adds proof and context before the price might lift first-time visitors from paid social and slow down returning customers who already know the product. Averaged together, the test can read "no change," and you lose a win for one group. For more on this, see our guides to product pages for cold traffic and different product pages for Meta ads.

Segment personalization asks a different question: what should each segment see? You keep several versions live at once, each built for a segment, and measure each one against the original page for that segment, not against the other versions. Intelligems and Shoplift both offer personalization alongside testing, and our personalization app roundup compares more options.

Ariven is a personalization app, not an A/B testing tool. You build an alternate version of a page in the Shopify theme editor, then create a flow that says which segment sees it, such as first visit, returning customer, or a specific ad or campaign. Ariven routes each visitor in that segment to your version on the same URL, and everyone else sees the original. Each flow is measured against the original page, so you know whether the segment's version is earning its place. If your question is "which single product page should everyone see," use Rollouts or a testing app. If it's "should first-time visitors from Meta see a different page than returning customers," that's a flow.

Sources

• Shopify Help Center: Rollouts overview, requirements and considerations, types of rollouts and changes, creating a rollout, managing rollouts, rollout analytics, reach and external channels
• Shopify Changelog: storefront changes with Rollouts (March 31, 2026), themes and checkout configurations (June 5, 2026), redesigned controls (September 22, 2026), discounts (October 2, 2026), catalogs (October 23, 2026)
• Shopify: Winter '26 Edition, pricing (checked November 2026), alternate templates
• Shoplift: pricing, App Store listing, template testing, test outcomes, Rollouts critique (March 23, 2026)
• Intelligems: pricing, template tests, statistical significance
• Visually: App Store listing, template testing, odds to win
• Convert: split testing Shopify templates, statistical methods, pricing, App Store listing
• Evan Miller: How Not to Run an A/B Test, sample size calculator
• Google Search Central: minimize A/B testing impact in Google Search
• Chrome for Developers: flicker-free client-side A/B testing
• Sadeghi et al., Microsoft, Novelty and Primacy: A Long-Term Estimator for Online Experiments (2021)