A/B testing campaign scripts

Adding a second script

1. 1

   Open the campaign

   Go to the Scripts and A/B tests tab.

2. 2

   Add script B

   Click Test a second script, then Add script B. The new script starts as a copy of script A, so you only need to change what you want to test, for example a different opening line.

3. 3

   Set the split

   Each script has a New players (%) field. New players are split evenly by default, but you can change the percentages. Use Add another script to go up to five scripts, A to E.

Once a player has been assigned a script, they keep it. That includes callbacks, so a player who calls back hears the same pitch again.

Script history

Every time you save a script, Milo stores a new version. Open Script history to see each version, view it, and Restore an older one. Restoring saves the old text as the newest version, so nothing is lost.

How each script performs

The How each script performs table compares every script against script A:

• new players, called, answered and accepted
• logged in, deposited and deposit rate
• revenue and cost per depositor

Each script gets one of these verdicts:

• Baseline: script A, which the others are compared against
• Not enough data yet: a script needs 100 players called before it's compared
• Better or Worse: shown only when Milo is at least 95% sure the difference in deposits is real
• No clear difference yet

When you have a winner, click Use only this. Every new player gets that script, and players already queued are switched over to it.

Customer.io

If you've connected Customer.io, Milo tells it which script each player heard:

• milo_last_script on the person, holding the letter, for example B
• script_variant and script_version on the milo_call_completed event
• script_variant on outcome events such as milo_offer_accepted, milo_offer_declined and milo_callback_requested

You can then split your follow-up emails or texts by script, or report on them in Customer.io.

Duplicating and archiving

To start a new campaign from scripts that work, use Copy on the campaign list. It copies the settings and scripts without any players, and the new campaign starts paused. When a campaign is done, Archive it to keep your list tidy. Its results stay available under the Archived tab.