Run an A/B experiment

Split new players between 2 to 4 variants, call variant() in your plugin, and read the verdict. Operator and Network.

Using the dashboardUpdated 2 min read

Experiments are under Growth, Experiments (Operator: 2 running at once; Network: unlimited).

Set up

  1. Create an experiment on a server: a key that your plugin will pass, a name, 2 to 4 variants with weights (the first is the control), who is included (new players only or everyone) and the goal metric: day-1 retention, day-7 retention, first purchase, revenue per player, playtime, or a custom event.
  2. Start it. The signed server config now lists it, and agents pick it up.
  3. In your plugin ask which variant the player gets, for example in Java ServerStats.get().variant(player, "welcome_kit", "control"). The first call per player records an exposure. The assignment is computed in the agent from a hash, so a player always gets the same variant and their id never leaves the server.

Read the result

A player counts in the variant of their first exposure, and only once their window has passed: 2 days for day-1 retention, 8 days for day-7, 7 days for the others. ServerStats uses a two-proportion test for rates and a Welch t-test for averages, adjusts for multiple variants, and makes no claim until each variant has 100 measured players. It also warns when the split is lopsided (sample ratio mismatch).

Ship the winner

Ship stops the experiment and serves the winning variant to everyone, with no more exposure events. It keeps working after a downgrade and does not count against your running limit.

Which call to use per game, and the assignment algorithm: developer SDK reference.

Related

Using the dashboardGrowthWhere players come from, store revenue, campaigns, A/B experiments and countries.Using the dashboardSend your own events and chart themCall track() from your plugin, then Make chart under Build. Names are snake_case; properties are flat numbers, booleans and short strings.
Still stuck? Email [email protected] with the server name and what you see. For a rejected claim code or no data, include the output of the agent’s status command.