Configuring A/B Testing
A/B testing helps marketers test treatments (for example, different marketing Calls to Action in an email) to determine which treatments are performing better. With Treasure Data's A/B Testing, you can compare one to eight versions of something to determine which performs better. To accomplish this test, at a minimum, you need one test case and a control group to validate the results. A control group is important because it provides a benchmark to measure performance against. A control group highlights what works as much as what doesn't.
INFO
The steps to configure A/B Testing apply only to draft journeys.
This topic contains:
- Prerequisites
- Limitations
- Verify the Status of a Journey
- Create A/B Tests in a Journey
- Modify an A/B Test
- Delete an A/B Test
- A/B Testing Statistics
Prerequisites
- A parent segment.
- A new draft journey or an existing draft journey.
- Basic understanding of Treasure Data.
Limitations
- Each stage of a journey can only have two layers of an A/B test added.
Verify the Status of a Journey
You can verify the current status of a draft journey to which you want to add or update A/B tests. The following example does not include an A/B test because we are going to add one in this tutorial.
In the case the journey included an existing A/B test, the request retrieves the name, variant, and allocation information for the specified journey.
This example uses the following values
| Method | GET |
| Endpoint | [ https://{BaseUrl}/entities/journeys/:journeyId ] |
| Header | Authorization: TD1 1/456789xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |
| Header | Content-Type: application/json |
Sample RequestSample Response
curl --location --request GET 'https:{{baseUrl}}/entities/journeys/:journeyId' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: TD1 123/456789xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \{
"data": {
"id": "117385",
"type": "journey",
"attributes": {
"audienceId": "57350",
"name": "Purchase journey",
"description": "",
"state": "draft",
"goal": null,
"createdAt": "2023-09-11T20:47:29.086Z",
"updatedAt": "2023-09-11T20:47:29.086Z",
"launchedAt": null,
"allowReentry": false,
"paused": false,
"pausedAt": null,
"journeyStages": [
{
"id": "149348",
"name": "Awareness",
"description": null,
"steps": {
"4e7dcb3e-4f3a-4a6c-be3f-a37b09acbac7": {
"type": "Activation",
"next": "2903c0a0-e43c-4ed6-8e0c-1b682be1cc03",
"description": null,
"name": "act_1",
"journeyActivationStepId": "153668"
},
"2903c0a0-e43c-4ed6-8e0c-1b682be1cc03": {
"type": "WaitStep",
"next": null,
"waitStep": 1,
"waitStepUnit": "day"
}
},
"rootStep": "4e7dcb3e-4f3a-4a6c-be3f-a37b09acbac7",
"entryCriteria": {
"name": "browse",
"segmentId": "912056",
"description": ""
},
"milestone": null,
"exitCriterias": null
}
]
},
"relationships": {
"parentFolder": {
"data": {
"id": "183985",
"type": "folder-segment"
}
},
"jumpFrom": {
"data": []
},
"jumpTo": {
"data": []
}
}
}
}About this request:
- Replace {baseUrl} with the appropriate endpoint for your region.
- Replace {journeyId} with the draft journey to which you want to add or update A/B tests.
- Replace {TD1 123/456789xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} with your personal API key.
About this response object:
- The name of the A/B test and its variants are contained within the journey stage.
Create A/B Tests in a Journey
You add an A/B test to any decision point; there is no limit to the number of A/B tests within a journey. The following example sets up a test name, number of variants, percentage allocation to each variable, and a stable ID key.
Sample RequestSample Response
curl --location --request POST 'https:{{baseUrl}}/entities/journeys/:journeyId' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: TD1 123/456789xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \
data raw '{
"type": "ABTest",
"name": <A/B Test Name>,
"customizedSplit": true or false
"uniqueId": <Attribute Name>,
"variants": [
{
"id": <UUID>,
"name": <Variant Name>,
"percentage": 10,
"next": <Next Step UUID>
}
]
}{
"data": {
"id": "117385",
"type": "journey",
"attributes": {
"audienceId": "57350",
"name": "Purchase journey",
"description": "",
"state": "draft",
"goal": null,
"createdAt": "2023-09-11T20:47:29.086Z",
"updatedAt": "2023-09-11T20:47:29.086Z",
"launchedAt": null,
"allowReentry": false,
"paused": false,
"pausedAt": null,
"journeyStages": [
{
"id": "149348",
"name": "Awareness",
"description": null,
"steps": {
"4e7dcb3e-4f3a-4a6c-be3f-a37b09acbac7": {
"type": "Activation",
"next": "2903c0a0-e43c-4ed6-8e0c-1b682be1cc03",
"description": null,
"name": "act_1",
"journeyActivationStepId": "153668"
},
"2903c0a0-e43c-4ed6-8e0c-1b682be1cc03": {
"type": "WaitStep",
"next": "e48c0c6f-7de2-4d94-9b22-c6c3eb51105e",
"waitStep": 1,
"waitStepUnit": "day"
},
"e48c0c6f-7de2-4d94-9b22-c6c3eb51105e": {
"type": "ABTest",
"variants": [
{
"id": "ee9aa667-60c7-42e0-b8b1-5cee9a2edfb3",
"name": "Variant A",
"percentage": 45,
"next": null
},
{
"id": "d4179b47-8d9a-495c-bddd-5dcf25c295c6",
"name": "Variant B",
"percentage": 5,
"next": null
},
{
"id": "bbf9c797-efb0-41ae-83b0-53082d8337a8",
"name": "Control",
"percentage": 50,
"next": null
}
],
"name": "test",
"customizedSplit": true,
"uniqueId": "cdp_customer_id"
}
},
"rootStep": "4e7dcb3e-4f3a-4a6c-be3f-a37b09acbac7",
"entryCriteria": {
"name": "browse",
"segmentId": "912056",
"description": ""
},
"milestone": null,
"exitCriterias": null
}
]
},
"relationships": {
"parentFolder": {
"data": {
"id": "183985",
"type": "folder-segment"
}
},
"createdBy": {
"data": {
"id": "1020",
"type": "user"
}
},
"updatedBy": {
"data": {
"id": "1020",
"type": "user"
}
},
"activations": {
"data": []
},
"journeyActivationSteps": {
"data": [
{
"id": "153668",
"type": "journeyActivationStep"
}
]
},
"jumpFrom": {
"data": []
},
"jumpTo": {
"data": []
}
}
}
}About this request:
- Replace {baseUrl} with the appropriate endpoint for your region.
- Replace {journeyId} with the specified journey ID.
- Replace {TD1 123/456789xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} with your personal API key.
About this response object:
- The name of the A/B test and its variants are adding to the journey stage.
Modify an A/B Test
You can make changes to an A/B test while it is in draft mode. For instance, you can rename the test or its variants, delete variants or the A/B test, change the Unique ID field, or the allocation percentage.
Sample RequestSample Response
curl --location --request PATCH 'https:{{baseUrl}}/entities/journeys/:journeyId' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: TD1 123/456789xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \
data raw '{
"type": "ABTest",
"name": <A/B Test Name>,
"customizedSplit": true or false
"uniqueId": <Attribute Name>,
"variants": [
{
"id": <UUID>,
"name": <Variant Name>,
"percentage": 10,
"next": <Next Step UUID>
}
]
} "data": {
"id": "117385",
"type": "journey",
"attributes": {
"audienceId": "57350",
"name": "Purchase journey",
"description": "",
"state": "draft",
"goal": null,
"createdAt": "2023-09-11T20:47:29.086Z",
"updatedAt": "2023-09-11T20:47:29.086Z",
"launchedAt": null,
"allowReentry": false,
"paused": false,
"pausedAt": null,
"journeyStages": [
{
"id": "149348",
"name": "Awareness",
"description": null,
"steps": {
"4e7dcb3e-4f3a-4a6c-be3f-a37b09acbac7": {
"type": "Activation",
"next": "2903c0a0-e43c-4ed6-8e0c-1b682be1cc03",
"description": null,
"name": "act_1",
"journeyActivationStepId": "153668"
},
"2903c0a0-e43c-4ed6-8e0c-1b682be1cc03": {
"type": "WaitStep",
"next": "e48c0c6f-7de2-4d94-9b22-c6c3eb51105e",
"waitStep": 1,
"waitStepUnit": "day"
},
"e48c0c6f-7de2-4d94-9b22-c6c3eb51105e": {
"type": "ABTest",
"variants": [
{
"id": "51527fb6-72c4-4e85-a4bc-f3b7094100a2",
"name": "Semi monthly",
"percentage": 17,
"next": null
},
{
"id": "d4179b47-8d9a-495c-bddd-5dcf25c295c6",
"name": "Semi weekly",
"percentage": 17,
"next": null
},
{
"id": "ee9aa667-60c7-42e0-b8b1-5cee9a2edfb3",
"name": "Weekly Once",
"percentage": 17,
"next": null
},
{
"id": "bbf9c797-efb0-41ae-83b0-53082d8337a8",
"name": "Control",
"percentage": 49,
"next": null
}
],
"name": "Sending promotion emails",
"customizedSplit": false,
"uniqueId": "cdp_customer_id"
}
},
"rootStep": "4e7dcb3e-4f3a-4a6c-be3f-a37b09acbac7",
"entryCriteria": {
"name": "browse",
"segmentId": "912056",
"description": ""
},
"milestone": null,
"exitCriterias": null
}
]
},
"relationships": {
"parentFolder": {
"data": {
"id": "183985",
"type": "folder-segment"
}
},
"createdBy": {
"data": {
"id": "1020",
"type": "user"
}
},
"updatedBy": {
"data": {
"id": "1020",
"type": "user"
}
},
"activations": {
"data": []
},
"journeyActivationSteps": {
"data": [
{
"id": "153668",
"type": "journeyActivationStep"
}
]
},
"jumpFrom": {
"data": []
},
"jumpTo": {
"data": []
}
}
}
}About this request:
- Replace {baseUrl} with the appropriate endpoint for your region.
- Replace {journeyId} with journey you want to add or update A/B tests.
- Replace {TD1 123/456789xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} with your personal API key.
About this response object:
- The name of the A/B test and its variants have been modified.
- The number of variants is increased in the response from 2 to 3.
-
The customized splits is changed to
false. - The percentage allocated to each variant has also been modified.
Delete an A/B Test
You can delete an A/B test while it is in draft mode.
The section contains:
Verify the A/B Test in the Journey Stage
Delete an A/B test Within a Journey Stage
Confirming Journey Status
Verify the A/B Test in the Journey Stage
To confirm which A/B test to delete, you first need to confirm an A/B Test applied to a stage. In the example that follows, you can view the journey stage and A/B tests for journey 117385.
Sample Request
curl --location --request GET 'https:{{baseUrl}}/entities/journeys/:journeyId' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: TD1 123/456789xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \
data raw '{
"data": {
"id": "117385",
"type": "journey",
"attributes": {
"audienceId": "57350",
"name": "Purchase journey",
"description": "",
"state": "draft",
"goal": null,
"createdAt": "2023-09-11T20:47:29.086Z",
"updatedAt": "2023-09-11T20:47:29.086Z",
"launchedAt": null,
"allowReentry": false,
"paused": false,
"pausedAt": null,
"journeyStages": [
{
"id": "149348",
"name": "Awareness",
"description": null,
"steps": {
"4e7dcb3e-4f3a-4a6c-be3f-a37b09acbac7": {
"type": "Activation",
"next": "2903c0a0-e43c-4ed6-8e0c-1b682be1cc03",
"description": null,
"name": "act_1",
"journeyActivationStepId": "153668"
},
"2903c0a0-e43c-4ed6-8e0c-1b682be1cc03": {
"type": "WaitStep",
"next": "e48c0c6f-7de2-4d94-9b22-c6c3eb51105e",
"waitStep": 1,
"waitStepUnit": "day"
},
"e48c0c6f-7de2-4d94-9b22-c6c3eb51105e": {
"type": "ABTest",
"variants": [
{
"id": "51527fb6-72c4-4e85-a4bc-f3b7094100a2",
"name": "Semi monthly",
"percentage": 17,
"next": null
},
{
"id": "d4179b47-8d9a-495c-bddd-5dcf25c295c6",
"name": "Semi weekly",
"percentage": 17,
"next": null
},
{
"id": "ee9aa667-60c7-42e0-b8b1-5cee9a2edfb3",
"name": "Weekly Once",
"percentage": 17,
"next": null
},
{
"id": "bbf9c797-efb0-41ae-83b0-53082d8337a8",
"name": "Control",
"percentage": 49,
"next": null
}
],
"name": "Sending promotion emails",
"customizedSplit": false,
"uniqueId": "cdp_customer_id"
}
},
"rootStep": "4e7dcb3e-4f3a-4a6c-be3f-a37b09acbac7",
"entryCriteria": {
"name": "browse",
"segmentId": "912056",
"description": ""
},
"milestone": null,
"exitCriterias": null
}
]
},
"relationships": {
"parentFolder": {
"data": {
"id": "183985",
"type": "folder-segment"
}
},
"createdBy": {
"data": {
"id": "1020",
"type": "user"
}
},
"updatedBy": {
"data": {
"id": "1020",
"type": "user"
}
},
"activations": {
"data": []
},
"journeyActivationSteps": {
"data": [
{
"id": "153668",
"type": "journeyActivationStep"
}
]
},
"jumpFrom": {
"data": []
},
"jumpTo": {
"data": []
}
}
}
}About this request: An A/B test can be deleted from a journey stage by omitting the test in the same PATCH request. This request confirms the specified journey has an A/B Test in Stage 1.
- Replace {baseUrl} with the appropriate endpoint for your region.
- Replace {journeyId} with journey you want to add or update A/B tests.
- Replace {TD1 123/456789xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} with your personal API key.
Delete an A/B Test Within a Journey Stage
To delete the A/B test, the information about the A/B test is omitted from the PATCH. In the example that follows, the A/B test and its associated variants are absent from the payload for journey 117385.
curl --location --request PATCH 'https:{{baseUrl}}/entities/journeys/:journeyId' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: TD1 123/456789xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \
data raw '{
"id": "117385",
"type": "journey",
"attributes": {
"audienceId": "57350",
"name": "Purchase journey",
"description": "",
"state": "draft",
"goal": null,
"createdAt": "2023-09-11T20:47:29.086Z",
"updatedAt": "2023-09-11T20:47:29.086Z",
"launchedAt": null,
"allowReentry": false,
"paused": false,
"pausedAt": null,
"journeyStages": [
{
"id": "149348",
"name": "Awareness",
"description": null,
"steps": {
"4e7dcb3e-4f3a-4a6c-be3f-a37b09acbac7": {
"type": "Activation",
"next": "2903c0a0-e43c-4ed6-8e0c-1b682be1cc03",
"description": null,
"name": "act_1",
"journeyActivationStepId": "153668"
},
"2903c0a0-e43c-4ed6-8e0c-1b682be1cc03": {
"type": "WaitStep",
"next": null,
"waitStep": 1,
"waitStepUnit": "day"
}
},
"rootStep": "4e7dcb3e-4f3a-4a6c-be3f-a37b09acbac7",
"entryCriteria": {
"name": "browse",
"segmentId": "912056",
"description": ""
},
"milestone": null,
"exitCriterias": null
}
]
},
"relationships": {
"parentFolder": {
"data": {
"id": "183985",
"type": "folder-segment"
}
},
"createdBy": {
"data": {
"id": "1020",
"type": "user"
}
},
"updatedBy": {
"data": {
"id": "1020",
"type": "user"
}
},
"activations": {
"data": []
},
"journeyActivationSteps": {
"data": [
{
"id": "153668",
"type": "journeyActivationStep"
}
]
},
"jumpFrom": {
"data": []
},
"jumpTo": {
"data": []
}
}
}About this request
- The name of the A/B test and its variants have been removed from the PATCH request.
- The "next" step is marked as "null" in the object preceding the A/B test.
- Replace {baseUrl} with the appropriate endpoint for your region.
- Replace {journeyId} with journey from which you want to remove an A/B test.
- Replace {TD1 123/456789xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} with your personal API key.
Sample Response
curl --location --request GET 'https:{{baseUrl}}/entities/journeys/:journeyId' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: TD1 123/456789xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \
data raw '{
"data": {
"id": "117385",
"type": "journey",
"attributes": {
"audienceId": "57350",
"name": "Purchase journey",
"description": "",
"state": "draft",
"goal": null,
"createdAt": "2023-09-11T20:47:29.086Z",
"updatedAt": "2023-09-11T20:47:29.086Z",
"launchedAt": null,
"allowReentry": false,
"paused": false,
"pausedAt": null,
"journeyStages": [
{
"id": "149348",
"name": "Awareness",
"description": null,
"steps": {
"4e7dcb3e-4f3a-4a6c-be3f-a37b09acbac7": {
"type": "Activation",
"next": "2903c0a0-e43c-4ed6-8e0c-1b682be1cc03",
"description": null,
"name": "act_1",
"journeyActivationStepId": "153668"
},
"2903c0a0-e43c-4ed6-8e0c-1b682be1cc03": {
"type": "WaitStep",
"next": null,
"waitStep": 1,
"waitStepUnit": "day"
}
},
"rootStep": "4e7dcb3e-4f3a-4a6c-be3f-a37b09acbac7",
"entryCriteria": {
"name": "browse",
"segmentId": "912056",
"description": ""
},
"milestone": null,
"exitCriterias": null
}
]
},
"relationships": {
"parentFolder": {
"data": {
"id": "183985",
"type": "folder-segment"
}
},
"createdBy": {
"data": {
"id": "1020",
"type": "user"
}
},
"updatedBy": {
"data": {
"id": "1020",
"type": "user"
}
},
"activations": {
"data": []
},
"journeyActivationSteps": {
"data": [
{
"id": "153668",
"type": "journeyActivationStep"
}
]
},
"jumpFrom": {
"data": []
},
"jumpTo": {
"data": []
}
}
}
}About this response:
- The response confirms the A/B test and its variants have been removed from the journey stage.
Confirming Journey Status
If additional confirmation apart from the response is required, you can do a GET to the endpoint entities/journeys/{journey_id}. A similar response as the PATCH request's response is displayed when a GET to the same journey in performed. The journey attributes do not display any A/B test information, as it was successfully deleted during the PATCH request.
curl --location --request GET 'https:{{baseUrl}}/entities/journeys/:journeyId' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: TD1 123/456789xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \
data raw '{
"data": {
"id": "117385",
"type": "journey",
"attributes": {
"audienceId": "57350",
"name": "Purchase journey",
"description": "",
"state": "draft",
"goal": null,
"createdAt": "2023-09-11T20:47:29.086Z",
"updatedAt": "2023-09-11T20:47:29.086Z",
"launchedAt": null,
"allowReentry": false,
"paused": false,
"pausedAt": null,
"journeyStages": [
{
"id": "149348",
"name": "Awareness",
"description": null,
"steps": {
"4e7dcb3e-4f3a-4a6c-be3f-a37b09acbac7": {
"type": "Activation",
"next": "2903c0a0-e43c-4ed6-8e0c-1b682be1cc03",
"description": null,
"name": "act_1",
"journeyActivationStepId": "153668"
},
"2903c0a0-e43c-4ed6-8e0c-1b682be1cc03": {
"type": "WaitStep",
"next": null,
"waitStep": 1,
"waitStepUnit": "day"
}
},
"rootStep": "4e7dcb3e-4f3a-4a6c-be3f-a37b09acbac7",
"entryCriteria": {
"name": "browse",
"segmentId": "912056",
"description": ""
},
"milestone": null,
"exitCriterias": null
}
]
},
"relationships": {
"parentFolder": {
"data": {
"id": "183985",
"type": "folder-segment"
}
},
"createdBy": {
"data": {
"id": "1020",
"type": "user"
}
},
"updatedBy": {
"data": {
"id": "1020",
"type": "user"
}
},
"activations": {
"data": []
},
"journeyActivationSteps": {
"data": [
{
"id": "153668",
"type": "journeyActivationStep"
}
]
},
"jumpFrom": {
"data": []
},
"jumpTo": {
"data": []
}
}
}
}A/B Testing Statistics
You can choose to show the number of profiles in a particular A/B test variant within the same parent segment using the following rule.
{
"type": "And",
"conditions": [
{
"type": "Value",
"exclude": true,
"leftValue": {
"journeyId": 1,
"journeyStageId": 2,
"location": "ab_test_<UUID with underscore>_variant_<Variant UUID>",
"tense": "intime"
},
"operator": {
"type": "IsNull",
"not": true
}
}
]
}