Manage Your Experiments Tool
Amazon has introduced a new tool to help vendors and sellers optimise their content. The Manage Your Experiments (MYE) Tool allows you to create two different product detail pages for the same product, testing exactly what shoppers are drawn towards and using the insights to optimise future content and drive more sales.
This tool runs A/B Tests, also known as Split Tests, allowing you to use shopper behaviour, rather than guessing, to help you create content that leads to sales. Currently, this tool, which has been available in the US for almost two years and recently became available in Europe, gives vendors and sellers the ability to change product titles and product images and A+ content. You can simply submit a new version of your A+ content, perhaps with a comparison chart or using only lifestyle images, and see which one performs better.
What is the Manage Your Experiments Tool?
The MYE tool allows vendors and sellers on Amazon to change certain aspects of their content and run two versions side-by-side. The shoppers are randomly divided into two groups and assigned to see either version A or B of the content. Amazon then observes the performance of each version over the duration of the experiment (usually about 8 to 10 weeks) and offers weekly results as well as the better-performing version at the end of the experiment.
The content available to be edited within this tool includes the product images and product titles on the product detail pages as well as the A+ content. Product titles and images help shoppers decide what product pages they will visit. Testing these aspects can therefore help drive traffic to your listings.
The additional imagery, product and brand descriptions of A+ content help to tell the brand story. A+ content also allows enhanced images and text placements. It is becoming the new standard on Amazon and most every successful product detail page includes A+ content. Therefore, it will be extremely valuable to understand what kinds of A+ content shoppers respond best to and can ultimately help increase conversion.
How to Run an Experiment
Step 1: Select an eligible ASIN (aka a Reference ASIN) to perform the experiment on.
Variational ASINs are possible but some child ASINs may not be eligible and therefore will not even show up as a possibility for an experiment.
Step 2: Add experiment details including name, hypothesis, duration and start dates. Name – Used to help you keep experiment results organized. Hypothesis – asks the question that you want the experiment to answer. For example, if I make my product title more specific, sales will increase. Hypotheses can help you gain learnings that you can then apply to future product content. Duration and start dates – Amazon recommends running an experiment between 8-10 weeks long. However, you can always change the duration and end the experiment early. Keep in mind, the longer an experiment runs, the more accurate the results.
Step 3: Select the content for your experiment.
Enter the proposed title and/or images that you would like to test into the associated box. Keep in mind that you can use different titles and images for variational child ASINs, but you will also need to submit content for the parent ASIN. Create new content for version B by clicking the link, “Start by duplicating Version A.” From here, you can make the content changes you would like to test.
Step 4: Submit your experiment. Once it is reviewed and validated by Amazon, it will begin.
Tip: Be sure to continuously check your MYE to ensure it was validated. If it failed, review the content to make sure it is compliant with Amazon regulations. Check out some possible reasons for it failing and how to avoid that in the following sections.
Content Validation & Editing Your Experiments
Just as all Amazon content is subject to review by Amazon, the MYE content is no different. Every bit of content, whether a test or not, must abide by Amazon guidelines. Keep in mind that titles must have no more than 200 characters (sometimes less, depending on the category) and the image quality must be high, with clear images that are easy to understand, informative and aesthetically pleasing.
As far as editing your experiments, different extents of editing are possible at different phases of the experiment process. The following stages permit the corresponding edits:
- Scheduled and waiting for validation: Can edit any detail
- Failed content validation: Revise the content and make edits to it abides by Amazon regulations
- Scheduled and successfully validated: Can edit content, but must select a new start date because the validation process must be repeated
- An experiment in progress: Content cannot be edited but the hypothesis and duration can be changed. In order to edit the content, you must either cancel the experiment (terminate before the end date) and create a new one
Cancelling an Experiment
If you decide to cancel an experiment, i.e. to end it before the scheduled end date, you must go to experimental details and select Cancel Experiment and indicate why you chose to cancel (for example, content error or desired result realised before the end date). Once an experiment is terminated, all shoppers will be shown only the original content.
On the other hand, a completed experiment is one that was carried out through its original end date. At the end of a completed experiment, all shoppers will once again see only the original content regardless of the winner until you decide which content to publish.
Results and How to Interpret Them
The experiment results are updated once a week during the entire duration of the experiment. You can access these results via the dashboard by clicking on the name of the experiment that you would like to review. These weekly results are available in different forms.
Firstly, you can see how each version is performing and provide a probability that one version is better than the other. For example, if there is a 75% probability that version A is better, this means that one version shows that 75% of possible impacts is a likely positive units/sales lift if you choose to publish version A.
For each version of the content posted, you also have the option to see different KPIs including units, sales, conversion rate, sample size and units sold per unique visitor.
Conversion rate is the ratio between the number of unique visitors who saw the experimental content (either version A or version B) and those who subsequently made a purchase.
The sample size is the number of unique visitors who saw each version of the content.
Units per unique visitor = Units/sample size
Finally, for completed experiments, you can see the projected one-year impact of your content. This is calculated by Amazon as an estimate of possible incremental units and sales throughout the year following the publication of the winning content.
Note: These projections (calculated by multiplying the average daily sales increase by 365) are only based on the experiment results and are not a guarantee of actual results. Amazon does not take seasonality, price changes or other affecting factors into account when performing these calculations.
Inconclusive results are also a possibility, however, these results are still valuable when you evaluate the reason for the inconclusiveness. The reasons could be:
- The change made was too small to lead to a noticeable customer behaviour change
- Perhaps your product didn’t have enough traffic to determine a winning version with high confidence
- The two versions were similarly effective in driving sales
- The change you made to your content is not the influential factor that influences customer decisions.
If this is the case, you may need to reevaluate your hypothesis to determine what aspect is ineffective in influencing customer decisions.
Experiment Methodology & Tips
It is one thing to run these experiments to determine what version of your content performs best, however, understanding WHY that version succeeded is extremely valuable. Running multiple experiments will help you confirm what you believe to be true and will help you determine which factors help Amazon determine a winning content version.
The same customer will continue to see one version of your content throughout the entirety of the experiment, regardless of the device they are using. Amazon automatically removes unidentified customers and statistical outliers to ensure maximum accuracy.
To make sure your experiment performs as well as possible and produces the best possible results, consider the following tips:
- Make sure your experimental content is very different from the original so you can determine definite causes for certain impacts
- Let your experiment run for the entire duration. It may be tempting to end it early due to promising-looking results, but this could lead you to select the wrong version
- Valuable points to test include: reducing the product title length or incorporating your brand name, product attributes, etc. into your title. You could also test more creative product imagery, i.e. lifestyle images only or standard images only
- When managing your results, do not check them too often because they may be misleading before the experiment ends. However, continue to test on different ASINs, change your hypothesis for the same ASIN and gain a well-rounded understanding of how customers interact with your product content
Insights
View all insights
Questions?
Partner