Seller Handbook

Advice and inspiration for successfully running your Etsy shop

Building Better Tools With Experiments

Testing new features and tools helps us make Etsy the best it can be. Learn more about our approach.

By Taylor Combs Apr 18, 2017
Photo by Bourbon Moth

Etsy’s mission is to reimagine commerce in ways that build a more fulfilling and lasting world, and every day we work to build tools to help creative entrepreneurs do just that. Everything we do centers on supporting you and your business because when you succeed, we succeed. Experiments are a tool we use to ensure that when we make updates on Etsy, they’ll have a positive impact on sales and traffic or make Etsy’s tools easier to use.

What do we mean we say experiments? Experiments are one way we gather data about how Etsy buyers and sellers interact with a new feature and measure how changes to that page or tool affect an end result. By testing new features with a small percentage of Etsy users, we gather valuable data that helps us ensure that the changes we make will have a positive impact for the majority of Etsy sellers. In most cases, we use experiments to confirm a hypothesis that we formed with the help of research and data, not to test a completely new idea. “Experiment ideas come from a combination of your feedback, our data analysis, and Etsy’s product teams,” says David Bendet, analytics manager at Etsy. “We always have a clear hypothesis for what the outcome of a change will be.”

When we make updates, our first priority is building a better Etsy for everyone. We use experiments to test almost every change we make to the buyer experience on Etsy to ensure that they increase sales, boost traffic, or make a shopper’s experience on Etsy better. When we do experiments on the tools you use to run your business, it’s because we want to be sure that the changes we’re making will make it easier for you to manage your shop and find the tools you need.

We know that you rely on Etsy’s tools to run your business, so having to change your workflow because you’re part of an experiment can feel like an unnecessary addition to your already packed to-do list. But every time you encounter an experiment on Etsy, you’re helping to build better tools for the entire community.

An experiment case study

Let’s look at an example of how we used an experiment to test our hypothesis about improvements to the listing process.

Our hypothesis: During research studies, we discovered that many sellers like to upload photos to their listing before adding a description or other elements, so they could look to them for reference while completing the rest of their listing. Knowing this, it didn’t feel right to us that uploading photos came later in the listing creation process. Our hypothesis: making photos the first step of creating a listing would save sellers time.

During the experiment, 25 percent of sellers saw this version of the listing creation page with photos as the first step.

How we confirmed our hypothesis: Before making a change to the listing process that would affect all sellers, we wanted to confirm that the change would be positive. So, we designed an experiment during which 25 percent of sellers would see the photo step first at the top of the listing creation page, 25 percent would see the photo step at the bottom of the page, and 50 percent would see the photo step in the middle of the page, where it had always been. This option, called the control, would help us prove that moving the photo step to the top of the page is what actually caused the positive result.

The results: After testing the three versions of the listing process for a few days, we found that our hypothesis was correct. Sellers who saw the photo option first completed new listings more quickly and created more listings than the sellers in the other groups.

By running a short experiment, we determined that the change would have a positive impact on a majority of sellers. “We take the fact that sellers run their businesses on our platform very seriously,” says Nickey Skarstad, Etsy’s director of product development. “We only do experiments when we think they’re going to have a positive impact.”

How we talk about experiments

Logging into your Etsy shop and finding something out of place can be disconcerting, but experiments are an integral part of building great tools. They allow us to quickly confirm our hypotheses, so we can make impactful changes on a regular basis. We know it can be frustrating to be surprised by a change in your shop when you’re part of a test, but announcing experiments before they happen can bias the results. If you knew we were testing a new tool in Shop Stats, for example, you might use the tool differently, impacting the results of the experiment.

We can’t pre-announce all of the tests we might run while building new tools for Etsy shop owners, but this year we are focusing on making sure you have the information you need, when you need it. We might run experiments around where information is displayed and where different tools are located in your Shop Manager. We're also interested in testing new tools to help you improve your listings. We regularly test changes to the buyer experience on Etsy to help make it easier for buyers to complete their purchases, so you can make more sales. Something as seemingly small as the color or location of a button or the language used in a menu can have a measurable impact on the number of completed purchases.

Other ways we gather feedback

When we test tools that might mean big changes to your workflow, like the recent launch of Shop Manager, we use prototypes to gather feedback and test our hypothesis so that we don’t risk asking you to change your processes for a tool that might still need a little work. Prototypes are one way you can choose to test a new Etsy tool to help make it even better before it’s launched to all sellers.

Unlike a prototype, you can’t opt-out of an experiment. Whether you have one sale or thousands, you could be part of an experiment. By including all kinds of shops, we can feel confident that the results of the test will have a positive impact on everyone.

We also use other kinds of research methods like interviewing sellers about their processes or watching a small group of sellers interact with a new tool to gather feedback to help us make informed decisions.

Author

Taylor Combs

Taylor Combs is a writer and editor for Etsy's Seller Handbook.

Comments

Continue reading in Site updates