Are you climbing the right mountain?

Editor’s note: This article is from WeChat public account “Redwoods” (ID: Sequoiacap), author Hong Cedar.

As an entrepreneur or company executive, you are no stranger to A/B testing. Every year, tens of thousands of such tests are used by many Internet companies for online optimization iterations. But the A/B test we rely on may not be reliable. Andre Morys, founder and CEO of Web Arts, even asserted: “90% of these test results are fake.”

Sure, the A/B test essentially asks a question: Which is better for A and B? But what if you ask the wrong question? If A and B are not good, the user just picks one that is not so annoying.

How to ensure that you do not blindly use A/B testing? This article presents two key questions that may be instructive.

Ask the right question and do the A/B test

If you are a data-driven person, you probably won’t think of anything and do it right away, but instead tend to do an A/B test and bring the winning version to market. This is really a good way. But if you started, you chose the wrong test direction? In order to avoid wasting money and time on product iterations, before the next test, you may want to answer the two questions raised in this article.

A/B test

A/B test is a random test. Compare the two versions of a single feature on the basis of keeping other factors constant. This feature may be a new button, or a different font, or a change in marketing information, and so on. Customers are randomly divided into two groups, each of which sees a different version. Collect user feedback, compare the two sets of views on different versions, and finally run the version that will reach your goal. Usually version A is the current version (control group) and version B is the version with new features (experimental group).

A successful A/B test requires three elements:

1. A goal to be achieved and a corresponding indicator to measure change.

2. The distribution of version A and version B is random.

3. Control other variables to be invariant and measure only one characteristic.

Now we already know the definition of A/B testing, let’s take a look at the first question.

Not all products are suitable for A/B testing

Question 1: Are you climbing the right mountain?

In other words, is A/B testing the right way to achieve your goals?

The essence of A/B testing is product optimization. Suppose you are climbing and want to try different routes to see which one can climb to the top faster. But what if you chose the wrong mountain at the beginning? What you climb is actually just a small hill. You are trapped in a limited area, wasting a lot of time and can only find a sub-optimal solution.

Not all products are suitable for A/B testing

Marissa Mayer, when serving as vice president of Google, conducted several A/B tests on 40 different shades of blue to find the best color for Google links. A/B testing is great for this type of task, because there are only so many blues, and Google clearly knows what they want: find a color that gets the most clicks.

This situation is undoubtedly suitable for A/B testing. But if you use the same treatment for each problem, you will find yourself indulged in the excitement of successfully climbing the hills, but miss the North Star that is visible in the climbing process.

For example: A DVD rental site wants to test different login page designs. Suppose they really find the perfect design, and how much revenue growth can they bring? Is it because the design is not beautiful enough to lack customers? Or is it because people prefer to watch the show on Netflix than renting a DVD? If the entire industry is being updated and your business model is out of date, then A/B testing can’t tell you the real reason for failure.

How to find the right mountain?

1. Remember where you are going.

We all like the fruits hanging on the lower branches. They are so tempting, especially when you are lost and need to develop your business quickly. But are they consistent with your product goals?

Easy victory will take you in multiple directions. But getting 10 simple victories in 10 directions will quickly exhaust your resources. What you should do is take a small step in the right direction – focus on the most important goals, and make a sense of direction. .

2. Talk to real customers.

Sometimes you need to forget big data.

We are trapped in the world of big data, butForgetting the data is nothing but a concentration of the customer’s voice. Talking to some real customers may not help you scale up your business, but it will bring you closer to real human needs, and it can help you discover problems that are truly valuable.

You may be worried: What if the answers to a small group of people are too specific to be promoted to a larger customer base? The good news is that the world is similar in many ways, so one’s struggle can resonate with many people in the world. In user research, this is seen as the power of small data.

3. Find the real bottleneck before solving the problem.

A/B testing will only give customers a limited choice. If the real solution is neither A nor B, then the two solutions customers will not like it, and you will eventually launch only one that they are not so annoying, and the real problem remains unresolved.

If you are using A/B testing to fix something, make sure you have found the root cause of the problem.

For example, you noticed that the abandonment rate on the payment page has increased, and I hope to fix this by improving the content on the page. However, the increase in the abandonment rate may be due to a problem with the home page, or too many pop-ups that distract the customer, or the page itself is loading too long. It may not require testing to improve features such as “buy” buttons or messages on the payment page – problems may occur before the customer sees the content!

Question 2: When you climb a mountain, does the landscape change?

In other words, have you controlled other factors to reduce the interference in the test?

Let us refer to all the factors in your experiment as “environment.” When you’re testing two versions of a button, other factors include browsers, device types, regions, months, promotions, and competitors’ activities.

The following picture is an analogy: You want to compare route A and route B which route will allow you to climb to the top faster. The only difference you want to test is the path. However, during your test, both paths have different wind, rainfall, air, and vegetation patterns. Because this environment and climate may change during other seasons beyond the testing phase, this makes the test results meaningless at other times of the year.

Not all products are suitable for A/B testing

If other factors have changed in your experiment of controlling variables, then you are comparing apples and oranges.

How to manage a changing environment

There are two types of environments:

  • The things you can control are called internal environments. Such as the company’s marketing activities, program code changes, product planning, and so on.

  • What you can’t control is called the external environment. Such as seasonal changes, industry trends, competitors, customer behavior, etc.

Smart people control what they can control, knowing the difference between what they can control and what they can’t control.

Note two common internal environmental changes

  • Marketing activities. People are affected by marketing activities, and the results of your tests are invalid. Improved signals may be overstated or lost in marketing. So, if the company is doing promotional activities, postpone your experiment until the end of the event.

  • Changes in the product. You can’t test the effect of a new product image while launching a new banner ad at the top of each page. The banner will consume the customer’s attention, and the product image you are testing may not be able to get enough traffic. When you analyze the results of the test, you can’t know if your new product image or the combination of banner + product image drives the change.

How to control the internal environment

  • When testing features, make sure other factors don’t change. If you want to test multiple features, please communicate with your team, in order.

  • Communicate with other teams to ensure they are not testing competitive features or changing any features. You should have a shared document to check each other’s test schedule.

Of course, when multiple factors change, you can use multivariate regression to analyze the test results, but there may be more interference items. So don’t be greedy, only test one feature at a time, unless you are an expert. Remember, taking a small step in the right direction is better than aiming at a big step.

External environment

Now you have determined that the feature you are testing is the only different thing on the site, let’s look at two external environments that you can’t control.

1. “Adjustable” external environment: seasonality and trends

  • Seasonal: This is a repeating pattern. Seasonal effects include spring, summer, autumn and winter, back to school, holidaysEtc.

  • Trends: It will change over time, but it is not cyclical, nor necessarily linear.

The impact of seasonality and trends is not just a simple multiplier effect; their impact on testing can show different results. For example, your feature may have no effect in April, but it increased sales by 10% in June and 5% in August.

  • How to control:

Build a time series regression model to adjust for seasonal effects and trends. If there is not enough data to build a time series model, you can collect historical data from the industry or past performance of the product as a baseline.

2. “Unadjustable” external environment: competitors, customer behavior and industry changes

Competitors may introduce a new feature that competes directly with the features you are testing; your customers may clear browser cookies or change browsers; your industry is experiencing something similar from physical to digital, from oil and gas The transition to electricity, from labor-intensive to automated, your test results may be “contaminated” by this shift.

What to do:

In this case, you can use simulation data to estimate the extent of the external impact and to assess the risk as much as possible.

Last, remember that there may not be a perfect mountain or a perfect static environment in the world, but you should check your options and choose the one that is closest to the target. Follow the instructions in this article to perform a successful A/B test.

Information

#5 key points against stress#

How do the founders manage stress?

Not all products are suitable for A/B testing

  • Continuous learning is important. While it seems counterintuitive, staying busy can actually help manage stress. When your brain is active and learning new things, you can focus on your daily work.

  • Balance your time with being alone and socializing with others. There are many things you can solve on your own, but there are many things that need help from others. In this case, working with the board, team, and family is a better option. When you need to rely on othersPlease, please be sure to vent or seek feedback.

  • Follow your health. As a founder of a busy startup, it’s easy to put health in a secondary position. Improve your eating habits and keep exercising.

  • Improve your travel habits. Traveling is the easiest way to get out of your daily life and disrupt your physical and mental health. In the journey, introduce some simple products and habits into your daily life: before going to bed, wash your face, make a cup of hot tea, and reply to the last email of the day. Keep exercising before the morning meeting. Try to avoid getting sick on the road.

  • Give yourself a time away from work. Every once in a while, pressing the “Pause” button and temporarily leaving the working state is the best way to get you back to an efficient state.

#7 Effective Mind Maps#

How to choose the most suitable mind map?

The mind map is one of the most common data visualization methods. They are often used as a process of expression thinking because you can create “branch” from a central idea to explore different relevant content. Here are 7 common mind maps and application scenarios:

Not all products are suitable for A/B testing

  • Flowchart. Typically used to visualize a process, progress, or a set of instructions.

  • Multi-function flow chart. Suitable for determining the cause and impact of a particular event.

  • Bracket map. Used as a real object or situation, not a concept or an idea.

  • Tree diagram. Ideal for classifying and organizing information, the design itself represents an actual tree.

  • Circle. A circle centered on the central idea, each circle contains the relevant ideas of the previous layer.

  • Bubble chart. Explain a subject or topic by using relevant adjectives and descriptions.

  • Double bubble chart. Also known as Venn diagram, it combines two bubble diagrams to show related themes.