SurveyMonkey
Feature Importance (MaxDiff)
The Feature Prioritization solution helps you understand what items or features matter most to your audience. The solution is based on the MaxDiff methodology which helps to prioritize new products or features, and lets you know what your customers care about most.
How Feature Prioritization Works
Feature Prioritization builds your survey questions using the items you enter. You don’t have to create each MaxDiff question yourself—you enter the items you want to include in your survey, and we create the questions for you.
Feature Prioritization Experimental Design
In a Feature Prioritization (MaxDiff) study, respondents should see each item about the same number of times as the other items. This solution creates 30 versions of the survey and distributes them randomly to respondents to achieve the most bias-free results. However, you can choose how many items are included in each set and how many questions each respondent sees.
We consider these metrics when creating the different versions of your survey. These metrics ensure that your items are represented fairly to reduce bias in your results:
- One-way frequencies: Each item should ideally appear an equal number of times to the respondent as they take the survey.
- Two-way frequencies: Each item should ideally appear in the same set with each other item an equal number of times.
- Connectivity: Each item should directly or indirectly be shown with every other item across the thirty versions of the survey.
- Within-set positional balancing: Each item should ideally appear an equal number of times in the top, middle, and bottom of the question when shown to respondents.
- Across-set positional balancing: The same item shouldn’t be shown in successive or nearby questions.
Setting Up Your Survey
Learn how to build your Feature Prioritization study.
Get Started
To set up your study:
- Get started by adding a Study title.
- Select Next: Set up MaxDiff to add survey components.
Set Up MaxDiff
To add survey components:
- Add Maxdiff Introduction to let respondents know exactly what you want them to do while completing your survey.
- Add MaxDiff question text to let respondents know how to respond to your MaxDiff question. This text goes above your MaxDiff question. Make sure people know to choose one best and one worst from each set of items.
- Choose the label to use for the “Best” and “Worst” options in your MaxDiff questions. “Best” and “Worst” are selected by default. You can also create your own labels.
- Add items to MaxDiff to use in your questions. You can add items one at a time or import items in bulk.
- Select + Add an item to enter a new item. Add a label and image, if you want.
- Respondents see the Description, so add any context they should know.
- Select Import items to use a list of items. Preview the items on the right. When you’re done, select Import.
- From Set up Maxdiff Settings, edit Items per set, and Sets per respondent. This option determines how many items to include in each question and how many question sets you want respondents to see. When you’re done, you can Preview MaxDiff Question below.
- Items per set: The number of items shown in each MaxDiff question set. Add at least 5 items.
- Sets per respondent: The number of MaxDiff question sets each respondent sees. Review the help text above Sets per respondent as it changes based on the number of items you add.
- Image scale: The size of the image you’ve added to your items. The scale applies to all images. Review the image in Preview MaxDiff Question.
- Preview MaxDiff Question: See what your questions look like in the survey. Select Single set to see what a single question will look like. Select All items to preview all items together in one set.
- Select View report to see the experimental design report. This report tells you how we balance your study to avoid biased results. To save a copy of this report, select Download report (.csv).
- Select Next: Customize survey.
Customize and Preview Your Survery
You can add additional custom questions to your survey to gather other important information from your target audience. You can:
- Use the Question bank to add pre-written questions, certified by survey methodologists.
- Add logic for the questions you add so only some people see certain questions.
- Style your survey, choose a theme, or brand it with your company’s colors and logo.
- Add custom variables to track known-data about your respondents through the survey and use them to filter your results in Analyze.
- Add translations to your survey so people can take it in another language.
Preview Survey
Once your project is set up, select Preview survey from the top-right corner to test your survey in a new window and see what it looks like to respondents. You can even share the preview with others to gather feedback.
Toggle between Desktop, Tablet, and Mobile views to see how your survey looks on various devices.Once your survey looks good to you, select Next: Collect responses.
Collecting Responses
When you’re ready to send your survey, select the Collect icon from the left-side menu.
There are 2 ways to collect responses:
- Share a survey link: Create a Web link to send your survey to your own audience.
- Buy responses from your ideal respondents: Target certain demographics with SurveyMonkey Audience.
You can create multiple collectors for your study.
Share a Survey Link
Create a web link you can send however you want. To share a survey link:
- On the Collect page, select Add new collector, then select Web Link collector.
- Customize your Web Link settings. You can:
- Set a custom survey end page.
- Allow multiple responses.
- Set a cutoff date and time.
- Set response limits.
- Copy your Web link and paste it where you’d like to to send your survey.
Buy Responses from Your Ideal Respondents
SurveyMonkey Audience lets you choose respondents based on hundreds of targeting options, so you can target respondents based on country, demographics, employment status, hobbies, religion, and more.
To buy target responses:
- Go to the Collect page.
- Select Choose an audience or select Add new collector, then select Buy Responses.
- Select your target audience by choosing Country, Gender, Age Range, and Household Income criteria.
- Select Targeting criteria to browse and choose from hundreds of other targeting options.
- Choose how many complete responses you need. We provide a recommendation based on the number of items in your study.
- (Optional) Select whether to Add custom screening questions. Screening questions help you narrow down your target audience and disqualify people who aren't the best fit for your survey.
- If you add a screening question, estimate how many people you expect to qualify for your survey.
- If you choose to include a screening question in your survey, open the Customize survey page in a new tab.
- Add a qualifying question to the beginning of your survey. We recommend either Multiple Choice or Checkboxes.
- Add skip logic that disqualifies people if they select certain answer choices.
- Return to the Target audience setup page to estimate your Qualification Rate.
- If you want to apply exclusions or schedule your Audience launch for a later date and time, see Additional Audience Settings.
- Review your target audience summary.
- When you’re ready, select Checkout.
- Review the details of your order.
- Under Payment Method, select to pay with Credit or Debit Card, or with My Credits.
- Enter your Billing Details, review the total, and select Confirm.
Once you submit payment, we start gathering responses for your survey right away.
Analyzing Results
Select the chart icon on the left side of the screen to start analyzing your results. In the Analyze section, there are Overview, Counts, Empirical Bayes, TURF, and Survey Results.
- Overview: Track your survey status and see how many responses have been collected by each collector.
- Counts: Get a simple view of how often items were chosen as Best or Worst.
- Empirical bayes: Learn how each item performed compared to others.
- TURF: Review item combinations that are likely to appeal to your target audience.
- Survey results: View charts and data for your custom questions.
Counts Analysis
Counts analysis shows you how often items were chosen as Best or Worst. This data helps you quickly understand how respondents rated each item. You can view a few different data sets in the chart: Best counts, Worst counts, Best and Worst counts, or Simple counts.
- Best counts: How many times an item was chosen as Best.
- Worst counts: How many times an item was chosen as Worst.
- Best and Worst counts: Compare how many times an item was chosen as Best and Worst.
- Simple counts: Total Best ratings minus Worst ratings for an item.
View all data for each item in the table below your chart. The table also includes the Count proportions metric, which is the quantity of Best or Worst counts divided by the number of times people saw the item.
Empirical Bayes Analysis
Empirical Bayes shows how respondents feel about each item. It estimates each item's likelihood of being chosen as Best, relative to other items.
Empirical Bayes calculates a utility score for each item, which is a measure of how well an item performed. We calculate this score across all sets and respondents. We use the following data to calculate the utility score for each item:
- Number of times chosen as Best
- Number of times chosen as Worst
- Number of times shown
First, we combine all responses to find the utility score for each item. Then, we calculate an item's utility value for each respondent. If someone didn't see all items, we use Bayesian Pooling (or "Shrinkage") to estimate how someone would have responded to the item. We assume that they would answer similarly to the aggregate score, since it combines all responses. We "shrink" the respondent's utility value toward the aggregate score for the item they didn't see. These scores help us estimate how likely it is that an item is chosen as Best.
The chart shows the utility score for each item. A higher score means that an item is more likely to be chosen as Best and may be more important to your target audience.
- Using Empirical Bayes
- Zero-centered
The table below the chart shows each item’s score and the 95% confidence interval. The confidence interval is a range of scores that contains the score we'd see a certain percentage of times if we did the survey over and over. For example, an item has a 95% confidence interval of [13–14]. In other words, the item would score between 13 and 14 in 95 out of 100 repetitions of the survey.
TURF Analysis
TURF stands for Total Unduplicated Reach and Frequency. It's a technique that can help you understand how different groups of products or features appeal to audiences.
Our MaxDiff TURF analysis tool simulates item combinations that are likely to appeal to your target audience. Use this data to prioritize products or features to reach the most people.
Our TURF analysis tool ranks combinations based on two key metrics: reach and frequency.
- Reach: the percentage of respondents with at least 1 appealing item in the combination. A “reached” respondent has an appealing item in the combination. A combination with high reach has a mix of items that appeal to the most people.
- Frequency: the average number of items in the combination that respondents found appealing. Frequency is always a number between 1 and 2. Higher frequency means that a combination’s items were more popular.
- Using TURF Analysis
Filtering Your Results
On any page, select the Filters button above the chart to filter your data. Any filters you apply to one chart also apply to others. For example, if you add a Web link collector filter to your Counts analysis, we’ll also apply it to your Empirical Bayes analysis.
Exporting Data
You can export Counts data, aggregated Empirical Bayes data, TURF analysis data, full response data, or individual responses.
- Counts or Empirical Bayes data
- Full response data
- Individual responses