SurveyMonkey
Feature Importance (MaxDiff)
ADD-ON FEATURE: You can use this solution with our Audience collector on any SurveyMonkey plan type. Get started now, or explore our other market research solutions. This solution is available for US Data Center accounts only.
The Web Link collector is available on Advantage, Premier, and Enterprise plans.
The MaxDiff solution helps you understand what items or features matter most to your audience. This is helpful when you’re prioritizing new products or features and want to know what your customers care about most.
The solution is based on MaxDiff methodology. We create your survey questions using the items you enter. You don’t have to create each MaxDiff question yourself – you enter the items you want to include in your survey, and we create the questions.
How MaxDiff Works (Experimental Design)
The MaxDiff solution builds your survey questions using the items you enter. You don’t have to create each MaxDiff question yourself – you enter the items you want to include in your survey, and we create the questions.
MaxDiff is not a predictive model. It uses algorithmic predictions to estimate outcomes based on multiple-choice scenarios. We recommend that you provide your own notices about these tools to your respondents.
Expand the dropdown to learn more about the experimental design.
- MaxDiff experimental design
Data Retention
Results from your study are retained for 3 years. After 3 years, the results are deleted.
Setting Up Your Survey
Learn how to build your MaxDiff study.
Get Started
To set up your study:
- Create a new MaxDiff study.
- Replace Untitled study with a study title on the Get Started page. Review what to expect in setup and after launching your study.
- Select Next: Set up MaxDiff.
Set Up MaxDiff
To add survey components:
- Add an introduction to let survey takers know exactly what you want them to do while completing your survey.
- Add MaxDiff question text to tell survey takers how to respond to your MaxDiff question. This text goes above your MaxDiff question. Make sure people know to choose one best and one worst from each set of items.
- Choose the label to use for the “Best” and “Worst” options in your MaxDiff questions. “Best” and “Worst” are selected by default. You can also create your own labels.
- Add items to MaxDiff to use in your questions. You can add items one at a time or import items in bulk.
- Select Add an item to enter a new item. Add a label and image if you want.
The Item description is what survey takers will see. - Select Import items to use a list of items. Preview the items on the right. When you’re done, select Import.
- Select Add an item to enter a new item. Add a label and image if you want.
- Edit items per set, totals sets. This option determines how many items to include in each question and how many question sets you want survey takers to see. When you’re done, you can Preview MaxDiff Question below.
- Items per set: The number of items shown in each MaxDiff question set. Add at least 5 items.
- Sets per respondent: The number of MaxDiff question sets each survey taker will see. Read the help text above the Sets per respondent field; it changes based on the number of items you add.
- Image scale: The size of the image you’ve added to your items. The scale applies to all images. View the image in the Preview MaxDiff Question section.
- Preview MaxDiff Question: See what your questions will look like in the survey. Select Single set to see what a single question will look like. Select All items to preview all items together in one set.
- Select View report to see the experimental design report. This report tells you how we balance your study to avoid biased results. To save a copy of this report, select Download report (.csv).
- Select Next: Add custom questions.
Add Custom Questions
You can add additional custom questions to your survey to gather other important information from your target audience. You can:
- Use the Question Bank to add pre-written questions, if you want.
- Implement logic for the questions you add so only some people see certain questions.
- Brand your survey so the survey includes your brand colors and logo.
- Add custom variables to track data about your respondents. You can use custom variables as a filter when you analyze results.
- Add translations to your survey so people can take it in another language. Learn more about setting up survey translations.
MaxDiff questions are locked to protect our proven methodology and can’t be edited. You can preview them but can’t change them.
Preview Survey
Select Preview survey from the top right corner to test your survey in a new window and see what it will look like to respondents. You can even share the preview with others to gather feedback.
Select Next: Collect Responses when you’re ready to send your study.
Collecting Responses
When you’re ready to send your survey, select the Collect icon from the left-side menu.
There are two ways to collect responses:
- Share a survey link: Create a web link to send your survey to your own audience. This feature is only available on certain plans.
- Target your ideal respondents: Buy responses from certain demographics using the SurveyMonkey Audience panel. This feature is available on all plans.
You can create multiple collectors for your MaxDiff study.
Share a Survey Link
PAID FEATURE: available on Advantage, Premier, and Enterprise plans. Free plans can use SurveyMonkey Audience to collect responses.
Create a web link that you can send your way. To share a survey link:
- On the Collect page, select Add new collector, then select Web link collector.
- Customize your Web link settings. You can:
- Set a custom survey end page
- Allow multiple responses
- Allow response editing
- Set a cutoff date and time
- Set response limits
- Copy your web link when you’re ready to send your survey.
Target your Ideal Respondents
SurveyMonkey Audience panelists have been sorted based on hundreds of targeting options so you can target your respondents based on country, demographics, employment status, hobbies, religion, and more.
To choose your target audience:
- On the Collect page, select Add new collector, then select Buy Responses.
- Select your target audience by choosing Country, Gender, Age, and Income criteria.
- Select More Targeting Options to browse and choose from hundreds of other targeting options.
- Choose how many complete responses you need. We’ll provide a recommendation based on the number of items in your study.
- (Optional) Select whether to add a screening question. Screening questions help you narrow down your target audience and disqualify people who aren't the best fit for your survey.
- If you add a screening question, estimate how many people you expect to qualify for your survey.
- If you choose to include a screening question in your survey, open the Customize survey page in a new tab.
- Add a qualifying question to the beginning of your survey. We recommend either Multiple Choice or Checkboxes.
- Add skip logic that disqualifies people if they select certain answer choices.
- Return to the Target audience setup page to estimate your Qualification Rate.
- Review your target audience summary.
- When you’re ready, select Checkout.
- Review the details of your order.
- Under Payment Method, select to pay with Credit or Debit Card, or with My Credits.
- Enter your Billing Details, review the total, and select Confirm.
Once you submit payment, we'll start gathering responses for your survey right away.
Analyzing Results
Select the chart icon on the left side of the screen to start analyzing your results. In the Analyze section, there’s an Overview, Counts, Empirical Bayes, Survey results, and Individual Responses.
- Overview: Track your Audience project's status, view how many responses are collected, and see a summary of your targeting criteria.
- Counts: Get a simple view of how often items were chosen as Best or Worst.
- Empirical Bayes: Learn how each item performed compared to others.
- TURF: Review item combinations that are likely to appeal to your target audience.
- Survey results: View charts and data for your custom questions.
- Individual responses: View each survey taker’s response.
Counts Analysis
Counts analysis shows you how often items were chosen as Best or Worst. This data helps you quickly understand how survey takers rated each item. You can view a few different data sets in the chart: Simple counts, Best counts, Worst counts, or Best and Worst counts.
- Simple counts: Total Best ratings for an item minus Worst ratings.
- Best counts: How many times an item was chosen as Best.
- Worst counts: How many times an item was chosen as Worst.
- Best and Worst counts: Compare how many times an item was chosen as Best and Worst.
View all data for each item in the table below your chart. The table also includes Count proportion, which is the quantity of Best or Worst counts divided by the number of times people saw the item.
Empirical Bayes Analysis
Empirical Bayes shows how survey takers feel about each item. It estimates each item's likelihood of being chosen as Best, relative to other items.
Empirical Bayes calculates a utility score for each item, which is a measure of how well an item performed. We calculate this score across all sets and survey takers. We use the following data to calculate utility score for each item:
- Number of times chosen as Best
- Number of times chosen as Worst
- Number of times shown
First, we combine all responses to find the utility score for each item. Then, we calculate an item's utility value for each respondent. If someone didn't see all items, we use Bayesian Pooling (or "Shrinkage") to estimate how someone would have responded to the item. We assume that they would answer similarly to the aggregate score, since it combines all responses. We "shrink" the survey taker's utility value toward the aggregate score for the item they didn't see. These scores help us estimate how likely it is that an item will be chosen as Best.
The chart shows the utility score for each item. A higher score means that an item is more likely to be chosen as Best and may be more important to your target audience.
Using Empirical Bayes
Select a scale in the top right corner to change how you view the data:
- Probability-scaled: Shows the likelihood that people will choose an item as Best. We show each utility score as a percentage. The highest scoring item is most likely to be chosen as best. The scores add up to 100.
- Zero-centered: Adjusts the scale to use 0 as the average score. A positive score means the item performed better than average. A negative score means the item performed worse than average.
- Raw scores: View the raw utility scores for each item. Raw scores reflect the actual choices people made during the survey. Raw scores are not probabilities.
The table below the chart shows each item’s score and the 95% confidence interval. The confidence interval is a range of scores that contains the score we'd see a certain percentage of times if we did the survey over and over. For example, an item has a 95% confidence interval of [13–14]. In other words, the item would score between 13 and 14 in 95 out of 100 repetitions of the survey.
TURF Analysis
TURF stands for Total Unduplicated Reach and Frequency. It's a technique that can help you understand how different groups of products or features appeal to audiences.
Our MaxDiff TURF analysis tool simulates item combinations that are likely to appeal to your target audience. Use this data to prioritize products or features that will reach the most people.
Our TURF analysis tool ranks combinations based on two key metrics: reach and frequency.
- Reach: the percentage of respondents with at least 1 appealing item in the combination. A “reached” respondent has an appealing item in the combination. A combination with high reach has a mix of items that appeal to the most people.
- Frequency: the average number of items in the combination that respondents found appealing. Frequency will always be a number between 1 and 2. Higher frequency means that a combination’s items were more popular.
Using TURF Analysis
Use the Items and Items per combination dropdowns to simulate different combinations. Consider simulating multiple combination sizes to learn how different items impact reach and frequency.
Filtering Results
On any page, select the Filters button above the chart to filter your data. Any filters you apply to one chart will also apply to others. For example, if you add an Age filter to your Counts analysis, we’ll also apply it to your Empirical Bayes analysis.
Exporting Data
You can export Counts data, aggregated Empirical Bayes data, full response data, or individual responses.
- Counts or Empirical Bayes data
- Full response data
- Individual responses