“More than half of our ideas deliver no value, we just don’t know which half” – John Wanamaker (3, pg 78)
Product Testing Statistics
Experiment Statistics (3, pg 79)
- 65% of feature are rarely or never used
- At Google and Bings, only 10% to 20% of experiments generate positive results (Harvard Business Review)
- At Microsoft 1/3 have positive results, 1/3 have neutral results, 1/3 have negative results
Product Testing Grids
Qualitative Tests |
Quantitative Tests |
|
Marketing Tests |
Marketing Materials | Landing page/ smoke test
Explainer video Ad campaign Marketing a/b tests Crowdfunding |
Product Tests |
Wireframes
Mockups Interactive prototype Wizard of Oz & Concierge Live Product |
Fake door/ 404 page
Product analytics & A/B tests |
MVP Test Grid (1, pg 93)
Qualitative Tests |
Quantitative Tests |
|
Behavioural |
Usability testing | A/B testing Analytics |
Attitudinal |
User Interviews | Surveys |
Research Methods Framework (1, pg 230)
Qualitative Marketing Tests
Marketing Materials (1, pg 93)
- To understand which benefits resonate with customers
- To understand how they react to different ways of showing the benefits
- Aim is to understand how they find the marketing material and why
- Marketing material can be landing page, video, advert, email
Quantitative Marketing Tests
Landing page/ smoke test (1, pg 94)
- Traffic is directed to a landing page
- On this page they are asked to show interest (e.g. a sign up button, or a plans and pricing page)
- There is no product yet
- A ‘coming soon’ message is often displayed to those who show interest
Explainer video (1, pg 94)
- Same as landing page
- For products that are harder to explain on a landing page (e.g. dropbox)
Ad campaign (1, pg 94)
- As adverts don’t allow you to display a lot, this is more appropriate for optimising customer acquisition and not product-market fit
- Can advertise to different demographics to check hypothesis about target market
- Measure clickthrough rate to measure which ads (and from which demographics) prove more successful
Marketing A/B testing (1, pg 94)
- Test two alternative designs to compare how they perform
- Run the tests in parallel with 50% of the traffic to each for simplicity
Crowdfunding (1, pg 94)
- Advertising your product on a site like Kickstarter and asking people to pay for the product in advance of it being made
- Set a minimum threshold for funding where you do not build your product before you have raised £X of funding
- The donators to your product will then get your product once it has been built (i.e. pre-order the product with a discount)
Qualitative Product Tests
Wireframes/ Mockups/ Interactive Prototypes (1, pg 100) (2, pg 124)
- Demonstrate or show concepts to user to gauge their feedback (e.g. wireframes)
- Have an ‘ask’ as a definitive pass-or-fail criteria
- Commitment, monetary value, time, or another investment to show that they are interested
- E.g. dropbox did a video of their concept (advert as if they had built it) to convince investors
- Variations in interactivity and fidelity
- (fidelity refers to how closely the artifact looks like the final product)
Low fidelity prototype (4, page 49)
- Start with the a persona
- Draw the homepage and ask what actions the user wants to do from there
- For each action draw a box (each box is a story)
- Continue until the persona has completed their actions (including exploring edge cases) and then start with another persona
Concierge (2, pg 122)
- Deliver the end result to your customer manually
- Customer understands that it is being done manually and there is no appearance of a final solution to them
- Conduct with just enough users as this is labour intensive
Wizard of Oz (2, pg 123)
- Deliver the end result to your customer manually
- Customer is not aware that it is manual behind the scenes and thinks they are using the end product
- Tempting to leave them up as if successful you will get value from it, but it is expensive to run
- Can be combined with A/B testing
Quantitative Product Tests
Fake Door/ 404 page (1, page 100)
- Good to test demand for a new feature
- Include a link or button on the product to direct customers to a new feature
- The link leads to a page saying it hasn’t been built yet and asking for why they would find this feature valuable
- Overuse will make customers unhappy
Product A/B tests (1, page 100)
- Used to compare performance of two alternative user experiences in your product
Qualitative Behavioural Tests
Usability testing
- Online tools can be used to give a user a task and record them completing the task
- Users are asked to talk through how easy it is to complete a task
Quantitative Behavioural Tests
A/B Testing
- Two different versions of the product are shown to the user
- Differences in behaviour are tracked (e.g. conversion percentage)
Analytics
- Tracking on the product of the users behaviour
- Data can then be analysed to see if hypothesis was acheived
Qualitative Attitudinal Tests
User Interviews
- One-on-one interview with a user
- Coming soon: Tips on User interviews
Quantitative Attitudinal Tests
User Surveys
- Coming soon: Tips of User Surveys
References
- Lean Product Playbook by Dan Olsen
- Escaping the Build Trap by Melissa Perri
- Mastering Professional Scrum by Stephanie Ockerman and Simon Reindl
- User Stories Applied by Mike Cohn
