Evidence Driven Design

On a recent project, I noticed how my team struggled to prioritize users needs, features and disruptive innovation. Of particular concern was the familiar problem: How can we get our users to validate a new concept or idea that they had never seen or even thought about?

Saying "Let's put it in front of users and see what they think?" is a goal we all support while remaining occasionally skeptical about its relevance.  Can we get valid feedback about a revolutionary idea that is disruptive enough to be:

• Difficult to adequately describe without building the entire product or related features.
• Disturbing users who are uncomfortable with sudden change and may skew otherwise promising results.

It dawned on us: User testing at different design stages may be premature, the results gathered may be misleading or user testing at that moment may just be too darn expensive.




We identified a potential extension to Design Thinking that would give us equivalent answers in less time. We call it:  Evidence Driven Design 🔍

It's an idea stretches back to the Magna Carta. OK, that might be too much of a stretch. Evidence Driven Design is more like going to Small Claims Court  👩🏻‍⚖️.

It says:

1. Anyone on the team can provide evidence that supports a new idea or design.  Their burden of proof is simple: They must provide enough compelling evidence to adequately prove their idea is sound and worth addressing.
2. The quality of evidence presented must be equivalent to that expected at this point in the design process.  

The result is (we hope): Faster and better design decisions with greater team engagement.

Elephant in the Room

Evidence Driven Design let's us address Design Thinking concerns that we occasionally have, but don't always want to talk about.  This includes issues like:

• User testing given when there is not enough context to generate reliable feedback.
• Biases that can be amplified by other errors including story setup, low test fidelity or even individual user biases. 
• Sometimes, everyone just knows that an idea is great.

What's New?

We want to give all team members the permission to contribute great ideas.  All they need to do is: prove their case.

Here are our Design Thinking extensions that we think enable Evidence Driven Design:

1. Anyone on the team can make a case for the prosecution in favor of a new idea.
2. The burden of proof is on that team member. They must convince the judges (the team stakeholders) that their idea is sound and is supported by evidence.
3. They must provide evidence that matches the stage in the design process that they are attempting to replace. If a simple survey was warranted, then a description of user intent with some bug listing may be enough. If full user testing was appropriate, then their burden is higher: Perhaps a market analysis and comparison of competitive features with a suggested improvements.
4. Time matters. You cannot waste the valuable time of other team members. Winning your case will often require lots of preparation and a clear presentation of value.  You must prepare your case.

Time will tell if this  Design Thinking extension works for us.  One immediate upside is clear: Design is now a direct responsibility of everyone.

Make it Fast

As I have said in the past, the three most important attributes of search are: Relevance, Relevance and Relevance (with apologies to my friends in real estate 😀)

Speed

The next most important search feature has got to be SPEED 🚅.  Slow results are nearly fatal in any search application.  Irrelevant and slow results will have your users heading for the exits.



Check out Google. They do everything they can to show results in less than one second. Studies from several SEO firms suggest that 3 seconds is an upper limit. Google likes results in under two seconds. My experience from A/B tests says users start getting agitated at around one second (the heal of the hockey stick above). You have until about 2.5 seconds to turn off the hour glass and show some results. If not, your user is inclined to look somewhere else. You don't need complete results. Offering some type of meaningful response is a critical action.

Value

If you think of value as a metric that integrates time and relevance, you have an even bigger concern.


You have more time - but still limited time - to start offering value. Remember: Your users don't really want to use search. They want to have used it. Get them their answer and get moving. Past experience suggests that application type matters: users of complex applications are more lenient. Just watch out for simple searches on the desktop. You need to get to a good answer in 10 seconds or less. If you fail big - or fail too often - your user is not just inclined to leave. There's a good chance they will exit and never come back.

To add some urgency to the mix: Mobile users have less patience. They can leave - never to return - in as little as 5 seconds.

So don't forget to focus on search result relevance.  Just make it fast.