Evidence Driven Design

On a recent project, I noticed how my team struggled to prioritize users needs, features and disruptive innovation. Of particular concern was the familiar problem: How can we get our users to validate a new concept or idea that they had never seen or even thought about?

Saying "Let's put it in front of users and see what they think?" is a goal we all support while remaining occasionally skeptical about its relevance.  Can we get valid feedback about a revolutionary idea that is disruptive enough to be:

• Difficult to adequately describe without building the entire product or related features.
• Disturbing users who are uncomfortable with sudden change and may skew otherwise promising results.

It dawned on us: User testing at different design stages may be premature, the results gathered may be misleading or user testing at that moment may just be too darn expensive.




We identified a potential extension to Design Thinking that would give us equivalent answers in less time. We call it:  Evidence Driven Design 🔍

It's an idea stretches back to the Magna Carta. OK, that might be too much of a stretch. Evidence Driven Design is more like going to Small Claims Court  👩🏻‍⚖️.

It says:

1. Anyone on the team can provide evidence that supports a new idea or design.  Their burden of proof is simple: They must provide enough compelling evidence to adequately prove their idea is sound and worth addressing.
2. The quality of evidence presented must be equivalent to that expected at this point in the design process.  

The result is (we hope): Faster and better design decisions with greater team engagement.

Elephant in the Room

Evidence Driven Design let's us address Design Thinking concerns that we occasionally have, but don't always want to talk about.  This includes issues like:

• User testing given when there is not enough context to generate reliable feedback.
• Biases that can be amplified by other errors including story setup, low test fidelity or even individual user biases. 
• Sometimes, everyone just knows that an idea is great.

What's New?

We want to give all team members the permission to contribute great ideas.  All they need to do is: prove their case.

Here are our Design Thinking extensions that we think enable Evidence Driven Design:

1. Anyone on the team can make a case for the prosecution in favor of a new idea.
2. The burden of proof is on that team member. They must convince the judges (the team stakeholders) that their idea is sound and is supported by evidence.
3. They must provide evidence that matches the stage in the design process that they are attempting to replace. If a simple survey was warranted, then a description of user intent with some bug listing may be enough. If full user testing was appropriate, then their burden is higher: Perhaps a market analysis and comparison of competitive features with a suggested improvements.
4. Time matters. You cannot waste the valuable time of other team members. Winning your case will often require lots of preparation and a clear presentation of value.  You must prepare your case.

Time will tell if this  Design Thinking extension works for us.  One immediate upside is clear: Design is now a direct responsibility of everyone.

Make it Fast

As I have said in the past, the three most important attributes of search are: Relevance, Relevance and Relevance (with apologies to my friends in real estate 😀)

Speed

The next most important search feature has got to be SPEED 🚅.  Slow results are nearly fatal in any search application.  Irrelevant and slow results will have your users heading for the exits.



Check out Google. They do everything they can to show results in less than one second. Studies from several SEO firms suggest that 3 seconds is an upper limit. Google likes results in under two seconds. My experience from A/B tests says users start getting agitated at around one second (the heal of the hockey stick above). You have until about 2.5 seconds to turn off the hour glass and show some results. If not, your user is inclined to look somewhere else. You don't need complete results. Offering some type of meaningful response is a critical action.

Value

If you think of value as a metric that integrates time and relevance, you have an even bigger concern.


You have more time - but still limited time - to start offering value. Remember: Your users don't really want to use search. They want to have used it. Get them their answer and get moving. Past experience suggests that application type matters: users of complex applications are more lenient. Just watch out for simple searches on the desktop. You need to get to a good answer in 10 seconds or less. If you fail big - or fail too often - your user is not just inclined to leave. There's a good chance they will exit and never come back.

To add some urgency to the mix: Mobile users have less patience. They can leave - never to return - in as little as 5 seconds.

So don't forget to focus on search result relevance.  Just make it fast.

Natural Language Processing

Natural Language Processing (NLP) is a technology that Gartner reported as a mature and usable technology as early as 2014. Since then, the focus has been on vertical integration. Products like the Amazon Alexa do a remarkable job interpreting spoken phrases and delivering value through skills.  

An area of slower advancement is (and therefore opportunity!): Natural Language Question and Answering (NLQA)

Links:
Gartner 2016 Emerging Technology Key Trends
Forbes Summary

NLQA is currently in the dreaded Trough of Disillusionment (i.e., not meeting user expectations). Vendors like ThoughtSpot are defining the market but so far no one has delivered on the promise (think Hal in 2001: A Space Odyssey ♜ 😎)

The technology behind NLP is mature. If you studied it in school be prepared for a bit of a shock. The days of building and connecting your own parsers, lemma tools and Named Entity Recognizers (NER) are gone.  NLP-as-a-Service (NaaS) is here. Driven by machine learning, modern NaaS easily beats the best academic products, like the venerable Stanford NLP Parser, that were state-of-the-art just a few years ago.

If you want to jump to the head of the NLP class, consider using services like the Google Cloud Natural Language API.  It works remarkably well without custom training for both text ingestion and question parsing. It implements several high level functions that help developers execute commands and answer questions with greater accuracy and precision. One of its best features: dependency graphs that link verb (action) and noun (thing) phrases to create powerful interpretations without custom programming.

Here is an example of dependency parsing using the Google API:



The moral: NLP is more mature than you might think. Start at the top using NLP services and you can meet NLQA challenges faster than you (and your boss) may have thought possible.

The Search Mindset

Consider your intent when using a search function. I like to think of two possible expectations:

1. Discovery -  finding data and concepts related to search terms. In other words, "I am not sure what is out there. I want to learn more".  Search results typically get wider when showing something like "related concepts" concepts panel. 
2. Refinement - filtering to reduce results according to given search criteria. In other words, "I want to see only related data and concepts. Search results typically get narrower when showing  something like a faceted result.

This may not seem like a big distinction but discovery and refinement generally focus on different outcomes. Mixing both in a single result list can be confusing.  

With that said, the ability to "pivot" and let a user seamlessly switch from "narrowing" to "widening" activities is a critical feature that distinguishes awesome search engines from the also-rans. 

Relevance, Relevance, Relevance,

Question: What do you think is the NUMBER 1 capability in every great search engine?

For me, the answer never changes.
It is: RELEVANCE!

Finding data with search is easy. Finding too much data is unavoidable. Filtering, sorting, prioritizing and ranking to create the most relevant results has been a primary search goal for decades.

How many times have you found the best answer on the 4th page of Google results? If this was common I am sure you would have switched to something else long ago. Getting relevant results - with the best answers first - is what brings you back for more :).

Irrelevant results rapidly erode user confidence. Search engines that provide poor answers take users from hope to despair in only a few clicks.  Once trust is lost, it can be very hard to get back.


Relevance on the Web - Learning from Google

Google has always been the standard bearer for good sets of ranked results. But frankly, they have had an easier job than most enterprise search products. Web content can be sorted and prioritized using straightforward statistics - some as simple as counting the number of sites that point to a given page. Google also gets a boost from hyperlink text that describes a link target. The text in a link literally offers a curated description of the page it points to. Finally, web content is routinely stored with relatively large passages of unstructured text that makes context and meaning easier to determine.


Enterprise Search Analytics can learn from Google (again)

Modern enterprise search systems need to find alternative ways to filter and rank their results. Page popularity and link text aren't nearly enough.  Once again, Google shows us some of the options because they no longer exclusively rely on web ranking methods.  Let's review some of the recent advancements we have all seen but don't necessarily think of as "relevance builders".

Consider a Google search for the single term "mercury".  You get results like:

Google features now include:

1. Disambiguation of search terms. Note how this is not simply dictionary autocomplete (which is also good). It is a proactive listing of related concepts that answer the question: "Did you mean X?"
2. Knowledge graph of attributes related to the default concept.  It helps you understand that you are asking about the right thing before even looking at the results.
3. List of related topics. In other words, "Did you also know X?"
4. What other people searched for.  Learning from colleagues is often the quickest route to an answer - especially a high value curated answer.
5. And finally a "feedback" link to make sure items 1-4 are correct.  Identifying inaccurate outcomes is crucial to user happiness (and the underlying machine learning algorithms too).

Oh yeah: there are the search results too.  But you already expected that :)  

It used to be that enterprise search solutions needed to be different than Google. Nowadays, being like Google is a good place to start.

What's new?

Back in the Saddle

No blog posts for 3 and half years.  Time to fix that!   🔛

Today I embark on a new career at qlik.com focusing on search driven analytics. What's with that? I did some of the first commercial work in search driven BI 15 years ago.  Isn't this a step backward 🔁?

Search Driven Analytics - Ready for Prime Time.

So much has happened in the Business Intelligence world. Innovators like Qlik and Tableau have brought BI and analytics to the more people than ever before. In the meantime, big data, Hadoop /Spark, NoSQL, Natural Language Processing (NLP) and Machine Learning / Artificial Intelligence have become commonplace. Open Source has also moved to the forefront.  Facets, dimensions and clustering of terms is implemented in several world class open source packages.  

Search technology is finally positioned to leverage these advancements in ways that we never envisioned. Conveniently (for me at least), I have spent the last ten years researching and developing apps in all these areas. I am excited at the prospect and can't wait to share some new insights about the current state-of-the-art and where I think we can go. 

Case Study: "What are Important Big Data Concepts?"

**** Looking BACK to 2013 ****
Much of this stuff is still true :)

The State of Text

Natural Language Processing (NLP) is a core technology used in text analytics to understand the meaning of unparsed text passages. This type of analytics has come a long way in the past decade.  Deloitte does a nice job describing value in Text Analytics - The three minute guide. Just beware: The accuracy and reliability of many text mining technologies diminishes when the length of text passages get smaller [Bland et al,  Furlan et al].  Problems related to short text samples remains important since social networks, including Twitter, Facebook and Reddit, routinely create text excerpts that are too short to be reliably understood by traditional NLP algorithms [Iorica et al].

I wholeheartedly expect this situation to improve. Statistical text analytics is viable today. And deeper analysis in vertical domains, like healthcare, is quickly becoming a big data success story.  Just set your expectations accordingly.  Betting on text analytics across multiple business domains may leave you wanting in 2013.    

The State of Structured and Semi-Structured Text

Readers of this blog know: The more data is structured, the more I like it. Not because of quantity but because of quality.  Data that has been curated and prepared in structured or semi-structured form, is routinely superior [Hanig et al].  Balancing your analysis to include all types of data remains a very good practice for the foreseeable future.

Simple Case Study:
"Finding Important Big Data Concepts"

A few months ago, I wanted to build a list of the top ten most useful big data concepts. 

Unstructured searches, based on keyword terms, using Google, Bing and WolframAlpha yielded a 'diverse' set of results that described hardware provisioning and Hadoop along with various solution providers offering services and consulting. Conclusion: These simple search oriented queries produced somewhat underwhelming results.

Inconclusive search engine results is a common problem that I see when trying to dig into a broad subject area like "big data". To get more definitive results, I built a data set using various 'value proposition' statements from the public websites.  I pulled data for 50 Big Data Startups based on web searches, including lists from various technology sites. The results: concise data that was more interesting too.

Using topic clusters, I was able to build a collection of about 30 concepts. I then reduced that list to a 'top 10 ranking' (a process I will describe in an upcoming post). 

Here is my list of  Top 10 Big Data Concepts:

Why is the list potentially better? The intriguing aspect of this result set is: the data sources. Each source is curated in some fashion. Data from sources from public sites, give me links to other sites worthy of investigation. Publicly disclosed mission statements and value propositions from these referenced  vendors, give me definitive text that is easy to parse and organize into related topics.  It is not structured data - just data that is easier to structure.
   
Is the list actionable? When I present this kind of list in public presentations, I see that people are very interested. They copy down the results, thinking it offers great insight. 

My opinion: This list might be very significant.  But until we do more analysis, I would say this list contains more speculative than actionable facts.