Who

Data driven design is dumb

26 October

Hashtag “data-driven”

It’s basically misinformation.

A UX study says almost nothing.

You have to take all the studies together,

and the accessibility principles that have been established over decades of research and real-world application with diverse users,

and how it works in your own experiences for yourself, for your colleagues, for customers, for prospects, and so on.

Put that all together and you have evidence-based, data-driven UX.

If you’re design choices are just based on your latest study, you’re being “hashtag data-driven”. To be data-driven, actually data-driven, interpret your new study in the context of all the other data.

Hashtag “science”

Still, your study says almost nothing.

  1. Does your study have the power for the conclusions you draw?
  2. Are you aware of the confounders?
  3. Did you use techniques to nullify them, and
  4. if so, are they validated techniques? Regardless,
  5. did you account for the amount of inaccuracy they contribute in your power calculations?

Absent the rigor of science, the best your own studies should contribute to your designs is a

“woah, I just saw someone have an issue with that, let’s consider what happened but also maybe not”

or a

“I didn’t see anyone have an issue with that, but then again, let’s not jump to any conclusions.”

Hashtag “okay, then what?”

The first thing is to be hyper aware of your study’s limitations, how it changes the context in which your participant’s operate, how its phrasing creates biased behaviors, how its existence biases behaviors further, how language differences affect your interpretation of their responses… and the many many more biases.

Second is to be exact about the results. Consider the results for what they are and no more. A talking dog is still a talking dog even if there’s only one in existence. But that doesn’t mean there’s more than one, nor can that dog be discounted just because no other talking dogs could be found.

Third is to almost entierly rely on well-established principles. It’s not that the concensus is surely right, it’s that your studies aren’t rigorous enough to show otherwise.

Be skeptical of yourself.