Hypothesis driven products

Something we are doing now, in a very considered way.

How to implement hypothesis driven development

From a design perspective, tieing things to expected outcomes and behaviours that can (and will) be measured sure is a nice way of focusing decisions and building credibility.

The ‘ole “fulfil a client requirement listed in bullet-points” thing must die. No one, not even the client, is ever truly satisfied with that approach.

Analysis (and theory*)

I’ve been reading Sam Ladner’s Practical Ethnography. It’s full of gems that I plan to summarise, but Chapter 9 is especially good. Especially this bit: “It is theory, not method, that differentiates good research from poor research”.

Sam Ladner writes that theory will help guide you through your mess of notes and photos and observations.

For example,

  • Goffman’s theory of the presentation of the self
  • Butler’s notion that gender is a performance
  • Bourdieu’s idea of cultural capital.

So many new ideas to follow, and so much happiness that we can all come out of the closet and acknowledge that theory has practical applications when interpreting data.

A bit of Foucalt, anyone?

*An aside: Once I went and spoke to a designer research academic about perhaps moving back into academia.

“You keep talking about theory”, she said, at one point, in quite a sharp tone. “But it’s theories. There’s not just *one* theory“.

I think that was the moment I realised I could never return to academia because academia is fundamentally ridiculous.

How to split out a super large excel spreadsheet

I am working with an excel spreadsheet of about 132,000 rows.

Or, rather, I am not working with that excel spreadsheet because every time I try and manipulate the data excel crashes.

Manually splitting the spreadsheet into smaller, tinier spreadsheets was not something I was going to entertain.

Then I found this fantastic, super-awesome macro. How to split a file.  Go get it! Thank me later.


How we work together (research and design) or who do you think you are?

At work, we’ve been trialling a way of working in a product team where we have a dedicated researcher (me) as well as a dedicated designer (my design buddy).

I think the question we’re really working out the answer to at the moment is “just who do these research people think they are” and also “why exactly do we need them?”.

We’re still in the process of nutting that out.

These are the things I’m thinking at the moment, as a result of this trial 

1.  The researcher and the designer need to work side-by-side.

2. Researchers bring a particular mindset to a problem. In Practical Ethnography Sam Ladner describes this as the “emic mindset” (or, the perspective of the person using the product).

3. When it’s just you trying to cover both the research and the design (fairly usual practice when you’re working in a less established design team) switching between the two mindsets is hard. Even if you’re good at both, it’s two very different ways of working.

4. Not everyone is good at everything. If you are working alongside someone with complementary skills, your product will benefit.

5. The researcher doesn’t really just do research. The designer doesn’t really just do design. At a talk recently Leisa Reicheldt estimated that, for user researchers, about 30% of time spent is on the core practice, 70% of the time is spent talking (talking to each other, in meetings, presentations, discussions with the product team, the broader community of practice, that guy in the hallway … )

6. Good research helps frame the design problem in a way that can be quantified by the business, as well as enacted on by the designer.

7. People are worried that the researchers will become “too academic” and “critical of the designer’s choices” and generally be a pain in the bum. I have many thoughts about that, but most of them boil down to “hire people who can play nicely. Don’t hire arseholes. Not sure about the academic thing.”


Interesting techniques for rolling the user research onto the product roadmap.

How to build a product roadmap based on user research


Mapping the research insights
Mapping the research insights

I found the approach to mapping the relevant insights quite useful.

As well as this set of principles further down in the article:



My advice is to select your projects in that order:


Design studio chaos

Last week my designer-buddy and I ran a design studio with our team.

Some things about our team.

1. They’re located in another country.
2. We have about 3 hours of working day overlap.
3. We talk a lot on Slack and via online video.

The two of us, the designers, are based in Melbourne, along with the product manager. But everyone else is in the other office.

In the past we’ve run design studios with the team. We’ve used very narrow scenarios, we’ve had all the answers lined up to all of the questions and the output has been very similar screen designs or user flows.

This time we took a design scenario to the team that we didn’t have all the answers to, we were still iterating on ourselves. When they asked us, we put it back to them. “We want to know what you think. It’s OK. We won’t implement your ideas straight away. But we want your designs to help us see what we haven’t thought through”.

It backfired.

There were 30 minutes of questions, which we couldn’t answer. You could hear, through the skype call, the session slowly derailling.

The designs the team came up with were great. The greatest stuff, and the most useful for us, that had ever been produced from a design studio. It felt, from my perspective at least, it was a proper design studio, and not just an exercise in how people would lay out different elements on the screen.

But, from the team’s perspective, it was not a good experience. There were too many unanswered questions. There was too much uncertainty. And the uncertainty has now bled into other parts of the work we’ve been working on. Too many “open questions”. “Did we do any user research?”

What makes that even tricker is, because we’re not co-located, you kind of get hints that things are going south, but you’re not 100% and then when things do end up in the South Pole it takes another day or two to get everyone in the same meeting to talk things through.

I guess it was a fail, overall, if the goal was to engage the team in the “design” process.
But it was a real success from our point of view. Their work actively contributed to the way we were thinking of the problem.

I’m still on the fence about what the next steps should be here. I’m leaning towards not running a design studio like that again. And that’s partly due to the distance. It’s too hard to explain things and guide them, after a session, so that you can help people reflect and learn from a session that wasn’t quite what they expected.

It’s a shame, because I thought it was a good step forward. But it’s been a step back really, for everyone. Perhaps it’s best if we go back to more narrow and tightly-defined design studios that are really an exercise in helping them see what we have already seen, rather than an exercise in helping us see the begins of something better than we could have come up with by ourselves.

Right technique, wrong time

20 per cent of essential feedback is lost during user interviews

User interviews are a great way of gaining insights at the start of a design cycle.
User interviews help us understand how people think, their opinions, their attitudes and the way they are.

User interviews that take place in someone’s office, at their workplace, out on a client visit, in context, let us observe other information too.

I work on an application used by the construction industry. When we visit our users on site, it’s always cold, dusty and there’s alway an issue with the wifi.
It’s a world away from my office, where it’s warm, there’s beer taps in the kitchen (seriously) and me and most of my colleagues are tapped into wireless on at least 2 or 3 different devices.

But using interviews to test an actual design is not quite the right technique. Once you’ve taken in all of the information you can, through interviews and other methods, added this information into documents, onto sticky notes, into your design artefacts, and start designing your prototype, then you need to test the way people behave, not the way they think.

These two things are very different. I’ll give you an example:

If you interviewed me about my attitudes to exercise, and how important I think it is, and how much I prioritise it, I’m going to tell you “yes, yes, very important, very prioritised”.
If you come back and show me your prototype, I’m going to say “great, looks cool, nice colours, I don’t like the green though” and you’ll make those changes and release and you’d be fairly confident I’m going to be a user of your product.
Except if you had asked me to use a prototype of your application for a week, and tracked my behaviour with the prototype and how much I used it when I was exercising, or even how much I exercise full-stop, then you’re going to find out that I am a: a little bit not-honest with myself about how much I exercise, I don’t really use tracking tools when I do exercise and I didn’t want to hurt your feelings because you’ve worked hard and done a good job with your product. I’m nowhere near a target market for your app.

So I dunno about the article about asking people’s opinions about a new design, and whether online or offline testing is going to make much of a difference (except that people will be nicer to your face).