Yeah I see what you mean. I wanted to include other examples in the post but just struggled to come up with them and/or articulate them. Plus I didn’t want to spent too much time on this post. If you or anyone else have any though, I’d love to hear them!
You talk about user research at the start of the post, but then you transition into talking about thought-experimentation. I note that these are not only not the same thing, but not even the same sort of thing!
Agreed. Like with the examples, this really is just a failure on my part as the writer for not clarifying and talking about this. I did in a previous version of the post, but had some trouble fitting it into this version.
What I was going for is something like this. In theory it would be great if you were able to do proper user research. But in practice that would involve an investment of time that might not be reasonable. And so as an alternative, the thought experiment thing is another option that is useful.
Attempting to do user testing as a thought experiment, instead of doing it for real, would be useless. I can tell you, from personal experience, that the value of user testing comes from the fact that users can, and will, surprise you. You can sit back and imagine users interacting with your system as much as you like, and you will never be able to imagine the sorts of things that actual users will actually do when they try to use it. Never.
This strikes me as thinking in absolutes. You won’t be able to imagine all of the things that users will actually do, but you’ll still be able to imagine some of them. And that portion of things you are able to imagine serves some use. Right?
Note that this is especially so given that what we imagine it is like to work with one or another sort of code, is affected quite strongly by our existing views on the design patterns (or other properties of a codebase) in question.
I agree. This is something to be aware of, and I think it would have been good for me to talk about in the post. But to be clear, I think that these biases are something that we can manage. “I recognize that I am biased against DRY code and therefore shift my beliefs accordingly.”
The third user
Yeah, I agree with all of this. And I did think a bit about it when I was writing the post. It didn’t feel like a path worth going down though:
It seems a little tangential and distracting to the core point of my post. I think my point applies to many of the interpretations of “easy to work with”, and talking about what makes code easy to work with is a large conversation.
I was sorta hoping/envisioning the word “easy” in “easy to work with” would do enough of the heavy lifting. If you are working in a codebase for five years and the code makes it easy to onboard but difficult to work with in the long run, then in aggregate, it’s not actually easy to work with since a good proportion of the time is spent as the Third User.
The words weren’t coming to me. I don’t want to shy away from acknowledging this. A writer with more skill and/or time would probably be able to incorporate the third user stuff into the post in such a way where it is a net positive.
This strikes me as thinking in absolutes. You won’t be able to imagine all of the things that users will actually do, but you’ll still be able to imagine some of them. And that portion of things you are able to imagine serves some use. Right?
It’s not impossible that imagining what users will do can serve some use. Of course, to the extent that such an exercise is useful at all, it’s best done systematically. This is the point of such usability evaluation techniques as the cognitive walkthrough, heuristic analysis, etc.
The danger—and it is a grave danger, which has been the ruin of many a project—is that (a) what you imagine users will do is not actually something they will do, and (b) what users will actually do is not something you imagine they will do. But you will think that you have learned something; and so your thought experiment will mislead you, and leave you worse off than before.
It is very easy to “imagine” yourself into creating something that nobody (not even you!) will want to use.
There is really no substitute for actual user testing—not even a partial one.
One question is whether 3 > 2. A separate question is 2 > 1. I am making the/a point in this post that 2 > 1.[2]
But since 3 > 2, shouldn’t we always be doing 3 instead of 2, making the point that 2 > 1 moot? I don’t think so.
3 takes more time than 2 and is not always practical. A given design is composed of lots of little decisions that are made[3], and there isn’t time to do proper user research on each of these component decisions. And so in practice, the status quo is that people currently do not do user research on all of these component decisions.
In which case, I think the question is whether 2 > 1. Or rather, as I mention in my second footnote to this comment, whether 2 can be used in addition to 1, adding value. I think it can and often does. Furthermore, I think that this point is underutilized/underappreciated/under-understood.
And there’s probably some sort of more general point to be made here that I’m struggling to think about and articulate that extends past usability and software design.
Although you seem to feel more strongly about how much 3 > 2 than I do. To make up numbers, how strongly I feel about it’s importance, I’d say is like a 7⁄10, whereas you seem to be more like a 9.5/10 or 10⁄10.
Yeah I see what you mean. I wanted to include other examples in the post but just struggled to come up with them and/or articulate them. Plus I didn’t want to spent too much time on this post. If you or anyone else have any though, I’d love to hear them!
Agreed. Like with the examples, this really is just a failure on my part as the writer for not clarifying and talking about this. I did in a previous version of the post, but had some trouble fitting it into this version.
What I was going for is something like this. In theory it would be great if you were able to do proper user research. But in practice that would involve an investment of time that might not be reasonable. And so as an alternative, the thought experiment thing is another option that is useful.
This strikes me as thinking in absolutes. You won’t be able to imagine all of the things that users will actually do, but you’ll still be able to imagine some of them. And that portion of things you are able to imagine serves some use. Right?
I agree. This is something to be aware of, and I think it would have been good for me to talk about in the post. But to be clear, I think that these biases are something that we can manage. “I recognize that I am biased against DRY code and therefore shift my beliefs accordingly.”
Yeah, I agree with all of this. And I did think a bit about it when I was writing the post. It didn’t feel like a path worth going down though:
It seems a little tangential and distracting to the core point of my post. I think my point applies to many of the interpretations of “easy to work with”, and talking about what makes code easy to work with is a large conversation.
I was sorta hoping/envisioning the word “easy” in “easy to work with” would do enough of the heavy lifting. If you are working in a codebase for five years and the code makes it easy to onboard but difficult to work with in the long run, then in aggregate, it’s not actually easy to work with since a good proportion of the time is spent as the Third User.
The words weren’t coming to me. I don’t want to shy away from acknowledging this. A writer with more skill and/or time would probably be able to incorporate the third user stuff into the post in such a way where it is a net positive.
It’s not impossible that imagining what users will do can serve some use. Of course, to the extent that such an exercise is useful at all, it’s best done systematically. This is the point of such usability evaluation techniques as the cognitive walkthrough, heuristic analysis, etc.
The danger—and it is a grave danger, which has been the ruin of many a project—is that (a) what you imagine users will do is not actually something they will do, and (b) what users will actually do is not something you imagine they will do. But you will think that you have learned something; and so your thought experiment will mislead you, and leave you worse off than before.
It is very easy to “imagine” yourself into creating something that nobody (not even you!) will want to use.
There is really no substitute for actual user testing—not even a partial one.
I have a feeling that we mostly agree with each other and are thinking about two different questions. Consider three options:
Think about whether a design matches various heuristics.
Think concretely about whether a design will actually be understandable to users (user research thought experiment).
Do user research to find out whether a design will actually be understandable to users (actual user research).
I think we are in agreement that 3 > 2 > 1.[1]
One question is whether 3 > 2. A separate question is 2 > 1. I am making the/a point in this post that 2 > 1.[2]
But since 3 > 2, shouldn’t we always be doing 3 instead of 2, making the point that 2 > 1 moot? I don’t think so.
3 takes more time than 2 and is not always practical. A given design is composed of lots of little decisions that are made[3], and there isn’t time to do proper user research on each of these component decisions. And so in practice, the status quo is that people currently do not do user research on all of these component decisions.
In which case, I think the question is whether 2 > 1. Or rather, as I mention in my second footnote to this comment, whether 2 can be used in addition to 1, adding value. I think it can and often does. Furthermore, I think that this point is underutilized/underappreciated/under-understood.
And there’s probably some sort of more general point to be made here that I’m struggling to think about and articulate that extends past usability and software design.
Although you seem to feel more strongly about how much 3 > 2 than I do. To make up numbers, how strongly I feel about it’s importance, I’d say is like a 7⁄10, whereas you seem to be more like a 9.5/10 or 10⁄10.
Well, they all work in conjunction with each other, but the idea of 3 > 2 > 1 still gestures at what I am trying to get at, I hope.
I’m having trouble articulating what I mean by this. Maybe you get it?
First, note that 1 is properly not just heuristic analysis, but also all the other formal and semi-formal methods of evaluation. That said:
1 > 2. (Because 1 is just 2 but systematized, with checklists, corrections for common biases, non-obvious considerations, etc.)
3 <> 1. (That is: 3 is incomparable with 1.) (As you say, a design is composed of many decisions, too many to effectively user-test, etc.)
3+1 > any of { 1, 2, 3 }. (Note that this constitutes the standard prescription for UX design practice.)