Core77

[HarveyJames]

Reading this thread, I picked up on two points that appear to be opposition to each other, regarding the use of surveys in user research: viewtopic.php?t=23462&

tarngerine says that surveys should be for quantitative data, whilst iab says they can be for qualitative data. Until tarngerine's comments, my own thoughts were that surveys could be useful for gathering various comments and views about a problem; unless a pool of respondents was large enough and sampled without bias, the data might not be significant enough to use to inform choices.

I'm currently due to start my first, full project (from start to prototype). It's not a big project. It's one of three problem spaces that we've been offered. I chose "holiday travel with children under five". Rather than make up a problem, I feel more comfortable starting from scratch, so I want to find out more about just what is involved. I've vaguely decided to:
i) interview two or three parents (perhaps including a nursery nurse) to just talk through their experience of travelling with young kids
ii) search for and look through online discussions and mentions of travelling with kids
iii) produce a survey to solicit comments from a broader group of people (I know more people online than I do IRL, I think)

But I'm wondering if the survey is really necessary at this point, whilst I'm still in search of a set of potential problems? Particularly if, in interviews, I can drill down on particular comments that might signpost to significant details. Would the survey be better off used once I've identified a potential problem and want to really define its boundaries? If that's the case, presumably it would be useful in setting the brief and user requirements/ constraints?

Any input... gratefully received

[iab]

i) interview two or three parents (perhaps including a nursery nurse) to just talk through their experience of travelling with young kids


But I'm wondering if the survey is really necessary at this point, whilst I'm still in search of a set of potential problems? Particularly if, in interviews, I can drill down on particular comments that might signpost to significant details.

Since I was quoted, I think I will respond.

By all means, continue with the interviews. But let me ask this, at the same time you are doing those interviews, do you think other people could answer the exact same questions with an online survey?

Which is my point. Every research methodology has its good and bad attributes. The key is to recognize the strengths and use them to fulfill your objectives.

I have no idea if I would recommend having or not having a survey without knowing your objectives. And while interviews can have great depth, they have no breadth. Surveys can provide the breadth and can be very valuable to confirm the data you obtained from your interviews.

And for the record, a bad interviewer can absolutely ruin the data received from the interview. Just as a poorly written survey can ruin its data.

[Generatewhatsnext]

Just to add to the conversation - I somewhat agree with IAB - the time to conduct live interviews might be better spent conducting observational research instead. Use an online survey to get raw data in the meantime. Granted, your topic is time-frame sensitive but, in our experience, seeing actual events while they happen has been the most useful, unbiased, detail-rich data gathering tool.

[HarveyJames]

Thanks for your replies.

I conducted a couple of interviews today. They were slightly awkward (actually, they were horribly awkward and a bit stilted) and I felt like I should have better prepared for them. I purposely set out to do an unstructured interview, but I think a structured set of questions might have been more useful, diverging where necessary. I probably came away with a 'whole' idea of how some parents travel with kids, but it was hard to identify specific issues for a physical product to address. I did pick out common activities and needs (if only from two interviews so far). I have one other interview lined up for tomorrow, albeit an email exchange.

I set aside some time tomorrow to go out and just watch people at a train station, bus stops and a shopping area/ public area. I'm hoping that will give me some in-situ observations to make and question.

What I wanted to avoid was just doing user surveys because that's how all of the other students have started the project. If that's still a good way to start, that's fine - I just want to have a good, justified reason for choosing it, rather than the implicit "that's what everyone else is doing, so maybe I should, too!"

Generatewhatsnext - you mentioned about doing online surveys to get some raw data: is the quickest and cheapest way of doing this simply a case of (almost) begging others to take part, via Facebook, forums etc? Fewer questions would presumably get more results, so would it be reasonable to jump to questions about describing problems they experience when travelling with kids?

[iab]

seeing actual events while they happen has been the most useful, unbiased, detail-rich data gathering tool.

So this person is going to join in on a holiday trip with children under five??? You don't think having a stranger along wouldn't skew the results?

I purposely set out to do an unstructured interview

Whoever taught you to this should have their teaching license revoked. A clear objective is a necessity, not an option. To create that objective, have you narrowed your scope?

Holiday travel with children under five.
By plane, train or automobile?
With family, friends or other?
What is the duration, 1 day, 1 weekend, 1 week?
Infant or toddler?
How many children?
First time traveling with child?
Staying in home or hotel?

That is what comes to mind without any thought. With an hour, I'm sure 100 more questions would come to mind. All good interviews are structured and well planned. They may not appear so, but they are well researched prior to the interview. Winging it will produce the bad data I mentioned.

Have an objective. Have someone with experience review your objective. Determine your target respondent. Have someone with experience review your choices. Create a script. Have someone with experience review your script. If it all possible, do dry runs of your script with that someone with experience. Don't use the person who thought an unstructured interview was a good idea.

[iab]

And who is your client?

Online travel site?
Airline?
Bus company?
Car manufacturer?
Luggage manufacturer?
Stroller manufacturer?
Adult clothing manufacturer?
Baby clothing manufacturer?
Diaper bag manufacturer?
Tire manufacturer?

Don't laugh, think Michelin with a baby in their tire.

[Generatewhatsnext]

seeing actual events while they happen has been the most useful, unbiased, detail-rich data gathering tool.

So this person is going to join in on a holiday trip with children under five??? You don't think having a stranger along wouldn't skew the results?

I purposely set out to do an unstructured interview

IAB - It is not impossible to watch parents pack a car for a trip or pack suitcases and gather trip 'stuff' for flights, to observe and gather comments as a family loads up for the departure - or to meet a family at the airport as they depart or return. A quick phone call during a trip or meeting a visiting family in person during a trip. Properly vetted candidates (or family & friends if on a low budget) will yield a wealth of data.

HARVEYJAMES - An unstructured approach will throw you into a tailspin when you're in a time-limited face to face situation with research subjects you've never met previously. Stick to a well filtered script unless an ah-hah moment presents itself.

[yo]

I'm not usually a fan of online surveys as they feel like tests, and often times when people take things that feel like tests they answer with what they perceive is the right answer.... however, when I want to prove a point to people who like numbers I will use a very carefully worded and structured online survey that is designed to yield a particular insight.... as the old saying goes "98% of all statistics are made up"

I prefer observational research with a loose script loaded with suggested questions and prompts (show me how you play a DVD, now show me how your wife plays a DVD, mow your lawn, whatever....) to keep things going in the awkward moments. I find that if an observation session goes past 3 hours it will be weird, especially in an in-home situation. It is a little different if you are observing a professional (FedEx guy, MailMan, Professional Basketball Player, Surgeon) those can run longer without you just getting in the way. Do 10-15 2 hour session and then abstract out the commonalities and differences between them to find trends and outliers. Look for user created hacks and work arounds to identify key problems that could be solved. This is good input to inspire design but not to chain it. It should be a spring board not a set of boundaries.

[HarveyJames]

The idea to do an unstructured interview came from reading some of the dSchool materials, amongst various other materials. Because there isn't a specified problem or client, I felt that I wanted to know more about the space in order to uncover possible problems to look in to, so I thought an interview to elicit comments and experiences might help with that. I had an idea of the questions I wanted to ask and the detail I wanted to uncover, but I thought that not sticking to a pre-determined set of questions would allow me to probe further if an interviewee mentions something of interest (and we did diverge now and again to discuss particular points of interest).

I spent some further time since the interviews observing at a train station bus stop and in a public space (popular with tourists and families), to see how parents and children cope with getting around. It wasn't particularly interesting and I realised that perhaps the problem with my choice is that it's just one snapshot of a whole journey or experience. Still, I saw how people struggled with juggling kids and luggage, and stressed parents trying to keep kids happy. I realise there are various other situations that could be chosen and I'll also be going out on a short trip with an adult and children in her care. I'd like to do what Yo mentions, but I think the time constraints don't really allow me to do that number of observations. I'm also starting to be mindful that I should try to move towards developing ideas.

So I think at this point, I feel that I know a little bit more about how parents travel with kids on holiday. I've identified some issues and needs - keeping kids occupied, handling kids and luggage, finding toilets, storing and carrying items - that I can look at further, narrow my scope and start to curate a brief and sets of requirements.

So once my brief and requirements are set, I might use a survey to help me to sculpt choices about my ideas? Would that seem appropriate? Or not at all? Is a survey a fundamental part of design research (ie, almost always used)?

I realise these might sound like silly statements to more experienced designers, but I am learning and failing, and trying to learn from my fails.

[iab]

IAB - It is not impossible to watch parents pack a car for a trip or pack suitcases and gather trip 'stuff' for flights, to observe and gather comments as a family loads up for the departure - or to meet a family at the airport as they depart or return. A quick phone call during a trip or meeting a visiting family in person during a trip. Properly vetted candidates (or family & friends if on a low budget) will yield a wealth of data.

In my experience, placing an observer into a stressful non-routine event will yield questionable results. In the last five years, I have spent probably months in the ICU (for work). Highly stressful work, hundreds of alarms and people literally dying every day. The difference is for those who work there, it is, for a lack of a better word, routine. And even in that case, it takes at least a day for them to ignore me. Hawthorne effect, Heisenberg effect, whatever you want to call it. That first day of observation rarely, if ever, offers any good nuggets.

But this problem is travel with children under 5. Hardly a routine, just as stressful and is a very short duration. Adding another person will add another level of stress. The additional person will never disappear and I think is likely going to have a significant effect on the experience.

I do like the OP's notion of going to a train or bus terminal and observing. I didn't realize this project was so exploratory. I wouldn't recommend either an interview or survey in this case. This is literally throwing spaghetti at the wall. Once the OP sees what sticks, that will provide the direction for any future interview or survey.

And I'd like to read the paper on the "unstructured" interview. If there is a link, please post it.

[HarveyJames]

I went back to my bookmarks. The Designing with People site was a seed site I used cursorily (http://designingwithpeople.rca.ac.uk/methods/interview), whilst the dSchool stuff helped with the structure of the project and methods (http://dschool.stanford.edu/wp-content/ ... 2010v6.pdf). Most of my other reading material is textbooks.

It would probably be fair to say the interviews I did were semi-unstructured, rather than completely unstructured. I had questions and types of questions prepared that I wanted to ask, but I didn't set them out in order because of the conversational style I wanted to attempt (it didn't work as well as I wanted it to).

I'm working on an empathy map today, to draw together what I found and generate the insights from that, which I hope will drive some specific problems to go at.

[yo]

IAB - It is not impossible to watch parents pack a car for a trip or pack suitcases and gather trip 'stuff' for flights, to observe and gather comments as a family loads up for the departure - or to meet a family at the airport as they depart or return. A quick phone call during a trip or meeting a visiting family in person during a trip. Properly vetted candidates (or family & friends if on a low budget) will yield a wealth of data.

In my experience, placing an observer into a stressful non-routine event will yield questionable results. In the last five years, I have spent probably months in the ICU (for work). Highly stressful work, hundreds of alarms and people literally dying every day. The difference is for those who work there, it is, for a lack of a better word, routine. And even in that case, it takes at least a day for them to ignore me. Hawthorne effect, Heisenberg effect, whatever you want to call it. That first day of observation rarely, if ever, offers any good nuggets.

But this problem is travel with children under 5. Hardly a routine, just as stressful and is a very short duration. Adding another person will add another level of stress. The additional person will never disappear and I think is likely going to have a significant effect on the experience.

Good point, I agree. Even though the results would be skewed, do you think it is better than nothing?

I'm also a big fan of more casual observational research. We used to go to local basketball courts and just watch were stresses were on product, how things were abused, did they show up wearing their shoes, or did they change into them? Simple things. The airport was another good spot, we used to do shoe counts to see what percentage of each brand we saw, how new were they....now designing headphones we do similar things when in urban areas, busses, air travel. Sometimes we do retail intercepts, talking with people right after they purchased something... stuff like that.

[iab]

Good point, I agree. Even though the results would be skewed, do you think it is better than nothing?

Depends. How am I managing my time?

As I wrote, I did not realize at the time of the first post that this project is completely on the exploratory end of the spectrum. Knowing that now, again, I would never recommend an interview or survey. You need direction for those methodologies (hence the term directional research).

For where the OP is, I would recommend secondary research and observation. But as you put it, I would go with the "casual observational research". I would not recruit a respondent and follow them for the day. As you and the OP suggested, going to a train or bus or airline terminal where the subjects are unaware of being observed would be the best for time management. You will get the most bang for your buck.

Once the OP gets a direction, e.g. pre-planning activities with children under 5 through a travel site, the interview and survey can come into play. I still think recruiting the "right" person for one-on-one observation is more likely to fail than succeed and would be a huge time sink and doing nothing in that case would be better.

And while the Designing with People site claims you can have an unstructured interview, the Stanford site says,

"While we always must allow room for the spontaneous, blissful serendipity of a user-guided conversation, we should never abdicate our responsibility to prepare for interviews"

I know a handful of people who could possibly pull of an unstructured interview, but they have been doing research for 20+ years and are some the best at it. I have been doing research for 20+ years and I certainly know I could never pull it off well, so I prepare. Having a novice try an unstructured interview would in no doubt result in poor data.

[yo]

Reminds me of the old Picasso story;

Woman bumps into Picasso on the street and says "OMG its Picasso! Could you draw a portrait of me?"
He replies "But of course" and casually scribbles a few lines, shows her, and then stuffs the sketch into his man purse.
The woman brazenly asks, "Can I have it?"
To which Picasso replies, "Indeed, for $30,000."
Shocked the woman responds "What? But it only took you a few seconds!"
And Picasso astutely points out "On the contrary Madam, that drawing took me my entire life to create!" and he walked away.

The story is a farce I'm sure, but the point is that those 20 year veterans who can pull off an unstructured interview are drawing on instincts and experiences honed over 20 years of conducting structured interviews.

[dosullivan]

It would probably be fair to say the interviews I did were semi-unstructured, rather than completely unstructured. I had questions and types of questions prepared that I wanted to ask, but I didn't set them out in order because of the conversational style I wanted to attempt (it didn't work as well as I wanted it to).

Sounds perfect to me, that's the beauty of interviews - you can go off-script and explore the domain organically with your interviewee. The questions provide a bit of a safety net for the interviewer and interviewee, but often the best bits are a bit of a surprise (i.e. you might not have known to ask that question).
_

Damon - web | tweets