Research Resources

I read a survey in the furniture forum where the writer had 10 open-ended questions as the survey. This type of question is great when the respondent is being recorded in an interview or focus group, but it is obvious he/she doesn’t realize that is unacceptable for a written questionnaire. I have several years of experience of putting together research programs and know what to do but is there a resource (book, web, etc.) for people who don’t know what’s best? Personally, I don’t know of any. I would be happy to write some general guidelines if anyone is interested.

On a side note, I have never liked the word respondent. Does anyone have a better word?

No takers? Well, as the sketching war takes place overhead, I feel it is as important to do research as well as sketching, CAD, etc. Some of the things I have seen on this forum and other places that try and pass as research are just sad and unprofessional. I have some time, I also want to clarify my thoughts, and so I will attempt to give a research tutorial.

My only qualifier is I have never conducted academic research; I have only observed and read reports. My experience is solely with corporations, big and small, with time and money being the two biggest drivers. Also, as with anything, different areas of research can be turned up or down, I am going to try and be relatively comprehensive. As with any specialty, there is lingo that some people use and others don’t. Please feel free to ask questions or put in your own .02.

There are many ways to approach this, but I will start by lumping research into two categories – primary and secondary. Primary research is directly questioning the end users, influencers and decision makers. Secondary research is using other resources to get answers to your questions. Generally, primary is excellent for specific questions and secondary is for broader information. Primary is much more interesting but secondary is far cheaper to conduct. I will discuss both, but I will start with secondary because it is easier to conduct.

As I said before, secondary research is excellent for discovering general information but can be used more specific information about a product, its market and importantly, the competition. Below is a list of my favorite resources for secondary research.
● The company you are working for
● The Internet
● Publications (trade and consumer)
● Collateral materials (brochures, sales sheets, etc.)
● Trade shows
● Vendors

I think most important to that list is the company producing the product. They should be able to give you (it always seems after a lot of calls) their info to date about the product including business plans, marketing plans, market research, competitive research, customer satisfaction data and their collateral materials. If you are able to read all of the material without falling asleep again and again, you will now be able to have an incredibly long and tedious meeting with the key players from nearly all of the company departments. You will want people form marketing, engineering, design, R&D, sales, manufacturing, distribution and customer support. If the company is small enough, you will also need the corporate officers.

The true goal of this meeting is to become as “smart” as everyone with regards to the product and its market. I will get into questions later, but needless to say this meeting will take anywhere form 2-4 hours to conduct and will be very painful. There is nothing like locking 20 people in a poorly ventilated conference room drinking coffee, eating crappy deli and sweating. I recommend having all of the people together, you will discover more that way. You can break the groups apart (nobody has time for a four hour meeting). Put marketing, sales, and corporate officers in one group for the big picture and design, engineering, R&D, manufacturing & customer support for the minutia. With the help from a key product manager, you could probably create a survey to distribute to the different areas. I would only recommend that if people were unable to attend the meeting. Once the discussion starts, it can lead in many directions that can’t be captured with a survey. It’s your job to rein it in when things diverge too much.

Please start the meeting by sucking up by thanking everyone and telling them how their input is important. Also, don’t forget to tell them why they are there. As for the questions, start broad and narrow the focus. Question the company’s, division’s, product family’s and product’s position in the market. What values do you want to be identified by? Who are the end users, influencers and decision makers (this can be one person or three different people)? What are their needs and concerns with regards to the product, company, etc.? Who are your competitors? What do they offer? What are key features and benefits of the product, company, etc.? Are there any specifications and requirements? There are many ways to ask the questions, I’ll leave that up to you. You don’t have to ask a lot of questions. As I said before, the discussion will naturally cover almost everything. Also, have this presentation PROOFED. There is nothing worse than people getting hung up on a typo. It is very embarrassing.

The second most important resource for secondary research is the Internet. Hopefully you used it before the meeting to answer the questions above. People will appreciate you have done your homework and the meeting is more of a “review” than a question and answer. Also, if the “answers” are visible it is easier to get the discussion started.

By using the company, internet and the other resources, you should be the go to guy about the product. I can’t emphasize enough how ALL of this information is important when developing the product.

That’s enough tonight, I am tired. The next installment will discuss primary research – woohoo (now there’s a combination of words rarely used together).

Wow, this is one hot discussion. The air is thick with anticipation for, dare I say, a tutorial for primary research. This is probably due to the writing style being equivalent to plain white toast. Here goes:

First, I forgot to mention quantitative and qualitative research. In developing products I have NEVER seen a company pony up to do qualitative research up front. I do mostly medical devices and qualitative is done for validation for the FDA (510K, pharma trials, etc.). You can also use third party qualitative research reports in the development process. The government (census bureau and other stuff) or academia usually has some good stuff for sniping. Otherwise, this discussion will be focused on quantitative research.

Hopefully you have determined the research goal and objectives when you had the long meeting to kick off the project. If not, write it down and get buy in. The goal and objectives should tell you the scope and methodology of the research. The scope will tell you who you want – end users, influencers and/or decision makers and how many of them. The total number of respondents is directly related to your budget and methodology. Generally, you need an honorarium for the respondents. Just as you get paid for your time, so should these people. The most expensive is if you are doing one-on-one interviews with physicians who are the leaders in their respective fields. You pay those guys $200/hour and pray they don’t laugh in your face because you are low balling them (generally, they will do the research only if they are interested in the subject). The cheapest is doing an online survey with general consumers. You offer them a chance to win the electronic device de jour and you are out only a few bucks or if it is short enough, they may do it for free. You don’t have to offer an honorarium – it just takes longer to hit your target number of respondents. As the saying goes; better, faster, cheaper – pick two. I will get into more specifics with budget when I write individual methodologies.

Sorry to go on a tangent, but there are a couple more general things before I focus on methodologies. Another way to categorize research is by the phase of development the product is entering. This way research can be called exploratory, directional and confirmatory. Think of a funnel with exploratory on top, directional in the middle, confirmatory on the bottom and what comes out is the product (or product strategy, depending on your original goal). Exploratory is used for gathering general info about the market. For example, when Apple decided to start development on the ipod, it would have done exploratory research. They looked at the growing mp3 market, benchmarks of portable music players (e.g. Walkman) and determined combining the two was a good idea. I have no idea who came up with the original thought but it could have come from R&D, engineering, marketing or design and some exec gave it some budget. 20 years ago you would assume it came from the skunkworks, but I rarely see companies use that methodology. Anyway, Apple determined the ipod was a good idea on paper. They would then use directional research in defining the overall requirements of the product. Things like how big is it, how much music can it hold, how long will the batteries last, interface, etc. You can also use directional research to assist in developing the general look and feel. I am assuming all of the requirements are assessed to Apple’s core positioning strategy of innovation. Finally, after putting specifications to the requirements, Apple would have used confirmatory research to finalize the device. Generally this is done with respondents evaluating only the final few concepts.

Also, I put respondents into four categories; thought leaders, end users, decision makers and influencers. As I said before, this can be one person or four different people (I forgot thought leaders on my previous post). You can subcategorize these people into other groups like early adaptors, laggards, etc. It is important to discover who they are and the relationships between these individuals with your research. It will create a better understanding of the potential impact of your product. For examples, a thought leader for the ipod may be a band or a DJ like Howard Stern or for old folks like me, Jim Coates. End users are pretty obvious. Decision makers are the ones to pull the trigger, it could be the parents. Influencers are more like unimportant thought leaders, generally your peers.

As for methodologies, I will keep it simple. You can either look at the respondents or you can listen to them (I put reading what they have written a category of listening). I suppose you could touch, smell or taste them, but that would probably fall into quantitative research which, as I stated before, is very rare in product development. So you have three choices; you watch or listen or, most preferred, a combination of both.

Since I consider myself as an amateur voyeur, I will start with watching respondents. Watching is always better than listening (unless you get some weird pleasure at a tennis match). Watching, of course is much more time intensive and cost more but that is something between you and the budget. It is also best if you are watching the respondent in their environment as opposed to a focus group or third party location. You all should know this is called ethnography. When I didn’t know any better, we called them site visits. Please have a plan before you arrive. I know from personal experience if you just show up and say, “I am here to watch.” you will be hauled away in cuffs. Once again, you have to do your homework. All this is essentially is a workflow analysis. You want to watch the respondent’s activities surrounding the use of the product. You should have a very good idea what they do before you get there. This helps in how much of their activities you want to observe and increase the intelligence of any questions you may have. Generally, big picture activities don’t lead to innovation, it is in the details that rarely get mentioned and are second nature to the respondent. You also want to carefully record and confirm the workflow. This will enable you to understand how your product will affect their day-to-day activities. I don’t want to tell anyone how to design, but people are resistant to change and need a good reason to do so. By understanding the respondent’s workflow and its drivers you should be able to focus the direction of your concepts. I usually put each of the respondent’s activities on a piece of paper, stick them to a wall and do my best to understand the issues with regards to the workflow.

As for the physical act of observing, I am a believer of quantum mechanics and the Heisenberg Uncertainty Principle. If a respondent knows they are being watched, they will act different. Once again, this is another reason to do your homework. If the respondent doesn’t have to hold your hand through the process they will become more comfortable and become more natural with their actions. There is a line to cross – don’t be a know-it-all, if the respondent dislikes you, you might as well leave and you will not get good data. Some researchers use recording devices when observing the respondents. Sometimes a camera or voice recorder is less intrusive as a person. The recorded information can be played repeatedly for a larger group of people for further analysis. A couple of things to remember: Always get permission and remind (at the start of the session) the respondent they are being recorded. As you probably already know, people will be put off by the camera. This will go away after only a little time especially since cameras are now very discreet. You can also set up permanent cameras. This is a purely observational methodology and there is no interaction between the respondent and researcher. It is also expensive and very boring because of the time involved to review the tapes. Some places will refuse the use of cameras. This is especially true with medical devices due to laws like HIPPA. You can choose to interview the respondent while doing your research. Once again, this is a little more intrusive and can affect their behavior but you can get a better understanding of their process if they talk about it. I will discuss interview techniques a little later.

Focus groups are another observational methodology. I have never conducted a focus group in a respondent’s natural environment. I would find it interesting if any of you have done that “twist” on a focus group. I have done groups ranging from just asking questions (no props) to trying to approximate their natural environment. Usually there are pictures and models to show the respondents. I have also never done a focus group without some sort of an interview. I would again be interested in stories about purely observational focus groups. Focus groups have gotten a bad rap usually because there are done poorly. You want the group to interact and have them contribute to the process; they will be much more engaging. You can’t just throw an idea in front of them and ask “What do you think?” or “How would you rate this?” You don’t need them to create the product, that is your job, but you do want them to feel vested in the process with the questions you ask. How to do that is proprietary and I don’t want give away the keys. Having the respondents create the product is a method used. Its kind of what you may have seen on The Apprentice. Have an IDer sketch or CAD their ideas. I have seen this work with limited success but there are better ways for a lively group.

If that wasn’t dull enough, next are some general guidelines for a focus group. Ideally, you want 6-8 respondents per group, anything over 12 is chaos. Keep the session under 2 hours and have a 5 minute break halfway through. Give them food and drink. Do your homework. Develop a script. Get signoff for the script from the client. Stick to the script. Don’t let the client bully you into adding questions to extend past the 2 hours. Have one of your people in the room leave every 20-30 minutes to get additional questions from the viewing room. Answer a respondents question with a question – e.g., Does this come with a flux-capacitor? You reply – Would you like it to come with a flux capacitor? Use visual aids – sketches, renderings, models, prototypes. Have the visual aids appropriate to the research stage (don’t have functioning prototypes if you are telling the respondents you need their assistance in developing the product). Don’t give a PowerPoint. Have the respondents write their answer before opening the discussion. You will always get 1-2 people who want to dominate the discussion. Don’t let 1-2 people dominate the discussion. Keep the respondents peers, don’t mix different groups. An example is not to mix physicians with anyone else, they think they’re Gods and will dismiss others very quickly. Have a good mix of questions – open ended, rating, prioritizing, multiple choice, etc. Thank the respondents (you may use them later). Don’t use a respondent more than once every 6 months.

That’s all for now. As always, keep those questions and comments coming. Next I will get into interviewing, questionnaires, research tools, recruiting, analysis and other fun and wacky things in the world of research.

At least I am not being heckled. The UX guys complain about not receiving the love like shoes. I aspire to be as hot as UX. I think this thread is equivalent to Fat Bastard saying he’s dead sexy.

Couple of things I failed to mention about structuring a focus group. The following is a general outline of activities. You should be able to put a time limit to each part. You can go a maximum of 10-15 minutes over the time allotted for the session.
● Introduction and welcome
● Workshop guidelines
● Recording and confidentiality
● Purpose
● Respondent introductions
● Agenda
● Respondent questions
● Warm-up exercise
● Concept(s) review or other research focus
● Wrap-up and thank you
Also, the true key to a good focus group is the facilitator.

On to the interview. They can be held anywhere or anytime, formal or informal. I would say the majority of upstream research is marketing “interviewing” customers over dinner. Unfortunately, most marketing/product managers blow this opportunity. They keep it too casual and don’t probe (I hate that word, I can’t believe they branded a car with it) enough. I have met the grand total of two product mangers who actually write a script before such dinners. The conversation should be kept casual but you should be able to control its direction.

The interview, IMO, during ethnography should also scripted and casual. As I said before, do your homework and have a very good idea what you are going to see. This way you can ask more valuable questions instead of wasting time with the easy stuff. You observe the what, where, when and how. The questions are really for the why, the stuff that goes unsaid. Ethnography is about the observation; the interview is for fill in. Since you are verbally questioning the respondent, open-ended questions are preferred. If you have a written questionnaire (for rating things, evaluating concepts, etc.), I prefer to do it at the end of the session. You can mix it in throughout the session but then you are breaking their natural workflow process which is really against what you are trying to accomplish. Once again, this is by far the best way to discover the needs and concerns of the respondent. They are in their element and are talking about themselves, try to be as least disruptive as possible. Also, since they are comfortable, I have gone up to 2 hours over the time scheduled for the session.

By my definition, interviews also take place during focus groups. Obviously these are well scripted but have openings for the facilitator to improvise and to have questions brought in from the back room. Again, open-ended questions are best for verbal answers. Rating, prioritizing, true/false, multiple-choice, matching and short answer (1 to 3 words) questions are best in a written format. Voting or multi-voting on something can go either way. Also, as a reminder, do the written questions before opening the floor to discussion.

Most common are the one-on-one interviews. These can take place anywhere; at their location, at your location, at a third-part location (tradeshows are great). They can be done in person, over the phone or over the Internet (I will call a chat an interview, not to be confused with an Internet questionnaire). Which reminds me, I totally forgot to mention blogs and customer reviews on the Internet. These are excellent resources for secondary research. Interviews can be videoed or voice recorded. Knowing that you will get 1-2 yahoos trying to dominate a focus group (a good facilitator will reduce their control), one-on-ones are great for people to open-up and not worry other opinions. You can use them in combination with focus groups to get both the undeterred answer and the ability for the respondents to feed off of each other. It is also very easy to mix questionnaires into the one-on-one process. It can’t be done with the phone or Internet interviews, but you can also observe body language in a one-on-one interview. Same rules apply for the interview – open-ended questions for verbal answers, the other type of questions (see above) for written questions. Almost forgot, my favorite is the quick human factors study – you hand the respondent something and ask them to hold it or read it or whatever and watch and listen to the response.

Questionnaires are the most versatile research methodology. They give the respondent the most flexibility to complete the questions. The questionnaire can be written, faxed, on the Internet, a PDA, a Blackberry, in your email, sent as direct mail, a part of a coupon, in your tradeshow literature, on a warranty – there are many more options. The format can be anything, what is important is writing a good questionnaire. Nobody liked the essay question in school and the open-ended question is equivalent to the essay. The amount of open-ended questions is directly proportional to the amount of time a respondent has to do the questionnaire which is nearly directional proportional to the size of the honorarium. There are always exceptions. Most of the time it doesn’t matter what you pay them, they either won’t fill out those questions or they will give 1-3 word answers. Sometimes you will get paragraphs form people who get little compensation, but that is very rare. My rule of thumb (I have broken it many times) is to have 3-5 open-ended questions per 15 minutes of questionnaire time. There are many ways to ask a question and kept it a rating or multiple choices. You will get a greater amount of data for the time spent. Also, keep the open-ended questions at the start to get top of mind responses. As for rating I like to use either a 5 or 7 scale. If the questions are objective I will use a 5 scale (e.g. 1 being completely disagree, 3 is neutral and 5 is completely agree). If the questions are more subjective, I will use a 7 scale (e.g. 1 being completely dislike, 4 is neutral and 7 is completely like). Don’t mix the scale number in a survey, keep it 5 or 7, never both. With multiple-choice, don’t go beyond 5 possible answers. For prioritization, I would keep the maximum number of things to 10. For multi-voting, I give a maximum of votes equal to a third of the objects. If there a lot of objects, I will give out less than a third.

That’s it for methodology, its pretty simple. You either go to secondary sources or you get it from the horse’s mouth. You can either watch the horse, interview it of give it a questionnaire or a combination of the three, that’s it. The only thing left are the more tactical elements of research.

I HATE RECRUITING. I have said it before and I will to the day I die. I hate it even more than doing a PowerPoint filled with graphs from the data. Most of the research I have conducted has been for the medical device industry. I need very specific professionals. It is not the same as doing a mall intercept for a consumer product where just about any slob on the street will do. Side note: For those who don’t know, in just about every mall in America (including the Mall of America), there is a research firm who has facilities in the mall. A mall intercept is when they will grab people and ask them your questions. First you need to get a good list of possible respondents. My database has been developed for several years yet the client parameters usually require a whole different group of people. The list from the client usually is as bad as mine. Either they are completely biased towards the client or they don’t exist. I would say I bat .200-.300 with the client’s list (which is usually VERY small, and yes girls, size does matter). That leaves professional societies but cold calling, faxing and emailing take a lot of time. Then, if you are getting respondents from Europe and Asia-Pac, you get little to no sleep. They also have the audacity not to speak English – how dare they! All in all, I HATE RECRUITING.

Speaking of size, the size of the study depends. If you are doing something very specific and you land a 2 hour interview with the head of cardiology at the Cleveland Clinic, who gives a sh*t about anyone else. If you are doing a quick concept test of a consumer product via a web survey, you probably would want 100-200 responses (more than that it starts to be quantitative). There are too many variables to give you an exact answer.

Next are research tools. You will need scripts for the recruiting, interviews and site visits. You will need a questionnaire(s) (the questionnaire can be tailored to the different respondents). Charts, sketches, renderings, models, prototypes and other images need to be appropriate to the research phase.

I will go into analysis next. Don’t worry, it will be short and sweet. Not like all of the bullsh*t I have written so far.

Coming into this busy conversation a little late, but wanted to make some times.

-There are some companies (a few) that do qualitative and quantitative research as the initial phase of a design project. I do it on practically every software design project that we do.

-Focus groups, interviews and surveys are okay, but the greatest value comes from direct observation of the individuals doing their work, using a product, etc. Interviews and the like are a way to get a deeper understanding, but what a person says, or even thinks, should not be confused with what he or she does.

-“Participants” is an alternative to “respondents”

May I ask, what is the scope and methodology of the quantitative research? What is the typical goal of the research. Its not observational, is it?

I almost totally agree when doing product research. Of course there are exceptions, especially when it is not practical to integrate your concepts into the workplace. When doing research for strategy or product positioning, I think focus groups work the best. If well facilitated, a group think works better than individual interviews.

Anything else? My objection is that both are kind of cold.

hi, Im doing a project involving childrens furniture for preschool aged children in their classrooms and daycare centers. Im putting together a survey to go forth into the world and learn why i need to design more options for these people, children and caretakers. Ive got some of the basic safety, use and Pricing covered as well as cleaning, how about some ideas of other areas i might be missing.

also how about “pertinent area experts” for your participants

Suggestion: Do NOT do a survey, conduct observational research instead. Watch how the kids use the furniture. Talk to the teachers/aides and find out about how they go about their day and directing the kids through their activities. Learn about kids “cognitive” learning styles & phases. This type of research will be MUCH more impactful. If you want to learn more, PM me.