Wow, this is one hot discussion. The air is thick with anticipation for, dare I say, a tutorial for primary research. This is probably due to the writing style being equivalent to plain white toast. Here goes:
First, I forgot to mention quantitative and qualitative research. In developing products I have NEVER seen a company pony up to do qualitative research up front. I do mostly medical devices and qualitative is done for validation for the FDA (510K, pharma trials, etc.). You can also use third party qualitative research reports in the development process. The government (census bureau and other stuff) or academia usually has some good stuff for sniping. Otherwise, this discussion will be focused on quantitative research.
Hopefully you have determined the research goal and objectives when you had the long meeting to kick off the project. If not, write it down and get buy in. The goal and objectives should tell you the scope and methodology of the research. The scope will tell you who you want – end users, influencers and/or decision makers and how many of them. The total number of respondents is directly related to your budget and methodology. Generally, you need an honorarium for the respondents. Just as you get paid for your time, so should these people. The most expensive is if you are doing one-on-one interviews with physicians who are the leaders in their respective fields. You pay those guys $200/hour and pray they don’t laugh in your face because you are low balling them (generally, they will do the research only if they are interested in the subject). The cheapest is doing an online survey with general consumers. You offer them a chance to win the electronic device de jour and you are out only a few bucks or if it is short enough, they may do it for free. You don’t have to offer an honorarium – it just takes longer to hit your target number of respondents. As the saying goes; better, faster, cheaper – pick two. I will get into more specifics with budget when I write individual methodologies.
Sorry to go on a tangent, but there are a couple more general things before I focus on methodologies. Another way to categorize research is by the phase of development the product is entering. This way research can be called exploratory, directional and confirmatory. Think of a funnel with exploratory on top, directional in the middle, confirmatory on the bottom and what comes out is the product (or product strategy, depending on your original goal). Exploratory is used for gathering general info about the market. For example, when Apple decided to start development on the ipod, it would have done exploratory research. They looked at the growing mp3 market, benchmarks of portable music players (e.g. Walkman) and determined combining the two was a good idea. I have no idea who came up with the original thought but it could have come from R&D, engineering, marketing or design and some exec gave it some budget. 20 years ago you would assume it came from the skunkworks, but I rarely see companies use that methodology. Anyway, Apple determined the ipod was a good idea on paper. They would then use directional research in defining the overall requirements of the product. Things like how big is it, how much music can it hold, how long will the batteries last, interface, etc. You can also use directional research to assist in developing the general look and feel. I am assuming all of the requirements are assessed to Apple’s core positioning strategy of innovation. Finally, after putting specifications to the requirements, Apple would have used confirmatory research to finalize the device. Generally this is done with respondents evaluating only the final few concepts.
Also, I put respondents into four categories; thought leaders, end users, decision makers and influencers. As I said before, this can be one person or four different people (I forgot thought leaders on my previous post). You can subcategorize these people into other groups like early adaptors, laggards, etc. It is important to discover who they are and the relationships between these individuals with your research. It will create a better understanding of the potential impact of your product. For examples, a thought leader for the ipod may be a band or a DJ like Howard Stern or for old folks like me, Jim Coates. End users are pretty obvious. Decision makers are the ones to pull the trigger, it could be the parents. Influencers are more like unimportant thought leaders, generally your peers.
As for methodologies, I will keep it simple. You can either look at the respondents or you can listen to them (I put reading what they have written a category of listening). I suppose you could touch, smell or taste them, but that would probably fall into quantitative research which, as I stated before, is very rare in product development. So you have three choices; you watch or listen or, most preferred, a combination of both.
Since I consider myself as an amateur voyeur, I will start with watching respondents. Watching is always better than listening (unless you get some weird pleasure at a tennis match). Watching, of course is much more time intensive and cost more but that is something between you and the budget. It is also best if you are watching the respondent in their environment as opposed to a focus group or third party location. You all should know this is called ethnography. When I didn’t know any better, we called them site visits. Please have a plan before you arrive. I know from personal experience if you just show up and say, “I am here to watch.†you will be hauled away in cuffs. Once again, you have to do your homework. All this is essentially is a workflow analysis. You want to watch the respondent’s activities surrounding the use of the product. You should have a very good idea what they do before you get there. This helps in how much of their activities you want to observe and increase the intelligence of any questions you may have. Generally, big picture activities don’t lead to innovation, it is in the details that rarely get mentioned and are second nature to the respondent. You also want to carefully record and confirm the workflow. This will enable you to understand how your product will affect their day-to-day activities. I don’t want to tell anyone how to design, but people are resistant to change and need a good reason to do so. By understanding the respondent’s workflow and its drivers you should be able to focus the direction of your concepts. I usually put each of the respondent’s activities on a piece of paper, stick them to a wall and do my best to understand the issues with regards to the workflow.
As for the physical act of observing, I am a believer of quantum mechanics and the Heisenberg Uncertainty Principle. If a respondent knows they are being watched, they will act different. Once again, this is another reason to do your homework. If the respondent doesn’t have to hold your hand through the process they will become more comfortable and become more natural with their actions. There is a line to cross – don’t be a know-it-all, if the respondent dislikes you, you might as well leave and you will not get good data. Some researchers use recording devices when observing the respondents. Sometimes a camera or voice recorder is less intrusive as a person. The recorded information can be played repeatedly for a larger group of people for further analysis. A couple of things to remember: Always get permission and remind (at the start of the session) the respondent they are being recorded. As you probably already know, people will be put off by the camera. This will go away after only a little time especially since cameras are now very discreet. You can also set up permanent cameras. This is a purely observational methodology and there is no interaction between the respondent and researcher. It is also expensive and very boring because of the time involved to review the tapes. Some places will refuse the use of cameras. This is especially true with medical devices due to laws like HIPPA. You can choose to interview the respondent while doing your research. Once again, this is a little more intrusive and can affect their behavior but you can get a better understanding of their process if they talk about it. I will discuss interview techniques a little later.
Focus groups are another observational methodology. I have never conducted a focus group in a respondent’s natural environment. I would find it interesting if any of you have done that “twist†on a focus group. I have done groups ranging from just asking questions (no props) to trying to approximate their natural environment. Usually there are pictures and models to show the respondents. I have also never done a focus group without some sort of an interview. I would again be interested in stories about purely observational focus groups. Focus groups have gotten a bad rap usually because there are done poorly. You want the group to interact and have them contribute to the process; they will be much more engaging. You can’t just throw an idea in front of them and ask “What do you think?†or “How would you rate this?†You don’t need them to create the product, that is your job, but you do want them to feel vested in the process with the questions you ask. How to do that is proprietary and I don’t want give away the keys. Having the respondents create the product is a method used. Its kind of what you may have seen on The Apprentice. Have an IDer sketch or CAD their ideas. I have seen this work with limited success but there are better ways for a lively group.
If that wasn’t dull enough, next are some general guidelines for a focus group. Ideally, you want 6-8 respondents per group, anything over 12 is chaos. Keep the session under 2 hours and have a 5 minute break halfway through. Give them food and drink. Do your homework. Develop a script. Get signoff for the script from the client. Stick to the script. Don’t let the client bully you into adding questions to extend past the 2 hours. Have one of your people in the room leave every 20-30 minutes to get additional questions from the viewing room. Answer a respondents question with a question – e.g., Does this come with a flux-capacitor? You reply – Would you like it to come with a flux capacitor? Use visual aids – sketches, renderings, models, prototypes. Have the visual aids appropriate to the research stage (don’t have functioning prototypes if you are telling the respondents you need their assistance in developing the product). Don’t give a PowerPoint. Have the respondents write their answer before opening the discussion. You will always get 1-2 people who want to dominate the discussion. Don’t let 1-2 people dominate the discussion. Keep the respondents peers, don’t mix different groups. An example is not to mix physicians with anyone else, they think they’re Gods and will dismiss others very quickly. Have a good mix of questions – open ended, rating, prioritizing, multiple choice, etc. Thank the respondents (you may use them later). Don’t use a respondent more than once every 6 months.
That’s all for now. As always, keep those questions and comments coming. Next I will get into interviewing, questionnaires, research tools, recruiting, analysis and other fun and wacky things in the world of research.