gaurav1987
Gaurav Garg
Qualitative Methods and Rigorous Management Research: (How) Are They Compatible?
IntroductionThe role, benefits and appropriate use of qualitative research methods in the basic and applied social and clinical sciences have been discussed extensively in the research literature. The field of health services research, in particular, has benefited from several insightful, comprehensive discussions of qualitative research methods and their appropriate use. Proponents have convincingly argued that qualitative methods contribute findings and insights that cannot be derived from "conventional" or "quantitative" research methods, and that research in the clinical, social and policy sciences requires careful application of both types of approaches to properly study their phenomena of interest
Although most discussions of qualitative research methods (including qualitative data collection and analysis methods) are careful to avoid over-simplifying or stereotyping, the dominant conception of qualitative methods continues to equate these methods with research perspectives and study goals often labeled "non-traditional" or "unconventional." Included in this broad category are exploratory or hypothesis-generating (inductive) studies, interpretive research, historical research and several other forms of knowledge creation distinct from deductive, hypothesis-testing research conducted within the positivist tradition. This view of qualitative methods is not without foundation: most authors view inductive, interpretive and related applications of qualitative methods as their strength and area of unique contribution, given their superiority over quantitative methods in developing insights into actors' values, beliefs, understandings and interpretations of events and other phenomena, or in explaining historical occurrences. Despite these strengths, however, the contributions of qualitative methods to deductive research are no less significant or unique, and are no less important nor valuable in these realms than in their traditional fields of usage. The contributions and role of qualitative methods in deductive research are often overlooked, however, or fall short of their potential when they are applied, due to shortcomings in their use. These shortcomings (1) limit the value contributed by qualitative methods and (2) undermine perceptions of their importance and applicability, further contributing to a cycle of under-use and misuse.
This paper addresses issues in the use of qualitative research methods in hypothesis-testing, deductive research (sometimes labeled "conventional" empirical research). The paper aims to illustrate how and why such research can benefit from increased application of qualitative methods, and how these methods can be used in a manner consistent with accepted standards of rigor and validity. The paper is a motivated by the belief that researchers who are not trained in qualitative methods (and are therefore accustomed to conducting empirical research using quantitative methods alone) are less likely to be interested in applying qualitative methods in inductive or interpretive research, but can—and should—be interested in applying qualitative methods to enhance the "conventional" forms of empirical research they are already conducting. The paper is further motivated by the observation that qualitative methods are too often applied inappropriately when used as a complement to quantitative methods in hypothesis-testing studies. The consequences of this misuse include threats to the validity of the study and its findings, as well as threats to the reputation or perception of qualitative methods as a valuable set of tools and approaches for a diverse set of researchers and research projects.
The paper does not attempt to provide an overview or tutorial in qualitative methods (given the existence of numerous excellent books and papers performing this role), nor does it discuss "mainstream" applications of qualitative methods in exploratory, interpretive or historical research. Instead, the paper has a far more modest aim, discussing, illustrating and advocating for the appropriate application of qualitative methods in hypothesis-testing research—including the types of studies often conducted in management research and in its reference disciplines in the social and behavioral sciences. A few key examples of these methods are listed in an Appendix.
Management research, qualitative methods and rigor
Threats to the validity and value of qualitative methods within management research arise throughout the entire process of their application, including (1) study design and conceptualization, (2) data collection instrument and protocol design and development, (3) data collection, and (4) analysis and interpretation. Most of these threats relate to need for rigor and for explicit, a priori goals, plans and implementation of qualitative methods. When harnessed for inductive, interpretive research, qualitative methods are typically used in a flexible, emergent manner, in which a priori specification of concepts, measures and data sources—and explicit limitations in the domains addresses and questions asked—would act to impede discovery of important phenomena and insights, thereby weakening achievement of the research goals. In such applications, qualitative analysis methods rely on formal, explicit techniques as well as less formal, implicit, intuitive interpretation. In deductive research, however, over-reliance on emergent, informal application of qualitative research methods and implicit analysis methods typically weakens their value, often producing inconsistent data and results with questionable validity.
Study design and conceptualization.
When applied to conventional, hypothesis-testing research, qualitative methods must be used within a study design and framework incorporating each of the key elements required in studies involving only quantitative methods. These elements include careful reviews of relevant theoretical and empirical research, derivation or development of formal hypotheses within an explicit theoretical framework, and a priori specification of variables and measures. Theory-based hypotheses specify expected causal relationships, guide identification of the specific variables deemed relevant and requiring measurement and guide analysis and interpretation of data and results. Reviews of relevant literature also provide guidance in conceptualization and measurement of key variables, including identification of specific data sources, measurement tools with known characteristics and guidance in use of these tools.
Failure to adequately specify hypotheses and relevant variables typically results in unfocused measurement and analyses, including a failure to identify and measure key variables and concepts, and a failure to recognize or to properly interpret important findings. While use of formal hypotheses is not consistent with the use of qualitative methods in inductive or interpretive research, it is critical in the use of these methods in deductive research, whether this research involves quantitative data and analysis methods or qualitative methods.
Data collection instrument/protocol development
Data collection instruments and protocols in qualitative research are often informal, flexible and subject to large variations in application. While flexibility represents a strength in traditional qualitative research, it produces unfocused data collection and variable data quality when qualitative methods are applied in deductive research. For example, interview guides specifying general topics of interest, using broad, open-ended questions can be very effective in assessing interview subjects' assessment of important concept and issues and their beliefs and values, but ineffective in ensuring that comparable measures of identified variables are collected from a range of subjects (e.g., assessing organizational participants' views or their ratings of concepts or variables deemed important by the research team). In part, the distinction here is between data collection approaches designed to develop frameworks for understanding and describing the phenomena of interest, versus applying a priori frameworks to collect pre-defined data and test aspects of these frameworks. Similar problems result from the use of observation guides or protocols lacking adequate specificity and a firm foundation in a priori hypotheses and clearly-identified variables: such protocols often produce inconsistent data by (1) encouraging the observer to record events as they unfold and to record a wide range of attributes of the situation under study (whether or not they are deemed relevant to the hypotheses of interest), (2) limiting the likelihood that the observer will note the significance of events that do not occur, and (3) limiting the likelihood that the observer will collect complete, consistent data required for direct comparisons across observation samples.
Considerations of validity, intrusiveness or subject reactivity (Hawthorne effects) and triangulation (to minimize bias) are also too-often neglected in deductive applications of qualitative methods. Distinctions between subjective and objective data and between formal and informal organizational structures and processes are also frequently neglected, threatening the validity of study conclusions.
Avoiding these problems requires careful design of data collection plans, based on study goals and hypotheses, and involving use of systematic tables or other methods for specifying key variables and suitable, multiple measures. Depending on the importance of each variable and the validity of available measures, two or more data sources are typically needed in qualitative research. Data planning tables listing concepts or variables, definitions and data sources are effective in ensuring appropriate rigor; data collection instruments (including document coding forms, survey questions and other data specifications) can be developed directly from these tables.
Rigor and validity are also enhanced through development and use of data collection instrument specifications and training protocols, including variable and measure definitions and instructions in instrument use. When used in management research, such protocols should include plans and instructions for approaching sites, making contacts, arranging interviews/visits, identifying and obtaining documents, following-up (to obtain documents and other post-visit/call information), managing informed consent and confidentiality, etc. Adequate pilot testing helps ensure the appropriateness of the data sources and measures, although data collection protocols must be flexible and allow for changes in data collection plans and strategies, when pilot testing fails to reveal valuable new data sources or validity problems with the sources in use.
Finally, study validity is further enhanced through development of data analysis protocols and plans together with the actual instruments, rather than after completion of data collection. Data planning tables created to guide data collection activities can be used to develop data reporting templates and specifications for translating raw data into variables and preparing for analyses; management data are often reported in a standardized “organizational profile” or other comparative format. These profiles store raw data and summary variables from all data sources, which are then converted to tables for analysis.
Data collection and data management
Use of qualitative methods, including interviews and observation, is subject to wide variations and interviewer/observer bias and interpretation. Steps to minimize these biases include adequate training of data collection staff; comprehensive plans for data collection, validation and storage; and frequent reviews of data quality and interpretation. Data collection and management plans should include immediate post-collection coding and review of data that are time- or memory-sensitive (e.g., interviews and observation). While data validity and completeness can be enhanced through tape recording of interviews, other methods may be more cost-effective, including real-time survey data entry and editing, use of paired interviewers, post-interview debriefing, and other methods. Quality assurance methods should be considered and operationalized for each instrument and data sources. Problems such as incomplete, missing, unusable data should be identified and resolved during the data collection phase, rather than after its completion.
Analysis and Interpretation
Analysis of qualitative data should be guided by the pre-specified, model-based hypotheses and detailed analysis plans developed at the outset of the study. Unfortunately, while quantitative analysis methods are well-established and accepted, methods for analysis of qualitative data are subject to variability and lack of consensus. Analyses of qualitative data are too-often informal, ad-hoc and emergent, with low reliability and validity. These threats can be countered through the use of formal table approaches, in which key variables relevant to each hypotheses are listed in tables and manipulated in a blinded fashion, using qualitative pattern-identification and non-parametric quantitative techniques. The analysis tables summarizing and synthesizing information from diverse sources in a standardized format may also serve as reporting tools, in papers and reports.
Analysis and interpretation follows the study hypotheses and research questions, but will often include detailed causal explanations and exploratory questions and findings as well, taking advantage of qualitative data's value in these areas. Combining the use of qualitative methods for hypothesis-testing and interpretive, inductive applications in this manner represents a powerful application of these methods, using their strengths to enhance management studies and other empirical research in important ways.
Conclusion
The origins and development of qualitative research methods and their close association with inductive, interpretive and historical research have led many researchers to associate these methods exclusively with these forms of research, and to fail to recognize their value in conventional deductive empirical research. Yet the rigorous application of qualitative methods in deductive research requires many modifications and adaptations. Qualitative methods, when properly applied, can contribute significant value to management and organizational research and research in health services more generally.
Appendix.
Outline of key qualitative research methodsMail, telephone surveys
• Effective for brief, fact-oriented, simple short-answer questions
• Do not work well for exploratory, open-ended, inductive questions
• Facilitate consistency, objectivity, completeness
• Mail vs. telephone considerations include development vs. administration effort
Telephone, in-person interviews
• Required for elite interviews, opinion and long-answer questions
• Infeasible for high-volume data collection
• Reduce consistency, objectivity, completeness
• Telephone vs. in-person considerations include sensitivity of information, need for candor; need for stories; risk of bias, social desirability, cues; interview-interviewee relationship
Collection and analysis of documents and other archival, administrative data
• May be more objective, consistent, complete, but incompleteness is more typical
• Interpretation and use require knowledge of creation conditions and factors
• Explanatory or supplementary interviews are generally needed
Observation
• Observation ranges from walk-throughs (to assess physical layout and facility characteristics), to attendance/observation at meetings, in routine work settings
• Observation is generally very expensive, often infeasible
• Observer’s biases, organization’s reactions to observers may affect validity of data, yet observation complements the biases, validity problems inherent in respondent reports