Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Complexity of the Paradigms Present in Quality Criteria of Qualitative Research Grids:

Complexity of the Paradigms Present in Quality Criteria of Qualitative Research Grids: With qualitative methods being increasingly used in health science fields, numerous grids proposing criteria to evaluate the quality of this type of research have been produced. Expert evaluators deem that there is a lack of consensual tools to evaluate qualitative research. Based on the review of 133 quality criteria grids for qualitative research in health sciences, the authors present the results of a computerized lexicometric analysis, which confirms the variety of intra- and inter-grid constructions, including within the same field. This variety is linked to the authors’ paradigmatic references underlying the criteria proposed. These references seem to be built intuitively, reflecting internal representations of qualitative research, thus making the grids and their criteria hard to compare. Consequently, the consensus on the definitions and the number of criteria becomes problematic. The paradigmatic and theoretical references of the grids should be specified so that users could better assess their contributions and limitations. Keywords qualitative research quality, evaluation criteria, guidelines, paradigms, Alceste® lexicometric analysis As part of an ongoing research (this document represents Introduction part of a larger research funded by the Swiss National Science The use of qualitative research methods in the social and Foundation (www.snf.ch), 2011-2014: “Quality of Qualitative human sciences considerably increased as of the 1990s Research in the Health Sciences: Which Evaluation Criteria?”), (Chamberlain, Stephens, & Lyons, 1997; Flick, 2007; Miles & we examined 133 evaluation grids in five major health science Huberman, 1994; Murray, 2000; Silverman, 1993; Smith, fields: “medicine,” “public health,” “nursing,” “psychiatry Harré, & Van Langenhove, 1995). This dissemination quickly and psychology,” and “research methodology.” To better reached the health field through nursing, medicine, and public understand their variability and its origins, we conducted a health. This renewed interest in qualitative research promoted computer-assisted lexicometric analysis, our premise being the publication of numerous “quality criteria” grids/guidelines that the diversity of the grids should not only be linked to the (we use the words criteria grids, guidelines, and checklists as various disciplinary fields but is also intrinsic to various synonyms) to evaluate the quality of a qualitative research, qualitative research conceptions among grid designers. In project, or article (e.g., in scientific journals such as the British this article, we start by presenting a brief status report on the Medical Journal, Psychology and Health, Family Practice, health science research field, in terms of quality criteria. We Sociology of Health and Illness, etc.). This abundance of then present our methodology. In the third part, we present guidelines, however, resulted in the use of numerous terms our main results. Finally, we discuss the inherent limitations and grids, which failed to contribute to the clarification of the main consensual criteria to evaluate qualitative research. This leads to a certain lack of recognition of this work in publica- tions and for fundraising purposes, the solicited reviewers University of Lausanne, Switzerland declaring that they are ill-informed or do not have the tools to Corresponding Author: evaluate qualitative research (Green & Britten, 1998; Christine Bruchez, Institute of Psychology, University of Lausanne, Greenhalgh & Taylor, 1997; Mays & Pope, 2000; Popay, Geopolis, 1015 Lausanne, Switzerland. Rogers, & Williams, 1998). Email: christine.bruchez@unil.ch Creative Commons CC-BY: This article is distributed under the terms of the Creative Commons Attribution 3.0 License (http://www.creativecommons.org/licenses/by/3.0/) which permits any use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access pages (https://us.sagepub.com/en-us/nam/open-access-at-sage). 2 SAGE Open of the various research conceptions highlighted within the Given these facts, our research focused on analyzing the different disciplines they represent. links between the health science disciplinary fields and the major paradigms guiding grid designers in determining the criteria used to evaluate qualitative research. We used Problem: A Shift in the Quarrels From Kuhn’s (1962) general definition, which states that a para- the Last Century? digm is a set of disciplinary convictions shared by the scien- tific community at a given time during its history. Any In a context of an increase in the publication of qualitative science is stabilized by sharing the same paradigm, which studies both in books and recognized scientific journals, marks its field and investigation procedures. This paradigm debates on the relevance of developing (or not) specific cri- is developed by groups of socially dominating researchers teria for qualitative methodologies have evolved continu- who control the transmission of knowledge through research ously (Devers, 1999; Dixon-Woods, Shaw, Agarwal, & Smith, funds, positions, and publications. This definition captures 2004; Mays & Pope, 2000; Yardley, 2008). Certain authors the epistemological and paradigmatic choice that is made in deem that the validity and reliability concepts stemming from all research conducted, whether this choice is deliberate or quantitative methodologies are suitable to reach a high level of the outcome of reflection, or much more implicit. rigor in qualitative research (Miles & Huberman, 1994; Morse, Barrett, Mayan, Olson, & Spiers, 2002; Porter, 2002); how- ever, another group of authors finds that applying the “reli- Method ability” and “validity” criteria directly is unacceptable for qualitative studies. This last group argues for differentiated Make-Up of the Corpus criteria for qualitative work based on specific philosophical In total, 133 grids (checklists or guidelines) were examined and methodological foundations. They argue their position by in the various health fields, from the MEDLINE, PsychInfo, insisting on the structured thought needed in all paradigm CINAHL, PeruniL, ScienceDirect, and Web of Sciences data- transfer, especially to apply the same criteria to social and bases, using the keywords “health,” “qualitative research,” human sciences as those applied to natural sciences (Cicourel, “assessment,” “appraising,” “quality criteria,” “peer review 1964; Garfinkel, 1967; Hamberg, Johansson, Lindgren, & process,” “guideline,” “evaluation,” and “standards.” Data Westman, 1994; Schutz, 1962; Van den Hoonaard et al., 2004). were used according to the following inclusion criteria: (a) Somewhere in the middle, a third group of authors argues that grids published for health scientists (medicine, nursing, evaluation criteria can be transferred from quantitative to qual- health psychology, health sociology and anthropology, pub- itative methods, as long as they are adapted (Pope & Mays, lic health, etc.), (b) guidelines that were sufficiently devel- 2006). oped and commented, and (c) work published by health In following these debates, we notice that the discrepan- journals, institutes, and organizations whose scientific qual- cies regarding the quality criteria seem to reflect different ity has been validated. conceptions of qualitative research (Santiago-Delefosse, Whether they are taken from scientific journals (instruc- 2007; Schneider, 1998). Yet, these discrepancies may tions to authors), articles for authors and researchers to explain why it is so hard to build consensus that will allow improve their qualitative work or as an evaluation support for the evaluation of qualitative research quality within the tool, or methodology manuals, most of the grids examined are scientific community, especially because they remain presented as lists of criteria to meet to produce “good qualita- obscure in literature on quality criteria grids for qualitative tive research” that is rigorous and reliable. Most of these grids research. are accompanied by a text explaining the criteria (definitions Some work highlights the variability of existing grids and uses) and/or a theoretical or methodological text that is (Cohen & Crabtree, 2008; Flick, 2007; Mays & Pope, 2000; sufficiently detailed to support the quality criteria analysis. Popay et al., 1998) and the difficulty in comparing them. The Most of the grids examined come from publications dat- inconsistency of the number and objectives of the criteria ing from 1993 to 2011, which correspond to the increase in used, the absence of consensual and explicit definitions, and debates on the quality of qualitative research in the 1990s. the absence of construction consistency between the grids of Most grids come from medical journals (45), nursing litera- one disciplinary field were identified in a previous publica- ture (39), research methodology journals/literature (20), pub- tion (Santiago-Delefosse, Bruchez, Gavin, & Stephen, 2015). lic health literature (19), and psychiatry/psychology literature Therefore, research conceptions underlying these grids may (10). Most evaluation grids were published in medical and be questioned as well as the epistemological and paradig- nursing literature. matic positions that guided their designers. This questioning We submitted our corpus to two analyses: a written analy- is still highly relevant as the lack of consensus regarding a sis with inter-evaluator agreement that allowed us to deter- certain number of criteria, even minimal, is an argument to mine coding categories for our second analysis and a which article and fundraising evaluators refer and which computer-assisted lexicometric analysis (Alceste®). motivates the rejection of projects. Santiago-Delefosse et al. 3 “post-positivism/interpretivism” bringing together grids that Thematic Analysis of the Content and Theoretical combine both approaches; the third category entitled “con- and Methodological Foundations of the Analysis structivism/interpretivism” was built upon a similar basis; Categories the fourth one was “critical constructivism”; and the fifth one Using Alceste® requires prior coding of the text to show the was the “post-modern” approach including grids with crite- software which variables must be identified in the text. To do ria dealing with radical relativism/constructivism (consider- this and identify the variables that would be most relevant to ing, for example, that qualitative research is too singular to our questioning, we conducted a first thematic analysis of the be evaluated). No grid was identified in the “positivist” epis- grid content. This first content analysis helps to identify the temological approach. “contextual variables” specific to each text (i.e., author, dis- cipline, and paradigm; “paradigm” here refers to a system of Presentation of Alceste® and Relevance of the fundamental beliefs belonging to the researchers and used as Lexicometric Analysis Based on Our Hypothesis the basis for their theoretical and methodological choices in We used an analysis produced by Alceste® to process the the conception of the grids). The variables are then coded to contents of this corpus. Alceste® is textual or statistical data make them compatible with the computerized analysis. The analysis software, originally designed by Max Reinert, of the content analysis was conducted independently by three CNRS, in France (Reinert, 1990, 1993). It has been used in researchers. The results were triangulated and discussed col- the human and social sciences since the 1990s. It counts lectively to come to an inter-evaluator agreement (Braun & words in a frequential manner and helps to obtain analysis Clarke, 2006; Robert & Bouillaguet, 2002). units that are based on formal criteria. It uses an inductive The reference framework used for this content analysis and recursive approach and helps to identify co-occurrences, was the one used for the work conducted by Denzin, Guba, or word associations in a sentence, by processing the words and Lincoln. Indeed, besides their heuristic scope, they are according to their similarities and differences. Technically, the most often cited in literature on the importance of para- Alceste® breaks the corpus into fragments that are relatively digms and epistemologies in qualitative methods (Denzin & similar in size, referred to as “context units” (e.c.u. the small- Lincoln, 1994/2011, Santiago-Delefosse & Rouan, 2001). est statistical units created by the software, based on a com- In 1994, Denzin and Lincoln proposed a classification promise between the syntactic form [proper punctuation] and that included two major paradigms subdivided into four epis- the statistical constraints [these units must be of comparable temological research positions: the positivist/post-positivist size]). These same fragments are then reclassified statisti- paradigm (including the positivist or post-positivist episte- cally and split into classes whose specific vocabulary is as mological approaches) and the interpretivist/constructionist different as possible. These classes are called “lexical paradigm (including the critical or constructivist epistemo- worlds” and are considered to present an “image” of the rep- logical approaches). Notice that in 2011, they added a “par- resentations contained in the text, the ideas and the corpus’ ticipative” epistemological position to the interpretivist/ major themes (Garric & Capdevielle-Mougnibas, 2009). constructivist paradigm. A certain number of criteria differ- Second, it defines a link to what software developers refer to entiate these epistemological and paradigmatic approaches. as “contextual variables” (data elements known by the Among these criteria, three major axes caught our attention researchers and used as instructions for the software so that as they can be operationalized and help to compare the crite- the lexical analysis can be conducted with these variables in ria in the grids: (a) the form and nature of reality (following mind). These variables are used to identify the texts and are the preexisting or co-constructed one), (b) the researcher’s related to their content. The researchers introduce them type of position regarding research (outside observer vs. par- according to their relevance to the research questions. The ticipant involved in the co-construction), and (c) the use of classes identified by the software must then be examined and the research (discovery of inherent truths vs. work on subjec- linked to the co-occurrences to give them meaning and tivity and the impact of its construction on the world, work explain their differences (Aubert-Lotarski & Capdevielle- on the co-construction of realities, work on dominating Mougnibas, 2002). Notice that the software’s results are forces and empowerment). We added a fourth axis regarding based on frequencies rather than weighting, which could be the presence of ethical questions, which also indicate the considered a limitation. researcher’s position toward the research (absent when the We deemed it relevant to use this software because of its research is considered an objective report on what exists vs. textual classification abilities, given that we hypothesized always present when the researcher, his or her values and the that the differences between the grids were not solely linked context are considered a part of the research at all times). to disciplinary fields in the health sciences, but represented We analyzed our corpus by focusing on these four major paradigmatic and epistemological positions within certain axes, which help to report on the conceptions underlying the disciplines, connecting certain grids of disciplines that dif- research in the 133 grids examined. As a result, we identified fered from those from which they stemmed. If this hypothe- five paradigmatic and epistemological approaches: The first sis was validated, we would obtain a classification of grids category was defined as “post-positivism”; the second one, 4 SAGE Open Table 1. Contextual Variables Used for Alceste® Coding. Five paradigmatic and epistemological approaches Paradigmatic position Post-positivism Constructivism, critical constructivism, and post-modernism Epistemological Post-positivism/ Constructivism/ position Post-positivism interpretivism interpretivism Critical constructivism Post-modernism Alceste® coding pp1 pp2 c1 c2 c3 The acknowledgement of Mixed grids mixing Grids using a knowledge Refers to a Grids that highlight an external reality to be criteria stemming approach according to constructivism social constructivism discovered using both both from a post- which there apparently approach that according to which our qualitative and quantitative positivist and exists multiple realities. adds a critical understanding of the methods that are capable interpretivist approach Our image of reality is dimension whose world not only reflects of “representing” or (hermeneutics, a construction of the goal is to bring it but also organizes “approximating” this psychoanalytic . . . human spirit, which must about a reform of and orders it. Possibly reality. The introduction ). It designates an be interpreted in a social, knowledge-building knowing what “the of qualitative methods approach according cultural and temporal in a sociocultural reality” is questioned, corresponds to “a way to which there exists context. According context. and brings us closer to to capture” this reality one dynamic and to this approach, the a form of relativism, “as best as possible”. The evolutionary reality to researcher cannot be where the researcher’s researcher and his or be discovered through separated from his or values and judgments her topic are considered the discourses. her subject and interacts change his or her independent. with it. subject. Five health disciplinary fields Disciplinary fields Medicine Nursing Public health Psychiatry/psychology Research methodology Alceste® coding Med Nurs Pubh Psy Meth Note. No grid examined corresponded to a pure positivist epistemological approach (which is normal for grids targeting the evolution of qualitative research). that would be more connected to paradigmatic/epistemologi- perspective: There are some in medicine, research methodol- cal positions than to disciplinary fields. This does not mean ogy and public health, one in nursing, but none in psychol- that certain disciplinary fields cannot be consistent among ogy/psychiatry. The post-positivist/interpretivist approach one another (i.e., the older and more solid ones). (pp2) is the most important in our corpus (57/133) and con- The following variables were used for the lexicometric cerns grids that include criteria close to the positivist field analysis conducted as part of this work: and more interpretative criteria: They are mostly found in medical journals. Constructivist/interpretivist grids (c1) are •• The five health disciplinary fields in which grids were not as numerous (31/133) and propose criteria specific to published: medicine, nursing, public health, psychia- qualitative research, integrating contextualized constructiv- try/psychology, and research methodology. ism and the researcher’s interpretative position: They are •• The five paradigmatic and epistemological approaches mostly found in the fields of nursing and medicine. Grids as identified in the thematic analysis of the content: classified in the critical constructivism category (c2) are post-positivism, post-positivism/interpretivism, con- slightly less numerous (27/133) and include criteria specific structivism/interpretivism, critical constructivism and to qualitative research, with a constructivist approach: They post-modernism. are found in all disciplines, predominantly in nursing. Finally, there is a small group of five grids that is harder to classify in Table 1 presents the “contextual variables” used and the previous categories, with a profile that is often explicitly adapted for Alceste® coding and stemming from our written geared toward a post-modern approach (c3) and published content analysis. mainly in nursing. We note that the grids are mainly split into two large groups. The “post-positivist/interpretivist (pp2)” group Results represents half of the corpus (57/133), and then the sum of During the written analysis and according to the contextual the two “constructivist (interpretivist and critical)” also variables used, we obtained a distribution of grids by “disci- represents half of the corpus (58/133). Consequently, the plinary field” and “paradigmatic and epistemological issue regarding the discrepancies between the grids may approach” (Table 2). stem from the difference between these two large groups, There are only a few post-positivist (pp1) grids in the cor- especially in the definition of the two “interpretivist” pus. They generally propose similar criteria from a positivist positions. Santiago-Delefosse et al. 5 Table 2. Distribution of the Grids by Disciplinary Field and identify those of each class. This analysis highlights the spe- Paradigmatic and Epistemological Position. cific identity of the grids and the differences/similarities between them. The dendrogram (Figure 2) shows how the six Alceste® Codes classes are organized and linked. Disciplines (number of grids) pp1 pp2 c1 c2 c3 The six classes are split into two main branches including on one hand Classes 1, 4, and 5 (Quadrants C + D) and on the Medicine (45) 4 23 11 7 0 other Classes 2, 6, and 3 (Quadrants A + B). Nursing (39) 1 9 14 11 4 In the first branch (Quadrants C + D), two classes contain- Research methodology (20) 4 10 1 4 1 ing psychology/psychiatry grids stand out: Class 1 with its Psychology/psychiatry (10) 0 4 3 3 0 constructivist/interpretivist approach (c1) and its criteria Public health (19) 4 11 2 2 0 focused on experience, the meaning of things and compre- Total = 133 13 57 31 27 5 hension; and Class 5, which is more focused on validation Note. No grid examined corresponded to a pure positivist epistemological criteria (triangulation, member-checking) and a constructiv- approach (which is normal for grids targeting the evaluation of qualitative ist approach mixed with post-positivist criteria (c1, pp2). research). pp1 = post-positivism; pp2 = post-positivism/interpretivism; Some medicine, nursing, and public health grids (Class 4) c1 = constructivism/interpretivism; c2 = critical constructivism; c3 = post- modernism. can also be found in the constructivist/interpretivist group (c1), using criteria that are more based on practices, care, research use value, and clinical contributions. Lexicometric Analysis and Distribution of the The second branch (Quadrants A + B) includes grids deal- Corpus Among Six Classes ing with a post-positivist approach: On one hand, public health grids (Class 6) with a post-positivist approach, using Once Alceste® processed the corpus, the latter was split into certain criteria specific to the interpretivist field (pp2) and six distinct classes. Figure 1 details the class distribution and describing mostly sample selection methods and inclusion/ indicates the five most significant terms (χ ) given by the exclusion criteria, and on the other nursing grids (Class 2) software for each of the six classes as well as the contextual focused on literature, work framework, research process, and variables related to them: discipline and paradigmatic being a little more related to the constructivist field (pp2, c2). approach. To make it easier to read the analysis, we also As for the research methodology grids in Class 3, they pres- numbered the quadrants of the factorial design (A, B, C, D). ent very diverse approaches, ranging from post-positivism to The factorial design (Figure 1) shows the interrelations post-modern (pp1, c3, c2), with their content focused on between the classes and their respective positions in the practices, tools and research techniques. In that respect, these quadrants. Certain classes, namely, Classes 1 and 5 in grids, compared with the other groups, seem to be more Quadrant C, and Classes 2 and 6 in Quadrant B, are closely linked through criteria based on methodological rigor alone, linked to one another, almost interwoven. As for Classes 3 than through the discipline or paradigm. and 4 (Quadrants A and D), they are more well-defined and independent. We thus notice that the classes do not necessar- It is worth noting that Quadrant D (Classes 1 and 4) ily cover the disciplines, as there are two psychology/psy- mainly corresponds to the constructivist/interpretivist posi- chiatry groups with different approaches, as well as two tion (c1) while Quadrant B (Classes 2 and 6) mainly corre- nursing grids (Classes 2 and 4) or public health grids (Classes sponds to the post-positivist/interpretivist position (pp2). 4 and 6) characterized by different paradigmatic approaches. These two quadrants include criteria from grids from four of We also notice that Class 4 includes grids from three disci- the five disciplinary fields; however, Quadrant C corre- plines: medicine, public health, and nursing. sponds to certain criteria from the psychology grids only The factorial design also indicates a bipartite distribution (Class 5) and Quadrant A to certain criteria from the research between the grids underlying approaches that are essentially methodology grids (Class 3). constructivist (c1) and focused on the researcher’s qualities These initial results show that the characteristics of the as an integral part of the research in quadrants of C and D grids do not necessarily cover a classification based on the (Classes 1, 4 and 5), and the grids with more post-positivist disciplines and that their differences or similarities are more approaches that highlight the “good research practices” linked to the paradigmatic and epistemological positions that needed to ensure the research’s validity and objectivity, in underlie them. Only two classes (5 and 3) seem to be more Quadrants A and B (Classes 2, 3, and 6). determined by disciplinary proximity (Class 5 to a lesser extent though). Consequently, these differences must be examined to determine their meaning. The following analy- Dendrogram of the Six Classes and Significant sis provides further detail on the six classes based on their Terms elementary context units (e.c.u., semantic content) and con- textual variables (disciplines and paradigms) to better under- The examination of the significant terms (χ ) present in each stand their weight in the differentiation of the grids. class can explain the specificities in the factorial design and 6 SAGE Open Figure 1. Distribution of the six classes and contextual variables. are linked to the assumptions of the researcher who is deemed Analysis of the Classes According to the to be integrated to the research both socially and culturally, Significant Terms and Contextual Variables and interacting with the subject in its context to co-construct Quadrants C and D (Classes 1, 5, and 4): “Mainly” constructivist the interpretations and help to build new theories. approaches This class proposes specific evaluation criteria for quali- Class 1 (psy, c1). This class includes a group of grids that tative research. They are linked to the meaning produced, in stems mainly from psychology/psychiatry (psy, χ = 60), a context explained by researchers who clarified the implica- with a constructivist/interpretivist approach (c1, χ = 8). This tions, and demonstrated their training in terms of interpreta- group of grids is characterized by terms such as experience, tive theories. In this group of grids, and in keeping with the meaning, effect, self, understand, change, live, preconcep- constructivist paradigm as defined, the goal of the research is tion, reflexivity, affect, instrument, and so on (i.e., in the grids to understand and “reconstruct” its initial subject, as a con- of Curtin & Fossey, 2007; Elliott, Fischer, & Rennie, 1999; sensus between the researchers and participants, and be open or Fossey, Harvey, McDermott, & Davidson, 2002). A joint to new interpretations. As such, the validation of the research review of the class’s elementary context units shows that the places a high value on the researchers’ training and experi- epistemological approaches of this class’s criteria are geared ence, as well as on the consideration of their own “biases” toward the notions of the experience lived by a subject in its and the characteristics that are likely to influence the context of meaning, culture, interpretation, and empowerment research. The criteria proposed in these grids are usually as an impact on the research participants or reader. In general, adapted to qualitative or mixed methodologies. “authenticity” and “trust” are based on the principle that the research accurately reflects the meanings and experiences lived and Class 5 (psy, c1, pp2). A second group of psychology/psy- perceived by participants. On the other hand, a self-reflective atti- chiatry grids (χ = 3), different from the first, holds a distinc- tude is recommended. Indeed, comprehension and interpretation tive place in this class. This group is also dominated by a Santiago-Delefosse et al. 7 Figure 2. Dendrogram: Distribution of the corpus into six classes with significant terms (χ ). constructivist/interpretivist approach (c1, χ = 9), but with ple, it includes the Cohen and Crabtree (2008) grid published a post-positivist-interpretivist component (pp2, χ = 4) also in medicine, the Walsh and Downe (2006) grid for nursing present, which provides this group with an approach that is and the Dixon-Woods et al. (2004) grid published in pub- increasingly marked by research bias validation and reduc- lic health. Various significant terms given by the software, tion notions (e.g., Meyrick, 2006 or Stiles, 1999). This class such as practice, knowledge, discipline, contribute, policy, presents significant terms such as member, establish, valid, social, practitioner, standard, academic, and develop, direct different, confirm, external, corroborate, truth, triangulation, the grids in this class toward notions dealing with the appli- and audit. With the examination of the class’s elementary con- cability of the research and its relevance to the disciplines text units, they help to specify the preferred objectives in these concerned. The epistemological and paradigmatic position grids that promote the intersection of numerous perspectives of these grids is constructivist/interpretivist (c1), and their (triangulation) to reach a sufficient level of trust (credibility) elementary context units generally show an interest in devel- to validate the research. In each case, the strategy involves opment, interdisciplinarity, originality, the meaning given to cross-checking the results with other sources and perspectives. events in a specific content, and a foundation built on exist- Criteria that are more specific to qualitative research and the ing knowledge. Two major axes have been identified. The constructivist approach are also proposed: member-checking first deals with the evaluation of qualitative research (reach- to see whether the interpretations are consistent, conducting ing “trustworthiness,” a sufficient level of reliability and an iterative analysis, or looking for the saturation/recurrence contextualization). The second deals with its practical and/or of the themes presented in the results. scientific use in a broad way (its purpose). In this group, as per the constructivist paradigm, research is Most grids in the class highlight the importance of the considered to be a collaborative and collective co-construction connection between the study conducted and the existing endeavor whose researcher and participants are tools in the knowledge, the fact that any new knowledge can contribute search for consistency and the integrity of the reports and to new conceptualizations or questions, lead to the develop- inferences. The content and vocabulary used by the quality cri- ment of basic theories or hypotheses, question existing theo- teria of the grids, however, also indicate a need to objectify the ries or provide methodological information leading to research through validation techniques stemming from more improvements in the practice. Certain criteria, especially in positivist approaches. The criteria proposed in these grids are public health, focus on the social mandate, moral obligation adapted to qualitative or mixed methodologies. as a rationale for health sciences. Like for the grids in Class 1, Class 6 displays a paradigmatic constructivist approach in Class 4 (med, nurs, pubh, c1). A group of grids in medicine which the goal of the research is the “reconstruction” of its 2 2 2 (χ = 10), public health (χ = 5) and nursing journals (χ = subject and the development of new interpretations. The cri- 2) holds a distinctive place in Quadrant D, while still being teria proposed in these grids are adapted to qualitative or close to Classes 1 (psy, c1) and 2 (nurs, pp2, c2). For exam- mixed methodologies. 8 SAGE Open Quadrants A and B (Classes 2, 6, and 3): “Mainly” post-positivist and data collection are also preferred in guaranteeing the approaches rigor of the research as a whole. The criteria in these grids are Class 2 (nurs, pp2, c2). This class includes a group of grids usually generally adapted to quantitative or mixed method- stemming from the nursing field (χ = 11), mainly displaying ologies both because of the paradigmatic approach (pp2) and a post-positivist/interpretivist approach (pp2, χ = 4), with the discipline concerned (pubh). criteria stemming from a critical constructivist approach (c2, χ = 3). For example, there are Cobb and Hagemaster Class 3 (meth, pp1, c3, c2). This class is well specified and (1987), Russell and Gregory (2003) and Beck’s (2009) grids. does not overlap onto any other class. On the disciplinary This class’s significant lexical content includes terms such as front, there are many grids stemming from research meth- framework, question, paper, study, review, reference, discuss, odology articles and/or literature (meth, χ = 94). From a purpose, rationale, and explain. A complete examination of paradigmatic and epistemological point of view, however, it the significant terms in Class 2 shows that the approach is remains very diversified, with an essentially post-positivist mostly marked by notions dealing with the research anchor predominance (pp1, χ = 29), but also including the pres- in a clear theoretical framework, and scientific validation ence of criteria linked to a post-modern (c3, χ = 9) or critical through a comprehensive literature review that must be used constructivist approach (c2, χ = 6). For example, there are to place the new study in a broader context and show its grids by Flick (2006) or Sandelowski and Barroso (2007). specificity. The objectives and steps must be explained and The most significant terms in this class are linked to the qual- the connections between the research and the clinic high- ity of the data (from their collection to their retention) and lighted, as well as the impact on health practices and policies the training of the researchers in terms of the methods used: (a common aspect with Class 4). In this group, the researcher record, data, note, document, collect, consent, transcript, explains the research’s logical steps to demonstrate the valid- procedure, audio, observe, confidential, or protect. ity and reliability of the reasoning. Another major objective The grids in this class provide a thorough description of of the grids in this group is to bring about new models and the research process, emphasizing the data collection tools theories and contribute to international debates. Comparing and data retention and protection methods. Indeed, data stor- the results obtained with those of other studies must ensure age must allow for independent control, and the specific their transferability. description of the process must show how the data are pro- In a mainly post-positivist/interpretivist perspective, the cessed and how they correspond to the goal of the study. In criteria in these grids focus on the reasoning of the researcher keeping with the post-positivist paradigm as described, con- and methodological rigor, with, as its validation terms and con- cern for validation is reflected through issues related to the ditions, the faithful reporting of the logic of the steps, sampling proof and quality control (of the data, processing, and train- and data collection. The criteria proposed in these grids are ing of the researcher). There are also ethical criteria among generally adapted to quantitative or mixed methodologies. the quality criteria. In this class, this criterion constantly refers back to the consent of participants as well as to the Class 6 (pubh, pp2). This class is closely intertwined information they receive on the objectives and process of the with the one formed by the nursing grids described previ- research, more than to the researcher’s position. Generally ously (Class 2). It also includes grids with a post-positivist/ speaking, the content of the grids refers back to general cri- interpretivist approach (pp2, χ = 14), but mostly stemming teria for qualitative research, and to criteria specifically from public health journals (pubh, χ = 13), such as the grids adapted to quantitative or mixed methodologies. Finally, this by Tong, Sainsbury, and Craig (2007), Daly et al. (2007), or class stands out from the rest because of its roots in issues the Critical Appraisal Skills Programme (CASP; 2006). This related to methods, dominant both among the authors of the group of grids is composed of criteria that differ from those grids stemming from positivist approaches and constructivist in the public health grids of Class 4 (med, nurs, pubh, c1), approaches. This may explain its diverse character, given and are more closely linked to methodology and sampling that it includes grids from post-positivist (pp1), post-modern issues. The main significant terms of this class highlight the (c3), and critical constructivist (c2) paradigms. technical considerations surrounding sampling and its justi- Our results show a certain order among the six classes. In fication: sample, select, sampling, size, inclusion, purposive, Quadrants C and D, there is a first group that includes grids case, exclusion, convenience, age, extreme, or recruit. In whose dominating theoretical model is identified as con- this group, it is important to explain the logic governing the structivist, that is, they base their relationship with reality on selection of the target sample, what the inclusion and exclu- the idea that the human being is constructed collectively, and sion choices are, and why and how the populations, events, that the best we can do is interpret in a more or less shifting people, and so on were selected. social, cultural, or temporal context. The criteria of this Often closely linked to positivist criteria, the criteria in group of grids present the researcher as an integral part of his Class 6 marks the need to represent the population (some- or her study subject, and in constant interaction with it. As a times related to statistical analysis) to ensure the transferabil- guarantee of the “authenticity” of the research data and ity of the results. Like for the grids in Class 2, step description results, these grids emphasize the faithful reporting of the Santiago-Delefosse et al. 9 Table 3. Distribution of the Paradigmatic and Epistemological Positions According to a Continuum: Summary of the Results. Paradigmatic position Post-positivism (with different dimensions) Constructivism, critical constructivism, and post-modernism Epistemological Post-positivism/ Constructivism/ positions Post-positivism Interpretivism Interpretivism Critical constructivism Post-modernism Alceste® code pp1 pp2 c1 c2 c3 Number of grids 13 grids 57 grids 31 grids 27 grids Five grids Characteristics Two methodologies Mixed criteria, some Specific qualitative criteria Specific qualitative Explicit demand for a can be linked in one from the interpretivist applied to qualitative criteria applied to post-modern approach research, or adapted field and others from research only. Used in a qualitative research (depends on the criteria stemming from the constructivist field. general interpretative-type only. Constructivist authors’ declaration) a positivist field: Mixed Mixed epistemologies approach and/or critical and explicit methods (not very explicit) approaches Quadrants Quadrant A Quadrant B Quadrants C + D Quadrants A + B Quadrant A Class detail Class 3 (partial) Classes 2, 6 (5 partial) Classes 1, 4, (5 partial) Classes 2, 3 (partial) Class 3 (partial) Procedures/methods Procedures/methods Meaning/context/use Procedures/methods Procedures/methods Disciplinary field Research methodology Nursing, public health, Psychiatry/psychology, Nursing, research Research methodology psychiatry/psychology medicine, public health, methodology nursing Significant terms Procedure, data, record, Literature, framework, Experience, meanings, Phenomenon, Procedure, data, record, interview, document, clinical, rationale, cultural, reflexivity, explicit, philosophy, interview, document, protect, ethics sample, select, recruit, practice, contribute, observation, protect, ethics random patients, useful procedure, confidential experiences lived by participants and the clarification of the a situation using an appropriate sample selection method. In researcher’s personal implications and biases, as well as his the second case, the interpretation is linked to meaning, in or her experience and training in terms of interpretative theo- connection with cultural comprehension; it calls on the ries. The second group (Quadrants A and B) includes grids researcher’s reflexivity and commitment, helps to clarify the whose dominating theoretical model was identified as post- practice, and allows for useful applicability for patients. positivist, with the addition of criteria stemming from post- More generally, we also notice that few grids were classified positivist and interpretivist approaches for some. The criteria in/under the “post-positivist” and “post-modern” paradigms. in this group of grids suggest looking at reality as a series of By contrast, a greater number were included in the columns existing facts and/or events irrespectively of the researcher. that bring together mixed approaches. An important finding The goal is to discover or at least approximate. Therefore, stemming from the classification of grids based on their methodological rigor is a guarantee of validity, with empha- underlying research conceptions was their distribution along sis put on the logic of thought and the transparent description a continuum from post-positivism to post-modernism, of the research process. including interpretivist and constructivist approaches, and Table 3 summarizes the results and highlights how the representing a Bell curve (Table 3). grids are split according to a paradigmatic and epistemologi- cal continuum, all the while showing that the boundaries Discussion between post-positivism and constructivism are not clearly specified. This result is even more important in that the sum Our content analysis and Alceste® lexicometric analysis of grids pp2 (post-positivism/interpretivism) and c1 (con- confirm the presence of major discrepancies between the structivism/interpretivism) is 88/133. As such, the “interpre- quality criteria grids examined and their relationship with the tive” position seems to represent the quality criteria grids for various paradigmatic and epistemological approaches identi- qualitative research the most. Moreover, given that the soft- fied in the grids. References to the major research paradigms ware Alceste® sorted the data “Interpretation” into two sepa- or epistemological positions of the researchers are generally rate categories, we acknowledged that there are at least two implicit, and are not easily identified because of the com- distinct meanings for this concept. The first is more or less plexity of the content and structure of the grids. Indeed, on solipsistic, that is, the researcher knows and can interpret the one hand, the same approach can be identified in grids data using the right theory. The second is more specific in belonging to different classes, and on the other hand, differ- that the interpretation is actually a co-constructed language ent approaches can be merged into the same grid. Although between the researcher/subject context. In the first case, the the six classes stemming from the lexicometric analysis can interpretation is linked to the examination of a preexisting be placed in a consistent order based on the vertical partition reality. Linked with the literature and theory, it must explain of the previous factorial design (Figure 1) and the significant 10 SAGE Open terms related to them (Figure 2), this distribution remains appear, it is directly related to the methodological pre- global. Indeed, the dispersion of items from the same grid to caution issues in a post-positivist context. Hence, the different classes reveals a lack of uniformity in the structure realization that ethics is more about the researcher of the grids and the underlying theoretical approaches in the making sure that he or she completed all the adminis- content of the criteria. This diversity shows how unclear the trative work falling under it, than a question of position underlying paradigmatic and epistemological approaches in with regards to the research. Therefore, it seems that the evaluation grids are and how complex their links are. the term reflexivity more often refers back to a notion Consequently, it is hard to draw a clear line between them close to ethics, which is then taking the research into and to place them in only one category. We can hypothesize account, its impacts, use, and acceptability among the that the authors of the grids themselves seem to have over- actors and determining the co-constructed meaning. looked this construction diversity. Yet, it may “objectively” 4. Finally, our results show the need to differentiate explain the experts’ “subjective” difficulty when they say “mixed methodologies” and “mixed epistemologies” that they do not have the tools needed to evaluate qualitative as they do not have the same consequences. The call research. The latter notices the lack of consistency between for mixed methodologies conducted rigorously is now the different grids and even sometimes within the same crite- more established (Creswell, 2013; Creswell & Plano ria grid. Still, beyond this lack of consistency, our results Clark, 2010; Paterson & Pentland, 2008; Tashakkori highlight various key points. & Teddlie, 2010). On the contrary, the reference to mixed epistemologies is more complex and only 1. The 133 grids can be categorized on a continuum. slightly identified. Yet, the compatibility of the para- First, the grids underlie theoretical approaches close digms is not only a question of philosophy but also to quantitative work, with the criteria “translated” refers back to the practical use of the grids as certain into qualitative, while being rather focused on the criteria are contradictory. Among these grids, some description of the research steps, with an emphasis on intersect with various paradigmatic and epistemologi- methodology and virtually no criteria referring back cal positions, and their authors propose a combination to the theories, epistemology, and values. Second, of the quality criteria that correspond both to positiv- grids underlie theoretical approaches and criteria that ist and constructivist approaches, with, for each grid, are more specific to qualitative work, with reference a conception that slightly differs from the combina- to the researchers’ training and their research posi- tion to adopt. The fact that criteria linked to different tion, little or no description of the steps, and an epistemologies can coexist in the same grid, visibly emphasis on the values, epistemology, and theoreti- unbeknownst to their author, reveals the existence of cal analysis. confusion between the terms used and questions the 2. The notion of interpretation is a pivotal point that dif- validity of their use. This confusion can stem from (a) ferentiates the research positions, while contributing the different meanings of a term (as seen for the terms to their semantic confusion. It seems that interpretiv- interpretive and ethics); or (b) the authors’ action of ism can be given at least two meanings. It can be an successively borrowing from different grids to create interpretation by the researcher, according to his or a new one, without thinking of the consequences of her theories regarding the words spoken and the this action; or (c) a lack of knowledge of the philo- observations made of a preexisting reality, and can sophical notions and frameworks of the terms used for also be a co-constructed search for meaning between the paradigms, methods, and theories; or finally (d) the researcher (reflexive) and actors, according to a diversified knowledge of the notions based on the dis- context and in a practical use objective. This last defi- ciplinary origin of each group. All of these parameters nition could also correspond, in part, to the “partici- contribute to creating a diverse and ultimately very pative” position proposed by Lincoln, Lynham, and nonconsensual image of the evaluation grids for qual- Guba (2011). itative research in the health sciences. 3. We had considered the call for “ethical” criteria in the grids as an indicator of the researcher’s position with Conclusion regards to the research, its absence indicating an objectivist position and its presence a position that The results of our research explain in large part the difficul- takes into account the values of the research and the ties in coming to a consensus on the quality criteria for quali- impacts. Our results question this premise. It also tative research. The conception diversity, both inter- and seems here that the presence of criteria linked to eth- intra-grid, prevents the coherence of the grids. Our results ics in the grids indicates at least two different mean- show that the grids examined are not representative of a dis- ings, which would also make this axis a pivotal point cipline but rather of a position with regards to qualitative among the grids. Indeed, the term ethics only rarely research and objectives inherent to each one. They thus seems significant in the corpus and when it does translate the vision that the authors of the grids have of Santiago-Delefosse et al. 11 qualitative research, one that is partial at times, and almost Foundation for the financial support (Grant N° CR13Ι1_126983) provided to this study. always explicit, as per the diverse results obtained. Yet, the goal is now to provide researchers with explicit and adequate References tools to assess their work, and to provide experts (and read- ers) with grids that allow them to evaluate the quality of Aubert-Lotarski, A., & Capdevielle-Mougnibas, V. (2002, March). qualitative work in various health research fields according Dialogue méthodologique autour de l’utilisation du logiciel to comparable criteria. Although standardizing the grids is Alceste en sciences humaines et sociales: “lisibilité” du cor- pus et interprétation des résultats [Methodological dialogue not the objective, their acknowledgement requires clarifying on the use of Alceste software in social and human sciences: the terms and comparing them to show the validity of their “Readability” of the corpus and interpretation of results]. 6èmes use and their scientific reliability. The identification of the Journées Internationales d’Analyse Textuelle [6th International paradigms underlying the quality criteria conducted in this Conference on Textual Analysis], Saint-Malo, France. document is a first step toward understanding the diversity of Beck, C. T. (2009). Critiquing qualitative research. AORN Journal, the qualitative research evaluation grids in the health sci- 90, 543-554. ences. As this article is only one part of a more extended Braun, V., & Clarke, V. (2006). Using thematic analysis in psychol- research project conducted from 2011 to 2014, these findings ogy. Qualitative Research in Psychology, 3, 77-101. therefore cover one aspect of our study. Another part of the Chamberlain, K., Stephens, C., & Lyons, A. C. (1997). research project included the participation of international Encompassing experience: Meanings and methods in health experts in qualitative research in the health sciences. Each psychology. Psychology and Health, 12, 691-709. Cicourel, A. (1964). Method and measurement in sociology. New expert and their peers (56 persons) assessed the grids in their York, NY: Free Press. own fields. As a result, these experts considered 12 quality Cobb, A. K., & Hagemaster, J. N. (1987). Ten criteria for evaluating criteria consensual. The main outcomes have been published qualitative research proposals. Journal of Nursing Education, on a website aimed at the dissemination of this research: 26, 138-143. https://wp.unil.ch/qualityofqualitativeresearch. However, Cohen, D. J., & Crabtree, B. F. (2008). Evaluative criteria for quali- despite its insightful results, our research has two main limi- tative research in health care: Controversies and recommenda- tations. First, this study was carried out on the basis of 133 tions. The Annals of Family Medicine, 6, 331-339. quality grids published in the health sciences field, but we Creswell, J. W. (2013). Research design: Qualitative, quantitative, did not conduct a systematic meta-analysis on all the grids and mixed methods approaches. Thousand Oaks, CA: SAGE. published in the field. This may have influenced the results Creswell, J. W., & Plano Clark, V. L. (2010). Designing and con- with regard to the distribution of grids by disciplines and the ducting mixed methods research. Thousand Oaks, CA: SAGE. Critical Appraisal Skills Programme. (2006). Qualitative Research number and diversity of the criteria. Second, we chose Checklist. Oxford, UK: Public Health Resource Unit. Retrieved Denzin and Lincoln’s (1994/2011) classification, close to our from http://media.wix.com/ugd/dded87_29c5b002d99342f788 own approach, and this choice resulted in our classification c6ac670e49f274.pdf of the epistemological approaches underlying the grids that Curtin, M., & Fossey, E. (2007). Appraising the trustworthiness would have perhaps differed if based on other fundamental of qualitative studies: Guidelines for occupational therapists. works on paradigms in qualitative health. Australian Occupational Therapy Journal, 54, 88-94. In conclusion, more work remains to be done, however, Daly, J., Willis, K., Small, R., Green, J., Welch, N., Kealy, M., for as long as the concepts considered to be major by the & Hughes, E. (2007). A hierarchy of evidence for assessing authors of the grids are not clearly explained, the grids will qualitative health research. Journal of Clinical Epidemiology, only be of limited use in improving and establishing qualita- 60, 43-49. tive research in the health sciences. Denzin, N. K., & Lincoln, Y. S. (Eds.). (2011). The SAGE handbook of qualitative research. London, England: SAGE. (Original work published 1994) Acknowledgments Devers, K. J. (1999). How will we know “Good” qualitative The authors would like to thank Dr Valérie Capdevielle-Mougnibas research when we see it? Beginning the dialogue in health ser- from the University of Toulouse (France), for her valuable help in vices research. Health Services Research, 34, 1153-1188. processing the data for Alceste®, and Dr Maria del Rio Carral from Dixon-Woods, M., Shaw, R. L., Agarwal, S., & Smith, J. A. (2004). the University of Lausanne for her careful reading of this manuscript. The problem of appraising qualitative research. Quality Safety Health Care, 13, 223-225. Declaration of Conflicting Interests Elliott, R., Fischer, C. T., & Rennie, D. L. (1999). Evolving guide- lines for publication of qualitative research studies in psychol- The author(s) declared no potential conflicts of interest with respect ogy and related fields. British Journal of Clinical Psychology, to the research, authorship, and/or publication of this article. 38, 215-229. Flick, U. (2006). Quality criteria in qualitative research. In U. Flick Funding (Ed.), An introduction to qualitative research (pp. 367-389). The author(s) disclosed receipt of the following financial support London, England: SAGE. for the research, authorship, and/or publication of this article: The Flick, U. (2007). Managing quality in qualitative research. author(s) express their gratitude to the Swiss National Science Thousand Oaks, CA: SAGE. 12 SAGE Open Fossey, E., Harvey, C., McDermott, F., & Davidson, L. (2002). [The “lexical worlds” and their “logics” through the statistical Understanding and evaluating qualitative research. Australian analysis of a corpus of stories of nightmares]. Langage et & New Zealand Journal of Psychiatry, 36, 717-732. société/Language & Society, 60, 5-39. Garfinkel, H. (1967). Studies in Ethnomethodology. Cambridge, Robert, A., & Bouillaguet, A. (2002). L’analyse de contenu MA: Polity Press. [Content analysis]. Paris: Presses Universitaires de France. Garric, N., & Capdevielle-Mougnibas, V. (2009). La variation Russell, C. K., & Gregory, D. M. (2003). Evaluation of qualitative comme principe d’exploration de corpus: Intérêts et limites de research studies. Evidence-Based Nursing, 6, 36-40. l’analyse lexicométrique interdisciplinaire pour l’étude du dis- Sandelowski, M., & Barroso, J. (2007). Appraising reports of cours [Variation as a principle to explore the research corpus: qualitative studies. In M. Sandelowski & J. Barroso (Eds.), Interests and limitations of the interdisciplinary lexicometric Handbook for synthesizing qualitative research (pp. 75-132). analysis to study discursive material]. Corpus, 8, 105-128. New York, NY: Springer. Green, J., & Britten, N. (1998). Qualitative research and evidence Santiago-Delefosse, M. (2007). Recherche en psychologie et based medicine. British Medical Journal, 316, 1230-1232. turbulences paradigmatiques [Research in psychology and Greenhalgh, T., & Taylor, R. (1997). How to read a paper: Papers Paradigmatic turbulences]. Bulletin de Psychologie/Bulletin of that go beyond numbers (qualitative research). British Medical Psychology, Hors série/Special issue, 97-102. Journal, 315, 740-743. Santiago-Delefosse, M., Bruchez, C., Gavin, A., & Stephen, S. Hamberg, K., Johansson, E., Lindgren, G., & Westman, G. (1994). L. (2015). Diversity of the Quality Criteria in Qualitative Scientific rigour in qualitative research: Examples from a study Research in the Health Sciences: Lessons from a Lexicometric of women’s health in family practice. Family Practice, 11, Analysis Composed of 133 Guidelines. Forum: Qualitative 176-181. Social Research, 16(2), Art.11. Kuhn, T. (1962). The structure of scientific revolutions. Chicago, Santiago-Delefosse, M., & Rouan, G. (Director). (2001). Les IL: University of Chicago Press. méthodes qualitatives en psychologie [Qualitative methods in Lincoln, Y. S., Lynham, S. A., & Guba, E. G. (2011). Paradigmatic psychology]. Paris: Dunod. controversies, contradictions, and emerging confluences, revis- Schneider, K. J. (1998). Toward a science of the heart. Romanticism ited. In N. K. Denzin & Y. S. Lincoln (Eds.), The SAGE handbook and the revival of psychology. American Psychologist, 53, of qualitative research (pp. 97-128). London, England: SAGE. 277-289. Mays, N., & Pope, C. (2000). Qualitative research in health care: Schutz, A. (1962). Collected papers (2 Vols.). The Hague, The Assessing quality in qualitative research. British Medical Netherlands: Martinus Nijhoff. Journal, 320, 50-52. Silverman, D. (1993). Interpreting qualitative data: Methods for Meyrick, J. (2006). What is good qualitative research? A first step analyzing talk, text and interaction. London, England: SAGE. towards a comprehensive approach to judging rigour/quality. Smith, J., Harré, R., & Van Langenhove, L. (1995). Rethinking Journal of Health Psychology, 11, 799-808. methods in psychology. London, England: SAGE. Miles, M. B., & Huberman, A. M. (1994). Qualitative data analy- Stiles, W. B. (1999). Evaluating qualitative research. Evidence- sis: An expanded sourcebook. Thousand Oaks, CA: SAGE. Based Mental Health, 2, 99-101. Morse, J. M., Barrett, M., Mayan, M., Olson, K., & Spiers, J. (2002). Tashakkori, A., & Teddlie, C. (Eds.). (2010). SAGE handbook of Verification strategies for establishing reliability and validity mixed methods in social & behavioural research. Thousand in qualitative research. International Journal of Qualitative Oaks, CA: SAGE. Tong, A., Sainsbury, P., & Craig, J. (2007). Consolidated criteria Methods, 1(2), 1-19. for reporting qualitative research (COREQ): A 32-item check- Murray, M. (2000). Level of narrative analysis in health psychol- list for interviews and focus groups. International Journal for ogy. Journal of Health Psychology, 5, 331-342. Quality in Health Care, 19, 349-357. Paterson, M., & Pentland, W. (2008). The use of hermeneutics in Van den Hoonaard, W., Given, L., Lévy, J., McGinn, M., O’Neill, P., a mixed methods design. The Qualitative Report, 13, 116-134. & Palys, T. (2004). Giving voice to the spectrum: Report of the Retrieved from http://www.nova.edu/ssss/QR/QR13-1/vonz- social sciences and humanities Research Ethics Special Working weck.pdf Committee to the Interagency Advisory Panel on Research Popay, J., Rogers, A., & Williams, G. (1998). Rationale and stan- Ethics. Ottawa, Ontario, Canada: Interagency Advisory Panel dards for the systematic review of qualitative literature in health and Secretariat on Research Ethics. Retrieved from http://www. services research. Qualitative Health Research, 8, 341-351. sfu.ca/~palys/SSHWC-GivingVoice-2004.pdf Pope, C., & Mays, N. (2006). Qualitative research in health care. Walsh, D., & Downe, S. (2006). Appraising the quality of qualita- Oxford, UK: Blackwell. tive research. Midwifery, 22, 108-119. Porter, S. (2002). Critical realist ethnography. In T. May (Ed.), Yardley, L. (2008). Demonstrating validity in qualitative psychol- Qualitative research in action (pp. 53-72). London, England: ogy. In J. A. Smith (Ed.), Qualitative psychology: A practical SAGE. guide to research methods (pp. 235-251). London, England: Reinert, M. (1990). Alceste. Une méthodologie d’analyse des SAGE. données textuelles et une application: Aurélia de Gérard de Nerval [Alceste. A methodology for analyzing textual data Author Biographies and its related application: Aurélia by Gérard de Nerval]. Bulletin de méthodologie sociologique/Bulletin of Sociological Marie Santiago-Delefosse has been a full professor in health psy- Methodology, 26, 24-54. chology at the University of Lausanne (Switzerland) since 2003. Reinert, M. (1993). Les “mondes lexicaux” et leur “logique” à trav- Her empirical research (fertility, HIV, chronic pain, etc.) has led her ers l’analyse statistique d’un corpus de récits de cauchemars to propose an alternative theorization of the biopsychosocial model. Santiago-Delefosse et al. 13 Her work pays particular attention to the history of ideas and to allowed her to develop skills and interests in various domains, such epistemology in psychology; it is mostly qualitative and based on a as sexology, couple counseling, ethics, critical health psychology, historico-cultural and phenomenological approach. She is the and qualitative research among others. author of more than 100 articles and several books. Sarah Lilian Stephen worked as Junior SNSF (Swiss National Christine Bruchez has been a teaching assistant, then research Science Foundation) researcher in an SNSF-funded project directed coordinator, in health psychology at the University of Lausanne by Prof. Santiago-Delefosse at the Faculty of Social and Political since January 2004. She has completed a DESS (advanced graduate Sciences at the University of Lausanne. Since 2011, she is also a diploma) in clinical psychology, and is currently a PhD candidate PhD student based at the Faculty of Business and Economics at the under the supervision of Prof. Marie Santiago-Delefosse. Her University of Lausanne. Her interests include quality of quantita- research interests in epistemology and methodology have led her tive and qualitative research and interdisciplinary research meth- toward the comparison of qualitative methods and the experiment- ods, as well as paradigms underlying research. ing with new research tools in psychology. Pauline Roux has been a Junior Lecturer at the University of Amaelle Gavin has been research coordinator in health psychology Lausanne (Switzerland) since March 2015. She presented her thesis at the University of Lausanne since October 2012. She is currently in social psychology in 2013 at the University of Lyon (France). training in clinical sexology and since February 2015, she occupies Her research interests include patient-doctor relationship and com- a position in the university as a PhD candidate under the supervi- munication, medically unexplained illnesses, and qualitative meth- sion of Prof. Marie Santiago-Delefosse. Her different trainings odology and triangulation. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png SAGE Open SAGE

Complexity of the Paradigms Present in Quality Criteria of Qualitative Research Grids:

Download PDF
13 pages

Loading next page...
 
/lp/sage/complexity-of-the-paradigms-present-in-quality-criteria-of-qualitative-Oce5QyRiXR

References (54)

Publisher
SAGE
Copyright
Copyright © 2022 by SAGE Publications Inc, unless otherwise noted. Manuscript content on this site is licensed under Creative Commons Licenses.
ISSN
2158-2440
eISSN
2158-2440
DOI
10.1177/2158244015621350
Publisher site
See Article on Publisher Site

Abstract

With qualitative methods being increasingly used in health science fields, numerous grids proposing criteria to evaluate the quality of this type of research have been produced. Expert evaluators deem that there is a lack of consensual tools to evaluate qualitative research. Based on the review of 133 quality criteria grids for qualitative research in health sciences, the authors present the results of a computerized lexicometric analysis, which confirms the variety of intra- and inter-grid constructions, including within the same field. This variety is linked to the authors’ paradigmatic references underlying the criteria proposed. These references seem to be built intuitively, reflecting internal representations of qualitative research, thus making the grids and their criteria hard to compare. Consequently, the consensus on the definitions and the number of criteria becomes problematic. The paradigmatic and theoretical references of the grids should be specified so that users could better assess their contributions and limitations. Keywords qualitative research quality, evaluation criteria, guidelines, paradigms, Alceste® lexicometric analysis As part of an ongoing research (this document represents Introduction part of a larger research funded by the Swiss National Science The use of qualitative research methods in the social and Foundation (www.snf.ch), 2011-2014: “Quality of Qualitative human sciences considerably increased as of the 1990s Research in the Health Sciences: Which Evaluation Criteria?”), (Chamberlain, Stephens, & Lyons, 1997; Flick, 2007; Miles & we examined 133 evaluation grids in five major health science Huberman, 1994; Murray, 2000; Silverman, 1993; Smith, fields: “medicine,” “public health,” “nursing,” “psychiatry Harré, & Van Langenhove, 1995). This dissemination quickly and psychology,” and “research methodology.” To better reached the health field through nursing, medicine, and public understand their variability and its origins, we conducted a health. This renewed interest in qualitative research promoted computer-assisted lexicometric analysis, our premise being the publication of numerous “quality criteria” grids/guidelines that the diversity of the grids should not only be linked to the (we use the words criteria grids, guidelines, and checklists as various disciplinary fields but is also intrinsic to various synonyms) to evaluate the quality of a qualitative research, qualitative research conceptions among grid designers. In project, or article (e.g., in scientific journals such as the British this article, we start by presenting a brief status report on the Medical Journal, Psychology and Health, Family Practice, health science research field, in terms of quality criteria. We Sociology of Health and Illness, etc.). This abundance of then present our methodology. In the third part, we present guidelines, however, resulted in the use of numerous terms our main results. Finally, we discuss the inherent limitations and grids, which failed to contribute to the clarification of the main consensual criteria to evaluate qualitative research. This leads to a certain lack of recognition of this work in publica- tions and for fundraising purposes, the solicited reviewers University of Lausanne, Switzerland declaring that they are ill-informed or do not have the tools to Corresponding Author: evaluate qualitative research (Green & Britten, 1998; Christine Bruchez, Institute of Psychology, University of Lausanne, Greenhalgh & Taylor, 1997; Mays & Pope, 2000; Popay, Geopolis, 1015 Lausanne, Switzerland. Rogers, & Williams, 1998). Email: christine.bruchez@unil.ch Creative Commons CC-BY: This article is distributed under the terms of the Creative Commons Attribution 3.0 License (http://www.creativecommons.org/licenses/by/3.0/) which permits any use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access pages (https://us.sagepub.com/en-us/nam/open-access-at-sage). 2 SAGE Open of the various research conceptions highlighted within the Given these facts, our research focused on analyzing the different disciplines they represent. links between the health science disciplinary fields and the major paradigms guiding grid designers in determining the criteria used to evaluate qualitative research. We used Problem: A Shift in the Quarrels From Kuhn’s (1962) general definition, which states that a para- the Last Century? digm is a set of disciplinary convictions shared by the scien- tific community at a given time during its history. Any In a context of an increase in the publication of qualitative science is stabilized by sharing the same paradigm, which studies both in books and recognized scientific journals, marks its field and investigation procedures. This paradigm debates on the relevance of developing (or not) specific cri- is developed by groups of socially dominating researchers teria for qualitative methodologies have evolved continu- who control the transmission of knowledge through research ously (Devers, 1999; Dixon-Woods, Shaw, Agarwal, & Smith, funds, positions, and publications. This definition captures 2004; Mays & Pope, 2000; Yardley, 2008). Certain authors the epistemological and paradigmatic choice that is made in deem that the validity and reliability concepts stemming from all research conducted, whether this choice is deliberate or quantitative methodologies are suitable to reach a high level of the outcome of reflection, or much more implicit. rigor in qualitative research (Miles & Huberman, 1994; Morse, Barrett, Mayan, Olson, & Spiers, 2002; Porter, 2002); how- ever, another group of authors finds that applying the “reli- Method ability” and “validity” criteria directly is unacceptable for qualitative studies. This last group argues for differentiated Make-Up of the Corpus criteria for qualitative work based on specific philosophical In total, 133 grids (checklists or guidelines) were examined and methodological foundations. They argue their position by in the various health fields, from the MEDLINE, PsychInfo, insisting on the structured thought needed in all paradigm CINAHL, PeruniL, ScienceDirect, and Web of Sciences data- transfer, especially to apply the same criteria to social and bases, using the keywords “health,” “qualitative research,” human sciences as those applied to natural sciences (Cicourel, “assessment,” “appraising,” “quality criteria,” “peer review 1964; Garfinkel, 1967; Hamberg, Johansson, Lindgren, & process,” “guideline,” “evaluation,” and “standards.” Data Westman, 1994; Schutz, 1962; Van den Hoonaard et al., 2004). were used according to the following inclusion criteria: (a) Somewhere in the middle, a third group of authors argues that grids published for health scientists (medicine, nursing, evaluation criteria can be transferred from quantitative to qual- health psychology, health sociology and anthropology, pub- itative methods, as long as they are adapted (Pope & Mays, lic health, etc.), (b) guidelines that were sufficiently devel- 2006). oped and commented, and (c) work published by health In following these debates, we notice that the discrepan- journals, institutes, and organizations whose scientific qual- cies regarding the quality criteria seem to reflect different ity has been validated. conceptions of qualitative research (Santiago-Delefosse, Whether they are taken from scientific journals (instruc- 2007; Schneider, 1998). Yet, these discrepancies may tions to authors), articles for authors and researchers to explain why it is so hard to build consensus that will allow improve their qualitative work or as an evaluation support for the evaluation of qualitative research quality within the tool, or methodology manuals, most of the grids examined are scientific community, especially because they remain presented as lists of criteria to meet to produce “good qualita- obscure in literature on quality criteria grids for qualitative tive research” that is rigorous and reliable. Most of these grids research. are accompanied by a text explaining the criteria (definitions Some work highlights the variability of existing grids and uses) and/or a theoretical or methodological text that is (Cohen & Crabtree, 2008; Flick, 2007; Mays & Pope, 2000; sufficiently detailed to support the quality criteria analysis. Popay et al., 1998) and the difficulty in comparing them. The Most of the grids examined come from publications dat- inconsistency of the number and objectives of the criteria ing from 1993 to 2011, which correspond to the increase in used, the absence of consensual and explicit definitions, and debates on the quality of qualitative research in the 1990s. the absence of construction consistency between the grids of Most grids come from medical journals (45), nursing litera- one disciplinary field were identified in a previous publica- ture (39), research methodology journals/literature (20), pub- tion (Santiago-Delefosse, Bruchez, Gavin, & Stephen, 2015). lic health literature (19), and psychiatry/psychology literature Therefore, research conceptions underlying these grids may (10). Most evaluation grids were published in medical and be questioned as well as the epistemological and paradig- nursing literature. matic positions that guided their designers. This questioning We submitted our corpus to two analyses: a written analy- is still highly relevant as the lack of consensus regarding a sis with inter-evaluator agreement that allowed us to deter- certain number of criteria, even minimal, is an argument to mine coding categories for our second analysis and a which article and fundraising evaluators refer and which computer-assisted lexicometric analysis (Alceste®). motivates the rejection of projects. Santiago-Delefosse et al. 3 “post-positivism/interpretivism” bringing together grids that Thematic Analysis of the Content and Theoretical combine both approaches; the third category entitled “con- and Methodological Foundations of the Analysis structivism/interpretivism” was built upon a similar basis; Categories the fourth one was “critical constructivism”; and the fifth one Using Alceste® requires prior coding of the text to show the was the “post-modern” approach including grids with crite- software which variables must be identified in the text. To do ria dealing with radical relativism/constructivism (consider- this and identify the variables that would be most relevant to ing, for example, that qualitative research is too singular to our questioning, we conducted a first thematic analysis of the be evaluated). No grid was identified in the “positivist” epis- grid content. This first content analysis helps to identify the temological approach. “contextual variables” specific to each text (i.e., author, dis- cipline, and paradigm; “paradigm” here refers to a system of Presentation of Alceste® and Relevance of the fundamental beliefs belonging to the researchers and used as Lexicometric Analysis Based on Our Hypothesis the basis for their theoretical and methodological choices in We used an analysis produced by Alceste® to process the the conception of the grids). The variables are then coded to contents of this corpus. Alceste® is textual or statistical data make them compatible with the computerized analysis. The analysis software, originally designed by Max Reinert, of the content analysis was conducted independently by three CNRS, in France (Reinert, 1990, 1993). It has been used in researchers. The results were triangulated and discussed col- the human and social sciences since the 1990s. It counts lectively to come to an inter-evaluator agreement (Braun & words in a frequential manner and helps to obtain analysis Clarke, 2006; Robert & Bouillaguet, 2002). units that are based on formal criteria. It uses an inductive The reference framework used for this content analysis and recursive approach and helps to identify co-occurrences, was the one used for the work conducted by Denzin, Guba, or word associations in a sentence, by processing the words and Lincoln. Indeed, besides their heuristic scope, they are according to their similarities and differences. Technically, the most often cited in literature on the importance of para- Alceste® breaks the corpus into fragments that are relatively digms and epistemologies in qualitative methods (Denzin & similar in size, referred to as “context units” (e.c.u. the small- Lincoln, 1994/2011, Santiago-Delefosse & Rouan, 2001). est statistical units created by the software, based on a com- In 1994, Denzin and Lincoln proposed a classification promise between the syntactic form [proper punctuation] and that included two major paradigms subdivided into four epis- the statistical constraints [these units must be of comparable temological research positions: the positivist/post-positivist size]). These same fragments are then reclassified statisti- paradigm (including the positivist or post-positivist episte- cally and split into classes whose specific vocabulary is as mological approaches) and the interpretivist/constructionist different as possible. These classes are called “lexical paradigm (including the critical or constructivist epistemo- worlds” and are considered to present an “image” of the rep- logical approaches). Notice that in 2011, they added a “par- resentations contained in the text, the ideas and the corpus’ ticipative” epistemological position to the interpretivist/ major themes (Garric & Capdevielle-Mougnibas, 2009). constructivist paradigm. A certain number of criteria differ- Second, it defines a link to what software developers refer to entiate these epistemological and paradigmatic approaches. as “contextual variables” (data elements known by the Among these criteria, three major axes caught our attention researchers and used as instructions for the software so that as they can be operationalized and help to compare the crite- the lexical analysis can be conducted with these variables in ria in the grids: (a) the form and nature of reality (following mind). These variables are used to identify the texts and are the preexisting or co-constructed one), (b) the researcher’s related to their content. The researchers introduce them type of position regarding research (outside observer vs. par- according to their relevance to the research questions. The ticipant involved in the co-construction), and (c) the use of classes identified by the software must then be examined and the research (discovery of inherent truths vs. work on subjec- linked to the co-occurrences to give them meaning and tivity and the impact of its construction on the world, work explain their differences (Aubert-Lotarski & Capdevielle- on the co-construction of realities, work on dominating Mougnibas, 2002). Notice that the software’s results are forces and empowerment). We added a fourth axis regarding based on frequencies rather than weighting, which could be the presence of ethical questions, which also indicate the considered a limitation. researcher’s position toward the research (absent when the We deemed it relevant to use this software because of its research is considered an objective report on what exists vs. textual classification abilities, given that we hypothesized always present when the researcher, his or her values and the that the differences between the grids were not solely linked context are considered a part of the research at all times). to disciplinary fields in the health sciences, but represented We analyzed our corpus by focusing on these four major paradigmatic and epistemological positions within certain axes, which help to report on the conceptions underlying the disciplines, connecting certain grids of disciplines that dif- research in the 133 grids examined. As a result, we identified fered from those from which they stemmed. If this hypothe- five paradigmatic and epistemological approaches: The first sis was validated, we would obtain a classification of grids category was defined as “post-positivism”; the second one, 4 SAGE Open Table 1. Contextual Variables Used for Alceste® Coding. Five paradigmatic and epistemological approaches Paradigmatic position Post-positivism Constructivism, critical constructivism, and post-modernism Epistemological Post-positivism/ Constructivism/ position Post-positivism interpretivism interpretivism Critical constructivism Post-modernism Alceste® coding pp1 pp2 c1 c2 c3 The acknowledgement of Mixed grids mixing Grids using a knowledge Refers to a Grids that highlight an external reality to be criteria stemming approach according to constructivism social constructivism discovered using both both from a post- which there apparently approach that according to which our qualitative and quantitative positivist and exists multiple realities. adds a critical understanding of the methods that are capable interpretivist approach Our image of reality is dimension whose world not only reflects of “representing” or (hermeneutics, a construction of the goal is to bring it but also organizes “approximating” this psychoanalytic . . . human spirit, which must about a reform of and orders it. Possibly reality. The introduction ). It designates an be interpreted in a social, knowledge-building knowing what “the of qualitative methods approach according cultural and temporal in a sociocultural reality” is questioned, corresponds to “a way to which there exists context. According context. and brings us closer to to capture” this reality one dynamic and to this approach, the a form of relativism, “as best as possible”. The evolutionary reality to researcher cannot be where the researcher’s researcher and his or be discovered through separated from his or values and judgments her topic are considered the discourses. her subject and interacts change his or her independent. with it. subject. Five health disciplinary fields Disciplinary fields Medicine Nursing Public health Psychiatry/psychology Research methodology Alceste® coding Med Nurs Pubh Psy Meth Note. No grid examined corresponded to a pure positivist epistemological approach (which is normal for grids targeting the evolution of qualitative research). that would be more connected to paradigmatic/epistemologi- perspective: There are some in medicine, research methodol- cal positions than to disciplinary fields. This does not mean ogy and public health, one in nursing, but none in psychol- that certain disciplinary fields cannot be consistent among ogy/psychiatry. The post-positivist/interpretivist approach one another (i.e., the older and more solid ones). (pp2) is the most important in our corpus (57/133) and con- The following variables were used for the lexicometric cerns grids that include criteria close to the positivist field analysis conducted as part of this work: and more interpretative criteria: They are mostly found in medical journals. Constructivist/interpretivist grids (c1) are •• The five health disciplinary fields in which grids were not as numerous (31/133) and propose criteria specific to published: medicine, nursing, public health, psychia- qualitative research, integrating contextualized constructiv- try/psychology, and research methodology. ism and the researcher’s interpretative position: They are •• The five paradigmatic and epistemological approaches mostly found in the fields of nursing and medicine. Grids as identified in the thematic analysis of the content: classified in the critical constructivism category (c2) are post-positivism, post-positivism/interpretivism, con- slightly less numerous (27/133) and include criteria specific structivism/interpretivism, critical constructivism and to qualitative research, with a constructivist approach: They post-modernism. are found in all disciplines, predominantly in nursing. Finally, there is a small group of five grids that is harder to classify in Table 1 presents the “contextual variables” used and the previous categories, with a profile that is often explicitly adapted for Alceste® coding and stemming from our written geared toward a post-modern approach (c3) and published content analysis. mainly in nursing. We note that the grids are mainly split into two large groups. The “post-positivist/interpretivist (pp2)” group Results represents half of the corpus (57/133), and then the sum of During the written analysis and according to the contextual the two “constructivist (interpretivist and critical)” also variables used, we obtained a distribution of grids by “disci- represents half of the corpus (58/133). Consequently, the plinary field” and “paradigmatic and epistemological issue regarding the discrepancies between the grids may approach” (Table 2). stem from the difference between these two large groups, There are only a few post-positivist (pp1) grids in the cor- especially in the definition of the two “interpretivist” pus. They generally propose similar criteria from a positivist positions. Santiago-Delefosse et al. 5 Table 2. Distribution of the Grids by Disciplinary Field and identify those of each class. This analysis highlights the spe- Paradigmatic and Epistemological Position. cific identity of the grids and the differences/similarities between them. The dendrogram (Figure 2) shows how the six Alceste® Codes classes are organized and linked. Disciplines (number of grids) pp1 pp2 c1 c2 c3 The six classes are split into two main branches including on one hand Classes 1, 4, and 5 (Quadrants C + D) and on the Medicine (45) 4 23 11 7 0 other Classes 2, 6, and 3 (Quadrants A + B). Nursing (39) 1 9 14 11 4 In the first branch (Quadrants C + D), two classes contain- Research methodology (20) 4 10 1 4 1 ing psychology/psychiatry grids stand out: Class 1 with its Psychology/psychiatry (10) 0 4 3 3 0 constructivist/interpretivist approach (c1) and its criteria Public health (19) 4 11 2 2 0 focused on experience, the meaning of things and compre- Total = 133 13 57 31 27 5 hension; and Class 5, which is more focused on validation Note. No grid examined corresponded to a pure positivist epistemological criteria (triangulation, member-checking) and a constructiv- approach (which is normal for grids targeting the evaluation of qualitative ist approach mixed with post-positivist criteria (c1, pp2). research). pp1 = post-positivism; pp2 = post-positivism/interpretivism; Some medicine, nursing, and public health grids (Class 4) c1 = constructivism/interpretivism; c2 = critical constructivism; c3 = post- modernism. can also be found in the constructivist/interpretivist group (c1), using criteria that are more based on practices, care, research use value, and clinical contributions. Lexicometric Analysis and Distribution of the The second branch (Quadrants A + B) includes grids deal- Corpus Among Six Classes ing with a post-positivist approach: On one hand, public health grids (Class 6) with a post-positivist approach, using Once Alceste® processed the corpus, the latter was split into certain criteria specific to the interpretivist field (pp2) and six distinct classes. Figure 1 details the class distribution and describing mostly sample selection methods and inclusion/ indicates the five most significant terms (χ ) given by the exclusion criteria, and on the other nursing grids (Class 2) software for each of the six classes as well as the contextual focused on literature, work framework, research process, and variables related to them: discipline and paradigmatic being a little more related to the constructivist field (pp2, c2). approach. To make it easier to read the analysis, we also As for the research methodology grids in Class 3, they pres- numbered the quadrants of the factorial design (A, B, C, D). ent very diverse approaches, ranging from post-positivism to The factorial design (Figure 1) shows the interrelations post-modern (pp1, c3, c2), with their content focused on between the classes and their respective positions in the practices, tools and research techniques. In that respect, these quadrants. Certain classes, namely, Classes 1 and 5 in grids, compared with the other groups, seem to be more Quadrant C, and Classes 2 and 6 in Quadrant B, are closely linked through criteria based on methodological rigor alone, linked to one another, almost interwoven. As for Classes 3 than through the discipline or paradigm. and 4 (Quadrants A and D), they are more well-defined and independent. We thus notice that the classes do not necessar- It is worth noting that Quadrant D (Classes 1 and 4) ily cover the disciplines, as there are two psychology/psy- mainly corresponds to the constructivist/interpretivist posi- chiatry groups with different approaches, as well as two tion (c1) while Quadrant B (Classes 2 and 6) mainly corre- nursing grids (Classes 2 and 4) or public health grids (Classes sponds to the post-positivist/interpretivist position (pp2). 4 and 6) characterized by different paradigmatic approaches. These two quadrants include criteria from grids from four of We also notice that Class 4 includes grids from three disci- the five disciplinary fields; however, Quadrant C corre- plines: medicine, public health, and nursing. sponds to certain criteria from the psychology grids only The factorial design also indicates a bipartite distribution (Class 5) and Quadrant A to certain criteria from the research between the grids underlying approaches that are essentially methodology grids (Class 3). constructivist (c1) and focused on the researcher’s qualities These initial results show that the characteristics of the as an integral part of the research in quadrants of C and D grids do not necessarily cover a classification based on the (Classes 1, 4 and 5), and the grids with more post-positivist disciplines and that their differences or similarities are more approaches that highlight the “good research practices” linked to the paradigmatic and epistemological positions that needed to ensure the research’s validity and objectivity, in underlie them. Only two classes (5 and 3) seem to be more Quadrants A and B (Classes 2, 3, and 6). determined by disciplinary proximity (Class 5 to a lesser extent though). Consequently, these differences must be examined to determine their meaning. The following analy- Dendrogram of the Six Classes and Significant sis provides further detail on the six classes based on their Terms elementary context units (e.c.u., semantic content) and con- textual variables (disciplines and paradigms) to better under- The examination of the significant terms (χ ) present in each stand their weight in the differentiation of the grids. class can explain the specificities in the factorial design and 6 SAGE Open Figure 1. Distribution of the six classes and contextual variables. are linked to the assumptions of the researcher who is deemed Analysis of the Classes According to the to be integrated to the research both socially and culturally, Significant Terms and Contextual Variables and interacting with the subject in its context to co-construct Quadrants C and D (Classes 1, 5, and 4): “Mainly” constructivist the interpretations and help to build new theories. approaches This class proposes specific evaluation criteria for quali- Class 1 (psy, c1). This class includes a group of grids that tative research. They are linked to the meaning produced, in stems mainly from psychology/psychiatry (psy, χ = 60), a context explained by researchers who clarified the implica- with a constructivist/interpretivist approach (c1, χ = 8). This tions, and demonstrated their training in terms of interpreta- group of grids is characterized by terms such as experience, tive theories. In this group of grids, and in keeping with the meaning, effect, self, understand, change, live, preconcep- constructivist paradigm as defined, the goal of the research is tion, reflexivity, affect, instrument, and so on (i.e., in the grids to understand and “reconstruct” its initial subject, as a con- of Curtin & Fossey, 2007; Elliott, Fischer, & Rennie, 1999; sensus between the researchers and participants, and be open or Fossey, Harvey, McDermott, & Davidson, 2002). A joint to new interpretations. As such, the validation of the research review of the class’s elementary context units shows that the places a high value on the researchers’ training and experi- epistemological approaches of this class’s criteria are geared ence, as well as on the consideration of their own “biases” toward the notions of the experience lived by a subject in its and the characteristics that are likely to influence the context of meaning, culture, interpretation, and empowerment research. The criteria proposed in these grids are usually as an impact on the research participants or reader. In general, adapted to qualitative or mixed methodologies. “authenticity” and “trust” are based on the principle that the research accurately reflects the meanings and experiences lived and Class 5 (psy, c1, pp2). A second group of psychology/psy- perceived by participants. On the other hand, a self-reflective atti- chiatry grids (χ = 3), different from the first, holds a distinc- tude is recommended. Indeed, comprehension and interpretation tive place in this class. This group is also dominated by a Santiago-Delefosse et al. 7 Figure 2. Dendrogram: Distribution of the corpus into six classes with significant terms (χ ). constructivist/interpretivist approach (c1, χ = 9), but with ple, it includes the Cohen and Crabtree (2008) grid published a post-positivist-interpretivist component (pp2, χ = 4) also in medicine, the Walsh and Downe (2006) grid for nursing present, which provides this group with an approach that is and the Dixon-Woods et al. (2004) grid published in pub- increasingly marked by research bias validation and reduc- lic health. Various significant terms given by the software, tion notions (e.g., Meyrick, 2006 or Stiles, 1999). This class such as practice, knowledge, discipline, contribute, policy, presents significant terms such as member, establish, valid, social, practitioner, standard, academic, and develop, direct different, confirm, external, corroborate, truth, triangulation, the grids in this class toward notions dealing with the appli- and audit. With the examination of the class’s elementary con- cability of the research and its relevance to the disciplines text units, they help to specify the preferred objectives in these concerned. The epistemological and paradigmatic position grids that promote the intersection of numerous perspectives of these grids is constructivist/interpretivist (c1), and their (triangulation) to reach a sufficient level of trust (credibility) elementary context units generally show an interest in devel- to validate the research. In each case, the strategy involves opment, interdisciplinarity, originality, the meaning given to cross-checking the results with other sources and perspectives. events in a specific content, and a foundation built on exist- Criteria that are more specific to qualitative research and the ing knowledge. Two major axes have been identified. The constructivist approach are also proposed: member-checking first deals with the evaluation of qualitative research (reach- to see whether the interpretations are consistent, conducting ing “trustworthiness,” a sufficient level of reliability and an iterative analysis, or looking for the saturation/recurrence contextualization). The second deals with its practical and/or of the themes presented in the results. scientific use in a broad way (its purpose). In this group, as per the constructivist paradigm, research is Most grids in the class highlight the importance of the considered to be a collaborative and collective co-construction connection between the study conducted and the existing endeavor whose researcher and participants are tools in the knowledge, the fact that any new knowledge can contribute search for consistency and the integrity of the reports and to new conceptualizations or questions, lead to the develop- inferences. The content and vocabulary used by the quality cri- ment of basic theories or hypotheses, question existing theo- teria of the grids, however, also indicate a need to objectify the ries or provide methodological information leading to research through validation techniques stemming from more improvements in the practice. Certain criteria, especially in positivist approaches. The criteria proposed in these grids are public health, focus on the social mandate, moral obligation adapted to qualitative or mixed methodologies. as a rationale for health sciences. Like for the grids in Class 1, Class 6 displays a paradigmatic constructivist approach in Class 4 (med, nurs, pubh, c1). A group of grids in medicine which the goal of the research is the “reconstruction” of its 2 2 2 (χ = 10), public health (χ = 5) and nursing journals (χ = subject and the development of new interpretations. The cri- 2) holds a distinctive place in Quadrant D, while still being teria proposed in these grids are adapted to qualitative or close to Classes 1 (psy, c1) and 2 (nurs, pp2, c2). For exam- mixed methodologies. 8 SAGE Open Quadrants A and B (Classes 2, 6, and 3): “Mainly” post-positivist and data collection are also preferred in guaranteeing the approaches rigor of the research as a whole. The criteria in these grids are Class 2 (nurs, pp2, c2). This class includes a group of grids usually generally adapted to quantitative or mixed method- stemming from the nursing field (χ = 11), mainly displaying ologies both because of the paradigmatic approach (pp2) and a post-positivist/interpretivist approach (pp2, χ = 4), with the discipline concerned (pubh). criteria stemming from a critical constructivist approach (c2, χ = 3). For example, there are Cobb and Hagemaster Class 3 (meth, pp1, c3, c2). This class is well specified and (1987), Russell and Gregory (2003) and Beck’s (2009) grids. does not overlap onto any other class. On the disciplinary This class’s significant lexical content includes terms such as front, there are many grids stemming from research meth- framework, question, paper, study, review, reference, discuss, odology articles and/or literature (meth, χ = 94). From a purpose, rationale, and explain. A complete examination of paradigmatic and epistemological point of view, however, it the significant terms in Class 2 shows that the approach is remains very diversified, with an essentially post-positivist mostly marked by notions dealing with the research anchor predominance (pp1, χ = 29), but also including the pres- in a clear theoretical framework, and scientific validation ence of criteria linked to a post-modern (c3, χ = 9) or critical through a comprehensive literature review that must be used constructivist approach (c2, χ = 6). For example, there are to place the new study in a broader context and show its grids by Flick (2006) or Sandelowski and Barroso (2007). specificity. The objectives and steps must be explained and The most significant terms in this class are linked to the qual- the connections between the research and the clinic high- ity of the data (from their collection to their retention) and lighted, as well as the impact on health practices and policies the training of the researchers in terms of the methods used: (a common aspect with Class 4). In this group, the researcher record, data, note, document, collect, consent, transcript, explains the research’s logical steps to demonstrate the valid- procedure, audio, observe, confidential, or protect. ity and reliability of the reasoning. Another major objective The grids in this class provide a thorough description of of the grids in this group is to bring about new models and the research process, emphasizing the data collection tools theories and contribute to international debates. Comparing and data retention and protection methods. Indeed, data stor- the results obtained with those of other studies must ensure age must allow for independent control, and the specific their transferability. description of the process must show how the data are pro- In a mainly post-positivist/interpretivist perspective, the cessed and how they correspond to the goal of the study. In criteria in these grids focus on the reasoning of the researcher keeping with the post-positivist paradigm as described, con- and methodological rigor, with, as its validation terms and con- cern for validation is reflected through issues related to the ditions, the faithful reporting of the logic of the steps, sampling proof and quality control (of the data, processing, and train- and data collection. The criteria proposed in these grids are ing of the researcher). There are also ethical criteria among generally adapted to quantitative or mixed methodologies. the quality criteria. In this class, this criterion constantly refers back to the consent of participants as well as to the Class 6 (pubh, pp2). This class is closely intertwined information they receive on the objectives and process of the with the one formed by the nursing grids described previ- research, more than to the researcher’s position. Generally ously (Class 2). It also includes grids with a post-positivist/ speaking, the content of the grids refers back to general cri- interpretivist approach (pp2, χ = 14), but mostly stemming teria for qualitative research, and to criteria specifically from public health journals (pubh, χ = 13), such as the grids adapted to quantitative or mixed methodologies. Finally, this by Tong, Sainsbury, and Craig (2007), Daly et al. (2007), or class stands out from the rest because of its roots in issues the Critical Appraisal Skills Programme (CASP; 2006). This related to methods, dominant both among the authors of the group of grids is composed of criteria that differ from those grids stemming from positivist approaches and constructivist in the public health grids of Class 4 (med, nurs, pubh, c1), approaches. This may explain its diverse character, given and are more closely linked to methodology and sampling that it includes grids from post-positivist (pp1), post-modern issues. The main significant terms of this class highlight the (c3), and critical constructivist (c2) paradigms. technical considerations surrounding sampling and its justi- Our results show a certain order among the six classes. In fication: sample, select, sampling, size, inclusion, purposive, Quadrants C and D, there is a first group that includes grids case, exclusion, convenience, age, extreme, or recruit. In whose dominating theoretical model is identified as con- this group, it is important to explain the logic governing the structivist, that is, they base their relationship with reality on selection of the target sample, what the inclusion and exclu- the idea that the human being is constructed collectively, and sion choices are, and why and how the populations, events, that the best we can do is interpret in a more or less shifting people, and so on were selected. social, cultural, or temporal context. The criteria of this Often closely linked to positivist criteria, the criteria in group of grids present the researcher as an integral part of his Class 6 marks the need to represent the population (some- or her study subject, and in constant interaction with it. As a times related to statistical analysis) to ensure the transferabil- guarantee of the “authenticity” of the research data and ity of the results. Like for the grids in Class 2, step description results, these grids emphasize the faithful reporting of the Santiago-Delefosse et al. 9 Table 3. Distribution of the Paradigmatic and Epistemological Positions According to a Continuum: Summary of the Results. Paradigmatic position Post-positivism (with different dimensions) Constructivism, critical constructivism, and post-modernism Epistemological Post-positivism/ Constructivism/ positions Post-positivism Interpretivism Interpretivism Critical constructivism Post-modernism Alceste® code pp1 pp2 c1 c2 c3 Number of grids 13 grids 57 grids 31 grids 27 grids Five grids Characteristics Two methodologies Mixed criteria, some Specific qualitative criteria Specific qualitative Explicit demand for a can be linked in one from the interpretivist applied to qualitative criteria applied to post-modern approach research, or adapted field and others from research only. Used in a qualitative research (depends on the criteria stemming from the constructivist field. general interpretative-type only. Constructivist authors’ declaration) a positivist field: Mixed Mixed epistemologies approach and/or critical and explicit methods (not very explicit) approaches Quadrants Quadrant A Quadrant B Quadrants C + D Quadrants A + B Quadrant A Class detail Class 3 (partial) Classes 2, 6 (5 partial) Classes 1, 4, (5 partial) Classes 2, 3 (partial) Class 3 (partial) Procedures/methods Procedures/methods Meaning/context/use Procedures/methods Procedures/methods Disciplinary field Research methodology Nursing, public health, Psychiatry/psychology, Nursing, research Research methodology psychiatry/psychology medicine, public health, methodology nursing Significant terms Procedure, data, record, Literature, framework, Experience, meanings, Phenomenon, Procedure, data, record, interview, document, clinical, rationale, cultural, reflexivity, explicit, philosophy, interview, document, protect, ethics sample, select, recruit, practice, contribute, observation, protect, ethics random patients, useful procedure, confidential experiences lived by participants and the clarification of the a situation using an appropriate sample selection method. In researcher’s personal implications and biases, as well as his the second case, the interpretation is linked to meaning, in or her experience and training in terms of interpretative theo- connection with cultural comprehension; it calls on the ries. The second group (Quadrants A and B) includes grids researcher’s reflexivity and commitment, helps to clarify the whose dominating theoretical model was identified as post- practice, and allows for useful applicability for patients. positivist, with the addition of criteria stemming from post- More generally, we also notice that few grids were classified positivist and interpretivist approaches for some. The criteria in/under the “post-positivist” and “post-modern” paradigms. in this group of grids suggest looking at reality as a series of By contrast, a greater number were included in the columns existing facts and/or events irrespectively of the researcher. that bring together mixed approaches. An important finding The goal is to discover or at least approximate. Therefore, stemming from the classification of grids based on their methodological rigor is a guarantee of validity, with empha- underlying research conceptions was their distribution along sis put on the logic of thought and the transparent description a continuum from post-positivism to post-modernism, of the research process. including interpretivist and constructivist approaches, and Table 3 summarizes the results and highlights how the representing a Bell curve (Table 3). grids are split according to a paradigmatic and epistemologi- cal continuum, all the while showing that the boundaries Discussion between post-positivism and constructivism are not clearly specified. This result is even more important in that the sum Our content analysis and Alceste® lexicometric analysis of grids pp2 (post-positivism/interpretivism) and c1 (con- confirm the presence of major discrepancies between the structivism/interpretivism) is 88/133. As such, the “interpre- quality criteria grids examined and their relationship with the tive” position seems to represent the quality criteria grids for various paradigmatic and epistemological approaches identi- qualitative research the most. Moreover, given that the soft- fied in the grids. References to the major research paradigms ware Alceste® sorted the data “Interpretation” into two sepa- or epistemological positions of the researchers are generally rate categories, we acknowledged that there are at least two implicit, and are not easily identified because of the com- distinct meanings for this concept. The first is more or less plexity of the content and structure of the grids. Indeed, on solipsistic, that is, the researcher knows and can interpret the one hand, the same approach can be identified in grids data using the right theory. The second is more specific in belonging to different classes, and on the other hand, differ- that the interpretation is actually a co-constructed language ent approaches can be merged into the same grid. Although between the researcher/subject context. In the first case, the the six classes stemming from the lexicometric analysis can interpretation is linked to the examination of a preexisting be placed in a consistent order based on the vertical partition reality. Linked with the literature and theory, it must explain of the previous factorial design (Figure 1) and the significant 10 SAGE Open terms related to them (Figure 2), this distribution remains appear, it is directly related to the methodological pre- global. Indeed, the dispersion of items from the same grid to caution issues in a post-positivist context. Hence, the different classes reveals a lack of uniformity in the structure realization that ethics is more about the researcher of the grids and the underlying theoretical approaches in the making sure that he or she completed all the adminis- content of the criteria. This diversity shows how unclear the trative work falling under it, than a question of position underlying paradigmatic and epistemological approaches in with regards to the research. Therefore, it seems that the evaluation grids are and how complex their links are. the term reflexivity more often refers back to a notion Consequently, it is hard to draw a clear line between them close to ethics, which is then taking the research into and to place them in only one category. We can hypothesize account, its impacts, use, and acceptability among the that the authors of the grids themselves seem to have over- actors and determining the co-constructed meaning. looked this construction diversity. Yet, it may “objectively” 4. Finally, our results show the need to differentiate explain the experts’ “subjective” difficulty when they say “mixed methodologies” and “mixed epistemologies” that they do not have the tools needed to evaluate qualitative as they do not have the same consequences. The call research. The latter notices the lack of consistency between for mixed methodologies conducted rigorously is now the different grids and even sometimes within the same crite- more established (Creswell, 2013; Creswell & Plano ria grid. Still, beyond this lack of consistency, our results Clark, 2010; Paterson & Pentland, 2008; Tashakkori highlight various key points. & Teddlie, 2010). On the contrary, the reference to mixed epistemologies is more complex and only 1. The 133 grids can be categorized on a continuum. slightly identified. Yet, the compatibility of the para- First, the grids underlie theoretical approaches close digms is not only a question of philosophy but also to quantitative work, with the criteria “translated” refers back to the practical use of the grids as certain into qualitative, while being rather focused on the criteria are contradictory. Among these grids, some description of the research steps, with an emphasis on intersect with various paradigmatic and epistemologi- methodology and virtually no criteria referring back cal positions, and their authors propose a combination to the theories, epistemology, and values. Second, of the quality criteria that correspond both to positiv- grids underlie theoretical approaches and criteria that ist and constructivist approaches, with, for each grid, are more specific to qualitative work, with reference a conception that slightly differs from the combina- to the researchers’ training and their research posi- tion to adopt. The fact that criteria linked to different tion, little or no description of the steps, and an epistemologies can coexist in the same grid, visibly emphasis on the values, epistemology, and theoreti- unbeknownst to their author, reveals the existence of cal analysis. confusion between the terms used and questions the 2. The notion of interpretation is a pivotal point that dif- validity of their use. This confusion can stem from (a) ferentiates the research positions, while contributing the different meanings of a term (as seen for the terms to their semantic confusion. It seems that interpretiv- interpretive and ethics); or (b) the authors’ action of ism can be given at least two meanings. It can be an successively borrowing from different grids to create interpretation by the researcher, according to his or a new one, without thinking of the consequences of her theories regarding the words spoken and the this action; or (c) a lack of knowledge of the philo- observations made of a preexisting reality, and can sophical notions and frameworks of the terms used for also be a co-constructed search for meaning between the paradigms, methods, and theories; or finally (d) the researcher (reflexive) and actors, according to a diversified knowledge of the notions based on the dis- context and in a practical use objective. This last defi- ciplinary origin of each group. All of these parameters nition could also correspond, in part, to the “partici- contribute to creating a diverse and ultimately very pative” position proposed by Lincoln, Lynham, and nonconsensual image of the evaluation grids for qual- Guba (2011). itative research in the health sciences. 3. We had considered the call for “ethical” criteria in the grids as an indicator of the researcher’s position with Conclusion regards to the research, its absence indicating an objectivist position and its presence a position that The results of our research explain in large part the difficul- takes into account the values of the research and the ties in coming to a consensus on the quality criteria for quali- impacts. Our results question this premise. It also tative research. The conception diversity, both inter- and seems here that the presence of criteria linked to eth- intra-grid, prevents the coherence of the grids. Our results ics in the grids indicates at least two different mean- show that the grids examined are not representative of a dis- ings, which would also make this axis a pivotal point cipline but rather of a position with regards to qualitative among the grids. Indeed, the term ethics only rarely research and objectives inherent to each one. They thus seems significant in the corpus and when it does translate the vision that the authors of the grids have of Santiago-Delefosse et al. 11 qualitative research, one that is partial at times, and almost Foundation for the financial support (Grant N° CR13Ι1_126983) provided to this study. always explicit, as per the diverse results obtained. Yet, the goal is now to provide researchers with explicit and adequate References tools to assess their work, and to provide experts (and read- ers) with grids that allow them to evaluate the quality of Aubert-Lotarski, A., & Capdevielle-Mougnibas, V. (2002, March). qualitative work in various health research fields according Dialogue méthodologique autour de l’utilisation du logiciel to comparable criteria. Although standardizing the grids is Alceste en sciences humaines et sociales: “lisibilité” du cor- pus et interprétation des résultats [Methodological dialogue not the objective, their acknowledgement requires clarifying on the use of Alceste software in social and human sciences: the terms and comparing them to show the validity of their “Readability” of the corpus and interpretation of results]. 6èmes use and their scientific reliability. The identification of the Journées Internationales d’Analyse Textuelle [6th International paradigms underlying the quality criteria conducted in this Conference on Textual Analysis], Saint-Malo, France. document is a first step toward understanding the diversity of Beck, C. T. (2009). Critiquing qualitative research. AORN Journal, the qualitative research evaluation grids in the health sci- 90, 543-554. ences. As this article is only one part of a more extended Braun, V., & Clarke, V. (2006). Using thematic analysis in psychol- research project conducted from 2011 to 2014, these findings ogy. Qualitative Research in Psychology, 3, 77-101. therefore cover one aspect of our study. Another part of the Chamberlain, K., Stephens, C., & Lyons, A. C. (1997). research project included the participation of international Encompassing experience: Meanings and methods in health experts in qualitative research in the health sciences. Each psychology. Psychology and Health, 12, 691-709. Cicourel, A. (1964). Method and measurement in sociology. New expert and their peers (56 persons) assessed the grids in their York, NY: Free Press. own fields. As a result, these experts considered 12 quality Cobb, A. K., & Hagemaster, J. N. (1987). Ten criteria for evaluating criteria consensual. The main outcomes have been published qualitative research proposals. Journal of Nursing Education, on a website aimed at the dissemination of this research: 26, 138-143. https://wp.unil.ch/qualityofqualitativeresearch. However, Cohen, D. J., & Crabtree, B. F. (2008). Evaluative criteria for quali- despite its insightful results, our research has two main limi- tative research in health care: Controversies and recommenda- tations. First, this study was carried out on the basis of 133 tions. The Annals of Family Medicine, 6, 331-339. quality grids published in the health sciences field, but we Creswell, J. W. (2013). Research design: Qualitative, quantitative, did not conduct a systematic meta-analysis on all the grids and mixed methods approaches. Thousand Oaks, CA: SAGE. published in the field. This may have influenced the results Creswell, J. W., & Plano Clark, V. L. (2010). Designing and con- with regard to the distribution of grids by disciplines and the ducting mixed methods research. Thousand Oaks, CA: SAGE. Critical Appraisal Skills Programme. (2006). Qualitative Research number and diversity of the criteria. Second, we chose Checklist. Oxford, UK: Public Health Resource Unit. Retrieved Denzin and Lincoln’s (1994/2011) classification, close to our from http://media.wix.com/ugd/dded87_29c5b002d99342f788 own approach, and this choice resulted in our classification c6ac670e49f274.pdf of the epistemological approaches underlying the grids that Curtin, M., & Fossey, E. (2007). Appraising the trustworthiness would have perhaps differed if based on other fundamental of qualitative studies: Guidelines for occupational therapists. works on paradigms in qualitative health. Australian Occupational Therapy Journal, 54, 88-94. In conclusion, more work remains to be done, however, Daly, J., Willis, K., Small, R., Green, J., Welch, N., Kealy, M., for as long as the concepts considered to be major by the & Hughes, E. (2007). A hierarchy of evidence for assessing authors of the grids are not clearly explained, the grids will qualitative health research. Journal of Clinical Epidemiology, only be of limited use in improving and establishing qualita- 60, 43-49. tive research in the health sciences. Denzin, N. K., & Lincoln, Y. S. (Eds.). (2011). The SAGE handbook of qualitative research. London, England: SAGE. (Original work published 1994) Acknowledgments Devers, K. J. (1999). How will we know “Good” qualitative The authors would like to thank Dr Valérie Capdevielle-Mougnibas research when we see it? Beginning the dialogue in health ser- from the University of Toulouse (France), for her valuable help in vices research. Health Services Research, 34, 1153-1188. processing the data for Alceste®, and Dr Maria del Rio Carral from Dixon-Woods, M., Shaw, R. L., Agarwal, S., & Smith, J. A. (2004). the University of Lausanne for her careful reading of this manuscript. The problem of appraising qualitative research. Quality Safety Health Care, 13, 223-225. Declaration of Conflicting Interests Elliott, R., Fischer, C. T., & Rennie, D. L. (1999). Evolving guide- lines for publication of qualitative research studies in psychol- The author(s) declared no potential conflicts of interest with respect ogy and related fields. British Journal of Clinical Psychology, to the research, authorship, and/or publication of this article. 38, 215-229. Flick, U. (2006). Quality criteria in qualitative research. In U. Flick Funding (Ed.), An introduction to qualitative research (pp. 367-389). The author(s) disclosed receipt of the following financial support London, England: SAGE. for the research, authorship, and/or publication of this article: The Flick, U. (2007). Managing quality in qualitative research. author(s) express their gratitude to the Swiss National Science Thousand Oaks, CA: SAGE. 12 SAGE Open Fossey, E., Harvey, C., McDermott, F., & Davidson, L. (2002). [The “lexical worlds” and their “logics” through the statistical Understanding and evaluating qualitative research. Australian analysis of a corpus of stories of nightmares]. Langage et & New Zealand Journal of Psychiatry, 36, 717-732. société/Language & Society, 60, 5-39. Garfinkel, H. (1967). Studies in Ethnomethodology. Cambridge, Robert, A., & Bouillaguet, A. (2002). L’analyse de contenu MA: Polity Press. [Content analysis]. Paris: Presses Universitaires de France. Garric, N., & Capdevielle-Mougnibas, V. (2009). La variation Russell, C. K., & Gregory, D. M. (2003). Evaluation of qualitative comme principe d’exploration de corpus: Intérêts et limites de research studies. Evidence-Based Nursing, 6, 36-40. l’analyse lexicométrique interdisciplinaire pour l’étude du dis- Sandelowski, M., & Barroso, J. (2007). Appraising reports of cours [Variation as a principle to explore the research corpus: qualitative studies. In M. Sandelowski & J. Barroso (Eds.), Interests and limitations of the interdisciplinary lexicometric Handbook for synthesizing qualitative research (pp. 75-132). analysis to study discursive material]. Corpus, 8, 105-128. New York, NY: Springer. Green, J., & Britten, N. (1998). Qualitative research and evidence Santiago-Delefosse, M. (2007). Recherche en psychologie et based medicine. British Medical Journal, 316, 1230-1232. turbulences paradigmatiques [Research in psychology and Greenhalgh, T., & Taylor, R. (1997). How to read a paper: Papers Paradigmatic turbulences]. Bulletin de Psychologie/Bulletin of that go beyond numbers (qualitative research). British Medical Psychology, Hors série/Special issue, 97-102. Journal, 315, 740-743. Santiago-Delefosse, M., Bruchez, C., Gavin, A., & Stephen, S. Hamberg, K., Johansson, E., Lindgren, G., & Westman, G. (1994). L. (2015). Diversity of the Quality Criteria in Qualitative Scientific rigour in qualitative research: Examples from a study Research in the Health Sciences: Lessons from a Lexicometric of women’s health in family practice. Family Practice, 11, Analysis Composed of 133 Guidelines. Forum: Qualitative 176-181. Social Research, 16(2), Art.11. Kuhn, T. (1962). The structure of scientific revolutions. Chicago, Santiago-Delefosse, M., & Rouan, G. (Director). (2001). Les IL: University of Chicago Press. méthodes qualitatives en psychologie [Qualitative methods in Lincoln, Y. S., Lynham, S. A., & Guba, E. G. (2011). Paradigmatic psychology]. Paris: Dunod. controversies, contradictions, and emerging confluences, revis- Schneider, K. J. (1998). Toward a science of the heart. Romanticism ited. In N. K. Denzin & Y. S. Lincoln (Eds.), The SAGE handbook and the revival of psychology. American Psychologist, 53, of qualitative research (pp. 97-128). London, England: SAGE. 277-289. Mays, N., & Pope, C. (2000). Qualitative research in health care: Schutz, A. (1962). Collected papers (2 Vols.). The Hague, The Assessing quality in qualitative research. British Medical Netherlands: Martinus Nijhoff. Journal, 320, 50-52. Silverman, D. (1993). Interpreting qualitative data: Methods for Meyrick, J. (2006). What is good qualitative research? A first step analyzing talk, text and interaction. London, England: SAGE. towards a comprehensive approach to judging rigour/quality. Smith, J., Harré, R., & Van Langenhove, L. (1995). Rethinking Journal of Health Psychology, 11, 799-808. methods in psychology. London, England: SAGE. Miles, M. B., & Huberman, A. M. (1994). Qualitative data analy- Stiles, W. B. (1999). Evaluating qualitative research. Evidence- sis: An expanded sourcebook. Thousand Oaks, CA: SAGE. Based Mental Health, 2, 99-101. Morse, J. M., Barrett, M., Mayan, M., Olson, K., & Spiers, J. (2002). Tashakkori, A., & Teddlie, C. (Eds.). (2010). SAGE handbook of Verification strategies for establishing reliability and validity mixed methods in social & behavioural research. Thousand in qualitative research. International Journal of Qualitative Oaks, CA: SAGE. Tong, A., Sainsbury, P., & Craig, J. (2007). Consolidated criteria Methods, 1(2), 1-19. for reporting qualitative research (COREQ): A 32-item check- Murray, M. (2000). Level of narrative analysis in health psychol- list for interviews and focus groups. International Journal for ogy. Journal of Health Psychology, 5, 331-342. Quality in Health Care, 19, 349-357. Paterson, M., & Pentland, W. (2008). The use of hermeneutics in Van den Hoonaard, W., Given, L., Lévy, J., McGinn, M., O’Neill, P., a mixed methods design. The Qualitative Report, 13, 116-134. & Palys, T. (2004). Giving voice to the spectrum: Report of the Retrieved from http://www.nova.edu/ssss/QR/QR13-1/vonz- social sciences and humanities Research Ethics Special Working weck.pdf Committee to the Interagency Advisory Panel on Research Popay, J., Rogers, A., & Williams, G. (1998). Rationale and stan- Ethics. Ottawa, Ontario, Canada: Interagency Advisory Panel dards for the systematic review of qualitative literature in health and Secretariat on Research Ethics. Retrieved from http://www. services research. Qualitative Health Research, 8, 341-351. sfu.ca/~palys/SSHWC-GivingVoice-2004.pdf Pope, C., & Mays, N. (2006). Qualitative research in health care. Walsh, D., & Downe, S. (2006). Appraising the quality of qualita- Oxford, UK: Blackwell. tive research. Midwifery, 22, 108-119. Porter, S. (2002). Critical realist ethnography. In T. May (Ed.), Yardley, L. (2008). Demonstrating validity in qualitative psychol- Qualitative research in action (pp. 53-72). London, England: ogy. In J. A. Smith (Ed.), Qualitative psychology: A practical SAGE. guide to research methods (pp. 235-251). London, England: Reinert, M. (1990). Alceste. Une méthodologie d’analyse des SAGE. données textuelles et une application: Aurélia de Gérard de Nerval [Alceste. A methodology for analyzing textual data Author Biographies and its related application: Aurélia by Gérard de Nerval]. Bulletin de méthodologie sociologique/Bulletin of Sociological Marie Santiago-Delefosse has been a full professor in health psy- Methodology, 26, 24-54. chology at the University of Lausanne (Switzerland) since 2003. Reinert, M. (1993). Les “mondes lexicaux” et leur “logique” à trav- Her empirical research (fertility, HIV, chronic pain, etc.) has led her ers l’analyse statistique d’un corpus de récits de cauchemars to propose an alternative theorization of the biopsychosocial model. Santiago-Delefosse et al. 13 Her work pays particular attention to the history of ideas and to allowed her to develop skills and interests in various domains, such epistemology in psychology; it is mostly qualitative and based on a as sexology, couple counseling, ethics, critical health psychology, historico-cultural and phenomenological approach. She is the and qualitative research among others. author of more than 100 articles and several books. Sarah Lilian Stephen worked as Junior SNSF (Swiss National Christine Bruchez has been a teaching assistant, then research Science Foundation) researcher in an SNSF-funded project directed coordinator, in health psychology at the University of Lausanne by Prof. Santiago-Delefosse at the Faculty of Social and Political since January 2004. She has completed a DESS (advanced graduate Sciences at the University of Lausanne. Since 2011, she is also a diploma) in clinical psychology, and is currently a PhD candidate PhD student based at the Faculty of Business and Economics at the under the supervision of Prof. Marie Santiago-Delefosse. Her University of Lausanne. Her interests include quality of quantita- research interests in epistemology and methodology have led her tive and qualitative research and interdisciplinary research meth- toward the comparison of qualitative methods and the experiment- ods, as well as paradigms underlying research. ing with new research tools in psychology. Pauline Roux has been a Junior Lecturer at the University of Amaelle Gavin has been research coordinator in health psychology Lausanne (Switzerland) since March 2015. She presented her thesis at the University of Lausanne since October 2012. She is currently in social psychology in 2013 at the University of Lyon (France). training in clinical sexology and since February 2015, she occupies Her research interests include patient-doctor relationship and com- a position in the university as a PhD candidate under the supervi- munication, medically unexplained illnesses, and qualitative meth- sion of Prof. Marie Santiago-Delefosse. Her different trainings odology and triangulation.

Journal

SAGE Open – SAGE

Published: Dec 28, 2015

Keywords: qualitative research quality; evaluation criteria; guidelines; paradigms; Alceste® lexicometric analysis

There are no references for this article.