University of Groningen
Using Q methodology
Lundberg, Adrian; de Leeuw , Renske; Aliani, Renata
Published in:
Educational Research Review
DOI:
10.1016/j.edurev.2020.100361
IMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite from
it. Please check the document version below.
Document Version
Publisher's PDF, also known as Version of record
Publication date:
2020
Link to publication in University of Groningen/UMCG research database
Citation for published version (APA):
Lundberg, A., de Leeuw , R., & Aliani, R. (2020). Using Q methodology: Sorting out subjectivity in
educational research. Educational Research Review, 31, [100361].
https://doi.org/10.1016/j.edurev.2020.100361
Copyright
Other than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).
Take-down policy
If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.
Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons the number of authors shown on this cover page is limited to 10 maximum.
Educational Research Review 31 (2020) 100361
Available online 28 September 2020
1747-938X/© 2020 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
Using Q methodology: Sorting out subjectivity in
educational research
Adrian Lundberg
a,*, Renske de Leeuw
b,1, Renata Aliani
c aMalm¨o University, SwedenbUniversity of Groningen, the Netherlands cUniversity of Melbourne, Australia
A R T I C L E I N F O Keywords: Compulsory education Mixed method Research review Participatory research
Science of learning and development (SoLD)
A B S T R A C T
Understanding subjective perspectives and lived experiences of different stakeholders can improve pupils’ learning environment in compulsory school settings. Q methodology is an inherently mixed method approach and regarded as the basis for the science of subjectivity. The present paper reviewed recent Q methodological publications in compulsory education research. Seventy-four studies reporting from context in twenty countries met the inclusion criteria and showed a wide-ranging and diverse application of the research methodology. The subjectivity of teachers, pupils and others were explored on topics related to the science of learning and development and teacher characteristics. This review showcases how Q methodology is applied to access subjectivity in educational research and provides an overview of Q methodological find-ings and implications for the field. The wide range of the application of Q studies in compulsory education attests to the flexibility and suitability of this research method in educational research.
Education is characterized by dynamic, complex and ever-changing contexts due to a myriad of social interactions. Therefore, educational researchers perennially face difficulties of local conditions limiting generalizations and theory building (Berliner, 2002). Consequently, as a complement to so-called objective science, studying subjectivity to gather reliable evidence is crucial. Implying a dualistic worldview, subjective has long been understood as “unreliable, undependable and unpredictable” (Good, 2010, p. 232) as opposed to objective being its testable and reliable counterpart. While objective implies something existing “beyond the inner expe-riences of any single person” (Good, 2010, p. 232) and potentially being “observed by others” (Stephenson, 1953, p. 22), subjective represents an inner experience and is much harder to observe. Notwithstanding, because subjectivity “can be seen as the epitome of a person’s dispositions and capabilities” (Harteis et al., 2006, p. 125) and “human actions depend on what humans think they are doing” (Eisenhart & DeHaan, 2005, p. 5), subjectivity in research is both powerful and important for a comprehensive representation of educational issues. Educational institutions have to adapt their structures to respond to linguistic and ethnic minorities and equitable access to education is a major issue of concern (OECD, 2012). Simultaneously, educational researchers increasingly need to possess a repertoire of methodologies that provides equitable access to research participation. The present paper reviews a means to objectively identify participants’ subjectivity with or without the application of a participatory research approach.
* Corresponding author. Department of School Development and Leadership, Nordenski¨oldsgatan 10, 205 06, Malm¨o, Sweden.
E-mail addresses: adrian.lundberg@mau.se (A. Lundberg), r.r.de.leeuw@rug.nl (R. de Leeuw), renata.aliani@unimelb.edu.au (R. Aliani). 1 Present affiliation: University of Applied Sciences, Saxion, the Netherlands; r.r.deleeuw@saxion.nl.
Contents lists available at ScienceDirect
Educational Research Review
journal homepage: www.elsevier.com/locate/edurev
https://doi.org/10.1016/j.edurev.2020.100361
1. Theoretical and conceptual foundations
1.1. Subjectivity and Q methodology
Aspects of subjectivity, such as views, beliefs, assumptions or in fact everything individuals say about any matter of personal or social importance (McKeown & Thomas, 2013) are often investigated using Likert scales or questionnaire instruments. To reduce the undesired reproduction of researchers’ modeling of subjective constructs, open interview questions with a conservative interpretation process seem more appropriate (Harteis et al., 2006). The present review centers on an alternative approach to participants’ emic perspectives, that resists being pigeonholed within broad research classifications. Here, subjectivity is not employed in the sense of an opposition to presumed scientific objectivity, but in line with individual “self-referent notions” (Stephenson, 1953, p. 248). Q methodology (henceforth Q) has been proclaimed as “the best-developed paradigm for the investigation of human subjectivity” (Dryzek & Holmes, 2002, p. 20). William Stephenson introduced the principles of Q in a letter to Nature in 1935 and claimed the methodology to be “especially valuable in experimental aesthetics and in educational psychology, no less than in pure psychology” (Stephenson, 1935, p. 297). Gooding and Wilbur (1971) promisingly concluded that Q “can become an increasingly valuable tool in the educational researcher’s repertoire” (p. 46) and Montgomery (2010) recently claimed that Q in educational research “is our window to learning about subjective reactions or responses to the issues confronting professionals daily” (p. 1). Certain principles, presented below, facilitate the understanding of subjectivity in line with Q methodology.
1.2. Q methodological principles and terminology
Whereas purely quantitative methods tend to conceal marginalized viewpoints (Dryzek, 2005) by reporting averages across different demographic variables, Q increases the likelihood of the emergence of any participant’s actual thinking (Brown, 2006) and ensures that all potentially relevant voices are heard (Howe, 2004). Hence, some of the methodology’s pivotal strengths are its established status as an inherently mixed method (Ramlo, 2020; Teddlie & Tashakkori, 2009), the opportunity to explore minority voices (Pike et al., 2015) and conduct participatory research (Militello et al., 2016). Fig. 1 illustrates six methodological steps of Q, which are explained in the next paragraphs for researchers less familiar with this methodology.
1.2.1. From concourse to Q set
The construction of the data collection instrument in Q, a set of items to be ranked by participants, is crucial to the process. It consists of various steps, which require thorough consideration and this can be time-consuming. They begin with collecting the concourse of communicability (step 1), which is all that can be said about the subject matter and shared with members of the same culture or society (Brown, 2019b). This confirms the highly contextual nature of the understanding of subjectivity in Q. In a so-phisticated structuring process (see e.g. Brown et al., 2019), items are then culled to present a balanced and representative sample, equal to the sample of persons in an R methodological study (step 2). Crucial to understanding Q is the self-reference of these statements, as they are subjected to participants’ feelings (Watts & Stenner, 2012).
1.2.2. Q Sorting and post-sorting activities
A typical Q study involves participants in a sorting activity (step 3) that allows them to communicate their subjectivity towards a specific topic (Brown, 1980; Watts & Stenner, 2012) without the need to produce language in written or spoken form. Instead of written statements, pictures, objects, audio or video files can be ranked by the respondents, allowing the participation of many otherwise marginalized groups of people (Kelly, 2007). Young participants have expressed their appreciation towards the rather playful Q sorting activity (de Leeuw et al., 2019) and the methodology has been termed as ethical, critical, respectful and person-centered (Hughes, 2016).
During the sorting process, guided by a condition of instruction derived from the research question, participants assign meaning to each of the items as they place them on a position in the grid. These items are sorted relative to each other into a predominantly predetermined quasi-normal distribution, providing a holistic configuration of the participants’ view, with minimal influence and bias from the researcher (Stephenson, 2014).
It is recommended to instruct participants to elaborate on the items placed towards the extremes of the sorting continuum in a post- sorting activity (step 4) in the form of interviews (Shemmings & Ellingsen, 2012) or written responses (Watts & Stenner, 2012). This additional information will eventually enrich the qualitative description of factor arrays and can be collected in different ways.
1.2.3. From Q factor analysis to factor interpretation
With dedicated software packages and informed decisions of the researcher, the by-person factor analysis compares and groups all participants’ sorts (step 5). This data condensation technique yields a few representative shared factors. These are rotated until the researcher symbolically views the subject matter from the participants’ standpoint and receives a clear and interpretable structure
(Zabala et al., 2018). The qualitative interpretation of factors (step 6) is done in a holistic manner, accounting for the entire item configuration captured by particular factors.
1.2.4. Potential weaknesses of Q methodology
Apart from being a rather time-consuming and demanding research approach, Q’s methodological principles and practices and in particular its inherently mixed-method character have long been controversially discussed (Ramlo, 2016). In fact, the very assertion of Q being mixed method or a hybrid of anything has sparked a series of communications between Q researchers (Brown, 2019a). Regardless of the methodological label Q inherits based on its potentially unfamiliar philosophical underpinnings, scholars are well-advised not to judge terminological distraction as a weakness, but see the opportunities this novel approach offers.
A historical summary of the steady stream of criticism from outside the community of active Q practitioners and responses to some of the issues raised is presented in (Brown et al., 2015). Illustrative of existing misunderstandings concerning Q is the way validity is treated within Q methodology. Since participants’ Q sorts are neither right nor wrong, but constructed through respondents’ rank-ordering of self-referent items, validity in line with quantitative tenets of research is of no concern in Q (Brown, 1980).
1.3. Educational research and the science of learning and development
To enable the thematic interpretation of the studies reviewed in this paper, the scientifically grounded framework proposed by
Darling-Hammond et al. (2019) was selected. Their deeply integrated approach is based on a comprehensive research synthesis (Cantor et al., 2019; Osher et al., 2020) and acknowledges both, the localized and interconnected nature of educational settings and the importance of subjective perspectives of stakeholders. The framework supports pupils’ learning and development by bringing together
four interactive and interrelated areas, each consisting of a range of principles of practice in the science of learning and development (henceforth SoLD).
a) Supportive Environments, are characterized by relational trust, continuous and consistent structures reduce anxiety among pupils. Strong, positive and sustained relationships between different actors in educational contexts are central. Furthermore, classroom communities that permit personalized learning contribute to pupils’ increased feeling of safety, belonging and purpose. b) Productive Instructional Strategies encompass motivated and self-directed learning well scaffolded by the teacher. Taking pupils’
prior knowledge into account and providing formative feedback in rich and engaging tasks are equally essential as collaborative learning opportunities that encourage pupils to share ideas and develop knowledge cooperatively.
c) Social and Emotional Development is advanced by an appropriate environment, where pupils develop habits that foster perseverance, resilience, agency, and self-direction. A growth mindset will further support pupils’ learning and development.
d) Systems of Support grant pupils access to a range of services that enable their healthy development. Extending learning opportunities for some, others need more extensive support. Sometimes learning barriers are best broken down by multi-tiered systems of support, ranging from measures of prevention to selective or intensive intervention applied as special education.
1.4. Present study
The purpose of the present paper is twofold. Firstly, reviewing characteristics of Q methodological studies in compulsory education illustrates how the methodology has been practically applied in research. Secondly, a synthesis of findings and implications from included studies within compulsory educational settings demonstrates and exemplifies outcomes produced through a Q methodo-logical approach. The guiding research questions read as follows:
1. What are the characteristics of Q methodological studies in compulsory education research? 2. What findings and implications for educational settings can be synthesized from the included studies?
2. Method
A systematic approach (Newman & Gough, 2020) was followed during the selection process and data extraction phase. Based on the PRISMA flow diagram (Moher et al., 2009), Fig. 2 illustrates the literature search and processing of records. A narrative approach (Popay et al., 2006; Snilstveit et al., 2012) was chosen to synthesize the findings and implications of included studies due to the predominantly descriptive nature of Q methodological results.
2.1. Study selection process
Three bibliographic databases were consulted on January 10, 2020, to retrieve the widest range of educational research conducted with Q: 1) ERIC (Educational Resources Information Center) via EBSCO, 2) Web of Science, and 3) Scopus. The following search parameters were used in titles, abstracts and keywords: (Q method*) AND (Q-method*). A time period restriction was applied to finding records published after 2009, to ensure a more contemporary review of research. Examining 10 years of research conducted with Q was considered to provide a thorough understanding of its applications in educational research. To minimize publication bias and report the most comprehensive result possible, considering gray literature could be justified (Alexander, 2020). The search results were limited to certain document types (academic journal papers, dissertations, books and book chapters) and subject areas (edu-cation, social sciences, humanities and multidisciplinary research). 688 publications were forwarded to the screening phase.
In addition to a publication date after 2009, the following inclusion criteria were used:
•Studies investigate a compulsory educational setting and/or teacher education for compulsory education. •Studies follow the broader conceptual philosophical framework of Q: including Q sorting and Q factor analysis. •Studies report new empirical results.
In an attempt to increase its comprehensiveness, the present systematic review was extended by snowballing (Wohlin, 2014), where thirty-nine studies were added by hand-searching Q’s official, but not indexed journal Operant Subjectivity, and some univer-sities’ online repositories. Next, 135 records were assessed for eligibility by applying the inclusion criteria. If a record was not accessible via one of the three authors’ library system, the corresponding author was contacted to obtain the manuscript. Two records had to be excluded because the full text was not available. To establish a reliable screening, any potentially excluded studies were screened by a second author. Discrepancies were resolved through consensus, resulting in the inclusion of 74 studies.
2.2. Data extraction and analysis
To respond to the first research question, a codebook (Pigott & Polanin, 2020) was collectively created. The codebook was piloted, by independently extracting data from a random sample of 15 papers. A comparison of the coded information yielded some necessary adaptations to the codebook. All included papers were then systematically coded accordingly:
1. Participant set (P-set) characteristics (educational setting, role, size) 2. Study focus (research question, condition of instruction)
3. Q sampling characteristics (approach, sources, instrument validation, size of final Q set) 4. Q sorting characteristics (distribution grid, items, post-sorting activities)
5. Q factor analysis (software, extraction method, rotation method) 6. Results (number of factors, labelling of factors)
7. Others (rationale for and limitations of Q, aspects of participatory research approach) Ambiguous information was discussed until a joint decision.
To respond to the second research question, SoLD principles of practice (Darling-Hammond et al., 2019) were applied as an analytical framework. Due to the tight interrelatedness of various principles of practice, studies could be assigned to more than one area. Studies that fit multiple principles were coded with the predominant principle. Studies with a focus on anything other than pupils’ learning and developmental needs were clustered into additional areas.
3. Results
3.1. Characteristics of Q methodological studies in compulsory education research
After applying the inclusion criteria, 74 studies from 50 different sources were included. Apart from one book, 15 doctoral dis-sertations from eight different universities, met the inclusion criteria. The noticeable majority of the 74 included studies was published towards the end of the decade (see Fig. 3), illustrating the continuation of Q’s growing popularity in educational research (Irie et al., 2018). Participants were mostly located in the USA (n = 36) and the UK (n = 13). With Australia (n = 5) and South Korea (n = 4), only two more countries were represented more than twice. Twelve study contexts were located in Europe, two in Africa and two in South America. Three multicultural studies were included, with two investigating contexts in the USA and South Korea, and one compared results from Bulgaria and Croatia.
3.1.1. Explored voices with Q methodological studies
With compulsory education settings as an inclusion criterion, only voices connected to elementary (n = 15) and secondary (n = 29) schooling were represented. Twenty-seven studies included participants representing both settings. Three studies did not mention the setting they investigated but made clear that it was concerning compulsory education.
Table 1 presents an overview of the participants whose voices and perspectives were explored. The total number of studies exceeds 74, because 13 studies combined different cohorts and are counted several times. The most explored voices are those of teachers. When in- and pre-service teachers are combined, this cohort covers 41% of all the included studies. The second-largest cohort is the pupils’ voices (23%). Voices of school staff members with a leading role, such as principals or school administrators were subsumed into the principal class, equal 14% of all included publications (n = 13). There were additional single cohort studies with members of special education, school counselors and teaching assistants. The voices of academics, committee members or teacher educators were only investigated in conjunction with other cohorts.
3.1.2. Categorization of study focus
A qualitative interpretation of stated research questions, research aims and conditions of instructions allowed clustering the studies into six different categories:
A. Representation
One third of all studies (n = 27) investigated participants’ representation of a subject matter, asking them how the issue under scrutiny is typically understood.
B. Attitudes and Values
Research questions (n = 21) that focused more on a personal preference towards a topic were categorized as attitudes and values. Typically, asking participants to investigate what they think and like/dislike about the subject matter.
C. Critical reflection
Thirteen studies used Q as a tool to allow participants to critically reflect about their situation or characteristics. D. Evaluation
Table 1
Participants’ cohorts and study foci.
# Cohort Studies per
cohort A: Represen- tation B: Attitudes & values C: Critical reflection D: Evaluation E: Response F: Decision making 1 Teachers 1a - In service 27 13 7 5 2 3 – 1b - Pre-service 11 3 3 2 2 1 – 2 Pupils 21 9 4 2 4 2 – 3 Principal class 13 7 4 2 – – – 4 Special education 4a - teachers 6 – 2 2 1 1 – 4b - coordinators 1 1 – – – – – 5 Parents 5 3 – 1 – – 1 6 School counsellors 3 1 2 – – – – 7 Teaching assistants 2 – – 1 1 – – 8 Academics 2 2 – – – – – 9 Committee members 1 1 – – – – – 10 Teacher educators 1 – 1 – – – –
Note: In the study focus categories, in-service teachers are represented 30 times, because three studies (Lundberg, 2019a/b and Wirth, 2014) have two components with different study foci.
In nine studies, Q was chosen to evaluate an educational issue. E. Response
Six studies investigated participants’ preferences in terms of how to respond to a certain subject matter. F. Decision making
One study used Q to describe and analyze the process of making decisions.
A total number of 77 studies are represented, as the publications by Lundberg (2019a, Lundberg, 2019b and Wirth (2014) consisted of two components with separate research questions and conditions of instructions, falling within different categories. Table 1 il-lustrates the distribution of study foci according to individual participant cohorts. Especially pupils are often asked to evaluate an issue, while in-service teachers are the participant cohort most often asked to critically reflect.
3.1.3. Characteristics of research design and analysis
In terms of the research design (Fig. 4), the average P-set in all the included studies consist of slightly more than 37 participants. Four studies conducted a Q study with 10 or fewer participants and two publications were conducted with more than 90. The number of Q items in the Q-sets was somewhat higher with an average of 40.3. In 28 studies, authors list a combination of sources for their Q-
Table 2
Summary of Compulsory Education Studies using Q methodology 2010–2019.
SoLD Authors, (year) Participant cohort
(location) Education setting P-set size Study focus Q-set size Part. appr. Type of extraction Type of rotation Factor
(N) (N) (N)
SEC
Bang and Montgomery
(2010) 1a, 4a (USA/South Korea) Ele. & Sec. 24 C 47 C BH 4
Beck (2017) 2 (USA) Elementary 48 B 42 yes ns ns 4
Berry et al. (2012) 1b (USA) Ele. & Sec. 77 A 29 C Va 4
Brown and Militello
(2016) 3 (USA) Ele. & Sec. 34 B 34 yes PCA Va 3
Coladonato (2013) 1a, 3 (USA) Ele. & Sec. 43 B 54 PCA Va 5
Davis (2018) 3 (USA) Elementary 16 B 35 ns ns 5
Dhillon et al. (2020) 3, 8 (UK) Elementary 14 A 23 PCA ns 4
Duncan and Owens
(2011) 2 (UK) Secondary 28 A 35 yes PCA ns 2
Frearson (2013) 2 (UK) Secondary 25 E 64 yes C Com 5
Goodrich (2017) 6 (USA) Secondary 91 B 21 PCA Va 5
Hobbs (2011) 1a (USA) Secondary 100 B 40 PCA Va 3
Kemp (2014) 1a, 2, 3, 5,9
(Australia) Secondary 26 A 64 C Va 3/3
Kim and Bang (2017) 5 (South Korea) Ele. & Sec. 36 C 47 C Va 4
Levine (2013) 4a (USA) Ele. & Sec. 25 C 40 PCA Va 3
Lo Bianco and Aliani
(2013) 2 (Australia) Secondary 48 B 25 ns ns 3/3
Lundberg, 2019b 1a (Switzerland) Elementary 67 A, E 39/
32 PCA Va 2/6
Lundberg (2019a) 1a (Sweden) Elementary 40 A, E 39/
32 PCA/C Va 3/3
Militello and Janson
(2014) 6 (USA) Ele. & Sec. 61 B 33 PCA Va 3
Militello, Bass, et al.
(2013) 3 (USA) Ele. & Sec. 79 C 43 PCA Va 1
Nzahabwanayo et al.
(2019) 1a (Rwanda) Secondary 58 A 50 PCA Va 4
Parker (2015) 5 (USA) Elementary 35 F 36 PCA Va 3
Provost et al. (2010) 3 (USA) Ele. & Sec. 30 A 21 ns ns 1
Slaughter et al. (2020) 3 (Australia) Elementary 6 B 48 ns ns 3
SticklHaugen et al.
(2019) 2 (USA) Secondary 43 A 39 C N 3
Stollery (2013) 2 (UK) Secondary 30 D 46 C Com 4
Swetnam (2010) 2 (USA) Secondary 32 A 32 PCA Va 3
Xi et al. (2016) 2 (China) Secondary 53 A 40 PCA Com 1
Yang and Montgomery
(2013) 1b, 10 (USA) Elementary 43 B 47 PCA Va 2
Yeboah et al. (2017) 2, 5 (Ghana) Secondary 72 A 34 PCA Va 2
PIS
Anderson and Jacobson
(2018) 1a (Ecuador) Ele. & Sec. 25 A 25 ns ns 3
Baltrinic et al. (2016) 1a (USA) Secondary 15 A 31 yes PCA Va 3
Bang and Kim (2016) 4a (USA/South
Korea) Ele. & Sec. 38 B 47 C Va 5
Barnes et al. (2015) 1a, 1b (USA) Ele. & Sec. 40 E 40 yes ns ns 3
Baron et al. (2019) 4a (USA) Ele. & Sec. 29 D 60 C Va 4/4
Bonar (2018) 2 (Australia) Secondary 52 B 36 PCA Va 5
Burke O’Connell et al.
(2019) 2 (Ireland) Secondary 24 B 48 PCA Va 2/3
Çirak Kurt & Yildirim
(2018) 1b (Turkey) Elementary 31 A 18 PCA ns 1
Everman (2016) 1b (USA) Elementary 38 B 43 ns ns 6
Hock et al. (2015) 2 (Malaysia) Elementary 30 A 34 ns ns 2
Jelizakova (2015) 1a (Bulgaria/
Croatia) Secondary 34 A 41 ns ns 5/4
Kotul’´akov´a (2019) 1a (Slovakia) Secondary 34 B 51 C Va 3
Levitt & Red Owl (2013) 1a (USA) Ele. & Sec. 21 B 53 yes PAF Va 3
McLain (2018) 1a (UK) Secondary 7 B 62 yes ns ns 1
Militello, Bass, et al.
(2013) 1a, 3 (USA) Ele. & Sec. 62 C 23 ns Va 1/1
Nauman et al. (2011) 1a, 1b (USA) Ele. & Sec. 60 A 31 ns ns 3
Ramlo (2019) 2 (USA) Secondary 46 D 40 ns ns 3/3
set. In studies with only one source, literature was found to be the most common origin, with participant-generated Q sets a close second, indicating aspects of a participatory approach. While 26 publications state external experts as a validation procedure to create Q sets, pilot studies to validate Q sets were mentioned 25 times.
The majority of authors (n = 55) used physical cards and were present during the sorting activity. Sixty-one studies mention post- sorting activity, whereof most aimed at collecting participants’ rationale for their decisions about the extreme value choices. The factor analytical procedure was mostly (n = 55) performed using a version of PQMethod (Schmolck, 2014). The most common analytical choices were principal components analysis (PCA; n = 30) as an extraction method and Varimax rotation (n = 38). Studies not clearly stating their statistical procedure were common for both, the extraction and the rotation method (Fig. 5).
The majority of studies retained and described three factors, followed by four-factor-solutions and two-factor-solutions (Fig. 5). If the studies are broken up into their respective components, especially three-factor-solutions and four-factor-solutions gain in count.
3.1.4. Less conventional study designs
Most studies in this review report one group of factors, stemming from one set of items, which participants, sorted according to one Table 2 (continued)
SoLD Authors, (year) Participant cohort
(location) Education setting P-set size Study focus Q-set size Part. appr. Type of extraction Type of rotation Factor
(N) (N) (N)
Sklarwitz (2017) 2 (USA) Secondary 55 C 25 PCA Va 3/4/4
Vandeyar (2014) 1a (South Africa) Elementary 23 B 78 PCA Va 3
SED
Atkinson and Rowley
(2019) 2 (UK) Ele. & Sec. 9 D 37 yes ns Com 3
Heffernan (2017) 2 (UK) Secondary 38 A 52 C Com 4
Kim and Oh (2017) 1a, 2, 3, 5, 6 (South
Korea) Secondary 56 A 36 PCA ns 4
SoS
Atkin (2019) 7 (UK) Ele. & Sec. 30 D 39 C BH 3
Berry (2010) 4a (USA) Ele. & Sec. 60 B 24 C Va 3
Boscardin et al. (2018) 3 (USA) ns 43 A 58 PCA BH 2
Brown (2016) 1a (UK) Elementary 26 B 48 C Va 2
(de Leeuw et al. (2019)) 2 (the Netherlands) Elementary 45 E 15 C Va 4/4
Heasly (2017) 2 (UK) Secondary 21 C 47 yes C BH 6
Ramsay et al. (2018) 4b (UK) Secondary 20 A 40 ns Va 4
Sabo et al. (2018) 1a, 3 (Slovakia) Elementary 32 A 57 ns Va 5
Subba et al. (2016) 2 (Norway) Ele. & Sec. 26 D 30 C BH 3
Subba et al. (2017) 4a (Norway) Ele. & Sec. 25 E 30 ns BH 3
Tudryn (2012) 3 (USA) Ele. & Sec. 30 A 40 PCA ns 2
TC
Collins and Liang (2014) 1a (USA) Ele. & Sec. 13 D 36 PCA Va 2
Cooper (2018) 7 (UK) Ele. & Sec. 38 C 64 C Com 2/5
Demir (2016) 1b (Turkey) ns 40 B 65 ns ns 3
DeVore-Wedding et al.
(2018) 1a (USA) Ele. & Sec. 29 A 47 yes PCA/C Va 3
Dobrica-Tudor &
Th´eorˆet, 2017 1a (Canada) Secondary 13 C 14 PCA ns 4
Irie et al. (2018) 1b (Austria) ns 51 C 56 ns ns 3
Levine and
VanSlyke-Briggs (2014) 1a (USA) Ele. & Sec. 26 C 44 C BH 2
Pruslow & Red Owl
(2012) 1a, 1b (USA) Ele. & Sec. ns D 45 PCA Va 7
Ramlo (2012) 1a (USA) Secondary 20 A 30 ns ns 1
Ramlo (2017) 1b (USA) Secondary 15 D 48 PCA Va 2
Spendlove et al. (2012) 1b (UK) Secondary 59 C 39 ns Va 3
Wirth (2014) 1a (USA) Secondary 18 A, C 41 PCA Va 3/3
Note: With the aim of encapsulating the review “into a more readable and easily analyzed form” (Alexander, 2020, p. 17), several data in this table is abbreviated.
The table is structured by SoLD area; SEC: Supportive environmental conditions, PIS: Productive instructional strategies, SED: Social and Emotional Development, SoS: Systems of Support, TC: Teacher characteristics.
Cohort legend: 1a: in-service teachers, 1b: pre-service teachers, 2: pupils, 3: principal class, 4a: special education teachers, 4b: special education coordinators, 5: parents, 6: school counsellors, 7: teaching assistants, 8: academics, 9: committee members, 10: teacher educators.
Education setting legend: Ele. & Sec.: Elementary and Secondary.
Study focus legend: A: Representation, B: Attitudes & Values, C: Critical reflection, D: Evaluation, E: Response, F: Decision making. Part. appr.: Participatory Research Approach.
Type of extraction legend: PCA: Principal Component Analysis C: Centroid Analysis, PAF: Principal Axis Factoring, ns: not stated. Type of rotation legend: Va: Varimax Rotations, BH: By-hand or Judgmental rotations, Com: Varimax and By-hand, N: none.
condition of instruction at one point in time. However, 13 studies reported factor solutions from multiple study components (see
Fig. 5). In three studies, participants were instructed to sort the same Q set according to two different research questions (Cooper, 2018;
de Leeuw et al., 2019; Wirth, 2014). Lundberg (2019a, Lundberg, 2019b constructed individual Q sets for each condition of in-struction. Lo Bianco and Aliani (2013) used the same condition of instruction with different Q sets and individually analyzed separate cohorts of participants, just like five other studies with one Q set and only one condition of instruction (Burke O’Connell et al., 2019;
Jelizakova, 2015; Kemp, 2014; Militello, Bass, et al., 2013; Sklarwitz, 2017). Additionally, two studies applied a pre/post research design (Baron et al., 2019; Ramlo, 2019).
3.1.5. Studies with a participatory approach
Eleven studies incorporated a participatory approach in at least one step of the research process. Mostly, that included participants’ contribution to the Q sampling, as experts during the culling process or as pilot study participants (Atkinson & Rowley, 2019; Baltrinic et al., 2016; Beck, 2017; DeVore-Wedding et al., 2018; Heasly, 2017; McLain, 2018). Studies using items based on previous interview or questionnaire studies were not coded as participatory approach, because this does not actively involve participants. Other appli-cations of participatory approach in Q are gathering the input of participants on the factor descriptions and interpretations, including participants in the factor interpretation to validate the results (Barnes et al., 2015; Brown & Militello, 2016; Duncan & Owens, 2011;
Frearson, 2013) or asking participants to interpret the factors (Pruslow et al, 2012).
3.2. Synthesis of implications for educational settings
The studies were clustered according to the SoLD principles of practice put forward by Darling-Hammond et al. (2019). The categorization is visible in Table 2. As Q was also applied in studies that did not concentrate on SoLD’s central focal point, the pupil, an additional category consisting of implications relevant for teacher characteristics was created.
3.2.1. Supportive environmental conditions
Several included studies are located within the SoLD principle of practice to offer a safe and personalized setting for learning. For example, by asking teachers to foster democratic and participatory skills among students to socialize them for their role as community members (Pruslow & Red Owl, 2012) or by investigating how teachers’ emotional intensity influences the classroom climate (Bang & Montgomery, 2010). A multifaceted sense of belonging and its connection to safety is researched in Stickl Haugen et al. (2019) and
Goodrich (2017). Furthermore, a sense of compulsory heteronormativity is found among girls’ construction of popularity (Duncan & Owens, 2011). Xi et al. (2016) add the notion of cultural differences in terms of how popularity among (Western and Asian) girls is perceived. In support of strong attachments and positive relationships, SoLD suggests longer grade spans and a reduction of different teachers which can be connected to the challenging transition from one education segment to the next (Lo Bianco & Aliani, 2013).
Implications for a culturally responsive pedagogy are found in studies focusing on teachers’ views about multilingualism ( Lund-berg, 2019a; Lundberg, 2019b), pupil’s use of their first language (Stollery, 2013), the suggestion of a prayer room at schools for Muslim girls (Frearson, 2013) or the use of works of art to engage and motivate culturally disadvantaged students (Beck, 2017). The creation of a secure learning environment for pre-service teachers to explore and discuss linguistic diversity is suggested after the detection of consensual, but somewhat insecure attitudes towards more support for linguistic minority students in Yang and Mont-gomery (2013). Finally, teachers’ self-perception of their cultural competence is studied by Hobbs (2011), who concludes with the call for an improved definition of the concept.
A range of studies are assigned to the SoLD principle of practice to strengthen relational trust and family engagement. In addition to the importance of a climate of hope to create a supportive administration-faculty relationship and eventually minimize teacher attrition (Levine, 2013), skillful school leaders are a comparatively well-researched topic with Q methodology. Leadership styles are suggested to be adapted to the realities in actual practice (Militello, Fusarelli, et al., 2013) and embedded in more in-school on-site learning opportunities for principals (Davis, 2018). Furthermore, Provost et al. (2010) conclude that effective principal leadership, was found to be characterized by high expectations for staff performance, communicating instructional goals, developing school goals, and systematically observing teachers’ instructional methods. A finding largely in line with one by Dhillon et al. (2020), describing the importance of taking decisive action to address the poor performance of staff. Related to that, Brown and Militello (2016) suggest an increase in principals’ ambition to take an active role in teachers’ development, especially regarding sustainability and collaboration. Family engagement is regarded as an important topic by parents during their process of choosing a school (Parker, 2015) and school-family interaction could be increased through on-campus family resource centers (Swetnam, 2010). Engaging parents to promote trust, safety, and belonging crucial for students’ successful educational career is supported by Yeboah et al. (2017), indicating parents’ strong influence on their children’s desirable job after schooling and Kim and Bang (2017) on parents’ educational aspiration for their children.
3.2.2. Productive instructional strategies
To teach pupils in the zone of proximal development and provide scaffolding as suggested by Darling-Hammond et al. (2019), teacher demonstration, including modeling and explaining (McLain, 2018) and the creation of a rich learning environment and collaboration through field trips (Ramlo, 2019) are suggested. Baron et al. (2019) conclude that teachers’ PD programs difficulty to acknowledge participants’ various historical identities, perspectives and epistemological positions can easily be transported into compulsory education. In search of a definition of excellent teaching practices, Baltrinic et al. (2016) show teachers’ views are informed by their experiences in multiple roles, such as counselors, liaisons and guides to pupils and families.
Various studies were assigned to the SoLD area of learning opportunities. Everman (2016) illustrates how literature can make science more approachable, build excitement, and encourage pupils to become more engaged. Levitt and Red Owl (2013) focus on the importance of early home and school literacy environments for reading competence and attitudes toward reading activities in school. The application of digital technologies in compulsory education (Burke O’Connell et al., 2019; Çirak Kurt & Yildirim, 2018) and inclusive dialogues about interest-based learning opportunities (Bonar, 2018; Lo Bianco & Aliani, 2013) are addressed. Other studies provide findings relevant to conceptual understanding, engagement and motivation in science (Barnes et al., 2015), citizenship ed-ucation in post-communist European countries (Jelizakova, 2015) and geometry (Hock et al., 2015).
Four studies focus on the importance of thoughtful feedback, which is uncontested for the success of pupils’ learning and devel-opment: Nauman et al. (2011) about pupil writing, Bang and Kim (2016) about practices of praising in the USA and Korea, Kotul’´akov´a (2019) about global citizenship and Militello, Bass, et al. (2013) analyze how schools use and misuse data and provide implications to improve school leadership, teacher pedagogy and equitable pupil learning.
3.2.3. Social and emotional development
Three studies provide implications most relevant for the area of social and emotional development. Heffernan (2017) reports that resilience among young women is understood differently and that there might be an even larger difference between teachers and pupils. In terms of offering educative and restorative behavior support, Atkinson and Rowley (2019) suggest a person-centered approach to pupils’ reintegration into mainstream education and list generally accepted strategies. Kim and Oh (2017) investigate different stakeholders’ perceptions of a zero-tolerance policy for school violence in South Korea and conclude with a need for educational interventions instead of punishments for a positive transformation via self-reflection.
3.2.4. Systems of support
The holistic SoLD framework also includes addressing pupils’ individual needs to overcome learning and development barriers. De Leeuw et al. (2019) investigate pupils’ perspectives on resolving social exclusion. Subba et al. (2016, 2017) find that teachers attach comparatively less significance in their practices, to autonomy support for pupils with learning difficulties. Individual needs is a controversial issue, as teachers opposed to inclusion practices believe they have to focus their attention on the majority of the class (Brown, 2016), while special education teachers regard inclusion as a natural quality of schooling (Sabo et al., 2018). As a consequence of lacking teacher self-efficacy regarding teaching in inclusive classrooms, Berry (2010) demands more opportunities for teacher education that meet the needs of a wide range of pupils and how to access and effectively manage resources.
Two different special education leadership models were proposed in the included studies. Firstly, a distributed leadership con-tinuum (Tudryn, 2012) focusing on collaboration and secondly, a transitional action model (Boscardin et al., 2018) that allows special education leaders to use facilitation to cultivate relationships, empower others, understand multiple perspectives and navigate complex, dynamic, organizational systems. Ramsay et al. (2018) reinforce the importance of collaboration among school staff regarding interventions. Heasly (2017) suggests incorporating educational psychologists in regular school routines, to support pupils to have a voice and prevent burnout among teaching staff (Atkin, 2019). Militello and Janson (2014) illustrate that there exists a dissonance between the ideal practice and the current practice of school counselors, often grounded in organizational constraints.
3.2.5. Teacher characteristics
Four out of the 74 included studies focus on teachers’ professional identity: Levine and VanSlyke-Briggs (2014) on being a renegade teacher, Wirth (2014) about the existence of pluralistic ignorance amongst secondary teachers in terms of influences on their teaching practices, Demir (2016) about Turkish pre-service teachers overall satisfaction with their choice of education and Dobrica-Tudor & Th´eorˆet, 2017 instruct teachers to critically reflect on their well-being and conclude with the importance of conditions and constraints inherent to the workplace.
Included studies also focused on teachers’ role as policy arbiters in discussion about the use of digital technologies in schools (Vandeyar, 2014) and local language planning processes (Kemp, 2014; Slaughter et al., 2020). Ramlo illustrates how Q can be used for pupils’ evaluation of teaching (2017) or the determination of in-service teachers views about learning during their professional development (2012).
4. Discussion
The present paper aimed to systematically review academic publications that used Q methodology in compulsory education set-tings to identify participants’ subjective viewpoints. The study selection process yielded 74 studies. The characteristics of these Q studies published between 2010 and 2019 were presented first, then, a narrative analysis based on the SoLD principles of practice, as put forward by Darling-Hammond et al. (2019), served as an analytical framework for the included studies’ results and implications. The discussion presents delimitations and limitations, followed by the most relevant results of the current review study. The paper concludes with directions for further research and implications for practice.
4.1. Delimitations and limitations
First, even though no exclusion criterion for non-English publications was applied in this review, studies not using the English terms
Q method or Q methodology, but a translation of them, might not have been identified. In addition, it has to be acknowledged that
review, because they reported findings based on a different analytical procedure than Q factor analysis and were therefore “not in line with the original aim of Q methodology” (Brown, 2019b, p. 568). However, 18 of them used Q sorting as a data collection technique and might have provided additional insights into characteristics of Q sampling and Q sorting procedures. Third, the review study included not only traditional literatures but also gray ones. Even though the latter are often associated with a lack of quality, the current study was set out to illustrate and represent how Q has been used recently. It is important to note some included studies did not state all information about study characteristics reviewed in the present paper. Furthermore, when the initial literature search was conducted in January 2020, some of the publications were listed as in press. With an updated reference list, two articles (Dhillon et al., 2020; Slaughter et al., 2020) are now listed as 2020 publications. Finally, the review of the rationale for choosing Q in the included papers and the limitations of the methodology could not be carried out as originally intended. In the included publications, these sections were largely based on standard literature (see e.g. Brown, 1980; McKeown & Thomas, 2013; Watts & Stenner, 2012), and do not stem from the experience of author(s) in the individual studies.
4.2. Flexibility of Q methodology
Stephenson (1980) argued that Q was not prescriptive and items used in a sort could mean different things to different people and, depending on the situation, have different meanings for the same person. Presenting the diverse and wide-ranging applications in educational research, this review further showcased Q’s flexibility. The included studies comprised elementary and secondary levels of compulsory education. Q publications that did not meet the inclusion criteria encompassed studies in preschools and tertiary settings. The wide range of participants also evidences the methodology is flexible enough to be used with adults, older pupils and younger pupils. The fact that sorting items vary in number, from 14 to 78 items, and in format, including text but also images, are other features that support Q’s flexibility. In addition, the result section about research foci further presents the methodology’s application for various projects. However, the full potential of Q’s flexibility in terms of study designs is not yet tapped. For example, more studies using items other than written statements could further strengthen the methodology’s application for pupils with disabilities.
Whilst for researchers who are familiar with Q methodology its flexibility can be considered an advantage, for the novice Q researcher who might require clear guidance while designing their study, this flexibility and wide range of applications could seem to be a disadvantage. As an example, culling Q items (see step 2 in section 1.2.1) is an integral aspect of the Q process. However, both the art of reducing the concourse and the lack of clear requirements regarding the size of the final Q set is a common problem faced by Q researchers (Brown et al., 2019).
4.3. A methodology for the otherwise marginalized
The variety of participant cohorts showcases Q’s suitability as a research method for all members of a school setting, without a limitation in age and verbal understanding. It can therefore be concluded that Q is sufficient to provide a holistic and comprehensive collection of perspectives in educational research. An unexpected finding is that the cohort of pupils was the second most frequently studied group of participants. This finding can be regarded as a confirmation of Q’s potential to explore subjectivity and perspectives of otherwise often marginalized pupils (Brown, 2006; Ellingsen et al., 2014). On the other hand, voices of school staff other than teachers or members of the principal class and specifically studies investigating teacher educators’ subjectivity are underrepresented. This is surprising as several included studies call for an adaptation of pre- and/or in-service teacher education. The review study uncovers a lack of communication between teacher education and teachers’ actual work environment. However, it needs to be noted that this conclusion only concerns Q studies. Teacher educators’ perspectives might have been gathered with alternative research methods.
4.4. A participatory approach to policy formation
Across different subjects, included studies are demanding a participatory approach to policy formulation and implementation, illustrating how Q offers insight into perspectives and subjectivity of different actors. Applying Q, especially in a participatory fashion, where respondents are given the possibility to contribute to various stages in research, could considerably increase the sense of accountability and transparency, thicken its democratic dimension (Howe, 2004) and be beneficial for educational research. This is of particular relevance during reformation processes regarding inclusive education policies, pupils’ well-being and their educational success. By granting pupils access to otherwise entirely adult conversations due to the methodology’s characteristic of a non-verbal approach, policies are expected to meet pupils’ needs even more accurately (Heasly, 2017) and teachers are better equipped to successfully implement inclusive education. In summary, Q is a potent way to either include stakeholders into policy formation processes or at least inform and educate policy-makers and the larger community about educational issues from participants’ sub-jective point of view.
4.5. Implications for practice
Educational practice can be improved by systematic and empirical evidence about subjectivity gathered by Q research. Results in this review study illustrate how this engaging and at times provocative methodology visualizes the honest and valuable standpoints of stakeholders in educational settings and takes their situated complexity into account. To move from researchers’ practice to that of teachers, a range of included studies have suggested the application of Q, and especially its item sorting technique, as an educational tool for critical reflection or for evaluating a particular aspect of learning (see Table 2: categories C and D). As an articulation tool
within classroom dialogues or an instrument for formative feedback, a prominent component of SoLD on productive instructional strategies, Q allows the conjunction of teachers’ and pupils’ subjectivity in a constructive way and supports their learning and development. Q sorting was also applied for teachers to gain a better understanding of their pupils, as it allows them to group in-dividuals (Beck, 2017), understand where they are coming from and continually assess if they are making growth in their competence (Sklarwitz, 2017). The review showcases Q’s sheer limitless potential as an educational tool to support pupils’ learning and devel-opment due to the methodology’s flexibility in terms of study focus and design. However, more research is needed to investigate how educators with limited or no knowledge of Q can integrate Q sorting in their teaching practices. Based on the experience and moti-vation afforded by post-sorting discussion with learners, educators might move on to using Q in line with its original aim of elucidating shared subjective viewpoints.
4.6. Directions for future research
Even though the present review expands insights into the potential of Q to study subjectivity in educational research, a range of questions remains for future research. First, while included studies focused especially on areas of supportive environmental conditions and productive instructional strategies, the SoLD area of social and emotional development of pupils is underrepresented. With its reputation to include marginalized voices, Q is considered to be well-suited for research illuminating pupils’ skills, habits and mindsets for successful learning and development. Strengthening this research trajectory would further develop the understanding of how to meet pupils’ individual social and emotional needs.
Second, a considerable number of studies were excluded because they analyzed data gathered in Q sorting activities with other approaches than Q factor analysis. The carried-out analysis was mostly descriptive statistics and Q factor analysis would provide a fruitful basis for future research and further highlight the strengths and weaknesses of the complete methodology. Moreover, it can be expected to see further implications of potential applications of Q sorting as an educational tool. It is therefore recommended that future studies explore the possibilities of individual elements of Q methodology, especially its sorting technique, as a tool in educa-tional practices.
Third, whereas teacher and pupil perspectives are highly represented in the included studies, parent and teacher educator per-spectives, either as a single cohort or as part of mixed participant groups, are largely missing. This gap suggests the need for new research to supplement current educational research on improving communication between teacher education providers and their recipients thus highlighting more voices.
Fourth, the review reveals a complete absence of Q methodological single case studies in educational research within its time span from 2010 until 2019. Future Q research in the field might aim to further tap the methodology’s potential by investigating single subjects under multiple conditions of instruction. Finally, it needs to be noted that the review was specifically focused on Q studies executed in compulsory education and on teacher training for compulsory education. Q studies from other educational settings, such as daycare and higher education or other research fields were therefore excluded. Future reviews exploring the characteristics and implications of Q studies in these additional settings could prove to further contribute to a science of subjectivity.
5. Conclusion
Focusing on compulsory education, the characteristics of Q methodological studies since 2010 were systematically reviewed. In this ten-year timespan, 74 included studies illustrated wide-ranging and diverse ways in which researchers have been using Q as a means to analyze subjectivity, including participants who tend to be marginalized. To identify the implications that the selected Q studies have for educational research purposes, the studies were grouped within the SoLD framework (Darling-Hammond et al., 2019). According to its inventor, Q “stands for discovery in subjectivity, of reality in nature, made possible by technique” (Stephenson, 2014, p. 43, emphasis in the original). This methodology allows representing a person’s internal standpoint or subjective point of view as empirically observable, meaningful and relational, hence operantly subjective in the form of a Q sort (Stephenson, 1968). As illustrated in this review study, Q offers a flexible and systematic approach to data collection and analysis about subjectivity and is suited for educational research. For researchers wishing to identify the subjective viewpoints of their participants without imposing their own, this review study offers an overview of how Q methodology has been used in compulsory education to explore a range of diverse, dynamic and complex contexts in this field.
Submission declaration
The article is not under consideration elsewhere and all authors have approved the final manuscript. A previous form of the article is published in the corresponding author’s doctoral thesis (Lundberg, 2020).
CRediT authorship contribution statement
Adrian Lundberg: Conceptualization, Methodology, Validation, Resources, Formal analysis, Investigation, Writing - original draft,
Visualization, Project administration. Renske de Leeuw: Conceptualization, Methodology, Validation, Formal analysis, Investigation, Writing - review & editing, Visualization. Renata Aliani: Conceptualization, Methodology, Validation, Formal analysis, Investigation, Writing - review & editing.
Declaration of competing interest
None.
References
References marked with an asterisk indicate studies included in the review sample.
Alexander, P. A. (2020). Methodological guidance paper: The art and science of quality systematic reviews. Review of Educational Research, 90(1), 6–23. https://doi. org/10.3102/0034654319854352
* Anderson, C., & Jacobson, S. (2018). Barriers to environmental education: How do teachers’ perceptions in rural Ecuador fit into a global analysis?. Environmental
Education Research, 24(12), 1684–1696. https://doi.org/10.1080/13504622.2018.1477120.
* Atkin, L. V. (2019). Support and supervision for emotional literacy support assistants (ELSAs): Gathering ELSAs views about the support offered to them. Doctoral dissertation. University of Sheffield.
* Atkinson, G., & Rowley, J. (2019). Pupils’ views on mainstream reintegration from alternative provision: a Q methodological study. Emotional & Behavioural
Difficulties, 24(4), 339–356. https://doi.org/10.1080/13632752.2019.1625245.
* Baltrinic, E. R., Jencius, M., & Brown, S. R. (2016). Excellent teaching as viewed by international teaching elites. Operant Subjectivity, 38(3–4), 16–28. https://doi.
org/10.15133/j.os.2016.009.
* Bang, H., & Kim, J. (2016). Korean and American teachers’ praising styles and teaching practices. Mid-Western Educational Researcher, 28(1), 28–54.
* Bang, H., & Montgomery, D. (2010). Exploring Korean and American teachers’ preferred emotional types. Roeper Review, 32, 176–188. https://doi.
org/10.1080/02783193.2010.485305.
* Barnes, C., Angle, J., & Montgomery, D. (2015). Teachers describe epistemologies of science instruction through Q methodology. School Science & Mathematics, 115 (3), 141–150.
* Baron, C., Sklarwitz, S., Bang, H., & Shatara, H. (2019). Understanding what teachers gain from professional development at historic sites. Theory & Research in
Social Education, 47(1), 76–107. https://doi.org/10.1080/00933104.2018.1489927.
* Beck, P. D. (2017). Using works of art to give a voice to culturally diverse students: Q-methodology study. Journal of Leadership and Instruction, 16(2), 18–25.
Berliner, D. C. (2002). Educational research: The hardest science of all. Educational Researcher, 31(8), 18–20.
* Berry, R. A. W. (2010). Preservice and early career teachers’ attitudes toward inclusion, instructional accommodations, and fairness: Three profiles. The Teacher
Educator, 45, 75–95. https://doi.org/10.1080/08878731003623677.
* Berry, R. A. W., Shields, B. A., Krickovich, S. R., & Sadler, J. T. (2012). Prospective student teachers’ concerns regarding the student-teaching experience. Operant
Subjectivity, 35(3), 159–175.
* Bonar, G. (2018). When I think of learning about Asia, … Students’ views of Asia, and their motivations towards Asia-related learning in two Australian secondary schools. Doctoral dissertation. Monash University.
* Boscardin, M. L., Schulze, R., Rude, H., & Tudryn, P. (2018). Do leaders of special education share similar perceptions of standards guiding the leadership and administration of special education?. Journal of Special Education Leadership, 31(2), 61–84.
Brown, S. R. (1980). Political subjectivity. Yale University Press.
Brown, S. R. (2006). A match made in heaven: A marginalized methodology for studying the marginalized. Quality and Quantity, 40(3), 361–382. https://doi.org/ 10.1007/s11135-005-8828-2
Brown, S. R., Danielson, S., & van Exel, J. (2015). Overly ambitious critics and the medici effect: A reply to Kampen and Tam´as. Quality and Quantity, 49, 523–537.
https://doi.org/10.1007/s11135-014-0007-x
* Brown, Z. (2016). “We just have to get on with it”. Inclusive teaching in a standards driven system: The design decisions of a Q methodological study. Operant
Subjectivity, 38(1), 1–14. https://doi.org/10.15133/j.os.2016.001.
Brown, S. R. (2019a). A time for manifestos? Q method mailing- and discussion list. January.
Brown, S. R. (2019b). Subjectivity in the human sciences. Psychological Record, 69, 565–579. https://doi.org/10.1007/s40732-019-00354-5
Brown, S. R., Baltrinic, E., & Jencius, M. (2019). From concourse to Q sample to testing theory. Operant Subjectivity, 41, 1–17. https://doi.org/10.15133/j.os.2019.002 * Brown, C., & Militello, M. (2016). Principal’s perceptions of effective professional development in schools. Journal of Educational Administration, 54(6), 703–726.
https://doi.org/10.1108/JEA-09-2014-0109.
* Burke O’Connell, N., Dempsey, M., & O’Shea, A. (2019). An investigation of students’ attitudes to science, mathematics and the use of technology in lower secondary education. International Journal of Education in Mathematics, Science and Technology, 7(4), 319–334.
Cantor, P., Osher, D., Berg, J., Steyer, L., & Rose, G. (2019). Malleability, plasticity, and individuality: How children learn and develop in context. Applied
Developmental Science, 23(4), 307–337. https://doi.org/10.1080/10888691.2017.1398649
* Çirak Kurt, S., & Yildirim, I. (2018). The students’ perceptions on blended learning: A Q method analysis. Educational Sciences: Theory and Practice, 18(2), 427–446.
https://doi.org/10.12738/estp.2018.2.0002.
* Coladonato, J. A. (2013). Attitudes about second-career teachers: An exploratory Q study of school administrators and classroom teachers. Doctoral dissertation. Long Island University.
* Collins, L. J., & Liang, X. (2014). Task relevance in the design of online professional development for teachers of ELLs: A Q methodology study. The Turkish Online
Journal of Distance Education, 15(3), 268–281.
* Cooper, R. (2018). A step towards clarifying TA ambiguity: A Q methodological study to elicit the views of teaching assistants regarding their current and ideal roles. Doctoral
dissertation. University of Sheffield.
Darling-Hammond, L., Flook, L., Cook-Harvey, C., Barron, B., & Osher, D. (2019). Implications for educational practice of the science of learning and development.
Applied Developmental Science. https://doi.org/10.1080/10888691.2018.1537791
* Davis, L. M. (2018). Urban elementary school principals’ views of principal professional development: A Q methodology study. North Carolina State University. Doctoral dissertation.
* De Leeuw, R. R., de Boer, A. A., Beckmann, E. J., van Exel, J., & Minnaert, A. E. M. G. (2019). Young children’s perspectives on resolving social exclusion within inclusive classrooms. International Journal of Educational Reserach, 98, 324–335. https://doi.org/10.1016/j.ijer.2019.09.009.
* Demir, S. (2016). An analysis of pre-service teachers’ attitudes and opinions regarding the teaching profession via Q-methodology. European Journal of Contemporary
Education, 17(3), 295–310. https://doi.org/10.13187/ejced.2016.17.295.
* DeVore-Wedding, B., Thomas, J., & Montgomery, D. (2018). Determining optimal professional development formats: A Q-methodology study of science teachers’ preferences. Operant Subjectivity, 40(1–2), 1–20. https://doi.org/10.15133/j.os.2018.001.
* Dhillon, J. K., Howard, C., & Holt, J. (2020). Outstanding leadership in primary education: Perceptions of school leaders in English primary schools. Management in
Education, 34(2), 61–68. https://doi.org/10.1177/0892020619885942.
* Dobrica-Tudor, V., & Th´eorˆet, M. (2017). La m´ethode Q dans l’´etude des r´eflexions des enseignants du secondaire sur leurs pratiques professionnelles. Mesure et ´
evaluation en ´education, 40(2), 125–153. https://doi.org/10.7202/1043570ar.
Dryzek, J. S. (2005). Handle with care: The deadly hermeneutics of deliberative instrumentation. Acta Politica, 40, 197–211. https://doi.org/10.1057/palgrave. ap.5500099
Dryzek, J. S., & Holmes, L. (2002). Post-communist democratization: Political discourses across thirteen countries. Cambridge University Press.
* Duncan, N., & Owens, L. (2011). Bullying, social power and heteronormativitiy: Girls’ constructions of popularity. Children & Society, 25, 306–316. https://doi.
org/10.1111/j.1099-0860.2011.00378.x.
