Faculty of Social and Behavioural Sciences
Graduate School of Childhood Development and Education
‘Good research’ within professional higher education
institutes and traditional universities
– the criteria of lecturers
Research Master Educational Sciences Thesis 2
Bart Roosenboom
Supervised by Prof. Sjoerd Karsten & Didi Griffioen
Abstract
Introduction: In Europe, both at traditional universities and non-university institutions of higher education, research and knowledge-generation plays an (increasingly) important role both within the curricula and outside the teaching-context. Little is known about how lecturers (faculty whose main task is to teach and not to research) perceive research. Their views and beliefs are known to be very influential when it comes to the direction and the assessment of (student) research. One of the relevant questions concerns the criteria they apply in evaluating whether research is ‘good’. Method: To explore this question, a qualitative mixed-methods study was conducted with 25 lecturers from Dutch institutes of higher professional education (‘hogescholen’) and 20 lecturers from Dutch traditional universities. The participants were asked, either in groups (hogeschool lecturers) or individually (university lecturers) to elaborate on personal examples of ‘good’ and ‘non-good’ research. The transcripts of the discussions and interviews were transcribed ad
verbatim. The statements and criteria they used to qualify research were labeled, categorized
and compared, according to the method of grounded theory. Results: the criteria of both groups of lecturers concerned 6 main aspects (ordered from most prevalent to less prevalent): 1) the design of research; 2) the final product presenting the result of a study; 3) the value of the research and its outcome; 4) the researcher as a person; 5) the way the research was
conducted; and 6) the theme or problem that was researched. Both groups of lecturers
brought up the same aspects, only some differences were found in the underlying themes of criteria. The most prominent difference was that hogeschool lecturers attached much value to the utility value of research and the practical relevance, whereas the university lecturers were more concerned with the scientific value and the scientific relevance of research.
1. Introduction
During the last decades, teaching and research have gone hand in hand at Europe’s traditional universities (Rupp, 1997). Over the last years, also non-university institutions of higher education have increasingly begun to conduct research all throughout Europe (Kyvik & Skodvin, 2003; Lepori & Kyvik, 2010; Vos, Borghof & Staa, 2007). These institutions are transforming from teaching-only institutions into institutions that generate new knowledge and information, which mainly takes place through research (Kyvik & Skodvin, 2003). In practice, this means that many lecturers who previously only educated profession-specific courses now also have to teach research methods, supervise research projects and in some cases also conduct research of their own (Skoie, 2000). At traditional universities it is very common for most faculty to conduct both teaching and research tasks, although there are also non-researching lecturers at these institutes.
For the professional institutions to make the transition from only teaching to hybrids of teaching and research, it is imperative that their lecturers are taken into account. Modern theories of organizational and educational change describe how an organization’s employees belong to the most important stakeholders and acknowledge that their cooperation is therefore crucial for any change to succeed, as is the case for lecturers in educational organizations (see for example Fullan, 2007 or Jones, 2010). In the end lecturers are the persons that will be working in these transformed institutions, and they are the sole connection between the institution and its students. Hence, it is crucial that they support and work with these changes. It is therefore relevant to understand how lecturers think about research and what they believe ‘good research’ entails. Previous studies on research conceptions, however, have always either focused on students or researchers (see theoretical framework). Little is therefore known about lecturers’ conceptions of ‘good research’, not to mention the fact that no research-conception studies have looked at non-university higher education institutes. This
study will therefore extend the current scientific knowledge of research conceptions by researching the views of lecturers in Dutch higher education institutes, specifically focusing on the question what lecturers at both traditional universities and higher professional education institutes (hogescholen) believe constitutes ‘good research’.
2. Theoretical Framework
A vast amount of research has been conducted on the question ‘what (good) research is’. These studies can be divided in two categories: 1) research into people’s individual conception of (good) research, and 2) research to establish standardized lists of objective criteria to rank research. The goals of these categories of research are very distinct; whereas the first category of research tries to understand (or map) different perspectives, the second category intents to establish a consensus and create an utilizable final product. Most research on (good) research falls within this second category. Especially within the medical/epidemiological sciences, many of these second-category studies have been conducted to establish criteria lists to assess the quality of research for conducting systematic reviews (see for example Verhagen, de Vet, de Bie, Kessels, Boers, Bouter & Knipschild, 1998; The Cochrane Collaboration, 2010). Since this category of research does not offer a better understanding of the different perspectives on research (perspectives are only used as a tool to create standardized criteria lists), the results of this type of studies is not relevant for understanding the views of lecturers.
The first category of research studies, however, is very relevant for understanding individuals’ perspectives of research. However, only a small amount of studies have been conducted in this area, of which the majority focuses on people’s research conceptions (i.e. investigating how they perceive the concept of research). This research has only been conducted at universities, either focusing on scientific staff (Kiley & Mullins, 2005 ;
Visser-Wijnveen, van Driel, van der Rijst, Verloop & Visser, 2010) or students and PHD-students (Meyer, Shanahan & Laugksch, 2005 ; Pitcher, 2011).
Only a handful of studies looked specifically at people’s perception of ‘good research’. These studies can be divided in two types of approaches: open-ended and closed-ended. Table 1 provides an overview of these different studies, with their participants and the relevant research question(s) and outcomes. Some of these studies had several research questions and outcomes, only those relevant to this study have been included in the table.
Table 1
Overview of different studies exploring researchers’ conceptions of (good) research
Note. Only relevant research questions/outcomes were included
Authors Participants Research question Outcome
Closed-ended Albert, Laberge & McGuire (2012) Scientists (Social, biomedical and clinical scientists)
Assess the quality of different types of research, based on:
Funding Publications Presentations
Most researchers (regardless of academic
discipline) prefer classic forms of ‘quality control: Non-business (‘neutral’) funding preferred over
business-funding
Peer review publications preferred over publications in non-scientific mediums Academic conferences preferred over any other
type of presentation or (television) appearance
Open-ended Hemlin (1991) as referred to in Langfeldt (2001) Disciplinary communities of scientists
What are the aspects and attributes of good research in different disciplinary scientific communities?
Aspects: research problems, the methods and the results.
Attributes:
Stringency and correctness Novelty
Depth and breadth
Intra- and extra-scientific relevance Productivity Kiley and Mullins (2005) University faculty (research supervisors) What are characteristics of ‘good research’?
Four types of good research characteristics: 1) originality and creativity
2) technical quality aspects of the research; its design and conduct
3) honesty and ethically just 4) communication about the results 5) usefulness and applicability of research Guetzkow, Lamont and Mallard (2004) Peer-reviewers (researchers) from social sciences and humanities
What does the criterion ‘originality’ mean to researchers in peer-evaluation panels?
Using a new approach, theory or method Using new data
Studying a new topic, doing research in an understudied area
Producing new findings Langfeldt
(2001)
Theory / previous research
What are the types of ‘biases’ of individual research evaluators?
1) Preconceptions of good and valuable research 2) Professional research interests: taking effects on economic and political standing of the field/research area into consideration. 3) General or personal cognitive constraints 4) Personal interests: taking effects on personal situation or situation of friends into account Lamont and Mallard (2005) Peer-reviewers (researchers) How do researchers in peer-evaluation panels evaluate the quality of research?
Evaluations are mainly based on whether the epistemology of the research matches with the epistemology supported by researchers.
Four types of research epistemologies were found: 1) constructivist (research to give voice to groups of people, to understand their perspective) 2) comprehensive (theoretical approach to understand social phenomena)
3) positivist (uses hypotheses and quantitative techniques to answer a question)
4) utilitarian (goal is to generate useful, policy-oriented, knowledge)
These studies show a number of things. Firstly, the studies of Hemlin (1991) and Kiley & Mullins (2005) found that there are a number of criteria that researchers use to evaluate research as good, or not. Both found that researchers took the following factors into account: 1) the set-up of the research (methods/design/creativity/correctness), 2) the research problem or topic (originality/creativity/depth/breadth), 3) the relevance of the study (intra- and extra-scientific relevance), and 4) the study’s results (productivity/depth/correctness /originality). Lastly, Kiley and Mullins also found that 5) the way the results are communicated influences the opinions of researchers. This was confirmed by Albert, Laberge & McGuire (2012), who found that researchers –regardless of their discipline- evaluate studies published in scientific articles higher, especially when these are published in peer-reviewed journals.
With regard to the criterion (or sub-criterion) of originality, Guetzkow, Lamont and Mallard (2004) found that this can mean a lot of things: “using a new approach, a new theory, a new method, or new data, studying a new topic, doing research in an understudied area, and producing new findings” (Lamont & Mallard, 2005, p.11). Originality can therefore be taken into account by researchers in any of the five factors that were just presented.
Lastly, Langfeldt (2001) and Lamont & Mallard (2005) found that also biases and preferences play a role for researchers in determining the quality of research. The first type of bias described by Langfeldt (a preconception of good and valuable research) accords with Lamont & Mallard’s conclusion that researchers’ adherence to certain epistemologies play an important role. These epistemological preferences are often related discipline-specific. According to Langfeldt, also other personal and professional interests and constraints can play a role in researchers’ evaluation of the quality of research.
All these studies looked at lecturers’ conceptions of good research. The conceptions of other academics, such as students or lecturers, have not been subject of research. The
perspectives of lecturers, relatively neutral experts who are nevertheless very well-informed about research, can be very helpful in understanding the concept of good research within the context of higher education. Furthermore, none of the studies looked at the perspectives of people in non-university higher education institutes. Since research does take place there, as explained in the introduction, it is very relevant to see whether the conceptions of academic staff at these institutes differ from the conceptions of university staff.
To increase our understanding of ‘good research’ conceptions, this study will explore two research questions:
1) What are the aspects of good research according to lecturers in higher education institutes in the Netherlands?
2) What are the differences between the conceptions of lecturers at traditional universities and non-university higher education institutes (hogescholen)?
In other words: this research will investigate lecturers’ conceptions of ‘good research’ and the underlying criteria aspects. These type of conceptions are not something solid. Depending on the situation, context and/or time, a person often holds different conceptions (Holstein & Gubrium, 1995). Hence: conceptions are context-specific. The goal of this research is therefore to get an overview of the different criteria lecturers use when being asked to reflect on ‘good’ (and ‘non-good’) research as a lecturer in the context of their university or hogeschool.
3. Methods
3.1. Design
Lack of previous research on research perspectives of this particular group made it impossible to construct hypotheses. An explorative qualitative design was therefore deemed most suitable to investigate the lecturers’ conceptions of good research and their underlying
criteria. Because of this, information was only gathered through open-ended and unstructured questions. Since two different groups of lecturers were studied, the data-collection consisted of two distinct phases: data on conceptions of the hogeschool-lecturers’ was gathered first, data on the university-lecturers’ conceptions was gathered at a later stage.
3.2. Method of data collection
Data on the criteria of hogeschool lecturers was gathered through five focus group sessions. This method was chosen because it was expected that lecturers would be better able to construct and formulate their thoughts through interaction with other lecturers (see for example Bloor, Frankland, Thomas & Robson, 2001), since most lecturers in hogescholen are not used to talk about research criteria. To avoid underrepresentation of intra-group variations, the sessions were split in an individual and a group-wise part, as will be further explained in the procedure.
Data on the criteria of the university lecturers was gathered through individual interviews1. The main reason that this method was chosen was of practical nature and had to do with the limited time available for data-gathering. It was expected to be less probable to find an adequate number of university lecturers to participate in a focus group sessions at a previously determined time and location, whereas it was anticipated to be more likely to find enough participants to participate in an individual interview at a time and location of their choosing. Furthermore, in contrast to the hogeschool lecturers, university lecturers are expected to be far more experienced in discussing research. They are therefore expected to have more well-developed opinions and therefore have less need for a co-constructivist method to form and express their opinion.
1 The use of these two different methods of data gathering does pose some limitations on the comparability of
3.3. Sample
The two groups of participants were approached differently.
The Hogeschool-lecturers that were approached had all participated in a large scale survey and indicated to be willing to participate in future research. Lecturers from five relatively large-scale hogescholen were selected on basis of their willingness and ability to join at the scheduled time (Nh = 25; for characteristics see table 2). One or two focus group meetings were held at each hogeschool (with 4 to 8 participants per meeting).
The university-lecturers were gathered using the snowball-method, based on the network of the researchers, and via the secretariats of several educational departments at the university of Amsterdam. Like the hogeschool-lecturers, the university lecturers had to spend the majority of their working hours on education-related tasks (as opposed to research-related tasks). As table 2 shows, the lecturers were matched to the hogeschool-sample in a 2x2 design: 50 % from a social sciences/gamma-disciplines and 50% from a sciences/beta-discipline, half of the lecturers had more than 5 years teaching experience (seniors) and half had less than 5 years of teaching experience (juniors). Since it proved to be very difficult to find junior (inexperienced) lecturers in a beta-discipline, two PhD-students were included that were also lecturer/teaching assistant during a few courses each year.
Table 2
Characteristics of the participants
3.4. Procedure
Both the interviews and the focus group sessions were set-up in a similar, open-ended, way. At the beginning of every focus group session and each interview, the participants were asked to come up with one example of good research and one of non-good research. This last phrasing was chosen instead of “bad research”, since the term bad research is more plural-interpretable than non-good (see also Kiley en Mullins, 2005). In other words: there is a thin line between good and non-good, whereas there is an entire continuum between good and bad.
3.4.1. Focus Groups
During the focus groups sessions, each lecturer, taking turns, explained what their example of good and non-good research was and why they qualified it that way. The other participants were asked not to react, but write down their questions and comments for later in
_________________________________________________ Lecturers Hogeschool University
(Nh = 25) (Nu = 20) _________________________________________________ Sex Male 15 11 Female 10 9 Teaching Discipline Social Sciences/ Gamma 13 10 Sciences/Beta 10 9 Multi-disciplinary 2 1 Teaching experience < 5 years 8 10 > 5 years 17 10 Education level HBO 9 - University 13 14 PHD 3 6
Conducts Research as part of job
Yes 15 9
No 10 11
the meeting. If the examples were apparently unclear or if the participants only explained one of the two examples, the focus group moderator would ask for further explanation. After all focus-group participants were done explaining their examples (usually after half an hour), the lecturers were asked, with as little interference as possible from the moderator, to discuss what they consider good and non-good research. Only when the discussion went off-topic for a long time (more than a few minutes), the moderator would lead the discussion back to good and non-good research. The discussion during the second half mostly lasted one hour. At some moment, the group would begin to repeat previously discussed topics and realise this themselves. At that moment, the moderator would step in and conclude the meeting.
3.4.2. Interviews
During the interviews, the university-lecturers were also asked to explain and elaborate on one example of good and one example of non-good research. However, often the lecturers started with one example and the interviewer asked clarification, probing and follow-up questions about the first example for 20-30 minutes. The other example, and underlying reasoning, was discussed during the last 10-15 minutes of the interview. Several times during the interview, the interviewer would sum-up what the interviewee said about good/non-good research and ask whether there are still other criteria/aspects of (non-)good research that were not brought up by the interviewer. The interview would end when the interviewee indicated that the summary was complete and that there were no other aspects. In three cases, the interviews also ended because the interviewee ran out of time.
All focus group meetings and interviews were recorded with a voice recorder.
3.5. Analysis
All focus group sessions and interviews were transcribed at verbatim. The resulting transcripts were analysed in three steps, using ‘content analysis’ in accordance with the analysis method of grounded theory (Charmaz, 2006). The focus group transcripts were
analysed first, the interview transcripts were analysed later. First, all separate arguments and statements concerning criteria for good and non-good research of the first part of the focus group meetings were labelled in vivo. Afterwards, codes covering similar themes were grouped and classified. After extensive discussions between the two coders, this resulted in a classification system of seven themes that covered almost all in vivo labels (see results section), by which the full focus group transcripts were coded. The university-lecturer interview transcripts were also coded using this classification system. However, a number of prevalent criteria in these transcripts did not fit the classification system. Therefore two categories were reformulated to incorporate these new criteria. Another small category was made into a subcategory, which resulted in a final classification system of 6 categories. All coding and structuring was done using Atlas.TI
In-depth qualitative descriptions of the different categories of criteria were composed by looking at similarities, differences and trends in the statements of all lecturers that were labeled as belonging to that category. The (quantitative) prevalence of each category was determined on basis of the amount of focus group meetings and interviews in which statements regarding that category were brought up.
4. Results
To be able to understand and interpret the aspects of ‘good research’, it is necessary to know what type of research the lecturers brought up in their examples of ‘good’ and ‘non-good’ research. Therefore, the differences in types of research will first shortly be introduced. Afterwards the six aspects of ‘good research’ will be presented. Thirdly, each aspect will be described in greater detail. Lastly, some differences between lecturers in the application of the underlying criteria will be discussed.
4.1. Different types of research
An important difference was visible between the two groups of lecturers. Most of the
hogeschool lecturers immediately started talking about ‘student research’ in their examples,
but also throughout the rest of the sessions. Most university lecturers, however, talked about research conducted by academic scholars or professional researchers, both in their examples of good and non-good research. This is likely to influence the specific criteria brought up by the two groups.
4.2. The six aspects of good research
Categorizing of the lecturers’ criteria showed that lecturers distinguish six aspects of ‘good research’, each of which had two or more underlying themes or types of criteria. Table 3 lists all aspects and underlying themes, in order of prevalence. Aspect 1: design of the
research was discussed during every single interview and focus group meeting, aspect 2: quality of the final product and 3: conduct of research were only slightly less prevalently
brought up. Aspect 4: value of the research and its outcome and 5: researcher related criteria
were discussed in 80% and 75% of all meetings and interviews. The sixth aspect: topic of the
research, was discussed in four out of five hogeschool focus group meetings, but in only 7
interviews with university lecturers. Hence, the first five aspects are important to almost all participating lecturers. However, different underlying criteria were brought up for each of the
aspects, therefore all aspects and their included criteria will now be described and explicated with examples. A detailed schematic overview of all aspects, the themes and their underlying criteria can be found in appendix I.
Table 3
List of aspects and themes of ‘good research’ criteria
4.2.1. Aspect 1: the design of the research
Every lecturer brought up criteria concerning this aspect. Two themes were prominent amongst these criteria: a) theoretical background and research questions, b) methodological/technical set-up. Especially criteria belonging to the second category were very prevalent in every interview and focus group session. As table 1 shows, both groups of lecturers discussed both themes. Each theme will now shortly be discussed.
With regards to the first theme, a number of criteria were put forward by the lecturers: research questions, sub-questions and hypotheses should be well-formulated and clearly demarcated, researchers should be aware of previous studies and literature on the research
1. Design of the research
a. Theoretical background and research questions hu b. Methodological/technical set-up hu
2. Quality of the final product a. Style and form hu b. The content hu
c. The argumentation and foundation hu d. Transparency about the conduct and set-up hu 3. Conduct of the research
a. Correct conduct hu b. Thorough conduct hu c. Dealing with uncertainties h 4. Value of the research and its outcome
a. Utility value h b. Scientific value u 5. Researcher related criteria
a. Positioning of the researcher towards the research hu b. Qualities of the researcher hu
6. Topic of the research a. Topic origin hu
b. Social and scientific relevance u
Note. h = brought up by hogeschool lecturs
u = brought up by university lecturers
topic, all (theoretical) concepts used in the research should be well-defined beforehand and researchers should not base their research on unfounded personal assumptions or biases. As one lecturer (respu 8) explained:
“A good preparation is knowing what [research on the topic] has been done and how
your research question fits within that context. If you are not aware of what has been done, you are either inventing a wheel or researching something that is useless because previous research has shown that it is impossible. So you are wasting time and money.“
With regards to the second theme, most lecturers agreed on two main criteria: 1) the chosen research methods should be suitable to study to research questions or test the hypotheses, and 2) the chosen methods should be sound, reliable and valid. The specific requirements that should be met for this second criterion differed much between the lecturers. However, these differences appeared to be related to disciplinary differences, but not to the
hogeschool or university context. Amongst others, the following requirements were brought
up several times by both groups of lecturers: 1) size and representativeness of the sample/data, 2) appropriate statistical techniques, 3) a good experimental design, 4) the use of multiple research methods (triangulation), 5) using a qualitative or quantitative design, 6) conducting research systematically, cyclical and continuously, 7) using an un-biased method that can convince even the most skeptic readers, and 8) taking into account the context and other influential factors.
4.2.2. Aspect 2: quality of the final product
Almost all lecturers made clear that they look at the way the research is presented to them in the final product when separating good from non-good research. Most hogeschool lecturers talked about students’ research rapports, but also advice rapports and (academic) articles came up. The university lecturers seemed to refer mainly to academic articles. Four
themes were distinguished: a) style and form of the final product, b) its’ content, c) the argumentation and substantiation and d) transparency about the research conduct.
With regards to the first theme, many lecturers stated that a good research rapport is well-structured, has cohesion, has no spelling or grammatical errors, and is written understandably. One lecturer (respu 13) explained:
“It’s all about convincing [your reader] (…) When you have done a very good study
but report it very badly, then (…) people will think ‘I understand it might be a relevant study, but I cannot get through the article, it is written down so badly’. Then people will put the article aside and [your contribution to the field] is lost.”
Furthermore, some university lecturers made clear that articles published in peer-reviewed journals are often good research, since a checks and balances have established that it has a certain quality.
With regards to the second theme, most lecturers (Nu = 20; Nh = 20) explained how they determine the quality of research on basis of content of the final product. One university lecturer (Respu 5) explained that “good research supplies you with all the information you
need to be able to judge whether it is trustworthy research”. When explaining what content
should be present in the final product, lecturers stated that it should contain an answer to the research question, recommendations, a good story, and different perspectives on the findings. Most university lecturers stated that the good research articles should contain new insights, the hogeschool lecturers did not bring this up. Several hogeschool lecturers believed that a rapport should be tailor-made to the organization for which the research was conducted and include utilizable recommendations for that specific organization.
Many lecturers, mostly working at a hogeschool, brought up criteria concerning the third theme: the argumentation and substantiation. Three criteria were brought up: 1) the results should be theoretically supported by theories/findings from good quality sources, 2) the arguments used in rapport/article should be valid and applicable, and 3) the conclusions should be based on the right data and results. Several university lecturers criticized that bad
research often draws conclusions that are too general or too far-fetching to be supported by the research results.
With regards to the fourth theme, most lecturers (Nh = 18, Nu = 16) stressed that transparency about steps taken in the research, and openness about the choices that were made by the researcher, are a prerequisite of good research. One lecturer (resph 1, session 2) stated that “the most important thing is to account for the different choices made, but what
those choices exactly are is less important”. Most lecturers explained that this transparency is
connected to the replicability or duplicability of a study. Another form of transparency that was discussed concerned a transparent argumentation and clearly visible results.
4.2.3. Aspect 3: the conduct of the research
Almost all lecturers (Nh = 22, Nu = 13) stated that they believe a study needs to be well-executed in order to be considered ‘good research’. Three themes are visible in the lecturers’ statements: a) the correctness of the conduct, b) the thoroughness of the conduct, and c) dealing with uncertainties.
Regarding ‘correctness of the conduct’, most lecturers stated that data should be correctly processed, material and methods should be correctly applied and analyses should be conducted correctly. The opposite of this, non-correct execution of the research was brought up by several university lecturers in their example of non-good research. Their examples ranged from inaccurateness and not being careful to data manipulation and even plain fraud.
With regards to the second theme, several lecturers stated that it is wrong to quickly jump to conclusions. One lecturer (resph 5, session 5) explained that many of his students do “quick and dirty” research, in which they come up with a solution for a problem without properly studying the problem itself first. Several university lecturers also complained about
the “lazy research” of some of their colleagues who always take the same approach to research even when it is not relevant. It lacks depth. One lecturer (respu 3) explained:
“Some persons have specialised themselves in one small sub-element of [my scientific
field], from which they try to explain everything, including answering the ‘big questions’ (…) because of that, they are always doing very negligible research that does not have much value, because it is often just plainly wrong.”
The third theme was only brought up by a small number of hogeschool lecturers. They stated that when confronted with difficulties, student-researchers should dare to make choices pro-actively. Choices that were brought up were 1) to stop the research, if there are not going to be any useful results, 2) to change the (methodological) design, if the chosen set-up is not working, and 3) to change the research questions, if another question turns out to be more relevant.
4.2.4. Aspect 4: the value of the research and its outcome
Almost all lecturers stated that good research needs to have a certain value. However, the two groups of lecturers talked about a different two type of values. The hogeschool lecturers (and two university lecturers) talked about the utility value, whereas most university lecturers discussed the criterion of scientific value.
Most hogeschool lecturers considered the utility value to be the factor that distincts
hogeschool research from university research. For most lecturers, the utility value meant that
a study should result in an applicable or utilizable outcome. Several uses of research (outcomes) were brought up: 1) some lecturers explained explicitly that research (outcome) should be useful for businesses or professional organizations, especially when the research was conducted on their behalf (by students), 2) others talked about the value research could have for their own teaching or their teaching department, 3) some lecturers also discussed the value of research in the perspective of student-researchers, for instance the value for their
future professional careers. One lecturer (resph 7, session 3) clearly explained his idea that the utility value is the difference between hogeschool and university research:
“The difference between research at hogescholen and research within the academic
world is that the higher professional education-sector is more focused on a profession in practice. (…) [hogeschool research] intends to improve that professional reality, whereas university research might only satisfy the curiosity of one individual researcher.“
The second type of value, the scientific value, was what most university lecturers found very important. Most university lecturers explained that research should be valuable in their scientific field in order to be considered ‘good research’. In most cases, this meant that the research should come up with a finding that is a useful contribution to the existing knowledge and theories in that field, for example by providing new theories, models, discoveries or insights. Several lecturers described that ‘very good’ research answers ‘the significant questions’ that need answers for their discipline to progress.
4.2.5. Aspect 5: researcher-related criteria
The lecturers’ statements about how researcher-related criteria demark good and non-good research can be divided in two types: 1) criteria concerning personal qualities of the researcher, and 2) criteria concerning the positioning of a researcher towards his/her research. Two-thirds of all statements about characteristics of the researcher concerned specific qualities a researcher should have to be able to conduct good research. Many lecturers talked about specific knowledge that a researcher should have, about research methodology, statistical analysis, or about the topic they are researching. Several hogeschool lecturers stressed that a good researcher should have a curious nature and a critical research-mentality. With regards to non-student researchers, several hogeschool and university lecturers stated that ability to share your knowledge, to involve others and to work cooperatively was an important characteristic of a good researcher. As one lecturer (respu 14) explained:
“A researcher should be critical enough to think ‘my idea is just an idea, I am enthusiastic about it, but I have to judge it critically. This is often done by talking about it with colleagues, visiting conferences… at the moment you publish it, you should have taken all possible tests.”
Some hogeschool and university lecturers that the intelligence of a researcher is important to be able to do research. “[To be able to] think outside the box, wield multiple perspectives and
handle a bigger complexity, you simply need more brainpower” (resp 4, session 4).
Furthermore, several hogeschool lecturers agreed that good information skills, such as the ability to read research articles and get the right information out of them, are a prerequisite of being a good researcher. In addition, several hogeschool lecturers stressed that a good researcher is able to look at problems from multiple perspectives and can see multiple ways to research a problem. Lastly, creativity and the ability to see and draw connections between different theories/studies within the research field were brought up by several university lecturers as an important characteristic a researcher should have.
When talking about the other researcher-related theme, the positioning of the researcher towards the research, several other criteria were brought up by both groups of lecturers. Firstly, several lecturers stated that good research can only be conducted by researchers who can think and act independently and objectively, who are critical about their own research and who are aware of their biases or assumptions. Secondly, some lecturers differed on whether researchers should be involved and passionate about their topic, or whether they should not be involved within their topic in real-life, to remain able to study it from a distance. Thirdly, with regards to (student) research that was commissioned by an external company or organization, several hogeschool lecturers phrased that a researcher should act independently from (incompany) instructors, while other hogeschool lecturers stated that a good researcher should be adaptive to the needs/wishes of these instructors. Fourthly, three university lecturers stated that in good research, the researcher should be aware of the context in which the research takes place, which includes awareness of (funding)
stakeholders involved in the research, but also awareness over other influencing (contextual) factors. Lastly, several hogeschool and university lecturers put forward the notion that research conducted by a researcher who is prejudiced on the outcomes can never be good.
4.2.6. Aspect 6: the topic of the research and its origin
A large group of lecturers stated that the topic of a study is important in their determination whether it is ‘good research’ or not. A smaller group of lecturers concluded that their perception of the quality of a study is not related to the topic or theme at all. Among the first group, two themes were discussed concerning this aspect: 1) the origin of the research topic/theme and 2) the social and scientific relevance of the research topic.
With regards to the origin of the research topic/theme, lecturers from hogescholen brought up different criteria than the university lecturers. Three criteria were brought up by
hogeschool lecturers: 1) studies that investigate problems brought up by external
organizations or businesses are good research, because it is helpful to the organizations that the students will work at in the future, 2) research focusing on topics brought up by lecturers is good research, because that can improve their teaching or the study’s curriculum, or 3) student research investigating problems related to students’ own interests is good research, because if the students are not interested in a topic the research will likely be of lower quality. Whereas a preference for the first criterion 1 seemed to be associated with teaching in business-related disciplines, whether or not a lecturer brought up criteria 2 and 3 did not seem to be associated with his/her disciplinary background. Not one of the university lecturers discussed any of these 3 criteria. Two university lecturers did talk about research brought up by external organizations or businesses, however they related this to the funding of the research. Research that was funded by corporations or other stakeholders with a preference for a certain outcome is never as good as more neutrally-funded research, they explained.
Most university lecturers (Nu=14), but none of the hogeschool lecturers, stated that good research should focus on a topic or theme that is scientifically relevant. Several teachers explained this as the research should intent to investigate something new, something that can contribute to the scientific field or scientific knowledge in general. This is highly related to the scientific value of research, which was discussed before, however, scientific relevance has to do with the intention to contribute when setting up the research, instead of judging its’ value on basis of the outcome of the research. Furthermore, several lecturers (Nu=5, Nh=1) brought up the criterion that good research should (also) be socially relevant. It should provide new insights or evidence for discussions happening in politics or problems occurring within society.
4.3. The standards for ‘good research’ in different contexts
Differences between lecturers were found concerning the extent to which their criteria of good research are applicable in every context. Whereas most lecturers seemed to adapt their standards of the criteria in certain contexts (e.g. student research), other lecturers were very strict and explained that research in these context simply cannot be good research. Both groups of lecturers will now shortly be described:
4.3.1. Lecturers with flexible criteria.
The group of lecturers that adapt their criteria to the context can be characterised as flexible in their conception of good research. Although both hogeschool and university lecturers belong to this group, the question ‘what is good research in different contexts’ was discussed extensively in every hogeschool group meeting, whereas only some university lecturers brought up this question themselves. The hogeschool lecturers discussed, for example, how good research in a hogeschool setting is different from good research conducted at universities and that therefore different standards should be set for the criteria apply. One lecturer (resph 5, session 5) gave the following example:
“In the research conducted in the university master program, we demand of all
students that they provide a good overview of all international literature on the topic of their research. If they did not do that, they simply failed their research project. […] At the bachelor study at the hogeschool we do not set such high standards. At least, they do not have to give an overview of academic literature. They do need to provide a good description of their research topic.”
Interestingly, the university lecturers never spoke about the context of hogeschool research. Two university lecturers did state that they believed applied research is lower quality research, because the research is less neutral as there are (funding) stakeholders and that have preference for a certain outcome, which limits the reliability of the research and its outcome.
Furthermore, several hogeschool and university lecturers stated that research conducted by students should be judged by different criteria than research conducted by professional researchers. Some lecturers used criteria that were roughly the same, but they believed that the extent to which they should be met differed per context. In other words: they use the same criteria, but with different standards.
4.3.2. Lecturers with rigid criteria.
A smaller group of lecturers, both from universities and hogescholen, could be characterised as very rigid in their conception of good research. They did not use flexible criteria and stated that research could only be good when all the criteria they discussed were met. They indicated that several examples they heard during the session were therefore ‘simply’ not good research. One hogeschool lecturer (resph 6, session 4) explained:
“the requirements [of research at hogescholen] should be the same as those that are
set for traditional university research. What you measure should be equally valid and reliable.”
5. Conclusion and discussion
The goal of this study was to explore the criteria used by lecturers in higher professional (hogeschool) education and university education to demark ‘good’ from ‘non-good’ research. The results show that the criteria of both groups of lecturers concern 6 main aspects (ordered from most prevalent to less prevalent): 1) the design of research; 2) the final product presenting the result of a study; 3) the value of the research and its outcome; 4) the researcher as a person; 5) the way the research was conducted; and 6) the theme or problem that was researched.
Although the criteria used by individual lecturers varied greatly (large intra-group variation), the differences between the hogeschool and the university sample were only minor (small inter-group variation). The criteria of both groups concerned the same six aspects, but some differences could be found in the underlying criteria. With regards to aspect 4, the value
of the research and its results, the hogeschool lecturers were concerned with the utility value
whereas the university sample looked at the scientific value. In aspect 6, the topic of the
research and its origin, some university lecturers were interested in the funding of a study
and its’ influences on the neutrality of the research. No hogeschool lecturer discussed this, but several hogeschool lecturers did state that the person or organization that brought the research topic up can influence their perception of a study being good research or not.
These differences can partly be explained by the fact that research at hogescholen in the Netherlands mostly focuses on themes and problems related to the professional context for which they educate their students. These themes are often brought up by professional organizations or education departments and these ‘suppliers’ would like the research to come up with a useful and often utilizable answer to their problem. Most research at traditional universities, however, intends to contribute to a scientific field. As shown by Langfeldt (2001), it is likely that the ‘good research‘ conceptions of the lecturers are influenced by their
professional research interests. In other words: the conceptions lecturers are likely to be positively biased towards the research conducted in their direct environment.
5.1. Relation to previous research.
The six aspects of ‘good research’ that were found are largely in accordance with the criteria/aspects of researchers/university faculty that were found in previous research. Most aspects of the lecturers are similar to the aspects of ‘good research’ that were discussed in the theoretical framework: 1) the design of the research is similar to the set-up of the research of Hemlin (1991) and Kiley & Mullins (2005). The importance of 2) the final product in ‘good research’ was also found by Kiley & Mullins (2005) and Albert, Laberge & McGuire (2012). 3) The utility value of the research is somewhat related to the study’s results and the
relevance of the study of Hemlin (1991) and Kiley & Mullins (2005). Aspect 5) the way the research was conducted was also brought up by the researchers in the study of Kiley and
Mullins (2005). Lastly, 6) the theme or problem that was researched (including the relevance of the topic) was also found to be an important aspect by Hemlin (1991) and Kiley & Mullins (2005).
The fourth aspect, the researcher as a person, was not found to be an important aspect in any of the previous studies. Perhaps this can be explained by the fact that many of the participating lecturers were not involved in research, but do see students and professional researchers conduct research in their daily work environment. This might give them a unique outsider perspective on the role of researchers that the participants in previous studies did not have, since those participants were researchers themselves (see table 1).
An aspect that was found to be important in previous research is the originality and
creativity of a study (Guetzkow, Lamont and Mallard, 2004; Kiley & Mullins, 2005).
Although several lecturers in the current study made statements about this, it was not a prominent theme amongst the lecturers. When creativity/originality was brought up, it was
mostly used in relation to one of the six aspects (e.g. creativity as one of important qualities a good researcher should have, or originality as a requirement for the research question in order for the research to have scientific value) .
5.2. Limitations and recommendations
There are certain limitations to the design used in this study. Most importantly, the comparability of transcripts from individual interviews and focus group sessions is limited. The focus groups resulted in criteria that groups of lecturers come up with, both indicating what norms exist among the lecturers in those meetings and what criteria caused discussion between the participants (Bloor et al., 2001). The interviews, on the other hand, allowed the university lecturers to voice their personal perspective and opinion, which may be very different from the norm. This limitation was partially averted by splitting the focus groups sessions in two parts, as was explained in the procedure-section. The first part in which the lecturers voiced their personal opinion without interruptions was very similar to the university lecturer interviews.
A second limitation concerns the explorative and open-ended nature of this study. Although many differences were found between the lecturers, it is uncertain whether these can be contributed to disciplinary differences, seniority, gender or other factors. A quantitative design or structured interviews in which all possible (categories of) criteria are presented to each lecturer could have provided more data on these individual differences and possible explanations. However, since the goal of the study was to come up with all possible ‘good research’ aspects, an open ended approach was the right choice. More structured research, however, is needed to test the newfound categories, and to possibly extent or merge the categories.
In future research, the link between conceptions and actions of lecturers needs to be researched as well. A study looking at lecturers’ application of the criteria can give insight
into how the different criteria are weighed in relation to each other when lecturers evaluate research. Such a follow-up study can also give an indication of the relation between conceptions and actions of lecturers, since in practice actions do not always logically follow conceptions (Visser-Wijnveen, 2009). A better understanding of the actions of lecturers by knowing their conceptions of research will help a focused implementation of research in higher professional education.
References
Albert M., Laberge S., McGuire W. (2012) Criteria for assessing quality in academic research: the view of biomedical scientists, clinical scientists and social scientists. Higher Education. doi: 10.1007/s10734-012-9519-2.
Bloor, M., Frankland, J., Thomas & Robson, K. (2001). Focus groups in social research. London: Sage.
Charmaz, K. (2006). Constructing Grounded Theory. A practical guide through Qualitative Analysis. Los Angeles: Sage Publications.
Fullan, M. (2007). Chapter 5 - The Causes and Problems of Implementation and
Continuation. In: The New Meaning Of Educational Change (4th ed.). New York: Teachers College Press.
Guetzkow, J., Lamont, M. & Mallard, G. (2004). What is Originality in the Social Sciences and the Humanities? American Sociological Review, 69, 190-212.
Hemlin, S. (1991). Quality in Science. Researchers' Conceptions and Judgements. Göteborg: University of Göteborg, Department of Psychology, Doctoral Dissertation.
Holstein, J. A., & Gubrium, J. F. (1995). Chapter 2 - The Active Interview in Perspective. In: The active interview (pp. 7-18). Thousand Oaks: Sage.
Jones, G. J. (2010). Chapter 10 - Types and Forms of Organizational Change. In: Organizational theory, design and change (6th ed.). Upper Saddle River, NJ: Pearson Education.
Kiley, M., & Mullins, G. (2005). Supervisors' Conceptions of Research. Scandinavian
Journal of Educational Research, 49(3), 245-262.
Kyvik, S., & Skodvin, O.-J. (2003). Research in Non-university Higher Education Sector - Tensions and Dilemmas. Higher Education, 45, 203-222.
Lamont, M. & Mallard, G. (2005). Peer Evaluation in the Social Sciences and the
Humanities Compared: The United States , the United Kingdom, and France. Ottowa:
Social Sciences and Humanities Research Council of Canada.
Langfeldt, L. ( 2002). Section 2.4: The concept of ‘bias’ in research evaluation. In:
Decision-making in expert panels evaluation research – Constraints, processes and bias (pp.
61-69). Oslo: GCS AS.
Lepori, B. & Kyvic, S. (2010). The Research Mission of Universities of Applied Sciences and the Future Configuration of Higher Education Systems in Europe. Higher
Education Policy, 23, 295-316.
Meyer, J. H. F., Shanahan, M. P., & Laugksch, R. C. (2005). Students' Conceptions of Research: A qualitative and quantitative analysis. Scandinavian Journal of
Educational Research, 49(3), 225-244.
Pitcher, R. (2011). Doctoral Students' Conceptions of Research. The Qualitative Report,
16(4), 971-983.
Rupp, J. C. C. (1997). Van oude en nieuwe universiteiten. De verdringing van Duitse door
Amerikaanse invloeden op de wetenschapsbeoefening en het hoger onderwijs in Nederland, 1945-1995. Den Haag: Sdu Uitgevers.
Skoie, H. (2000). Faculty involvement in research in mass higher education: current practice and future perspectives in the Scandinavian countries. Science and Public Policy,
27(6), 409-419.
The Cochrane Collaboration. (2011). Cochrane Handbook for Systematic Reviews of
Interventions (version 5.1.0). Available from www.cochrane-handbook.org.
Verhagen, A.P., de Vet, H.C.W., de Bie, R.A., Kessels, A.G.H., Boers, M., Bouter, L.M. & Knipschild, P.G. (1998). The Delphi List: A Criteria List for Quality Assessment of Randomized Clinical Trials for Conducting Systematic Reviews Developed by Delphi Consensus. Journal of Clinical Epidemiology, 51(12), 1235-1241.
Van der Vos, J., Borgdorff, H. & van Staa, A. (2007). Kennis in context. Onderzoek aan hogescholen. Retrieved from http://www.scienceguide.nl/pdf/KennisinContext.pdf Visser-Wijnveen, J., van Driel, J.H., van der Rijst, R.M., Verloop, N. and Visser, A. ( 2010).
The ideal research-teaching nexus in the eyes of academics: Building profiles. Higher
Sub-category Criteria Hogeschool University Design of the research
Theoretical Background and
Research Questions
Well-formulated Research questions, sub-questions and
hypotheses
Aware of previous research
Well-defined concepts
Not based on unfounded personal assumptions / biases Methodological and
Technical Set-up
Suitable research methods to study research question(s)
Reliable and valid methods
Discipline-specific methods
Researcher is aware of the method’s pros and cons Student research: feasible planning
Quality of the final product
Style and Form Writing, structure & cohesion
Publishable article
Content Answer to the research question
New insights
Recommendations
A convincing story
Different perspectives on the findings Argumentation &
Substantiation
Conclusions based on results
Arguments supported by right sources
Valid and applicable arguments
Transparency & Accountability
Transparency about steps taken
Replicable and duplicable
Openness about choices made
Transparent argumentation
Clearly visible results
Conduct of the Research
Correct conduct Correct execution of all phases of a study
No manipulation
Thorough conduct No quick and dirty research
No lazy research
Dealing with uncertainties
Pro-active choice making
Value of the research (outcome)
Utility value Applicable or utilizable outcome Valuable for businesses, organizations, society
Valuable for teaching
Valuable for students’ professional careers Scientific value Valuable contribution to scientific field
Setting things in motion (new research)
Researcher-related criteria
Qualities of the researcher
Knowledge of the researcher
Communication and cooperation skills
Intelligence of the researcher
Ability to wield multiple perspectives
Nature of the researcher
Creativity
Positioning of the researcher
Think and act independent
Personal involvement with topic
Positioning towards organization that commissions the
research
Awareness of (funding) stakeholders
The topic of the research and its origin
Topic origin Problems brought up by businesses
Topics brought up by lecturers
Topics of students’ interest
Funding of a study
Topic relevance Social relevance
Scientific relevance
Note. = brought up by most lecturers in this group
= brought up by some lecturers in this group = brought up by no lecturers in this group
