Stage-Gate in Vocational Education: Foundation of a Qualification System?

(1)

Stage-Gate in Vocational Education:

Foundation of a Qualification System?

MSC Technology Management and Operations

Kai Oord

University of Groningen

Faculty Economics and Business

2019

J.C. Kapteynlaan 7

9714 CL Groningen

K.C.M.Oord@student.rug.nl

Student number: s2713047

Supervisor, University of Groningen: Prof. Dr. J. Riezebos

Co-Assessor, University of Groningen: Dr. Ir. D.J. van der Zee

(2)

2

Abstract

In Vocational Education, there is an extensive discussion going on about the way students are assessed. Linking learning processes with student assessment processes (here stated as a part of Qualification Systems in education) has proven to be one of the key challenges in the Vocational Education area. Frameworks that can serve as a foundation of Qualification Systems are not generally developed yet due to the context-specific nature of Vocational Education. In this article, it is

(3)

3

Table of Figures and Tables

Figure 2.1.: Example of a Stage-Gate model………9

Table 3.1.: Overview Stage-Gate aspects and available sources………14

Table 3.2.: Example of Validation Education Stage-Gate………..15

Table 4.1.: Validation of Requirements for QS………..18

Table 4.2.: Completion of S-G aspects in different articles………19

Table 4.3.: Reciprocal Translation………..20

Table 4.4.: Overview of which S-G aspect can be found in what data source………...23

(5)

5

1. Introduction

In Vocational Education, linking learning outcomes with Qualification Systems (QS) has become a crucial part of learning strategies and the assessment of learning outcomes (Bohlinger, 2012). Qualification Systems is an umbrella term for all the processes that lead up to the learning of competences by students (Bohlinger, 2012). In this research, the student-assessment and learning process sides of these Qualification Systems are investigated. There is an extensive discussion going on about the content of Vocational Education and part of it is about student assessment (Buisman and van der Velde, 2017). Researchers argue that schools need to systematically draft, document and evaluate the Qualification Systems in order to guarantee high quality learning (Gulikers, Baartman and Biemans, 2010). This means that practices, guidelines and methods are needed as an underlying foundation of Quality Systems (Bohlinger, 2012; Gulikers, Baartman and Biemans, 2010). One of the key challenges in vocational education is to link learning processes and outcomes with underlying foundation methods for Qualification Systems (Bohlinger, 2012). A Qualification System needs to contain well specified aspects such as the determination of assessors (who evaluate students), decisions possible, student assessment criteria, specified moments of assessment and more. At the moment, many kinds of Qualification Systems exist (Mulder, Weigel and Collins, 2008). However, most of these systems are focused on just assessment criteria and leave out the other aspects (Mulder, Weigel and Collins, 2008). A well-known, general method to include the aspects mentioned at the beginning of this paragraph has not been found yet. Such a method needs to serve as the foundation of the Qualification System.

The fact that there is no general method does not mean that there are no attempts to create

Qualification Systems with underlying methods that a Qualification System is based on. A vocational school in the Netherlands has created a Qualification System wherein different criteria for student competency learning are drafted. However, the school has difficulties in finding a foundation for this Qualification System. The school searches for a general method that can help structure the process of setting up their Qualification System, so they asked the researcher to find such a method.

This article tries to tackle the problem of not having a general, valid method for setting up a Qualification System that covers all important aspects. In the educational context, it is hard to standardize a certain Qualification System (assessment criteria of students are mostly drafted by schools and therefore Qualification Systems are often context specific) (Epstein, 2007). This calls for searching for a method that brings structure to the drafting process. However, this search does not need to be limited to methods developed in an educational context, as other disciplines might have had similar needs with respect to their Quality Systems. In the Technology Management sector, many methods that can set-up Qualification Systems have been created that can be used in several different situations. The sector provides different tools and frameworks that can be used for quality assessment such as Service Quality (SERVQUAL-method) (Parasuraman et al., 1988), Project Selection (Stage-Gate) (Cooper, 2008), Lifecycle and Knowledge Management (Wu et al., 2014) and more. Because of the richness of Quality Frameworks and validation methods it is worth researching whether

Technology Management tools can be applied to an educational context.

(6)

6 Stage-Gate model divides innovation processes of products or projects into a predefined set of stages (separated by certain “gates”), all with certain criteria that have to be met in order to continue to the next step of innovation (Cooper, 1990). A project travels through stages and gates whilst being developed (Cooper, 2008). In addition to that, when carrying out a Stage-Gate process, aspects like go/redo decisions, assessment criteria, deliverables and outputs are all used in order to develop projects. The overall goal of a Stage-Gate process is to bring order in the process of developing projects from idea to launch; it maps out what needs to be done in order to develop a well-matured project. Another goal of the Stage-Gate method is to manage a variety of individual projects while still maintaining a general process of assessing these projects (Cooper, 2008).

The development-process which a project undergoes in a Stage-Gate context might be similar to the process a student goes through in vocational education. A student passes through phases of

development and can only go on to the next phase when he or she meets the levels of competences that are needed to proceed to the next phase of education. Because of this, it is interesting to research whether important aspects of Stage-Gate models can serve as a foundation for a Qualification System that is used to evaluate the students that go through a learning process. Filling in Stage-Gate aspects in an educational setting might just prove to be a good method that structures the drafting process of a Qualification System. Whether the Stage-Gate is actually suited to apply to this educational context is researched in this article.

Several researchers have applied the Stage-Gate method to other contexts: Soenksen and Yazdi (2017) have created a Stage-Gate model in the context of Medical Innovation. In that article, they created a Stage-Gate Model together with professional entrepreneurs who are experts in the medical section. This model was meant to guide entrepreneurs in choosing medical innovation projects that are worth investing in. Instead of focusing the Stage-Gate model on the development of project, they applied it as a helpful tool for investment decisions. Whynacht and Duinker (2015) applied Stage-Gate thinking into the context of Strategic Environmental Assessment. They proposed an eight-step Stage-Gate model in which different plans or policies must pass all stages before a firm will implement it. And lastly, Enevoldsen and Sovacool (2015) applied a Stage-Gate view onto the development of social acceptance of onshore wind projects in France. This article shows that even a social acceptance process can be divided in different stages and gates. From these examples, it is clear that Stage-Gate models already have been used in other context than originally meant for. However, whether Stage-Gate can be applied to an educational context remains a question.

Looking at the general problem of vocational education and the specific problem of the vocational school in the Netherlands, the following research questions were generated:

 How can Stage-Gate aspects be translated to an educational context and serve as a theoretical foundation for Qualification Systems?

 To what extend can Stage-Gate aspects be validated in the Qualification System of a Vocational School in the Netherlands in order to verify if Stage-Gate aspects can serve as a foundation of a Qualification System?

(7)

7 will be created. This is done in order to create a process that brings structure to the drafting process of a Qualification System. A quicker and more efficient way of decision making within this process along with a more complete decision making (not forgetting to fill in certain aspects) will be the result of such a roadmap. If the Stage-Gate aspects are present within the Qualification System, it means that filling in these Stage-Gate aspects can serve as the foundation-method in creating a Qualification System.

The remainder of this article is as follows: in the next section, the theoretical background is written. In section 3, the methodology is described. In section 4, the results of the research will be stated. In chapter 5, the conclusion and recommendations are stated. Lastly, limitations and suggestions for further research are given in chapter 6.

2. Background

In this section of the article, the theoretical background will provide the reader with several definitions and explanations of concepts that are relevant to this research. Firstly, a general explanation about the Stage-Gate model will be given. Alongside this, an overview of important aspects of Stage-Gate models will be given. Secondly, requirements that a Qualification Framework has to comply with are stated.

2.1. Stage-Gate models

In this section, a general explanation of a Stage-Gate model will be given. The general goal of using a Stage-Gate model will be described and important aspects of a Stage-Gate model will be stated here as well. These aspects are: stages, deliverables, gates, assessment criteria, go/redo decisions, fluid and rigid stages, gatekeepers and lastly the projects themselves.

General explanation: Stage-Gate methodologies are recognized and embraced by companies all over the world as a method that brings order to a product innovation process (Grönlund, 2010). The focus of such Stage-Gate methodologies is to improve the effectiveness and efficiency of innovation processes by integrating discipline into the process (Grönlund, 2010). Predefined activities that are credited to a certain stage must ensure that the process of product or project development advances in an orderly and effective manner. A Stage-Gate process exists of different development-stages

separated by certain decision points (gates) (Cooper, 2008). A stage can be seen as a set of activities that is needed to move the product or project along to the next gate (Cooper, 2008). Additionally, a gate is a go/redo decision point. Here, it is decided whether a project has met the criteria to move along to the next stage. If the project has met the criteria, a “go decision” is given. If the project has not met the criteria, a “redo” decision is given (Cooper, 2008). At the gates, three types of information are needed: deliverables (which is the project itself in its current state), criteria (defined in advance, are the minimal requirements a project must have in order to move on to the next stage) and outputs (the go/redo decision) (Grönlund, 2010). At the gate, “gatekeepers” (which are mostly senior managers or project leaders) make the “go or redo” decision (Grönlund, 2010).

(8)

8 preceding one (Cooper, 2008). That is the counterpart of the reduction of risks after each stage. After the activities are carried out, an integrated analysis of the result of these activities is made (Cooper, 2008). When the analysis is done, all analyses-reports are bundled into a deliverable.

Deliverables: When all reports and analyses that were made in a certain stage are bundled, a

deliverable is created. In other words, a deliverable is what the project team brings to the gate (Cooper, 2008). It is a visible end-product of all activities in one stage, which will be under assessment at the gates.

Gate: After all the activities and analyses of a stage are carried out, a gate is the next “huddle” in the process. Gates serve as quality control points where go/redo decisions are made by the gatekeepers (Cooper, 2008). Whether the project team is allowed to further develop the project or if the project is discontinued is decided at these gates.

Assessment criteria: At the gates, projects are evaluated or “judged” based on predefined assessment criteria (Cooper, 2008). The assessment criteria are often divided in must-meet and should-meet criteria (Gronlund, 2010). A method to assess the projects on the criteria is a Cut-off Scorecard (de Souza and Borsato, 2016). Here, projects are scored based on must-meet and should meet criteria. Furthermore, a certain Cut-off Score is stated. When a project scores lower than this Cut-off Score, it is stopped directly (de Souza and Borsato, 2016). Examples of criteria are: strategic fit or expected financial returns (Grönlund, 2010).Must-meet criteria are drafted in order to determine whether a project should be further developed or should be discontinued (Cooper, 2008). If a project is a misfit, must-meet criteria must be drafted in such a way that this misfit project can be quickly disposed. Should-meet criteria are used to prioritize projects (Cooper, 2008). Mostly, a point count system is used to score projects based on the should-meet criteria (Grönlund, 2010).

Output: Outputs are the results of the gate review (Grönlund, 2010). Outputs of the gate contain a go/redo decision and an execution plan (which are mostly made by the gatekeepers) with predefined activities and deliverables for the next stage (Grönlund, 2010).

Go/Redo Decision: At a gate, it is decided whether the project can go on to the next stage (go decision), whether the project must be stopped (stop decision) or whether the project may be further developed but first need some iterations (hold or redo decision) (Cooper, 2008). The decision is based on the must-meet criteria (Cooper, 2008).

Flexible or rigid stages: Many companies have implemented their own Stage-Gate process or method (Sommer, Hedegaard, Dubovska-Popovska and Steger-Jensen, 2015). Stage-Gate processes are adaptable, no activity or stage is mandatory (Cooper, 2008). Examples are that not every stage has to take the same amount of time or that not every activity stated for a stage has to be carried out for every project. Stage-Gate processes are not linear; stages can overlap, projects can be send back to a

(9)

9 Projects: The projects must reach the last stage in order to say that development has been successful. All activities within the stages are drafted in order to develop the projects itself (Cooper, 2008). Projects can take many shapes, from product development to investment decision projects. Below, a general representation of a Stage-Gate model is given. This is just an empty Stage-Gate model, with only the first and last activity defined. This is due to the fact that different versions of Stage-Gate models exist, and no stage (or gate) is set in stone (Cooper, 2008).

As can be seen, from the ideation (in project development context) phase, a project has to pass different stages and gates in order to come to the launch phase. At every stage that is carried out, project uncertainties and risks are reduced (Cooper, 2008). In addition to that, the further a project proceeds through the different stages and gates, the more it is defined and elaborated. A successfully designed project that is deemed as mature enough will come at the “launch” stage and eventually leave the Stage-Gate model (Cooper, 2008).

2.2. Requirements for a Qualification System

In order to be able to create Stage-Gate aspects that can be used in education and serve as a foundation for Qualification Systems, it is important to state the minimal requirements for a Qualification System. Below, requirements a Qualification System has to comply with are stated. This is done in order to create a benchmark that can be used to verify whether the translated Stage-Gate aspects actually can serve as a foundation for a Qualification System. The translated aspects must meet all requirements, will they be theoretically validated as a foundation for a QS.

The aspects stated below are derived from different leading articles on student assessment. The requirements are used by leading international education agencies such as the Association to Advance Collegiate Schools of Business, which is a leading associating for business school. More importantly however, the Dutch Education Inspection (CITO) states that the requirements listed below are crucial for a well-developed Qualification System. In their article “Toetsen op school, middelbaar

beroepsonderwijs”, written by Sanders (2016), requirements for exams, assessors, learning and assessment processes are stated. These requirements are similar to the requirements stated below. Some requirements are even directly copied from articles stated below. As CITO is the leading organization in the Netherlands when it comes to Vocational Education (among others) it can be concluded that the requirements stated below are quite complete when it comes to which aspects of a Qualification System need to be drafted. In the CITO-report, six chapters were written. These chapters were about defining learning goals, content of tests and assignments, assessment methods, assessors, documentation and criteria from which the requirements below were derived.

(10)

10 A definition of learning goals and activities (Stivers, 2009; Booth, 2003): Every Qualification System has to contain defined learning goals in order to describe the desired learnings of students. Students have to work towards learning goals in order to get a degree. In addition to that, the way students can learn must be defined as well. It must be defined which kind of activities students must carry out during the learning process.

Specific Learning goals (Stivers, 2009; Booth, 2003): These stem from the general learning goals. They are the targets for assessment of student learning. It describes particular actions a student has to perform or states particular requirements about a student’s project. Basically, they are assessment criteria for a student’s actions or projects.

Documentation of Assessment criteria (Stivers, 2009): This is a visual curriculum map which people who evaluate the students can use to find the assessment criteria.

Definition of assessment teams or persons (Stivers, 2009): A Qualification System must have a definition of a team or persons who evaluate the students. This means that it must be clear who evaluates certain students.

Coordination of data collection (Stivers, 2009): A certain way of collecting all the data (which means student assignments) must be defined. A point in time must be established to which all projects must be handed in. Furthermore, the way students can hand in their work must be established.

Measurements Methods (Stivers, 2009; (Banta, Lund, Black, & Oblander, 1996; Huba & Freed, 2000; C. A. Palomba & Banta, 1999, 2001; Walvoord, 2004; Booth, 2003): The way different deliverables from students are assessed must be stated as well. For example: are the deliverables assessed

quantitatively or by qualitative opinions of assessors?

Feedback, planning and execution plans (Palomba & Banta, 1999, 2001; Black, 2006): Mechanisms for feedback from assessors to students must be defined as well. This must be done in order to ensure that students learn from the assessment they receive. This means that not only projects must be assessed, but an explanation about the assessment must be stated as well. Planning and execution plans must be defined as well, in order for the students to know what activities must be carried out.

(11)

11

3. Methodology

In this section of the article the methodology of both research questions will be elaborated. The section is divided in the methodology to answer the first question and the methodology to answer the second question.

3.1. Question 1

A literature study will provide the answers to this question. This kind of research design is chosen because a literature study can provide concepts and constructs that we can reuse in this research (Ahlstrom, 2009). We can borrow different methods or frameworks and use them in this research (Ahlstrom, 2009). This is exactly what this question is about; translating (or applying) Stage-Gate models to another context. There are multiple ways of conducting a literature study. In the next paragraph, the specific literature study that is used in this research is described.

3.1.1. Qualitative Meta-Synthesis

To answer the first question a Qualitative Meta-Synthesis will be carried out. This kind of research is suited to build a common frame of reference (Jensen, 1996). The goal of a Qualitative Meta-Synthesis is to develop a new framework (in this case; aspects of Stage-Gate in an educational context) through carrying out a process of translation and synthesis (Jensen, 1996). In this research, it means that different articles about Stage-Gate in education are researched and compared in order to combine and translate them into a new educational Stage-Gate Framework. In the framework, the main aspects of Stage-Gate are put into an educational context. Below, the different steps that have to be carried out according to the Qualitative Meta Synthesis-method (Jensen, 1996) are described:

Locating relevant studies and deciding what to include (Jensen, 1996): The articles will be collected through the use of different search engines. The four that are used most commonly in this research are “Business Source Premier”, “Google Scholar”, “Web of Science” and “ERIC”. Through the use of key words, articles will be searched that have to do with applying Stage-Gate models to educational contexts. Keywords that will be used are “Stage-Gate models”, “Education”, “Vocational”,

“Assessment”, “Qualification System” and “Validation”. Via a screening procedure, the number of articles that comes up when using the stated search words will be reduced in order to only keep articles that are relevant. Criterion for articles to be included in the research is that the Stage-Gate model has to be applied to education in some way. For example, a course at a university is taught to the students by applying different stages and gates. When it is clear in the abstract that this is the case, the articles are taken along in the research.

Compare and contrast, determining what “Stage-Gate” contexts are related or different (Jensen, 1996): In the articles that remain, the contexts in which the Stage-Gate processes are used are compared. This means that all the important aspects of Stage-Gate that are stated in the theoretical background are compared in the different articles. After that, an overview of the completion of every aspect in every article is given. This will be done by creating a table in which the usage of Stage-Gate aspects in the different articles is stated.

(12)

12 Syntheses of Translation (Jensen, 1996): In this step of the translation, all the translated aspects of all the Stage-Gate context will be compared and clustered together, so that a new refined theory (in this case, aspects of Stage-Gate) will occur. This will eventually result in a table in which all the aspects of the newly formed educational Stage-Gate context are stated. A new Stage-Gate context is created. This is done by listing all the important (translated) aspects of the Stage-Gate model and giving the final definitions of the aspects.

Last step: From the translation, educational Stage-Gate aspects arise with clear definitions. However, in order to verify whether this translated model can be used as a foundation for a Qualification System, the model has to be compared with the requirements for a QS that were stated in the theoretical

background. In order to be verified as a foundation for Qualification System, the translated Stage-Gate aspects need to contain all requirements stated in section 2.2. A self-generated checklist will be created in which criteria are stated that contain specific guidelines. These guidelines are used to decide

whether requirements are present in the definitions or not. This checklist can be found in appendix A. The requirements of a Qualification System and the Stage-Gate aspects will be depicted in a binary matrix. This kind of matrix is used to indicate whether a translated Stage-Gate aspect contains a Qualification System requirement (depicted with the number “1”) or whether it does not contain this (depicted with the number “0”). Then, the numbers in the Qualification System rows are added up. Because the translated Stage-Gate aspects can only be verified as a foundation for a Qualification System when all the requirements for QS are present, the total added up number at the end of the Qualification System rows cannot be zero (which would indicate a requirement is not present in the newly drafted context). This kind of matrix is used because it can easily indicate whether a

relationship exist between two variables (Namey et al., 2008). In this context, it means that it can easily conclude whether al QS requirements are present by just looking at the total numbers.

3.2. Question 2

In this part of the research, the methodology of research question 2 will be explained. First, come context about the current Qualification System of the vocational school is given. After this is done, the section is divided in subsections: research design, data collection, measurement/instruments and data analysis.

3.2.1. Qualification System Vocational School in the Netherlands

The vocational school in the Netherlands has created a Qualification System with different

competencies and criteria drafted in relation to these competencies. They use this Qualification System to monitor and evaluate the learning process of their students. However, the problem that this

vocational school faces is that the drafting process of this Qualification System is unstructured. At the moment, the unstructured way of setting up the QS causes ineffective decision making, which takes up more time than necessary. It also causes that some tasks that must be carried out in order to create a QS that meets the requirements by the Dutch Educational Board are omitted. The school is searching for a method to get a structured drafting process for the Qualification System, as well as documenting, validating and finding a foundation for it.

(13)

13 Communication is the next competency, where students are evaluated based on their presentation skills, knowledge of different communication means and asking for feedback. The next competency is Craftsmanship. Students are evaluated based on skills they must have acquired within their field of work. The last competency is awareness of context. Here, students are evaluated on aspects like working towards a goal with structure, being perceptive to feedback or suggestion and presenting their work in a way that other people understand what the project or work was about.

The vocational school divides the learning process into different phases: the learning process starts in phase 0, when a student applies for an education at the school. Then, through phases one to three, students have to acquire skills and knowledge in the field of one of the five competences. Eventually, they all have to be at a certain level (in all five competences) to be able to go to phase four. In phase four, an end-project has to be made in order to show whether the student mastered the right level of competency in order to graduate.

3.2.2. Research Design

In order to answer the second research question, a case study will be executed at the vocational school in Zwolle. The translated aspects that were created in research question one must be validated not only in theory, but also in practice. The original Stage-Gate model is translated into an educational context, with different aspects that are specially drafted for this context. These different aspects must be validated in order to come to know whether the Stage-Gate aspects can be deemed as legit aspects that can serve as foundation of the Qualification System. In order to do this, assessment processes (QS-forms) of students at the vocational school in Zwolle are compared with the newly drafted aspects of the educational Stage-Gate Framework. For example, it is checked whether during the assessment and learning process of a student, gatekeepers are used, or can be used for assessment at the school. When all the aspects are validated, it means that Stage-Gate aspects can serve as a foundation for

Qualification System instead of just drafting this out of the blue like it is done now. The final result will then be a roadmap with different steps that describe what activities need to be carried out in order to draft the Qualification System. This roadmap must bring order in the drafting process and provide a way to have a more complete Qualification System, which means that no steps are omitted in the drafting process.

Documents (in which the Qualification System and student assessment process are stated) of 50 students of the school will be evaluated and compared with the newly drafted educational Stage-Gate aspects in order to come to know whether this new Stage-Gate model can be used with student assessment. For every student assessment process in the case study, it is checked whether the

educational aspects of stages, deliverables, gates, assessment criteria, go/redo decisions, fluid and rigid stages, gatekeepers and the projects itself can be found.

There are several data sources that are used, these are listed below:

1. An Excel-File in which all the grades of the students are stated, accompanied with codes that correspond to student assignments and activities.

2. The exam-regulation form. 3. Course manuals.

4. Feedback forms, written by teachers and communicated to students. These are about learning outcomes of a student.

5. Written correspondence with members from the vocational school.

(14)

Excel-14 File correspond to the different student assignments that are stated in the course manuals. This is the reason that one aspect can be found in multiple sources. Note that these data sources are available upon request. This has to do with confidentiality reasons.

Below, an overview of the aspects and available resources is given.

Aspects Available sources

Stage Assessment Forms

Deliverables Exam Regulation Document

Gate Course Manuals

Assessment Criteria Feedback Forms

Output Correspondence with school

Go/Redo Decisions Flexible or Rigid Gatekeepers Projects

Table 3.1.: Overview Stage-Gate Aspects and Available Sources

3.2.3. Data Collection

The population of this case study consists of the assessment forms students from the vocational school in Zwolle, along with the other data sources. As the school offers different kinds of educational training, different studies within the school are taken along in this research. The school offers seven kinds of studies, from which the five biggest studies (based on student numbers) are included in the study. The remaining two studies are not taken along in the study because of the fact that they contain a limited number of students. Furthermore, the data available from these studies is too limited. The fact that different studies are taken along in th is research is because the researcher wants to know whether the assessment procedures of the studies differ when it comes to the data that is available (grades, moment of assessing, assessment criteria, ways of documenting etc.). Because of the fact that vocational schools offer job specific studies, the study assignments and way of assessing might differ per study. By taking multiple studies into account, the researcher wants to generate conclusions that are representable for the entire school. From each study, the assessment forms of ten students are researched. This was decided based on several meetings with the representors of the vocational school; they assured that the learning and assessment processes of students within the same study was the same. The researcher wanted equal representation of the studies within the sample size (because the process that will be drafted must work for every study) and as the data of roughly 300 students was available, it was decided that 10 assessment processes of 5 studies were taken along. This means that 50 assessment processes are researched. In addition to that, the vocational school stated that

researching more than 50 assessment processes would be unnecessary. This is because of the fact that all students are assessed in the same way. When differences in the assessment processes are found, it’s most likely caused by missing data. Because of the statements, the researcher decided that 50

assessment forms would give a representative view of the entire population.

The data used in the study will be secondary data, as the student assessment forms are already

collected by people from the vocational school. Within these forms, it will be evaluated whether there are aspects of the newly formed educational Stage-Gate context present, be it in other words or forms.

3.2.4. Instruments/Measurement

(15)

15 of the aspects of the educational Stage-Gate model, a binary matrix in which all the aspects and forms are stated will be generated (Namey et al., 2008). In this matrix is indicated whether an aspect was present in the form. Eventually, a matrix like the one beneath will be generated:

Aspect/Form 1 2 3 4 5 .. 50 Total

Stages 1 1 1 1 1 .. 1

Gates 0 1 0 0 1 .. 1

Gatekeeper 0 0 0 0 1 .. 1

Deliverables 1 1 1 1 1 .. 1

Table 3.2: Example of Validation Education Stage-Gate

Per aspect, it is indicated whether it was present in the form. Measurement will be indicated in the theoretical framework in order to be able to assign a negative (0, absent) or positive (1, present) value to an aspect. For example, a guideline for “stages” can be: “check 1 when the assessment of students is divided in years or semesters, check 0 when not”.

3.2.5. Data Analysis

After the collection of all the data is completed, it is checked whether the Stage-Gate aspects can be validated as foundations of a Qualification System. This is done by looking at the total (added up) number at the end of the Stage-Gate aspect-rows. Because of the fact that this kind of research has no well-known preceding researches (applying Stage-Gate aspects to Qualification Systems), there are no previously stated acceptance criteria. This means that the researcher can state acceptance rates

(Wielders, 2016).

It is decided that a Stage-Gate aspect is validated as usable in a QS when in 100 percent of the forms, this Stage-Gate aspect is found. When all numbers are added up, it can be concluded which aspects are usable as foundation for a QS. It was also decided that only when all aspects are validated, the Stage-Gate aspects can be adopted. This is because all “ingredients” of Stage-Stage-Gate must be present in order to draft the QS according to the Stage-Gate aspects in the future. When an aspect cannot be found in one of the data sources, further research will be the deciding factor as to which an aspect is present in the current QS of the vocational school. This further research will take place in the shape of

(16)

16

4. Results

In this section of the report the results of the first and second research question will be described. First, the Qualitative Meta-Synthesis will be described (including validation of the model). Second, the validation of the translated Stage-Gate aspects in the Qualification System of the vocational school will be elaborated. Lastly, some remarks about the current Qualification System of the school are given. These remarks are about “shortcomings” the current QS has in contrast to a QS that is drafted according to the Stage-Gate principles.

4.1. Question 1: Qualitative Meta Synthesis

Step 1: Locating relevant studies and deciding what to include

After a recursive and iterative literature search eventually six articles were included in the research. The first step was to use the search word “stage-gate” and 361 articles were found. The second step was to add “education” to the search and 18 articles were found. However, not all the articles were suited to be included in the research because in some articles the Stage-Gate model was only mentioned as an example framework, but not actually applied to education . After several search attempts with varying (additional) terms like “qualification system”, “assessment” and “validation”, six articles remained that were taken along in the research. These articles all applied Stage-Gate models to an educational context. However, the application and completion of these Stage-Gate models were slightly different in every article. The articles of Domschke and Blaho (2017),

Reifschneider et al. (2015), Farris and Lane (2006), Huet et al. (2008), Yazdi and Archaya (2013) and Dean and van Bossuyt (2014) were deemed as usable in this research.

Step 2: Compare and contrast determining what “Stage-Gate” contexts are related or different

In step 2, a table was created in order to give an overview of the completion of all the Stage-Gate aspects in the six articles. The results of this step can be seen in table 4.2 on page 19.

Step 3: Reciprocal Translation

In this step, all the terms that were used in the different articles are translated to similar, overarching terms. The generation of these overarching terms was an iterative process. Sometimes the translation was straightforward when the terms used in all the articles were the same and sometimes the writer had to adjust the terms a little. At the same time, it was checked which terms are used the most across articles. The term that is most used is called the overarching term. The terms used in the articles were labeled as “overarching term” (indicated by a green color) when it was obvious that the term was used widely across most of the articles. Furthermore, a term could be labeled “similar to overarching term” (indicated by the color yellow) when a term had similar aspects to the overarching term, but was just slightly different. In this case, the writer decided whether the components of this term would be used in the synthesis of translation. Lastly, a term could be labeled as “different to overarching term” when a term was completely different (indicated by a red color). When this happened, the term would not be taken along in the synthesis of translation. The results of this step are stated in table 4.3 on page 20.

Step 4: Synthesis of Translation

(17)

17 Stage: A Stage is represented by a semester or academic year, in which specific “stage activities” or student assignments have to be carried out in order to reach specific learning goals. These stages are separated by gates.

Deliverables: The deliverables are represented by all bundled student assignments that have to be completed at the end of a stage. This includes written documents, project outcomes and oral

presentations. The bundled student assignments must be handed in at the “gate”, at a predefined date. Gates: A gate is a predefined meeting at the end of a semester or academic year in which the students and student assignments are reviewed, based on the learning goals and additional measurement methods (that are relevant when assessing students or student assignments). The way of reviewing (measurement methods) and the learning goals must be stated in Qualification System documents. Assessment Criteria: Assessment Criteria are predefined by the faculty and concern criteria about the student assignments and advancement of student learning, as well as methods to measure these criteria. Again, these are stated in Qualification System documents and must be determined prior to the beginning of the educational “Stage-Gate Model” or in this case, a semester or academic year. Output: The output of gate reviews consist of both feedback (about the student assignments and student development) and the definition of all activities that have to be carried out by the students in the next stage. Both the feedback and the plan for the next semester have to be communicated to the students prior to the next stage.

Go/Redo Decision: The go or redo decision concern whether a student may advance to the next stage. It is based on the results of the student assignments and advancement in student development. The decision must be taken by a gatekeeper. In addition to this, the decision can state that the student can go on to the next stage, has to redo the previous stage or has to be removed from the study due to bad results.

Flexible or rigid: All stages are rigid, with predefined begin and end dates. The stages cannot overlap. Gatekeepers: Gatekeepers are faculty members (i.e. teachers) who are qualified to assess the

(assignment of) students and/or any extern project partner that is involved in the development of a student or student project.

Projects: Projects are all the student assignments that must be carried out by the students, along with a definition of all activities that have to be carried out by the students when executing the assignments. Criteria with measurement methods at which the projects and assignments are assessed must be documented as well.

Together, these translated Stage-Gate aspects make up the newly drafted Stage-Gate context that might be suitable to use when developing a Qualification System.

Validation of new Stage-Gate aspects

Below, the results of the validation process are described.

(18)

18 The “specific learning goals” requirement is found in the “gate”, “assessment criteria” and “projects” aspects. These are the targets of student learning, the student assignments itself and assessment criteria to which the students are assessed. “Assessment Criteria” are a part of the “gates” and contain the specific criteria and learning goals. The projects itself are the student assignments. Because these three aspects contain the “definition of specific learning goals”, a “1” can be put in the binary matrix. The third requirement (documentation of assessment criteria) is found in the definition of “assessment criteria”, which clearly states that assessment criteria must be described in the course manuals and Qualification System documents. In other words, it means that assessment criteria need to be documented.

The “definition-of-assessment-teams-or-persons”-requirement can be discovered in the “gatekeepers” aspect, which states that faculty members that are authorized to assess the students need to be

appointed to assess certain students and student assignments. This was the criteria that needed to be met in order to put a “1” in the binary matrix.

The fifth requirement “coordination of data collection” is found in the definitions of “deliverables” and “gates”. The “deliverables” aspect states the student assignments need to be bundled in some sort of way. Furthermore, the “gate” aspect specifies when all the data need to be collected.

The definition of “measurement” methods can be traced in the “gate” and “assessment criteria” aspects, which both state that measurement methods need to be specified when drafting the QS. The last requirement (feedback, planning and execution plans) is found in both the “output” and “go/redo” decision aspects. The “output” aspect states that students need to get feedback and activities for the next semester. The “go/redo” decision also states that feedback needs to be communicated to the students.

The results of this validation step are depicted in table 4.1. As can be seen in the table, all

requirements for a Qualification System are present in the Gate aspects. This means that Stage-Gate aspects can indeed be translated to an educational context and theoretically serve as the

foundation of a Qualification System.

QS demands S-G aspects

Stage Deliverables Gate Assessment Criteria Output Go/Redo decision Flexible/Rigid Gatekeepers Projects Total Valid

Definition of learning goals and activities 1 1 1 0 0 0 0 0 1 4 Yes

Specific Learning Goals 0 0 1 1 0 0 0 0 1 3 Yes

Documentation of Assessment Criteria 0 0 0 1 0 0 0 0 0 1 Yes

Definition of Assessment Teams or Persons 0 0 0 0 0 0 0 1 0 1 Yes

Coordination of Data Collection 0 0 1 0 0 0 0 0 0 1 Yes

Measurement Methods 0 0 1 1 0 0 0 0 0 2 Yes

Feedback, Planning and execution plans 0 0 0 0 1 1 0 0 0 2 Yes

(19)

Aspect Article Domschke and Blaho (2017) Reifschneider et al. (2012) Farris and Lane (2006) Stage Semesters with predefined activities and/or

assignments

Product Development Stages, divided over the course of ten weeks (one semester)

Stages are divided over the duration of one course with predefined student activities

Deliverables Bundled Student Assignments Outcomes Bundled outcomes of execution student product development assignments

Bundled outcomes of predefined product development assignments

Gates Student review meeting at end of Semester. Review based on documented criteria and measurement methods.

Not explicitly stated (mostly review deliverables at the end of product stage)

At the end of a Stage, all student assignments are reviewed in the gate based on documented criteria and measurement methods

Assessment Criteria

Review Student Assignment based required level of knowledge gaining along with methods to measure this.

Criteria stated by course professors (different criteria per stage), along with documented assessment methods

Predefined Project Development criteria (different per stage) along with documented measurement of criteria

Output Guidelines for student activities next semester

Feedback on projects and guidelines activities next period

Not mentioned

Go/Redo Decision

Moving Project to a next semester Moving project to next development stage Moving project to next development stage

Flexible or Rigid

Rigid (Semesters) Rigid (predefined time-length of stages over the course of one semester)

Flexible (Stage ends as soon all the assignments are carried out properly)

Gatekeepers College Faculty and industry partners Instructors and professors of the course A team of faculty members

Projects Student Assignments and Student

Development along with predefined activities and assessment methods

Predefined Product Development activities (assignments) students have to carry out plus assessment method

Predefined Product Development activities (assignments) students have to carry out

Aspect Article Huet et al. (2008) Yazdi and Archaya (2013) Dean and van Bossuyt (2014)

Stage Years Semesters with predefined student

activities and assignments

Humanitarian Engineering Project Development phases divided over two semesters

Deliverables Bundled Outcomes of Student Engineering Assignments

Bundled Outcomes (Summaries) of Medical Student Project Assignments (Predefined per Semester)

Bundled Social, Technical, Financial and Sustainability Assignments

Gates End-of-Year meeting + Student-Teacher meetings (students assessment based on documented assessment methods)

Predefined Meeting by CBID at the end of a Semester where student (projects) are reviewed based on documented assessment criteria and methods

Predefined Gate Review and Presentations (every four weeks) with documented measurement methods and criteria

Assessment Criteria

Predefined Criteria regarding Personal, Product skills and Disciplinary Knowledge + Assessment Logbook + documented project assessment method

Predefined Assessment Criteria (different per semester) drafted by the CBID faculty plus assessment methods

Predefined Assessment Criteria (Group and Individual) plus assessment method

Output Feedback from students and teachers plus required activities next year

Not explicitely mentioned Feedback to students, next assignment (stated in course description) and go/no go

Go/Redo Decision

Decision regarding advancement student in learning process

Moving Project to a next semester Moving Project to next phase

Flexible or Rigid

Rigid (Years) Rigid (Semesters) Flexible

Gatekeepers Teachers (and students when peer reviewing) CBID faculty, mentors and partners Faculty Members and NGO partners

Projects Predefined Student Assignments + Project Development Assignment + Logbook + assessment method of project

Medical Innovation Project Assignments carried out by students plus project assessment method

Student Assignments for Project Development (Individual and group) plus project assessment method

(20)

20

Article Domschke and Blaho (2017) Reifschneider et al. (2012) Aspect

Stage Semester with predefined activities Semester with predefined activities

Deliverables Bundled Student Assignments Bundled Student Assignments

Gates Student Review at end of Semester with documented assessment method Student Assignment Review with documented assessment criteria and method Assessment Criteria Predefined Assessment Criteria stated by Faculty and documented assessment method Predefined Assessment Criteria stated by Faculty and documented assessment method

Output Guidelines Guidelines and Feedback

Go/Redo Decision Moving Student to next Semester Moving Student and project to next stage

Flexible or Rigid Rigid Rigid

Gatekeepers Teachers, Faculty Members and (Project) Partners Teachers, Faculty Members and (Project) Partners

Projects Student Assignments with assessment method Student Assignments with assessment method

Article Farris and Lane (2006) Huet et al. (2008) Aspect

Stage Courses Years with predefined activities

Gates Student Assignment Review with documented assessment criteria and method Student Assignment Review with documented assessment criteria and method Assessment Criteria Predefined Assessment Criteria stated by Faculty and documented assessment method Predefined Assessment Criteria stated by Faculty and documented assessment method

Output - Feedback and Guidelines

Go/Redo Decision Moving Student and project to next stage Moving Student to next semester

Flexible or Rigid Flexible Rigid

Article Yazdi and Archaya (2013) Dean and van Bossuyt (2014) Aspect

Stage Semester with predefined activities Semester with predefined activities

Gates Student Assignment Review with documented assessment criteria and method Student Assignment Review with documented assessment criteria and method Assessment Criteria Predefined Assessment Criteria stated by Faculty and documented assessment method Predefined Assessment Criteria stated by Faculty and documented assessment method

Output - Feedback and Guidelines

Go/Redo Decision Moving Student and Project to next stage Moving Student and Project to next stage

Flexible or Rigid Rigid Flexible

Overarching term

Similar Definition to overarching term

Different Definition to overarching term

(21)

4.2. Question 2: Practical Validation

In this part of the report, the practical validation of the Stage-Gate aspects within the QS of the vocational school will be described. For every aspect, it will be described whether it is validated. Furthermore, the process that leads up to the conclusion for these aspects is described as well.

Validation of Stage-Gate aspects

The process of validating the Stage-Gate aspects starts with drafting a checklist in which criteria for granting a positive value (1) or a negative value (0) to a Stage-Gate aspect (per student). This checklist can be found in appendix B. Next, it was checked (according to the checklist) whether a Stage-Gate aspect was present in the learning process of a certain student (and therefore in the QS itself). This generated a binary matrix which can be found in appendix C. Below; the results of the validation process of the Stage-Gate aspects are described:

Stage: In the QS of the vocational school, this aspect can be found back in all instances that were researched. That means; out of the 50 assessment and learning processes that were under investigation, 50 processes contained stages. While researching this aspect, it was checked whether a clear phasing of the learning and assessment process could be distinguished. In all 50 instances, it could be

concluded that the learning and assessment processes were divided into semester-phases and academic years. In the assessment forms, this can be seen by the description of the aspects on which the students are constantly graded. This description often contained an explanation of the kind assignment that was graded and an indication when this assignment was carried out (in which semester of which year). General grade-averages of the different competences (and their corresponding assignments) were often entered onto the assessment forms every semester. In addition to that, the average of all assignments together was often displayed at the end of the grade-list, together with a decision about whether the student could proceed to the next year of the study. This all indicates that the learning process is divided in different phases (or stages) and therefore this aspect is validated.

Deliverables: In order to validate this aspect, it was checked whether the assignments were bundled together in one deliverable (stated in a course manual) and whether for the competences a weighted average could be found in the student assessment form. This weighted average had to relate to the deliverable. In all 50 instances, this was the case. For all five studies, examples of deliverables could be found in the course manual. For example: in “craftsmanship” a student has to make a planning for crafting a certain product, make the concept, make the product and then deliver it as a whole at the end of the semester. Because of the fact that for all 50 instances, deliverables were found, it can be

concluded that this aspect is validated.

(22)

22 Assessment Criteria: The validation of this aspect was done by looking at both the assessment forms of the students and the course manuals. In the course manuals, assessment criteria are stated regarding the five different competences, the assignments that the students have to carry out and a grading (or measurement) method. These assessment criteria have certain codes that were also used in the assessment forms of the students. In addition to this, results of the students with predefined measurement methods had to be present as well. When all these aspects are present in the different documents, it means that the Qualification System of this school makes use of predefined assessment criteria. In all instances, the codes in the assessment forms corresponded to the ones in the course manual. In addition to that, the results with predefined measurement methods were also present. This means that this aspect is validated as present in the Qualification System. In addition to that, course manuals (in which the codes are stated) were made available prior to the academic year, which means that assessment criteria (and assignments) are communicated to the students prior to the academic year, which is stated in the “assessment criteria” definition.

Output: In order to check whether the right output was given at the gates, it was checked whether in the assessment processes final decisions about the advancement of the student in the learning process were given. In addition to the decisions, it was also checked whether the codes of the courses in the assessment forms corresponded to the codes stated in the course manual. This is because that would mean that the courses were communicated to the students prior to the semester and/or academic years. In other words, this means that feedforward, or a plan for the next semester is communicated to the students. In addition to that, the presence of feedback forms was also checked. In all instances, a final decision was found in the assessment forms and the codes were the same as stated in the course manuals. This means that this aspect is present in the current QS of the school.

Go/Redo decision: The process of validating this aspect was all about looking whether the final decisions about advancement of the students in the learning processes were present on the assessment forms. In all 50 instances this was the case. However, not all the decisions were digitalised, but filled in manually on paper forms. An extra explanation of the vocational school stated that there is indeed a yearly decision about all students. There is also a meeting at the end of each semester where a

temporary decision is given, however only the end-of-the-year-decision is binding for the student. This indicates that go or redo decision are indeed present in the system. They are represented as 1 (go to the next year), 2 (go, but under supervision), 3 (go, but think about other options) and 4 (redo year or go to another study).

Flexible or Rigid: In order to validate this aspect, the conclusion must be that this Qualification System has rigid stages, as can be seen in the definition of the educational stage-gate aspects. This validation process is carried out by looking at whether the activities of one semester do not overlap with the activities of the next semester. This would indicate that a “stage” stops at a predefined time and that there is no overlap between the stages. In all instances, this was the case. This means that it can be concluded that this aspect is present in the QS and therefore validated.

(23)

23 matrix for every student. This means that every grade is entered onto the assessment forms by a gatekeeper and as every student that was researched received grades, this aspect is validated.

Project: The last aspect that must be validated is the projects. This aspect can be validated by checking whether the assignments that were graded in the assessment forms are the same as the ones stated in the course manuals. Furthermore, the assignment must be graded by a predefined method. These two criteria are reviewed because they would indicate that this QS makes use of predefined student

activities and assignment, along with predefined measurement methods, which is a requirement for the newly drafted Stage-Gate aspect. In all instances, these criteria were found, which leads to the

conclusion that this aspect is validated.

In conclusion, all nine Stage-Gate aspects can be traced in the Qualification System of the vocational school. The implication of this conclusion, together with a short analysis and different

recommendations addressed to the vocational school are to be found in the next section. Below, a table that described which aspect was validated by searching in what source is given.

Aspect Source

Stage Assessment Forms

Deliverables Assessment Forms, Course Manuals

Gate Assessment Forms, Course Manuals, Exam

Regulations

Assessment Criteria Assessment Forms, Course Manuals

Output Assessment Forms, Course Manuals, Feedback

Forms

Go/Redo decision Assessment Forms, Exam Regulations,

Correspondence with School

Flexible or Rigid Assessment Forms, Course Manuals

Gatekeepers Assessment Forms, Exam Regulations,

Correspondence with School

Projects Assessment Forms, Course Manuals

Table 4.4.: Overview of which S-G aspect can be found in what data source

4.3. Remarks in relation to current Qualification System

It was concluded that all Stage-Gate aspects are present within the Qualification System of the vocational school and therefore there is an opportunity to use these aspects when drafting the Qualification System with its learning and assessment processes. However, this is not the case at the moment. Despite of the fact that all aspects are present, they are not used to draft the Qualification System. In this part of the report, some remarks about the current interpretation of the Stage-Gate aspects and statements about how to use them right are given.

(24)

24 Definition of activities to be carried out in the next stage: The output of a gate review exists of both feedback about the learning advancement of a student and a definition of activities that have to be carried out by the student in the next stage. However, the activities that have to be carried out are mostly drafted prior to the whole learning process and stated in a course manual without explicitly being communicated to the students after a gate review. In order to cope with the requirements of the Stage-Gate aspects better, activities need to be drafted (or redrafted) after every gate review and then be communicated to the students. The activities cannot just be drafted prior to the academic year and then put in a course manual which the students have to look up themselves. Because of the fact that the activities are communicated to the students in some sort of way, a “1” is filled in for every student in the binary matrix. However, the activities can be communicated to the students more actively (i.e. email). Actively communicating the activities for the next semester is important (in Stage-Gate) because that means that students will know what activities need to be carried out and which learning goals they have to reach in a timely manner. This brings more structure into the learning and

assessment processes.

(25)

25

5. Conclusions and Recommendations

In this part of the report, the conclusions drawn based on the results of the two main questions are given. After that, recommendations to the school are stated in order to show what improvements can be made if the school wants to draft its Qualification System based on the Stage-Gate aspects.

5.1. Conclusions

5.1.1. Question 1

The results of the first main question show that the Stage-Gate aspect can indeed be translated to an educational context. The analysis of the different articles shows the way the different aspects are translated from a project development context into an educational setting. The final definitions of these aspects show how the Stage-Gate aspects must be interpreted in the educational context. This part of the process showed the end-result of the translation-method to an educational context, which answered the first part of the first question. The Stage-Gate aspects can be translated to an educational context by a Qualitative Meta-Analysis. By clustering the completion of the Stage-Gate aspects in the different articles, general definitions have been drafted. For some aspects, an overarching term could be easily drafted; which was the case with deliverables, gatekeepers, assessment criteria, go/redo decisions and projects. In all articles, the completion of these aspects was the same. For other aspects, like stages, gates, output and flexible or rigid semesters, the researcher had to decide on the

overarching term himself. The most widely used completion of the aspect became the overarching term in that case.

The last step of this analysis was to check whether the translated aspects could theoretically serve as a foundation of a Qualification System. All the requirements a Qualification System has to comply with are present in the definitions of the aspects. This means that the aspects can be used as a foundation for a Qualification System, and that filling in the aspects will create a structured drafting process. The Qualification System can be drafted according to the Stage-Gate principles. By determining how long the semesters (stages), the drafting process is started. Stating when the students have to hand in their assignment, in what form (deliverables) and when they are assessed (gates) continues the process. By assigning different assessment criteria to the student assignment (projects), the “empty” stage-gate model is filled (semesters get content and important dates of student-teacher meetings). The

determination of feedback (way of communicating and what kind of), go/redo decisions and who will be assessing the students (gatekeepers) makes the whole Stage-Gate context complete. The learning and assessment processes of the students can now be seen as a Stage-Gate process, because all the components are there to draft it according to the Stage-Gate principles.

The new Stage-Gate aspects comply with the requirements for a QS and filling in the aspects brings structure into the drafting process. From these results, it can be concluded that in theory, Stage-Gate aspects serve as a foundation for a Qualification System.

5.1.2. Question 2

(26)

26 answer to the question whether the Stage-Gate aspects are present and can practically serve as a foundation of a QS. However, there will be given some recommendations to the vocational school in order to set an example of how the Stage-Gate aspects can be used.

The results of this analysis are that all Stage-Gate aspects are present in the current Qualification System of the vocational school. However, the fact that they are all present does not mean that the aspects are systematically used for the drafting process of the Qualification System. It merely means that the aspects are there, and can in the future be used to set-up the QS. The results do show that the Stage-Gate aspects are also present in a real-life Qualification System and are therefore used as a method to draft the Qualification System.

The vocational school already divides the learning processes into different stages. Learning processes are divided into different study years. The study years are divided into different semesters, which shows that a clear phasing of the learning process (as defined in the Qualification System) does exist, which is one of the characteristics of a Stage-Gate process. The vocational school also makes use of certain deliverables. At the end of a stage, all assignments that the students have been carrying out that semester are bundled together. The assignments are reviewed and eventually an average grade is given, which essentially decides whether a student may proceed to the next study year, or in this case: stage. Gates are also present in this Qualification System. At the end of each semester and academic year, the deliverables of the students are reviewed in a meeting with all the teachers involved in the development of certain students. The grades of the students are checked against predefined criteria and a decision about the advancement of the students in the learning process is made. These two

components (review of assignment and decision about students) are exactly what is normally done in a gate review. The output of the teacher meetings does comply with the requirements set in the

Stage-Gate in Vocational Education: Foundation of a Qualification System?