The assessment process and effectiveness of hybrid learning
Chapter 6 | 107 Figure 20: Model of hybrid learning environments
There is a strong relationship between learning and assessment (Baartman,
Bastiaens, Kirschner & Van der Vleuten, 2006). Therefore, we studied the assessment process of hybrid learning environments in higher education. We determined the quality of the assessment process and consequently determined the effectiveness of hybrid learning environments. Two research questions were addressed in this article:
§ How can the assessment process of a hybrid learning environment be characterised?
§ How effective is a hybrid learning environment?
6.2 Method
The method we chose to answer the research questions is the case study. The strength of the case study method is its ability to examine, in-depth, a “case” within its “real-life” context (Yin, 2005).
6.2.1 Selected case study
The studied educational context was situated in a large Dutch University of Applied Sciences. The institute consists of six independent faculties. The research presented here took place at the faculty of Communication and Journalism, at the department of Digital Communication. The studied learning environments were two subsequent study years of the final, obligatory course for first year students with as topic 'system development'. During an eight-week course students worked in small project teams (three to four students) on the design and development of a website for an external client. In 2006, 125 learners, 12 educators and six external clients participated; in 2007, 170 learners, 14 educators and seven external clients. The external clients were of the small and medium sized business domain or of the non-profit sector. The learners were divided into sub-groups, one sub-group for each client (six in 2006;
seven in 2007). In each sub-group, about eight project-teams worked in parallel for the same external client.
Reality Simulation
Acquisition
Participation
Learning Task Learning
Task
Learning Task
Hybridlearningenvironments:
Sequence of related learning tasks positioned in different quadrants
Learning Task
Chapter 6 | 108
The competencies, defined above as an integration of knowledge, skills and attitudes (Van Merriënboer et al., 2002) were described in the educational material as
followed.
§ You will design and develop a dynamic website for an external client.
§ In your future profession you will be confronted with the core tasks of advising, designing and developing. These three core tasks will be dealt with in a specific setting.
§ It was explicitly chosen to develop a website, since there is a strong chance that you will participate in projects to design and develop websites in your future professional practice.
The above case was selected for a number of reasons. The first reason was that the educational institute and the context in question explicitly had competence-based learning as underlying educational rational. The selected context represented a good example of a learning environment based on competence-based learning. It was designed fifteen years ago and has evolved and improved since. Lastly, the learning environment involved a large number of learners and was divided into several sub-groups. Such a large-scale environment would help to gather relatively much data from one case. Two subsequent study-years were studied. The learning environment and the assessment process remained the same in both years and were considered to be a single case. The learners differed in the subsequent years, data from the learner experiences and the learner outcomes of both years were studied.
6.2.2 Data collection
Educational material. To characterise the learning environment and determine the quality of the assessment process, the educational material was collected. All documents and material specifying the learning environment were collected, both paper and digital material. The documents and material consisted of student manuals, hand-outs, obligatory resources (e.g. books, links to online material), formats, examples, announcements in the accompanying online learning environment and so on.
Questionnaire: experiences and learning outcomes. A questionnaire was used to evaluate the learning environment. The questionnaire consisted of seven, separate topics with statements. For each statement, learners were asked to score to what degree they thought the statement applied to the learning environment in question.
A Likert-scale from 1-5 was used, with on one end 'not at all' and on the other end 'completely'. It was based on the standard evaluation questionnaire used in the educational institute in question. The institute has a standard evaluation procedure to monitor the quality of learning environments. The evaluation questionnaire was part of that quality process. The questionnaire was distributed to all participating students through the online learning environment accompanying the physical environment. The response to the questionnaire was 26% in 2006 and there was a response of 48% in 2007.
The following data were used from the questionnaire:
§ From one topic ('Feedback, Assessment & Reflection') two statements about the assessment process were used to assess the quality of the assessment process.
§ From another, separate topic ('System development') three statements to gather data about the learning outcomes as experienced by the learners were used.
Chapter 6 | 109
§ Furthermore, the answers to the final open questions were used.
Effect measures - Self-reported learning outcomes. The above mentioned three statements and answers to the open questions about the learning outcomes as experienced by learners.
Effect measures - Online websites. After the presentation of the final websites, the external clients selected the project-team with the best website. The clients could request the winning project-team to implement the website to go online. The number of websites that went online was used as an indicator of the level of competence and thus of the overall effectiveness of the learning environment.
Effect measures - Formal grades. All the learners were summatively assessed twice, after the first phase and after the second, final phase of the project. Learners were graded on a scale of 1-10. Grades of six and higher are passing grades, grades of five and lower are fail grades. The formal assessment criteria were established by the educators beforehand and were available to the learners in the educational material.
The formal assessment criteria are representative of the final level of first-year students of a University of Applied Sciences.
6.3 Results
Before determining the effectiveness of the hybrid learning environments with different sources of authentic assessment data, the quality of the assessment process was assessed. We expected to find a relation between the quality of the assessment process and the learning outcomes. The presented data confirmed these
expectations.
6.3.1 Characterisation of hybrid learning environment and assessment process The learning environment was characterised with the model of hybrid learning environments by positioning the different elements of the learning environment in the four quadrants. These elements are described, including the assessment process taking place in each quadrant [see figure 21].
Figure 21: Characterisation of the hybrid learning environment.
Realistic Simulation
Acquisition
Participation Acquisition/Simulation:
•Lectures
•Obligatory books for self-study
Participation/Simulation:
•Project management of a compact project
Acquisition/Realistic:
•Access to online professional sources
•Access to educators in role of senior professional
•Access to senior peers in role of account manager
Participation/Realistic
•Access to online professional communities
•Functional, Graphical & Technical design
•Prototype of website
•Advise implementation & use
•Final website
•Interaction with external client
Chapter 6 | 110
Acquisition/Simulation. In this quadrant, we could find elements with an emphasis on conceptual and procedural knowledge, combined with a context that did not resemble professional practice. The lectures and obligatory books for website development and project management can be positioned here. These elements were not assessed separately, for example in the form of a knowledge test. Learners were expected to use the conceptual and procedural knowledge from the lectures and books to produce the results in the participation/simulation and participation/reality quadrants.
Acquisition/Reality. In this quadrant, we could find elements with an emphasis on conceptual and procedural knowledge, combined with a realistic setting which resembled professional practice. In the studied learning environments, the access to online professional sources, educators and senior peers (second-year learners) in the role of (senior) professional, could be positioned in this quadrant. This interaction was not assessed separately. However, the educators and senior peers also enacted the role of assessor. The whole group of students was divided into sub-groups (six in 2006, seven in 2007). A duo of educators and a senior-peer provided the guidance, expertise and assessment for each sub-group. In the role of assessor during the assessment of the results in the participation/simulation and the
participation/reality quadrant, they would take their interaction with the learners into account.
Participation/Simulation. In this quadrant, learning could be characterised as a situated, ongoing, participatory activity in a classroom setting that slightly resembled professional practice. The project management activities, which were to result in project management documents, could be positioned here. Though these activities and results could be characterised as participatory, the context had a simulation character. The context was not as complex as in a real professional setting.
A selection and the order of project management documents which had to be produced were specified by the educators. The required project management documents, namely, a global project plan and a detailed project plan, were assessed by the assessors. The assessors would assess the results and discuss them to reach consensus. Consequently, each project team had a meeting with the assessors. A combination of a project management document, the project management process and results from the participation/reality quadrant would be discussed during such a meeting. When necessary, the project teams were given the opportunity to improve their results. They would then have to hand in the final versions shortly afterwards.
Above, the formative aspects of the assessment process in this quadrant are described. The two project management documents and the overall project management process were also part of the summative assessment as described in the participation/reality-section.
Participation/Reality. Most of the elements of the studied learning environments could be positioned in the participation/reality quadrant. The project teams were expected to make a functional, graphical and technical design of a website. They also had to develop a prototype of the website and the final website, and gave advice about the implementation and use of the website. The learners were expected to use concepts and theory from the other quadrants in this quadrant. Besides, learners interacted with online professional communities in particular about the technical
Chapter 6 | 111
aspects of the website development. The learners worked in classrooms which were used as collaborative workspaces for the project teams.
The educators scheduled three meetings between the client and the teams at the educational institute. In the first meeting, the client briefed the project teams. The teams presented their prototypes in the second meeting and the final website during the third meeting. A senior-peer enacted the role of account manager. S/he was responsible for managing the contact between the project teams and the client in between these three scheduled meetings (by telephone and e-mail).
The assessment process in this quadrant was similar to the process of the
participation/simulation quadrant. Formative meetings were carried out three times during the project and summative assessments took place twice [see figure 22]. The formative meetings took place before the meetings with the client, which gave the project teams the chance to improve the client-results. After the presentation of the prototype and the website, the summative assessments took place. After the presentations of the final websites, the external client selected which website s/he considered best.
Figure 22: Assessment process
6.3.2 Assessment of the quality of the assessment process
The quality of the assessment process was assessed with a set of quality criteria, besides, data from the questionnaire about the assessment process as experienced by the learners was used.
Assessment as experienced by learners. The data from the questionnaire show that learners were quite satisfied with the assessment process. They felt that the process reflected the content of the learning environment and that the assessment was based on clear criteria [see table 18].
Briefing client Presentation prototype Presentation website + best website
Formative assessment: