Quick scan of learning outcomes assessment instruments: report for the OECD

(1)

Quick scan of learning outcomes assessment

instruments

Report for the OECD

Nadine Zeeman Renze Kolster Rainer Mensing Don Westerheijden Hans Vossensteyn August 2016 Center for Higher Education Policy Studies Universiteit Twente P.O. Box 217 NL-7500 AE Enschede www.utwente.nl/cheps

Reference: …

(2)

1 Rationale and Introduction ... 3

2 Learning outcomes assessment instruments in higher education characterised ... 4

2.1 What are general skills? ... 4

2.2 How are general skills being tested? ... 5

3 Methodology of the desk study ... 6

4 Results and conclusion ... 7

5 Learning outcomes assessment instruments ... 9

5.1 Template ... 9

5.2 Listing of general learning outcomes test instruments ... 10

GRE revised General Test ... 10

Graduate Aptitude Test in Engineering (GATE) ... 11

ACT Collegiate Assessment of Academic Proficiency (ACT CAAP) ... 12

Collegiate Learning Assessment (CLA+) ... 13

Graduate Skills Assessment (GSA) ... 15

Valid Assessment of Learning in Undergraduate Education (VALUE) ... 17

ETS Major Field Test (MFT) in Business ... 19

Degree Qualifications Profile (DQP) ... 21

Alternate Uses Test (AUT) ... 23

Torrance Tests of Creative Thinking (TTCT) ... 24

Remote Associates Test (RAT) ... 25

ETS Proficiency Profile Test ... 26

Mathematics Self-Efficacy Scale ... 28

National Student Performance Exam (ENADE) ... 29

EQ-i 2.0 ... 31

Literacy and Numeracy Test for Initial Teacher Education Students (LANTITE) ... 32

(3)

(4)

A quick scan of learning outcomes assessment instruments

1 Rationale and Introduction

For a long time already, the OECD has been interested in comparing countries’ performances in many

areas of the economy and society, including education. Internationally-applied tests of diverse

performances of learners such as PIRLS, PISA and TIMMS (and PIAA) have been received in many countries

with much interest, and—for better or for worse—have influenced national policies with regard to

curriculum changes in primary and secondary education. PIRLS is an assessment tool administrated at

children in the fourth, fifth or sixth grade of primary education. PISA is a test administrated to 15-year-old

pupils in secondary education and TIMMS is a test administrated to pupils at the fourth and eighth grades

of secondary education. PIAA measures skills of adults beyond traditional school age. Students in tertiary,

postsecondary or higher education (for short, in this report we will use the term ‘higher education’ for

every kind of post-secondary or tertiary education) have not been addressed by these tests.

In the 21

st

_{century knowledge societies, attention of policy makers around the world extends increasingly}

to higher education as well, as it is seen to be necessary to prepare countries’ populations for living and

working in these new and exciting times, which need more, new and more complex skills both at work

and at home. Gaining insight into how their own country performs in relation to others becomes ever

more important to many decision makers in higher education: is the country’s higher education system

doing all than can—and needs to be—done for the wellbeing of the population and the country? The

OECD has responded to this increased attention for higher education’s system-level performance at first

by developing and piloting a set of tests to compare performance of students near the end of their

undergraduate education, called AHELO. While this was successful as a ‘proof of concept’, the

implementation of such a large-scale international test was not continued beyond the pilot, for practical

(financial) reasons but also due to critique of the concept of uniform testing of students studying in very

different contexts and cultures.

The current report is intended to assist OECD to respond to both types of criticism by listing a number of

available learning outcomes assessment instruments across the world that may be useful in an

international comparative context.

The type of learning outcomes assessment instruments on which we focused were those of general skills

and competencies rather than those focusing on a single discipline or field of knowledge. There are two

main reasons for this choice. First, for comparative reasons: the interest for international comparison is

in developing a general picture of achievement of undergraduate students of a country, rather than

detailed analysis of specific disciplines. Second and more a matter of principle, it is generally asserted that

in the knowledge society, higher-order skills and competencies (‘21

st

_{century skills’) are more important}

than specific performance in specific subjects—although at the same time the importance of the STEM

areas is emphasized as well (Science, Technology, Engineering and Mathematics). It is not our intention

to make a choice in this type of discussion; we only aim to survey the field of available general skills tests.

(5)

In the following pages, we next discuss a number of issues around learning outcomes assessment

instruments—their history and major dimensions along which they can be differentiated. That leads over

into a section on the methodology of our survey: how did we select the cases included in our report, and

how did we describe them (the dimensions used in more detail)? The textual part of the report ends with

a brief conclusion, after which we list the ‘short list’ of tests described according to our full format, and

the rest of the ‘long list’ which were described more succinctly using part of the format.

2 Learning outcomes assessment instruments in higher education

characterised

2.1 What are general skills?

In a brief blog entry, education scientist Paul Kirschner at once criticized the hype around ‘21

st

_century

skills’—because almost all of them were already relevant in the 20

th

_{century as well—listed those general}

skills in a systematic graph (Fig. 1) and pointed out which ones were the 21

st

_{century additions to the}

general skills, i.e. information management

(ability to collect and manage information in

multiple formats); knowledge management

(enhancing

communication,

information

transfer, and collaboration; and publication

management (the ability to understand the

dissemination of knowledge / work and new

publishing modes) (Kirschner, 2015).

Note that Kirschner goes beyond the common ‘4

Cs’ of communication, collaboration, critical

thinking and creativity as the essential 21

st

century skills. He (like some others) adds

self-direction and life-long learning as well as social

responsibility, next to singling digital literacy out

from communication in general (see Fig. 1). His

figure also adds the major aims of these skills: to

become an engaged thinker and ethical citizen,

beyond showing entrepreneurial spirit.

Instruments testing individual students cannot

directly assess the social aspects of the skills involved (communication, collaboration). Just like the two

higher-order competencies of self-direction/life-long learning and social responsibility, these social skills

can only be approached indirectly through paper and pencil tests or their online variants. Consequently,

most attention in developing test instruments has been on the more individual parts of the 4 Cs: critical

thinking and creativity.

The two concepts of critical thinking and creativity have some commonality, if one looks at current

definitions. Both focus on higher-order elements of Bloom’s taxonomy of learning or its modern variants:

Fig. 1 General skills for 21st century learners (source: Kirschner, 2015)

(6)

synthesis, evaluation and (tautologically) creation play a role. Moreover, Lau & Chan, in their Hong Kong

University sponsored website on critical thinking, convincingly argue (Lau & Chan, 2004-2016):

Some people believe that critical thinking hinders creativity because it requires following the rules of logic and rationality, but creativity might require breaking rules. This is a misconception. Critical thinking is quite compatible with thinking "out-of-the-box", challenging consensus and pursuing less popular approaches. If anything, critical thinking is an essential part of creativity because we need critical thinking to evaluate and improve our creative ideas.

For our report, following this reasoning we will focus both on instruments measuring critical learning and

measuring creativity.

Critical thinking apparently was first conceptualized by Glaser as ‘(1) an attitude of being disposed to

consider in a thoughtful way the problems and subjects that come within the range of one's experiences,

(2) knowledge of the methods of logical inquiry and reasoning, and (3) some skill in applying those

methods’ (Glaser, 1941). Some authors add a value component to its definition, e.g. Elder: ‘Critical

thinking is self-guided, self-disciplined thinking which attempts to reason at the highest level of quality in

a fair-minded way’ (quoted in www.criticalthinking.org/pages/defining-critical-thinking/766; emphasis

added), while others including Scriven & Paul (on the same website) separate the skill from the motive,

which in their view can be selfish as well as altruistic. Whatever the motive, with a view to developing

testing instruments the skills and operational levels are more important. Benjamin et al. (2012, p. 5) cited

Pascarella and Terenzini, who in their major review volume stated that critical thinking definitions

operationally converged on:

an individual’s capability to do some or all of the following: identify central issues and assumptions in an argument, recognize important relationships, make correct references from the data, deduce conclusions from information or data provided, interpret whether conclusions are warranted based on given data, evaluate evidence of authority, make self-corrections, and solve problems

Creative thinking or creativity—often used interchangeably—can be defined as ‘the tendency to generate

or recognize ideas, alternatives, or possibilities that may be useful in solving problems, communicating

with others, and entertaining ourselves and others’ (Franken, 1993, p. 396). Others too emphasise that

creativity involves new, innovative or unusual ideas, products and behaviour (e.g. Csikszentmihalyi, 2014;

Piirto, 2011). These researchers, and still others, stress that creativity is a skill not (fully) innate but at least

partially learned, time-dependent and related to a social situation—and that it is amenable to

psychometric testing.

2.2 How are general skills being tested?

Defining learning outcomes, in particular of 21

st

_{century skills, is a recent development in higher education}

around the world. Even more recent is the desire at a national systems level to devise test instruments to

assess achievement of those outcomes or skills independently from the teachers involved in teaching the

students within the higher education institutions. Until then, teachers were implicitly trusted to teach and

assess their students—if people thought of assessment as a separate activity at all. Psychometric

instruments were used only for research or for private companies’ decision making, e.g. in hiring

(7)

personnel. In the latter type of application, rigour of methodological underpinning of tests (reliability and

validity) was often but not always assured.

The USA since the mid-1980s led the movement towards student assessment (Lubinescu, Ratcliff, &

Gaffney, 2001), while in Australia or in Europe parallel developments ensued. In Australia, the discussion

was fuelled by the quality regulator of the system, TEQSA’s suggestion to consider tests of graduate

learning outcomes (cf. AAGLO, 2011a). In Europe, interest in learning outcomes developed especially since

the articulation of the European Standards and Guidelines for quality assurance (or ESG; see European

Association for Quality Assurance in Higher Education, 2005; European Association for Quality Assurance

in Higher Education et al., 2015). A country that introduced nation-wide testing of students was Brazil,

which introduced its Provão in 1996, and replaced it with a National Examination of Student Performance

(Enade) from 2004 onwards (in selected courses every year), which is described in our listing below.

However, the main source of tests remained the USA. The growing demand for accountability has

increased the interest for educational performance outcomes. Higher education institutions have reached

agreement on common outcomes for their undergraduate education. This has heightened the

dependence on especially standardized tests to assess the skills of college students (Lakin et al., 2012).

USA

is

where

standardised

testing

has

been

pioneered

(see

https://www.timeshighereducation.com/news/standard-tests-students-big-leap-forwards-or-moving-wrong-direction) and is also the country where the tests carry most weight in institutional policy-making,

decision making by students and parents (connection to quality assurance through the Voluntary System

of Accountability), while state and federal governments variously prescribe or stimulate using such tests.

3 Methodology of the desk study

To get an overview of available instruments to assess tertiary students’ general skills and competencies,

especially critical thinking and creative thinking, we performed a desk study, basing ourselves on literature

and web search. In essence, our search strategy included:

§

Higher education and educational science books and journals from about 2005, traceable through

libraries and databases such as Scopus, WoS, Google Scholar, on the development and/or

evaluation of testing instruments;

§

Accreditation and other quality assurance agencies’ handbooks and/or websites available in

major international languages (English, French, Spanish, German);

§

Testing services’ handbooks and/or websites available in major international languages (ETS, CLA,

etc.);

§

International higher education projects sponsored by major authorities and agencies. Next to

OECD itself, these included EU programmes such as Erasmus+ and KP6/KP7/Horizon2020

including Tuning-projects; US DoE, CHEA; Lumina Foundation; Carnegie Commission; UNESCO and

its national commissions; SEAMEO in APEC; and INQAAHE (with subnetworks such as APQN,

ENQA).

(8)

Through these sources we found at least 78 instruments, which were described briefly as a source for

further selection. To create the ‘long list’ of general skills assessment instruments, we applied as criteria

that each instrument be:

§

Applied across higher education systems (i.e. across multiple jurisdictions, to include e.g.

instruments used in federal systems such as the USA)

§

Assessed general knowledge, skills and competencies and fit for use in several fields of knowledge

§

Used since 2005

This resulted in a table (excel file) including 51 general test and 4 that were both general and specific.

From this set of 55, we selected a ‘short list’ of 16 cases for in-depth description. The main criterion for

inclusion in the ‘short list’ was that we aimed to cover the breadth of variety in the ‘long list’, while the

available time was a boundary condition on the number of cases we could include.

To allow for standardised comparison, the presented cases were described using a fixed format. The

format includes the name of the test, a short description of main topic, the target group, the main aim

and stakes, the assessed learning outcomes, used methods and advised sampling, where it is applied, the

starting year, who developed the tests and under whose auspices, the costs to use the test (usually per

test taker), the validity and evaluation of the test. The template is shown in section 5.1.

4 Results and conclusion

The sixteen tests are described along the template in one- to two-page tables in section 5.2, below. On a

general level, we present the following conclusions and suggestions.

Most standardised instruments to assess higher education students’ general skills were found to originate

in the USA. Most tests focus on cognitive skills, such as critical thinking and, associated with it, problem

solving. Fewer address non-cognitive skills, such as communication (writing skills) and creativity. Only one

that that we included in our shortlist, the Mathematics Self-Efficacy Scale, appears to cover attitudes

toward learning. With respect to field specific knowledge, it appears to be easier to assess the learning

outcomes of study programmes with a core curriculum that is more standardised across institutions (e.g.

engineering).

Other countries (e.g. Australia and in Europe) focus not on assessing learning outcomes, but more on

making sure that all students are actually taught the expected learning outcomes. Instruments to do so

include qualification frameworks, the different Tuning projects to achieve consensus regarding learning

outcomes to be aimed for at the discipline level, and learning outcome rubrics.

Tests may have several pros and cons, depending on the principal’s view on several dimensions. In

general terms, these considerations are related to:

• Validity of multiple choice vs. open questions; note that some tests use a mixed methods

approach (Steedle, Kugelmass, & Nemeth, 2010). Multiple choice tests are easier to implement

and to compare; some hold that they are less valid with regard to the concept of critical thinking

than tests involving open questions and essays written by students (Benjamin et al., 2012).

(9)

• Time investment and motivation of test takers

1

_{. Low-stakes test situations tend to show}

significantly lower levels of performance (hence lower levels of valued-added) than high-stakes

test situations (Liu, Bridgeman, & Adler, 2012)

• Skills actually tested vs. self-reported skills

• Standardized tests vs. real work assessment (AAGLO, 2011b, 2011c)

• Online / digital test taking vs. paper-pencil test taking

• Measuring learning outcomes vs. conditions in which outcomes are achieved, e.g. student

satisfaction as in many European national student surveys vs. student engagement as in the US

NSSE (Kuh, 2001).

• Not all the considered tests appear to measure learning outcomes that can be directly related to

higher education. For example, creativity. Is it fair to measure these (highly relevant—no doubt

about that from a social point of view!) outcomes across higher education systems, even if they

are not explicitly included as intended learning outcomes?

• Conclusion [to be elaborated in definitive version]:

o Which are the most valid tests around? …

o Are they also valid in cross-national/cross-cultural use? Hard to say; if validity was

researched, it was mostly/always? In a single, Western-world setting.

o For cognitive and non-cognitive skills? Yes.

o Do they cover all of the ‘21

st

century skills’? …

o Did we find more instruments intended for STEM areas (but not elaborated in the short

list as it was our remit to look at general learning outcomes)? …

1_{Motivation to participate in tests is very important, see:} http://www.tandfonline.com/doi/pdf/10.1080/10627197.2015.1127753

(10)

5 Learning outcomes assessment instruments

5.1 Template

Name test

_Name of test

Short description of main topic What is the main topic of the test: what does it attempt to test (e.g. problem solving skills)

Target group Age To which age group is the test targeted?

Level first cycle/undergraduate / second cycle/postgraduate students / year of study / undefined Field (which field and/or general?) Main aim & stakes What is the main aim of the instrument? e.g. Summative assessment for final grading – access to a profession – access to postgraduate study – formative assessment – benchmark of a study programme – required to access to postgraduate study, or no stakes (just for national statistical purposes Learning outcomes which learning outcomes are assessed? Knowledge Cognitive skills Non-cognitive skills Attitudes Methods Which methods does the instrument use? E.g. Paper and pencil tests, designs, assessment of existing student work? Sampling Which sampling principles does the instrument use? e.g. All students in the programme/school/university – randomly selected – volunteers – selected by previous GPA – obligatory / voluntarily participation Applied in Region: Country: HEI: Types of higher education institutions (or parts of higher education institutions: (schools/faculties or study programmes) has the instrument actually been used?) Number of users: (How many times has the test been taken?) Starting year When has it been used? [We propose limiting our search to instruments used in the most recent dozen of years, i.e. since about 2005.] Auspices Under whose auspices is the instrument used, who is the principal? (With contact data) Developer Agency or research group that developed the test (With contact data) Cost As reported by developer (e.g. cost per user / cost of using test) Validity and evaluation of the test - How long after taking the test do test results remain valid? - Source of evaluation (e.g. journal publications): - Assessment of thoroughness of evaluation design (experiment, survey, case study etc.): - Main results of evaluation Final remarks Pros and cons of the test (self-reported) References Included if necessary

(11)

5.2 Listing of general learning outcomes test instruments

Name test

_{GRE revised General Test}

Short description of main topic Graduate Record Examination’s revised General Test focuses on student’s ability to perform in second or third cycle programmes. It is meant to supplement and support students’ application to education on these levels. It tests reasoning and analytical writing skills.

Target group Age Not specified

Level First cycle/undergraduate students Field Graduate schools and business schools (MBA, specialized master's in business or doctoral degrees) Main aim & stakes Summative assessment to show preparedness for second and third cycle study programmes. Learning outcomes Knowledge - Cognitive skills - Verbal Reasoning (ability to analyse and evaluate written material and synthesize information obtained from it, analyse relationships among component parts of sentences and recognize relationships among words and concepts. - Quantitative Reasoning (problem-solving ability using basic concepts of arithmetic, algebra, geometry and data analysis.) - Analytical Writing (critical thinking and analytical writing skills, specifically ability to articulate and support complex ideas clearly and effectively) Non-cognitive skills - Attitudes - Methods - Computer-delivered test (length: three hours and 45 minutes with short breaks) - Paper-delivered test - The GRE revised General Test is available at more than 1,000 test centres in more than 160 countries. Sampling Voluntarily participation; students are to register for the test

Applied in Region International

Country -

HEI: graduate schools / business schools (HEIs using GRE have a specific code)

Number of users: Number of users: 766,185 tests administered worldwide (2015)

Starting year 1949

Auspices -

Developer Educational Testing Service (ETS): https://www.ets.org/ Website test: http://www.ets.org/gre/ Cost $205 per test (paid by test taker) Validity and evaluation of the test - GRE scores are valid for five years after the testing year - Source of evaluation: Kuncel, N. R., Hezlett, S. A., & Ones, D. S. (2001). A comprehensive meta-analysis of the predictive validity of the graduate record examinations: implications for graduate student selection and performance. Psychological bulletin, 127(1), 162. - Assessment of thoroughness of evaluation design: 1,753 independent samples - Main results of evaluation: “The results indicated that the GRE and UGPA are generalizably valid predictors of graduate grade point average, 1st-year graduate grade point average, comprehensive examination scores, publication citation counts, and faculty ratings. GRE correlations with degree attainment and research productivity were consistently positive; however, some lower 90% credibility intervals included 0. Subject Tests tended to be better predictors than the Verbal, Quantitative, and Analytical tests.” Final remarks -

(12)

Name test

_{Graduate Aptitude Test in Engineering (GATE)}

Short description of main topic Test takers pick a paper topic related to their field of study or the field or study for which they intent to apply. Tested is their knowledge of the field through specific questions and their general Aptitude (Language and Analytical Skills).

Level Final year undergraduate students or degree holders in engineering, technology & science programmes Field Engineering, technology & science Main aim & stakes The summative assessment results in a score that is used for admissions to various post-graduate education programs (e.g. Master of Engineering, Master of Technology, Doctor of Philosophy) in Indian higher education institutes. Successful test takers may also be awarded (government) scholarships. Test results may also be used for entry-level engineering jobs.

Learning outcomes Knowledge - Field specific knowledge

Cognitive skills - General Aptitude (Language and Analytical Skills) Non-cognitive skills -

Attitudes -

Methods ONLINE Computer Based Test (CBT) (length: three hours), tested at fixed locations.

Sampling Voluntarily participation; students are to register for the test

Country India

HEI: Indian Institutes of Technology

Number of users: 818,850 (in 2016)

Auspices -

Developer Indian Institute of Science & Indian Institutes of Technology (http://www.gate.iisc.ernet.in/)

Cost Approximately between $11 and $22 per test (paid by test taker) Validity and evaluation of the test - Test results remain valid three years from the date they are announced. - The validity of the test has not been evaluated. This is perhaps because the questions change annually. - The high reliance on multiple choice questions is a weakness of the GATE test (Hegde, et al., 2014) - There have been (old) reports that the exam is unable to differentiate between brilliant students and highly task-committed students. Consequently, the test may be more of a test of endurance than of intelligence (Jayaram, 2011). Final remarks - References Hegde, A., Ghosh, N., & Kumar, V. (2014). Multiple Choice Questions with Justifications. In Technology for Education (T4E), 2014 IEEE Sixth International Conference on (pp. 176-177). Jayaram, N. (2011). Chapter 6: Towards world-class status the IIT system and IIT Bombay, in Altbach, P. G., & Salmi, J. (Eds.). The road to academic excellence: The making of world-class research universities. World Bank Publications.

(13)

Name test

_{ACT Collegiate Assessment of Academic Proficiency (ACT CAAP)}

Short description of main topic The American College Testing (ACT) Collegiate Assessment of Academic Proficiency (CAAP) enables postsecondary institutions to assess, evaluate, and enhance student learning and general education program outcomes.

Level Undergraduate students Field Postsecondary science curriculum Main aim & stakes ACT CAAP offers six independent test modules covering a range of cognitive skills. CAAP tests are used by postsecondary institutions to measure the academic progress of students and to help determine the educational development of individual students. The outcomes provide institutions / programmes evidence that education objectives are being met. For test takers the outcomes (proven by a Certificate of Achievement) indicate their readiness for further education, need for interventions, and assure some specified level of skill mastery prior to program completion. Learning outcomes Knowledge - Cognitive skills - Reading (Reasoning Skills, Referring Skills) - Science (Analysis, Generalization, Understanding) - Writing Skills (Punctuation, Grammar, Sentence Structure, Strategy, Organization, Style) - Critical Thinking (Analysis of Elements of Arguments, Evaluation of Arguments, Extension of Arguments) - Mathematics (Prealgebra, Elementary Algebra, Intermediate, Algebra, Coordinate Geometry, College Algebra, Trigonometry) - Writing Essay (organizing, developing, and conveying in standard written English the writer's ideas about the topic) Non-cognitive skills - Attitudes - Methods Pencil and paper based, using open and multiple choice questions (length: each modules takes 40 minutes) Sampling Based on requirements set by institutions (e.g. all or a random sample of students from certain program are to participate)

Country USA HEI: - Number of users: More than 250 institutions use ACT CAAP each year. Number of test takers not specified. Starting year 1990 (operational) Auspices -

Developer ACT incorp:

http://www.act.org

CAAP: http://www.act.org/content/act/en/products-and-services/act-collegiate-assessment-of-academic-proficiency/about-act-collegiate-assessment-of-academic-proficiency.html Cost Cost per taken test: Students Taking a Single Test: $14.75 (<500 test taker) or 13.75 (>500 test takers) Students Taking a Single Test: $22.00 (<500 test taker) or 20.80 (>500 test takers) Writing Essay test fee: $14.75 Additional costs for reporting by ACT Validity and evaluation of the test - More than 250 institutions use ACT CAAP each year, providing norms tables with national comparative data. ACT CAAP norms are updated every fall using test scores from the past three years and are categorized by type of institution (two-year or four-year), year of students tested (freshman, sophomore, junior, senior), and institutional ownership (public, private). - Tests are designed and checked by teams of experts. - Numerous academic publications use ACT CAAP as a data source. Final remarks -

(14)

Name test

_{Collegiate Learning Assessment (CLA+)}

Short description of main topic CAE pioneered the use of performance-based assessments, such as the Collegiate Learning Assessment (CLA+), to challenge students to demonstrate skills rather than to simply recall information. CLA+ is one of the few “standardized test that students can take, add to their transcript, and send to prospective employers to assist them in prescreening applicants” (CLA+ Student Guide).

Level Undergraduate students (freshmen and/or final year students) Field General Main aim & stakes - In 2002, CAE launched a national effort to assess the quality of undergraduate education by directly measuring student learning outcomes through performance tasks. The assessments are delivered online and use real-world problem-solving tasks to measure students' critical-thinking skills. - CLA+ allows students to compare themselves to others at their institution and across other institutions, as well as the opportunity to prove themselves to potential employers. The student report is a credential that you send potential employers and/or graduate schools as you approach graduation. - The institutions that participate in the CLA+ receive insights into how, and by how much, they contribute to the development of their students’ higher-order skills. Collecting this information is one step in the process of improving teaching and learning practices. Learning outcomes Knowledge - Cognitive skills - Critical Thinking - Problem Solving - Analytic Reasoning - Effective Communication - Scientific Reasoning - Quantitative Reasoning - Critical Reading Non-cognitive skills - Attitudes - Methods Computer administrated, either online or in fixed location. The former uses webcams to allow for monitoring by a proctor. The test consists of two parts: the performance task and selected-response questions. Students have 60 minutes to complete a Performance Task (evaluate information, critical thinking, writing). Students do so based on a provided document library. The selected-response questions take 30 minutes to complete (critical reading & evaluation, scientific & quantitative reasoning, critique an argument). Sampling Not explained in detail, but not all test takers may need to take all aspects of the CLA. Hence, some of the outcomes on institutional level are based on sampling.

Country USA HEI: - Number of users: Over 150 colleges and universities annually participate in the CLA+. The number of test takers has not been specified. Starting year 2002 (development) - 2004 (operational) Auspices -

Developer Council for Aid to Education (CAE): http://cae.org/

(15)

Validity and evaluation of the test - For validity discussion see: http://cae.org/images/uploads/pdf/Reliability_and_Validity_of_CLA_Plus.pdf, Klein, et al. (2007) and Possin (2013) - CAE+ has been endorsed by a number of higher education stakeholders in the US (e.g. US Department of Education). - CAE+ has been tested in other contexts ( http://cae.org/images/uploads/pdf/A_Case_Study_of_an_International_Performance-Based_Assessment_of_Critical_Thinking_Skills.pdf) Final remarks Pro and cons of the test: Usage of opened questions, the test measures achievement directly, rather than relying of self-reported achievement, the tests are largely assessed by computers, only 10% of the scores are checked by graders (Possin, 2013) References Klein, S., Benjamin, R., Shavelson, R., & Bolus, R. (2007). The Collegiate Learning Assessment Facts and Fantasies. Evaluation Review, 31(5), 415-439. Possin, K. (2013). A fatal flaw in the Collegiate Learning Assessment Test. Assessment Update, 25(1), 8-9.

(16)

Name test

_{Graduate Skills Assessment (GSA)}

Short description of main topic The Graduate Skills Assessment was developed under the Australian Higher Education Innovation Program. The test has been designed to assess generic skills of students when they begin at university and just before they graduate

Level University entry and exit level Field General Main aim & stakes - Universities may wish to use the test to compare differences in student profiles between fields of study and are likely to be interested in changes between entry and exit points for students in different courses. - At entry level, universities may also use the test diagnostically to identify, for example, those who write poorly or have trouble dealing with text-based critical thinking items or quantitative problem solving items. Such students may be followed up and offered assistance. - At exit level, results of the test may be used as an additional criterion for entry into post-graduate courses or as an indication of generic skills to an employer. - Universities receive an electronic data file of candidates' results on the five scales of: Critical thinking, Problem solving, Interpersonal understandings, Report writing, Argument writing. - Every student sitting the GSA receives a personalised report from ACER. This gives the score for each component, on a scale divided into three levels, describing the skills demonstrated at each level. - The report also provides comparative data, comparing the scores with all other students who have sat for the test, as well as more specifically with results from other students studying in the same or similar disciplines. Learning outcomes Knowledge - Cognitive skills - Problem Solving: analysis of information and its application to problems and decision-making - Critical Thinking comprehension, analysis and evaluation of viewpoints and arguments in order to aid decision-making. - Interpersonal Understandings: analysis and evaluation of work and other interpersonal situations. - Written Communication: ability to present a well-organised and clearly expressed response based on an analysis of the information provided. Two writing tasks are required: a Report Writing task and an Argument Writing task Non-cognitive skills - Attitudes - Methods The test consists of a multiple-choice test and two writing tasks. The multiple-choice test assesses Critical thinking, Problem solving and Interpersonal understanding, and is two hours long. The writing test includes a reporting task and an argument task. It assesses Language and expression (e.g. control of language conventions, clarity and effectiveness of expression) and Organisation and thought (e.g. effectiveness and purposefulness of organisation, depth of analysis of issues or information). The written test is sixty minutes long. Sampling Not specified Applied in Region - Country Australia HEI: - Number of users: In 2000, there were originally 2,000 students and 20 universities that participated in the Generic Skills Assessment. However, participation of students and universities fell steadily thereafter. Starting year 2000 (exit level assessment) - 2001 (entry level assessment) (current status is unclear, it may have been discontinued around 2010; Yorke & Vidovich, 2016)

(17)

Auspices Department of Education, Training and Youth Affairs (DETYA) (now Department of Education, Employment and Workplace Relations and Department of Innovation, Industry, Science and Research)

Developer Australian Council for Educational Research (ACER): https://www.acer.edu.au/gsa

Cost - The Australian Government provided $3.5 million in funding to support the administration of the Graduate Skills Assessment between 2000-2009. - Each test taker was charged $20 (covering around one-quarter of the administration costs) Validity and evaluation of the test - For validity discussion see: https://www.acer.edu.au/files/GSA_ValidityStudy.pdf - After widespread criticism, the GSA was not considered in the 2010 Indicator Framework Discussion paper by the Australian government (Yorke & Vidovich, 2016) Final remarks - References Yorke, J., & Vidovich, L. (2016). Learning Standards and the Assessment of Quality in Higher Education: Contested Policy Trajectories (Vol. 7). Springer. Also see: Assessment of Generic Skills Discussion Paper December 2011 http://goo.gl/yebXVZ

(18)

Name test

_{Valid Assessment of Learning in Undergraduate Education (VALUE)}

Short description of main topic VALUE provides generalized rubrics for three assessment domains: intellectual and practical skills, personal and social responsibility, and integrative and applied learning. Within the domains, there are a total of 15 measurable outcomes and a generalized rubric for measuring these outcomes. These generalized rubrics are adaptable to individual institutions, and provide a list of criteria and achievement descriptions for each outcome.

Level Associate and undergraduate levels Field General Main aim & stakes - VALUE rubrics or scoring guides provide needed tools to assess students’ own authentic work, produced across their diverse learning progressions and institutions, to determine whether and how well students are meeting graduation level achievement in learning outcomes that both employers and faculty consider essential. - The VALUE rubrics are being used to help institutions demonstrate, share, and assess student accomplishment of progressively more advanced and integrative learning. - The VALUE rubrics have been approved for use in meeting national standards for accountability established by the Voluntary System of Accountability (VSA). - Not only can individual students know their progress and proficiency levels, but institutions can aggregate assessment data and get a sense of the learning that is happening in a particular course, department, or school. Over time, the institution can chart learning outcomes improvement as a whole using the evaluation data from one or more VALUE rubrics. Learning outcomes Knowledge - Cognitive skills - Intellectual and practical skills (Inquiry and analysis, Critical thinking, Creative thinking, Written communication, Oral communication, Reading, Quantitative literacy, Information literacy, Teamwork, Problem Solving)) - Personal and social responsibility (Civic knowledge and engagement - local and global, Intercultural knowledge and competence, Ethical reasoning, Foundations and skills for lifelong learning) - Integrative and applied learning Non-cognitive skills - Attitudes - Methods The assessment rubrics are applied to students’ prior work (e.g. from courses in the first two years) that are collected in an electronic portfolio. The work in the portfolio is than assessed using the VALUE rubrics. Ideally, more than one evaluator assesses the piece of work, and the level of mastery is determined by the group. Sampling If an institution has made it mandatory for students to collect their work in a portfolio, the VALUE rubrics can be applied to all.

Country USA

HEI: 4,200 unique institutions, including more than 2,800 colleges and universities

Number of users: 42,000 users (December 2015)

Auspices -

Developer American Association of Colleges and Universities (AACU): http://assessment.aa.ufl.edu/value-rubrics

Cost The VALUE rubrics are made available free of charge. There are no costs for students, since the assessment is based on prior work. Institutional costs include the assessment time by the evaluators and possibly expenses for trainings to use the VALUE rubrics effectively. As indicated by AACU: “Initial individual campus findings indicate that a VALUE rubric/e-portfolio approach is actually cheaper than when they use standardized tests.” (http://www.aacu.org/value-faq#question9)

(19)

Validity and evaluation of the test - For validity discussion see: http://www.aacu.org/peerreview/2011-2012/fall-winter/finley - VALUE rubrics have been used and evaluated in several academic publications. Final remarks Self-reported pros of the test include: - All students are assessed, using material that they had incentives to worked on (unlike standardized test that take a snapshot of a sample of students, by having them take a test for which they may not be fully motived as it hold limited consequences). - Good psychometric practice rejects the idea of using any single measure as a proxy either for individual student proficiency or for institutional evaluation. Information from a particular test, because it is disconnected from specific curricula, provides little help for students or faculty to identify specific areas in which to focus their own efforts to achieve higher levels of mastery. - By assessing students’ best work done in their curricula, those who evaluate the level of achievement get a fuller picture of how much a student’s knowledge and skills have grown and matured during college. - See: https://www.aacu.org/value-faq

(20)

Name test

_{ETS Major Field Test (MFT) in Business}

Short description

of main topic

Measure student learning outcomes for bachelor's degree business programs for use around the world.

Level Bachelor level (for Business also available for Associate and Master level) Field Business (also available for Biology, Mathematics, Chemistry, Music, Computer Science, Physics, Criminal Justice, Political Science, Economics, Psychology, Literature in English, Sociology) Main aim & stakes The Business Test has five key objectives: - Measure students’ knowledge of the multidisciplinary subject matter representative of undergraduate business education - Provide information to students regarding their level of achievement - Provide information on student achievement to faculty to enable institutions to assess their performance relative to their specific mission and objectives - Provide information to facilitate development of appropriate goals and educational programs - Strengthen the accountability of undergraduate business education Institutions use the Major Field Tests to measure students' mastery of their chosen field of study, assess the effectiveness of major programs of study and improve curricula and student learning outcomes.

Learning outcomes Knowledge - Subject knowledge (accounting, economics, finance, law, management, marketing, information systems, quantitative analysis and international business) Cognitive skills - Ability to apply facts, concepts, theories and analytical methods - Students’ abilities to analyse and solve problems, understand relationships and interpret material. Non-cognitive skills - Attitudes - Methods - The test can be taken online or by paper-and-pencil and are monitored by on or off-campus proctors. - The Major Field Test in Business contains 120 multiple-choice items; some are based on case studies. - Academic departments may add up to two subgroups and as many as 50 additional locally written questions to test areas of the discipline that may be unique to the department or institution. - To ensure fairness and content relevance, the test is revised approximately every four to five years. - The programs can select the time and venue of the two-hour test, which may be split into two sessions. This test must be given by a proctor - Sample questions: https://www.ets.org/Media/Tests/MFT/pdf/mft_samp_questions_business.pdf Sampling Participating institutions can select a cohort of students that take the test.

Country USA HEI: Not specified Number of users: 132,647 individuals at 618 different institutions completed the MFT-B between 2005 and 2009 (Bielinska-Kwapisz & Brown, 2013) Starting year 1990 Auspices -

Developer ETS: https://www.ets.org/s/mft/pdf/mft_testdesc_business.pdf

Cost - Undergraduate online test on-campus: $25 (1-99 test takers) - $24 (100 or more test takers) - Undergraduate online test off-campus: $20.50

(21)

- Institutions may ask test takers to contribute to the fee. - Extra costs for reports in addition to the included standard ETS Major Field Test Reports. Validity and evaluation of the test - The validity of the MFT in Business appears not to be widely tested. One critical publication recognizes three major flaws that are connected to the scaling of the test’s scores, determinants of student performance on test’s scores, and the unknown comparison group (Green, et al., 2014). Based on these flaws the authors advise against the use of the MFT in Business to assess student learning or to alter curricula. - Particularly the MFT in Business has been used as a data source in several academic publications. - The MFT in Business is widely used by business schools in the US, and is an acceptable measure by business programme accreditation agencies. Final remarks - References Bielinska-Kwapisz, A., & Brown, F. W. (2013). Differential gender performance on the major field test– business. Journal of Education for Business, 88(3), 159-166. Green, J. J., Stone, C. C., & Zegeye, A. (2014). The Major Field Test in Business: A Solution to the Problem of Assurance of Learning Assessment?. Journal of Education for Business, 89(1), 20-26.

(22)

Name test

_{Degree Qualifications Profile (DQP)}

Short description of main topic The DQP provides students with useful information about their current skills, knowledge, and competencies. Enables the university to evaluate the effectiveness of its academic courses and programs in terms of achieving the desired learning outcomes for its students. Is used for continuous improvement at all levels of the institution. Ensures that students are prepared for success in work and citizenship in a diverse, global society. The Degree Qualifications Profile (DQP) outlines a set of reference points for what students should know and be able to do upon completion of associate, bachelor’s and master’s degrees – in any field of study.

Level Associate, bachelor’s and master’s level Field General Main aim & stakes Similar to the VALUE rubrics, the DQP is an ex-ante stocktaking of students’ prior work. The major difference between the two is that DQP outlines the kind of tasks students should do to develop expected proficiencies, while the VALUE rubrics address the question of “how well” the student demonstrates key intellectual proficiencies. In this respect the DQP is much like the European Qualification Framework. The DQP describes generic forms of student performance appropriate for each degree level through clear reference points that indicate the incremental, integrative and cumulative nature of learning. As such, DQP goals include: - An emerging common vocabulary for sharing good practice in degree granting by U.S. higher education institutions. - A foundation for better public understanding of what institutions of higher education actually do in their instructional and learning assessment roles. - Reference points for accountability that at least complement — and ideally, replace — less revealing measures of improvements in student learning such as test scores or tallies of graduates, research dollars, student satisfaction ratings, or job placements and average salaries. - Benchmarks for improving the quality of learning in terms of integration and application — because the DQP defines proficiencies in ways that emphasize both the cumulative integration of learning from many sources and the application of learning in a variety of settings. - Enhanced institutional assessment practices and resources — because every learning outcome should lead to and support a provider’s capacity to gather evidence that stated proficiencies are achieved - See: http://degreeprofile.org/read-the-dqp/the-dqp-value-uses-contexts-and-reinforcement/the-need-for-the-dqp/

Learning outcomes Knowledge - Specialized knowledge in a major

- Broad knowledge of disciplines and fields outside the major Cognitive skills - Intellectual skills - Applied and Collaborative Learning - Civic and Global learning - See: http://degreeprofile.org/press_four/wp-content/uploads/2014/09/DQP-grid-download.pdf Non-cognitive skills - Attitudes - Methods - DQP is not so much an assessment tool, it rather is a tool to ensure students achieve the by the institutions set learning outcomes. Sampling DQP when implemented applies to all the students. Applied in Region - Country USA HEI: Not specified Number of users: More than 400 colleges and universities have used DQP Starting year 2011

(23)

Auspices -

Developer Lumina Foundation - Indianapolis, IN: https://www.luminafoundation.org/files/resources/dqp.pdf /

http://degreeprofile.org/ National Institute for Learning Outcomes Assessment (NILOA): http://www.learningoutcomeassessment.org/index.html the Institute for Evidence-Based Change (IEBC): http://www.iebcnow.org/ Cost - Not specified Validity and evaluation of the test - DQP evaluation by National Institute for Learning Outcomes Assessment: http://www.learningoutcomesassessment.org/documents/DQP_impact_study.pdf Final remarks -

(24)

Name test

_{Alternate Uses Test (AUT)}

Short description of main topic Alternate Uses (also known as Guilford's Alternate Uses Test/ Guilford's Alternative Uses Test) is a revised and improved form of the Unusual Uses Test, which was originally designed (Wilson, Guilford, Christensen & Lewis, 1954) to represent an expected factor of "flexibility of thinking" in an investigation of creative thinking. Each item represents the name of well-known object, such as a newspaper, with a statement of its ordinary use. The participant is to list as many as six other, uncommon uses for the object, in the time allowed.

Level 5th grade reading level and higher Field General Main aim & stakes The Alternate Uses Test aims to measure test takers’ creativity. The test developed a standardized way to do so. Learning outcomes Knowledge - Cognitive skills - Non-cognitive skills - Divergent thinking / Creativity evaluation (Originality, Fluency, Flexibility, Elaboration) Attitudes - Methods - Online assessment or via paper-and-pencil. The test takes on average 8 minutes to complete. - Test takers are asked to list as many possible alternative uses for common house hold items (e.g. bricks, paperclips, newspapers, etc). - Acceptable alternatives are: the use is possible for the object; must be different than the provided use; must not be vague or very general use - Unacceptable alternatives are: the use does not qualify as Acceptable - Scoring is based on originality, fluency, flexibility and elaboration. - Sample question: http://www.mindgarden.com/67-alternate-uses#horizontalTab3 Sampling Not specified

Country Not specified HEI: Not specified Number of users: Not specified Starting year Created in 1967 Auspices - Developer Mindgarden (authors: J.P. Guilford, Paul R. Christensen, Philip R. Merrifield, & Robert C. Wilson): http://www.mindgarden.com/67-alternate-uses Cost - Remote online survey license: $2 (minimal purchase of 50) - License to reproduce: $2 (minimal purchase of 50) - Additional costs for manuals. Validity and evaluation of the test - The Alternate Uses Test is widely used in academic publications to measure divergent thinking. Final remarks - References Wilson, R.C., Guilford, J.P., Christensen, P.R., & Lewis, D.J. (1954). A factor-analytic study of creative-thinking abilities, 19(4), 297-311.

(25)

Name test

_{Torrance Tests of Creative Thinking (TTCT)}

Short description of main topic The Torrance Tests of Creative Thinking invites test takers to draw and give a title to their drawings (pictures; figural test) or to write questions, reasons, consequences and different uses for objects (words; verbal test). These instruments have been used for identification of the creatively gifted and as a part of gifted matrices in states and districts in the USA, especially in multicultural settings, and for special populations around the world.

Level The Figural TTCT is appropriate at all levels, kindergarten through adult. The Verbal TTCT is appropriate for first graders through adults Field General Main aim & stakes The Torrance Tests of Creative Thinking provides insight into the creativity of test takers by quantifying figural and verbal creative strengths. A shortened version of the test – The Abbreviated Torrance Test for Adults (ATTA) – has been made by the same developer and is focused on measuring the creativity of adults. Learning outcomes Knowledge - Cognitive skills - Non-cognitive skills - Divergent thinking and creative strengths - Figural: 5 mental characteristics - Verbal: 3 mental characteristics Attitudes - Methods - Paper based test. - Figural: uses three picture-based exercises to assess five mental characteristics: fluency, resistance to premature closure, elaboration, abstractness of titles, originality. It takes 30 minutes to complete. - Verbal: Thinking Creatively with Words uses six word-based exercises to assess three mental characteristics: fluency, flexibility, originality. It takes 45 minutes to complete. Sampling Not specified

Country USA HEI: Not specified Number of users: Not specified Starting year Created in 1966, renormed in 1974, 1984, 1990 and 1998 (Kim, 2006) Auspices - Developer Scholastic Testing Service (STS) (Author: Dr. E. Paul Torrance): http://www.ststesting.com/ngifted.html Cost - TTCT Figural booklet is $7.40 per student booklet. - TTCT Verbal booklet is $9.30 per student booklet. Validity and evaluation of the test - Kim (2006, p. 11) concludes that the TTCT (focus on Figural component) “appears to be a good measure, not only for identifying and educating the gifted but also for discovering and encouraging everyday life creativity in the general population.” - The Torrance Tests of Creative Thinking is widely used in academic publications to measure creativity Final remarks Pro of the test: TTCT is the most widely used test of creativity test it has been translated into more than 35 languages (Kim, 2006) References Kim, K. H. (2006). Can we trust creativity tests? A review of the Torrance Tests of Creative Thinking (TTCT). Creativity research journal, 18(1), 3-14.

(26)

Name test

_{Remote Associates Test (RAT)}

Short description of main topic The Remote Associates Test (RAT) is a test of creative potential. Each RAT question presents three cue words that are linked by a fourth word, which is the correct answer. It measures test takers’ ability to see relationships between things that are only remotely associated. The test is used by psychologists to measure creative potential.

Level Not specified Field General Main aim & stakes Remote Associates Test aims to measure test takers’ creative potential. Learning outcomes Knowledge - Cognitive skills - Non-cognitive skills - Creative potential, creative thinking skills Attitudes - Methods - Online or paper based test. Sampling Not specified

Country USA HEI: Not specified Number of users: Not specified Starting year Created in 1962 Auspices - Developer Sarnoff Mednick Cost RAT does not appear to have been made into a standardized test. Hence, examples are freely available. For example, on: www.remote-associates-test.com Validity and evaluation of the test - The validity of the Remote Associates Test has been challenged. Worthen & Clark (1971) argue that making associations between words it is more a measure of sensitivity to language, than it is of creativity. In response, these authors created the Functionally Remote Associates Test (FRAT). Despite the challenges to its validity, the RAT test continues to be used in academic publications - The Remote Associates Test is widely used in academic publications to measure creativity Final remarks - References Worthen, B. R., & Clark, P. M. (1971). Toward an improved measure of remote associational ability. Journal of Educational Measurement, 8(2), 113-123.

(27)

Name test

_{ETS Proficiency Profile Test}

Short description of main topic The ETS Proficiency Profile Test is a test of college-level skills in reading, writing, critical thinking and mathematics designed to measure the academic skills developed through general education courses, rather than the subject knowledge specifically taught in those courses.

Level Associate through doctoral level Field General Main aim & stakes - The ETS Proficiency Profile was developed to measure and demonstrate the outcomes of general education programs in order to help institutions improve the quality of instruction and learning. - The test aims to help institutions to stay competitive by providing comparative data, promotes curriculum improvements by delivering criterion-referenced scores with actionable score reports, gauge student learning outcomes of traditional, blended learning and distance learning students and meet requirements for accreditation and program funding by measuring and documenting program effectiveness (see: https://www.ets.org/proficiencyprofile/about/vsa/ and

https://www.ets.org/proficiencyprofile/scores/usage/). - Students taking the Standard form can earn a Certificate of Achievement based on how well they perform. This is meant as an incentive to participate and to do well. Learning outcomes Knowledge - Cognitive skills - Reading (interpret the meaning of key terms, recognize the primary purpose of a passage, recognize explicitly presented information, make appropriate inferences, recognize rhetorical devices) - Writing (recognize the most grammatically correct revision of a clause, sentence or group of sentences, organize units of language for coherence and rhetorical effect, recognize and reword figurative language, organize elements of writing into larger units of meaning) - Critical thinking (distinguish between rhetoric and argumentation in a piece of nonfiction prose, recognize assumptions, recognize the best hypothesis to account for information presented, infer and interpret a relationship between variables, draw valid conclusions based on information presented) - Mathematics (recognize and interpret mathematical terms, read and interpret tables and graphs, evaluate formulas, order and compare large and small numbers, interpret ratios, proportions, and percentages, read scientific measuring instruments, recognize and use equivalent mathematical formulas or expressions) - Sample questions: https://www.ets.org/s/proficiencyprofile/pdf/sampleques.pdf Non-cognitive skills - Attitudes - Methods - Paper-and-pencil and online versions of the test are available, thus allowing for on and off campus testing. - The ETS Proficiency Profile is available in both Standard and Abbreviated forms to meet different institutional needs. Questions on the ETS Proficiency Profile are multiple choice and are arranged in blocks of three to eight questions in each skill area. Each section tests the same types of skills. This integrated design ensures that all skill areas are tested equally. - The Standard form of the ETS Proficiency Profile test is intended to provide information about individual students as well as groups of students. It consists of 108 questions (27 for each of the 4 tested skills), divided into two sections of 54 questions each. The two sections may be administered either in a single two-hour testing session or in two separate, one-hour testing sessions. - The Abbreviated form of the ETS Proficiency Profile test is intended to provide information about groups of at least 30 students. It is administered in a single 40-minute testing session. The abbreviated form does not provide information about individual students. The Abbreviated form is

(28)

created by dividing the Standard form into three smaller forms of 36 questions each (9 for each of the 4 tested skills). - Participating institutions can add up to 50 locally authored, multiple-choice questions and nine demographic questions to meet specific program needs. They may also add an essay (30 minutes test time) through which institutions can gain additional insight into students' general knowledge and critical thinking and writing skills. Essays can be scored by software developed by ETS. Sampling Institutions can decide to test a (random) sample of the student population. ETS does give instructions (see: https://www.ets.org/s/proficiencyprofile/pdf/Users_Guide.pdf), but does not guarantee that institutions use the same sampling methods. This is a limitation when making cross-institutional comparisons.

Country USA HEI: Not specified Number of users: Comparative data is available on more than 500 institutions and more than 550,000 students Starting year 2010 (successor of the Academic Profile 1990-2006 and Measure of Academic Proficiency and Progress 2007-2009) Auspices -

Developer Educational Testing Service (ETS): https://www.ets.org/

Website test: https://www.ets.org/proficiencyprofile/about/content/ Cost - The standard form (paper and online) costs: $16.50 (up to 499 test takers), $15.50 (500 to 99 test takers), $14.50 (1000+ test takers) - The abbreviated Form (paper and online) costs: $14.50 (up to 499 test takers), $13.50 (500 to 99 test takers), $12.50 (1000+ test takers) - The optional essay costs $5 per test taker. - Paper based test are sold in packages of 25 - Extra costs for additional ETS reports and off-campus testing. - Costs for taking of the test may be passed on test takers. Validity and evaluation of the test - The validity of the ETS Proficiency Profile has been tested by ETS and others, which established that the test is appropriate to measure the outcomes of general education programs (see: https://www.ets.org/proficiencyprofile/scores/guidelines/ and Liu, et al., 2012). Moreover, students with higher Proficiency Profile scores tend to have gained more course credits and performance is consistent with the skill requirements of their major fields of study, with humanities majors scoring higher than other students on critical thinking and writing and mathematics and engineering students scoring higher on mathematics (Liu, et al., 2012, p. 354). - The ETS Proficiency Profile has been used as a data source used in academic publications. For examples, see: https://www.ets.org/proficiencyprofile/about/research/ Final remarks - References Liu, O. L., Bridgeman, B., & Adler, R. M. (2012). Measuring learning outcomes in higher education motivation matters. Educational Researcher, 41(9), 352-362.

(29)

Name test

_{Mathematics Self-Efficacy Scale}

Short description

of main topic

The Mathematics Self-Efficacy Scale (MATH, also known as MSES) is intended to measure beliefs regarding ability to perform various math-related tasks and behaviours

Level Students in college or attempting a higher degree Field General Main aim & stakes The MATH inventory is based on three areas of math-related self-efficacy expectations: - The solving of math problems, that is, problems similar to those found on standardized tests of mathematical aptitude and achievement (i.e. Dowling's Mathematics Confidence Scale). - Mathematics behaviours used in everyday life, for example, balancing a check book, as represented by the Math Anxiety Rating Scale (MARS; Richardson & Suinn, 1972). - Capability of satisfactory performance in college courses requiring various degrees of mathematics knowledge and mastery. This aspect of behaviour has not previously been used in the study of attitudes toward math but was considered particularly appropriate for examination in a population of college students. The Mathematics Self-Efficacy Scale measures the confidence test takers have in their mathematical ability. Learning outcomes Knowledge - Cognitive skills - Non-cognitive skills - Attitudes - Self-efficacy in math problems, math tasks and mathematics-related college coursework - Sample questions: http://www.mindgarden.com/118-mathematics-self-efficacy-scale#horizontalTab2 Methods - The test has 34 items and takes on average 15 minutes to complete. Sampling Not specified

Country USA

HEI: Not specified

Number of users: Not specified

Starting year 1983 (Hall & Ponton, 2002)

Auspices -

Developer Mindgarden (Authors: Nancy Betz & Gail Hackett):

http://www.mindgarden.com/118-mathematics-self-efficacy-scale Cost - Remote online survey license: $2 (minimal purchase of 50) - License to reproduce: $2 (minimal purchase of 50) - Additional costs for survey hosting and manuals. Validity and evaluation of the test Testing students’ mathematics self-efficacy is grounded on the finding that mathematics self-efficacy is a predictor for performance in mathematics (Langenfeld & Pajares, 1993). Final remarks - References Hall, M., & Ponton, M. (2002). A comparative analysis of mathematics self-efficacy of developmental and non-developmental freshman mathematics students. In meeting of Louisiana/Mississippi section of the Mathematics Association of America. Langenfeld, T. E., & Pajares, F. (1993). The Mathematics Self-Efficacy Scale: A Validation Study. Richard, F.C., & Suinn, R.M. (1972). The Mathematics Anxiety Rating Scale: Psychometric data. Journal of Counselling Psychology, 19(6), 551-554.

Quick scan of learning outcomes assessment instruments: report for the OECD