Editorial article for the special issue on data-based decision making
around the world: from policy to practice to results
Kim Schildkampa*, Melanie Ehrenaand Mei Kuin Laib
a
University of Twente, Enschede, The Netherlands;bUniversity of Auckland, Auckland, New Zealand
This editorial article briefly examines the importance of data-based decision making. It discusses the definition as well as rationale for data-based decision making, its purposes, the use of data at different levels of the educational system, and possible promoting and hindering factors of effective data use. It also examines the effects of data use, intended effects (e.g., increased student achievement), as well as unintended consequences (e.g., cheating with tests). We end with suggestions for new research priorities.
Keywords: data-based decision making; data systems; professional learning communities; leadership; district policies; strategic use; school improvement; accountability
Introduction
Data-based decision making (hereafter called data use) continues to receive ample attention in education, partly because of the increasing emphasis on holding schools accountable for the education they provide, and because of recent evidence implicating data use in improvements in student achievement (Carlson, Borman, & Robinson, 2011). School leaders and teachers can use data (e.g., information on the functioning of their school) to change teaching, address existing (ineffective) programs in their schools, and improve the functioning of the school in terms of increased student achievement (Carlson et al., 2011; Lai, McNaughton, Timperley, & Hsiao, 2009; McNaughton, Lai, & Hsiao, this issue; Wohlstetter, Datnow, & Park, 2008).
This special issue brings together the current research conducted on data use across multiple countries using a wide range of methodologies from qualitative to quantitative studies. Some of these studies are ‘‘best practice’’ studies, where effective data use has led to improvements in student learning. Others provide insight into challenges in the district and school organizational context. Each study draws on recent research and literature in the field. Together, they take the current line of research a step further by studying different facets of data use across a range of different international contexts.
*Corresponding author. Email: k.schildkamp@utwente.nl Vol. 23, No. 2, June 2012, 123–131
ISSN 0924-3453 print/ISSN 1744-5124 online Ó 2012 Taylor & Francis
http://dx.doi.org/10.1080/09243453.2011.652122 http://www.tandfonline.com
The studies in this issue focus on different levels of data use, such as data use at the district, school, and classroom levels, and how these levels interact (Lee, Seashore Louis, & Anderson, this issue; Levin & Datnow, this issue; Wayman, Jimerson, & Cho, this issue). The studies focus on the use of different types of data, including assessment data, but also data such as teacher observations. Furthermore, this special issue considers both the positive effects of data use (e.g., school improvement in terms of increased student achievement) (McNaughton et al., this issue) and the potential unintended consequences (e.g., referring students to special education so that the average assessment results increase) (Ehren & Swanborn, this issue). Finally, two of the articles in this special issue focus also on the constraints and enablers of data use, such as the role of the school leader in data use (Levin & Datnow, this issue; Schildkamp, Rekers-Mombarg, & Harms, this issue).
There is great potential for data to support and improve education. But for many educators, the promise of improvement through the use of data is constrained and even thwarted by a wide array of factors, such as educators’ context (e.g., school leader support for data use, teacher/school leader collaboration, and vision and goals for data use), data characteristics (e.g., access to valid, reliable, and timely data), and user characteristics (e.g., the skills and knowledge to effectively use data and attitude towards data use) (see, e.g., Ikemoto & Marsh, 2007; Kerr, Marsh, Ikemoto, Darilek, & Barney, 2006; Levin & Datnow, this issue; Schildkamp et al., this issue; Schildkamp & Kuiper, 2010; Schildkamp & Teddlie, 2008, Schildkamp & Visscher, 2009; Visscher, 2002; Wayman, 2005; Wohlstetter et al., 2008). Too often, we are unaware of how these factors affect the use of data for the improvement of schools.
The articles in this special issue include examples on data use from the USA, The Netherlands, and New Zealand. These studies are by no means representative of all research on data use in these countries, nor of data use globally. There are many other examples of data use around the world that are not included in this special issue, such as data use in South Africa, Flanders, England, and Canada (Schildkamp, Lai, & Earl, in press). Rather, the studies in this special issue were selected to represent particular aspects of using data and to highlight particular policy issues influencing data. Together, these studies provide a strong picture of the state of affairs in data use across a range of international jurisdictions. These examples provide the reader with an opportunity to compare and contrast the findings towards insights that transcend any of the individual studies.
Different perspectives on data-based decision making
Data use by teachers and school leaders is often narrowly defined to the use of assessment data, particularly data from standardized assessments or national tests that are quantifiable. This view is reinforced by some policy-makers and researchers who focus on aggregated standardized test results as the primary source of data about schools and disregard other forms of data such as records of the quality of instruction in classrooms. In this special issue, we use a broader definition of data, which encompasses the multiple types of data that teachers and school leaders need for decision making. Ehren and Swanborn (this issue) distinguish between the use of data to change cognitive outcomes (student achievement), non-cognitive outcomes (absence or graduation rates), and educational practices (teaching and learning in the
school). In addition to these outcome and process data, input data, such as the demographics of the student population, are also relevant sources for decision making (Ikemoto & Marsh, 2007).
In short, data can be defined as all the relevant information that teachers and schools need for decision making (e.g., relevant information on students, schools, school leaders and teachers, such as assessment results and teacher surveys) and can include both qualitative and quantitative data. Data, however, are rarely used in the form in which they are presented (Cousins & Leithwood, 1993), as data on their own provide no judgment or interpretation and no basis for action. Only after data are interpreted and transferred into information (through contextualizing, categorizing, calculating, connecting, and summarizing the data) can they be used as a basis for decision making (Davenport & Prusak, 1998).
Rationale for data-based decision making
High expectations have risen with regard to the potential contribution of data use to improvement of schools. Many districts, school boards, principals, and teachers at different levels have put a lot of time and energy into making data accessible to schools and teachers and into using data, in training schools and teachers to use data for improvement, and in requiring schools and teachers to collect and publish data. The notion of data use is that good teaching is based on high quality data and tested assumptions about the needs of children to learn effectively (Robinson & Lai, 2006). Too often, teachers make decisions based on intuition and limited observations (Ingram, Louis, & Schroeder, 2004). Valuable time and resources are lost when, as a result of faulty intuitions, the implementation of new practices or curricula do not coincide with the needs of the students (Earl & Katz, 2006). The use of data is expected to prevent such wastage of resources.
Also, schools are dynamic institutions with changing school populations and new curricula to accommodate. Staff, students, and contexts change, so that what used to be effective and worked at one point in time may become ineffective when students change or new teachers come into the school (Lai, McNaughton, Timperley, et al., 2009). Teachers and school leaders need data to help monitor their constantly changing environment – what has changed and to what extent are curriculum aims and staff and student needs being met – so that they are able to react in a timely and evidence-based manner to uncover and solve new problems.
Data use is also considered to be an important aspect of professional development for teachers and principals. Recent research suggests that professional development is highly effective in improving student learning when school leaders and teachers gather data about their own students and their own practices and use these data to identify and solve the problems they are facing in their classrooms and schools (Timperley, Wilson, Barrar, & Fung, 2007).
Finally, and most importantly, data use can lead to sustainable improvements in student learning and achievement. Growing evidence exists that data use can lead to school improvement in terms of increased student achievement levels (Campbell & Levin, 2009; Carlson et al., 2011; Cawelti & Protheroe, 2001; Lai, McNaughton, Amituanai-Toloa, Turner, & Hsiao, 2009; Lai, McNaughton, Timperley, et al., 2009; McNaughton et al., this issue; Timperley & Parr, 2009; Wohlstetter et al., 2008). A recent synthesis of the literature on professional learning that makes a difference to student learning and achievement found that schools that used data to inquire into
the effectiveness of their teaching and school practices made significant improve-ments in achievement (Timperley et al., 2007).
Unintended consequences of data use
The results and rationales discussed previously present a very positive picture and seem to indicate that using data for decision making is the new panacea to the longstanding problems of low student performance, lack of improvement in schools, and the small (or lack of) effect of many improvement intervention programs. In this special issue, we, however, explicitly address the fact that using data in itself is not necessarily positive. Moreover, data use can also have unintended consequences if used in an incorrect manner or if misused to, for example, encourage low-performing students to drop out. Ehren and Swanborn in this issue describe how a high-stakes accountability context of data use may lead to harmful consequences for student learning or inflated test scores when schools use data to improve their status on the accountability indicators instead of improving teaching and learning in the school.
Summary of the articles
The first articles focus on data use in the USA, and primarily data use at the district level. The first article by Lee, Seashore Louis, and Anderson, examines the relationship between district leadership around data use and teachers’ assessments of their work environments in the USA. Results of this study suggest that the development of networks across schools has a positive relationship with instruction and subsequent learning, while district emphasis on learning targets and data use has a negative relationship with instruction.
Wayman, Jimerson, and Cho also studied the role of USA districts in the use of data, but from a different perspective of comparisons across districts. Their results suggest that districts experience very similar problems with regard to data use, including a lack of common understanding about district vision and data use, a lack of proper technology, the need for better principal leadership, and lack of support in terms of time and professional development.
Levin and Datnow, in their USA article, show how districts’, principals’, and students’ actions influence each other to co-construct the success of data use at the school level. The results of their study show, among other things, that principals can facilitate data use, for example, by formulating measurable goals that are specific to the needs of the school, by providing structures to support data use (e.g., structured time for data discussions, data analysis protocols), and by creating a climate of trust, collaboration, and data use (e.g., professional learning communities).
The study by McNaughton, Lai, and Hsiao, in contrast, focuses on the impact of interventions, where data use is an intervention focus, on student achievement. Their quasi-experimental study conducted over 5 years across different school contexts in New Zealand showed that when teachers and school leaders use data (assessment data and classroom observation data) to improve their teaching practices, there is a consistent and significant increase in student achievement.
The article by Schildkamp, Rekers-Mombarg, and Harms addresses the issue of how schools use two different types of (similar) student achievement data in The Netherlands. The article investigates the discrepancy between the results of school-based examination and central examinations for different student groups (e.g., school-based
on ethnicity) and whether schools analyze these discrepancies and take measures to decrease them and thereby improve their education. This article also discusses the factors that can either hinder (e.g., lack of data use knowledge and skills) or promote (school leader support for data use and teacher collaboration) effective data use within the school.
The final article addresses potential unintended consequences of data use and the conditions promoting such unintended consequences. Ehren and Swanborn describe how schools are inclined to strategically use data when these data are part of a high-stakes accountability context.
Contributions of the articles to the field of data-based decision making
The articles in this special issue provide us with valuable new insights into data-based decision making. The articles show how data use in schools is influenced by district policy (Lee et al., this issue; Levin & Datnow, this issue; Wayman et al., this issue) and by school organizational characteristics (Schildkamp et al., this issue). Having a clear vision and goals for data use and providing school staff with professional development on how to use data appear to be essential for effective data use (Levin & Datnow, this issue; Schildkamp et al., this issue; Wayman et al., this issue). However, districts’ use of targets and data (e.g., district assistance to schools or teacher professional development based on performance data) seems to generate certain negative pressures on teachers within the context of accountability (Lee et al., this issue). This does not imply that districts should avoid setting policies about learning targets or expectations that schools will use data for decisions. But it does imply that districts should try to reduce the amount of external pressure for data use as a strategy for improving student learning, as this might unintentionally reduce schools’ motivation to use data. Rather, once these policies (with regard to learning targets and data use) and practices are in place, the district should consider how to use sufficient (rather than undue) pressure as well as adequate support for schools in carrying them out, for example, by developing and encouraging networked learning communities and by providing professional development (Lee et al., this issue; Wayman et al., this issue).
The articles show that at the level of the school and the classroom, data use is not without problems either. The article by Schildkamp et al., for example, point to several factors that can either promote or hinder effective data use within schools. In most countries, for example, teacher training colleges do not pay attention to data collection, data analyzing, data interpreting, and data use for improvement. Therefore, we cannot expect our educators to be able to use data without proper professional development. Several studies in this issue point at the importance of providing schools with some form of professional development or training (Lee et al., this issue; Levin & Datnow, this issue; Schildkamp et al., this issue; Wayman et al., this issue). Moreover, technology and having the time to use data play an important role. Also, school leadership for data use is essential. Schools will not use data if data use is not advocated, stimulated, and encouraged by the school leader (Levin & Datnow, this issue). The principal has an essential role in advocating data use, and tight connections between the principal and the teachers help enable reform. Finally, collaboration in using data is essential. This can be collaboration between teachers or between teachers and school leaders from one school. It can also be very effective to organize collaboration between schools, as the study conducted by
Lee et al. show that ‘‘within-district networked community, as assessed by principals, has a significantly positive effect on teachers’ reported instructional behaviors (focused instruction), that is, in turn, significantly associated with student achievement’’. McNaughton et al. (this issue) similarly show the importance of using data in a professional learning community which can contribute to improvements in achievement.
Data use differs for the different levels of the education system. School leaders, for example, use different types of data for different purposes than teachers do. Wayman et al. (this issue) illustrate the importance of coming to a shared vision on data use. The results of their study show that in most districts a vision exists, but that each district struggled to communicate vision throughout the organization authentically. An articulated and clear vision exists usually at the upper levels of leadership but becomes scattered as it filtered down the chain; in particular, teachers in their study often were not able to express the school’s vision. What is also worrying is that, although there seems to be agreement on the importance of data use (albeit an exact vision and goals are often lacking), the actual use of data in schools seems to be very limited (Schildkamp et al., this issue) unless support is provided for its use (McNaughton et al., this issue).
Schools can also use data strategically in ways that do not benefit students or certain student groups (Ehren & Swanborn, this issue). Schools may use data in undesirable ways when they, for example, select only easy-to-use data for change and ignore data that involve more complicated long-term improvement such as developing teaching strategies for specific groups of students. Strategic data use also involves schools distorting data to improve their status on the accountability indicators. They, for example, teach to the test or exclude students from the test (Ehren & Swanborn, this issue). This type of use is considered to be unintended as schools ignore opportunities to improve and the data do not reflect the actual performance of students and, as a result, the outcome of schools.
Finally, this special issue shows that, when implemented properly, data use can be effective in improving education in terms of improving student achievement. The study conducted by McNaughton et al. (this issue) shows that the use of data in intervention models can be successful in raising student achievement across multiple school contexts, from rural to urban schools and from high- to low-achieving student populations. Using statistical growth modeling, the study shows how achievement has increased more than nationally expected progress in multiple contexts, therefore strengthening the evidence that interventions where use of data is central are effective in improving student achievement. In each case, the growth models estimated accelerated gains of between 3 to 4 months’ additional progress per year. In the intervention, achievement and teacher observation data were used to develop context-specific interventions and as part of discussions within professional learning communities. The intervention, replicated across different settings, also indicated some differential effects across contexts, thereby strengthening the need for interventions to continue to use data to continually refine the programs to meet student needs.
Directions for further research
This special issue points to new areas of research. Firstly, it is important to further study what types of data are being used, how data are used at the different levels of
the system (e.g., district level, school level, classroom level), and how these levels interact. As previously stated, actors at different levels of the system may need or want different types of data to use in different manners. A school leader, for example, may be interested in the overall achievement levels of the school to base policy decisions on. A teacher would be more interested in the achievement levels of his or her students and is more likely to change instructional strategies based on this type of data. It is also important to study how different types of data travel through these different levels of the system (e.g., the data paths; how data find their way to school leaders and teachers) and if actors of the different levels interact with each other concerning the use of data. The article by Levin and Datnow in this issue can serve as a starting point for thinking about these issues.
Almost all articles point to the importance and potential of data use but also conclude that data use practices are limited in most schools. This raises the issue of what preconditions and support need to be in place at the district, the school board, the school, and the classroom levels in order to increase effective data use practices. Some of the studies in this issue reveal important facilitating factors and barriers such as the importance of external support mechanisms for schools, but greater detail is needed on the needs at different levels (e.g., what support do district leaders need to help teachers effectively use data) and on how the needs at the different levels interact (e.g., how do the district level needs influence the school level use of data, and vice versa).
What is critical also is how schools interpret these supports and barriers from the different levels and how aligned and coherent these different types of support are. Why do some schools choose not to use data for improvement purposes even when they have access to relevant data, professional development, and other support designed to enable data use? Could this reluctance to use data be due to the mixed messages on the importance of data use across different parts of the education system? For example, in most countries, teacher colleges do not pay attention to use of data for improvement, which could mean that they do not value the use of data as much as some schools and districts do. Thus, teacher trainees are graduating without the skills in analyzing and using data and without the understanding of its relevance to their daily teaching practice.
More comparative studies are also needed on how a country’s policy and practices, such as district policies, vision, and goals for data use, characteristics of the data that are available (including technology), and user characteristics (e.g., skills to use data) influence data use practices at the different levels of the system. For example, the New Zealand study (McNaughton et al., this issue), which found that using data as part of interventions improved student achievement, took place in the context of a non-hierarchical education system which does not have high-stakes accountability systems at primary schools (e.g., no national testing at primary schools).1The lack of ‘‘national pressure’’ may have influenced the uptake of data use and focused the use of data on learning and improvement rather than accountability goals. It may be that high-stakes education accountability reduces the use of data for improvement purposes because the use of data for improvement purposes is subsumed by, or perhaps conflated with, accountability purposes. A strong accountability context may, for example, lead to schools fixating on a small range of accountability targets or schools distorting data to improve their status on the accountability measurement. Whilst the differential educational contexts of the articles in this issue can be compared, direct comparative studies are required to
precisely understand the influence of differing educational contexts on effective data use.
Finally, this special issue shows promising evidence that data use (as part of interventions) can lead to school improvement in terms of increased student achievement. However, further empirical studies using robust methodological designs over a longer period of time and in different educational contexts are required to confirm the benefits of using data on student outcomes. In addition, it is important to study whether the impact of data use on student outcomes is most effective within the context of interventions, or whether data use, if part of everyday teacher practice, is equally efficacious. It is also important to problematize what is considered ‘‘effective use of data’’. For example, are data only ‘‘effectively used’’ if they lead to improvements in student achievement outcomes, and if so, is this too narrow a definition?
In sum, this issue highlights the promise and pitfalls of using data and directs us to new research areas. There is a lot we have learned about effective data use, but there is also a lot we do not know yet. As data use is perceived as ‘‘the way forward’’, more effort is needed in studying how we can support schools in effectively using data for school improvement and, consequently, how to measure the effects of these support interventions in schools.
Note
1. This situation has recently changed with the introduction of National Standards in 2010.
References
Campbell, C., & Levin, B. (2009). Using data to support educational improvement. Educational Assessment, Evaluation and Accountability, 21, 47–65.
Carlson, D., Borman, G.D., & Robinson, M. (2011). A multistate district-level cluster randomized trial of the impact of data-driven reform on reading and mathematics achievement. Educational Evaluation and Policy Analysis, 33, 378–398.
Cawelti, G., & Protheroe, N. (2001). High student achievement: How six school districts changed into high-performance systems. Arlington, VA: Educational Research Service. Cousins, B.J., & Leithwood, K.A. (1993). Enhancing knowledge utilization as a strategy for
school improvement. Knowledge: Creation, Diffusion, Utilization, 14, 305–333.
Davenport, T.H., & Prusak, L. (1998). Working knowledge. How organizations manage what they know. Boston, MA: Harvard Business School Press.
Earl, L.M., & Katz, S. (2006). Leading schools in a data-rich world. Harnessing data for school improvement. Thousand Oaks, CA: Corwin press.
Ikemoto, G.S., & Marsh, J.A. (2007). Cutting through the data-driven mantra: Different conceptions of data-driven decision making. In P.A. Moss (Ed.), Evidence and decision making(pp. 105–131). Malden, MA: Wiley-Blackwell.
Ingram, D., Louis, S.K., & Schroeder, R.G. (2004). Accountability policies and teacher decision making: Barriers to the use of data to improve practice. Teachers College Record, 106, 1258–1287.
Kerr, K.A., Marsh, J.A., Ikemoto, G.S., Darilek, H., & Barney, H. (2006). Strategies to promote data use for instructional improvements: Actions, outcomes, and lessons from three urban districts. American Journal of Education, 112, 496–520.
Lai, M.K., McNaughton, S., Amituanai-Toloa, M., Turner, R., & Hsiao, S. (2009). Sustained acceleration of achievement in reading comprehension: The New Zealand experience. Reading Research Quarterly, 44, 30–56.
Lai, M.K., McNaughton, S., Timperley, H., & Hsiao, S. (2009). Sustaining continued acceleration in reading comprehension achievement following an intervention. Educational Assessment, Evaluation and Accountability, 21, 81–100.
Robinson, V., & Lai, M.K. (2006). Practitioner research for educators: A guide to improving classrooms and schools. Thousand Oaks, CA: Corwin Press.
Schildkamp, K., & Kuiper, W. (2010). Data-informed curriculum reform: Which data, what purposes, and promoting and hindering factors. Teaching and Teacher Education, 26, 482–496.
Schildkamp, K., Lai, M.K., & Earl, L. (in press). Data-based decision making in education: Challenges and opportunities. Dordrecht, The Netherlands: Springer.
Schildkamp, K., & Teddlie, C. (2008). School performance feedback systems in the USA and in The Netherlands: A comparison. Educational Research and Evaluation, 14, 255–282. Schildkamp, K., & Visscher, A.J. (2009). Factors influencing the utilisation of a school
self-evaluation instrument. Studies in Educational Evaluation, 35, 150–159.
Timperley, H., & Parr, J. (2009). Chain of influence from policy to practice in the New Zealand literacy strategy. Research Papers in Education, 24, 135–154.
Timperley, H., Wilson, A., Barrar, H., & Fung, I. (2007). Best evidence synthesis iterations (BES) on professional learning and development. Wellington, New Zealand: Ministry of Education.
Visscher, A.J. (2002). A framework for studying school performance feedback systems. In A.J. Visscher & R. Coe (Eds.), School improvement through performance feedback (pp. 41–72). Lisse, The Netherlands: Swets & Zeitlinger.
Wayman, J.C. (2005). Involving teachers in data-driven decision making: Using computer data systems to support teacher inquiry and reflection. Journal of Education for Students Placed at Risk, 10, 295–308.
Wohlstetter, P., Datnow, A., & Park, V. (2008). Creating a system for data-driven decision-making: Applying the principal-agent framework. School Effectiveness and School Improvement, 19, 239–259.
