Do peer reflection and self-reflection increase motivation for one’s
self-improvement: An empirical analysis in an online learning
environment
Tim Kuhlmann
Student number: S2968762
January 20, 2020
Business Administration
MSc Strategic Innovation Management
Supervisor: dr. T.L.J. Broekhuizen
Co-assessor: dr. J.D.R. Oehmichen
ABSTRACT
Building upon Stimulus-Response Theory (SRT), this study investigates the relationships between peer reflection and self-reflection, on the one hand, and the motivation for self-improvement, on the other hand. This study empirically investigates to what degree the quantity of giving feedback, the quantity and constructiveness of receiving feedback and self-reflection of one’s skills influence one’s motivation for self-improvement. This study contributes to the literature on peer reflection and self-reflection and empirically tests these relationships in an under-researched setting: online learning environments. Using a sample of 74 students who performed three presentation tasks in an online learning tool, the results show that peer reflection (i.e., receiving feedback – in terms of quantity not in terms of constructiveness) positively influences and self-reflection negatively influences one’s motivation to self-improve. Theoretical and managerial implications are discussed.
Keywords: Peer reflection, Self-reflection, Motivation for self-improvement, Online learning environment, Stimulus-Response Theory (SRT)
TABLE OF CONTENTS
I. INTRODUCTION ... 2
II. THEORETICAL BACKGROUND ... 5
Conceptual model ... 5
Hypotheses ... 6
Giving feedback ... 7
Receiving feedback ... 8
Self-reflection of mastered skills ... 9
III. METHODOLGY ... 11
Background setting: learning environment and learning objectives ... 11
Data collection ... 11 Data measures ... 12 Data analysis... 15 IV. RESULTS ... 18 Main results ... 18 Robustness checks ... 19
Alternative estimation techniques ... 19
Constructs tested separately ... 20
Alternative valence measurements ... 20
Alternative conceptual model ... 21
V. CONCLUSION AND DISCUSSION ... 22
Theoretical implications ... 23
Managerial implications ... 24
Limitations and opportunities for future research ... 25
I. INTRODUCTION
Peer feedback is an important aspect in enhancing learning and achievement of individuals (Hattie and Timperley, 2007). Feedback refers to information provided by an individual (e.g., teacher, peer, book, parent, self, experience) based on a performance or understanding of another individual (Hattie and Timperley, 2007). However, providing feedback is sometimes limited in its effectiveness because individuals do not always process the feedback received (Hounsell, 1987), or understand or use it (Gibbs and Simpson, 2004). Individuals generally show thirst for peer feedback as they can use it at their advantage to innovate and improve their own performance (O’Donovan, Price and Rust, 2001). Previous studies pointed out that peer feedback influences engagement (Bloxham and West, 2004), critical thinking (Sims, 1989), and motivation to learn (Topping, 2005). Besides, peer feedback related to persons’ tasks might affect, next to their motivation to learn, their improvement efforts (Ames, 1992). Furthermore, giving and/or receiving peer reflection (i.e. feedback from individuals to other similar individuals) might trigger self-reflection and enhance self-evaluation skills (Griesbaum and Görtz, 2010). Self-reflection is a distinctive construct that is crucial in understanding the role of feedback; it is a separate construct that also influences motivation and achievement (McMillan and Hearn, 2008). Without reflection it would be more difficult to motivate oneself to improve oneself. Hence, self-reviewing is an important dimension such that it could affect someone’s ability to change a person’s own thinking and behavior (Bandura, 1986; Griesbaum and Görtz, 2010; McMillan and Hearn, 2008). This study tries to investigate the role of providing peer feedback, receiving peer feedback, and reflecting oneself in triggering motivated behaviors to improve oneself (cf. van Dinther, Dochy and Segers, 2011).
and omnipresence of (digital) reviews might change the impact and role of peer reflection and self-reflection in online contexts.
Extant studies have shown the positive impact of receiving feedback and the positive impact of self-reflection on the motivation and performance of individuals in offline environments. Nevertheless, they have largely ignored the impact of quantity and constructiveness of feedback and self-reflection combined with the motivation for self-improvement in online environments. It seems that the studies done in an offline environment are a bit out of date and not the same compared to studies done in an online environment. Researching in an online environment emphasizes the timeliness of this research. Therefore, this research will be conducted to assess whether findings of traditional settings can be replicated. Table 1 provides an overview of existing empirical studies that investigated the impact of peer feedback given and received (PfGiv and PfRec), the quantity (Quantity) and constructiveness (Construc.) of feedback, self-reflection (Self-R) on any dependent variable (DV), like performance or motivation in an online or offline environment (Learning environment). This overview is provided in order to see to what extent this study is innovative and relevant in comparison to other recently done studies.
TABLE 1: Overview of empirical studies
Source
PfGiv
PfRec Quantity
Construc.
Self-R DV
Learning
environment
Berggren (2013)
X
X
X
Learning and
performance
Offline
Cantillon and Sargeant
(2008)
X
X
X
Performance
Offline
Cauley and McMillan
(2010)
X
X
Motivation and
achievement
Offline
Cleary and
Zimmerman (2004)
X
X
X
X
Motivation to learn
Offline
Corrin and De Barba
(2015)
X
X
X
X
Motivation to learn
Online
Griesbaum and Görtz
(2010)
X
X
X
Individual/
collaborative learning
Offline
Hsia, Huang and
Hwang (2016)
X
X
X
X
Performance and
motivation
Online
Lew and Schmidt
(2011)
X
X
X
Performance
Offline
Li, Xiong, Zang,
Kornhaber, Lyu,
Chung and Suen (2016)
X
X
X
Motivation to learn
Offline
Lizzio and Wilson
(2008)
X
X
Performance and
motivation
Offline
Lundstrom and Baker
(2009)
X
X
X
Learning and
performance
Offline
Phielix, Prins, and
Kirschner (2010)
Shin and Dickson
(2010)
X
X
X
Motivation and
performance
Online
Suen (2014)
X
X
X
Performance
Online
Thomas and Arnold
(2011)
X
X
Satisfaction and
performance
Offline
Zhu, Zhang, He, Kraut
and Kittur (2013)
X
X
Contribution
Online
This study
Motivation for
self-improvement
Online
This study aims to contribute to the literature by considering both directions of feedback (giving and receiving feedback), as well as the content (quantity and constructiveness) and self-reflection (self-reflection of mastered skills). Next, it also investigates a relatively new dependent variable that focuses on one’s willingness to self-improve and master specific skills (motivation for self-improvement) in a relatively under-researched environment (online context). Furthermore, this study investigates the quantity of feedback (both giving and receiving) since there is an apparent lack of research on the number of feedback messages exchanged. It is relevant to take the quantity of feedback (both giving and receiving) into this study because it can serve as a stimulus for individuals to self-improve. Moreover, many studies took motivation to learn, motivation and achievement or learning performance as dependent variables. Still, this study does not consider the standard outcome performance measures but investigates the role of motivation for self-improvement as dependent variable. As argued before, most studies are done in an offline environment since the environment becomes increasingly online. This study is done in a real setting and not in artificially created experimental setting, which therefore creates an advantage. In a real setting, the individuals have no knowledge of the research which makes the validity quite high (Cheng, 2001), and avoids potential demand biases (Gundlach, 1993).
The aim of the study is to answer the following research question: “Do peer feedback (quantity
and constructiveness) and self-reflection influence the motivation for self-improvement in an online learning environment?” To answer this research question, unique data is used from a sample of 74
This study contributes to the knowledge about learning tools with feedback options and a self-reflection of mastered skills format in an online environment. Software training developers and educationalists can use this knowledge in order to design and shape online learning tools for optimizing the motivation for self-improvement. Addressing the motivation for self-improvement leads to new and different insights. The motivation for self-improvement might be one step ahead of great performance/good quality end product and both the directions of feedback (giving and receiving) and the content (quantity and constructiveness) can contribute to this motivation. Focusing purely on the end product might ignore the pure inherent quality, which has a major impact on the end product. This will investigate whether the online training tool works in stimulating continuous improvement.
The structure of this thesis is as follows. The next session addresses the theoretical background and relevant literature to develop the conceptual model and underlying hypotheses. The methodology section provides information about the background setting, data collected, and data analysis. The results section represents the outcomes of the hypotheses.After the results, some robustness checks are going to be done. Furthermore, the conclusion and discussion section answers the research question, and provides the theoretical and managerial implications, as well as the study limitations and opportunities for future research.
II. THEORETICAL BACKGROUND Conceptual model
Figure 1 shows the conceptual model. The conceptual model is grounded in the Stimulus-Response literature which assumes that peer feedback can lead to motivational behavior for self-improvement. The Stimulus-Response Theory states that learning is behavioral change after a specific experience (Gage and Berliner, 1984). Learning is behavioral change after an interaction between stimulus and response, Slavin (1990) also states. An individual might have learned something if these individuals can demonstrate behavioral change. Regarding the Stimulus-Response Theory, the stimulus is the ‘input’ and the response is the ‘output’ (Nazir, 2018). That means that the response is a certain reaction to the stimulus (Nazir, 2018). The type of stimulus does not have to be the same but can vary in terms of quantity and constructiveness. Using the Stimulus-Response Theory in this particular study shows that quantity and constructiveness of feedback are the stimulus here and that the motivation for self-improvement is the response. The stimulus (peer feedback or self-reflection of mastered skills) might trigger a response, namely: the motivation for self-improvement. Hence, the focus of this study is on the motivation for self-improvement rather than the actual performance or quality of the end product. All hypothesized relationships are displayed in Figure 1.
FIGURE 1: Conceptual model
Hypotheses
The goal of feedback is giving information from one individual to another in order to change and improve the performance of the individual in question (Porter, 1982). Providing feedback has some requirements to be useful for individuals who are reviewed; (i) it should be sufficient in detail and frequency; (ii) it should be focused on performance or on personal characteristics; (iii) it should be given in time and (iv) the feedback should be focused on the core of the assignment (Gielen, Peeters, Dochy, Onghena and Struyven, 2010). Individuals give but also receive feedback. These are two directions of feedback and each direction very likely has a different impact on the motivation for self-improvement. Generally, there is a time difference between the receipt and provision of feedback (asynchronous communication) but there is no chronological order. Caffarella and Barnett (2000) argued that both giving and receiving are important in motivation and learning; hence, this study analyzes both. The quantity of feedback messages individuals give and receive can vary and there are two possible ways of providing feedback after one’s performance. The first way for the provider is to give feedback to some of the receiver’s areas in which the individual has a high or low-performance level. This type of feedback creates more consciousness about the skills the individual has or does not have, named praise or criticism which is non-constructive feedback (Thomas and Arnold, 2011). Secondly, if the provider notices points for improvement based on evaluating the receiver’s current performance, constructive feedback is therefore provided in order to suggest how the receiver can increase its performance (Thomas and Arnold, 2011). Afterwards, the receiver might be incentivized to improve its performance based on the feedback.
(Lew and Schmidt, 2011). The concept of self-reflection is quite broad and will be further specified such that this variable is more to the point for this study. Individuals engaging in self-reflection might differ to what extent individuals note that a specific skill is mastered. Mastery is about learning and the development of a particular skill (Ames and Archer, 1988). Therefore, this variable is specified as the self-reflection of mastered skills instead of self-reflection.
Furthermore, motivation is a very broad variable too and requires further specification, what makes it more to the point for this research. Motivation is further specified into the motivation for self-improvement, which can be elaborated based on the article of Abd-El-Fatta (2018). This article states that individuals might experience four kinds of motivations/goals to learn. Individuals can set mastery-goals (improve mastery of the skill) or performance mastery-goals (improve performance with respect to others), in combination with approach-goals (taking pleasure in improving oneself) and avoidance-goals (not liking being worse in something than others are) (Abd-El-Fatta, 2018). Since this research is taking the motivation for self-improvement (and not the performance) into consideration, the focus will be on mastery-goals.
Additionally, extant studies have uncovered relationships between feedback/self-reflection and motivation (Corrin and De Barba, 2015; Cleary and Zimmerman, 2004). Besides, relationships between feedback/self-reflection and performance (Lizzio and Wilson, 2008; Suen, 2014) have also been uncovered. Yet, motivation has also been linked with performance (Rau, Gao and Wu, 2008). Rau, Gao and Wu (2008) argued that encouraging motivation has a positive influence on the actual performance. So, the link between motivation (for self-improvement) and performance (quality end product) is assumed to be positive based on extant literature but is not going to be tested in this study (see dotted arrow in Figure 1). The coming sections will elaborate on the relationships between giving and receiving feedback and self-reflection on the motivation for self-improvement. As previously stated, giving and receiving feedback are both directions of feedback but are distinct from each other, indicating that each variable can impact the motivation to self-improve.
Giving feedback
strongly with the evaluation of another individual’s performance and seems less likely to affect one’s motivation to self-improve.
As shown in Figure 1, the first independent variable is the quantity of feedback that is given. The quantity reflects the number of feedback messages rather than specifying the valence of the feedback that is provided (Carpentier and Mageau, 2014). There is early evidence (in traditional setting) that giving feedback more often provides opportunities for self-improvements (Anderson, Kulhavy and Andre, 1971). The more individuals provide feedback to others, the more those individuals can potentially learn and reflect, which creates a higher chance to find points of improvement for themselves (Lundstrom and Baker, 2009). Feedback providers will evaluate their own “zone of proximal development” more likely than those of their peers. When these individuals have a higher chance of finding points for improvement, the feedback providers are more likely to notice gaps or deficiencies in their own performance. When feedback providers do notice a gap in performance, motivation to learn and self-improve will be encouraged (Locke and Latham, 2006). Thus, providing a higher number of feedback messages increases one’s motivation for self-improvement. Hence, the next hypothesis is formulated:
H1. The total number of feedback messages given increases one’s motivation for self-improvement. Receiving feedback
Receiving feedback is an interaction between two persons, in which the receiver receives information about his/her performance from another individual (Porter, 1982). Lots of individuals see receiving feedback as valuable, and desire to receive feedback in which their progress is confirmed and advice for further improvements is provided (Murdoch‐Eaton and Sargeant, 2012). When people receive feedback, the receivers may realize that a change in their project is feasible in order to improve their performance (Menachery, Knight, Kolodner and Wright, 2006). Feedback providers can set goals for others in order to improve their performance, which may motivate receivers to take action to change (Carless, 2006).
The second variable is the quantity of feedback received as shown in Figure 1. Individuals who receive a higher number of feedback messages experience a higher chance of useful feedback (White, Tiberius, Talbot, Schiralli and Rickett, 1991). Useful feedback is specific, relevant, timeless, clear, has a constructive nature and contains an overall assessment of the performance. More frequent and useful feedback offers more opportunities to learn or improve, more suggestions about how to transform weaknesses into strengths and gives the receiver a higher chance in further improving their performance (Fluckiger, Vigil, Pasco and Danielson, 2010). Being confronted with such deficiencies in one’s own performance and opportunities to improve, individuals will likely display a greater motivation to learn and improve oneself (Locke and Latham, 2006). This results in second hypothesis:
The third independent variable is the constructiveness of the feedback received. Non-constructive and Non-constructive feedback may evoke different reactions to the receiver (Ilgen, Fisher and Taylor, 1979). Non-constructive feedback could be either positive (praise) or negative (criticism). Praise and criticism respectively contain comments about correct things done in performance (Brookhart, 2017) and comments about imperfections in performance (Baron, 1988). Criticism might decrease a person’s motivation to learn (Atlas, Taggart and Goodell, 2004) and also praise might decrease one’s motivation to learn and improve (Mueller and Dweck, 1998). Therefore, non-constructive feedback is assumed to have a negative impact one’s motivation to self-improve.
Constructive feedback contains encouragement, support, direction for personal development and advice on how to resolve the receiver’s problems in an appropriate style, either positive or negative (Hamid and Mahmood, 2010). Therefore, constructive feedback provides information and solutions about how the receiver should change and improve its performance. Constructive feedback contains information about how the receiver can adjust, overwrite, or restructure its work in order to improve quality (Winnie and Butler, 1994). Therefore, individuals receiving feedback with a constructive nature are provided with remarks and suggestions for improvement. Based on the work of Locke and Latham (2006), providing individuals with feedback of a constructive nature contributes to the construction of the solution for the receiver in which the motivation to learn and self-improve will be enhanced. Thus, receiving feedback with a constructive nature has a positive impact on motivation for self-improvement (as compared with non-constructive feedback). It results in the following hypothesis:
H3. Receiving feedback with a constructive nature increases one’s motivation for self-improvement Self-reflection of mastered skills
Self-reflection plays as an important construct in (online) learning tools where individuals subjectively evaluate the interpretation of their own performance and achieved learning objectives. Self-reflection supports individuals in various ways: (i) it helps with reviewing the individual’s performance, (ii) people can apply their strategies in learning and solutions for problems in a different way and (iii) people can link obtained new knowledge to prior understanding in an easier way (Salomon and Perkins, 1989). Reflecting on the mastery of skills helps individuals to gain insights and to understand whether they have attained skills or not (Donovan, 2012). This study focuses on the individual’s mastery of skills and is the fourth independent variable. The self-reflection of mastered skills can either be positive or negative. Besides, it can have a positive or negative influence on motivated behaviors to improve oneself.
(Heckman, 2006). Whereas, individuals with a negative self-reflection of mastered skills do not feel the urge to improve at all, because they feel incompetent for example (Vallerand, Gauvin and Halliwell, 1986).
On the other hand, a negative relationship can exist, as it relates to the opportunities to further improve and self-confidence of individuals. Individuals with a highly positive self-reflection of mastered skills are assumed to be confident, positive about their attained or mastered skills (McFarland, Saunders and Allen, 2009), and feel little incentive to self-improve (Dunlosky and Rawson, 2012). Besides, the large majority of extant studies (McGraw, Mellers and Ritov, 2004; Siegle and Mccoach, 2005; Balduf, 2009; Schwarzer and Scholz, 2000; Mueller and Dweck, 1998) argue that individuals with a positive self-reflection of mastered skills experience less motivated behaviors to improve if they master each skill, think they master each skill (high level of confidence) or if goals are reached. By contrast, individuals with a negative reflection of mastered skills may feel a stronger urge to self-improve (Locke and Latham, 2006). This implies a negative relationship between self-reflection of mastered skills and motivation to self-improve. Figure 2 displays the positive (black arrow) and negative (white arrow) relationship.
FIGURE 2: Self-reflection of mastered skills relationships
Therefore, while both perspectives may be valid, this study hypothesizes that individuals indicating a positive self-reflection (via a high level of confidence) are expected to be less incentivized and motivated to improve. Thus, a positive reflection of mastered skills (from now on: self-reflection of mastered skills) negatively impacts the motivation for self-improvement, resulting in the following hypothesis;
III. METHODOLGY
Background setting: learning environment and learning objectives
This research is executed in an educational setting where students used TrainTool in order to improve their professional communication skills. TrainTool is an online learning tool which gives individuals the opportunity to practice professional communication skills. Peer feedback (both giving and receiving) and self-reflection are built in the TrainTool program, such that those elements will support students with continuously improving. Before using TrainTool, students participate in blended learning via a classroom part in which they get a lecture on the program and how to use TrainTool from teachers. After that, an online part follows in which they use a laptop or a smartphone to record themselves in a digital/online environment. Individuals are thus training themselves online and can try over and over until a certain aspiration level is met. Through video role play, individuals can practice with a job interview, with a professional pitch, with a network interview and will learn how an assessment works in practice. Theory and examples about the particular skill in the TrainTool program will be shown online. When recording, individuals are able to see how and what they did, and they can try again if the outcome is not to their satisfaction. When individuals are satisfied with their last attempt, it becomes their final product and they will upload the video.
But before uploading it, these individuals engage in self-reflection of mastered skills. There are some criteria, like structure or dynamics, they can check if they think that their video complied with these points. After self-reflection, individuals were satisfied with their final product, decided not to change it and uploaded the video. The final product can only be seen by assigned ‘friends’. Each student is instructed to pick three ‘friends’ who can see the final video and are able to give feedback. It also works the other way around; the individual’s ‘friends’ will upload videos and only the individuals with that person as a ‘friend’ are able to see it and give feedback. If the individuals are eager to receive some additional feedback, coaches, teachers or professionals can be asked to receive feedback from additional sources.
Data collection
The data from the online learning environment was used for the empirical analysis, including objective (behavioral) and subjective (self-reflection, feedback) data. These data were gathered, combined and analyzed.
Used data source
instructed to upload a final video for each of these modules with unlimited opportunities to redo the presentation and upload. Students presented their final pitch in class in the last week (week 7). The dataset contains data about the actual feedback given and received, which skills students master and the number of video record attempts per module per student. In line with the University’s and TrainTool’s demands all data were anonymized, such that linking additional personal data (gender, age, educational background, previous student results) to individual students was not possible.
To better understand the research setting, the researcher of this study used additional interviews and a questionnaire. The interviews were used to get a better understanding about the operations of and experiences with TrainTool. The interviews with 2 students contained questions about how TrainTool works exactly, what kind of exercises the students had to do, their personal experiences with the program and potential points for improvement. The interviews were useful to get some prior knowledge about TrainTool and to understand students’ possible behaviors or frustrations. The interviews were not used for this research. The questionnaire was used to see how students experienced working with TrainTool. Unfortunately, the limited response (N<5) did not allow for additional insights.
Data measures
Dependent variable: Motivation for self-improvement
Objective data was used to measure the student’s motivation for self-improvement: the total attempts students recorded and uploaded videos before ending modules. The number of attempts each student did per module and during the entire program were registered. This data is used as proxy to determine the motivation and effort of each student.
Independent variable: Giving feedback
In measuring the feedback students give to other students, the researcher obtained data on the exact online message (written feedback), the number of feedback messages (quantity) and number of characters (length) of feedback provided by each student to other students. Next to that, the researcher evaluated the messages in terms of content to determine the valence of feedback (constructive, non-constructive). Students could potentially select three friends and could give feedback for each of the three modules once which explains the maximum of nine.
Independent variable: Receiving feedback
The data about feedback received by students is similar to the feedback given. It shows the type of feedback received from other students, the number of feedback messages, the length of the message and the content of the messages received. Again, students could choose again a maximum of three friends for each of three modules, which explains the maximum of nine.
Following extant studies, the valence of feedback messages is divided into constructive (Kreitner and Kinicki, 2014) and non-constructive feedback (Brookhart, 2017; Baron,1988) which is given and received by students. These two types are further divided in positive feedforward, negative feedforward, mixed feedforward, praise, criticism and mixed non-constructive. Winnie and Butler (1994) argued that constructive feedback is information and solutions provided to the receiver in which the individual can adjust or change, rethink, restructure or add something to one’s performance in order to improve it. Constructive feedback is divided in three types based on the study of Kreitner and Kinicki (2014): positive feedforward, negative feedforward and mixed feedforward. Non-constructive feedback includes praise, criticism and mixed non-constructive.
Positive feedforward
When providing other students with positive feedforward, the points which are already done in a positive way and the things that can be improved are addressed (Baron, 1988). Some example statements from the positive feedforward provider include: “You did well, but I think that there is room for improvement in…”, “Excellent! But that specific area requires some more attention” and “You make good use of hand gestures, but I think it will boost your performance if…”
Negative feedforward
According to Duffy (2013), negative feedforward involves negative constructive feedback that will address concerns about what is experienced in a negative way and simultaneously involves a request for a specific new behavior to address the concern. An example statement could be: “You did not do a good job there, try to… next time”. Only a request for a specific new behavior (feedforward), without addressing positive or negative remarks, is rated as negative feedforward. It implies that the feedforward is based on a negative experience. An example statement is: “You need to… in the future”
Mixed feedforward
Mixed feedforward involves the provision of feedback that addresses points for improvement with both making references about what went well and/or wrong (positive and negative comments). According to Baron (1988) being constructive contains feedback about things that could be improved in the future. According to and Duffy (2013) being constructive means that a request is made for a specific new behavior.
Praise
Criticism
Baron (1988) argued that criticism is about pointing out the causes of bad performance delivering this in an inappropriate style, which might be associated with pressure and anger. Baron (1988) states that there are certain elements showing that criticism is purely negative (destructive). Giving negative comments about the performance and the result of the job or activity, having a thoughtless tone, possibly making threats to others are elements of criticism (destructive) and involve non-constructive feedback. Some examples of what the feedback provider might say is, “You did not try to…”, “You cannot seem to do that specific thing right” or “You will fail if you do not improve that on a short notice”.
Mixed non-constructive
The last type happens when a message contains both praise and criticism without (positive, negative or mixed) feedforward. Based on the work of Brookhart (2017), Baron (1988) and Duffy (2013), a mix of these types of feedback contains both points about things which are already done in a good way, but also addresses concerns about what is experienced by an evaluator. When both praise and criticism, without any type of feedforward, are simultaneously present in a feedback message, it will be classified as “mixed non-constructive”. The valence of feedback is visually displayed in Table 2 down below.
Table 2: Valence of feedback
Feedback Non-constructive Constructive
Positive Praise Positive feedforward
Negative Criticism Negative feedforward
Mixed Praise and criticism Positive and negative feedforward
Independent variable: Self-reflection of mastered skills
a positive self-reflection of mastered skills). Due to a system malfunction, some of the students were not able to complete module three (modules are noted in the ‘used data sources’ section). For these students, the study only uses the checked criteria scores for the first two modules. The independent variables’ descriptive statistics are shown in Table 3.
Control variable: Extensiveness of feedback messages
Data about the number of characters used in feedback messages (extensiveness of feedback messages) is available, but the average number of characters used in feedback messages per student will act as control variable here. The extensiveness of feedback messages says something about how detailed and elaborated (number of characters) feedback messages are that are given and received by individuals. It can be assumed that the more extensive (i.e., lengthy) the feedback message, the more effort the feedback provider has put in it. Putting significant effort in reviewing the performance of others may have a greater effect on learning and improving writing and reviewing skills compared to shorter peer reviews (Cho and MacArthur, 2011). Individuals feel more appreciated and acknowledged when peers put a significant amount of effort and time in providing other individuals with rich and detailed feedback (White et al., 1991). Acknowledgement creates reciprocal behavior (have the feeling to return a favor) and results in a higher level of motivation for the receiver to learn and improve (Mani, 2002).
Data analysis
The data is subsequently processed and analyzed, using the program SPSS (version 26). Descriptive statistics are presented in Table 3 and the variables are going to be checked for correlations to see if elements of a particular variable influence each other before testing the hypotheses (Table 4). Afterwards, the study analyzes which regression method fit the data best (based on the dispersion), since the dependent variable is a count variable.
model to check the goodness-of-fit, indicated by the R-squared. The R-squared says to what extent the variance in the dependent variable is explained by the independent variables (Gelman, Goodrich, Gabry and Vehtari, 2019). The model with the highest R-squared indicates the best fit. Consequently, one of the following techniques are considered: a Poisson distribution, a Negative binomial distribution with log link function and a Negative binomial distribution with log link function and an estimate parameter. The method that fits the data best, is going to be selected for the regression analysis.
But, before proceeding with the regression analysis, Table 3 represents the descriptive statistics for the dependent variable, each of the independent variables and the control variables. Students were supposed to give feedback to their selected friends. Nevertheless, some students failed to do so; this explains why the minimum number of feedback messages given and received could be 0.00. Next to that, the minimum for the average number of characters given per student and the average number of characters received per student is again 0.00 as they did not provide any feedback to others.
Table 4 provides the correlation matrix. The correlation matrix shows a negative association between self-reflection of mastered skills and the motivation for self-improvement; its effect will be tested in hypothesis 4. A strong correlation exists between the quantity of feedback received and the quantity of feedback given, suggesting that those that provide much feedback also receive more feedback. Furthermore, the number of constructive feedback messages received has strong correlations with the quantity of feedback given and the quantity of feedback received. Lastly, the control variable extensiveness of feedback messages received correlates with the number of constructive feedback messages received and the extensiveness of feedback messages given.
TABLE 3: Descriptive statistics
Variables N Mean Std. Dev. Min. Max.
Total attempts 74 16.73 13.78 1 91
Number of feedbacks given 74 4.26 2.23 .00 9.00
Number of feedbacks received 74 4.26 2.11 .00 9.00
Number constructive feedback received
74 2.69 1.63 .00 8.00
Positive self-reflection of mastered skills
74 0.74 .34 .00 1.00
Average number of characters given per student
74 290.44 173.07 .00 874
Average number of characters received per student
TABLE 4: Correlation matrix
Variables 1 2 3 4 5 6 7
(1) Motivation for self-improvement 1.000
(2) Quantity of feedback given .039 1.000
(3) Quantity of feedback received .141 .739*** 1.000
(4) Number of constructive feedbacks received .015 .585*** .763*** 1.000 (5) Positive self-reflection of mastered skills -.241** .013 .031 -.052 1.000
(6) Average number of characters given
.209* .011 .080 .103 .046 1.000
(7) Average number of characters received
.159 .089 .115 .197* .055 .496*** 1.000
IV. RESULTS Main results
The hypotheses are going to be tested in five different models shown in Table 4. The first model starts with the control variables, and in the subsequent models, every step an independent variable is added. Model 1 starts with testing the relationships between the two control variables ‘average number of characters given’ and ‘average number of characters received’ and the dependent variable ‘motivation for self-improvement’. Model 2 adds the first independent variable ‘quantity of feedback given,’ corresponding to H1. Model 3 adds the second independent variable ‘quantity of feedback received,’ to test H2; Model 4 adds the third independent variable ‘number of constructive feedback messages received,’ corresponding to H3. Model 4 leaves out the number of non-constructive feedback messages in equation because it is redundant (remaining category). Model 5 adds ‘self-reflection of mastered skills,’ corresponding to H4. Because the data are dispersed (variable) for both the Poisson distribution and the Negative binomial distribution with log link function, the results of the Negative binominal with log link function and an estimate parameter are used to test the hypotheses. Also, the BIC and AIC suggest that this method is preferred. Table 4 represents the regression outputs for each model. Considering the AIC, BIC and the R-square (17.5%), Model 5 including all the control and independent variables has the best model fit, and these are used to test the hypotheses.
TABLE 4: Negative binomial distribution with log link and an estimated parameter (NbLe) Motivation for
self-improvement
Model 1 Model 2 Model 3 Model 4 Model 5 Hypothesis
testing (CV) Number of characters given .001 .001 .001 .001 .001 (CV) Number of characters received .000 .000 .000 .000 .000 (H1) Quantity of feedback given .004 -.046 -.034 -.040 ✖ (H2) Quantity of feedback received .082 .143** .153** ✓ (H3) Number of constructive feedbacks received -.112* -.130* ✖ (H4) Positive self-reflection of mastered skills -.481** ✓ Pearson Chi-Square 94.804 95.051 92.943 85.956 83.359
Omnibus test: Chi-Square 4.591 4.534 6.923 9.834 15.336**
R-square .048 .049 .069 .096 .175
AIC 550.462 552.447 552.058 551.147 547.645
BIC 559.679 563.968 565.883 567.275 566.078
Notes: *p<.10, **p<.05, ***p<.01
CV=Control variable, H1=Hypothesis 1, H2=Hypothesis 2, H3=Hypothesis 3, H4=Hypothesis 4,
motivation for self-improvement (β = .153, p = .011). Thus, H2 is supported, since the quantity of feedback received significantly increases one’s motivation for self-improvement. In contrast to H3, the results do not show the expected positive relationship between the number of constructive feedback messages received is and the motivation for self-improvement is not confirmed, but actually a negative relationship (β = -.130, p = .056) that is significant at the less restrictive p < .10 level. Hence, no support is provided for H3, since the number of constructive feedback messages received does not increase but rather decreases one’s motivation for self-improvement. As suggested by H4, the results confirm the expected negative relationship between reflection of mastered skills and motivation for self-improvement by a strong negative regression coefficient (β = -.481, p = .019).
Robustness checks
Alternative estimation techniques
For checking the robustness of the outcomes above, multiple additional checks are conducted to perform a stronger test of the hypotheses tested. The first robustness check will test the hypotheses based on the two additional regression methods. Table 5 represent the results of the Poisson distribution and the Negative binomial distribution with log link function results are discussed afterwards. For both regression methods, Model 5 is the preferred model based on the AIC, BIC and R-squared.
TABLE 5: Poisson distribution (Pd) Motivation for
self-improvement
Model 1 Model 2 Model 3 Model 4 Model 5 Hypothesis
testing (CV) Number of characters given .001*** .001*** .001*** .001*** .001*** (CV) Number of characters received .000* .000* .001* .001** .001*** (H1) Quantity of feedback given .014 -.054*** -.053** -.060*** ✖ (H2) Quantity of feedback received .093*** .162*** .189*** ✓ (H3) Number of constructive feedbacks received -.121*** -.151*** ✖ (H4) Positive self-reflection of mastered skills -.608*** ✓ Notes: *p<.10, **p<.05, ***p<.01
CV=Control variable, H1=Hypothesis 1, H2=Hypothesis 2, H3=Hypothesis 3, H4=Hypothesis 4,
motivation for self-improvement is negative and significant. Besides, the number of constructive feedback messages received still decreases one’s motivation for self-improvement and is significant too. Subsequently, the Negative binomial distribution with log link regression found that H2 and H4 are nonsignificant, and therefore none of the hypotheses are significant. Though, the regression coefficients are admittedly equal, yet the outcomes are nonsignificant ones. Which means that all four hypotheses are not supported. Yet, the Poisson distribution and the Negative binomial distribution with log link regression are not suitable for testing the hypotheses due to the dispersion and high an AIC and BIC.
Constructs tested separately
The second robustness check tests all relationships between the independent and the dependent variable separately – in isolated tests – and is presented in Table 6. It has been found that again the Negative binomial distribution with log link and an estimate parameter has the best fit for each model and is thus reported. These results partially confirm the main results presented in Table 4. H2 is not supported, still H4 is supported.
TABLE 6: Constructs tested separately using NbLe Motivation for
self-improvement
Model 2 Model 3 Model 4 Model 5 Hypothesis
testing (CV) Number of characters given .001 .001 .001 .001 (CV) Number of characters received .000 .000 .000 .000 (H1) Quantity of feedback given .004 - - - ✖ (H2) Quantity of feedback received .047 - - ✖ (H3) Number of constructive feedbacks received -.018 - ✖ (H4) Positive self-reflection of mastered skills -.433** ✓ Notes: *p<.10, **p<.05, ***p<.01
CV=Control variable, H1=Hypothesis 1, H2=Hypothesis 2, H3=Hypothesis 3, H4=Hypothesis 4, Alternative valence measurements
TABLE 7: Alternative valance measurements
Motivation for self-improvement (1) (2) (3) (4) (5)
(CV) Number of characters given .001 .000 .001 .000 .001
(CV) Number of characters received .000 .001 .000 .000 .000
(H1) Quantity of feedback given -.040 -.048 -.052 -.042 -.048 (H2) Quantity of feedback received .023 .055 .114* .087* .080
*Alternative valence measurements* .130* .096 -.056 -.095 .042
(H4) Positive self-reflection of mastered skills
-.481** -.447** -.448** -.493** -.466**
Notes: *p<.10, **p<.05, ***p<.01
CV=Control variable, H1=Hypothesis 1, H2=Hypothesis 2, H3=Hypothesis 3, H4=Hypothesis 4,
The output of Table 7 shows that H1 is nonsignificant for each of the different valence measurements, which indicates the same result compared to the main results represented in Table 4. Table 7 indicates that H2’s positive and significant effect holds when incorporating the measurements positive feedforward (3) and mixed constructive (4) in the regression. A remarkable finding for H3 shows that the number of non-constructive feedbacks received (1) has a positive significant effect on the motivation for self-improvement. This indicates that receiving a higher number of non-constructive feedback messages increases one’s motivation for self-improvement. Therefore, praise and/or criticism might increase a person’s motivation to learn (Bracken, Jeffres and Neuendorf, 2004). This is in contrast with the findings of Atlas, Taggart and Goodell (2004) and Mueller and Dweck (1998) who argued that both criticism and praise have a negative impact on the motivation for self-improvement, Further, the results for H4 presented in Table 4 hold while depending on each of the five different measurements.
Alternative conceptual model
FIGURE 3: Alternative conceptual model
TABLE 8: Negative binomial distribution with log link and an estimated parameter (NbLe)
Self-reflection of mastered skills Model 1 Model 2 Model 3 Model 4 Model 5
(CV) Average number of characters given
-.001 -.001 -.001 -.001 -.001 (CV) Average number of characters
received
.001 .001 .001 .001 .001
Quantity of feedback given -.012 -.014 -.014 -.032
Quantity of feedback received .003 .014 .070
Number of constructive feedbacks received
-.023 -.078
Motivation for self-improvement -.016
Notes: *p<.10, **p<.05, ***p<.01
CV=Control variable, H1=Hypothesis 1, H2=Hypothesis 2, H3=Hypothesis 3, H4=Hypothesis 4,
V. CONCLUSION AND DISCUSSION
unavailability of data during the process) to see if students were more motivated to improve even more after attaining initial skills, the results seem to suggest that a higher confidence or quicker attainment of skills strongly reduces the number of attempts.
Theoretical implications
Initially, this study contributes to the knowledge for further development of learning tools in an online environment and it will help to understand the interlinkages between peer reflection, self-reflection and motivation for self-improvement and performance in the end. Some of these linkages have been investigated but not in an online context. Individuals that are highly motivated to improve, might have a higher chance of improving their own professional communication skills (Rau, Gao and Wu, 2008). Therefore, doing research in motivation for self-improvement is interesting because this might be an essential step to improve performance. This study has used the Stimulus-Response literature and contributes to the peer feedback literature.
In contrast to extant studies (Anderson, Kulhavy and Andre, 1971; Lundstrom and Baker, 2009; Cho and Cho, 2011; Tsui and Ng, 2000) that find that individuals who provide more feedback to others, reflect and evaluate their own zone of proximal development and competencies more, this study does not find that it leads to a greater number of attempts. As this study uses a somewhat different dependent variable (number of attempts instead of actual skill development), it suggests that the link between exerted effort in improving may be less strongly affected than actual skill development. Therefore, it could be that providing a higher number of feedback messages has an effect on increasing one’s own performance rather than the motivation to self-improve in terms of attempts. Being very motivated to provide feedback more frequently could indicate that the provider already has the necessary knowledge. This might result in missing the motivation to self-improve but might increase the actual skill development.
The second hypothesis arguing that the quantity of feedback received has a positive effect on the motivation for self-improvement is supported by this research and also supported by existing literature. Fluckiger et al. (2010) stated that individuals who received useful feedback more frequently are confronted with more opportunities to change and learn from previous performance. Being confronted with opportunities to improve, individuals likely show more motivation to learn and improve oneself (Locke and Latham, 2006). Therefore, the chance for further improving the performance is higher. Consequently, the more feedback an individual receives, the more motivated it is to self-improve.
(Winnie and Butler, 1994) and therefore increases one’s motivation to self-improve. However, the negative significant relationship indicates that one’s motivation for self-improvement decreases when an individual is confronted with a higher number of constructive feedback messages. This can be explained by at least four reasons: first, the receiver understands what s/he did wrong but no longer wants to improve this aspect as it is so obvious (and the learning has actually taken place but is not demonstrated in a future practice). Second, the receiver understands what s/he did wrong but feels incompetent and does not dare to take another attempt to fix the deficiency with the suggested feedback. Third, the student does not understand or disagrees with the feedback received and therefore reacts negatively against the (well-meant) suggestions. Fourth, if students receive very adequate feedback about how to solve issues, it may also be true that students more quickly learn with the help of very useful feedback. Besides, the results of this study show that non-constructive feedback increases a person’s motivation to self-improve which is similar to the results of Bracken, Jeffres and Neuendorf (2004). Moreover, the results are not in line with both Atlas, Taggart and Goodell (2004) and Mueller and Dweck (1998).
The role of self-confidence may play an important role in motivating individuals to further improve. The reflection of mastered skills has a negative effect on the motivation for self-improvement, which suggest that individuals having a positive self-reflection of mastered skills might show a lot of (over)confidence (McFarland, Saunders and Allen, 2009) and are less incentivized to further improve because those individuals do not see where to improve (Dunlosky and Rawson, 2012). These findings are in contrast with Wiegand and Geller (2005) who argued that positive self-reflection creates positive reinforcement during the learning process and therefore motivation to self-improve. Managerial implications
This study is relevant for developers of online learning tools and educationalists who can use this knowledge for shaping and designing the online learning tools in order to enhance motivation for self-improvement. Li et al. (2016) argued that learning in an online environment might enhance efficiency, especially for trainers in large classes. That makes this matter of great business interest for developers of online learning tools and educationalists, but also for users who want to improve their skills and performance in the end. Eventually, the individuals will be stimulated to provide feedback such that they will be motivated for self-improvement in the end.
should obligate the individuals to give feedback to other individuals. Giving a minimum number of feedback messages ensures that feedback receivers get more opportunities for improvement, resulting in more motivation for self-improvement. Furthermore, obligating individuals to provide a minimum number is advantageous. Some of the individuals of this study decided not to give feedback to the friends they selected due to a lack of discipline. Therefore, some students disadvantaged other students since they did not receive the number of feedback messages that was possible.
Further, the results for the number of constructive feedback messages on the motivation for self-improvement remarkably has a negative impact on the number of attempts. This relationship can be explained by the receiver does not improve because it is too obvious, feels incompetent, does not agree with or does not understand the feedback received or the individual already learned from very useful constructive feedback and is therefore not motivated to further improve. As has been shown by the results of this study, the instructions must emphasize that getting constructive feedback must be avoided when the aim is to motivate individually to further improve one’s skills. If individuals are told that it is good but not perfect yet without telling where to improve (non-constructive feedback), the behaviors of individuals can be steered. Like that, individuals will have to find out for themselves how and where they need to improve such that they become more motivated to self-improve.
Concerning the reflection of mastered skills, it turns out that motivation for self-improvement decreases if individuals are very self-confident (a positive self-reflection of mastered skills). It is up to developers and trainers to make individuals aware that they need to have very critical look at their performance and not settle for reasonable performance too quick. In order to increase the motivation for self-improvement, students should not be too confident and should be confronted more with things they can’t do yet, especially if it appears that the student’s actual skills are overrated by themselves. A solution for this might be the implementation of Artificial Intelligence, that can indicate where students make mistakes. Further, Campbell and Lee (1988) argued that self-reflections used developmentally (future-oriented focus) are expected to have a higher impact on the motivation to self-improve compared to evaluative self-reflections (past-oriented focus). Therefore, educationalists and trainers might want to stimulate students to focus on development (setting goals/future-oriented) rather than or next to evaluating (past-oriented) in order to encourage motivation to self-improve.
Limitations and opportunities for future research
similar online learning tools. Additional data including more students and different courses using online learning tools will provide additional results and will increase the reliability and generalizability.
Furthermore, missing a second person (e.g. trainers, coaches or lecturers) for rating the feedback messages regarding the constructiveness (H3) can also be considered as a limitation for this research. The only person who rated the feedback messages was the researcher and a second rater would improve reliability such that determining the type of feedback for each feedback message that is provided is more accurate. Subsequently, regarding H4 in this research, only data on self-reflection for the final uploads had been conducted by the individuals. Therefore, the researcher was not able to see whether the individuals engaged in subsequent uploads after receiving feedback from others or after engaging in self-reflection. Hence, it was not possible to distinguish feedback and self-confidence. Future research could therefore collect longitudinal data to see whether students changed behavior in response to pre-existing self-confidence or feedback. Furthermore, to what extent students strive to achieve goals has not been measured and can be considered as another limitation. According to Erez and Judge (2001) the self-reflection impacts the goal setting motivation. This research did not include to what extent students set goals and it therefore may be interesting for future research to investigate whether their self-reflection leads to a lower goal setting or faster satisfaction with achieving certain goals.
Moreover, due to restricted possibilities to use personal data of individuals, this study could not account for the students’ personal network or personal characteristics. Future research could assess whether the effects of peer-reflection and self-reflection differ across, gender, educational background or personal traits. Therefore, it could be interesting to see whether individuals with a difference in gender, educational background or personal traits provide more or a specific type of feedback. Next to that, future research could assess whether individuals with a difference in gender, educational backgrounds or personal traits engage in self-reflection of mastered skills differently.
VI. REFERENCES
Ames, C., & Archer, J. (1988). Achievement goals in the classroom: Students' learning strategies and motivation processes. Journal of educational psychology, 80 (3), 260.
Ames, C. (1992). Achievement goals and the classroom motivational climate. Student perceptions in
the classroom, 1, 327-348.
Anderson, R. C., Kulhavy, R. W., & Andre, T. (1971). Feedback procedures in programmed instruction. Journal of Educational Psychology, 62 (2), 148.
Artino Jr, A. R., & Jones II, K. D. (2012). Exploring the complex relations between achievement emotions and self-regulated learning behaviors in online learning. The Internet and Higher
Education, 15 (3), 170-175.
Atlas, G. D., Taggart, T., & Goodell, D. J. (2004). The effects of sensitivity to criticism on motivation and performance in music students. British Journal of Music Education, 21 (1), 81-87.
Balduf, M. (2009). Underachievement among college students. Journal of advanced academics, 20(2), 274-294.
Bandura, A., & Cervone, D. (1986). Differential engagement of self-reactive influences in cognitive motivation. Organizational behavior and human decision processes, 38 (1), 92-113.
Baron, R. A. (1988). Negative effects of destructive criticism: Impact on conflict, self-efficacy, and task performance. Journal of Applied Psychology, 73 (2), 199.
Bloxham, S., & West, A. (2004). Understanding the rules of the game: marking peer assessment as a medium for developing students' conceptions of assessment. Assessment & Evaluation in
Higher Education, 29 (6), 721-733.
Bracken, C. C., Jeffres, L. W., & Neuendorf, K. A. (2004). Criticism or praise? The impact of verbal versus text-only computer feedback on social presence, intrinsic motivation, and recall. Cyberpsychology & behavior, 7 (3), 349-357.
Brookhart, S. M. (2017). How to give effective feedback to your students. ASCD.
Burnham, K. P., & Anderson, D. R. (2004). Multimodel inference: understanding AIC and BIC in model selection. Sociological methods & research, 33 (2), 261-304.
Campbell, D. J., & Lee, C. (1988). Self-appraisal in performance evaluation: Development versus evaluation. Academy of Management Review, 13 (2), 302-314.
Carless, D. (2006). Differing perceptions in the feedback process. Studies in higher education, 31 (2), 219-233.
Carpentier, J., & Mageau, G. A. (2014). The role of coaches' passion and athletes' motivation in the prediction of change-oriented feedback quality and quantity. Psychology of Sport and
Exercise, 15 (4), 326-335.
Cheng, C. (2001). Assessing coping flexibility in real-life and laboratory settings: a multimethod approach. Journal of personality and social psychology, 80 (5), 814.
Cho, Y. H., & Cho, K. (2011). Peer reviewers learn from giving comments. Instructional Science, 39 (5), 629-643.
Cho, K., & MacArthur, C. (2010). Student revision with peer and expert reviewing. Learning and
instruction, 20 (4), 328-338.
Cho, K., & MacArthur, C. (2011). Learning by reviewing. Journal of Educational Psychology, 103 (1), 73.
Cleary, T. J., & Zimmerman, B. J. (2004). Self‐regulation empowerment program: A school‐based program to enhance self‐regulated and self‐motivated cycles of student learning. Psychology in
the Schools, 41 (5), 537-550.
van Dinther, M., Dochy, F., & Segers, M. (2011). Factors affecting students’ self-efficacy in higher education. Educational research review, 6 (2), 95-108.
Donovan, S. J. (2012). Improving dynamic decision making through training and self-reflection. Duffy, K. (2013). Providing constructive feedback to students during mentoring. Nursing Standard
(through 2013), 27 (31), 50.
Dunlosky, J., & Rawson, K. A. (2012). Overconfidence produces underachievement: Inaccurate self-evaluations undermine students’ learning and retention. Learning and Instruction, 22 (4), 271-280.
Fleishman, E., Thomson, J. R., Mac Nally, R., Murphy, D. D., & Fay, J. P. (2005). Using indicator species to predict species richness of multiple taxonomic groups. Conservation biology, 19 (4), 1125-1137.
Fluckiger, J., Vigil, Y. T. Y., Pasco, R., & Danielson, K. (2010). Formative feedback: Involving students as partners in assessment to enhance learning. College teaching, 58 (4), 136-140.
Gage, N. L., & Berliner, D. C. (1984). Educational psychological.
Gelman, A., Goodrich, B., Gabry, J., & Vehtari, A. (2019). R-squared for Bayesian regression models. The American Statistician, 1-7.
Gibbs, G., & Simpson, C. (2004). Does your assessment support your students’ learning? Journal of
Teaching and learning in Higher Education, 1 (1), 1-30.
Gielen, S., Peeters, E., Dochy, F., Onghena, P., & Struyven, K. (2010). Improving the effectiveness of peer feedback for learning. Learning and instruction, 20 (4), 304-315.
Griesbaum, J., & Görtz, M. (2010). Using feedback to enhance collaborative learning: an exploratory study concerning the added value of self-and peer-assessment by first-year students in a blended learning lecture. International Journal on E-learning, 9 (4), 481-503.
Gundlach, E. (1993). Demand bias as an explanation for structural change (No. 594). Kiel Working Paper.
Hamid, Y., & Mahmood, S. (2010). Understanding constructive feedback: A commitment between teachers and students for academic and professional development. J Pak Med Assoc, 60 (3), 224-227.
Hattie, J., & Timperley, H. (2007). Feedback power. Review of educational research, 77 (1), 81-112. Heckman, J. J. (2006). Skill formation and the economics of investing in disadvantaged
children. Science, 312 (5782), 1900-1902.
Hounsell, D. (1987). Essay writing and the quality of feedback. Student learning: Research in education
and cognitive psychology, 109-119.
Ilgen, D. R., Fisher, C. D., & Taylor, M. S. (1979). Consequences of individual feedback on behavior in organizations. Journal of applied psychology, 64 (4), 349.
Kokonendji, C. C. (2014). Over- and underdispersion models. Methods and Applications of Statistics in
Clinical Trials, 2, 506-526.
Kreitner, R., & Kinicki, A. (2014). Organizational Behavior.
Ku, D. T., & Chang, C. S. (2011). The effect of academic discipline and gender difference on Taiwanese college students' learning styles and strategies in web-based learning environments. Turkish
Online Journal of Educational Technology-TOJET, 10 (3), 265-272.
Lew, M. D., & Schmidt, H. G. (2011). Self-reflection and academic performance: Is there a relationship? Advances in Health Sciences Education, 16 (4), 529.
Li, H., Xiong, Y., Zang, X., L. Kornhaber, M., Lyu, Y., Chung, K. S., & K. Suen, H. (2016). Peer assessment in the digital age: a meta-analysis comparing peer and teacher ratings. Assessment
& Evaluation in Higher Education, 41 (2), 245-264.
Locke, E. A., & Latham, G. P. (2006). New directions in goal-setting theory. Current directions in
psychological science, 15 (5), 265-268.
Lundstrom, K., & Baker, W. (2009). To give is better than to receive: The benefits of peer review to the reviewer's own writing. Journal of second language writing, 18 (1), 30-43.
Mani, B. G. (2002). Performance appraisal systems, productivity, and motivation: A case study. Public
Personnel Management, 31 (2), 141-159.
McFarland, L., Saunders, R., & Allen, S. (2009). Reflective practice and self-evaluation in learning positive guidance: Experiences of early childhood practicum students. Early Childhood
Education Journal, 36 (6), 505-511.
McGraw, A. P., Mellers, B. A., & Ritov, I. (2004). The affective costs of overconfidence. Journal of
Behavioral Decision Making, 17 (4), 281-295.
McMillan, J. H., & Hearn, J. (2008). Student self-assessment: The key to stronger student motivation and higher achievement. Educational Horizons, 87 (1), 40-49.
Menachery, E. P., Knight, A. M., Kolodner, K., & Wright, S. M. (2006). Physician characteristics associated with proficiency in feedback skills. Journal of general internal medicine, 21 (5), 440-446.
