5.1 Defining a Task Difficulty Index
First we analyzed which of the indicators differs from the mean difficulty. In 16 out of 60 tasks we could observe that authors’ opinion on a task difficulty is quite different from the other factors. What we found astounding was that it was the factor contes-tants’ opinion that was closest to the mean difficulty. It differed least and also showed 96 J. Vaníček
the smallest variance. On the contrary the indicator that differed most was solving time (which means that according to our method it gives least information on a task difficulty).
In the next step, pairs of indicators of a task difficulty were compared to analyze to what extent they provide similar results. The greatest difference between all ten com-pared pairs of indicators was in the pair authors’ opinion – success rate. Indicators with most similar results were no answer – contestants’ opinion. This would mean that contestants mark as most difficult those tasks they skipped in the test and did not answer.
However, the criterion does not seem to be reasonable enough to use it for determining as the most difficult the task that put off most contestants. It is also quite interesting that the contestants’ opinion states a task difficulty more accurately than authors’ opinion.
We studied variance between different factors. In some tasks one indicator was significantly distant from the others and its omission would have resulted in a signif-icantly smaller variance. The values that were left out by this in the greatest number of tasks was the indicator authors’ opinion but surprisingly also the indicator success rate, which has so far been taken as the most dominant indicator of a task difficulty. The indicator that was excluded least often was the indicator no answer.
Based on results of the above described analysis, two most suitable indicators were used for definition of index of a task difficulty: incorrect answer ratio in answered tasks (signaling real difficulty) and no answer ratio (signaling perceived difficulty) according to Pozdnyakov [9]. These indicators described difficulty well together so their values were added. As the reason for not answering a task might be lack of time when solving the test causing that the contestant did not get to the task at all, this indicator does not give a task difficulty unequivocally and thus it is divided by two in the calculation.
Index of difficulty from absolute number of answers is expressed by the formula i¼ ð1 n cÞ=ð1 nÞ þ n=2
where i– is index of a task difficulty, c – correct answers ratio, n – no answer ratio.
This index was used by factors impact calculations on all of 283 tasks.
We were interested in how the two basic components of index of difficulty change with age of contestants. Graph on the left in Fig.2 shows a huge difference in the
Fig. 2. Relations of unanswered tasks ratio and difficulty index to a contestant’s age What Makes Situational Informatics Tasks Difficult? 97
number of test items with no answer at lower and upper secondary school level. The number increases abruptly at the age of 15 and almost doubles. This is in contrary to real difficulty of tasks (on the right). The reason for this might be that the contestants start to perceive tasks as difficult or that they understand the rules of the contest better and do not want to run the risk of losing points.
5.2 Factors Affecting a Task Difficulty
Proved Impact on A Task Difficulty. Statistical calculations prove impact of the followingfive factors:
– Presence of formal description increases a task difficulty. An informatics task is more difficult if it includes a code, programme, formula, chain of texts conveying some meaning. It can be inferred that the need to grasp formal description is connected to abstraction, generalization and non-trivial mental operations that the task demands from the contestant.
– Structural tasks are more difficult. In contrast e.g. to algorithmization or informa-tion comprehension, to looking for a structure of objects and phenomena, orien-tation in structures (including traditional structures as trees or containers) represent a difficulty for the contestants.
– Presence of optimization increases a task difficulty. Contestants face problems in the process of optimization when they are to select the best choice from a set of possible choices (which is often large).
– Demands of reading comprehension increase a task difficulty. It is very interesting to see that neither the length of the text, nor its difficulty have so much impact as demands on reading comprehension itself. More difficult are those tasks in which the data needed for its solution are harder to be detected in the text and remembered.
– Interactivity of a task lowers its difficulty. However, it depends on the type of interactivity. Of all the sub-factors into which interactivity factor was divided, only the types Drag and drop and Game actually decrease a task difficulty.
In all cases the presence of the particular factor in the task assignment brought about a significant change in the mean value of the difficulty index.
Unproven Impact on A Task Difficulty. Let us now present some potential factors whose impact on a task difficulty was not proved.
– Presence of an illustrative picture or example does not decrease a task difficulty.
We explain this by the fact that authors decide to use such a picture or example only if theyfind the assignment difficult. Thus these aids are not likely to be present in easy tasks.
– The need to work with a diagram, graph, map, scheme has no impact on a task difficulty. This could imply that children have no problems when working with visual data or that authors of tasks are very careful when making the decision whether to use these tasks in younger categories.
98 J. Vaníček
– Entering text into a textbox (surprisingly) has no significant impact to difficulty. But we have to keep in mind that we have used this type of answer very rarely.
– A negative question has no impact on a task difficulty. This might be caused by the fact that this kind of question is always alerted to by its format or other warning to make sure the contestants do not fail to notice it.
– Length of a text has no significant impact on real difficulty (Fig.3). The result of linear regression shows that no significant impact of the length of a text on the actual difficulty of a task (given by index of difficulty) was proved. The graph shows that in the set of 283 tasks only weak or moderate dependence of difficulty on length of the text was proved, coefficient of determination R2in linear regression model was only 0.11 and Pearson’s correlation coefficient 0.34.
– The length of the text has no significant impact on perceived difficulty. Significant impact of the length of the text on perceived difficult was not proved (given by no answer ratio). Pearson’s correlation coefficient 0.55 indicates moderate dependence.
So our results agree with [11] and disagree with [9] in this case.
Perceived Task Difficulty in Contrast to Its Real Difficulty. In 5 out of 20 assessed factors we could observe a difference between actual and perceived test task difficulty (given by no answer ratio). This means that the impact of some factors on real task difficulty did not affect the real difficulty statistically but affected the perceived diffi-culty (or vice versa).
Tasks of type problem solving (strategies, logical tasks) were perceived as easier by the contestants but the ratio of correct answer showed no difference. Multiple-choice tasks were perceived as more difficult but in reality they were not. Also subtype of interactive tasks with clicking on objects was perceived as more difficult. Optimization problems were not perceived as more difficult but in reality were more difficult. What we found most striking was that the presence of an illustrative picture makes the task appear more difficult to the contestants, as this type of tasks was answered by relatively
Fig. 3. A low dependence of a task difficulty index on the length of assignment text.
What Makes Situational Informatics Tasks Difficult? 99
fewer contestants. This might be explained by the assumption that the author decided to use this illustration out of their fear that the task was too difficult. The author seems to anticipate this difficulty thanks to some other signal. And contestants seem to perceive the task difficulty intuitively thanks to the same signal. It is this “other” signal, not the presence of an illustration or example that causes more refusals to solve the task. If the contestant decides to solve the problem in the end, the example will help them and the task will not be as difficult in the end.
6 Conclusion
Comparison of different indicators for stating difficulty of contest tasks shows that the criterion of success rate (i.e. what proportion of contestants chose the correct answer) does not always correspond to other indicators and that the indicator no answer describes a task difficulty very well. Thus a task difficulty will be more accurately predicted if we take into account not only the indicator success rate but also the indicator no answer. Thus our recommendation is to use the rule based on both of these indicators.
Presence of formalized description, structuring, optimization and demands of assignment reading comprehension are also substantial factors making a task more difficult. Interactivity of answering appears to be a significant factor making the task less difficult. On the other hand factors as the length of the text, use of technical terminology, algorithmization, diagrams, negative questions, illustrative pictures or examples did not prove to affect the difficulty of an informatics task.
It must be stressed at this point that the discovered statistical dependencies do not automatically mean there is some causality. We do not claim that the presence of any of these indicators and factors is responsible for a task difficulty. For example the cause of difficulty of structuring and formal description may be the higher level of necessary abstraction. However, this is hard to detect. We had to look for factors that can be detected in a task assignment more or less accurately and use such for determination of a task difficulty. The real difficulty of a task will always have to be detected only experimentally, i.e. when it is actually solved in the contest.
Results of this research study cast some light on the area of predicting difficulty of situational informatics tasks and at the same time indicate in which direction to con-tinue in research: to look for links between individual factors, to study these factors in different age and gender groups, in a child’s development, compare these factors across countries or school subjects.
Accurate determination of the difficulty of test tasks for an informatics contest will be of benefit when constructing contest tests, for development of new tasks designed for a particular age group or level of difficulty. If some elements of a task prove to contribute to its difficulty, it will be possible to modify the tasks and tune the test. Also findings on sources of difficulties of informatics tasks may contribute to development of higher quality informatics curricula.
Acknowledgment. The research was supported by the project GAJU 121/2016/S.
100 J. Vaníček
References
1. Dagienė, V.: What kinds of tasks are good for contests? In: 6th International Conference on Creativity in Mathematics Education and the Education of Gifted Students, Riga, Latvia, pp. 62–65 (2011). ISBN 9789984453606
2. Tomcsányiová, M., Kabátová, M.: Categorization of pictures in tasks of the Bebras contest.
In: Diethelm, I., Mittermeir, R.T. (eds.) ISSEP 2013. LNCS, vol. 7780, pp. 184–195.
Springer, Heidelberg (2013). ISBN 978-3-642-36616-1
3. Pohl, W., Hein, H.-W.: Aspects of quality in the presentation of informatics challenge tasks.
In: Jekovec, M. (ed.) The Proceedings of International Conference on Informatics in Schools: Situation, Evolution and Perspectives— ISSEP 2015, pp. 21–32. Založba FRI, Ljubljana (2015)
4. Tomcsányi, P., Vaníček, J.: International comparison of problems from an informatic contest. In: ICTE 2009: Information and Communication Technology in Education 2009, pp. 219–221. University of Ostrava, Ostrava (CZ) (2009). ISBN 978-80-7368-459-4 5. Dagienė, V., Stupurienė, G.: Bebras – a sustainable community building model for the
concept based learning of informatics and computational thinking. Inform. Educ. 15(1), 25–
44 (2016)
6. Tomcsányi, P.: Náročnosť úloh v súťaži Informatický bobor (Difficulty of tasks in Informatický bobr contest. In: Konferencia DidInfo 2009. Univerzita Mateja Bela, Banská Bystrica (SK) (2009). ISBN 978-80-8083-720-4
7. Tomcsányiová, M., Tomcsányi, P.: Analýza riešení úloh súťaže iBobor v školskom roku 2013/14. (Analysis of solutions in iBobor contest tasks in 2012/13 year). Konference DidactIG 2014. Technical University, Liberec (CZ) (2014)
8. van der Vegt, W.: Predicting the difficulty level of a Bebras task. Olympiads Inform. 7, 132–
139 (2013)
9. Yagunova, E., Podznyakov, S., Ryzhova, N., Razumovskaia, E., Korovkin, N.: Tasks classification and age differences in task perception. Case study of international on-line competition“Beaver”. In: Jekovec, M. (ed.) The Proceedings of International Conference on Informatics in Schools: Situation, Evolution and Perspectives— ISSEP 2015, pp. 33–43.
Založba FRI, Ljubljana (2015)
10. Gujberová, M.: Výber úloh do informatickej súťaže iBobor (Collection tasks for informatics contest iBobor. In: Konference DidactIG 2014. Technical Universoty, Liberec (CZ) (2014) 11. Vaníček, J., Křížová, M.: Kritéria obtížnosti testových otázek v informatické soutěži (criteria of informatics contest tasks difficulty). In: Lovászová, G. (ed.) DidInfo 2014, pp. 191–199.
Univerzita Mateja Béla, Banská Bystrica (SK) (2014)
12. Šimandl, V.: Odhalování podvádění v online soutěžích (detecting of cheating in online competitions). J. Technol. Inf. Educ. 6(2), 114–121 (2014). ISSN 1803-537X
13. Dagienė, V., Futschek, G.: Bebras international contest on informatics and computer literacy: criteria for good tasks. In: Mittermeir, R.T., Sysło, M.M. (eds.) ISSEP 2008. LNCS, vol. 5090, pp. 19–30. Springer, Heidelberg (2008)
What Makes Situational Informatics Tasks Difficult? 101