The goal of this intellectual output is to validate the effectiveness of the products developed in the first three intellectual outputs. For this purpose, we have applied different instruments for assessing the awareness of students of the importance of testing, as well as their knowledge of testing. The instruments are used in the courses of our pilot study in two phases, namely the baseline phase (before we started using out outputs) and the validation phase (after applying our outputs). We have developed three different instruments for this purpose: a rubric, a questionnaire and diagnostic assignments.

Thanks to the questionnaire and the diagnostic assignment, we can check whether there are some differences between students' self-assessment and a more objective assessment of their quality awareness. The rubric allows us to observe the evolution of the students throughout the course and we can check when there are changes in the behaviour of the students (e.g. improvement in the quality of their code).

• Rubric: it allows teachers from the different courses which are involved in the pilots to assess each assignment submitted by the students in terms of quality. It consists of 10 items (e.g., modularity, documentation, etc.) in a 4-Likert scale. All partners used this rubric in the baseline stage and are applying it during the validation stage. Thanks to this rubric, all students that participate in the project are assessed in the same way. To collect data in a homogenous format, the we have developed a web-based version of the rubric.
• Questionnaire: it allows us to know the perception of the students regarding the importance of quality issues when they programming. The questionnaire was the same for all the courses and was sent at the end of the term.
• Diagnostic assignments: the goal of such assignments is to observe which is the actual awareness and knowledge of quality issues of the students. Because each course is different, each lecturer in charge decided the questions and the format of her diagnostic assignment in detail, but all these assignments have a similar form and intention.

Summary results of the Open Universiteit Nederland (OU)

The course Advanced object-oriented programming is the second programming course and covers the more advanced OO topics: inheritance, abstract classes and interfaces, generic and enumeration types, error types, debugging, testing, robust programming, concurrency, GUI programming and MVC and the Observer pattern. After the topic robust programming, students make a hand-in assignment of 15 study hours concerning correct functionality, robustness and testing. Each student's solution is accessed in detail and students receive detailed feedback based on a rubric.

We did a baseline and a validation measurement. The baseline concerned the course as it has been given since 2018. About 80 students were involved. The validation (February 2023 to April 2023) concerned the course with an instructional design based on the procedural guidance: contracts and how to use them are explained and trained by exercises. Again, about 80 students registered for the course.

Based on the QPED rubric used, using a 4-points scale, in both measurements we observed a positive redistribution of students across the various score ranges. Where in the baseline phase the majority of students achieved scores of 3, in the validation phase there is a decrease in the proportion of students obtaining scores of 2 and 3 while there is a significantly higher percentage of students attaining the highest score of 4. This pattern recurs in the majority of the items, in particular test traceability, test completeness and robustness. The questionnaire suggests a positive impact of the learning tools implemented in the project on the skills acquired by the students.

Please, find more information in this short report. If you need additional information, please write to us in order to provide you with an in-depth report.

Summary results of the Phillips-Universität Marburg (UMR)

The Phillips-Universität Marburg is a classical full university. For the pilot study in the QPED project we have chosen the Bachelor course Object-Oriented Programming (9 ECTS), which is part of several study programs and usually followed by 170-200 (exams taken) students in the first or second semester. The baseline measurement for our diagnostic test (part of the mandatory exam) was performed in the run of 2020/21. For our student self-evaluation by means of a questionnaire and the rubric-based assessment of exercises, the baseline measurement took place in 2021/22. The validation measurement was in all cases made in the winter term 2022/23. After the first baseline measurement, already small adjustments have been made to the course in terms of initially introducing the TILE approach for some exercises. Thereafter, in winter 22/23 the course has been adjusted further, widely using TILE and the MASS feedback system. Some of the assignments were also rewritten to stipulate a step-wise solution approach, following a simplified form of the procedural guidance.

In comparison, the rubric-based ratings for the different concepts are similar for all assignments in both the baseline and validation runs. Consistently, the concepts “correctness” and “readability” were rated higher in the validation. For other concepts, in particular “test traceability” and “test completeness”, the trend varies between assignments. We expect that that the use of the rubric suffered from the fluctuation of student tutors who filled-in the rubric over the years. In our questionnaire, students rate their understanding of different quality-related concepts as more positive in the validation run. Lastly, we performed a diagnostic test, i.e., an assignment targeting the student knowledge w.r.t. software quality in the mandatory written exam, which was completed over three years by 180, 200 and 170 students, respectively. Here, we found a clear positive trend. With every year, the score in the diagnostic task was around 16.4 percent higher. This effect was statistically significant as well with a very low p-value at p < 0.001.

Please, find more information in this short report. If you need additional information, please write to us in order to provide you with an in-depth report.

Summary results of the Universitat Oberta de Catalunya (UOC)

The course "Programming for bioinformatics" (PB), which has 6 ECTS, belongs to the Master’s degree in Bioinformatics and Biostatistics and has, on average, 200 students per semester. It covers basic programming concepts (e.g. syntax, variables, conditionals, loops, functions, etc.) and some topics related to Bioinformatics, such as math and visualization libraries. Regarding the assessment policy, PB has five graded assignments (GA) that students submit throughout the semester.

We did a baseline and a validation measurement. The baseline, with 144 students, was conducted from 16/02/22 to 24/06/22 and concerned the course as it has been given since 2020. During the validation (28/09/22 - 29/01/23), the course was adapted to the TILE approach and used feedback tools. In this occasion, 222 students registered for the course.

In the light of the results, we can see that the use of the TILE approach and feedback tools was effective in order to improve students' skills related to quality software. The improvement was especially high for “output correctness” (+22.4% students with score 4 in validation's diagnostic test in comparison with baseline's diagnostic test) and “program robustness” (+28.6%). Likewise, the questionnaire suggests a positive impact of artefacts implemented in the project, since more than 72% students strongly agreed with testing is as important as programming and using tests improves the quality of the software. Moreover, 96% agreed or strongly agreed with the readability of the code is as important as the code runs correctly.

Please, find more information in this short report. If you need additional information, please write to us in order to provide you with an in-depth report.

Summary results of the Technische Universiteit Eindhoven (TU/e)

Both the lectures and instruction classes for the 5ECTs Introductory Programming course (2IP90) were augmented with TILE test and Procedural Guidance information. The instruction classes contained, besides ungraded exercises for practice, graded assignments which contributed to the final mark. For the instruction classes a user manual was provided with a simplified, practical version of the Procedural Guidance.

In all categories (Rubric, Questionnaire and final marks), we see positive trends: the QPED contributions were appreciated and improvements are noticeable. The differences were not very large: teaching at TU/e involved already aspects of the QPED approach like specification and example test runs, so this is consistent with expectations. A clear positive influence was that testing and Procedural Guidance were made more explicit in teaching and application by the students, contributing to a greater awareness among teachers and students.

Please, find more information in this short report. If you need additional information, please write to us in order to provide you with an in-depth report.

Available material and instruments used during the pilots

• Diagnostic tests
• Quality awareness questionnaire
• Rubric
• Web-based tool to fill in rubric tool (Git repository)
• Publications
  • S. Dick, T. Dreyer and C. Bockisch, "TILE and MASS, a retrospective", In Software Engineering im Unterricht an Hochschulen (SEUH 2024), Lecture Notes in Informatics (LNI), Gesellschaft für Informatik, Bonn, 2024. To Appear
  • N. Doorn, T. Vos, B. Marín, C. Bockisch,S. Dick and E. Barendsen, "Domain TILEs: Test Informed Learning with Examples from the Testing Domain". In: S. Nurcan, A. Opdahl, H Mouratidis, A. Tsohou (eds) Research Challenges in Information Science: Information Science and the Connected World (RCIS 2023), Lecture Notes in Business Information Processing, Vol 476. Springer, Cham. https://doi.org/10.1007/978-3-031-33080-3_31
  • S. Dick, S. Schulz, C. Bockisch, "A study on the quality mindedness of students", Software Engineering im Unterricht der Hochschulen (SEUH 2022), Gesellschaft für Informatik, Bonn, PISSN: 1617-5468. ISBN: 978-3-88579-715-9. pp. 119-124. Ergänzende Themen https://doi.org/10.18420/SEUH2022_12