Forensic Engineering: Learning by Accident - Teaching
Investigation Skills to graduate students using Real-Life
Accident Simulations
Gillian N. Saunders-Smits1, Michiel J. Schuurman2 and Calvin Rans3
Faculty of Aerospace Engineering, Delft University of Technology, Delft, 2629 HS the Netherlands
This paper relates the experiences of lecturers at Delft University of Technology in the designing and running of a Master course in Forensic Engineering. Rather than traditional face-to-face lectures, use of real-life evidence-based learning was made in the form of training for and execution of a mock aircraft accident investigation. The culmination of this learning experience for the students was the group exam, which took the form of the examination of a recreated accident scene. Students were required to organize their investigation groups, document the scene, and collect evidence while on scene. Subsequently, they were given time to analyze the results and prepare a standard accident investigation report. Evaluation results show satisfied students and good learning outcomes, making this a course worth repeating.
Nomenclature
mode = most frequently occurring score in a set of data n = sample sizep = p-value or significance of a test r = effect size
rs = Spearman’s rank correlation coefficient
S.D. = Standard Deviation
z = standardized test statistic in Wilcoxon Signed Rank test
I. Introduction
eaching students that reality is not like television is a challenge that educators of all age groups face. In this particular case the lecturers at the Faculty of Aerospace Engineering of Delft University of Technology in Delft, The Netherlands set out to challenge students’ misconceptions about accident investigation, influenced by news shows and ‘documentaries’ on various television channels, through a master course on Forensic Engineering.
Forensic science is defined as the use science for the purpose of law1. As Aerospace Engineering at Delft University of Technology focuses on aircraft, the primary principles are related to engineering. Therefore the term Forensic Engineering is introduced to use scientific and engineering methods of gathering and examining information related to aircraft.
To teach students about the world of Forensic Engineering the lecturers chose to use Real-Life, Problem-Based-Learning as a starting point, immersing students in the world of air accident investigation and having them experience what it is like to be an air crash investigator. This paper explains what the objectives of the course are, how the course is designed to meet those objectives and shares the experiences of the first iteration of running the
1 Assistant Professor, Aerospace Structures & Materials Department, Faculty of Aerospace Engineering, Delft University of Technology, Kluyverweg 1, 2629 HS Delft, The Netherlands, non-member.
2 Assistant Professor, Aerospace Structures & Materials Department, Faculty of Aerospace Engineering, Delft University of Technology, Kluyverweg 1, 2629 HS Delft, The Netherlands, non-member.
3 Assistant Professor, Aerospace Structures & Materials Department, Faculty of Aerospace Engineering, Delft University of Technology, Kluyverweg 1, 2629 HS Delft, The Netherlands, member.
course. Furthermore, a closer look is taken at the learning outcomes of the course for the students and a set of clear recommendations is given for colleagues who are inspired to run courses with a similar set up.
II. Faculty of Aerospace Engineering
The course discussed in this paper is an elective course with an 84-hour study load in the 2-year MSc course at the Faculty of Aerospace Engineering of Delft University of Technology in the Netherlands. Delft University of Technology (TU Delft) was founded by King William II in 1842 and has grown into one of the Netherlands’ 14 Research Oriented Universities. The university consist of 8 faculties offering 15 BSc degrees and 30 Master degrees in Engineering with over 20000 students (BSc & MSc, excluding PhD students). The Faculty of Aerospace Engineering was founded in 1940 and has grown to become the largest aerospace engineering faculty in Western Europe with over 1700 BSc Students and over 1000 MSc students divided over 4 departments. It offers students a broad 3-year Bachelor degree (work load of 5040h) in Aerospace Engineering followed by a specialized 2-year MSc degree (work load of 3360 hours) allowing students to specialize in one of 5 topics: Aerodynamics & Wind Energy, Flight Propulsion & Performance, Control & Operations, Aerospace Structures & Materials, and Space Flight. Approximately 34% of our students come from abroad. The faculty is a member of several education networks such as CDIO1 and GE32 and has an active learning philosophy when it comes to the curriculum2.
III. Teaching Real-life Courses
As stated previously active learning in all its facets is the core philosophy to the aerospace curricula taught at TU Delft. By active learning, it is meant that learning is the responsibility of the learner and that the learner must undertake more activities than just listening as discussed in Bonewell & Eison3. In their report they argue that the application of active learning methods leads to deeper learning, reaching higher levels of Blooms taxonomy4, by exposing students to life-like role play scenarios, case studies etcetera. This is where real-life learning comes in. A type of learning that emerses students in a life-like (as in the case discussed in this paper) or real situation to enable the learning4. Next to that the Faculty of Aerospace Engineering of TU Delft is a proud member of the CDIOTM network which aims to stress engineering fundamentals in the context of conceiving, designing, implementing and operating real-world systems and products which also advocate the use of active learning methods6. It is from these point-of-views that the course is designed with an aim to enhance the learning experience for the students and obtain deeper learning.
IV. Forensic Engineering Course Set-Up
The course was set up with a combination of lectures and case-based practical examples. In this forensic engineering course we set out to teach students forensic methods and principles by using air safety investigations as examples. In certain cases the accident is related to an engineering failure by which an investigation can reveal the root cause. In other investigations human error is a prominent cause but more often it is a combination theroreof. Only a non-biased investigation approach using facts will result in a substantiated conclusion. The learning objectives of the course are:
- To familiarize the students with different phases of an investigation and principles used in safety investigations
- To provide students an oversight of investigation methods and techniques within a range of disciplines, in particular with respect to diagnosing complex systems and technological concepts
- To familiarize students with the nature, scope and context of the findings and recommendations which can be derived from the various investigation process phases
- To clarify opportunities, threats, strengths, weaknesses in the investigation process with respect to its feedback of expertise and experiences in the engineering and design methodology
- To demonstrate the students the potential of case based and evidence based learning with respect to its potential for deriving design principles and knowledge deficiencies from these cases
- To provide the students with the basic competences to link safety investigation findings with systems safety enhancement strategies during design and operation
1 The CDIOTM initiative is an innovative educational framework for producing the next generation of engineers and the CDIOTM network is formed by a diverse range of institutions see www.cdio.org
2 The Global Engineering Education Exchange is a global exchange network for engineering students founded by research universities, see www.iie.org/Programs/GlobalE3
The course is based on case-based and evidence-based learning principles. The course aims at that capability to cooperate with various scientific and design disciplines in a specific case oriented context. The course is taught by two assistant professors one of which is also a senior air safety investigator working for Dutch Safety Board and has over 10 years of experience in hands on air accident investigation.
V. Course Delivery
In order to accomplish the intended learning outcomes, the courses consisted of 8 lectures, 2 practical sessions, and a final practical exam for which a simulated accident scene investigation was created. During the lectures the primary phases of an aircraft accident investigation were explained: starting from the fact-finding phase through the analysis phase and the final report-writing phase7.
During the Forensic Engineering lectures students learned how to procedurally conduct an (accident) investigation and apply forensic concepts to determine the (probable) cause of the event, but they also learned about valuing the variety of required knowledge and specialism. For the fact-finding phase of an investigation topics were explained on case-by-case bases using actual aircraft incidents (clased as serious or higher under ICAO classification terminology) and accidents as examples. The topics dealt with included: flight data analyses, fire investigation, and structural investigation based on the book8. Different engineering failure modes and characteristics were shown and discussed. Most lectures were primary technical and safety orientated in nature. However because in many cases humans (pilot, witness or other people) are involved in accidents it was found necessary to expose students to ‘living’ information sources. The theory and practical session on interviewing was an experience for students, some of whom had no idea of the difference between closed and open-ended questions.
Industry practice of working together in teams and the reality of pursuing several lines of investigation simultaneously were also given serious attention. This was done by running two practical sessions to prepare students for the final exam. The case studies were chosen based on actual accidents but some aspects were adapeted for educational purposes. Student received limited information on the event and were tasked to analyze the accident by determining the probable cause(s). On request, certain additional information was provided but only if the students were able to ask the right questions to the lecturer. The diversity of the accident scenarios required students to apply a broad range of disciplines ranging from aerodynamics to the failure of structures. This background knowledge waspresumed to be basic knowledge obtained in the bachelor phase of the students’ educational path.
The application of previous knowledge is often challenging for many students. This is exacerbated by the variety of bachelor education backgrounds (Aerospace, mechanical, physics and material science) combined with different MSc specializations track (varying from Aerodynamics to Structures or from Space to Wind Energy or from Control to Operations) of the students. Therefore, students automatically divided themselves into groups of ‘experts’ working out a single topic of the accident during the practical case studies. At the end each ‘expert group’ reports back to the class on that topic. The sharing of knowledge then opened up a discussion on what happened and why it happened. After the two practical sessions the use of the ‘expert group’ philosophy was notable and causes were found more quickly. This prepared the students for the final exam and the experience of a total immersion in a simulated accident scene investigation.
Another aspect which was brought to students attention was (information) bias and the need for critical thinking. Students were shown accident examples which at first glance seemed ‘easy’ to solve. The students were then asked to explain why they thought was the causes of the accident. During the discussion it became clear that the engineering background influenced students hypothesis, judgement and conclusions. By following the normal investigative process steps: finding, evaluating and analyzing facts; a substaniniated fact based conclusion can be drawn. In this case the outcome of the event was something that was not that obvious but could very well be substantiated if all the facts were gathered. One student commented after the lecture: “I never though I could be that wrong”, which in essence summerized the goal of the lecture.
VI. Assessment of the Students
The final group-based exam consisted of a 6 – 8 student team carrying out a simulated accident scene investigation (field part) and write a final accident report (report part). The group received a field grade which was 50% of the final student grade. Three lecturers assessed each group using a predefined set of observation and grading criteria. After the field part of the exam, students had two weeks to write a final accident report following Annex 13 format9. With regards to report grading, it was not necessary that the students obtain the conceived accident cause. It was well recognized that in setting up a simulated accident scene, extraneous details or misleading clues could be overlooked by the instructors. Thus, the primary goal of the report was to have students develop scenarios (hypotheses) and determine their credibility. For grading the final reports a assessment matrix was made
which took into account the ‘facts’ of the scenario and the clues left in the field by the lectures. It also took into account the formulation of hypotheses and the analyses. The final report counted for 50% towards the final grade
The accident scene was setup on the grass field behind the faculty building of Aerospace Engineering. The scenario involved two aircraft: one ultra-light and one glider positioned some distance apart on the field. See the photo in Fig. 1 to get a feel for the simulated accident scene.
Groups of students were brought to the accident site to start the exam and given one hour to investigate the simulated accident. Beforehand students were given a briefing indicating that the police had cordoned off the area and that both pilots were taken to hospital and thus unavailable for interviews. They had one hour available to them on the field. Three lecturers were present on the field to assist students if necessary; however, they did not answer any questions on the nature of the event or comment on what had occurred. The primary goal was to assess the students for the exam field grade.
The students’ reactions when arriving at the simulated accident scene made for interesting observations. Some students were visibly shocked and out of their comfort zone. What they had learned, how to search for facts and conduct interviews had not fully prepared them for the reality that was presented. The seeming overwhelming amount of information and the time pressure of getting it done in one hour created stress which students had to deal with.
A variety of items (facts), both relevant and non-relevant, were placed on the field to be found by students during the exam. Examples of relevant tems included a briefcase with pilot and aircraft documents. Non-relevant, but out-of-place items were also included to to assess the observational skills of the groups. An example of one such item was a rubber duck, which is normally found in your bathtub, that was positioned inside the glider, see Fig.1. This seemingly out-of place item was meant to trigger the observation skill and process of information by students. The duck was found multiple times during the exam, however another non-relevant item, a stone rabbit garden ornament strategically concealed in the grass, was not. On the other hand, much to the surprise of lectures, some items including a fork, were found by students which were not part of the exam.
Student’s natural presumption was further challenged by the scenario setup. As an example: The lady at the picnic table reading a book was assumed by everyone to be a witness to the incident. However her brief was that she arrived after the incident and was simply having a picnic out in the field. The notion that someone who is at the scene must always have useful information was one of the presumptions the students were challenged on. That this was necessary could be observed from the fact that every group complained about the witness who was ‘uncooperative and at times annoying in not answering questions’ when in fact she had no answers and was irrelevant for the investigation. This comment was also found in the course evaluation filled in by the students where two respondents wrote: “Provide more useful witnesses” (see table 1).
Another way of enhancing the life-like accident scenario was the fact the national press was invited to participate in the exam. As it was a slow news day, the news crews came out in force. The press had been briefed to ask the students relevant questions as they would during a normal accident, but at the same time not to push the students too hard. The teams of students needed to react to the presence of the press, and devise strategies for dealing with them while continuing their investigation. The life-like simulation was so real it made national television and radio throughout the day and various national newspapers reported on the event as well.
Figure 1. General overview simulated accident scene behind the faculty building of Aerospace Engineering (Left). The Rubber Duck hidden by lectures in the glider (Right).
The final accident report was assessed on facts (observations), the analyses of these facts and the conclusions drawn there from. Beforehand it was explained that a hypotheses (scenario) and the analyses of that scenario was the essence of the final report. How thoroughly was the analysis described and supported by facts? As an example in one of the final reports the student team hypothesized that the ultra-light was towing a banner (which was not found in the field). However, the students accurately recognized that the GPS data found onboard the aircraft showed a takeoff and direct flight path to the accident site. For banner towing a normal pickup would have occurred at the airport, showing a second low altitude pass at the airport, the pick-up banner loop was not recorded on the GPS. The hypothesis therefore was rejected on the available data, this example illustrates the line of thought of the student group.
VII. Course Evaluation
A. Research Set Up
At the end of the course the authors were extremely keen to find out if the learning objectives of the course had been met. Furthermore the goal was to try to make a more detailed analysis of the learning outcomes and use the feedback to further enhance the course in the future. To this end a computer-based questionnaire was designed and using our Virtual Learning Environment could be answered directly at the end of the course by students. The questionnaire consisted partly of the university’s standard course evaluations questions with additional questions asking students to assess themselves on their perceived skill level in a number of learning outcomes of the course. To meet the university’s ethical board requirements, students were asked if they wanted to take part in this research study. We informed that during this research they would be asked to fill in a computer-based questionnaire at the end of the course and that their answers would be linked to student performance. Of the 41 students who participated in the questionnaire, 29 students gave their permission to analyze their submitted answers for this research study. Out of the total number of students who completed the course (65) this yields a response rate of 45%. All results have been anonymised for the analysis performed.
B. Research Questions
In the general part of the course evaluation we wanted answers to the following questions: 1) Was the set up of the course right?
2) Was the workload of the course in-line with the number of credits? 3) Was the assessment method acceptable?
4) What parts of the course should be improved?
Our secondary aim in this research was to find out if students felt that the course had added to their forensic engineering skills. We also wanted to find out if there were any relationships between their grades and their growth in forensic skills as well investigate if any significant relationships exist between the students’ self-perceived ability to ask critical questions, observe an accident scene, record essential information, process information & come to preliminary conclusions, the ability to write a comprehensive report, and their grade. Our research questions in this part of the study are:
5) What did students feel they learned?
6) Does the self-perceived skill level of students’ forensic engineering skills change over the running of this course?
7) Does a correlation exist between the self-perceived skill level of students in forensic engineering and the grade the student received for the course?
8) Do correlations exist between the students’ self-perceived ability to ask critical questions, observe an accident scene, record essential information, process information & come to preliminary conclusions, the ability to write a comprehensive report and their grade?
To answer these questions a questionnaire was designed and the data from the questionnaire was subjected to number of statistical operations. The questionnaire is available from the authors upon request.
C. Data Analysis
All data was coded and made anonymous and put in SPSSTM in which also all analyses were carried out. A frequency distribution and the mode were calculated to investigate the research questions concerning set up, the workload, and the assessment method. A small ethnographic analysis was carried out on the open questions. For research question 6, 7, and 8 non-parametric tests were carried out, as the data was not normally distributed. For research question 6, a Wilcoxon Signed Rank test was carried out to see if there was a significant difference. Research question 7 and 8 were researched by calculating Spearman’s rho to see if any correlations existed.
D. Results
To answer the first research question whether the set up of the course was right students were asked to evaluate the number of lectures, the number of lectures on forensic theory, the number of guest lectures, the number of practical sessions, and whether the material in the lecture and practical’s prepared them sufficiently for the exam. Each question could be evaluated on a 5-point Likert scale. It can be observed from the results from figure 2 that most students would have liked more lectures, guest lectures, and practicals. However, this is most likely due to their motivation and interest for the subject as they also by a great majority agreed (more than 60%) that the materials provided in the lectures and practicals prepared them sufficiently for the exam.
The total workload for the course was set at 84 hours. 78% of the students agreed or strongly agreed that the course was within the specified workload and therefore doable.
We also wanted to find out whether or not the assessment methods used were acceptable to the students. We asked the students several questions on this theme as listed in fig. 3. From the results it can be seen that students overwhelmingly agreed that the assessment method (over 96%) was appropriate and that they prefer the hands on exam to a written exam (over 92%). They also liked the combination of the individual and group assignments and prefer the group exam to an individual exam. Although 25% of the students indicated they did not agree with the combination of individual assignment. We asked students if they had any advice for us to improve the course. This resulted in a total of 22 responses and a total of 35 response items. In table 1 the most frequent occurring items were listed. The most frequent comment was to say that they liked the course. In terms of advice students indicated they would like to work in smaller groups on the in-class assignments, that they would like more theory less examples. Interestingly 2 students also indicated they would like better witnesses, which means we will have to reinforce our teaching on the reality of scenes in that you may be confronted with a incident situation in which no or only very biased witnesses exist.
In order to investigate the learning outcomes we asked the students to list what they felt they had learned during
Figure 2. Evaluation of the numbers of lectures, guest lectures and practical’s and their relation to the exam (n=28)
the course. This was an open-ended question that 26 students answered and in total gave 71 response items of what they learned. In table 2 the most frequent occurring items are listed. Top of the list is what to look for in an accident and standard investigation methodology. Other high scoring items were hypothesis forming, interviewing and critical thinking skills, to question everything, how to critically evaluate, and how to decide what information is important.
We also wanted to investigate if the students’ self-perceived skill level went up over the course. In order to do that we asked the students to grade themselves on a 1-10 scale (with 1 - very poor and 10 – excellent) on their self-perceived skill level in forensic engineering before and after the course. The students rated themselves an average of 5.3 (n = 29, S.D. = 1.236) before the course and an average of 8.0 (n = 28, S.D. = 0.693) after the course. A Wilcoxon Signed Rank test was carried out to see if there was a significant increase and its results (z = 4.577, p < 0.05, r = 0.86) show that a significance increase in skill was perceived by the students with a large effect size10. Interestingly their assessment of their ability after the course is not that far off the average grade of the students received for the course, which was 7.7 (n = 29, S.D. = 0.4312) as the standard deviations overlap.
This then led us to the question whether a correlation exists between the self-perceived skill level of students in forensic engineering and the grade the student received for the course. A Spearman’s correlation coefficient was
calculated (rs = 0.081), the value of which was so low that no correlation may be assumed10.
Finally, we wanted to see if any correlation existed between the students’ self-perceived ability to ask critical questions, observe an accident scene, record essential information, process information & come to preliminary conclusions, the ability to write a comprehensive report, and their grade. Again Spearman’s correlation coefficient was calculated and small correlations were found between the grade and the ability to ask questions (rs = 0.258), the
ability to record essential information (rs = 0.288), the probability to process information (rs = 0.214), and the ability
to write a comprehensive report (rs = 0.286). No correlation was found between the grade and the ability to observe
an accident scene
VIII. Conclusion & Reflection
This was the first time we as lecturers endeavored to run a hands-on, case-based course. Evaluation results show that the learning objectives were reached, and that students felt their skill level in forensic engineering went up over the duration of the course and correlation appear to exist between the learning objectives and the grades. It was also clear that students felt that they learned a lot during this course. However, the results also give us reason to reflect on running such a course again. It is important that from the get-go we prepare students for what to expect during the course and the function each assignment has in relation to learning objectives of the course. Results also highlight
that learning that the authors would expect to have occurred in other courses (such as asking critical questions, critical thinking and forming hypotheses) has not necessarily taking place yet. As this is an elective, this is an issue for the wider faculty body to consider. Another issue highlighted for us as lecturers is the number of students in the
course. Sixty five students took the course. With the ever-increasing number of Master students at Aerospace Engineering at TU Delft there is a real risk of reaching critical mass. Having said that we will not hesitate to do it again.
What remains is that we feel that it is very worthwhile to run evidence, case-based learning course. The preparation workload is high but the results are definitely worth it.
Acknowledgments
The authors would like to thank everyone who helped this course become a reality and the students for allowing us to do something different. In particular the authors would like to thank: Anthony Brickhouse (Embry Riddle Aeronautical University), Joris Melkert, Michel van Baal, Gemma van der Windt, Lisette Volmer, Johan Boender and colleagues from the Dutch Safety Board.
References
1White, P.,"Crime Scene to Court: The Essentials of Forensic Science", Royal Society of Chemistry, 2010
2Kamp, A., “Master of Science Aerospace Engineering Self-Assessment Report 2013”, Delft University of Technology, 2013 3Bonwell, C., Eison, J. “Active Learning: Creating Excitement in the Classroom” AEHE-ERIC Higher Education Report No.
1. Washington, D.C.: Jossey-Bass, 1991
4Bloom, B., Taxonomy of Educational Objectives: The Classification of Educational Goals McKay, New York, 1956. 5Weert, van, T and Tatnall, A (Eds.), Information & Communication Technologies and Real-life Learning – New Education
for the Knowledge Society, Springer, New York, 2005
6Crawley, E. Malmqvist, J., Östlund, S. Brodeur, D. Rethinking Engineering Education: The CDIO approach, Springer, New
York, 2007.
7International Civil Aviation Organization (ICAO), "Manual of Aircraft Accident and Incident Investigation (Doc 9756):
Part III — Investigation 1st Edition”, International Civil Aviation Organization (ICAO) ,2012
8Richard H. Wood and Robert W. Sweginnis, “Aircraft Accident Investigation”, Endeavor Books , 1990
Table 1: What advice do you have for us (35 items out of 22 responses)?
What advice do you have for us? Count
1. None, I was happy with the course thank you! 5
2. Create smaller investigation groups 4
3. More theory less examples 3
4. Provide more useful, better witnesses 2
5. Add lecture on field work 2
6. Prepare in-class assignments in advance 2
7. Give more time to write report 2
8. More lectures on structural analysis of aircraft incidents 2
9. Also include non-aviation disasters 2
10. More on investigation techniques 2
11. More practice on flight data interpretation 1
12. Move exam back 1
13. Keep the enthusiasm 1
14. More practical assignments 1
15. More guest lectures 1
16. Include personal review and lessons learned 1
17. More analysis methods 1
18. Written exam 1
9International Civil Aviation Organization (ICAO),"International Standards And Recommended Practices Annex 13
To the Convention on International Civil Aviation Aircraft Accident and Incident Investigation 10TH EDITION”, International Civil Aviation Organization (ICOO), 2010
10Field, A., Discovering Statistics Using SPSS (and sex, drugs and rock ‘n’ roll), 2nd edition, Sage, London, 2006
Table 2: What did you learn in the course? (71 items out of 26 responses)
What did you learn in the course? Count
1. What to look for in a crash 9
2. Standard investigation methodology 8
3. How to interview people 5
4. How to form a hypotheses 5
5. Critical thinking 4
6. To question everything 4
7. How to decide what information is most important 4
8. How to critically evaluate 4
9. Importance of diversity of background knowledge on the part of the investigator 3
10. How to make an investigation report 2
11. That you must work as a team 2
12. The big picture of Forensic engineering 2
13. Broader knowledge of past aviation accidents 2
14. How to draw conclusions 2
15. How to make recommendations 2
16. That not everything is what it looks like/not to draw conclusions too early 1
17. To dig deep to get to the truth 1
18. Aircraft procedures 1
19. Towing procedures 1
20. How difficult it is to manage information on your own 1
21. How to curve-fit data 1
22. What information can be explained differently 1
23. How to deal with data that does not fit your theory 1
24. That not all information is available or observable 1
25. Common issues with aircraft 1
26. How to combine information 1
27. How to look at the future 1
