Dealing with anomalous data.

[vc_row][vc_column][vc_row_inner][vc_column_inner][vc_column_text]In a survey of Danish chemistry students I conducted with my colleague Frederik Voetmann Christiansen (reported in Johansen & Christiansen (2020)), about two thirds of the students acknowledged that they had removed an outlier or discarded an entire experiment only because the results somehow seemed wrong. This result is hardly surprising for anyone who has ever done experiments in a chemistry lab. It is difficult to do experiments. That’s why students spend a lot of time learning how to do them – and that’s why experiments often go wrong for inexplicable reasons. And yet, for researchers discarding or deleting results is a so-called questionable research practice.

More surprisingly, about one third of the students in the survey asserted that they had been encouraged by a teacher to delete or discard results. Taken at face value, this apparent result suggests that university teachers actively instruct students to engage in questionable practices. A closer analysis of the data, however, gave an important qualification to this result.[/vc_column_text][/vc_column_inner][/vc_row_inner][vc_row_inner css=”.vc_custom_1592560273619{margin-top: 32px !important;margin-right: 1px !important;margin-bottom: 32px !important;margin-left: 1px !important;padding-top: 16px !important;padding-right: 16px !important;padding-bottom: 16px !important;padding-left: 16px !important;background: #388bc2 url(http://h2020integrity.eu/wp-content/uploads/2020/06/INTEGRITY_Icon-Quote04.png?id=2644) !important;}”][vc_column_inner][nd_options_text nd_options_text_tag=”h2″ nd_options_text_weight=”normal” nd_options_text_family=”nd_options_first_font” nd_options_text=”About one third of the students in the survey asserted that they had been encouraged by a teacher to delete or discard results.” nd_options_text_line_height=”36″ nd_options_text_color=”#ffffff”][/vc_column_inner][/vc_row_inner][vc_row_inner][vc_column_inner][vc_column_text]In the survey, students who indicated that they had been encouraged by a teacher to discard data were asked to describe the situation where the encouragement occurred. In broad terms the students drew on four different types of arguments in their explanations, namely:

• Learning arguments, where students discarded and redid experiments in order to understand why the strange and unexpected result had appeared in the first place.

• Empirical arguments, where the students used other experiments done on-site or their background knowledge of the empirical context they worked within to argue that an anomalous result was probably caused by a mistake.

• Instrumental arguments, where students needed the product of the experiment e.g. a particular chemical compound to continue a longer sequence of experiments.

• Authoritarian arguments, where students referred to an authority such as the teacher or known theory.

From a lab practice point of view, the three first arguments may in fact offer sufficient justification for discarding or deleting a result; if you need a specific chemical compound to continue your experiment, you have no choice but to discard an experiment that failed to produce the right compound and try again. And if your understanding of the empirical nexus you are working with makes it very likely that an anomalous result was caused by a specific mistake, it seems sensible to discard the result in order not to contaminate your data (given that you are transparent and remember to mention the deleted data).[/vc_column_text][/vc_column_inner][/vc_row_inner][vc_row_inner css=”.vc_custom_1592560273619{margin-top: 32px !important;margin-right: 1px !important;margin-bottom: 32px !important;margin-left: 1px !important;padding-top: 16px !important;padding-right: 16px !important;padding-bottom: 16px !important;padding-left: 16px !important;background: #388bc2 url(http://h2020integrity.eu/wp-content/uploads/2020/06/INTEGRITY_Icon-Quote04.png?id=2644) !important;}”][vc_column_inner][nd_options_text nd_options_text_tag=”h2″ nd_options_text_weight=”normal” nd_options_text_family=”nd_options_first_font” nd_options_text=”In the survey, students who indicated that they had been encouraged by a teacher to discard data were asked to describe the situation where the encouragement occurred.” nd_options_text_line_height=”36″ nd_options_text_color=”#ffffff”][/vc_column_inner][/vc_row_inner][vc_row_inner][vc_column_inner][vc_column_text]Of course, it could argue that you ought to make sure and investigate precisely how the unexpected result occurred, but these kinds of mistakes are so common in real life laboratory work that you cannot examine all of them. As the fourth law of thermodynamics goes (according to Ravetz (1971 p.76)) “no experiment goes properly the first time”, so if you stop to investigate every strange and anomalous result, you will never get any important work done (see also Kuhn [1962] 1970, p. 82 for similar remarks). In real-life science there may, in other words, be valid reasons to discard or delete an anomalous result, and the teachers who encouraged the students to do so were not (necessarily) encouraging the students to engage in questionable practices, but only teaching them sound and sensible laboratory practice.

However, the arguments drawing on authority were more worrisome. In the empiricist ethos that governs the natural sciences today any kind of argument from authority is looked upon with suspicion, as empirical observation is supposed to be the only authority. It is, however, clear that this stance is not reasonable. Any advanced scientific knowledge production requires some kind of division of scientific labor. You cannot test and check everything by yourself, but will have to trust the authority of other scientists and scientific institutions. This is especially so for students who still have limited laboratory skills. Consequently, at least some of the arguments from authority presented by the students in the survey may be justifiable.[/vc_column_text][/vc_column_inner][/vc_row_inner][vc_row_inner css=”.vc_custom_1592560273619{margin-top: 32px !important;margin-right: 1px !important;margin-bottom: 32px !important;margin-left: 1px !important;padding-top: 16px !important;padding-right: 16px !important;padding-bottom: 16px !important;padding-left: 16px !important;background: #388bc2 url(http://h2020integrity.eu/wp-content/uploads/2020/06/INTEGRITY_Icon-Quote04.png?id=2644) !important;}”][vc_column_inner][nd_options_text nd_options_text_tag=”h2″ nd_options_text_weight=”normal” nd_options_text_family=”nd_options_first_font” nd_options_text=”The teachers who encouraged the students to do so were not (necessarily) encouraging the students to engage in questionable practices, but only teaching them sound and sensible laboratory practice.” nd_options_text_line_height=”36″ nd_options_text_color=”#ffffff”][/vc_column_inner][/vc_row_inner][vc_row_inner][vc_column_inner][vc_column_text]The arguments from authority, however, came in two qualitatively different types: In the first type, a teacher or the scientific record was seen as epistemic authorities. Those are the possibly justifiable cases. In the second type, the teacher was seen as an authority only because of the power he or she possessed to e.g. fail a student or give her a bad grade. In this last type of cases, students would in other words discard or delete a result when asked to do so by the teacher only because the students feared some kind of punishment if they failed to comply. Here, science is seen as a power game where those in control can dictate the results. This kind of thinking is clearly problematic from an integrity point of view, especially in the current situation where scientific knowledge production is under increased pressure from outside (non-epistemic) interests.[/vc_column_text][/vc_column_inner][/vc_row_inner][vc_row_inner css=”.vc_custom_1592560273619{margin-top: 32px !important;margin-right: 1px !important;margin-bottom: 32px !important;margin-left: 1px !important;padding-top: 16px !important;padding-right: 16px !important;padding-bottom: 16px !important;padding-left: 16px !important;background: #388bc2 url(http://h2020integrity.eu/wp-content/uploads/2020/06/INTEGRITY_Icon-Quote04.png?id=2644) !important;}”][vc_column_inner][nd_options_text nd_options_text_tag=”h2″ nd_options_text_weight=”normal” nd_options_text_family=”nd_options_first_font” nd_options_text=”This kind of thinking is clearly problematic from an integrity point of view, especially in the current situation where scientific knowledge production is under increased pressure from outside (non-epistemic) interests.” nd_options_text_line_height=”36″ nd_options_text_color=”#ffffff”][/vc_column_inner][/vc_row_inner][vc_row_inner][vc_column_inner][vc_column_text]From an integrity teaching point of view the main message of the study is that integrity cannot be taught as simple rules pointing particular actions out as not allowed. A rule such as “do not delete outliers” does not make sense in real-life lab practice, as it will depend on the context whether deleting or not deleting an outlier is the right thing to do. Instead of teaching rules aimed at the level of actions, it might be more productive to address the reasons behind the actions, and teach students what kind of reasons that can and cannot be used to justify a particular kind action – and how a justifiable action should be performed in order to comply with the norms of good study practice.[/vc_column_text][/vc_column_inner][/vc_row_inner][vc_row_inner css=”.vc_custom_1596705448088{margin-top: 32px !important;}”][vc_column_inner][vc_column_text]Read more in:
Johansen, M. W., & Christiansen, F. V. (2020). Handling Anomalous Data in the Lab: Students’ Perspectives on Deleting and Discarding. Science and Engineering Ethics.

Literature
Ravetz, J. R. (1971). Scientific Knowledge and its Social Problems. Oxford: Clarendon Press.
Kuhn, T. ([1962] 1970). The Structure of Scientific Revolutions. Second Edition. Chicago: The University of Chicago Press.

[/vc_column_text][/vc_column_inner][/vc_row_inner][/vc_column][/vc_row]