A study into evaluating the location of fingermarks on letters given activity level propositions

Delft University of Technology

A study into evaluating the location of fingermarks on letters given activity level

propositions

de Ronde, Anouk; van Aken, Marja; de Poot, Christianne J.; de Puit, Marcel

DOI

10.1016/j.forsciint.2020.110443

Publication date

2020

Document Version

Final published version

Published in

Forensic Science International

Citation (APA)

de Ronde, A., van Aken, M., de Poot, C. J., & de Puit, M. (2020). A study into evaluating the location of

fingermarks on letters given activity level propositions. Forensic Science International, 315, [110443].

https://doi.org/10.1016/j.forsciint.2020.110443

Important note

To cite this publication, please use the final published version (if applicable).

Please check the document version above.

Copyright

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons. Takedown policy

Please contact us and provide details if you believe this document breaches copyrights. We will remove access to the work immediately and investigate your claim.

This work is downloaded from Delft University of Technology.

A

study

into

evaluating

the

location

of

fingermarks

on

letters

given

activity

level

propositions

Anouk

de

Ronde

*

,

Marja

van

Aken

,

Christianne

J.

de

Poot

,

Marcel

de

Puit

a

AmsterdamUniversityofAppliedSciences,ForensicSciences,P.O.Box1025,1000BAAmsterdam,theNetherlands b

NetherlandsForensicInstitute,DigitalTechnologyandBiometrics,P.O.Box24044,2490AATheHague,theNetherlands c

VUUniversityAmsterdam,CriminologyDepartment,DeBoelelaan1105,1081HVAmsterdam,theNetherlands

d_{DelftUniversityofTechnology,FacultyofAppliedSciences,ChemicalEngineering,VanderMaasweg9,2629HZ,Delft,theNetherlands}

ARTICLE INFO Articlehistory:

Received17December2019 Receivedinrevisedform9July2020 Accepted30July2020

Availableonline2August2020 Keywords: Fingerprints Activitylevel Documentexamination Fingermarklocation ABSTRACT

Apreviouspaperpublishedinthisjournalproposedamodelforevaluatingthelocationoffingermarkson two-dimensionalitems(deRonde,vanAken,dePuitanddePoot(2019)).Inthispaper,weapplythe proposedmodeltoadatasetconsistingofletterstotestwhethertheactivityofwritingalettercanbe distinguishedfromthealternativeactivityofreadingaletterbasedonthelocationofthefingermarkson theletters.Anexperimentwasconductedinwhichparticipantswereaskedtoreadaletterandwritea letterasseparateactivitiesonA4-andA5-sizedpapers.Thefingermarksontheletterswerevisualized, andtheresultingimagesweretransformedintogridrepresentations.Abinaryclassificationmodelwas used toclassifythelettersintotheactivities ofreading andwriting basedonthe locationof the fingermarksinthegridrepresentations.Furthermore,thelimitationsofthemodelwerestudiedby testingtheinfluenceofthelengthoftheletter,theright-orleft-handednessofthedonorandthesizeof thepaperwithanadditionalactivityoffoldingthepaper.Theresultsshowthatthemodelcanpredictthe activitiesofreadingorwritingaletterbasedonthefingermarklocationsonA4-sizedlettersof right-handeddonorswith98%accuracy.Additionally,thelengthofthewrittenletterandthehandednessof thedonordidnotinfluencetheperformanceoftheclassificationmodel.Changingthesizeoftheletters andaddinganactivityoffoldingthepaperafterwritingonitdecreasedthemodel’saccuracy.Expanding thetrainingsetwithpartofthisnewsethadapositiveinfluenceonthemodel’saccuracy.Theresults demonstratethatthemodelproposedbydeRonde,vanAken,dePuitanddePoot(2019)canindeedbe appliedtoothertwo-dimensionalitemsonwhichthedisputedactivitieswouldbeexpectedtoleadto differentfingermarklocations.Moreover,weshowthatthelocationoffingermarksonlettersprovides valuableinformationabouttheactivitythatiscarriedout.

©2020TheAuthors.PublishedbyElsevierB.V.ThisisanopenaccessarticleundertheCCBYlicense (http://creativecommons.org/licenses/by/4.0/).

1.Introduction

Focusontheactivitythatwascarriedoutduringthedeposition ofevidencehasrecentlybecomeanimportantaspectinthefieldof forensicscience[1,2].Establishingalinkbetweenthedonorand thecrimescenebydeterminingthesourceofthetraceisoftennot sufficient to determine what happened at the crime scene. Frequently,thequestionincourtisabouttheactivitythatledto thedepositionofthetraces,whichrequirestheuseofactivitylevel propositionsinsteadofsourcelevelpropositions[3].For finger-markevidence,theevaluationofactivitylevel propositionsisa ratherunexploredterritory.However,recentresearchhasshown

thatevaluatingfingermarksgivenactivitylevelpropositionsmay addvaluableinformationwhenoneisreconstructingacrime[4]. An important variable for the evaluation of fingermarks at activity levelisthelocationofthefingermarks ontheobjectof interest.deRonde,vanAken,dePuitanddePoot[5]presenteda model for evaluating fingermark locations on pillowcases in relationship to the activity level questions of whether the pillowcasewasusedforsmotheringorwassimplychanged.The paper proposed that this model could be applied to all two-dimensionalitemsforwhichitisexpectedthatdifferentactivities resultindifferentfingermarklocations.Aninterestingapplication forthismodelistheevaluationofthelocationoffingermarkson handwritten letters since it might be expected that different activities—such as writing and reading—leave fingermarks on differentlocationsandthatthelocationoffingermarksonaletter canbeusedtodeterminewhatactivityhastakenplace.

* Correspondingauthorat:AmsterdamUniversityofAppliedSciences,Forensic Sciences,P.O.Box1025,1000BA,Amsterdam,theNetherlands.

E-mailaddresses:a.de.ronde2@hva.nl,a.de.ronde@nfi.nl(A.deRonde).

http://dx.doi.org/10.1016/j.forsciint.2020.110443

0379-0738/©2020TheAuthors.PublishedbyElsevierB.V.ThisisanopenaccessarticleundertheCCBYlicense(http://creativecommons.org/licenses/by/4.0/).

ForensicScienceInternational315(2020)110443

ContentslistsavailableatScienceDirect

Forensic

Science

International

Althoughexaminations ofhandwrittendocumentsseemless relevantasaforensicdisciplineinthedigitalworld,astudyintothe demandfordocumentexaminationshowedthatthismaynotbe thecase[6].Besidescasesoffraudorcounterfeiting,handwritten document examination is still considered very important in counter-terrorism because terrorists appear to prefer to use handwrittentextstoavoiddigitaltraces.Handwrittendocument examinationisalsostillconsideredrelevantwhentheauthenticity ofsuicidenotesisquestioned.AnexampleofthisisthecaseRv. StephenPort[7],inwhichPortwasconvictedoffourmurders.In oneofthesemurders,Portleftasuicidenotenexttothevictimin anattempttodivertsuspicion.Anotherapplicationofhandwritten documentexaminationisincasesinvolvingillegaldrugs.Evidence collected in these cases regularly includes handwritten notes describingthemanufacturingstepsforthesynthesisofdrugs1.For allthesecases,itmightberelevanttodeterminewhowrotethe notesorlettersdiscoveredatthecrimescene.Incasesregarding handwrittendocuments,a plausible alternativeexplanation for the presence of fingermarks on letters may be the activity of readingtheletterinsteadofwritingtheletter.

The current approach forevaluating these types of questions about handwritten documents is to perform a handwriting examination[8].Weproposeacomplementaryinnovativeapproach: theevaluationofthelocationofthefingermarksontheletter.

This study investigates whether the model proposed by de Ronde,vanAken,dePuitanddePoot[5]toanalyzethelocationof fingermarkscouldalsobeusedtodistinguishtheactivityofwriting aletterfromthealternativeactivityofreading aletter.For this purpose,wedesignedanexperimentinwhichparticipantscarried outtwotasks:readingapreprintedletterandwritingaletter.The fingermarks were visualized using conventional visualization techniques for fingermarks on paper. Afterwards, the binary classificationmodelproposedbydeRonde,vanAken,dePuitand dePoot[5]wasusedtocategorizethelettersintotheclassesof writingandreading.Inthisstudywehavefocussedonlyonthe fingermarks visualised and not any palm marks that have potentially been left during writing, normally referred to as writerspalm.Thismodelisbasedonthedistancebetweengrid representationsofthelettersandclassifieseachgridintooneof two classes that represent an activity by using quadratic discriminantanalysis.Themodelwasfirsttrainedusingatraining setconsistingofwrittenandreadletters.Thetrainedmodelwas thenusedtopredicttheclassofanunseentestset.

The previous studyof this model onpillowcases had a few limitations.First,theobjectsinthetrainingsetwerecreatedby exactly the same protocol as the objects that were tested. Furthermore,forpillowcases,itwasnotdeemedrelevanttostudy thedifferencebetweenleft-and right-handeddonorssincethe activitiesofsmotheringandchangingwerecarriedoutusingboth hands.However,forwrittenletters,thehandednessofthedonor maybeanimportantfactor.Inthis study,thelimitationsofthe modelwereinvestigatedbytestingtheinfluenceofthelengthof theletter,theleft-orright-handednessofthedonorandthesizeof thepaperwithanadditionalactivityoffoldingthepaperonthe model’sperformance.

2.Materialsandmethodsexperiment 2.1.Experimentaldesign

The study is divided into two experiments. In the first experiment,westudiedthepossibilityofdifferentiatingbetween

thetwoactivitiesofwritingandreadingbasedonthefingermark locationspresentonA4-sizedlettersforright-handeddonors.For thisexperiment,weusedadatasetof84right-handeddonorswho wrotealetterofregularlengthonA4-sizedpaperanddividedthis set intoa training set(70 %) and a test set (30 %) by random selection. The training set was used to train the classification model,andtheunseentestsetwasusedtostudytheperformance ofthemodel.Wealsotestedtheclassificationperformanceofthe modelwhenonlythefrontsideoftheletterwasusedtodetermine theinfluenceof thebackside oftheletterontheclassification performance.

Tostudythelimitationsofthemodelfordifferentvariationsof the letters, we conducted a second experiment in which the classificationperformanceofthetrainedmodelbasedonA4-sized lettersofregular lengthforright-handeddonorswas testedon threeextratestsets:

a)atestsetconsistingof13right-handeddonorswhowrotea full-pageletter;

b)a testsetconsistingof 12left-handeddonors,ofwhomtwo wroteafull-pageletter;and

c)atestsetconsistingof15donorswhousedA5-sizedpaperand foldedtheirlettersafterwritingthem.

2.2.ExperimentalprotocolforA4-sizedletters

Atotalof110studentsoftheAmsterdamUniversityofApplied SciencesreadaletteronA4-sizedpaperandwrotealetteron A4-sized paper. Theparticipantswerefirst presented witha letter printedononesideofthepaperthatwasplacedonatable.The participantswereaskedtopickuptheletterandreadit.Thisletter was printed bya printer that was loadedbya personwearing gloves with clean, brand-new paper. Next,the participant was givenanew,blanksheetofcleanpaperonwhichtheparticipant wasaskedtowrite.Sinceitwasobservedthattheletterswrittenby theparticipantsweremostlythelengthofhalfanA4-sizedpaper, weasked15participantstowritealetterthatwasthelengthofa fullA4-sizedpaper.

Tovisualize the fingermarks, the letters were treated with indanedione followed by ninhydrin. The results of one donor wereexcludedfromthedatasetduetoheavystainingonaletter asaresultofincorrectapplicationofthevisualizationmethod. After each treatment, the letters were documented using a scannerandeditedusingPhotoshopCSbycroppingtheimages andadjustingthe brightnessforoptimalcontrastbetween the fingermarksandthebackground.Thecustom-madesoftwaretool Lexie translatedthe pictures into grid representationsusinga segmentationprocess,asdescribed bydeRonde,vanAken,de PuitanddePoot[5](supplementarymaterial).Agrid represen-tation of 15
20 cells was used, which was found to be the optimalgridsize.

2.3.ExperimentalprotocolforA5-sizedpapers

To study the influence of the size of the paper on the performanceofthemodel,anexistingdatasetconsistingofgrids representingA5-sizedpaperwasused2 _{.Forthisexperiment,15}

participantswereaskedtoperformthree tasks:toreada letter printed on A5-sizedpaper, to writea threatening letter onA5 sized-paper and towrite a love letter on A5-sized paper. The experimentalprotocolusedforreadingtheletterwasthesameas

1

Caseexample:RbGelderland20December2018,ECLI:NL:RBGEL:2018:5606. Availableviawww.rechtspraak.nl,adatabaseofrandomlyselectedDutchverdicts.

2

FortheA5-sizeddata,wehaveonlyusedthegriddatathatweregeneratedfrom thisstudy.

that described in Section 2.2, whereas in the protocol for the writingscenario,anextrastepoffoldingthepaperwascarriedout byallparticipantsaftertheyfinishedwriting.Forthevisualization of the fingermarks, the paper was treated with indanedione followedbyninhydrinandanadditionaltreatmentwithphysical developer.Theseletterswerephotographedinsteadof scanned, and the photographs were manually transformed into a grid representationof15
20cells.

2.4.Materials

FortheA4-sizedpapers,cleanregularwhitepaperofthebrand CanonBlackLabelZerowasused.FortheA5-sizedpapers,clean, ruledpaperofthebrandStapleswasused.Forthedevelopmentof thefingermarks,1,2-indanedione,ninhydrinandphysical devel-operwereused.Indanedionesolutionwaspreparedbymixing8 mLstocksolutionofZnCl2with100mLof1,2-indanedionestock

solution(100mL),whichresultsinanIND-Znsolution(7,4%v/v). ThestocksolutionofZnCl2ispreparedbyadding0.8gZnCl2to

10mLEtOH,towhich1mLethylacetateand190mLHFE7100was added. The stock solution of 1,2-indanedione is prepared by mixing1.0g1,2-indanedionewith60mLethylacetate,towhich 10mLaceticacidand900mLHFE7100areaddedandstirredfor 20min.Theletterswereimmersedinthesolutionandairdried for 2min. Ninhydrin solution was prepared by mixing 5g of ninhydrinwith45mLofethanol,2mLofethylacetateand5mL aceticacid,towhich1LofHFE7100wasadded.Theletterswere immersedin thesolution andair driedfor2min.TheA5-sized documentswereadditionallytreatedwiththephysicaldeveloper technique as described by Wilson, Cantu, Antonopoulos and Surrency[9].Allsolutionswerepreparedfreshlybeforeuse,from pre-weighedreagentsexceptthesilvernitrate.Theapplicationof thedevelopersolutionoccurredonaslowshakingdeviceinorder tocircumventsilverdepositiononthebottomofthecontainer.All the glassware was salinized before use, to prevent silver deposition on the slightly acidic surface of the glass. ZnCl2

(>99 %), EtOH (absolute, >99 %), Ethyl acetate (>98 %) were obtained fromSigma Aldrich (Zwijndrecht, NL).HFE 7100 was obtained from 3M (Delft, NL). 1,2-indanedione (99 %) was obtained fromBVDA(Haarlem, NL). Silvernitrate, maleic acid, ironnitratemonohydrate,ammonium ironsulfatehexahydrate and citricacid monohydrate were obtained from Merck & Co (Darmstadt, Germany). n-Dodecylamine acetate was obtained fromICN/Hicol(AlisoViejo,CA)andSynperonicNfromBDH/VWR (Amsterdam,theNetherlands).

3.Analysis

All analyses were conducted using the software R, a freely availablesoftwareforstatisticalcomputing,version0.99.896[10]. 3.1.Constructionofthedatasets

Forthedatapre-processing,weusedthedesignshowninFig.1 forboththedatasetsofA4-sizedpapersandA5-sizedpapers.Each picturewastransformedintoagridrepresentationof15
20cells. Inthegridrepresentations,thepresenceofafingermarkinacellis denotedbya1andtheabsenceofafingermarkinacellisdenoted by a 0, resulting in a binary grid that represents the picture. Because the front side and the back side of each letter are considereddependent,wedecidedtoconcatenatethegridsintoa 30
20grid representing one letter, of which the left side representsthefrontsideoftheletterandtherightsiderepresents theback side of the letter. The finaldatasets consisted of one concatenatedgridforeachscenarioperdonor.

3.2.Visualanalysis

In order tovisualize the locationof thefingermarks onthe paperforthetwoscenariosreadingandwriting,wemakeuseof heatmaps.Aheatmapisagraphicalrepresentation,inwhichthe distributionoffingermarksforallgridsofonescenarioisvisually shown by the use of colors. From a heat map, the observed fingermarklocationsthatarecharacteristicforeachscenariocan directlybeobserved.

3.3.Classificationtask

Thepurposeoftheclassi_ficationmodelweusedistoassignthe objects(letters)toaclass(writingorreading)basedonthelocation ofthefingermarksontheletter.Thisisdonebytrainingthemodel withtheuseofatrainingset,forwhichforeveryletterisknownto whichclasstheletterbelongs.Thetrainedalgorithmisthenused topredicttheclassoflettersinanunseentestset.Theaccuracyof themodelisdeterminedbycomparingthemodelpredictionsof thetestsettotheknownclassesofthelettersinthetestset.Fig.2 showsthestructureofthedatasets.Inthefirstphase oftesting whetherwecandifferentiatebetweenthetwoactivitiesofwriting andreadingbasedonthelocationofthefingermarks,weusedthe trainingsetconsistingof59right-handeddonors(denotedinblue in Fig. 2) to train the classification model. An unseen test set consisting of 25 right-handed donors (also denoted in blue in Fig. 2) was used to study the performance of the model. The limitations of the model were studied by testing test sets consisting of different variations of the letters to see the performance of the model trained on right-handed A4-sized lettersofregularlengthonvariationsofthisdata,denotedbytest setsA,BandCinFig.2.

Fig.1.Dataconstructionofthegridsrepresentingtheletters. A.deRondeetal./ForensicScienceInternational315(2020)110443 3

3.4.Classificationmodel

Fortheanalysis,weusedtheclassificationmodeldeRonde,van Aken,dePuitanddePoot[5]proposed.Thisclassificationmodelis basedona similarity and distancemeasure betweengrids. For gridsthatbelongtothesameclassisexpectedthatthereisahigher similaritybetweenthemthanforgridsthatbelongtoadifferent class.Thesimilaritybetweengridsisrepresentedbythesimilarity index(SI)ofSokalandMichener[11]:

SI¼ aþ_nd ð1Þ

Inwhicharepresentsthenumberofcellsforwhichbothgrids containafingermark,drepresentsthenumberofcellsforwhich both grids contain no fingermark and n represents the total numberofcells.TheSIisusedtodeterminetheEuclideandistance (d)betweentwogrids,whichcanbeexpressedas:

d¼ pffiffiffiffiffiffiffiffiffiffiffiffiffi1SI ð2Þ

Thisdistancemeasureisusedtodeterminethedistanceofeach gridtoeachofthegridsinthetrainingsetconsistingofwriting lettersand its distancetoeach of thegirds in thetraining set consisting of reading letters. As a result, each grid can be representedasafeaturevector x1

x2

 

wherex1representsitsmean

distancetothetrainingsetofwritinglettersandx2representsits

mean distance to the training set of reading letters. The classificationisbasedontheexpectationthatagridrepresenting awritingletterhasalowerdistancetothetrainingsetconsistingof writing letters compared to its distance to the training set consistingofreading letters,andviceversa.Thefeaturevectors of all letters form a so-called feature space, which can be partitioned in classes with the use of a classification rule, for which we used Quadratic Discriminant Analysis (QDA). For a furtherexplanationofQDA,wereferthereadertoJames,Witten, HastieandTibshirani[12].

3.5.ProgramminginR

For the implementation of the analysis in R, the following packageswereused:

- Rasterforallgridcomputations[13]; - Ade4tocomputedistancemeasures[14]; - MASStoperformQDA[15];and

- MVNtotestassumptionsforQDA[16]. - ggplot2toproducethefigures[17].

4.Results

4.1.Right-handeddonorsonA4-sizedpaper

Figs.3and4showtheheatmapsforthe59right-handeddonors in the training set for the scenarios of reading and writing, respectively.Theheatmapsshowtheconcatenatedgridsofthe frontsidesandthebacksidesoftheletters.Fig.3showsthatforthe readletters,thefingermarksaremostlydistributedaroundtheleft andrightedges,onbothsidesofthepaper.Theheatmapforthe writtenlettersinFig.4showsthatonthefrontsideofthepaper, thefingermarksaremostlydistributedinanareaonthemiddletop ofthepaperandalongtheleftedge.Thefingermarksonthemiddle topofthepaperarecausedbytheplacementoftherightpalmon thepaperwhilewriting.Thefingermarksaroundtheleftedgeson thefrontsideofthepaperarecausedbyholdingthepaperwiththe lefthand.Therewerealmostnofingermarkobservationsonthe backsideofthepaper.

4.2.Theclassificationmodel

Foreachletterinthetrainingsset,itsmeandistancestothe trainingsetofwrittenlettersandtothetrainingsetofreadletters arecalculated.Fig.5showstheresultingfeaturespace,inwhich thedistancetothetrainingsetofwrittenlettersisplottedonthe x-axisandthedistancetothetrainingsetofreadlettersonthey-axis.

Fig.2.Structureofthedataset.

The reddots representthe read letters, and the bluetriangles representthewrittenletters.Fig.5showsthatthetwoclassesof readingandwritingformtworeasonablyseparateregions,raising theexpectationthataclassificationbasedonaQDAclassifieras usedin[5]maybeappropriateforthisdataset.

FortheuseoftheQDAclassifier,theassumptionisthatboth classesfollowamultivariatenormaldistribution.Thishypothesis istestedwiththeuseoftheMardiatestandbystudyingQQplots. TheMardiatestis usedtoassessmultivariatenormalityfor the separate classes writing and reading based on the Mardia’s multivariate skewness and kurtosis coefficients. For a further explanationoftheMardiatest,wereferthereadertoKres[18].The Mardiatest result showed that thedata werenot multivariate normallydistributed withinthe classes ofwriting and reading. Becausemultivariateoutliersmaybethereasonforviolationofthe multivariateGaussianassumption,westudiedtheQQplotofeach class, a widely used graphical approach to visually evaluate multivariatenormality[16].UsingaQQplotmakesitpossibleto directly observe outliers that may cause a violation of the multivariatenormalityassumption.FromtheQQplotshownin Fig.6fortheclassofwriting,weobservedthatoneoutlierdistorted thenormalityassumption.Asidefromthisoutlier,theMardiatest shows that the data are indeed distributed following the multivariate Gaussian assumption.TheQQ plotfor theclass of reading,showninFig.7,showsthreepossibleoutliers.Asidefrom themostextremeoutlierintheupperrightcorner,theMardiatest

showsthatthedataarealsodistributedfollowingthemultivariate Gaussianassumption.

4.3.Evaluationofthemodel

Table1showstheconfusionmatrixfortheQDAclassificationof thetestsetconsistingof25right-handeddonorswritingaletterof regularlengthandreadingaletter.Themodelclassified49ofthe 50letterscorrectly,representinganaccuracyof98.0%.Oneread letterwasmisclassi_fiedasbeingawrittenletter.Fig.8showsa visualrepresentationoftheconcatenatedgridofthefrontsideand thebacksideofthisletter,indicatingthatthefingermarksonthis letterarearoundtheedges,aswewouldexpectfromtheheatmap forreadletters,butadditionalfingermarksarefoundinthemiddle ofthefrontofthepaper,indicatedbyablackcircle.Weexpectthat thesefingermarksinthemiddleofthepapercausedthemodelto classifyitasawrittenletter.

SinceQDAclassificationisbasedontheposteriorprobabilities, theuseofaQDAclassifierallowsforthecalculationofalikelihood ratio for each object present in the test set using theformula

PrX¼x j Y¼writingÞ 

PrX¼x j Y¼readingÞ , in which x represents a feature vector of the

corresponding letter. Fig. 9 shows the log10 likelihood ratio

distributionsforbothclassesofboththetrainingsetandthetest set.Thedistributions fortheclasses ofwriting andreading are

Fig.4.Heatmapforthetrainingsetofthewritingscenario.

Fig.5.Featurespaceforthetrainingsetconsistingofright-handeddonors.

Fig.6. QQplotfortheclassofwriting.

quitewellseparated,althoughsomelettersobtainarelativelylow likelihoodratioinfavorofthewrongclass.Oneoftheseistheletter showninFig.8,and theotherthreeletterswerepresentinthe trainingsetonwhichthemodelistrained.Fromthedistributions, weobservethatthelikelihoodratiosreachextremevalues.This willbefurtherexplainedinthediscussion.

4.4.Onlythefrontsideoftheletter

BecausetheheatmapforthewritingscenarioinFig.4shows thattherewerealmostno_fingermark observationsontheback sideofthewrittenletters,thequestionofwhetherthemodelonly usestheemptybacksideoftheletterasanindicationfortheclass of writing or reading might arise. This would make the applicabilityof themodel questionable ifthe activitiesslightly

change such that the back side of the letter also contains fingermarksinthewritingscenario.Toaccountforthis,wetested theperformanceofthemodelwhenonlyusingthefrontsideofthe letters.TheconfusionmatrixshowninTable2demonstratesthat whenusingonlythefrontsideoftheletters,themodelclassified 48ofthe50letterscorrectly,anaccuracyof96%.Oneadditional readletterwasmisclassifiedasbeingawrittenletter.Theseresults showthatthemodelisabletoclassifythelettersbasedononlythe frontsideoftheletter;however,theaccuracyincreasesslightly whentakingthedependencybetweenthefrontandthebacksides ofthelettersintoaccountbyconcatenatingbothsides.

4.5.Full-pageletters(testsetA)

Fortheanalysisofthefull-pageletters,atestsetof13full-page letterswaspredicted bytheclassification modeltrainedonthe trainingsetconsistingofright-handeddonorswhowrotelettersof regularlength.Fig.10showstheheatmapforthefull-pageread letters,andFig.11showstheheatmapforthewrittenletters.The heatmapforthereadlettersshowsthesamecharacteristicsasthe heatmapforthetrainingsetshowninFig.3.Theheatmapforthe written lettersshows a somewhatdifferent distribution of the fingermarksthantheheatmapforthetrainingsetshowninFig.4. Theareaonthemiddletopofthepaperobservedfortheregular length lettersis more spread over thefront side of the letter. However,theheatmapsshowsomewhatthesamecharacteristics as theheat mapsused for thetraining set,which leadstothe expectationthatthistestsetwillbequitewellpredictedbythe model.

Table3showstheconfusionmatrixforthetestset.Theresults showthattheactivityofreadingandtheactivityofwritingwere predicted correctly in allcases, although the heat map for the writtenletterslookedslightlydifferent.Thisisbecausethewritten lettersarestillquitedifferentfromthereadletters.Whereasfor the writing scenario, fingermarks are mostly observed in the middleofthepaperandalmostnofingermarksareobservedonthe backsideofthepaper,thefingermarksforthescenarioofreading arestillmostlyplacedalongtheedgesofthepaperonbothsidesof thepaper.Theseresultsshowthatwritingafull-pageletterinstead of a shorter letter on A4-sized paper does not influence the performanceoftheclassificationmodel.

4.6.Left-handeddonors(testsetB)

Fortheanalysisofthelettersoftheleft-handeddonors,weused atestsetconsistingof12readandwrittenletters,ofwhichtwo donorswrotefull-pageletters.Thistestsetwasalsopredictedby theclassificationmodeltrainedonthetrainingsetconsistingof right-handeddonorswhowrotelettersofregularlength.Sincethe resultsinSection4.5showthatthelengthoftheletterdoesnot in_fluencetheperformanceofthemodel,thesetwofull-pageletters werealsoincludedintheleft-handedtestset.Figs.12and13show theheatmapsfortheleft-handeddonorsfortheclassesofreading andwriting,respectively.Fig.12showsthatforthereadletters, left-handeddonorshaveasimilarpatternasright-handeddonors. Fig.13showsthatforthewrittenletters,thefingermarksof left-handed donors are distributed over thewhole page, while for right-handeddonors,thefingermarksweremostlydistributedin anareaonthemiddletopoftheletterandalongtheleftedge.Since theheat maps for theleft-handed donorsshowsomewhat the samecharacteristicsastheheatmapsforthefull-pagelettersand thefull-pageletterswereallcorrectlypredicted,weexpectthatthe modelwillalsobeabletopredictthecorrectclassofmostofthe left-handeddonors.

Table4showstheconfusionmatrixforthetestsetconsistingof left-handed donors. The resultsshow that all read letters and

Fig.7.QQplotfortheclassofreading.

Table1

Confusionmatrixforthetestsetconsistingofright-handeddonorsonA4-sized paper.

Testset Reading Writing Readingpredicted 24 0 Writingpredicted 1 25

writtenletterswerepredictedcorrectly.Apparently,trainingthe modelwithadatasetconsistingofright-handedlettersdoesnot affectthe classificationof the left-handed letters,although the fingermarkpatternsdifferforthewritingscenario.

4.7.A5-sizedletters(testsetC)

Fortheanalysisofthesizeoftheletters,atestsetconsistingof 15read lettersand30 writtenletterswasalsopredictedbythe classificationmodeltrainedonthetrainingsetconsistingof right-handeddonorswhowrotelettersofregularlength.Figs.14and15 showtheheatmapsfortheseA5-sizedlettersforthescenarioof readingandthescenarioofwriting,respectively.Fig.14showsfor theA5-sizedreadletters,thefingermarksaremostlydistributed

alongtheedgesonbothsidesofthepaper,aswealsoobservedfor theA4-sizedreadletters.Additionally,somedonorsplacedtheir handsaroundthebottomofthepaper,whichwasalsoobservedfor theA4-sizedreadlettersinFig.3.TheheatmapfortheA5-sized written letters in Fig. 15 shows that the distribution of the fingermarksisclearlydifferentfromthedistributionweobserved fortheA4-sizedwrittenlettersinFig.4,forwhichweobserved that onthefrontside ofthepaper, thefingermarksare mostly distributedonthemiddletopoftheletterandalongtheleftedge. For the A5-sizedwritten letters,we observethat this areahas shiftedtothemiddlebottomofthepaperandisconcentratedon theentirewidthofthepaper,andalmostnofingermarksarefound inthemiddletopareaoftheletter.Anexplanationforthismaybe that the palm is placed lower onthepaper since the paper is smaller.Furthermore,thefingermarksaroundtheedgescausedby holdingthepaperwiththeotherhandmayinterferewiththepalm placement becausethe paper is narrower, sothe areas almost overlap.The_fingermarksonthebacksideofthewrittenletterscan beexplainedbytheadditionalactivityoffoldingthepaperbeforeit was putbackonthetable. Thisalsodiffersfromtheheatmap observed for the A4-sized written letters, since almost no fingermarkswerefoundonthebacksideofthepaper.

Fortheclassification,wetestedatestsetconsistingofall15 readlettersandall30writtenletters(lovelettersandthreatening letters).TheconfusionmatrixinTable5showsthatthemodelhad anaccuracy645%.All15readletterswerepredictedcorrectly,but the model had difficulty classifying the written letters. One explanationforthemodel’spoorclassificationaccuracyforthe A5-sizedlettersmightbetheinfluenceoftheadditionalpost-activity offoldingthepaperafterwritingonit.Sinceweexpectthatfolding thepapermostlyaffects_fingermarkstobepresentonthebackside oftheletter,theclassificationwasrepeated withonlyusingthe frontsidesoftheletters.Table6showstheclassificationresults. Althoughthemodelaccuracyincreasedto75.6%,themodelstill wronglypredicted11ofthewritingletters.Apossibleexplanation forthiswillbefurtherexplainedinthediscussion.

OnewaytoachievehigheraccuracyforA5-sizedlettersmaybe toexpandthetrainingsetconsistingofA4-sizedlettersbyadding A5-sizedletterstotrainthemodelforA5-sizedlettersaswell.For thisanalysis,70%ofthefirst15donorswhoreadandwrotealove letteronA5-sizedpaperwereaddedtothetrainingset(11donors). Theremaining30%ofthedonorsrepresentthetestset(4donors), together with the extra 15 threatening letters written by the donors. For this, we assumed that there is no difference in

Fig.9. Likelihoodratiodistributionforthecompletedataset.

Table2

Confusionmatrixforthetestsetconsistingofright-handeddonorsonA4-sized paperusingonlythefrontsideofthepaper.

Testset Reading Writing Readingpredicted 23 0 Writingpredicted 2 25

Fig.10.HeatmapofreadlettersfortestsetAconsistingoffull-pageletters.

fingermark deposition between the type of message (love or threatening)thatiswritten.Thenewtrainingsetwasusedtotrain the model, and afterward, the performance of the model was testedontheunseentestset.Table7showstheconfusionmatrix, whichindicatesthatfivewrittenlettersarewronglyclassifiedas read letters, resulting in an accuracy of 78.3 %, which is significantly increased compared to the accuracy of 64.4 % obtainedforatrainingsetconsistingofonlyA4-sizedletters. 5.Discussionandconclusion

Thisresearchstudiedwhetherthemodelfortheactivitylevel analysisofthelocationoffingermarksproposedbydeRonde,van Aken,de Puitand de Poot[5] couldalsobe usedonletters to distinguishtheactivityofwritingfromthealternativeactivityof reading.Theresultshaveshownthatthemodelcouldverywellbe

applied to fingermarks on letters of right-handed donors to differentiate between the two activities, with a classification accuracyof98.0%.Furthermore,weshowedthatthelengthofthe writtenletterandthehandednessofthedonordidnotinfluence theperformanceoftheclassificationmodel.Forlettersonasmaller sized paper (A5)and withan additional activityof folding the paperafterwritingonit,themodelaccuracydecreasedto64.4%.If thetraining set consistingof A4-sizedlettersused totrain the model is expanded with A5-sized letters, the model accuracy increases to 78.3 %. These results show that the location of fingermarks onlettersprovides valuableinformation aboutthe activitythatwascarriedout.

Despitethefactthattheheatmapforthewrittenlettersofthe left-handeddonorsshowedsignificantdifferencesfromtheheat map of written letters of the right-handed donors, all letters written byleft-handed donors werecorrectly predicted by the

Fig.11.HeatmapofwrittenlettersfortestsetAconsistingoffull-pageletters.

Table3

ConfusionmatrixfortestsetAconsistingoffull-pageletters.

Testset Reading Writing Readingpredicted 13 0 Writingpredicted 0 13

Fig.12.HeatmapofreadlettersfortestsetBconsistingoflettersbyleft-handed donors.

Fig.13.HeatmapofwrittenlettersfortestsetBconsistingoflettersbyleft-handed donors.

Table4

ConfusionmatrixfortestsetBconsistingoflettersbyleft-handeddonors. Testset Reading Writing Readingpredicted 12 0 Writingpredicted 0 12

modeltrainedonright-handeddonors.Thedifferencein finger-markpatternsforthescenarioofwritingbetweenleft-and right-handeddonorsmaybecausedbythevariationinhandplacement thatwasobservedforleft-handedpeoplewhentheywerewriting letters.Thereasonthattheclassificationwasnotaffectedbythese differentfingermarkpatternsisbecausethegridsthatrepresent thewrittenlettersoftheleft-handeddonorsstillhaveadistinctive patternfromthatofthereadletters.However,careshouldbetaken when testing left-handed donors on a right-handed–trained model.Sincewetestedonlyasmallsampleofleft-handeddonors (12donors),thepossibilityexiststhatnot allvariationsof left-handedwritingareincorporatedinourdataset,andvariationsthat arenotrepresented maybeclassifiedincorrectly.To correctfor this,alargersampleofleft-handeddonorsshouldbetested.

ThemodeltrainedonA4-sizedletterswronglypredictedmore than half of the written A5-sized letters. There can be two explanations:thedifferenceinactivitythatiscarriedoutandthe differenceinthesizeofthepaper.Anadditionalactivityoffolding thepaperwascarriedoutbytheparticipantsintheexperiment withA5-sizedpaper,causingtheappearanceoffingermarksonthe

backsideofthepaperinthewritingscenario.Sincetheresultsfor testingonlythefrontsideofthelettersfortheA5-sizedpapers have shownthat still 36.7%of thewrittenlettersarewrongly predictedbythemodel,thisextraactivityoffoldingthepaperdoes notexplainthepoorclassificationresultsonitselfandweexpect thatthedifferenceinthesizeofthepaperbetweenthetrainingset (A4)andthetestset(A5)isanimportantfactortoconsider.Since themodelisconstructedsuchthatthetrainingsetandthetestset havetocontaingridsofsimilardimensions,thenumberofcellsis thesameforbothsizesofletters(15
20),butthesizeofthecells differsbetweenthegridsfortheA4-sizedletters(1.5cm
1.5cm) andthegridsof theA5-sizedletters(1cm
1cm).However,the sizesofthefingermarksdonotchangewhenusingasmallerpaper, soonefingermarkmayfillmorecellsinthegridrepresenting A5-sizedpaperthanitdoesinthegridrepresentingA4-sizedpaper. Thismeansthatifthesizeoftheobjectspresentinthetrainingset significantlychangesfromthesizeoftheobjectbeingtested,the trainingsetwillprobablynotberepresentativeofthetestset.One solutionmaybetoexpandthedatasetwithnewdata,aswehave shown fortheA4-andA5-sizedletters.Anothermaybetonot workwithsquaredcellsbuttochooselargerareasontheletters thatarerepresentativefortheactivitiesofreadingandwritingand tostandardizedifferentsizesofpapertothisrepresentation.This maybeatopicforfurtherresearch.Fornow,weproposeexpanding thetrainingsetsothatthedimensionsoftheobjecttobetestedare alsorepresented.

Thelikelihoodratiovaluesthatwereprovidedasoutputfrom ourmodelareinahigherandlowerorderthenexpected,giventhe sizeofourdataset.SincetheassumptionsfortheuseofQDAwe havemadearebasedonalimiteddataset,wehavenoproofofthe applicabilityof QDAbeyondourdataset,which meansthatthe likelihood ratios provided by the system may be sensitive to extrapolation errors [19]. A solutionfor this is tocalibrate the likelihood ratio system that results fromthe model. There are several methods for performing this calibration [20]. Further researchisneededtodeterminewhichcalibrationmethodismost suitableforourdatasettoobtainlikelihoodratiovaluesthatcanbe directlyappliedtocasework.

Inthisresearch,thesourcelevelinformationofthefingermarks isnottakenintoaccount.Thismeansthatthemodelisnotonly basedonidentifiablefingermarkspresentontheletters,butalso on additional stains such as smears that were visualized. We decided to not work only with identifiable fingermarks since smears and stains are also a direct result of the activity. For example,a smearcreatedbytheplacementofthepalmonthe paperduringwritingmaynotresultinafingermarksuitablefor identification.However,thissmearprovidesinformationaboutthe placementofthehandduringtheactivity.Adrawbacktothisis that careshouldbetakenwhenusing thismodel onvisualized fingermarks: if the fingermark visualization method is not correctlyapplied, causing the appearanceof drops or spots on theobjectofinterest,thesedropsandspotswillalsobeinterpreted asmarks.

In this experiment, we exclusively tested the activities of writing andreadinga letter for the trainingset used, without testinganypre-orpost-activitiessuchasgrabbingthepaperor folding the paper. As a consequence of this, if this dataset is appliedincasework,itisofgreatimportancetoclearlystatethe activityhypothesestested,toknowexactlywhatactivitiesareat stake. Aswehaveshown,anyadditionalpre-orpost-activities mayslightlyinfluencetheperformanceofthemodel;addingan extrastepoffoldingthepapermayinfluencetheperformanceof themodelifthemodelistrainedbasedonatrainingsetthatdoes not involve this extra folding step. Thus, when applying this datasettocasework,itshouldbeconsideredwhetherthetraining setshouldbeexpandedwithappropriateexamplesofadditional

Fig.15.HeatmapofwrittenlettersfortestsetCconsistingofA5-sizedletters.

Table5

ConfusionmatrixforclassifyingtestsetCconsistingofA5-sizedlettersbasedona trainingsetofA4-sizedletters.

TestsetA5size Reading Writing Readingpredicted 15 16 Writingpredicted 0 14

Table6

ConfusionmatrixforclassifyingtestsetCconsistingofA5-sizedlettersbasedona trainingsetofA4-sizedletters,whenusingthefrontsideoftheletters.

TestsetA5size Reading Writing Readingpredicted 15 11 Writingpredicted 0 19

Table7

ConfusionmatrixforclassifyingatestsetconsistingofA5-sizedlettersbasedona trainingsetofA4-andA5-sizedletters.

TestsetA5size Reading Writing Readingpredicted 4 5 Writingpredicted 0 14

activitiesifanyextraactivitieswerecarriedoutintheparticular case.

Another factor to take into account when applying the generated data to casework is that in this study, we clearly separatedtheactivitiesofwritingandreading.Inrealcasework, thismaynotalwaysbeexpectedandtheseactivitiescouldhave occurred successively. However, by studying these activities separately,wehaveshownthatbothactivitiescauseadistinctive fingermarkpatternontheletter.Theheatmapsshowparticular areason theletter thatare representativeof writingtracesor readingtraces,makingitpossibletoselectthetracesonaletter thatarespeci_ficfortheactivityofwritingorfortheactivityof reading.Inthisway,theinvestigationcanfocusonthemarksthat provideanindicationofacertainactivity,andifnoidentifiable fingermarksarefound,atargetedsamplingforDNAispossible. Thefocusofthisstudywastodistinguishtheactivityofwriting aletterfromthealternativeactivityofreadingaletter,basedonthe variablelocation ofthefingermarks. AsdiscussedbydeRonde, Kokshoorn, de Poot and de Puit [4], there are several other variables that maybe of interest when evaluating fingermarks given activity level propositions.The data fromthe conducted experimentshowsthatthepresenceofalargeareaonthefront sideoftheletter,causedbyplacingthepalmonthepaperwhile writingaletter,isprobablyverydistinctivebetweenthedisputed activities writing and reading, raising the suspicion that the presenceofapalmprintonthefrontsideofthepapermayprovide valuableinformationontheactivitythatwascarriedoutwiththe paper. The variable area of friction ridge skin that left the fingermarkmay bean interesting variable for further research intofingermarksgivenactivitylevelpropositionsonletters.

Withthisresearch,wehaveconfirmedthatthemodelproposed [5]couldverywellbeappliedtoanytwo-dimensional itemfor which it is expected that different activities lead to different fingermarklocations.Insteadofusingpainttodirectlyvisualizethe fingermarksaswasdoneinthepreviousstudyonpillowcases[5], conventionaltechniquestovisualizefingermarksonpaperwere used, resulting in traces that represent fingermark traces that wouldbe obtainedin realcasework.We now haveaccess toa databaseconsistingofwrittenandreadlettersonA4-sizedpaper and A5-sized paper that represents the separate activities of readingandwriting.Nowthatwehaveshownthatthemodelis verywellabletodistinguishbetweentheactivitiesreadingand writing,thenextstepforimplementationtocaseworkwillbeto performfurtherresearchintomorerealisticscenariossuchas pre-andpost-activitiesorcarryingoutreadingandwritingsuccessively byperformingapseudo-operationaltrialonlettersthatwerenot collectedunderlabconditionstoseehowthemodelperformson morerealisticcaseworkmaterials.

CRediTauthorshipcontributionstatement

AnoukdeRonde:Conceptualization, Methodology,Software, Validation,Datacuration,Formalanalysis,Investigation,Writing -original draft,Writing- review &editing, Visualization, Project administration.MarjavanAken:Methodology,Software, Valida-tion,Data curation,Writing- originaldraft,Writing- review& editing,Visualization.ChristianneJ.dePoot:Conceptualization, Methodology,Writing-review&editing,Supervision.Marcelde Puit:Conceptualization,Methodology,Writing-review&editing, Supervision.

References

[1]S.Willis,L.McKenna,S.McDermott,G.O’Donell,A.Barrett,B.Rasmusson,A. Nordgaard, C. Berger, M. Sjerps, J. Lucena-Molina, ENFSI Guideline for Evaluative Reporting inForensic Science, EuropeanNetwork of Forensic ScienceInstitutes,2015.

[2]A.Biedermann,C.Champod,G.Jackson,P.Gill,D.Taylor,J.Butler,N.Morling,T. Hicks, J. Vuille, F. Taroni, Evaluation of Forensic DNA Traces When Propositions of Interest Relate to Activities: Analysis and Discussion of RecurrentConcerns,Front.Genet.7(2016)215.

[3]R.Cook,I.W.Evett,G.Jackson,P.Jones,J.Lambert,Ahierarchyofpropositions: decidingwhichleveltoaddressincasework,Sci.Justice38(1998)231–239. [4]A. de Ronde, B. Kokshoorn, C.J.de Poot, M.de Puit, The evaluation of fingermarksgivenactivitylevelpropositions,ForensicSci.Int.302(2019). [5]A.deRonde,M.vanAken,M.dePuit,C.dePoot,Astudyintofingermarksat

activitylevelonpillowcases,ForensicSci.Int.295(2019)113–120. [6]S.Flight,C.Wiebes,Handschriftonderzoekinhetkadervan

terrorismebes-trijding,WODC,2016.

[7]Rv.StephenPort,SentencingRemarksofMrJusticeOpenshaw,Judiciaryof EnglandandWales,CentralCriminalCourt,2016.

[8]R.N.Morris,ForensicHandwritingIdentification:Fundamentalconceptsand principles,Academicpress,2000.

[9]J.D.Wilson,A.A.Cantu,G.Antonopoulos,M.J.Surrency,Examinationofthe stepsleadinguptothephysicaldeveloperprocessfordevelopingfingerprints, J.ForensicSci.52(2)(2007)320–329.

[10]RCoreTeam,R:ALanguageandEnvironmentforStatisticalComputing,R FoundationforStatisticalComputing,Vienna,Austria,2016.

[11]R.R.Sokal,C.D.Michener, AStatisticalMethodforEvaluating Systematic Relationships,Univ.KansasSci.Bull.38(22)(1958).

[12]G.James,D.Witten,T.Hastie,R.Tibshirani,AnIntroductiontoStatistical LearningwithApplicationsinR,Springer,NewYork,2013.

[13]R.J.Hijmans,Raster:GeographicDataAnalysisandModeling,(2016). [14]S.Dray,A.B.Dufour,Theade4package:implementingthedualitydiagramfor

ecologists,J.Stat.Softw.22(4)(2007)1–20.

[15]W.N.Venables,B.D.Ripley,ModernAppliedStatisticswithS,4thed.,Springer, NewYork,2002.

[16]S.Korkmaz,D. Goksuluk,G. Zararsiz,MVN: AnRPackage for Assessing MultivariateNormality,RJ.6(2)(2014)151–162.

[17]H.Wickham,ggplot2:ElegantGraphicsforDataAnalysis,Springer,2016. [18]H.Kres,TheMardia-TestforMultivariateNormality,Skewness,andKurtosis

Tablesby,in:K.V.Mardia(Ed.),StatisticalTablesforMultivariateAnalysis, Springer,NewYork,NY,1983.

[19]P.Vergeer,A.vanEs,A.deJongh,I.Alberink,R.Stoel,Numericallikelihood ratiosoutputtedbyLRsystemsareoftenbasedonextrapolation:Whentostop extrapolating?Sci.Justice56(6)(2016)482–491.

[20]G.S.Morrison,N.Poh,Avoidingoverstatingthestrengthofforensicevidence: Shrunklikelihoodratios/Bayesfactors,Sci.Justice58(2018)200–218.