Consistency issues in the best worst method

(1)

Delft University of Technology

Consistency issues in the best worst method

Measurements and thresholds

Liang, Fuqi; Brunelli, Matteo; Rezaei, Jafar

DOI

10.1016/j.omega.2019.102175

Publication date

2020

Document Version

Final published version

Published in

Omega (United Kingdom)

Citation (APA)

Liang, F., Brunelli, M., & Rezaei, J. (2020). Consistency issues in the best worst method: Measurements

and thresholds. Omega (United Kingdom), 96, [102175]. https://doi.org/10.1016/j.omega.2019.102175

Important note

To cite this publication, please use the final published version (if applicable).

Please check the document version above.

Copyright

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons. Takedown policy

Please contact us and provide details if you believe this document breaches copyrights. We will remove access to the work immediately and investigate your claim.

This work is downloaded from Delft University of Technology.

(2)

ContentslistsavailableatScienceDirect

Omega

journalhomepage:www.elsevier.com/locate/omega

Consistency

issues

in

the

best

worst

method:

Measurements

and

thresholds

✩

Fuqi

Liang

a,∗

_,

_Matteo

_Brunelli

b

_,

_Jafar

_Rezaei

a

a Faculty of Technology, Policy and Management, Delft University of Technology, Delft, the Netherlands b Department of Industrial Engineering, University of Trento, Trento, Italy

a

r

t

i

c

l

e

i

n

f

o

Article history: Received 1 July 2019 Accepted 19 December 2019 Available online 20 December 2019

Keywords:

Best worst method Consistency threshold Consistency measurement

a

b

s

t

r

a

c

t

TheBest-WorstMethod(BWM)usesratiosoftherelativeimportanceofcriteriainpairsbasedonthe as-sessmentdonebydecision-makers.Whenadecision-makerprovidesthepairwisecomparisonsinBWM, checkingtheacceptableinconsistency,toensuretherationalityoftheassessments,isanimportantstep. AlthoughboththeoriginalandtheextendedversionsofBWMhaveproposedseveralconsistency mea-surements,there aresomedeﬁciencies, including: (i)the lack ofamechanism to provideimmediate feedbacktothedecision-makerregardingthe consistencyofthepairwisecomparisonsbeingprovided, (ii)theinabilitytoconsidertheordinalconsistencyintoaccount,and(iii)thelackofconsistency thresh-oldstodeterminethereliabilityoftheresults.Todealwiththeseproblems,thisstudystartsbyproposing acardinalconsistencymeasurementtoprovideimmediatefeedback,calledtheinput-basedconsistency measurement,afterwhichanordinalconsistencymeasurementisproposedtocheckthecoherenceofthe orderoftheresults(weights)againstthe orderofthepairwisecomparisonsprovided bythe decision-maker.Finally,amethodisproposedtobalancecardinalconsistencyratiounder ordinal-consistentand ordinal-inconsistentconditions,todeterminethethresholdsfortheproposedandtheoriginalconsistency ratios.

1. Introduction

The Best Worst Method (BWM), which is a Multi-Criteria De-cision Making (MCDM) method that was recently developed by Rezaei [37], uses ratios of the relative importance of criteria in pairwise comparisons provided by a decision-maker (DM), based on two evaluation vectors: the Best criterion against the Other criteria, and the Other criteria against the Worst criterion. The weights ofthecriteriaare obtainedbysolving anonlinear[37]or a linear model [38]. Compared to one of the mostpopular pair-wise comparison-based MCDM methods, Analytic Hierarchy Pro-cess(AHP),BWMrequiresfewercomparisondata,whilebeingable to generate more consistent comparisons, allowing it to produce more reliable results accordingto previous analyses [37]. Thanks to its simplicityandreliability, BWMhas beenwidely applied to addressahostofdifferentproblems[29,39,49].Formoredetailed

✩_{This paper was processed by Associate Editor Dias.} ∗ _{Corresponding author.}

E-mail addresses: f.liang-2@tudelft.nl (F. Liang), matteo.brunelli@unitn.it (M. Brunelli), J.Rezaei@tudelft.nl (J. Rezaei).

information,readersare referred toa recentsurvey on theBWM

[32].

BWMandother pairwise comparisonsmethods, like AHPand ANP(AnalyticalNetworkProcess),arebasedonaDM’sevaluations ofthe relativeprioritiesof thedecision-making elementsas cap-turedina completepairwise comparisonmatrix [41],incomplete pairwise comparison matrix [19] or vectors[37]. One ofthe ad-vantages of using pairwise comparisons is that they allow us to estimatetheinconsistencyofaDM’spreferences.Usually,the con-sistencylevelofthejudgementsisrelatedtotherationalityofthe DMandhis/herabilitytodiscriminatebetweencriteria/alternatives

[21]. The DM’s judgments have to meet the cardinal transitiv-ityconditiontobe perfectlyconsistent;otherwise, theDMisnot fullyconsistent,whichmayimplysomeirrationalityintherelative weightestimates.

Tocheckhowinconsistent(deviatingfromtheconditionoffull consistency)afullsetofpairwisecomparisonsmaybe,Saaty[40], in their seminal work on the AHP, proposed a consistency mea-surement (Saaty index), but since then, many other consistency indices have been proposed [10]. Basically, the existing consis-tency measurements can be divided into two groups: the input-based measurements and output-based measurements [28]. The

https://doi.org/10.1016/j.omega.2019.102175

(3)

measurements in the former group are based on the input, i.e. preferencesassignedtopairwisecomparisons,e.g.Koczkodajindex

[24],whiletheoutput-basedconsistencymeasurementsarebased ontheweights orrankings. Inthisgroup, thereare, forinstance, Saaty’sindex[40] andthe geometric consistency indexproposed byCrawfordandWilliams[13].

The consistencymeasurements mentionedabove wereinitially designedforcompletepairwisecomparisonmatricesandwe can-not use them to measure the consistency degree of incomplete pairwise comparison matrices where some judgments are miss-ing[19].To adaptthe consistency indicesto incompletepairwise comparison matrices, one of the most popular approaches is to completethepairwisecomparisonmatrices[16,47]andthen mea-sure their consistency in the traditional manner [19,28]. Instead ofcompletingthematrix,agraph-theoreticapproachcanbeused to generateall possible preferences by enumerating all spanning trees,afterwhichthevarianceofthesepreferencescanbeusedas ameasure ofinconsistency [6,31,44].Replacing triadswith cycles

[28]isanotherwaytoestimatetheinconsistency.

One might seeBWMas aspecial caseof incompletepairwise comparisonmatrix.Althoughthemethodonlyusesaspecific sub-set of 2n-3 comparisons gathered in two representative vectors, these preferences can be represented equivalentlyby an incom-pletepairwisecomparisonmatrix.Onecouldarguethat wecould thencompletethetwovectorstocreateafullmatrixandmeasure theinconsistencybyusingtheapproachesmentionedabove. How-ever,notonlywillthatmakethemeasurementmoredifficult (un-realistic),it willalso destroy thesimplification (non-redundancy) philosophyembeddedinBWM.Therefore,tochecktheconsistency byusingthisspecificmethod,Rezaei [37]proposed aconsistency measurement (sometimes referred to as inconsistency measure-ment) inthe original version ofBWM. Later, the extended BWM methods also provided corresponding consistency measurements similartotheoriginalconsistencymeasurement.Forexample,Mou etal.[35]extendedBWMtoincludeintuitionisticfuzzy multiplica-tive preference relations, and provided a new definition for the consistency algorithm to check consistency, while Guo andZhao

[18]proposedaconsistencyratio(alsoreferredtoasinconsistency ratio)forfuzzyBWM,andAboutorabetal.[1]explaineda corre-spondingconsistencyratiofortheZ-numbersBWM.

However, the existing studies on BWM lack a metric/tool to provide the DM/analyst with immediate feedback regarding the consistency of the pairwise comparisons. The consistency ratios obtainedby the existing consistency measurements of BWM are basedontheoutputs insteadofdirectlyontheinputs.A DMcan onlyobtain theconsistency ratioandcheck the consistency after theentireoptimizationprocessiscompleted,byusingtheexisting consistencymeasurements.However,ithasbeenshownthat con-frontingtheDMwiththeinconsistenciesinhis/herassessments af-terhe/shehasalreadygonethroughtheentireelicitationprocessis ineffective[34].Inaddition,theconsistencyratiosobtainedbythe originalBWM,graph-theoreticapproach[6,31,44]andthemethods ofreplacingtriadswithcycles[28]areoverallindicatorsthatshow theconsistencyofthepairwisecomparisonsystemasa whole,so theycannothelptheDMlocatetheirmostinconsistentjudgments. A proper consistency measurement should indeed assist the DM inidentifyingthemostinconsistentcomparisons[14]andachieve suﬃciently consistent preferences [17,36]. Although some input-basedconsistency measurements forgeneralincomplete pairwise comparisonmatrices, includingthe Koczkodajindex[24] andthe SaloandHämäläinenindex[43],canbe appliedtoBWM, someof theirpropertiesarenot asdesirableaswe expected, asdiscussed inSection3.

Moreover, the existingstudies onconsistency measurement in theBWMthus farfail totake ordinal consistencyinto considera-tion.Consistencyinpairwisecomparisonscanbedividedintotwo

categories: cardinalconsistency andordinal consistency [45]. The existing consistencyratiosof BWMonly measurecardinal consis-tency. However, even ifthe judgementshave a highlevel of car-dinalconsistency,they canbe stillcontradictory, accordingtothe researchofKwiesielewiczandVanUden[30].Thecontradictionis caused bythe violationof ordinal consistency, i.e.there isa dis-crepancy in the criteria importance rankings obtained from the two pairwise comparison vectorsin BWM. If the preferences are ordinal-consistent,thefinal rankingwill notchangewiththe car-dinalconsistency ratio,only the intensitycould vary; butif they are ordinal-inconsistent, a change in the cardinal consistency ra-tiocould affect thefinal ranking [45].Thus, inorder toensure a DMprovides a stablejudgement, itis importanttocheck his/her ordinal consistency status, andindicate to what extent the ordi-nalconsistencyhasbeenviolated.Thereareseveralordinal consis-tencymeasurementsforthecompletepairwisecomparison matri-ces,liketheordinalcoefficientproposedbyJensenandHicks[22], thedissonancemeasurementproposedbySirajetal.[45,46]. How-ever, they cannot be applied to incomplete pairwise comparison matricesorthetwovectorsusedinBWM.

Furthermore,thereisno thresholdfortheconsistencyratioof BWMinexisting literature.AlthoughBWMhasbeenwidely used and the consistency measurements help a DM check the relia-bility of his/her preferences, the absence of threshold associated withtheexistingconsistencymeasurementsmakesithardto pro-vide a meaningful interpretation. Without a consistency thresh-old, the DM/analyst is left with the major problem of having to decide when his/her judgments should be revised and when it shouldbeaccepted,nottomentiontheconsiderationofthe num-ber of criteria andthe scale of evaluation, making the situation even more complicated. The 10% rule of thumbof AHPhas long beencriticised [5,7,33],andeven Saatylater suggestedadditional threshold values of 5% and 8% for 3 and 4 criteria, respectively

[42].Althoughsome othermethods havebeenproposed to deter-mineconsistencythresholds[2,4,33],mostofthemareappliedin completepairwisecomparisonmatrices,whichcannotbeused di-rectlyforincompletepairwisecomparisonmatrices. Thus, design-ingathresholddeterminationalgorithmforBWMcanﬁllthisgap. Assuch,thecontributionofthisstudyisthreefold:(i) Develop-ingamechanismdesignedtoprovideaDMwithimmediate feed-back regardinghis/her consistencystatus andmaking the elicita-tionprocessmoreeffective.Tothisend,weproposeaninput-based consistencymeasurement,whichissimple touseandhasseveral desirable properties; (ii) Developing an ordinal consistency ratio thatshowsaDM’sviolationlevelinvolvingordinalconsistencyand complementsthecardinalconsistencymeasurement.Withthis ra-tio,a DM can revisehis/her judgmentsto meet theordinal con-sistency condition, which is a minimum requirement for a logi-calandrationalDM;(iii)Themostsigniﬁcantcontributionofthis studyistoestablishthresholdsfortheconsistencyratios(the pro-posed consistency ratio and the original consistency ratio) used inBWM.

The remainder of the paper is structured as follows: In

Section2,theoriginalBWMandits consistencymeasurementare introduced.Aninput-basedconsistencyratioisproposedasan al-ternative toreplacetheoriginal output-basedconsistencyratioin

Section 3. An ordinal consistency measurement is formulated in

Section4.ThethresholdtablesarepresentedinSection5,followed bytheconclusioninSection6.

2. Thebestworstmethodandconsistencymeasurement

Inthispart,thebasicstepsoftheoriginal BWMarebrieﬂy in-troduced,andtheoriginaloutput-basedconsistencymeasurement isreviewed.

(4)

2.1. ThebasicstepsofBWM

Asapairwisecomparisonmethod,BWMusesratiosofthe rel-ativeimportance ofcriteriainpairs estimatedbya DM,fromthe two evaluation vectors, ABO and AOW.The weights of the criteria can be obtainedby solving the linearornonlinear program [38]. ThebasicstepsoftheoriginalBWMcanbesummarizedasbelow:

Step1 Have the set of evaluationcriteria

{

C1,C2,· · · ,Cn

}

deter-minedbytheDM.

Step2 Havethebest(e.g.themostinﬂuentialorimportant)and the worst(e.g. theleastinﬂuentialorimportant) criteria determinedbytheDM.

Step3 Determine the preferencesof thebest overall the other criteria using a number from

{

1,2,...,9

}

. The obtained Best-to-Others vector is: ABO=

(

aB1,aB2,· · · ,aBn

)

, where aB jrepresentsthepreferenceofthebestcriterionCBover criterionCj, j=1,2,· · · ,n.

Step4 Determine the preferences of all the criteria over the worst criterion using a number from

{

1_,2_,_._._._,9

}

. The obtained Others-to-Worst vector is: AOW =

(

a1W,a2W,· · · ,anW

)

,whereajW representsthepreference ofcriterionC_jovertheworstcriterionCW, j=1,2,· · · ,n. Step5 Determine the weights

(

w∗₁,w∗₂,· · · ,w∗_n

)

by solving the

followingmodel: min max j

wB wj− a B j

,

ξ

∗

ξ

max

ξ

+

a2 BW− a BW

= 0 . (5)

TherangeofCRO_is_[0_,_1]_._The_closer_C_RO _is_to_0,_the_more con-sistent the judgments are. In particular, CRO₌₀ _means _that _the comparisonsarecardinallyconsistent.

3. Theproposedconsistencymeasurement

The consistency ratioproposed in the original BWM can only beobtainedaftertheentireelicitationprocesshasﬁnished,which meansitcannotprovideaDMwithimmediatefeedbackinvolving his/herconsistency.ToovercomethisproblemandtoprovideaDM witha clearandimmediate ideaof his/herconsistency level,we proposeaninput-basedconsistencymeasurementforBWMthatis easytocompute andhasclearandsimplealgebraicmeaningand interpretation.Furthermore,we will seethat it hasseveral desir-ableproperties(in comparisontotheexistingindices)andahigh correlationwiththeoutput-basedconsistencymeasurement.

Inaccordancewiththeoriginalindex,thenewinconsistency in-dexproposed inthefollowing section onlyattains value 1when, givenaBW,thereexistsaCjsuchthataB j=ajW =aBW.Thisis pos-siblebecausetheindexconsidersthemaximumviolationoflocal inconsistenciesandthevalue 1can actually be attained.None of theindices studiedby Kułakowski andTalaga[28] hasthis prop-erty.Besidesthissimilarity,wewillalsoshowtheresemblance be-tweentheoldandthenewindexusingsomenumericalanalyses.

3.1. Theinput-basedconsistencyratio

IncontrasttotheOutput-basedConsistencyRatio(CRO_),_the ra-tioweproposeinthispapercanimmediatelyindicateaDM’s con-sistencylevelbyusingtheinputhe/sheprovides,i.e.his/her pref-erences,insteadofgoingthrough theentireoptimizationprocess, whichiswhyitiscalledanInput-basedConsistencyRatio(CRI_):

Deﬁnition3(Input-basedConsistencyRatio). TheInput-based Con-sistencyRatioCRI _is_formulated_as_follows:

CRI_{= max} j CR I j (6) where CRI j=

_a B j× a jW− a BW

aBW × aBW − aBW aBW >1 0 aBW = 1 . (7)

CRI _is _the _global _input-based _consistency _ratio _for _all _criteria, CRI

jrepresentsthelocalconsistencylevelassociatedwithcriterion Cj.

Compared to the output-based consistency measurement, the input-basedconsistencymeasurementhasseveraladvantages:

(5)

Table 1

Input-based consistency ratio of each criterion.

Price Quality Comfort Safety Style

aB j 1 2 4 3 8

ajW 8 4 4 2 1

CRI

j 0 0 0.14 0.04 0

1.Itcanprovideimmediatefeedback.Theinput-basedconsistency measurementisbasedontheinput(preferences),whichmeans it is not necessary to complete the entire elicitation process. Theoutput-basedconsistencymeasurementontheotherhand, is basedon theoutput (weights), making ita diﬃcultwayto determine the consistency level.By using the simple calcula-tionoftheinput-based consistencymeasurement,itiseasyto provideaDMwithimmediatefeedback.

2. It iseasy tointerpret:it isthemaximumnormalized discrep-ancybetweenthe value ofaBW andits estimatedvalue calcu-latedastheindirectcomparisonaB j× ajW.

3. It can provide a DM with a clear guideline on the revision ofthe inconsistentjudgement(s).The CRO _indicates _the_global consistencylevel,butitcannotshowtheDMwhichjudgement should berevised.The localCRI_,_however,_displays_the consis-tency levels associated to individual criteria; after identifying the maximumlocal CRI_,_the _most_inconsistent _judgement_can belocated,afterwhichaDMcanrevisehis/herjudgements ac-cordingly,insteadofmodifyingthemwithoutaguideline. 4. Itismodel-independent.ThisCRI_can_be_applied_{independently}

tomeasuretheconsistencylevelinvariousformofBWM mod-els,e.g.anon-linearorlinearmodel,oramultiplicativemodel

[11].Forexample,thelinearBWMmodel[38]doesnothavean effectiveconsistencymeasurement,whilethe non-linearBWM model[37]hasadifferentinterpretationthanthemultiplicative BWM model [11]. By using the input-based consistency ratio, however, they are the same in all three models. Actually,the input-basedconsistencymeasurementdoesnotdependonthe optimizationmodels.

Example1. To illustrate the proposed consistency measurement, weadoptthecarevaluationexamplefromtheoriginalBWM[38], inwhichthe best criterionis priceandthe worst criterion style. ThepairwisecomparisonsvectorsofABOandAOW arepresentedin thesecond andthird rowsrespectively. Byusing theinput-based consistency measurement in Eq. (7), the CRI

js are represented in thelastrowofTable1.

FromTable1,byusingthemaximummeasurement(6),wecan obtain the global CRI_, _0.14. _One _of _the _advantages _of _the input-basedconsistencymeasurementisthatwecanimmediatelylocate themostinconsistentpairwise comparisonfromthistable,which inthiscaseisthepreferencesregardingthecriterioncomfort.Ifthe

CRI_{is too high, the DM’s preferences have to be modiﬁed .}

3.2.Propertiesoftheinput-basedconsistencymeasurement

AsindicatedbyBrunelli[8],itisimportantthatformal proper-tiesofinconsistencyindicesbe investigatedtochecktheir techni-calsoundnessandruleout possibleunreasonablebehaviours.The nextpropositionwillshow thatCRI _satisﬁes _a_number_of reason-ableproperties.

Proposition 1. The proposed consistency measurement, CRI₌ maxjCRIjsatisﬁes the following properties:

1.CRI₌₀_if_and_only_if_the_preferences_are_{cardinal-consistent}_. 2. CRI_{is invariant}_{with respect to a permutation of}_{the indices of the}

criteria.

3.CRI_{is normalized, i.e. 0}_{≤ C}_RI_{≤ 1}_.

4. Ifweconsiderafullyconsistentpairwisecomparisonsystem, mov-ing one of the preferences aB j or ajW away from their original valueintherange[1_,aBW]willresultinanincreaseofthevalue ofCRI_.

5. When aBW>1,CRI is a continuous functionwith respect to the valuesofa_{B j},a_jW,aBW forall j.

6. Ifwe remove a criterionwhich isneither thebest northeworst fromthedecisionproblem,thenthevalueofCRI_cannot_increase_.

Proof. Itisusefultoconsidertheorderedset

S =

CRI_j

|

j = 1 ,...,n

=

aB jajW − a BW

aBWaBW − a BW

|

j = 1 ,...,n

sothat wecanconsiderCRI _to_be_a_function_of_S,_i.e._C_RI

₍

_S

₎

_,_and, ultimately,ofthepreferencesofthedecision-maker.

1. If the preferences are consistent, then aB jajW=aBW, for all j, from which we obtain S₌

0_,_._._._,0

and CRI₌₀_. _In _the otherdirectionCRI₌₀_only_if_S₌

₀_,_._._._,₀

_._If_a

BW=1,CRIj=0, CRI₌₀_,_and_a

B j=ajW =1,aB jajW=1× 1=1=aBW,itisfully cardinal-consistent; If aBW =1, then the only case leading to S=

0,...,0

iswhenthenumeratorsoftheelementsofS are

allequaltozero,whichispossibleonlyifa_{B j}a_jW ₌aBW,forall j,whichistheconsistencycondition.

2. A reordering of the criteria corresponds to an application of apermutation map

σ

:

{

1_,_._._._,n

}

_→

{

1_,_._._._,n

}

totheindices j.

Thenewset

_to changes in a single comparison in the range [1,aBW]. In this case1≤ ajW,aB j≤ aBW,and we canconsider aBW a constant. LetusconsidertheeffectofavariationofaB j inCRIby taking itspartialderivative

∂

CRI

∂

aB j = ajW

(

aBWaBW − a BW

)

aB jajW − a BW

aB jajW− a BW

.

Wecanseethat

∂

CRI

∂

aB j

< 0 ,

aB jajW<aBW

> 0 ,

aB jajW>aBW , whichshows thatCRI

₍

_a

B j

)

isa U-shaped function in[1,aBW], withminimumintheconsistentcase(aB jajW=aBW).Thesame conclusionfollowsifweconsidera_jW insteadofa_{B j}.

5. Straightforward.CRI_is_a_continuous_function_for_all_a BW >1. 6. Ifwe assume that thecriterion which iseliminated, sayCi,is

neither the best nor the worst, then aBW remains unchanged andwecandeﬁneanewsetS_−iwhichdisregardsCi:

S_−i =

CRj

|

j ∈

{

1 ,...,n

}

\

{

i

}

.

(6)

Fig. 1. The relationship between a B j , a jW and CR in the input-based (a) and the output-based (b) consistency measurements when the maximum scale is 9.

Notethatthesepropertiesareadaptationsofwell-known prop-ertiesalreadyproposedandjustiﬁedintheframeworkofpairwise comparison matrices. In particular, Properties 1, 2,4 and5 stem fromthoseproposedbyBrunelliandFedrizzi[9],Property3from thenormalizationproposedbyKoczkodajetal.[26],andProperty 6fromthecontractionpropertyproposedbyKoczkodajandUrban

[25].

It isworth mentioning that Property6would notbe satisﬁed by an approach basedon theaverage ofthelocal inconsistencies likeSaloandHämäläinenindex[43].

3.3. Relationshipbetweentheinput-basedandoutput-based consistencyratio

In the input-based consistency measurement, when the num-berofcriterialargerthan2,fortwopairwisecomparisons,aB j and ajW ∈

{

1,2,...,9

}

,therelationshipbetweenthemandtheir corre-spondingCRI_s_is_shown_in_Fig.₁_(a)._Likewise,_we_can_calculate_the relationshipbetweena_{B j},a_jW _∈

{

1_,2_,_._._._,9

}

andtheirCRO_s_for_the output-based consistencymeasurement in BWM,which isshown inFig.1(b).

It is clear that these two relationship ﬁgures have similar shapes,whichindicatestheyshouldhaveahighcorrelation.

To determine the agreement between these two indices, we analyse them from a statisticalperspective by numerical simula-tions.Firstly,werandomlygeneratedasetof20,000pairsof pair-wise comparison vectors (ABO and AOW) in a 9 criteria problem with1–9scales to representthepreferences providedby DMsin BWM. Then we computedthe input-basedconsistency ratios and theoutput-basedconsistencyratios

(

CRI_,_C_RO

₎

_for_each_pair_of vec-torsinthis20,000randompairsset.Eachpair

(

CRI_,_C_RO

₎

_is repre-sentedbyapointinthescatterplotinFig.2.

AsaB j,ajW ∈

{

1,2,...,9

}

take valuesfromadiscretescale,the possible CRO_s _and_C_RI_s _are _limited. _Thus, _although _we _have ob-tained20,000CRO_s_and_C_RI_s_,_they_distribute_only_in_these_limited possibilities,whichiswhytherearemuchfewerthan20,000dots inthisscatterplot.

WecomputethePearson’scorrelationcoeﬃcientbetweenCRO_s andCRI_s_to_check _the_linear_correlation_between_them._The _result of Pearson’s correlation coeﬃcient in this case is 0.9942, which meanstheseCRO_s_and_C_RI_s_have_a_very_high_linear_correlation._We alsoconsidertheSpearman indextomeasuretheextenttowhich

CRO_s_and_C_RI_s_are_co-monotone._The_result_of_the_Spearman _index

is0.9963,which meansthesetwovariables arehighly monotoni-callyrelated.

When we calculate all the Pearson’s and Spearman’s corre-lation coefficients with respect to 3–9 criteria under maximal scale from 3 to 9, the minimum Pearson’s and the minimum Spearman’s correlation coefficients are 0.979 and 0.958, respec-tively. As such, based on these high correlation coefficients, the input-based consistency measurement and the output-based consistency measurement have a very good agreement, so they could be used interchangeably. Nevertheless, due to its advan-tages discussed in Section 3.1, there are valid reasons to prefer the input-based consistency measurement to the output-based consistencymeasurement.

4. Ordinalconsistencymeasurement

Inthissection,an ordinal consistencyratioisproposed to de-terminetheextenttowhichaDMviolatestheordinalconsistency. Some propertiesforthis ratioare presented andtherelationship betweenordinalconsistencyandcardinalconsistencyisanalysed.

(7)

Fig. 3. The percentage of ordinal-consistent paired vectors.

4.1.Ordinalconsistency

Kwiesielewicz andVan Uden [30] have shown that, even ifa pairwisecomparisonmatrixpassestheconsistencytest,itcanstill becontradictory. Therefore,inaddition tocalculatingthecardinal consistency,itisalsoimportanttocheckwhethertherankings of the criteria obtained from the two pairwise comparison vectors

ABO and AOW are the same in BWM, in what we call ordinal con-sistencycondition. The meaningof ordinalconsistency inBWM is slightlydifferentfromthatinearlystudies,whichismainlybased onthecirculartriads[20,23,27].Wedeﬁnetheordinalconsistency inBWMasbelow:

Deﬁnition4 (Ordinal consistency). In theBWM, a pairwise com-parisonsystemissaidtobeordinal-consistentiftheorderrelations ofthetwopairedcomparisonvectors(ABO andAOW)arethesame. Thatis,thefollowingconditionsshouldbesatisﬁed:

aBi− a B j

×

ajW− a iW

> 0 or

aBi= aB j= aiw

, for all i and j.

(8) Theordinalconsistencyistheusualweaktransitivitycondition whichshouldbetheminimumrequirementforalogicaland ratio-nalDM[48].Intuitively,onemightconsiderordinalconsistencyto beeasilysatisﬁed,butthatisnot true,especiallywhenthe num-berofcriteriaislarge.Toseehowitdevelops,werandomly gener-ated100,000pairedvectorsforeachcombinationofcriteria num-berfrom3to9to simulatethe preferencesforBWM.After cate-gorizing,we cansee thepercentage ofordinal-consistent pairs is reduceddramaticallyasthenumberofcriteriaincreases,asshown inFig.3.Inreality,thesituationisbetterthantherandomly gener-atedvectors,butaftercheckingthedatausedintheoriginalBWM, wefoundthatonly24.4%ofthemareordinal-consistent.

4.2.Ordinalconsistencyratio

Sincetheordinalconsistencyhasavitalimpactontheranking ofthecriteria,itisnecessarytocheckwhetherthepreferences vi-olatetheordinalconsistency, and,ifso,towhat extent.Todoso, weneedtodeﬁneanindex,whichwecallOrdinalConsistencyRatio

(hereaftersimplyOR)inthisstudy.

Deﬁnition 5 (Ordinal Consistency Ratio). The Ordinal Consistency RatioORofapairwisecomparisonsystemisdeﬁnedas:

OR= max

j ORj (9)

Table 2

Ordinal consistency ratio for each criterion.

Price Quality Comfort Safety Style

aB j 1 2 4 3 8 ajW 8 4 4 2 1 O R j 0 0 0.3 0.2 0 where ORj= 1 n n i=1 F

aBi− a B j

×

ajW− a iW

, for all i and j (10) whereF

(

x

)

isastepfunctiondeﬁnedas:

F

(

x

)

=

₁ _{i f}_x_<₀ 0 .5 i f x = 0 and 0 otherwise

aBi− a B j

= 0 or

ajW − a iW

= 0

. (11) The rationale of ORj formulation is that if criterion Cj over-weighs criterion C_i, then the ordinal consistency should satisfy

is equal to 0, we say that, in thissituation,itviolatesweakordinalrelation[12,15],butifboth areequalto0,itisordinal-consistent.

ORjis called localordinal consistencyratio,indicating the de-greeofconsistencywithrespecttothe jthcriterion.Withthis or-dinalconsistencyratio(ORj∈[0,1]),we canﬁndoutwhich crite-rionviolatestherelativeorder(andtowhatextent),andthehigher the ORj is,the more contradictory the preferences hasregarding thiscriterionC_j.

ORiscalledglobalordinal consistencyratio,which reﬂectsthe ordinalconsistencyofthepairwisecomparisonsystemprovidedby theDM.

Example2. We usethe carevaluationpreferences examplefrom theoriginalBWMagain(showedintheExample1inSection3.1) toexplain theordinalconsistency measurement.Fromthe prefer-encevectorABO,wecaneasilygettherankingofthecriteria:price quality safety comfort style. The ranking fromthe AWO vector:price quality∼ comfort safety style(“” means su-periorto,“_{∼” means}indifferentto). Theordersofthecriteriaare differentinthesetwovectors,thusthepreferencesofthisDM vio-latetheordinalconsistency.Byusingtheordinalconsistency mea-surementfromEqs.(9)-(11),wecanobtaintheordinalconsistency ratiosregardingeachcriterion inTable2,which representthe or-dinal violation level of each criterion. The globalordinal consis-tency ratios can be calculated from Eq.(9), which is 0.3in this case.

Combining the cardinal and ordinal consistency ratios, a DM cancheckhis/herrationalityduringthepreferenceelicitation pro-cess. ThisimmediatefeedbackhelpstheDMconfrontshis/her in-consistenciesassoon astheyarise, making thisprocessmore ef-fective[34].

4.3. Propertiesoftheordinalconsistencyratio

TheindexOR(Eq.(9))satisﬁesthreebasicproperties.To enun-ciatetheproperties,weneedtoacknowledgethateachvectorABO andAOW inducesanorderrelationonthesetofcriteria.Thatisto say,forexample,a_Bi_>a_{B j}_⇒i_{≺ j and}a_iW₌a_jW _⇒i_{∼ j.}

1. OR

(

ABO,AOW

)

=0ifandonly ifthe preferencesin thevectors ABO,AWOinducethesameorderrelationonthesetofcriteria. 2. ORisinvariantwithrespecttopermutationsofcriteria.

(8)

Fig. 4. The inclusion relation between ordinal and cardinal consistency.

3. Given two vectors ABO andAWO representing the same order relationonthesetofcriteria,whenwechooseone preference (componentofavector)andwemoveitawayfromitsoriginal valueintherange[1,aBW],thiscanonlyincrease thevalue of ORorleaveitunchanged.

SincethesepropertiesaresimilartothoseinProposition1,the associatedproofisomittedforthesakeofbrevity.

4.4. Therelationshipbetweenordinalconsistencyandcardinal consistency

Analysing the data used in the original BWM [37,38], we can obtain the inclusion relation between cardinal and ordinal (in)consistency of the preferences obtained from different DMs, whichisgraphicallypresentedinFig.4.Forexample,thepairwise comparisonsystemwithcardinalconsistencyisasubsetofwhich, withordinalconsistency,theordinalinconsistentsystemisa sub-setofcardinalinconsistency.

Theinclusionrelationbetweencardinalconsistencyandordinal consistencyshowninFig.4isformalizedintheProposition2and

Corollary1.

Proposition 2. If a pairwise comparison system is cardinal-consistent,itmustbeordinal-consistent.

Proof. Taking the cardinal consistency condition (aBi× aiW= aBW, aB j× ajW =aBW, where aBi, aiW, aB j, ajW, aBW ≥ 1), and ordinal consistency condition (

(

aBi− aB j

)

aBi× a iW= aBW,aB j× a jW= aBW,ajW= aBW aB j ,aiW= aBW aBi so, aBi× a jW− a B j× a jW− a Bi× a iW+ aB j× a iW = aBi× a BW aB j + aB j × aBW aBi − 2 aBW = aBW

a2 Bi+ a 2 B j

aBiaB j − 2 aBW = aBW

a2 Bi+ a2B j− 2 aB jaBi

aBiaB j = aBW

aBi− a B j

2 aBiaB j > 0 .

Therefore,thecomparisonisalsoordinal-consistent.

Corollary 1. Ifapairwisecomparisonsystemis ordinal-inconsistent, itmustbecardinal-inconsistent.

5. ThresholdsforBWM

Eventhoughwecaneasily identifytheinconsistentjudgement byusingtheconsistencymeasurementsproposedinthisstudy, re-quiringtheDMtoachieveperfectcardinalandordinalconsistency isunrealistic.However,thequestioninvolvingthedegreetowhich inconsistencycanbeacceptedhasfarbeenlackinginthestudyof BWM.Assuch,tobridgethisgap,athresholdhastobedeﬁned.In thefollowingsection,basedontheconceptofordinalandcardinal consistencymeasurement,amethodtoderiveconsistency thresh-oldsisproposed.

5.1. Amethodologyfordeterminingthethresholds

Inspired by Amenta etal.[3,4],we develop amethod for de-terminingthe thresholds forBWM, whichis based on the cardi-nalconsistencymeasurementandthedeﬁnitionofordinal consis-tency. The thresholds for BWM are established, not only for the input-based consistency measurement, but also for the output-basedconsistencymeasurement.However,weusetheinput-based consistencyratio(CRI₎_to_illustrate_this_approach.

Thebasicidea isthat,basedon theconceptofordinal consis-tency,ifa decision-makerisordinal-consistent,therankingofthe ﬁnal weights obtained fromthe two preferencevectors (ABO and AOW) will not changewithCRI, only the intensities may vary. In thissense, we can suggest that the preferences provided by the DMarereliable.

WeuseMonte-Carlomethodtosimulatetheprobability distri-butionofCRI_s_._In_this_study,_we _analyse_the_entire_problem_space coveringtheweightingproblems,withthenumberofcriteria rang-ingfrom3to9,andwherethepreferencescan beassignedwith thelargestevaluationgrade from3to9, wecall them3-scale to 9-scale.1_{Consequently,}_in_all,_there_are₇_{× 7}₌₄₉_combinations_to beanalysed.Foreachcombination,werandomlygenerated10,000 pairs of ordinal-consistent vectors, each pair acting as the two vectorsABO andAOW. We categorizedthisgroup asan acceptable group,andcalculatedall theCRI_s _of_this_group. _Likewise,_we ran-domlygenerated 10,000pairs of ordinal-inconsistent vectorsand calculatedtheirCRI_s_,_which_is_categorized_as_an_unacceptable_group_. Theoretically,wecanobtainallthepossibleCRI_s_of_the accept-ablegroup in each situation,taking themaximum asa boundary (boundary1),theCRI_s_above_this_boundary_are_not_acceptable, be-causethey canonly beordinal-inconsistent. Although,practically, itis verydiﬃcult totraverse allthe possibilities, westill assume thatthemaximumCRI_from_10,0₀₀_pair_of_vectors_as_the_boundary 1,becausethelikelihoodofhavingahighervaluethanthis bound-aryisverylow.Forexample,themaximumconsistencyvalueof 9-criterion and9-scale ordinal-consistent pairwise comparison vec-tors is 0.7639,which means that, forany judgments whoseCRI_s are biggerthan thisvalue in a 9-criterionand 9-scalesize prob-lem,theyshouldberejected.

However,thatdoesnotautomaticallymeanthattheCRI_s_within thatboundary arenecessarily acceptable,because theycould still be ordinal-inconsistent,and ordinal inconsistencyis what we set outto reject.Based onthisidea,the minimumCRI_could _be _used asaboundary (boundary 2),all oftheCRI_s _within _this_boundary areacceptable.Forexample,theminimumconsistencyvalueof 9-criterionand9-scaleordinal-inconsistent pairedvectorsis0.0694, iftheCRI_s_obtained_are_smaller_than_this_boundary,_they_should_be accepted.

ValuesofCRI_greater_than_boundary₁_are_assumed_to_be_totally unacceptable,whilevaluesbelowboundary2are assumedtotally

1 In each scale, we use discrete number from 1 to the largest grade which is actually the a BW . For example, if we use 7-scale, the grades used in A BO and A OW

(9)

Fig. 5. The kernel distribution of C R I s of the two groups (9-criteria 9-scale).

acceptable.Between boundary1 and2,we expect that there ex-istsathreshold,makingtheproportionofordinalinconsistencywe acceptassmallaspossible,andbeyondthethreshold,the propor-tionofordinalconsistencywe rejectshould beassmallas possi-ble.Instatisticalterms,our goalis tominimize thesum ofType Ierror(falsepositive)andTypeII error(falsenegative).Thisidea canbe moreclearlyvisualizedin akernelsmoothingdistribution in9-criteriaand9-scalecombination,asshowninFig.5.

Fromtheideaexplainedabove,theempiricalcumulative distri-butionfunctioncanbeusedtoachieveourpurpose.

Deﬁnition6. (Empiricalcumulativedistributionfunction).The em-piricalcumulativedistributionfunctionofCRI_can_be_deﬁned_as:

ˆ F

(

α

)

α

∈[0,1]isthepossiblethreshold.

We now distinguish the distribution function based on two groups:(1)fortheAcceptablegroup,thecumulativedistributionof

CRI _in_{ordinal-consistent}_situation_is_denoted_as_F_ˆA

₍

_α

₎

_;₍₂₎_for_the Unacceptablegroup, the cumulativedistribution ofCRI _in ordinal-inconsistentsituationisdenotedasFˆU

₍

re jected)andtheacceptedproportionoftheCRIsin theunacceptablegroup(PU

₎

. (15)

Fig. 6. The acceptance and rejection relative proportions of the two groups (9- criteria 9-scale).

The relationship between these two proportions is shown in

Fig. 6, which shows how the possibility of acceptance (red line withsquares)andrejection(bluelinewithcircles)distributeinthe twogroupsaccordingtotheselectedthresholdfrom0to1.

Thegoalistoobtainathresholdwhichmakestheredandblue areasinFig.5assmallaspossible,ormakes therelative propor-tions ofthetwogroupsinFig.6ascloseaspossible.Ifthere ex-ists aCRI _obtained _from _the _two _groups _which _makes _PA

re jected= PU

accepted,thetwolines inFig.6willintersectatthat point,which meansthattheproportionofrejectionintheacceptablegroupand the proportion of acceptance in the unacceptable group are the same.However, asthe obtainedCRI_s _are _discrete, _there _could _be noCRI_at_the_intersection_point,_which_means_that_we_need_to_ﬁnd out the intersecting coordinateof thetwo lines,using the corre-sponding CRI _as _the _threshold. _The _simulation _algorithm _for ob-tainingthethresholdisillustratedintheAppendix.

5.2. Approximatedthresholdsfortheinput-basedconsistencyratio

Basedon thealgorithmpresentedabove,we canﬁnally estab-lishthethresholdsforBWM.InTable3,wehaveobtainedthe con-sistencythresholdsforcombinationswhichrangefrom3–9criteria

(10)

Fig. 7. Thresholds for different combinations using input-based consistency mea- surement.

Table 3

Thresholds for different combinations using input-based consistency measurement. Criteria Scales 3 4 5 6 7 8 9 3 0.1667 0.1667 0.1667 0.1667 0.1667 0.1667 0.1667 4 0.1121 0.1529 0.1898 0.2206 0.2527 0.2577 0.2683 5 0.1354 0.1994 0.2306 0.2546 0.2716 0.2844 0.2960 6 0.1330 0.1990 0.2643 0.3044 0.3144 0.3221 0.3262 7 0.1294 0.2457 0.2819 0.3029 0.3144 0.3251 0.3403 8 0.1309 0.2521 0.2958 0.3154 0.3408 0.3620 0.3657 9 0.1359 0.2681 0.3062 0.3337 0.3517 0.3620 0.3662

with highest evaluationgrades from3 to 9 based on the input-basedconsistencymeasurement.

The thresholds in the combinations with 3-criteria and the combinationswith3-scaleare relativelyspecial.Thethresholdsin 3-scale problemremain unchangedeven the numberof criterion changes,because,nomatterhowmanycriteriathereare,the max-imumCRI_in_the_acceptable_group_and_the_minimum_C_RI_in_the un-acceptable groupareequalto0.1667. Inmostother cases,wecan seethatthethresholdshaveatendencytoincreasealongwiththe numberofcriteriaandwiththescaleofthepreferences,asshown inFig.7.2

5.3. Approximatedthresholdsfortheoutput-basedconsistencyratio

ByusingthesamealgorithmintheAppendix,wecanalso de-termine the thresholds for the CRO _in _different _{combinations,} _as showninTable4.3

Comparedtothethresholdsobtainedfromtheinput-based con-sistency measurement,thethresholdsoftheoutput-based consis-tencymeasurementareslightlyhigher.

Finally, by using the approximated consistencythresholds ob-tainedabove,wecancheckwhetherornottheconsistencyofthe DMisacceptable.Forinstance,sincetheoverallCRI_in_the illustra-tive exampleinSection 3.1is0.14,which islessthanthe thresh-old of0.2958 (in5-criteriaand8-scalecombination),asshownin

Table3,itisacceptable.IfweuseCRO_,_which_is_0.223,_we_can_see thatitisalsobelowthethresholdof0.4029,asshowninTable4.

2 The combinations with 2-scale for the C R I are not shown in _{Table 3}_and_{Fig. 7}_,

but it is worth mentioning that the threshold should be 0 in this case, because, when the preferences are ordinal-consistent, C R I = 0 . Therefore, the DM should re-

vise his or her preferences when C R I > 0 .

3 The threshold for the C R O in the combinations with 2-scale is 0, because when

the preferences are ordinal-consistent, C R O = 0 .

Table 4

Threshold for different combinations using output-based consistency measurement. Criteria Scales 3 4 5 6 7 8 9 3 0.2087 0.2087 0.2087 0.2087 0.2087 0.2087 0.2087 4 0.1581 0.2352 0.2738 0.2928 0.3102 0.3154 0.3273 5 0.2111 0.2848 0.3019 0.3309 0.3479 0.3611 0.3741 6 0.2164 0.2922 0.3565 0.3924 0.4061 0.4168 0.4225 7 0.2090 0.3313 0.3734 0.3931 0.4035 0.4108 0.4298 8 0.2267 0.3409 0.4029 0.4230 0.4379 0.4543 0.4599 9 0.2122 0.3653 0.4055 0.4225 0.4445 0.4587 0.4747

Thankstothesethresholds,CRI_and_C_RO_now_have_a_meaningful interpretation, because we can now determine whether they are acceptable or not. The thresholds forCRI _can _help _a _DM _check his/her pairwise comparisons before solving the optimization program.

6. Conclusion

In this paper, we addressed the consistency issue in BWM. First, we argued that the output-based consistencymeasurement in BWM cannot provide immediate feedback to a DM, and only informsthe DMaboutanyinconsistenciesinhis/herassessments after the entire elicitation process has ﬁnished, which has been proven to be ineffective. In addition,existing consistency indices designedfortheincompletepairwisecomparisonmatricesarenot asdesirable as we expected. To remedy that state of affairs, we proposean input-basedconsistency ratio,whichhasa numberof desirablepropertiesandahighcorrelationtotheoriginalratio,to indicate the DM’s consistency status during the preference elic-itation process. This input-based consistency ratio is simple and iseasy foraDMto identifyhis/hermostinconsistentjudgments. Then, to complement the cardinal consistency measurement, we proposed an ordinal consistency measurement to explicate the possible contradictions even in cases where the cardinal consis-tency of a DM’s pairwise comparisons is considered to be good enough. This rationot only showshow much a DM violates the ordinalconsistency,butalsoprovidesaconvenientwaytoidentify andcorrecttheconﬂictsinvolved.Finally,withthehelpof Monte-Carlo simulations, we determined the thresholds for the input-basedandoutput-basedconsistencyratiosindifferentscaleswith different numbers of criteria. The idea is to balance the ordinal consistencyandinconsistency,making theportionofthecardinal consistencyratiosthatviolateordinalconsistencytobeacceptedas smallaspossibleandtheportionofthecardinalconsistencyratios that satisfy ordinal consistency to be rejectedas small as possi-ble.Withthesethresholds,aDMcandecidewhetherornotto re-visehis/herearlierassessments.Andbecausetheinput-based con-sistencymeasurementcanindicatetheconsistencylevelregarding eachcriterion,itcanbeusedinthepreferencerevisionprocess.

The method of determining the thresholds only considers whetherthejudgmentsare ordinal-consistent ornotandhasnot takentheviolationlevelintoaccount.Thiswillbeexaminedin fu-turestudies. Similarly tothe approach what wasadopted inthis paper,thismethodcan alsobe appliedtofuzzyconsistency mea-surementstodeterminetheircorrespondingthresholds.

Acknowledgement

The authors gratefully acknowledge ﬁnancial support for PhD studyfromChinaScholarshipCouncil(No.201708440305).

(11)

Appendix

The algorithm for obtaining the threshold for the CRI _is il-lustrated asfollows and its graphical representation is shown in

Fig.8.

Step1Generate pairwise comparisonvectors. Suppose we have

n criteria (n=3,4,...,9), two random vectors ABO=

(

aB1,. . .,aBn

)

and AOW=

(

a1W,. . .,anW

)

with the maxi-mumscalem(m₌3_,4_,_...,9),arecreatedtorepresentthe pairwise comparisonsvectorsABO andAOW inBWM. The elementsin ABO andAOW areintegers randomlyselected fromdomain[1_,m].

Step2Establish the ordinal-consistent group. After creating a pair of vectors aB and aW, it will be assigned to the ordinal-consistent group ifit satisﬁesordinalconsistency condition(8),andi=i+1.

Step3Establishtheordinal-inconsistentgroup.Ifthepaired vec-tor generatedinStep1 doesnot satisfy theordinal

con-sistency condition, it will be assigned to the ordinal-inconsistentgroup,and j= j+1.

Step4 Continue to create the ordinal-consistent and ordinal-inconsistent groups through steps 1–3, until the size of bothgroupsis10,000.

Step5 Calculate theCRI _for_all _the_paired _vectors _in_these _two groupsbyusingEqs.(6),(7).

Step6 Calculatethe empiricalcumulativedistribution ofCRI_for thetwogroupsbyusingEqs.(12),(13).

Step7 Calculate the relative rejected proportion of the CRI_s in the acceptable group (PA

re jected) and the accepted proportionoftheCRI_s_in_the_unacceptable_group₍_PU

accepted) byusingEqs.(14),(15).

Step8 Ifthere exists aCRI

T making Pre jectedA =PacceptedU ,then this CRI

T is the threshold. If not, go to next step. Step9 IdentifythecrosspointofthelinesofPA

re jectedandP U accepted, theCRI_at_this_point_is_used_as_the_threshold.

(12)

References

[1] Aboutorab H , Saberi M , Asadabadi MR , Hussain O , Chang E . ZBWM: the Z-number extension of Best Worst Method and its application for supplier develop- ment. Expert Syst Appl 2018;107:115–25 .

[2] Aguarón J , Moreno-Jiménez JM . The geometric consistency index: approximated thresholds. Eur J Oper Res 2003;147(1):137–45 .

[3] Amenta P , Lucadamo A , Marcarelli G . Approximate thresholds for Sa- lo-Hamalainen index. IFAC-PapersOnLine 2018;51(11):1655–9 .

[4] Amenta P , Lucadamo A , Marcarelli G . On the transitivity and consistency approximated thresholds of some consistency indices for pairwise comparison matrices. Inf Sci 2020;507:274–87 .

[5] Bozóki S , Rapcsák T . On Saaty’s and Koczkodaj’s inconsistencies of pairwise comparison matrices. J Glob Optim 2008;42(2):157–75 .

[6] Bozóki S , Tsyganok V . The (logarithmic) least squares optimality of the arith- metic (geometric) mean of weight vectors calculated from all spanning trees for incomplete additive (multiplicative) pairwise comparison matrices. Int J Gener Syst 2019;48(4):362–81 .

[7] Bozóki S , Fülöp J , Poesz A . On reducing inconsistency of pairwise comparison matrices below an acceptance threshold. Cent Eur J Oper Res 2015;23(4):849–66 .

[8] Brunelli M . A survey of inconsistency indices for pairwise comparisons. Int J Gener Syst 2018;47(8):751–71 .

[9] Brunelli M , Fedrizzi M . Axiomatic properties of inconsistency indices for pairwise comparisons. J Oper Res Soc 2015;66(1):1–15 .

[10] Brunelli M , Fedrizzi M . A general formulation for some inconsistency indices of pairwise comparisons. Ann Oper Res 2018;274(1-2):155–69 .

[11] Brunelli M , Rezaei J . A multiplicative best-worst method for multi-criteria decision making. Oper Res Lett 2019;47(1):12–15 .

[12] Cavallo B , D’Apuzzo L , Basile L . Weak consistency for ensuring priority vectors reliability. J MultiCriteria Decis Anal 2016;23(3-4):126–38 .

[13] Crawford G , Williams C . A note on the analysis of subjective judgment matrices. J Math Psychol 1985;29(4):387–405 .

[14] Ergu D , Kou G , Peng Y , Shi Y . A simple method to improve the consistency ratio of the pair-wise comparison matrix in ANP. Eur J Oper Res 2011;213(1):246–59 .

[15] Escobar MT , Aguarón J , Moreno-Jiménez JM . Some extensions of the precise consistency consensus matrix. Decis Support Syst 2015;74:67–77 .

[16] Fedrizzi M , Giove S . Incomplete pairwise comparison and consistency optimization. Eur J Oper Res 2007;183(1):303–13 .

[17] Fishburn P . Preference structures and their numerical representations. Theor Comput Sci 1999;217(2):359–83 .

[18] Guo S , Zhao H . Fuzzy best-worst multi-criteria decision-making method and its applications. Knowl-Based Syst 2017;121:23–31 .

[19] Harker PT . Incomplete pairwise comparisons in the analytic hierarchy process. Math Model 1987;9(11):837–48 .

[20] Iida Y . Ordinality consistency test about items and notation of a pairwise comparison matrix in AHP. In: Proceedings of the international symposium on the analytic hierarchy process; 2009 .

[21] Irwin FW . An analysis of the concepts of discrimination and preference. Am J Psychol 1958;71(1):152–63 .

[22] Jensen RE , Hicks TE . Ordinal data AHP analysis: a proposed coeﬃcient of consistency and a nonparametric test. Math Comput Model 1993;17(4-5):135–50 .

[23] Kendall MG , Smith BB . On the method of paired comparisons. Biometrika 1940;31(3-4):324–45 .

[24] Koczkodaj WW . A new deﬁnition of consistency of pairwise comparisons. Math Comput Model 1993;18(7):79–84 .

[25] Koczkodaj WW , Urban R . Axiomatization of inconsistency indicators for pairwise comparisons. Int J Approx Reason 2018;94:18–29 .

[26] Koczkodaj WW , Magnot JP , Mazurek J , Peters JF , Rakhshani H , Soltys M , Strza- łka D , Szybowski J , Tozzi A . On normalization of inconsistency indicators in pairwise comparisons. Int J Approx Reason 2017;86:73–9 .

[27] Kułakowski K . Inconsistency in the ordinal pairwise comparisons method with and without ties. Eur J Oper Res 2018;270(1):314–27 .

[28] Kułakowski, K. & Talaga, D. (2019). Inconsistency indices for incomplete pairwise comparisons matrices. arXiv preprint arXiv:1903.11873.

[29] Kumar A , Gupta H . Evaluating green performance of the airports using hybrid BWM and VIKOR methodology. Tour Manag 2020;76:103941 .

[30] Kwiesielewicz M , Van Uden E . Inconsistent and contradictory judgements in pairwise comparison method in the AHP. Comput Oper Res 2004;31(5):713–19 .

[31] Lundy M , Siraj S , Greco S . The mathematical equivalence of the "spanning tree" and row geometric mean preference vectors and its implications for preference analysis. Eur J Oper Res 2017;257(1):197–208 .

[32] Mi X , Tang M , Liao H , Shen W , Lev B . The state-of-the-art survey on inte- grations and applications of the best worst method in decision making: why, what, what for and what’s next? Omega 2019;87:205–25 .

[33] Monsuur H . An intrinsic consistency threshold for reciprocal matrices. Eur J Oper Res 1997;96(2):387–91 .

[34] Monti S , Carenini G . Dealing with the expert inconsistency in probability elicitation. IEEE Trans Knowl Data Eng 20 0 0;12(4):499–508 .

[35] Mou Q , Xu Z , Liao H . An intuitionistic fuzzy multiplicative best-worst method for multi-criteria group decision making. Inf Sci 2016;374:224–39 .

[36] Pereira V , Costa HG . Nonlinear programming applied to the reduction of inconsistency in the AHP method. Ann Oper Res 2015;229(1):635–55 .

[37] Rezaei J . Best-worst multi-criteria decision-making method. Omega 2015;53:49–57 .

[38] Rezaei J . Best-worst multi-criteria decision-making method: Some properties and a linear model. Omega 2016;64:126–30 .

[39] Rezaei J , Kothadiya O , Tavasszy L , Kroesen M . Quality assessment of air- line baggage handling systems using SERVQUAL and BWM. Tour Manag 2018;66:85–93 .

[40] Saaty TL . A scaling method for priorities in hierarchical structures. J Math Psy- chol 1977;15(3):234–81 .

[41] Saaty TL . The analytic hierarchy process: planning, priority setting. Resour Al- loc 1980;2 .

[42] Saaty TL . Fundamentals of decision making and priority theory with the analytic hierarchy process. RWS publications; 20 0 0 .

[43] Salo AA , Hämäläinen RP . Preference programming through approximate ratio comparisons. Eur J Oper Res 1995;82(3):458–75 .

[44] Siraj S , Mikhailov L , Keane JA . Enumerating all spanning trees for pairwise comparisons. Comput Oper Res 2012;39(2):191–9 .

[45] Siraj S , Mikhailov L , Keane JA . Contribution of individual judgments toward inconsistency in pairwise comparisons. Eur J Oper Res 2015;242(2):557–67 .

[46] Siraj S , Mikhailov L , Keane J . A heuristic method to rectify intransitive judgments in pairwise comparison matrices. Eur J Oper Res 2012;216(2):420–8 .

[47] Ureña R , Chiclana F , Morente-Molinera JA , Herrera-Viedma E . Managing incomplete preference relations in decision making: a review and future trends. Inf Sci 2015;302:14–32 .

[48] Xu Y , Gupta JND , Wang H . The ordinal consistency of an incomplete reciprocal preference relation. Fuzzy Sets Syst 2014;246:62–77 .

[49] Yadav G , Mangla SK , Luthra S , Jakhar S . Hybrid BWM-ELECTRE-based decision framework for effective offshore outsourcing adoption: a case study. Int J Prod Res 2018;56(18):6259–78 .