L~31ro
The Fourth European Conference on the
Evaluation of Information Technology
Blb
l
lQth ••
k TU Delft
"
l
il 111/111
C 2344155
2409
640
The Fourth European Conference on the
Evaluation of Information Technology
October 30-31, Delft, The Netherlands
Delft University Press/1997
Editors
E. W. Berghout
and
Published end distributed by: Delft University Press Mekelweg 4 2628 CD Delft The Netherlands Telephone + 31 15 2783254 Fax +31152781661 ISBN 90-407-1546-7
Copyright © 1997 by the individual authors
All rights reserved. No part of the material protected by this copyright notice may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording or by any storage and retrieval system, without written permission from the publisher: Delft University Press.
The Organization Committee of the Fourth European Conference on the Evaluation of
Information Technology would like to acknowledge and thank the following organizations
whose support and financial contributions helped to make this conference possible.
City University Business School
Faculty of Infonnation Technology and Systems, Delft University
Infonnation Systems Department, Delft University
KPMG EDP Auditors
KPMG Nolan Norton Institute
Foreword
A
'mid-life
crisis' of an academie discipline?
Forewords of conference proceedings tend to
start
with the cliché that this year's conference
again had so many high quality submissions and how hard it was to choose from them. And
indeed, it was hard to choose from the papers submitted and this conference contains many
papers of excellent quality. However, a sense of unease remains. We sincerely believe that the
IT Evaluation discipline, or area of study, is currently experiencing a kind of 'mid-life crisis'
probably caused by its initial apparent success and its increasingly diminishing progress. This
vital area of study now requires a massive shot in the arm in the form of more challenging
hypotheses andlor theories.
All published papers are invariably very
'polite' ,
appreclatmg previous work and always
stating that a lot remains to be done. We seriously challenge both statements. The more we
know about the value of information systems, the less we actually appreciate or value previous
work, because, in retrospect, those efforts often appear to have been based on arguments that
are no longer tenable. With regards to what remains to he
done we are also very sceptical of
some of the routes or avenues of exploration which are being followed, such as in the area of
financial investment analysis of information systems. We question the amount of work that
remains to be done with respect to the IT evaluation calculus. Maybe we already know
everything there is to be known about discounting techniques and payback calculations, at
least in as far as they apply to IT investrnents?
We would therefore like to encourage researchers and practltlOners to write about more
challenging ideas and notions. This field of study is ready to have many old ideas stood on
their head and we believe that a shake-up of the old concepts will lead to more exciting
published work. An example of this might
he
a hypotheses about the multi-disciplinary
character of the concept of the value of information systems. Value has traditionally been
defined by economie theory, but behavioural theory studies the perception of value, and
poli tic al and organizational science considers the consequences for evaluation for the
structuring and functioning of organizations.
Although it is the intention of this foreword to be controversial, we hope that we will not
offend any of our colleagues. We do hope that our view is actually shared by some others and
that we can help spark a new level of research into the question of IT evaluation and its
consequences.
Next year's conference would be an excellent opportunity to take this new approach,
especially as we can then review five years of European IS evaluation conferences.
Egon Berghout,
Delft
Dan Remenyi,
Gothenburg
Reviewing process
In total 29 papers have been submitted to the conference. All of them were blindly reviewed
by two referees. After this, 15 were accepted to he presented as research papers and 6 will he
presented as research-in-progress paper
.
We would very much like to thank those who actually did the reviewing. Without your
contribution, this conference would not have been possible.
Name
Organization
Joan Ballantine
Warwick Business School
Erik Beulen
Originffilburg University
Carole Brooke
Durham University Business School
AnnBrown
City University Business School
Barbara Farhey
London School of Economics
Maarten Looijen
Delft University of Technology
Ronald Losekoot
Delft University of Technology
Roger Lundegard
Chalmers University of Technology
Paul Mantelaers
Delft University of Technology
Philip Powell
Warwick Business School
Anton Van Reeken
University of Maastricht
Dan Remenyi
Henley Management College
Theo-Jan Renkema
PhilipslEindhoven University of Technology
Michael Sherwood-Smith
University College Dublin
Vasilis Serafeimidis
University of Surrey
Frits Swinkels
RabobanklUniversity of Amsterdam
David Targett
Bath University
Table of contents
Evaluation in practice track
Development levels and IT evaluation models in Spanish eompanies: a cluster
analysis
H
.
Huerta Arribas and P.J. Sanches Inchusta
IT failure: a major challenge for South African management in the late 1990s
D
.
Remenyi and T
.
Schambreel
13
Does information technology matter? A study of IT involvement in strategie
investment decisions
T
.
C
.
Chou
,
R.G
.
Dy
s
on and P.L. Powell
23
Post-implementation track
Evaluating the needs and perceptions of small to medium sized enterprises in
relation to emerging teehnologies
L. McNutt and G
.
O
'
Donnell
37
Post-implementation evaluation of information systemsltechnology: a survey
of UK practice
K.
Miller and D. Dunn
47
Software product evaluation: eurrent status and future needs for eustomers
and industry
T. Punter, R. Van Solingen and 1. Trienekens
57
Developments in theory track
Option pricing for IT valuation: a dead end
B.
De Jong, P. Ribbers and H. Van der Zee
Control of the information systems function: the role of eost alloeation
P
.
Dirks and L. Van Lent
Managing the Iife cycle of information and communication technology
investments for added value
G.J.P. Swinkels
67
75
Applications of evaluation track
Evaluation of innovative IT projects: the dynamic interaction between design
and use
e. Dyrnoke-Bradshaw and B
.
Cox
95
Evaluation of information systems in health care: a framework and
it's application
H. Salrnela and P. Turunen
111
IT
economics applied to billing: a process oriented approach
M
.
P
.
Dieben
,
K.M
.
Duursrna
,
R
.
Klopman
,
RA.F Pronk and M
.
e. Schönfeld
121
Towards s
y
n
e
rch
y t
ra
c
k
Concept of value: a contextual moderator of information system business value
M.e. Cronk and E
.
P
.
Fitzgerald
131
The four P's revisited: appraising and managing the infrastructure impact of
IT investments
TJ.W.
Renkema
139
A typology of information and communication technology applications
from a management point of view
A.J
.
van Reeken
151
Research in progress: Methods in Movement track
Measuring and improving corporate information technology through the
balanced scorecard technique
W. van Grernbergen and R van Bruggen
163
Telecare: on the strategie assessment of telematics in health care
RR Peterson and D. De Wit
173
Deciding on inter-organizational telematics investments: the use of
IT evaluation methods
RH.J
.
Dernkes
181
Research in progress
:
Th
e
or
y i
n Movement track
A universal spanner in the works: economie issues of evaluating general
purpose technologies
EIS usage: a cross-cyltural study
M
. EIKordy, A. Leeming and O.M. Khalil
197
The impact of information technology on vertical coordination structure and
performance of firms
N.
Shin
205
Best practices track
IT performance improvement
J. Acohen and
L.
de Looff
Managing the evolution of information technology
H.T.M. van der Zee
211
Development levels and IT evaluation models in Spanish companies: a
cluster analysis
Huerta Arribas E.
l
and Sánchez Inchusta P.J.
2lDpt. Business Administration, 2Dpt. of Mathematic and Computer Science, Universidad
Póblica de Navarra, 31006 Pamplona, Spain, E-mail: inchusta@upna.es
The aim of this article is to study how Infonnation Technology (lT) evaluation is carried out among a group of Spanish companies, establish a typology or grouping of them which uses similar evaluation criteria, determine the relationship hetwen the level of IT maturity and the way in which it is evaluated and, finally, if there are any similarities in the forms of evaluation carried out by Spanish and foreign companies. The analysis will he based on a detailed study of 20 Spanish firms, taking into account the different stages of development of IT within each finn.
1 Introduction
The evaluation of Information Technology (lT) has been seen as a complex task owing both to the difficulty in determining the value of information as weil as to the special economic characteristics of the technology which produces it. One proof of this complexity is the dichotomy which exists between the practice and the theory.
Researchers even believe that a sound disparity
exists between the methods of evaluating
investment in IT, not only between different organizations but also within the same company (Farbey et al. 1993), based on the general belief that investment in IT is different from other forms of investment. This leads to the conclusion that financial techniques are not suitable to evaluate
investment in IT (Hayes & Albernathy 1980;
Strassman 1985; Shank & Govindarajan 1992;
Robson 1992; Hochstrasser & Griffiths 1990). However, little empirical evidence exists as to whether both types of investment are treated by
companies in the same way. The few existing
papers using the survey method conclude that in practice companies use similar procedures to justify all types of investment, whatever their nature (Wilner et al. 1992; Bacon 1992; Golden & Oflaherty 1996). Other works do not offer such conclusive results and their authors consider the reasons for similar or different treatment to be extremely diverse (Ballantine et al. 1995).
The aim of this article is to study how IT evaluation is carried out among a group of Spanish companies,
analyse the characteristics which unite or
differentiate them and determine whether we can
establish a typology or grouping of companies
which uses similar evaluation criteria. To do this, we pose five main questions to be answered: a) Is evaluation of IT investment viewed differently to
other generic investment? b) Is the level of IT development in the companies homogeonous? c) Can the companies he grouped into specific evaluation typologies? d) Can any relationship he
established between the level of IT maturity
-coordination betwen organizational and IT aims-and the way in which it is evaluated? e) Are there any similarities in the forms of evaluation carried out by Spanish and foreign companies?
To answer these questions we have analysed twenty Spanish companies. Given the complex numher of cases and variables involved in the compilation of quantitative and qualitative aspects, we have used cluster and correspondence analysis.
The structure of the paper is as follow: The second section involves a justification of the methodology used and the group of companies chosen. In the third section, the results are analysed regarding the similarities between IT evaluation and that of other investment, the levels of maturity and the evaluation typologies. In section four, we compare the results obtained with those of other studies and finally, in the fifth, the basic conclusions are drawn.
2 The empirical research
2.1 Methodology
Case studies have been consistently defended as a suitable methodology for the study of IT and, more so, for the analysis of their evaluation (Benbasat et al. 1987; Baroudi & Orlikowsli 1989; Lee 1989; Galliers 1991; Walsham 1993). The main reasons given are the complexity of the task, the multiple perspectives and interpretations which can he adopted, the advantageousness of considering content, context and evaluation process, the dynamic and transitory nature of this technology and the need to use quantitive and qualitive
information to understand these processes. The study of individual cases would seem to be suitable for the investigation of subjects which have not previously been studied, while multiple cases are recommended when the aim is to describe or construct theories, or to test them as this allows the cross analysis of cases. What is more, multiple cases are useful when the object of the research is not to generalize but understand individual processes or topics and recognize the capacity for success. To this end, specific examples of excellence are looked at and information is sought from qualified groups (Remenyi & Williams 1995).
However, when the number of cases is high, drawing up conc1usions can prove difficult if some statistical tools are not used (Gabie 1994).
We have studied a group of twenty companies and collected information by means of detailed interviews in order to obtain qualitive information from different levels and functions of the organizations. We have used the cluster analysis which allows us to describe common behavioural guidelines and bring companies together according to the characteristics which distinguish them most from the other groups. This technique has allowed us to establish niches or groups with certain characteristics in common, not only of a quantitive nature but also qualitive or nominal.
2.2 Characteristics of the companies
Twenty companies were chosen belonging to diverse economic sectors and with a variety of legal status, both national and multinational. In terms of workforce size, three companies are small (less than 100 workers), fourteen are medium-sized (between 100 and 150) and three are large (more than 500).
However, weighing up their incidence according to workforce size, 43% of workers are found in large companies, 53.7% in medium and 3.1% in small ones.
The companies were chosen with the aim to observe behaviour and specific situations. Thus, the chosen companies obey one or more of the following criteria:
a) Innovative technology such as EDI or JIT has been introduced (6 Companies).
b) Significant investment in IT has been carried out (3).
c) They are competitive and have a major involvement in the foreign sector (4).
d) They produce innovative and competitive products from an industrial point of view (4).
e) Other characteristics (3)
The information concerning these companies was principally obtained using open questionnaires as stipulated in case study guidelines, and was completed using information gathered from written documents, both from internal company sources as weil as published material. Following this, the information was put into c10sed question categories which allowed a statistical analysis. As a result, we obtained a large group of variables of which the majority are considered to be active (60) and six illustrative and which bring together primarily information of a qualitive nature. As for the interviewees and their corresponding hierarchical levels, four of the forty five people spoken to correspond to General Management, seventeen to Head of Computing, fifteen to Financial Director, one to Personnel Manager, four to Manufacturing Manager and the remaining four to other categories.
This information has been structured around three axes relating to content, context and evaluation process. The content refers to the values and criteria to be considered and what should be measured. The process is the way in which evaluation is carried out and who are the actors. The context can be external -competltlve environment- or internal organizational structure-. This is in line with contextual and interpretive theories since for a correct understanding of IT evaluation, not only must the values be analysed but also the organizational setting and those who carry out the process (Dav is & Hamann 1988; Symons 1990; Symons & Walshaml991; Farbey et al. 1994;
McBride & Fidler 1994; Serafeimidis & Smithson 1994; Brown 1994; Griffiths & Willcocks 1994;
Deitz 1994; Huerta & Sánchez 1996).
3 Empirical results
This research seeks to answer the questions which concern us and which we set out in the introduction, regarding how companies evaluate IT, what methods are used and the influence which the level of development of IT can have on the way in which it is evaluated. The main results are described.
3.1 Similarities between IT and other
generic investment evaluation
The main conc1usion is that II companies out of 20 believe that the evaluation of IT investment is different from other investment although no distinction is made to the different kinds of IT and
its wide range of objectives. Other considerations which demonstrate the peculiar nature of IT include the fact that evaluation deals on the whole with generic projects (16 out of 20), but only in a minority of cases with IT; also, within the financial techniques only one is employed (Cost/Benefit Ana/ysis or Net Present Va/ue in 15 companies) for generic investment while in the case of IT a
combination of techniques are used (13 companies).
Furthermore, different attitude is shown towards expenditure on hardware or on software; the former as investment and this as expenditure, which underlines th is peculiar nature.
3.2 Maturity levels of IT
We understand as IT maturity the degree to which companies incorporate IT in order to achieve organizational aims. A multitude of theoretical models exist although the one we have chosen (Ward et al. 1993; Galliers 1993; Earl
NOKlNAL CHARACTBRISTICS
VARIABLBS MODALITIBS
GROUP 1 "
MANAGER COMPROMISE Promoter
IT STRATEGIC PLANNING Rea1ized
1989;Galliers & Somogyi 1987; Wiseman 1985) is a simple but powerful one according to which we establish categories of IT for automation; IT to aid decision making; and IT for strategic purposes. These models are defined according to their
objectives of mechanizing tasks which were
previously carried out by hand thereby achieving cost efficiency; obtaining the information necessary to make decisions, and thus achieving effectiveness;
and obtaining competitive advantages, respectively.
The cluster analysis separates the companies studied into four groups each with distinguishing modalities. The variables and modalities in table I characterise each of the four groups with the level of confidence shown in T-Test and Prob. The ratio cla/mod means the percentaje of the companies who display this modalitie and who are included in the group; the ratio mod/cla means the contribution of the companies with these modalities to the group. (For more details see Simón & Sánchez 1996).
RATIOS t-TEST PROB.
CLA/MOD MOD/CLA
100.00 100.00 3.13 .001
60.00 100.00 2.38 .009 IT EXPENSES (1.5-2)% Turnover 100.00 66.67 2.15 .016 IT COMPETITIVE ADVANTAGE Implication 42.86 100.00 1. 87 .031
RELATION TERMINAL I PC No preponderance 42.86 100.00 1. 87 .031
N' PEOPLE IT DEPARTMENT >5 peop1e 66.67 66.67 1. 69 .046 IT EVALUATION Internal evaluation 37.50 100.00 1. 65 .049
GROUP ~"
DECISION APPLICATIONS Used 70.00 100.00 2.96 .002
STRATEGIC PLANNING No strategie p1ann. 83.33 71. 43 2.45 .007
TERMINALS COMPUTER No 53.85 100.00 2.01 .022
SALES >40 millions $ 60.00 85.71 1. 90 .029
N' COMPUTER I PERS ON Unitary 50.00 100.00 1. 70 .044
INTERNAL NETWORKS yes 54.55 85.71 1. 57 .058
TYPE OF INTERNAL NETWORKS business & factory 62.50 71.43 1. 62 .052
GROtlP 3 "
ORGANIZATIONAL DEPENDANCE
I
Don't exist 100.00 100.00 1. 64 .050FUNCTIONS Not specific 100.00 100.00 1. 64 .050
GROtlP
'I'
TERMINAL COMPUTERS Terminals YES 100.00 77.78 3.31 .000
RELATION TERMINALS/PCs Terminals 100.00 55.56 2.40 .008
STRATEGIC PLANNING Yes 64.29 100.00 2.26 .012
INTERNAL NETWORKS No 77.78 77.78 2.25 .012
N' COMPUTER I PERSON (50-80) % 100.00 44.44 1. 94 .026
EXTERNAL NETWORKS No 70.00 77.78 1. 81 .035
DECISION APPLICATIONS No 70.00 77.78 1. 81 .035
TYPE OF EXTERNAL NETWORKS No 70.00 77.78 1. 81 .035
The first group is made up of companies in which management plays an important role as promoter
and motivator in the incorporation of IT.
Furthermore, IT is seen as a strategic factor which serves to link up and coordinate with customers via the use of internal and external networks. The IT department is of considerable importance due both to its financial resources for expenditure and investment as weil as to the large number of staff who are employed there. With reference to the development levels adopted, we can broadly classify these systems as strategie systems (SIS). The group comprises three companies.
The second group is made up of those companies whose sales figures define them as being large and yet have no strategic plan. They have a high ratio of
computer equipmentlindirect personnel, the
equipment is connected to an internal network and they possess applications to aid decision making, albeit of a personal nature and via office automation applications. In other words, where IT is dedicated
to the internal working of the company,
strengthening interconnection and extending
information and where the system can be described
as management deeisions support (OSS). The group
is made up of seven companies.
The characterizing feature of group three is that IT is not considered strategic nor is there a computer
department, only users. The ratio of
computer/indirect personnel is very low and hardware structure is made up by individual microcomputers. One out of the three small companies is in this group.
In group four, IT is not considered important in the strategy of the company. Guidelines for behaviour are established which take no account of the role of IT which continues to deal exclusively with the
automation of tasks previously carried out
manually. These systems can be denominated systems of transaction processing or automation. Group four is made up of nine companies, almost half of those studied, and is the most numerous group.
If we compare groups two and four, systems for aiding decision making and automation systems, we see that they display opposite characteristics (Figure 1) and that they obey the rationale of the modeis. The only possible question mark is that companies in group four possess a strategic plan, an advanced level of formal planning, and yet the level of computerization is relatively immature. This is a real, and quite common situation and normally involves cases in which a strategie plan has been developed as a result of market needs and is
explained by the manufacture of a brand new product following the disappearance of demand (as in the case of spare parts for cars), by the crisis caused by domestic or international competition
and/or the withdrawal of foreign capital. The
company directors have adopted a strategie plan
which allows the company to survive, but is
normally developed by external agents and without any involvement of IT in the plan even when the lack of outside data has been identified as the weak point of the company.
Modalities Group2: Group4: DecisioD
Auto-SUDDOrt matioD
tratel!ic Plan no yes
Decision Suooort aoolication yes no
erminals no yes
Pe's yes
nternal network yes no
Ratio DelPersonnel 1 Medium
IExternal network no
Figure I. Modalities in groups 2 and 4
3.3 IT evaluation typologies
In order to analyse the evaluation of IT, the different techniques employed by the companies specifically in IT investment are described together with their content and their problems. We also look
at the evaluation process, dealing with such
questions as who carries it out, who makes the decision, what values are considered and what consideration is given to the evaluation itself. In short, the aim is to deduce which characteristics are held in common and which are not, in order to establish possible, real evaluation typologies. The information used is contained in twenty ni ne variables, the majority of which are of a qualitative nature, twenty seven being considered active and two iIIustrative, and which contain one hundred and two modalities. This consideration of active and iIIustrative variables is in our opinion the most suitable in the context of the theories regarding IT maturity and evaluation (Sánchez 1997).
The result of the cluster analysis is the c1assification of all the companies studied into four groups whose most outstanding modalities can be seen in table 2 and which we will subsequently look at in more detail.
IlOIlALITIBS lUIlD CllARACTllRISTICS RATlOS
GROtJP
1"
CLA/MOD MOD/CLA T Test PROB.WHO EVALUATE PROPOSER PROP 80.00 80.00 2.58 .005
STAKEHOLDER CUSTOMERS C9CL 80.00 80.00 2.58 .005
MADURITY LEVEL 1 NIV1 100.00 60.00 2.38 .009
TECHNI UES VARIOUS B6DI 100.00 60.00 2.38 .009
VALUES STRATEGIC ESTR 50.00 100.00 2.14 .016
EVALUATION*2 BENEFICIAL JBEN 57.14 80.00 1. 87 .031
STRATEGIC ESTRATEGIC SIES 41. 67 100.00 1. 63 .051
AUTHORITY OWNER BOARD COAD 100.00 40.00 1. 62 .053
QROtJP l / '
IMPORTANCE VERY IMPORTANT C2MU 100.00 100.00 2.56 .005
CRITERIA COMBINATION A4VA 50.00 100.00 1. 86 .032
NORMS OON'T EXIST B5NO 40.00 100.00 1. 62 .053
BENEFICS IDENT.INTAN+ UANT.FI IICF 40.00 100.00 1. 62 .053
GRAL. NORMS OON'T EXIST A1NO 33.33 100.00 1. 41 .079
PROBLEMS EE C8EE 100.00 50.00 1. 28 .100
TECHNI UES FEELING B7FE 100.00 50.00 1. 28 .100
EVALUATION*2 BENEFIClAL JBEN 28.57 100.00 1. 22 .1l1
GROtJP 3/6
EXTENSION UANTITATlVE IMPORTA A3CU 75.00 85.71 2.62 .004
IMPORTANCE INSIGNIFICANT BlIN 58.33 100.00 2.32 .010
FRE UENCY WHEN RE UIRED C1MA 100.00 42.86 1. 87 .031
CRITERIA PAYBACK A4PA 66.67 57.14 1.42 .078
WHO EVALUATE CEO FINA 55.56 71.43 1. 27 .102
BENEFICS IDENTIF. INTANG. IINO 55.56 71.43 1. 27 .102
VALUES NECESITY NECE 55.56 71. 43 1.27 .102
SIMlLARITY SIMlLAR B4S1 55.56 71. 43 1.27 .102
QROtJP
'I'
FINANClAL TECHNI UES USED A2S1 71. 43 83.33 2.45 .007
MADURITY LEVEL 4 NIV4 80.00 66.67 2.20 .014
WHO EVALUATE GEN. MANAGER GERE 100.00 50.00 2.11 .018
BENEFICS NON INTAN-FINAN NINF 100.00 50.00 2.11 .018
CRITERIA OTHERS A40T 62.50 83.33 2.10 .018
FRE UENCY NONE C1NO 55.56 83.33 1. 78 .038
EXTENSION NONE C3NO 55.56 83.33 1. 78 .038
EXTENSION NONE A3NO 75.00 50.00 1. 55 .061
Table 2. Evaluation models
GROUP 1. Strategie Eva/uation. This group comprises five companies which represent aquarter of all the companies studied. The companies belonging to th is group can be defined using the following characteristics which distinguish them from the other groups: the use of diverse and varied techniques in the evaluation of IT investment, depending naturallyon the different kinds of projects; the values observed are in general strategic and benefits which may affect the customer are included in a consideration of benefits to be gained. Furthermore, those who carry out the financial evaluation of investment are those who propose the investment and this evaluation is considered a beneficial process for the company.
The modalities which characterize this group from others, together with the level of confidence and the class/modality and modality/class relation are seen in the previously mentioned Table 2. It is worth pointing out that, all in all, over 50% of the companies who display each of these modalities are included in this group, with less than a 5% probability of error. The modality "maturity level \" stands out dramatically, emphasising the fact that
all the companies which correspond to this highest level of maturity are in this group. The same percentage is found regarding "use of different techniques to evaluate IT investment". In addition, of the companies whose characteristics include a consideration of the benefits to the customer brought about by IT, and/or an evaluation of investment carried out by the same person to propose it, 80% are found in this group. However, those companies who exclusively consider as a priority the values of cost or necessity are not included here.
In short, the consideration of strategic values in the evaluation of IT, the involvement of IT in company strategy and the attention paid to of one of the important competitive forces in this setting, the customer, lead us to denominate this group
Strategie Eva/uation.
GROUP 2. Eva/uation by Standard Financia/
Teehniques. This group consists of two companies. The characteristics which most stand out from the rest of the companies, and which in addition are held by all components of the group, are the use of
a combination of budgetary investment techniques in the evaluation of generic investment; the importanee given to these techniques in the evaluation of IT investment despite no written norm concerning its implementation; the evaluation of IT by quantifying economic benefits and identifying those which are most qualitive or intangible; and finally the view that the formal process of IT evaluation is beneficial to the company.
In short, the enormous importance given to the use of capital budgetary techniques to evaluate IT investment, together with the quantifying of economie benefits lead us to denominate this group Standard Financia/ Eva/uation.
GROUP 3. Eva/uation for reasons of Necessity.
This group is made up of seven companies,
representing over a third of the companies in this study. lI's distinguishing characteristics are the
following: The companies only use capital
budgetary techniques to evaluate generic investment involving important quantities, the most common of these techniques being the recovery period; expenditure and investment in IT is not a significant sum in relation to turnover or total investment; thus, when budgetary techniques of evaluation are used, they are used sporadically or as they are required and not during the elaboration of budgets.
Statistically, we can see that two thirds of the companies for whom IT investment is seen to be in significant are included in this group and all the companies in the group display this characteristic. As for the evaluation of generic investment, threequarters of those who only evaluate large investments are included here and this characteristic is dominant among the companies of this group. With regard to the frequency of evaluation , all
those who carry out evaluation sporadically belong to this group.
In this group there are no differences in the ways in which investment in IT and generic investment are
evaluated. The financial manager normally
performs the evaluation which, curiously, is limited to identifying intangible benefits. Investment in IT is as a result of necessity. 55% of those companies which display these modalities are included in this group which constitutes 70% of the group. For these reasons, we have labelled this group Eva/uatian for Reasans of Necessity.
GROUP 4. Cast Eva/uation. Six companies are included in this group. The characteristics which distinguish the group are the following: No financial techniques are used in the evaluation of generic investment nor in that specifically of IT. Nor are the criteria financial: Cost alone is
considered, with no attention paid neither to
financial nor qualitative benefits. Furthermore,
evaluation is normally carried out by the general manager.
All companies which only consider cost in their evaluation without paying attention to any kind of benefit and all those in which the general manager carries out the evaluation are included in this group. What is more, 80% of the companies in group 4 of maturity, or transaction processing are included here. We will look more closely at this point in the following section.
To sum up this epigraph, four different evaluation typo logies are found among the companies studied which we have labelled strategic evaluation, DSS financial, reasons of necessity and cost criteria. Their incidence within the group as a whole is unequal since evaluation based on cost and on necessity is found in 13 of the twenty cases studied.
3.4 Correspondenee between maturity
and evaluation levels
One question which was put forward in the introduction, and which we will now analyse is whether a relationship exists between the IT maturity level and the different evaluation systems, assuming as a theoretical hypothesis that the most advanced levels of maturity are accompanied by the best systems of evaluation. To do this, we will use the results of the cluster analysis from the previous epigraph and complete them with the simple correspondence analysis (Grandes & Abascal 1995; Escofier & Pages 1988).
Using the cluster analysis in the previous section we
noticed some interesting facts. Firstly, a
correspondence is confirmed between the strategie
information systems and their own strategie
evaluation. Thus we notice that all the companies pertaining to the group 1 maturity level are included in the evaluation group I. Secondly, in the other extreme, a correspondence exists between those companies using IT for automation and the companies which evaluate investment in IT according to cost. In this correspondence, half of the companies situated in group 4 maturity level are represented in the evaluation group 4 and make up almost two thirds of the companies in this evaluation group.
We have completed these results of the cluster analysis using simple correspondence analysis and the elaboration of the ad hoc statistie (tabie 3). We are able to see that five companies carry out a strategie type of evaluation. However, the maturity
level of this category does not correspond entirely to maturity group I since two other companies exist, one with OSS maturity level and the other, automation.
The second evaluation group, that of the standard
financial techniques is carried out in two
companies, one corresponding to OSS level of
development and the other to automation level. The third evaluation model, that of necessity, concerns seven companies of which four are found at OSS level and three at automation level. The fourth model, that based on cost, includes six companies, one with OSS level, four with automation level and one at an initial stage of computerization. This demonstrates that the evaluation only takes into account the cost of the investment.
To find out whether in our subgroup, a higher IT maturity level corresponds to a greater level or
sophistication of evaluation tools, we have
e1aborated an approximate ad hoc statistic, and we have calculated the sign and relative value of the differences between the real and expected outcomes to determine the existence of correspondence
between the modalities, sign and magnitude. Among
the values and sign of this ad hoc statistic we can point out 0 and I as the most significant values together with the positive and negative signs. The 0 value of this statistic indicates the independence between both modalities or lack of correspondence;
the positive signs represent positive
correspondences or attraction bet ween the
modalities and the negative signs, negative
correspondence or rejection. The value -I
demonstrates the inexistance of real values and instead estimated ones; the value + I demonstrates positive correspondence and in it the real values are double those expected.
We are aware of the inappropriateness of the usual statistics such as
X
2, given that the size of the sample is c1early insufficient to be representative, likewise its randomness -we have indicated the specific characteristics of the chosen
companies-(Canavos, 1988). This will prevent us from
generalizing our conclusions but will allow us to discover the correspondence in the subgroup studied
From this analysis we can deduce that a positive correspondence or attraction exists, one which is very strong, between the level of strategic maturity and the strategic evaluation system; a blatant attraction between the OSS level of maturity and evaluation of necessity; and equally between the automation level of maturity and cost evaluation.
However, a weak but nevertheless positive
correspondence exists between the OSS level of
maturity and the tinancial level of evaluation and
al most complete independence between the
automation level of maturity and the tinancial evaluation.
MATURITY
EVALUATION GI-SIS G3-DSSI G4-TP LlNE
G2-ITP TOTAL
PI-STRATEGIC
Real Value 3 J J 5
Expected V. ,8 2 2,3 25%
ReJative V& sign +29 -05 -0.57
102- FINANClAL
ReaJ Value 0 J I 2
Expected V. 0,3 ,8 ,9 10%
Relative V& sign -10 +0.25 +011
p3-NECESSITY
Real Value 0 4 3 7
Expected V. 1,1 2,8 3,2 35%
Relative V& sign -10 +043 -003
1G4-COST
ReaJ VaJue 0 2 4 6
Expected V. ,9 2,4 2,7 30%
Relative V& sign -10 ·017 +048
COLUMN 3 8 9 20
TOTAL 15% 40% 45% 100%
Table 3. Calculation and sign of the differences
With regard to the rejection or negative
correspondence, the level of strategic maturity is c1early opposed to the OSS tinancial types of evaluation as weil as those of necessity and cost. The OSS level of maturity displays a slight rejection of strategic evaluation and an al most complete independence to standard tinancial evaluation. In short, within the group of companies studied and bearing in mind the statistical limitations mentioned, we can observe a positive correspondence or attraction of considerable si ze, between the strategic evaluation form and the level of strategic maturity. The second attraction of significant proportions is found between cost evaluation and the automation level of maturity. A similar situation can be found between evaluation of necessity and the OSS level of maturity. However, standard tinancial evaluation has no obvious attraction with any level of maturity, excepting its negative relationship with the strategic level. From an economic perspective, we can conclude, tirstly, that a relationship exists between strategic maturity and strategic evaluation. In other words, it
reveals that those companies in which the
management plays an active and dynamic role in the introduction and planning of IT, where the function of the computer department is linked to company strategy, and where expenditure and investment in
IT is both of quantitive and qualitive importanee, employ evaluation methods of IT investment where strategie values take precedence, with the logical and subsequent medium and long term view. Furthermore, the setting in which the company operates is taken into account and the customer occupies an important position as beneficiary of the advantages which IT may originate. The evaluation of such investment is viewed differently from that of generic investment. Those who propose the investment are the same people to evaluate it and the process of evaluation itself is considered beneficial to the company. We have summed up these aspects in the so called strategie evaluation.
Secondly, there is a positive correspondence between the maturity level 2 or OSS and the evaluation of necessity. That allows us to conclude that those companies which do not possess a general strategie plan, but do possess computer applications to aid decision making, with a computer equipment structure connected via internal corporate networks which link up industrial plants, central offices and all areas of the company are companies which only evaluate large generic projects. In such cases the recovery period tooi is used in which expenditure and investment in IT is not substantial, evaluation is sporadic rather than regular and no difference is perceived between generic evaluation and that of IT. The financial manager who is responsible for carrying out the investment curiously identifies the intangible benefits although it is the principal of competitive necessity which has most influence on the decision.
In third place, the correspondence between cost evaluation and the automation level of maturity means that companies who have established a strategie plan for the purpose of emergency situations in the company, who are without an internalor extern al network linking their hardware structure, and whose applications are used to carry out regular transactions but not to make decisions, these companies do not employ financial techniques or criteria neither in the evaluation of generic investment nor IT investrnent. The only aspect to be taken into consideration is cost.
To sum up, we can affirm that a pOSltlve coincidence exists within the group of companies studied between strategie maturity and the strategie evaluation of IT; between the DSS level of maturity and the evaluation of necessity, and finally, between the transaction processing system and that of cost evaluation. In the other modalities, the independence, rejection or non-existence of correspondenee can be maintained.
4 Comparison with other empirical
results
Our final task, and one which we will deal with in this epigraph, is to check our results with those obtained in other international studies. In this area, the empirical evidence as to whether both types of investment are dealt with by companies in a similar or different way continues to be scarce. Work carried out using the survey method indicates that in practice companies use similar procedure to justify all types of investment, independent of their nature (Wilner et al. 1992; Golden & Oflaherty 1996; Bacon 1992).
Other authors believe that the reasons for similar or different treatment are extremely diverse (Ballantine et al. 1995) producing even disparate results, normally when the research is carried out by means of case study and is therefore more comprehensive and qualitative. Thus, research into sixteen English companies (Farbey et al. 1993) conc1udes that few companies consider investment in IT differently to other capital investment and, in general,quantifiable, tangible and short term benefits are taken into account. In nine Greek cases (Mirtidis & Serafeimidis 1994), IT investment is considered different but evaluation is the same due to the inability to find an alternative method. In fourteen South African cases (Sutherland 1994), the difference lies in who is the evaluator rather than the subject of the evaluation and thus the different types of IT projects are not evaluated differently.
However, a distinction is made when the evaluators are the general management or the IT manager, with qualitative criteria predominating in the former and purely financial criteria in the latter. In three cases from the low countries (Oeitz 1994) a great difference is acknowledged among the evaluation methods according to the objectives sought by the IT projects. Thus, of the four studies carried out using the case study method, three consider that IT evaluation requires different procedures from all other investment, coinciding with our own conclusion (Figure 2).
With regard to the maturity/evaluation technique question, other studies are not as concerned as ours to establish such a close relationship. However, we are able to identify other common points of interest. To do this, we consider Farbey's study dealing with an IT structure to aid decision making (OSS), Mirtidis' study dealing with automation and Deitz's study with strategy, and we compare them with characteristics from our own groups of strategie evaluation, standard financial, necessity and cost.
Author,Country IT & STRATEGY IMPACf JUSTIACATION OF QUANTIACATION DECISION OTHER
Frnns, Methodology INYESMENT MAKING CHARACfERISTICS
PROCESS
Farbey el al. (1993) Less than half possess IT AUlomalion 3, A1mosl half don'ljuslify 10 cases idenlify tangible Promoter, rnainly PrineipaIly ex anle.
England plans. Effeeliveness 8, (7116). The resl demand benefits. Only 3 quanlify managerand
16 One linked with global Coordinalion 6. profil juslifiealion Few inlangibles although 7 Cios,very Case sludy business plan. Potenlial and real impael eonsider !hem differenl from indicate them to continue tbe important in its
Theory-praeliee Dieholomy. dieholomy. o!her forms of eapilal, lillle investment. Short tenno approvaI, slandard juslifiealion, only quanlifiable and langible NOl Users.
4/16. benefits.
Mirtidis (1994) 5 possess business plan and All aulomalion. 3 no and 6 yes, 4 with IRR Diffieully in identifying and Promoter not All ex anI •. None ex pOSI.
Greece ITplan. and 2 non financial. quanlifying intangibles. importanl. Similar 10 Dukidis sludy
9 Low generaJ level of Differenl from o!her Differenl from o!her IT depends on (1992- 93) in \he same
Case sludy malurily. investment. investments. Don'l know how finaneial level. country and banking seclor 10 evaluale IT.
Su!herland (1994) Don 'I consider differenl IT Payback !he mosl used CIOS use differenlleehniques Users very Majority ex ante.
South Afriea types 10 evaluale !hem lechnique. 10 CEOS. Managers employ importanI as NOl usually ex pOSI.
14 (13 Cios y II Ceos) differently. Differenl depending on who intangibles. Cios payback controllers of
Case sludy evaluate. strietly. investmenl in IT.
Ooitz (1994) Stralegie role Stralegie role. Consideration as expendilure Slandard leehniques nol Priority given 10 Low Countries fundamentaJ in giving or investments according to important.Nor lhe assignmenl decision makers.
3 (5,10 inlerviwes*firm) priorily 10 or caneelling quantily or department. of qualilive values. Stralégie Promolor very
Case sludy investment in IT. Differs aecording 10 projecl Crileria and classifiealion. important.
I type.
Huerta y Sánehez (1997) 3 Stralegie, Differenlleehniques for IT 5 Stralegie erileria. Managemenl Except in necessi ty
Spain 7 DSS, than for generaJ inveslmenl 2 Slandard-finaneial e. importanI in lhe evaluation, wbere eeos
20, 9 Automation. (11120), due 10 ilS diffieulty 7 Necessity e. vision and role of predorninate.
Case sludy (30%, .NOI due lolT type 6 COSI c. IT.
(70%). In slandard leehniques several lools used I (I3nO).
Golden (1996) Support of explicil Organizalional, leenicaI, and Paybaek (62%) and IRR are CIO is main Similar 10 Bacon 's sludy Ireland objeclives of business and mainly finaneial erileria. In fundamenIal. Don'l konw promoIer . Users (1992) in 80 eompanies of 34 aid in management 15% also strategie criteria other more qualitative tools. only moderalely England, Auslralia, USA
Survey deeision making. and in 44% linked 10 70% use slandard- financial, involved. and N .Zealand.
- --- - ---- - No e2mpelilive advanlagec bu~~e~ _________ - - - M>.'!I>.",~re modem All ex anle. Few ex POSI. Source:Own research.
In Farbey's study a situation is described in which no IT planning exists. This makes it impossible to discover links with business strategy which means, as a consequence, that expected impact is low and not strategic.
Benefits, for this reason, are considered tangible and short term. Farbey confirms the dichotomy which exists between the potential impact of IT and that which companies see as genuinly viabIe and seeks to achieve. She also points to the progressive abandonment of formal financial techniques since only four out of sixteen cases use them. If we compare Farbey's study with our own necessity evaluation group, so far as contextual aspects are concemed, we can see that both studies display OSS or decision support levels of development, and in neither of the studies do we find any indication that IT is considered different from other types of investrnent. As for the investrnent process, the department and IT managers and not the users are those who have the biggest say in its go ahead. Similarly, in our cases the financial managers are those who in addition list intangibles among the benefits to be taken into consideration.
In Mirtidis' study, the purpose of IT is automation and the companies demonstrate a low level of maturity. The evaluation of IT is perceived to be different from that of other investrnent although the companies do not know how to identify and quantify the intangibles. Almost half the cases use OSS financial techniques and those who promote IT investment do not have a decisive say in the process which instead is the responsibility of the financial manager. In our cost study model, the aim of IT is also automation, or the attainment of efficiency. The evaluation of IT is considered different from other investment and therefore OSS financial techniques are not used. Only the cost is taken into account.The financial manager is responsible for investment in IT although the users are the main promoters of both its investment and use.
In comparison with Deitz's cases, a great similarity is observed since strategy evaluation is related contextually to those companies in which strategy plays a fundamental role in determining investment priorities and IT has an important strategic role. As for content, strategic criteria rather than financial techniques are important and as far as the process is concemed, the role of the promoter which, in our group, coincides with the management's vISIon of IT impact, is increasingly relevant.
5
Summary and conclusions
The questions put forward at the outset regarding different aspects of IT evaluation have been dealt with throughout this article and we will now conclude by summarizing our observations.
Firstly, regarding the methodological aspects, we have seen that when there is an abundance of qualitive information and the number of cases is considerable, the cluster analysis is useful for gathering companies together in groups with similar characteristics.
Secondly, using this tooI we have identified four maturity groups: strategic, supporting decision making, automation and initia!. The strategic maturity group is characterized by the involvement of IT in company strategy, by the management's role as promoter in the introduction of IT and by the competitive advantages of IT sought in relation to the customer. In the IS group to aid decision making, IT is organized so as to facilitate the intemal working of the company, support interconnection and spread the information necessary to make decisions. The group containing systems for automation uses IT without integrating it into the organizational strategy and seeks to mechanize tasks in order to achieve efficiency in the production of information. However, ten of the companies studied display a low level of maturity, only three at the strategic level and seven at that supporting decisions.
Thirdly, we have also formed four groups regarding evaluation typologies: strategic, formal-financial and those of necessity and cost. The primary consideration of strategic evaluation are the strategic values in the long term; on the other hand, financial evaluation uses exclusively standard financial techniques in the quantification of benefits. The so called necessity and cost evaluation groups cannot be entirely considered evaluation inasmuch as they only take into account part of the costlbenefit binomial: the availability of financial resources to carry out investment or the attainment of IT at the lowest price. Thirteen out of twenty companies still use low level evaluation methodology although five carry out strategic evaluation and two using standard financial methods.
Fourthly, we can conclude that within the group studied a positive correspondence exists between the level of strategic maturity and the
strategic evaluation of IT; between the level of DSS maturity and necessity evaluation; and finally, between the systems for automation and cost evaluation. In the rest of the modalities the independence, rejection or non-existence of correspondence can be maintained. It is worth pointing out that the standard financial techniques are of liule influence and do not correspond exactly to any specific context. This is justified by the opinion of eleven companies out of twenty that IT evaluation is different from the evaluation of other investments.
Fifthly, in comparing our conc1usions with those of other studies, we have confirmed differences exist when it is used a survey methodology but not using case study. The Golden (1996) and Bacon (1992) survey method reveals a significant use of standard financial techniques, reaching almost 70%, while in the case study this use is seen as very much a minority. Farbey quotes four out of sixteen cases which use financial techniques and in our work we discovered two out of twenty. With regard to the case studies, all agree that the general level of IT maturity in companies is low and that the evaluation methodologies used are also rudimentary. This means that the theoretic and strategic effects of the IT are non yet known by companies and, subsequently, they are not considered in the evaluation process. This confirms the dichotomy between the theory and the practice.
Finally, we are aware that the size of the sample is clearly insufficient to be representative and to generalize the conclusions. So, for the immedialy future research, we will study the maturity level of IT and the evaluation methodology used by a representative sample of Spanish companies of the service sector.
References.
Bacon C J (1992) The Use of Decision Criteria in Selecting IST Investments. MIS Quarterly,
16,3,335-349.
Ballantine J A et al. (1995) Daring To Be Different: Capital Appraisal and Technology Investments. Presented at European Conference On Information Systems (3rd,
Athens, Greece, June, 1-3, 1995)/ edited by Doukidis G et al., 87-97.
Baroudi J J and Orlikowsli WJ (1989) The Problem of Statistical Power in MIS Research.
MIS Quarterly, \3, \,87-106.
Benbasat 10K et al. (1987) The Case Research
Strategy in Studies of Information Systems. MIS Quarterly, 11,3, 369-86.
Brown A (1994) Appraising Intangible Benefits From LT. Investments. Presented at
European Conference On IT Investment Evaluation ,(Ist, Henley on Thames, England, 1994)/ Edited by Brown A and Remenyi D. Birminghan (U.K): Operational Research Society, 187-199.
Canavos G (1988) Probabilidad y Estadfstica: Aplicaciones y Métodos. McGraw-Hill, México.
Davis J G and Hamann J R (1988) In Context Information Systems Assesment: A Proposal and an Evaluation. In Information Systems
Assessment: Issues And Challenges
(Bjorn-Andersen N and Davis G B Eds.) 283-296. The Netherlands, Nort-Holland.
Deitz RMH (1994): Deciding IT-Investment: Multiple case descripcion, current practice &
implication. Presented at European Conference On IT Investment Evaluation (Ist., Henley on Thames, 1994)/ edited by Brown A and Remenyi D. Birmingham: Operational Research Society. 79-89.
Earl M (1989) Management Strategies for Information Technology. Prentice-Hall, New Jersey.
Escofier Band Pages J (1988) Analyses Factorielles Simples et Multiples Objectifs, methodes et interpretations. Dunod, Parfs.
Farbey B et al. (1994) A Taxonomy Of Evaluation Methods. Presented at European
Conference On IT Investment Evaluation (Ist,
Henley on Thames, England, 1994)/ Edited by Brown A and Remenyi D. Birminghan (U.K): Operational Research Society. 44-70. Farbey B et al. (1992) Evaluating Investments in IT.Journal of Information Technology, 7,
109-122.
Farbey B, et al. (1993) IT Investment. A Study Of Methods And Practice. Butterworth-Heinemann, Oxford.
Gable GG(l994 ) Integrating Case Study and Survey Research Methods: An Example in l.S.
European Journalof Information Systems, 3,
Hayes Rand Albernathy W (1980) Our way to Economic Decline. Harvard Business Review, July-August, 67-77.
Hochstrasser Band Griffiths C (1990) Regaining Control of IT Investment: A Handbook for Senior UK Management. Kobler Unit.
Huerta E and Sánchez P (1996) Evaluation of Information Technologies: Practical Cases in Spanish Firms. Presented at European Conference On Eva/uation of IT, (3rd, Bath University School of Management, England, November, I 996)/ Edited by Brown A and Remenyi D. Bath (U.K). 43-53. Lee AS, (1989) A Scientific Methology for MIS Case Studies. MIS Quarterly. March, 32-50. McBride N and Fidler C (1994) An Interpretative Approach to Justification of Investment in Executive Information Systems. Presented at European Conference On IT Investment Eva/uation (I st. Henley on Thames. 1994)/ edited by Brown A
and Remenyi D. Birmingham: Operational
Research Society. 16-26.
Mirtidis D and Serafeimidis V (1994) Evaluating I.T. Investments in Greece. Presented at European Conference On IT Investment Eva/uation (lst., Henley on Thames, 1994)/ edited by Brown A and Remenyi D Birmingham: Operational Research Society. 167-177.
Remenyi D and Williams B (1995) Some Aspects of Methodology for Research in Information Systems Journalof Information·Technology, 10, 191-201.
Robson W (1992) Strategic Management and Information Systems: An Integrated Approach. Pitman Publising.
Serafeimidis V and Smithson S(l994) Evaluation of ISIIT Investments: Understanding and Support.
Presented at European Conference On IT
Investment Evaluation (I st, Henley on Thames, 1994)/ edited by Brown A and Remenyi D. Birmingham: Operational Research Society. 135-147.
Shank J K and Govindarajan V (1992) Strategic Cost Analysis of Technological Investments. S/oan Management Review, Autumn, 3-51.
Sánchez P J (1997) Modelos para la Evaluaci6n de las Inversiones en Tecnologfas de la Informaci6n: Estudio de casos en empresas espailolas. PhD. Thesis, Universidad Publica de Navarra. Spain. Simon K and Sánchez P (1996) Information Technologies in Companies in Navarra: An Exploratory Study. Presented at European Conference On Information Systems (4th, Lisbon. Portugal, 2-4 July, 1996)/ edited by Dias Coellho J et al. 943-952.
Strassman P (1985) Information Payoff. The Free Press, New Y ork.
Sutherland F (1994) Some Current Practices In The Evaluation IT Benefits In S.African Organizations.
Presented at European Conference On IT
Investment Eva/uation. (I st.. Henley on Thames. England. 1994)/ Edited by Brown A y Remenyi D. Birminghan (U.K): Operational Research Society. 27-43.
Symons VJ (1990) Evaluation Of Information Systems: Multiples Perspectives. PhD. Thesis. University of Cambridge (U.K).
Symons VJ and Walsham G (1991) The evaluation of Information Systems: A critique. In The Economics of Information Systems and Software (R Veryard De.). 70-88. Buterwort-Heinemann. Oxford.
Walsham G (1993) Interpreting Information Systems in Organizations. Weley Series in Information Systems. Chichester.
Ward J et al. (1993) Strategie Planning for Information Systems. Wiley Series in Information Systems. Chichester.
Wilner N et al. (1992) Justification of High Technology Capital Investment, An Empirical Study. The Engineering Economist. 37,4, Summer. 341-353.
Wiseman C (1985) Strategy and Computers: Information Systems as a Competitive Weapons. Dow Jones-Irwin. Homewood. IIIinois.
