An Alternative Method for Determining Critical Success Factors of Information System Project

Historically, information system (IS) researchers have been determining deductively critical success factors (CSFs) since 1970s. Meanwhile, most of them have also been performing inductively the CSFs determination in their IS project performance studies. Definitely, each of both methods has strengths and weaknesses, but we argue that researchers and practitioners will get an alternative choice if the methods are combined to share caracteristics. This study has conducted to respond this issue through combination of the abovementioned CSFs determination methods to improve the result validity. It contributes into the debates on ensuring interrelationship between the IS project succes criteria and their CSFs, incorporating the new factors, decomposing complexity of the project, and understanding focus of the stakeholders in the IS project environment. Although, the proposed method has only a combination method, but its implementation feasibility will be a good reference point for the upcoming studies especially in the CSFs determination method selection.


Introduction
Project performance studies in IS environment have been the interest of both researchers and practitioners for many years [1], but few of them focused on the methodological aspects in particularly the CSFs determination method.This method described how to identify critical areas that affect significantly performances of the project [2].Although, this method as an IS methodology has introduced by John Rockart [3] using top-down approach in the late 1970s, but numerous researchers [4]- [12] have been using the bottom-up in their studies.Boynton and Zmud [2] described that the top down approach lends a sense of consistency and completeness.On the other side, they have also indicated that this nonautomated methodology is difficult to use and needs direction from a skilled analyst.It is may be one of the reasons why numerous researchers [4]- [12] utilized the bottom up in their studies because its easiness in the utilization, even by non-professional audiences [13].Nevertheless, researchers [14] who used this method in their empirical studies concluded that they did not successful to prove the critical connection between most of the used CSFs and the project outcome.However, each of these methods has strengths and weaknesses, but the challenges here are how the IS project managers can determine the right CSFs that affect significantly in their projects using the appropriate method.
In addition, researchers and practitioners will get an alternative choice to determine CSFs of their projects if both methods are combined in term of sharing the strengths and weaknesses.This combination is aimed to give a new perspective on understanding the IS project environment related to the organizational, managerial, and environmental issues.This paper presents combination of two methods abovementioned using the compatibility analysis.As described by Belout and Gauvreau [15] that most of models are developed using the previous theories rather than on empirical proofs.Based on this description, authors tried to make a logical sense to combine the top down [2] and the bottom up [4]- [12] methods.The aim is to provide an alternative method in order to deal a new method in the CSFs determination process.The following sections describe literature review, research method, the proposed method, its implementation in a case study, and lastly, this article is concluded with suggestion for the further studies.

Dimensions of the IS Project Success
Numerous scholars [5], [10], [27], [29]- [34] indicated that an overall framework can be developed through combining several perspectives in order to represent the success dimensions.Clearly, two meta-analysis studies by Petter et al. [30] and Urbach and Muller [34] described that the use of multidimensions will produce a high content validity.These studies represented that the multidimensional usage is more reasonable rather than single ones [6], [30], [34] in the project performance measurements, but a number of scholars [23]- [24], [35]- [38] used partially the project dimensions in their studies.Subiyakto and Ahlan [32] presented coherently combination of four dimensions to understand a project in information and communication technologi (ICT) environment.The four dimensions are resources, managerial, directional, and environmental dimensions.
First, Belasi and Tukel [39] concluded that capability of the resources ownership have consequences toward the success of a project.De Wit [24] described that this dimension regarding to the technical aspects in particularly to attain efficiency of the project [40].Several studies [27], [33], [41]- [43] indicated that the dimension is input of the project process.Clearly, McLeod and MacDonell mentioned that this dimension as one of five dimensions that affect software systems development project outcomes in their survey results.
Second, researches [24], [27], [38], [39], [44] mentioned that one of the dimensions which affects the project performance is managerial aspects.Meanwhile, De Witt [24] explained a contradiction; although the project is successful in this dimension, but there is probable that its product will futile.Therefore, scholars [24], [44] separated between the project performance and the project management performance to reduce the complexity.In the processional separation, a number of researchers [24], [27], [33], [44], [45] distinguished between the project life cycle and the product life cycle.
Third, numerous researchers [5], [22], [24], [37], [43] indicated that the directional issues of the project strategic management are part of the CSFs in their studies.For example, Belasi and Tukel [39] recorded that scholars classified CSFs based on strategic and tactical aspects.Wateridge [44] described that each of stakeholders has different attentions in each stage because of their natures.Therefore, the success can be measured based on various stakeholder interests [47] according to the technical issues (short-term), the tactical issues (medium-term), and the strategic issues (long-term).
Fourth, several scholars [19], [48] explained that CSFs are inherited from the particular environment where they have operated.Various studies [5], [22], [24], [37], [43] shown the environment influences affect significantly a project.Specifically, Lim and Mohammed [26] decbribed this dimension in two main parts: micro and macro environments.In addition, Howsawi et al. [10] designed a project measurement concept based on this dimension in four influence levels: context, business, deliverable, and project levels [10].They have also concluded that their concepts contribute towards the body of knowledge by highlighting the effect of the context-related criteria on the project success definitions and plans [10].In short, it is unavoidable that the use of multidimension concepts is more reasonable, rather than the single dimension usage in the IS project measurement studies [6], [30], [34].Utilization of this multidimension meaurement will ensure a high content validity in the project performance researches [30], [34], especially if one can develop coherently the interrelationship between success criteria and success factors as suggested by Stankovic et al. [14] who concluded that they did not represented this interrelationship in their studies.

Research Method
The research process is performed in four stages (Figure 1): Stage 1: Literature review; stage of this research has been performed through literature review of secondary sources that held during Desember 2012 till March 2013.The literatures consist of books, journals, and conference proceedings that have been published during the end 1970s to the early 2010s.Based on natures of these scholarly articles, authors identified and analyzed the CSFs determination methods in order to capture a holistic approach.As it has used by Xu et al. [23], the term IS project has also used interchangeably with information technology (IT) project referring to IS development for deploying business processes and services.In addition, the use of multiperspective approach has also implemented in this study  ISSN: 1693-6930 TELKOMNIKA Vol. 12, No. 3, September 2014: 665 -674 668 based on suggestions of the previous studies [5], [6], [10], [27], [29]- [34] in order to produce the higher result validity than the single perspective usage.
Stage 2: The method conceptualization; authors combined and drew the concepts into the proposed method.As described by Belout and Gauvreau [15] who shown that most of models are developed using the previous theories rather than on empirical proofs, authors tried to combine the two previous methods to share their strengths and weaknesses.This combination has developed using a compatibility analysis to measure the semantic relationship between the derivative CSFs and the identified ones.These works have been done during April 2013.At the end of this stage, authors proposed a new CSFs determination method (Figure 2).
Stage 3: Implementation of the proposed method within a case study; in order to ensure the feasibility of the method proposed for further studies, authors applied the method through an empirical study based on 25 various publications related to the field such as surveys reports, full research articles, literature reviews, and theoretical papers during 1987-2011.This case study has been done from May till August 2013.In addition, authors have also discussed the study with a number of colleagues including at least 5 doctoral students and 5 academicians who had experiences in the similar research field in order to get their comments and sugesstions.At the end of this stage authors formulated four project dimensions with 18 measured CSFs (Table 1).
Stage 4: Report writing; finally, documenting findings of the research have been carried out during September 2013.

The Explanation of the Proposed Method
This part elucidates combination of the top-down the bottom-up methods.Behind thought of this combination is to share strengths and weaknesses of both methods to produce an alternative method for researchers and practitioners in the IS project environment.Based on the literature review, several researchers [2], [3], [13], [28] indicated that the top-down method lends a sense of consistency and completeness related to complexity of the organizational issues, but it is relative difficult to use because needs expert in the implementation.On the other side, numerous studies [4]- [12] used the bottom-up method because of its easiness, but scholars [14] mentioned in their research conclusions that this method is deficient in strength to represent the critical connection between the success factors and its success dimensions.Combination of these two methods is expected to provide four main sharing points: (1) Lends a sense of consistency and completeness to cover the important organizational issues [2], [3], [13], [28] such as representation of the critical connection between the success factors and its success dimensions [2], [14], [19]; (2) Powerful to represent a complex project environment [13]; (3) Helpfulness as a communication tool between the project managers and the project stakeholders [2]; (4) Easy to incorporate new factors and to find which factor should be modified and close to the way of human perceiving it [13].The proposed method is consisting three steps (Figure 2): Stage 1: Implementation of the top-down method; this phase is performed through identification of the project success criteria based on the project dimensions and bringing down these criteria into the CFS level.This step is conducted for lending a sense of consistency and completeness the important organizational or managerial issues [2], [13] in order to ensure interrelationship between the success criteria and the success factors [2], [14], [19].Outputs of this stage are the project success criteria, its project dimensions, and the derived CSFs lists.
Stage 2: Implementation of the bottom-up method; this phase is performed through identify and collecting a number of CSFs from previous studies.Importance of this inductive method is that researchers and practitioners will get a number of the CSFs which their validities have been measured by the prior researchers.However, numerous project performance studies have conducted by previous researchers, but similarities between these studiesand the current study is needed for ensuring validity of the CSFs.Output of this stage are list of the identified CSFs which will use for the further analysis.
Stage 3: Implementation of the compatibility analysis; this stage is performed through analyzing the relationship between the first CSFs list and the second one in order to measure their compatibilities.The analysis is conducted by identifying etymologically whether "there is a relationship" or "no relationship" using a relationship table.The supporting idea for this CSFs compatibility analysis are validities of the identified CSFs from the previous studies and their relationships with the managerial/organizational aspects, as stated by Freund [49] that CSFs analysis was most effective when it is done top-down which begins by identifying the strategic issues into technical ones.Output of this last stage is list of the compatible CSFs that represented the project success criteria.
Stage 3: Implementation of the compatibility analysis; this analysis is conducted in order to identify compatibilities of two CSFs lists: the 18 derivative CSFs (output of the first stage) and the 176 CSFs as identified in the second stage as described by the proposed method.Degree of the relationships between both lists is represented by the following table: The above table shows that average of the relationship degree between the derived CSFs and the identified CSFs from the previous literatures is relative strong around 54%.This degree indicated that the derived CSFs represent the previous ones.In addition, authors argue that this result represents natural trends of a project as indicated by presvious researchers [24,36] that most of projects are conducted to produce a product on efficient and effective ways and how the product will fulfill the business functions [27].Therefore, it is reasonable if support to the short term direction represents the highest degree of the CSFs in 98.86 percentages.Conversely, support to the long term direction represents the lowest degree of the CSFs in 24.43 percentages.

Consultants
Methodologically, as mentioned in the method development section, result of this determination method demonstrates: (1) Interrelationship between the project success criteria and the CSFs levels to cover dissatisfactions from scholars who used the bottom-up method [14]; (2) Compartmental procedures to cover complexity of the IS/IT project environment [13]; (3) Helpfulness to understand the stakeholder focuses in each of the project phases [24], [27], [54]; (4) Easiness to incorporate the new CSFs [13].
These four sharing aspects indicated that this proposed method improves the two previous methods.However, there is a improvement in relation to the managerial, organizational, and environmental issues, but this study indicates that the proposed method is still have weaknesses related to the expertise requirements in its implementation [2] and tendencies of the human bias especially in the interpretation process [2], [14], [19].The first limitation is inevitable regarding characteristics of the organizational and managerial aspects, but utilization of the automatic analysis based on mathematical computation can be one of alternative solutions for the second limitation such as analytic hierarchy process (AHP) or fuzzy cognitive maps (FCM) [13] or index evaluation system [55].

Conclusion
Literature study in IS project performance studies shown that few of scholars concentrated in the methodological aspects especially in the CSFs determination methods.One of significant points of these methods is their result validities in line with nature of the project.Authors found two categories of this method: top-down and bottom-up methods.Definitely, each of these methods has strengths and weaknesses, but researchers and practitioners will get an alternative choice if both methods are combined to share their characteristics.This study was conducted to respond this issue in order to expand the alternative choices for the project managers.Therefore, the proposed method is the major contribution of this work.Although the result of the study is only a combination of both previous methods, but it has proposed with a case study to prove the proposition in order to show its implementation.The proposed method is consisting three steps through implementation of the top-down to produce a number of derivative CSFs, implementation of the bottom-up methods to identify CSFs from the previous studies, and implementation of the compatibility analysis in order to analyze compatibility of both CSF groups.As indicated in the case study, implementation of the method provided four main sharing points: (1) Interrelationship between the project success criteria and the CSFs levels that covered dissatisfactions from scholars who used the bottom-up method; (2) Compartmental procedures for disentangling complexity of the IS project environment; (3) Helpfulness for understanding focuses of the project stakeholders as long as the project life cycle; and (4) Easiness to incorporate the new CSFs in line with the domain growth.
However, these sharing points indicated improvement of the two previous method validities, but this study also represented weakneses regarding to the formulation of the project success criteria and the compatibility analysis method.The formulation needs the expertise requirements in its implementation and the analysis may be still contain bias interpretations.Therefore, in order to improve validity of this proposed method, it is recommended for the next studies to formulate the appropriate success criteria, to focus on the CSFs identification related to the study subject, and to improve the compatibility analysis.In brief, this method shows an alternative choice for both researchers and practitioners in the context for choosing the CSFs determination method.

Figure 3 .
Figure 3.The coherent relationship between the project success criteria, the project dimensions, and the CSFs (adopted from[32])

Table 1 .
List of CSFs compatibilies An Alternative Method for Determining Critical Success Factors of …. (A'ang Subiyakto) 671