Description
The study of how an investment is likely to perform and how suitable it is for a given investor. Investment analysis is key to any sound portfolio-management strategy.
Advancing Return on Investment Analysis for Government IT
A Public Value Framework
Anthony M. Cresswell G. Brian Burke Theresa A. Pardo
Center for Technology in Government University at Albany, SUNY 187 Wolf Road, Suite 301 Albany, NY 12205 Phone: (518) 442-3892 Fax: (518) 442-3886 E-mail: [email protected] www.ctg.albany.edu
September 2006
This material is based upon work supported by SAP . Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of SAP . ©2006 Center for Technology in Government The Center grants permission to reprint this document provided this cover page is included.
THIS PAGE INTENTIONALLY LEFT BLANK
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Acknowledgments
his paper and the project it is based on—”Public ROI: Advancing Return on Investment Analysis for Government IT”—was made possible with funding and guidance from SAP ,a provider of enterprise software for the public sector. In particular, we would like to thank Russ LeFevre and Peg Kates, who have been partners from the very beginning of this work, and special thanks to Tom Shirk, Ingo Hoffman, Rod Massey, Ian Swann, Mor Sagmon, Carsten Friedland, Alex Bratzler, and Bonnie Rothenstein. Several members of the Center for Technology in Government staff made many important contributions to this white paper and the overall project: Sharon Dawes, Jane KrummSchwan, Alison Heaphy, Paula Hauser, and Lucy Dadayan. Finally, this white paper and the case study reports depended on the enthusiastic participation of many government officials from the five case study sites. In particular, we would like to thank Christian Ihle from the Ministry of Finance in Austria, Adam Jansen from the Washington State Digital Archives, Ronny Jacobowitz and Yitshak Cohen from the Merkava Project in Israel, Andy McIntyre from the Integrated Enterprise System in the Commonwealth of Pennsylvania, and Darrell Fowler from Service New Brunswick in Canada. We thank them for their generous hospitality and commitment in helping us conduct our site visits and creating this resource for improving government IT investments.
T
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
THIS PAGE INTENTIONALLY LEFT BLANK
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Table of Contents
Executive Summary Introduction Section I Section II Section III 1 4 5 Why Assess Public ROI for Government IT 6 Research Summary of Public Value Case Studies 8 A Public Value Framework for Government IT Assessment 10 A. The Framework Strategy. 11 B. How Does Government IT Investment Link to the Public? 13 C. What Kinds of Impacts Matter for Public Value. 14 D. What is the Investment? Linking IT to Goals and Business Processes. 16 E. What Kinds of Public Value are Produced? 19 F. Who Receives Value? The Stakeholder Analysis. 20 G. How to Demonstrate the Value? Identifying Variables and Methods. 26 H. What Can Interfere With or Prevent Public Returns? The Risk Analysis. 29 I. Overview of Using the Framework. 32 J. How to Summarize and Present Results? Section IV Appendix 37 Reflections on the Framework and the Value of Public Value 38 The Research Basis for the New Framework 38 A. Consultative Workshop 40 B. Case Study Summaries E ndnotes 43
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
THIS PAGE INTENTIONALLY LEFT BLANK
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Executive Summary
he range and complexity of government information technology (IT) investments makes assessing their returns a daunting prospect. Projects can range from systemwide transformations, to improving financial transparency, to more efficient dog licensing. The returns may be large or small, obvious or obscure, and can run from a few minutes saved in a routine transaction to improving the trust and legitimacy of an entire government. In spite of the difficulty in assessment, however, knowledge about public returns can be vital to fully informed and justified IT investment decisions. Assessing these returns remains a core problem in IT planning and decision making. That problem results from shortcomings in the available methods and models for assessing public returns, what we call public return on investment (ROI). In looking over these existing methods and models we saw three significant shortcomings: Incomplete analysis of public value, resulting in too narrow a scope of what can be considered returns to the public. Lack of systematic attention to how government IT investments generate results of value from the point of view of the public.
T
Weak or absent methods for tailoring a public ROI assessment to the specific context and goals of a government IT investment. Existing methods also deal with risks to government IT investment, but primarily from the point of view of technology development and implementation. The risks that involve the public beneficiaries of the investments merit more attention. This white paper presents a public ROI assessment framework that addresses these shortcomings. We call it a public value framework to emphasize the point of view of the public, not the government, as the basis for the assessment. The framework describes how to identify and assess public value through the kinds of activities shown below in Figure 1. The framework’s strategy is simple in concept, but complex in application: connect what happens in the government (on the left) with the impacts on stakeholders in the public domain (to the right), then report and apply the results. The activities on the left identify the potential value mechanisms and outcome goals. Those are linked to stakeholder interests, impacts, and risks to the right. The curved arrows indicate that in
Figure 1. Public Value Framework
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
1
practice the process would seldom be linear, requiring reflection and backtracking to adjust for learning and new information. The full paper presents a detailed version of this schematic, showing the links among these activities. The public value proposition takes center stage. This value proposition must be broadly conceived to do justice to the scope of government and how it affects individuals, groups, and both public and private organizations. This framework presents a new and more comprehensive way of describing public value, based on six kinds of impacts government IT can have on the interests of public stakeholders: Financial – impacts on current or anticipated income, asset values, liabilities, entitlements, and other aspects of wealth or risks to any of the above. Political – impacts on personal or corporate influence on government actions or policy, role in political affairs, or influence in political parties or prospects for current of future public office. Social – impacts on family or community relationships, social mobility, status, and identity. Strategic – impacts on economic or political advantage or opportunities, goals, resources for innovation or planning.
Ideological – impacts on beliefs, moral or ethical commitments, alignment of government actions or policies or social outcomes with beliefs, or moral or ethical positions. Stewardship – impacts on the public’s view of government officials as faithful stewards or guardians of the value of the government itself in terms of public trust, integrity, and legitimacy. Expanding the view of stakeholder interests in this way brings into focus two distinct but equally impor tant types of public value: the delivery of benefits directly to citizens and enhancing the value of government itself as a public asset. An IT investment that makes government more transparent, more just, and a better steward has added public value, a non-financial but nonetheless important return. This framework describes how to include both in public value assessments. The framework also identifies the basic ways government IT investments link to public value. The simplest link results from an IT investment embedded directly in a service delivery process (Figure 2:A) in a way that enhances its value to citizens or officials. The total value may be a composite of several separate measures: cost savings, quality, enhanced trust. An indirect link (Figure 2:B) can result from infrastructure improvements to business processes. This link may be more difficult to trace and can involve more risk. Enterprise Resource
Figure 2. Direct and Indirect Links to Public Value
Citizen/ Official
V1 V2
government with old IT government with new IT
Citizen/ Official
V1 V2
old government business process new government business process
old IT infrastructure new IT infrastructure
Return = V2 – V1
Return = V2 – V1
A - Direct
B - Indirect
2
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Figure 3. Mix of Direct and Indirect Links to Public Value
environment Citizen/ Official Vi Citizen/ Official Vd government with new IT
Return = Vd + Vi Vi = indirect value Vd = direct value
Planning (ERP) software implementations in our case studies are excellent examples of this kind of investment. Value can also result from a mix of direct and indirect links (Figure 3) when new IT changes the environment. Identifying these links is just part of the story. Each link can involve several valuegenerating mechanisms. The framework describes how to trace these mechanisms working through specific business processes to produce different kinds of value. The value-generating mechanisms are: Increases in efficiency – obtaining increased outputs or goal attainment with the same resources, or obtaining the same outputs or goals with lower resource consumption. In our Austrian and Pennsylvania case studies, for example, new ERP systems helped achieve substantial efficiencies in financial management. Increases in effectiveness – increasing the quality and/or quantity of the desirable thing. Our case study of Service New Brunswick, for example, repor ts how an online registry for land data can contribute to improvements in proper ty tax administration. Enablement – providing means or allowing otherwise infeasible or prohibited desirable activity, or preventing or reducing undesirable events or outcomes. In our Washington
State Digital Archives case study, for example, putting birth and marriage records online enabled research by local historians and genealogists. Intrinsic enhancements – changing the environment or circumstances of a stakeholder in ways that are valued for their own sake. For example, our Israel case showed how enhanced financial accounting and reporting in the Merkava ERP system opened government financial decision making to greater transparency. An IT investment project can deliver public value through any or all of these mechanisms. The framework is deliberately presented at a moderate level of generality to make it most widely useful. Every government IT project will have its own unique goals, value propositions, and stakeholders. So this framework can be used to plan and guide a public value assessment, in combination with measurements, analysis tools, and repor ting techniques chosen for the specific situation. To aid in this regard, the paper presents an overview of more detailed and highly specified assessment methods that can be used in conjunction with this framework. This includes a more detailed flow char t for the assessment process, a summary of several ROI models and methods, and suggestions for analysis and repor ting of results.
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
3
Introduction
very investment decision requires a leap of faith—sometimes a large one— into an uncer tain future. When much of that uncertainty involves the value of the expected returns, risk increases and the investment decision is all the more difficult. This is an even bigger problem for government IT projects, which are notoriously risky and aimed at public value returns that are often very difficult to define and even harder to measure. This white paper takes on one important part of that problem: the question of describing and measuring public value. We chose to focus on public value because it is both the most poorly understood and the most complex par t of the overall problem. We can divide the overall analysis of return on investment (ROI) in government IT expenditure into four parts: costs, internal returns (agency efficiencies, cost avoidance, etc.), public returns, and risks. Of these, the public returns element receives the least amount of attention, though interest in this topic is growing. The key question about public returns, of course, is what do we mean by public value and how can we observe, measure, and document its creation. The goal of this white paper is to help answer that question.
E
The approach we take to answering that question takes a point of view based on our experience, analysis, and the background research for this paper. From this point of view we see two sources of public returns: (1) value to the public that results from improving the government itself, and (2) value that results from delivering specific benefits directly to persons or groups. We also see potential value creation that goes far beyond the traditional financial and service evaluation data. Value creation can come as much from increasing the integrity and transparency of government as from reducing costs through online tax payments. This expanded scope of value includes an often wide range of stakeholders, each with their special interests and expectations from government. This point of view dictates an expansive way of seeking public value. We call this a public value framework, meaning that it is less a specific measurement tool and more a way of identifying and as sessing as wide a range of public value as possible. This is not a small task, and cannot be completed in one such effor t. But this approach can advance the search for a way to effectively measure public value.
4
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Section I: Why Assess Public ROI for Government IT?
he scope of government investment in IT and the associated problems certainly deserve serious attention. Both the levels and growth rates in government IT spending are substantial. The most recent data available shows that the level of this investment in both the industrialized countries and the developing world has grown to a ver y large scale. European government IT spending is expected to increase from US$110 billion in 2005 to US$119 billion by 2007, with US$26 billion in the UK alone, which is about 40% above Germany or France. IT spending by Asia-Pacific region governments, excluding Japan, is expected to reach US$31.7 billion by 2010, from US$22.7 billion in 2006. The Chinese government expenditures of USD$5 billion in 2004 are expected to grow 16% in the following five years. India spent US$943 million on e-government in 2002, and this figure is predicted to increase by 15% a year to US$3.3 billion by 2009. The U.S. federal budget for fiscal year 2007 provides $64 billion in funding for IT investments, approximately a 3% increase from the 2006 enacted level of $62 billion. Total state and local government IT spending was about $55 billion in FY2004 and is expected to grow to $62.4 billion by FY2009. Though what is included in the IT category may vary across these examples, the overall scale of expenditure remains substantial.1 This level of expenditure is receiving increasing scrutiny. The reason for this scrutiny is clear, according to Andrea DiMaio, a Gartner Vice President focusing
T
on the public sector: “If governments do not accurately measure the full value of their IT investments, they risk a serious political backlash. They will be accused of wasting billions of pounds of taxpayers’ money on unnecessary technology.” This sentiment has reached the US Congress. In July of this year the Senate Appropriations Committee recommended no funding for the administration’s 2007 e-government initiatives, reporting that “… the committee has no confidence that the amounts being assessed have any relationship to the benefits anticipated to be returned.” And in the House of Representatives, the current appropriations bill calls for a cost-benefit analysis of all e-government initiatives. According to Mike Hettinger, staff director at the House Government Reform Committee’s Government Management, Finance and Accountability Subcommittee, “the language speaks for itself and reiterates what the subcommittee has been saying for the past year, that in order for this initiative to be successful, we need to have a better understanding of the costs and benefits and clearer guidance for the agencies to follow.” 2 Certainly this enhanced attention to ROI for IT must include a comprehensive and effective way to deal with the public value side of the problem.
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
5
Section II: Research Summary of Public Value Case Studies
his framework is based in part on the results of five case studies that examined how government IT investment projects came to deliver value to the public. 3 The projects were the Integrated Enterprise System in the Commonwealth of Pennsylvania and the Washington State Digital Archives in the US, the Merkava Project in Israel, the Austrian Federal Budgeting and Bookkeeping System in Austria, and Service New Brunswick in Canada. All five case study reports and the white paper are available on CTG’s Web site.2 In the case studies we examined how the IT investments were conceived and developed, with particular attention to the role of public value in the process. We took the view that government IT investment generates public value in two ways: By improving the value of the government itself from the perspective of the citizens, and By delivering specific benefits directly to persons, groups, or the public at large. The first is based on the idea that, assuming a benign government, the better it functions overall, the better off its citizens will be. The government is an asset to the community or nation that delivers a wide range of values. Internal improvements make it a more valuable asset to the public. The second way of generating value has three forms: financial, political, and social. Financial value results from lowering the cost or increasing the efficiency of government or delivering direct financial benefits to the citizens. Political value consists of increasing participation, fairness, transparency, legitimacy, or conferring political capital to elected officials or citizens. Social returns include increased social status, stronger relationships, or opportunities; increased safety, trust in government, and economic advantage.
T
These general understandings of public value guided the data collection and presentation of the case results. The case studies examined public returns to the overall IT investment, including returns generated by a government IT investment and the possible mechanisms to produce them. We did this by considering the links between goals, implemented systems, government performance, and public returns in terms of where they represent value in the chain shown in Figure 4 (next page). The public returns from the investment can flow from the internal improvements in the agency resulting in returns to individual citizens and the public at large (the main flow through the center). Other returns can flow to the political system and the economic environment (below the center), or through effects on other agencies (secondary performance gains). This general view of public returns informed the case studies and helped summarize the results.
6
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Figure 4. Public ROI Value Propositions
Public Cost Internal Agency Returns
*
IT Investment
Secondary Performance Gain
*
Business Goal or Problem Private Cost (?) Vendor/ Industry Benefit
* *
Public-atLarge Benefit
*
*
Direct Citizen Benefit
* * *
Policy/ Electoral Benefit
*
*
*
General Economic Development
* *
Development Risk Benefit Risk Direct Benefit Flow Indirect Benefit Flow Direct Benefit Accumulation Indirect Benefit Accumulation
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
7
Section III: A Public Value Framework for Government IT Assessment
T
his framework is designed to assist government IT executives and analysts in understanding and measuring the value to the public of government IT investments. The main goal is to produce an assessment of public returns that is credible, persuasive, and highly relevant to the investment decisions faced by the government. We use value here, rather than return to emphasize the broad scope of the framework. Most methods for assessing return on investment focus on financial or economic metrics; this approach includes a much broader view of how IT investments can produce results of value to citizens or to the society as a whole.4 This concept of value includes more than the usual financial or economic metrics common in ROI analysis. It is a new and expanded way of understanding the results of government IT expenditures. We call this set of ideas a framework to indicate that it is more than a particular method for public value assessment. It is broad in scope so that it can be applied to virtually any government IT investment, from simple Web sites to governmentwide information systems and architectures. It is broad in scope because this range of investments requires a comparable range of assessment methods. Our framework, therefore is a way of thinking about and organizing the analysis of a family of problems that can encompass many methods. A spreadsheet, for example, is a framework for working on a broad class of problems or analytical tasks. Any par ticular spreadsheet may include specific methods, such as scenario analysis or a net present value calculation. Instead of guiding the assessment process in terms of calculations in a matrix, however, this framework provides an analysis process that starts with a high level view of the IT investment and then drills down through successive steps to identify the specific measures and methods that will reveal and document public value.
In this way, the framework offers both a systematic way of thinking about public value and a way to apply that thinking to par ticular IT investment decisions. The drilling down process is necessary to tailor a specific public value assessment to the nature of a par ticular investment decision. The framework shows how to take into account how public value can change across the many interests of citizens and groups in interacting with governments. In the morning, for example, an executive doing business with the government may think about how IT speeds payments on her government contracts, in the evening while helping with homework she may observe how computers improve the quality of schools, and while watching the news on TV at night she might learn how a new crime mapping system makes the neighborhood safer. These ways of thinking about public returns include both easily measured value, like improved financial flows, as well as highly subjective ones like public safety, service quality, or government integrity. A framework for public value assessment must provide a way to deal with these many perspectives and possible measures of public value. Not every aspect of public value is relevant for a par ticular IT investment. The Washington State Digital Archives project, for example, had no particular public safety related goals, but is of considerable value to genealogists and local historians. Our framework begins the process of narrowing and focusing by starting with the three basic elements of analyzing public value: the investment, the government operations affected, and the stakeholders. At the beginning of a public value assessment, an analyst often knows that there are many connections among theses three elements, but those connections are poorly specified and understood. The situation might be described by the overlapping parts of
8
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Figure 5 below. The public value is to be found by unpacking, so to speak, the area of overlap. The task of the assessment, therefore, is to identify the connections and gather data about how the IT investment produces value for the relevant stakeholders. Close attention to all three elements is essential. The connections among these elements are the keys to a fully informed public value assessment. The framework provides a way of describing these connections to show how public value is generated and the risks involved therein. Focusing on one or two elements alone cannot reveal the necessary scope of public values involved in an IT investment or how they can be assessed. The risks of slighting one or another of these elements in an assessment can be substantial. In addition to missing significant public returns, such limited thinking can lead to stakeholder resistance, flawed technology decisions, or poor integration with or disruption of business processes.
Analyzing those connections can, in principle, start in any part of the problem. Ultimately, of course, these three elements must be considered together in order to ensure that the value assessment for any par ticular investment project or system is tailored to the specific value context, investment situation, and the beneficiaries involved. In practice, however, the assessment must begin somewhere. The framework presented below describes the logic and methods that guide the assessment process. The sequence of activities shown for an assessment is not designed or intended, however, to suggest how IT investment decisions should be made.
Figure 5. The Basic Elements in the Public Value Framework
The Stakeholders Public Value The Technology Government Investment Programs & Operations
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
9
A. The Framework Strategy
The framework describes public value assessment as a series of steps to gather specific data and use it to answer questions that lead to public ROI results. The results can include measures and documentation of public returns, risk analysis of the threats to achieving those results, and suggestions for presenting results to decision makers. Each component of the framework deals with a different set of questions and possible data to use in subsequent steps, and so for th. By working with the ideas and methods presented in the framework, the analysis can produce a public ROI assessment that is broad in scope, yet tailored to the par ticular investment of interest. The steps are not intended to be used in a lock-step manner. The questions and inquiry involved in one part of the framework may be obvious and easily answered in some cases and quite complex and difficult in others. It may be necessary to explore later steps before preceding ones are finished, and then cycle back. In
general, however, the sequence follows the path shown in Figure 6 below. The steps illustrate how the process of value identification and measurement carries through from the goals of an IT investment, through the value generating mechanisms of the business processes, connection with stakeholders, to specific data and reporting. These are the main components of the framework. By starting with a deliberately broad scope, then narrowing to specific questions for each project, this approach addresses one of the shortcomings of previous work on ROI for government IT, namely the narrow scope of possible value questions considered. By expanding the initial scope of analysis to include a broad view of public returns, the framework can lead to a more comprehensive result. The framework thus does not replace or supersede the existing methods developed by others for assessing returns to government IT investments. Rather the framework shows how these methods can be used in assessment, what additional value issues should be addressed, and how additional assessment methods may be used.
Figure 6. Steps in Applying the Public Value Framework
10
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
B. How Does Government IT Investment Link to the Public?
The question of how an IT investment can impact public value can be answered initially in terms of three impact mechanisms. These three, alone or in combination, describe the general ways IT investments can connect with the public, as revealed in our research. Identifying these linking mechanisms is an important step in the analysis process because it leads to a more detailed examination of impact pathways than might otherwise occur. IT investment can enhance public value in many ways, some of which are only indirectly related to the performance of a particular IT system. Attention to these indirect and more complex mechanisms can help ensure a full picture of public value results is developed.
Figure 7. Type 1: Direct Service Impact
government with old IT government with new IT
Citizen/ Official
V1 V2
Return = V2 – V1
value propositions for these investments go beyond the important but obvious cost and time savings to include attention to service quality, access, equity, and the full range of values described below.
2. Indirect Service Impacts
Indirect service impacts (Figure 8) occur when back office or infrastructure investments produces changes in a government business process. As in Type 1 mechanisms, the value to the public is reflected in the changed interaction or transaction with a government business process. However, the process change results from an IT investment at least one step removed from interaction with the public. Because of this indirect route, it may be more difficult in Type 2 cases to trace the links from the IT investment clearly and unambiguously to the public. Risks to the effectiveness of the investment are higher as well, due to dependence on business process changes that are independent of the IT itself. The IT investment may have potential results spread over many business process and may interact with other technologies, further obscuring the impact linkages. The ERP system implementations in Austria, Israel, and Pennsylvania that were examined in our case studies are excellent examples of this kind of investment. The primary impact of these systems is on the internal, back office operations of the government. How to identify and describe
1. Direct Service Impacts
Direct service impacts (Figure 7) occur when IT is embedded directly in a service delivery process, generating service changes that enhance value to the citizen or official involved in the service process. The net public value is just the difference between the value of the new and the old. Value in this sense, and in the other mechanisms as well, is assessed from the point of view of the public participant in the transaction, not the government. The total value may be a composite of several separate measures, such as lower cost to the citizen, plus increased satisfaction with quality of service, plus enhanced trust in government. The specific kinds of value involved are described in more detail in a later section. The e-government investments in our case studies and much of the literature on e-government provide many examples of this kind of investment. These include systems for fee and tax payments, license application and renewals, obtaining information, filing forms, etc. The public
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
11
Figure 8. Type 2: Indirect Service or Impact
Citizen/ Official
V1 V2
old government business process new government business process
old IT infrastructure new IT infrastructure
Return = V2 – V1
these linked back office operations is discussed in a later section of the framework in terms of business reference models (see Figure 10 on page 15). The impacts on these internal operations may be very positive and extensive, and still remain hard to trace to many kinds of value gains for individual citizens or organizations. This is an impor tant part of the public value problem dealt with in detail in the value types discussion to follow. ERP systems also contribute to public value generation through the Type 3 mechanism discussed below.
3. Mixed Service & Environmental Value Impact
In this third type of impact mechanism (Figure 9 below), the links to public value
are more complex. The direct value mechanism shown here as Vd, is the same as a Type 1 or Type 2 mechanism, producing a value increment for a citizen, official, or group (shown in a simplified way). In this type of system, the new IT is also linked to changes in the environment and relationships between the direct beneficiary and other entities (persons, groups, organizations) in the public arena. This indirect mechanism can result in additional public value flowing from interactions outside the government, between other persons or organizations. The changes in the environment may also produce direct value beyond the direct transactions. There are many examples of this type of value mechanism in our cases and in other settings. In the Washington State Digital Archives case at least two such impacts
Figure 9. Type 3: Mixed Direct, Indirect Service & Environmental Impact
environment Citizen/ Official Vi Citizen/ Official Vd government with new IT
Return = Vd + Vi Vi = indirect value Vd = direct value
12
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
occurred. The ability of the county auditors to shift responsibility for preserving digital records to the state archive produced savings in cost and workload at the county level (Vd). This allowed county auditors to improve services to their citizens in other areas (Vi). In addition, the accessibility of the government records online increases the overall environment of transparency for the government, benefiting all citizens. In the Service New Brunswick case, Web access to company registrations provided direct value to citizens such as accountants, lawyers, financial institutions, and the general public searching for such business information (Vd). In addition, the decision by CGI to locate its Global E-Government Headquarters in Fredericton, New Brunswick, due in large part to its strong partnership with Service New Brunswick, benefits the economic development of the city of Fredericton and surrounding areas in the province (Vi). In a different way, putting some forms of information online, such as the Toxic Release Inventory in the US, enable citizens to obtain benefits from other transactions, such as lawsuits against polluters (Vi).5
the same outputs or goals with lower resource consumption. In our Austria and Pennsylvania case studies, for example, new ERP systems helped achieve substantial efficiencies in financial management and other core administrative functions of government. Increases in effectiveness – increasing the quality and/or quantity of the desirable thing. Our case study of Service New Brunswick, for example, repor ts how an online registry for land data can contribute to improvements in proper ty tax administration. Enablement – providing means or allowing otherwise infeasible or prohibited desirable activity, or preventing or reducing undesirable events or outcomes. In our Washington State Digital Archives case study, for example, putting birth and marriage records online enabled research by local historians and genealogists. Intrinsic enhancements – changing the environment or circumstances of a stakeholder in ways that are valued for their own sake. For example, our Israel case showed how enhanced financial accounting and reporting in the Merkava ERP opened government financial decision making to greater transparency. The examples above illustrate public returns that accrue in addition to, and largely independently of, internal efficiency gains that may accompany the IT investment. These value generators can also operate together, increasing the overall return. Research by the local historians mentioned above was enabled by access to previously inaccessible records; online access to these records also made research much more efficient compared to paper records. Access to crime mapping information by citizens could also improve the efficiency of choosing a place to live or locate a business.
C. What Kinds of Impacts Matter for Public Value?
Just identifying these operational mechanisms in general terms, however, does not tell the full value story. Each mechanism can involve more than one kind of value generator. These mechanisms show how IT investments can link to increased public value, but not how that increase is produced. Each mechanism has the potential to generate more than one kind of public value increase, depending on the details of the situation. The framework recognizes four basic kinds of public value generators, listed below, each with a different range of measurements and implications for assessment: Increases in efficiency – obtaining increased outputs or goal attainment with the same resources, or obtaining
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
13
D. What is the Investment? Linking IT to Goals and Business Processes.
1. Links to the Business of Government
This analysis of public value generation requires identifying how the IT investment project links to government goals and activities. IT investments generate value in relation to the policy and administrative setting in which they operate. The goals and business processes provide links between how the technology operates and the interactions with stakeholders that generate value. The analysis, therefore, includes linking the investment to the relevant government goals, operations, and business processes. This linking process is more complex than it may appear, requiring a comprehensive and reasonably detailed picture of government goals and operations. Such a picture will ensure that all the relevant links between the technology and business processes will be identified. Many government IT investments have potential links across many agencies and processes. The Washington State Digital Archives, for example, collects records from hundreds of state and local agencies. The Merkava project in Israel will eventually involve all government agencies. The Pennsylvania IES supports human resources management, budgeting, and other administrative functions for all executive agencies in the Commonwealth. The relevant business process setting of an IT investment, therefore, can be quite extensive. To deal with this contingency, this part of the framework is based on just such a comprehensive picture. This is a useful analysis strategy because it helps draw attention to value generating aspects of the investment that might be missed because they result from indirect effects or complex interactions across many agencies or processes.
The first part of the linking takes advantage of the comprehensive integrated descriptions of government goals and processes that are found in the enterprise architecture work currently underway for US and other governments. For our purposes here, the most useful comprehensive description of government processes is in the US Federal Enterprise Architecture (FEA) Business Reference Model (seen in Figure 10).6 With some minor modifications, the components in this FEA model can be used to identify the business operations and general goals of an IT investment in most government agencies. Local governments, for example, do not typically have defense and national security operations. The supporting documents that form part of the Business Reference Model contain descriptions of each of the components. These descriptions can be used to clarify meaning and help identify which components are linked to a particular IT investment. While comprehensive, this FEA model is not the only enterprise architecture model that may be used. There are a number of comparable models or business process frameworks that could work as well, such as those developed for the EU, the UK, Hong Kong, and Malaysia.7 The National Association of State Chief Information Officers (NASCIO) has an enterprise architecture model, adapted for US state governments, that may also be used. The European Commission also suppor ts additional framework activity through the Athena Project, which has published similar documents.8 Using this model, the analysis proceedsby identifying where the IT investment links to government business. This linking can be based on the new technology’s stated goals, organizational location, and intended operational profile. The detailed descriptions of each Business Reference Model component are useful in identifying these links. If needed, those responsible for the analysis can enlist a team of IT and operational experts to help ensure all relevant links are identified.
14
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Figure 10. Elements in the FEA Business Reference Model
Services for Citizens Community and social services Defense and national security Economic development Energy Science and innovation Income security Law enforcement Natural resources Workforce management Information& technology management
Purpose of Government
*
Correctional activities Disaster management Education Environmental management Health International affairs & commerce Judicial activities Transportation Financial management
*
Mechanisms used to achieve purpose
Mode of Delivery Government Service Delivery Direct Services to Clients Knowledge creation & Management Public Goods Creation Regulatory compliance & enforcement
*
Financial Vehicles Financial assistance Credit & insurance Transfers to local governments
Government Operations Support Functions
Support Service Delivery Legislative relations Regulatory development Planning & resource allocation Internal risk management & mitigation
*
Public affairs Controls & oversight Revenue collection General government
Resource Management Functions
Management of Government Resources Supply chain management Human resource management Administrative management Financial management Information& technology management
*
*
*
To illustrate how to use the model, we marked several of the components in Figure 10 with asterisks. The asterisks mark an example of the business links we identified for an IT investment project in the Merkava case study used for this framework: an online reverse auction system for procurement. The agency conducting the online auction sets the terms of a procurement, publishes them, and qualifies vendors as potential bidders. At a predetermined time, a Web site is opened for bidding. All qualified vendors can then submit authenticated price bids to a public space on the site, visible to all other vendors. The bidding continues until the low bid remains unchanged for a predetermined time (e.g., five minutes), closing the auction.
The low price bidder wins the contract. The entire auction takes place in public view and is recorded in detail. For an illustrative example, consider the procurement of new police radios for a law enforcement agency. Based on the nature of the e-procurement system and the component descriptions, we see it as linked to the five marked components in Figure 10: law enforcement, financial management, supply chain management, direct service to clients (vendors), and planning and resource allocation.
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
15
2. Links to Business Processes
Making further links to business processes means connecting the larger goal or function from the Business Reference Model to specific activities identifiable as business processes. For this step, picking the appropriate level of detail is important. Choosing too general or too fine-grained a process analysis will tend to obscure stakeholder interactions that are necessary to identify public value. For this step in the analysis, we recommend another enterprise architecture tool: the Zachman Framework. 9 The descriptions in the first two rows of the Zachman Framework (Scope and Business Model) are a good guide for the most useful level of detail. This method identifies several business processes for the e-procurement example. They include: determining communication needs for the law enforcement agencies budgeting for the procurement recruiting and qualifying vendors conducting the auction paying the vendors
The list could be extended for fur ther analysis by adding deployment of the radios, training for users (human resource management), and evaluation of the impacts of the new radios use in the agencies and business processes where installed. To keep this example manageable, however, we are limiting the analysis to the processes listed below. The components identified from Figure 10 are linked to the business processes in the rows of Table 1 below. Identifying the business processes leads to two key questions: (1) How does a business process generate increased public value? and (2) For whom? A way of answering the first question is recorded in the third column of Table 1. Our analysis of public value generating mechanisms is shown in the discussion of value impact mechanisms below.
E. What Kinds of Public Value are Produced?
Identifying the linkages between business processes and public value generators described above carries the analysis a critical step farther: from how the IT changes a business process to the impact of those changes for a par ticular public or
Table 1. Business Component, Process, and Value Mechanism
Business Component Law enforcement Financial management Supply chain management Planning & resource allocation Direct ser vice to clients Financial management Business Process conduct auction budget for procurement qualify vendors for auction deter mine communication needs conduct auction pay vendors Public Value Mechanism citizen - type 3 citizen - type 3 citizen - type 3 vendor - type 1 citizen - type 3 gov't staf f - type 1 vendor - type 1 vendor - type 1
16
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
stakeholders. That step is accomplished by looking to the value of the impact in terms of the interests of one or another stakeholder. So the framework includes an analysis scheme for taking into account interests that can apply to the full range of stakeholders. The framework employs these six basic kinds of interests: Financial – impacts on current or anticipated income, asset values, liabilities, entitlements, and other aspects of wealth or risks to any of the above Political – impacts on personal or corporate influence on government actions or policy, role in political affairs, or influence in political parties or prospects for current or future public office Social – impacts on family or community relationships, social mobility, status, and identity. Strategic – impacts on economic or political advantage or oppor tunities, goals, and resources for innovation or planning Ideological – impacts on beliefs, moral or ethical commitments, alignment of government actions or policies or social outcomes with beliefs, or moral or ethical positions Stewardship – impacts on the public’s view of government officials as faithful stewards or guardians of the value of the government itself in terms of public trust, integrity, and legitimacy This last interest—stewardship—is deliberately not included in the ideology categor y, though it could logically fit there. Setting it off separately emphasizes its importance in the overall public value framework. Just as corporate managers and directors are responsible for stewardship of
a corporation’s integrity and assets on behalf of stockholders, public managers and elected officials have a parallel responsibility for government on behalf of the public. Recent financial scandals in the US have demonstrated how lack of stewardship in major corporations (e.g., Enron, WorldCom) can destroy their financial value. The government officials in our cases expressed a similar stewardship responsibility in terms of, for example, using IT to increase transparency and accountability. Public surveys of trust in government show that citizens have similar stewardship expectations of government.10 Clearly this list goes well beyond the internal efficiency or service quality impacts of an IT investment. Of course, attending to this expanded concept of value raises many problems of definition and measurement. However, our case research and results from many other kinds of inquiry show the importance of this more comprehensive treatment of value and interests. Many of the measurements and data implied by these interest types are found in existing ROI methods as well. However, none of the methods we reviewed for this white paper cover the full range represented here. Working with this expanded range of value types represents one impor tant contribution of this framework. It is also clear from this way of describing public value that it supplements rather than replaces methods that assess the efficiency gains or savings returns captured by government agencies internally. For example, it seems safe to say that a Department of Motor Vehicle’s internal cost savings from putting license renewals online are invisible and largely irrelevant to drivers. These savings would have no detectable impact on overall tax burdens or benefits resulting from a shift of government resources to some other service. The information about the savings, however, is another matter. If government officials make some political use of the cost savings information, that would represent strategic value to those officials or to their political
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
17
allies. The public learning about the savings might also improve their opinion of the government’s stewardship, fairness, or other ideological value.
thinking and include multiple opinions and information sources Identifying multiple stakeholder roles, internal and external to the organization setting (e.g., internal user, customer, vendor, developer, manager, oversight, politician, taxpayer, analyst, advocate, etc.) Identifying stakeholder expectations, influence potential, past and future par ticipation possibilities, and level of interest In this kind of analysis, continuing reference to the nature of the investment and impact mechanisms will help produce the needed focus and detail.11 The results of a stakeholder analysis have implications for value assessment and for understanding their potential to influence the investment project. These results can be presented in ways that show the multiple dimensions used in the analysis. An example of results from a hypothetical stakeholder analysis is shown in Figure 11. This figure combines data for 14 stakeholders’ roles (SA = Stakeholder A, etc.) and times of possible influence in a project’s lifecycle. This kind of display illustrates some of the complexities of stakeholder analysis and the possibilities for interactions among different stakeholder groups. A more complete analysis would include estimates of the stakeholder’s influence capabilities and specific interests in the project development and outcomes. These issues are discussed in more detail in the section on risk analysis (section III.H). To focus on stakeholder interests, the illustration in Table 2 below shows how the results of a simple stakeholder analysis can be linked to the business process, value types, and mechanisms. The business processes identified in Table 2 on page 20 link to a partial list of stakeholder types in the columns: citizens at
F. Who Receives Value? The Stakeholder Analysis.
Completing the analysis of value impacts requires identifying those with an interest in the value generating process: i.e., the stakeholders. The kind of stakeholder analysis required for this task has three main parts: (1) identifying the persons or groups (including organizations) whose interests are potentially affected, (2) identifying what their specific interests may be, and (3) assessing their role and potential influence in the deliver y of public value. The first two par ts are necessary for the analysis of value impacts and possible assessment strategies and so are discussed here. The third part is more relevant to the risk analysis discussed in more detail in section III.H below. Identifying stakeholders for the framework will depend on in-depth knowledge of the context of the investment and the agencies involved in its development and use. Typically, stakeholder analysis engages a group of participants with extensive knowledge of the political and organizational setting of the investment and, hopefully, with experience in this kind of analysis. Because the analysis is so context-dependent, there are no standard processes to follow. There are consistencies, however, among the many versions of stakeholder analysis methods found in the management, planning, and assessment literature. These include: Involving multiple participants with wide knowledge of the stakeholder environment Looking widely to identify all relevant stakeholders through brainstorming and related methods to stimulate divergent
18
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Figure 11. Stakeholder Analysis Matrix12
Type of Participation
Inform Initiation Identification Consult Partnership Control
SA SH SK
SC SB SF SM SN
SI SJ
SE SD
Stage in Lifecycle
Planning
Execution Implementation Controlling Monitoring and Evolution
SG SL SO SD SA
Sx = Stakeholder X
Closing
large, vendors, and elected officials. An actual assessment would include a more detailed list of stakeholders. The value types most likely to map onto each stakeholder type are entered in the cells. The basis for this mapping is linking the nature of the value as understood in that context with detailed knowledge of the stakeholder groups. This part of the assessment can include data collection about the interests and expectations of stakeholders through interviews, surveys, or focus groups if needed. The entries in the column for the citizens at large, for example, are based on the assumption that their interests in government stewardship would be served by the e-procurement system. While a reasonable assumption, this could be confirmed by collecting opinion data from a sample of citizens, or relying on previous opinion research. Similarly, the entries in the column for government IT staff are based on
assumptions about what advances the interests of those staff members. A successful procurement system would provide political value and some strategic advantage to elected officials, as well as enhance their reputation as good stewards of public resources.
G. How to Demonstrate the Value? Identifying Variables and Methods.
Using the framework up to this point has identified many kinds of data relevant to assessing public value. The next step is to choose the specific variables or points of observation for collecting the assessment data. This is as much a practical problem as an analytical one. Many variables may be relevant for each combination of
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
19
Table 2. Stakeholder Value Matrix
Stakeholders Business Process Value Generator Impact Type Citizens at Large Elected Officials
Vendors
Budget for procurement
intrinsic
3
stewardship
Deter mine needs
intrinsic, ef fectiveness
3
ideology, stewardship
political, strategic
Qualify vendors
ef ficiency, intrinsic ef ficiency, intrinsic, ef fectiveness ef ficiency, intrinsic
1,3
ideology, stewardship
financial, strategic
political, strategic political, strategic, stewardship political, stewardship
Conduct auction
1,3
ideology, stewardship
financial, strategic
Pay vendors
1,3
stewardship
financial, strategic
stakeholder and value type. A few variables that would be relevant to this example are shown in Table 3. This table illustrates two important aspects of its role in the framework and assessment process. First, it is highly unlikely that applying the framework to any IT investment would lead to variables in all, or even most of the cells of this matrix. This matrix can be thought of as an heuristic device, prompting questions about what might be useful and available variables for each row and column without expecting to fill them. The second is that identifying a specific variable relies on combining information about stakeholder interests, the value type, the impact mechanisms, and the context. This is a complex and demanding process. This section offers additional guidance on choosing the best public return variables for a given assessment. In addition, the other ROI methods described below include many
variables and additional methods that can be helpful in that task. However, the more the choice of variables can be tailored to the specific public value context, the more valid and persuasive the assessment is likely to be. An actual public value assessment should be based, of course, on the best available information. But actual assessments take place in practical settings of limited resources and access to data, plus being par t of the additional work needed for internal returns and costs. The priority setting described in the risk analysis section can narrow the field to only the most important public value outcomes. The section below describes additional strategies for choosing the appropriate variables and analysis methods.
20
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Table 3. Stakeholder Variables by Type
Stakeholder Impact Variables
Value Type Financial
Citizens at Large local economic development
Vendors increased revenue
Elected Officials
Political
suppor t from vendors
positive media coverage Social enhanced reputation in agencies receiving radios increase level of market competition increase in citizen access to infor mation about procurement: greater transparency Strategic increased market share enhanced reputation for being moder n & progressive enhanced reputation for promoting market competition
Ideological
increased knowledge about competition Stewardship increased trust in gover nment reputation for frugality & good management
1. Variables and Analysis Methods
The choices of variables and analysis methods for the empirical par ts of a public value assessment should be considered together. In terms of basic measurement and analytical methodology, what constitutes an appropriate analysis depends, in part, on the types of data and variables involved.
The scope of this framework, as applied to a particular IT project, could encompass a very wide range of data types. Many kinds of quantitative data from financial sources, operations research, and surveys are appropriate for statistical analysis, modeling, and simulations. Many of the social, political, and intrinsic value returns can be expressed in normalized scales, or may best be revealed in qualitative terms or in simple dimensions that are not suitable
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
21
for much quantitative manipulation. To help guide the assessment, given the very large number of possible public return data types and variables, the framework provides two kinds of resources. The first is a general scheme suggesting variable types and sources for different value types (see Table 4). The second consists of summaries of the approaches and variables available in a range of existing ROI methods that can be of value in completing an assessment (see Table 5). A wide range of possible public value data can be identified by the methods presented here. The framework approach is based on the assumption that virtually any kind of data can be useful in describing public value creation, from the most precise quantitative figures available from financial or physical measurements, to material as diverse as the content of blogs or observation of user or customer behavior. A conclusion about public value creation
requires an inference, since value does not stalk about wearing a label. Valid inferences about value can be formed from qualitative as well as quantitative data, content analysis as well as statistics. Taking these four principles into account, choices made about how a specific analysis proceeds should be based on three criteria: 1) What constitutes the best data? 2) What kind of analysis is appropriate to that type? 3) Who will be the audience for the conclusions reached? The best kind of data available will be specific to the operational and stakeholder context. The kinds of analysis appropriate to various data types are shown in summary form in Table 4.13 Beyond these general considerations, the choice of variables and analyses for an assessment can draw on a volume of existing work on ROI methods for guidance. These methods, summarized in Table 5, vary widely in the number and type of variables used, the scope of public value considered,
Table 4. Variable Type for Public Value Data
Variable Type Examples Analysis Methods averages, totals, dif ferences, percents, parametric & nonparametric statistics (cor relation, regression, etc.), simulation, time series, optimization averages (with assumptions), frequencies, scale analysis, nonparametric statistics (x2, etc.), cluster analysis cor relation, frequencies, scaling analysis frequencies, clustering, network analysis content analysis, word frequency, semantic analysis process maps, network analysis, behavior patter ns
ratio scale
financial data, time, age, distance, frequency, transaction volume
inter val scale
opinion scales, rating scales, grades, Liker t scales
ordinal scales nominal variables text/discourse data obser vational data
rankings, rating scales gender, ethnicity, location, relationships, user behaviors inter views, blogs, news repor ts user studies, field obser vation notes, photos, film/video
22
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Table 5. Summary of Selected ROI Methods and Models
Methods/Models Method Descriptors Time Orientation Time Frame Institutional Focus Gover nment Level Project Orientation Value Measuring Methodology, US Federal CIO Council prospective cross sectional public national/federal Social Retur n on Investment, REDF (US) retrospective longitudinal private n /a Best Value Perfor mance Indicators (ODPM, UK) retrospective mixed public local UK local gover nment programs predominantly quantitative +qualitative Gar tner Gover nment Perfor mance Framework retrospective cross sectional* public general general perfor mance orientation predominantly quantitative +qualitative highy quantitative, employing indexes mixed inter nal & exter nal
IT mixed, predominantly quantitative general categories + examples & suggestions
social ser vices
Variable Types
predominantly quantitative
Variable Specification
highly specific
highly specific
PROI Orientation
primarily inter nal, some public
mixed inter nal/exter nal with exter nal emphasis broad method fully specified par tial suppor t not included suppor t materials examples provided
mixed; many inter nal and exter nal variables large & diverse limited to variable definition not included not included none none
PROI Variable Set Analytical Methods Analytical Process Suppor t Risk Analysis Decision Suppor t Presentation Tools
nar row, limited some method suppor t modest suppor t included decision tool included minor suggestions
limited
method specified
unknown limited inclusion suppor t materials results matrices
*Includes some longitudinal variables
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
23
Table 5. Summary of Selected ROI Methods and Models (continued)
Methods/Models Perfor mance Reference Model (US Federal Enter prise Architecture) retrospective mixed public national-US program perfor mance mixed predominantly variable categories mixed ver y large & diverse not included not included n/a material included par tial
Method Descriptors
Demand and Value Assessment Model (Australia)
Public Ser vice Value Method, Accenture
MAREVA (Gover nment of France)
Time Orientation
prospective cross sectional public national/general general mixed, largely quantitative mixed categories and some specified mixed inter nal and exter nal; few political large, diverse highly specified & detailed highly specified & detailed included suppor t materials and guidelines included
combined longitudinal public general general predominantly quantitative mostly specific perfor mance & cost metrics primarily inter nal, some public nar row, limited analysis method specified included not included suppor t materials standard displays
retrospective cross sectional* public national-France IT mixed, largely quantitative
Time Frame Institutional Focus Gover nment Level Project Orientation
Variable Types
Variable Specification
largely specified
PROI Orientation
mixed, largely inter nal limited included included included material included par tial
PROI Variable Set Analytical Methods Analytical Process Suppor t Risk Analysis
Decision Suppor t Presentation Tools
*Includes some longitudinal variables
24
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
and the level of analytical detail and technique included. Some are intended for use prospectively, in planning for and justifying an investment. Others are aimed primarily at showing impacts of investments after the fact. They also vary in terms of the degree to which they deal with both internal and public value results of the investment, and whether they are designed specifically for IT or government investments generally. The summary of these methods or models in Table 5 can help in the selection of variables and analysis to fit the IT project. The SROI (Social Return on Investment) model, the only private sector oriented one in the summary, is included for its special features. Its private orientation refers not to the commercial sector but to a private philanthropic orientation. The method was developed by a San Francisco foundation to assess its social and economic development programs. It illustrates not so much how to assess an IT investment but rather how to deal with personal and community impacts in a systematic and comprehensive way.
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
25
H. What Can Interfere With or Prevent Public Returns? The Risk Analysis.
Government IT innovation is risky business. Evidence from many surveys and cases suggests that the threats and other challenges of IT projects often overwhelm the capabilities of the developers and implementers. Data from the private sector tells a similar story. For this paper, we focus our attention not on the full range of risks to government IT project success, but concentrate on the public return aspects of that problem. The distinction between the public return and more generic aspects of risk analysis is not, however, a simple one. The approach is the same: identify and evaluate threats, develop and evaluate response methods, and produce a summary analysis and
mitigation strategy. To identify and evaluate threats, we return to the overall value proposition schematic presented earlier (Figure 12). In this figure, we identified two kinds of risk: 1) development risk and 2) benefit risk. Development risk, simply put, is risk that the development and implementation of the IT will fail outright or will not perform as designed and intended. Benefit risk applies to whether the IT investment will fail to produce the envisioned benefits in spite of being successfully developed and implemented. For the public value framework we focus on benefit risk. Of course any threat to the development of the IT is an indirect threat to public returns. However, several of the ROI methods described below include adequate analysis methods for development risk. These methods do not deal adequately, however, with the additional threats and issues in benefit risk.
Figure 12. Public Return on Investment Value Chain
Secondary Performance Gain
Public Cost Internal Agency Returns
*
IT Investment
*
Business Goal or Problem Private Cost (?) Vendor/ Industry Benefit
* *
Public-atLarge Benefit
*
*
Direct Citizen Benefit
* * *
Policy/ Electoral Benefit
*
*
*
General Economic Development
* *
Development Risk Benefit Risk Direct Benefit Flow Indirect Benefit Flow Direct Benefit Accumulation Indirect Benefit Accumulation
26
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Table 6. Examples of Threats to Public Value
Sources of Threats Threat Type Theory Failure overestimate of public demand leads to non-use of ser vice; flawed understanding of impact mechanisms; misunderstanding of stakeholder interests Exogenous Factors confounding or cancelling ef fects: down-tur n in economy cancels financial advantages; changes in gover nment counter stewardship outcomes; political scandal cancels positive media repor ts indirect and second-order ef fects undetectable; data inaccessible due to privacy or sensitivity
To creation of retur ns
To detection of retur ns
lack of or er roneous baseline data; mis-specification of outcome variables; inappropriate analysis methods or models; accounting systems based on inappropriate data definitions and models
The benefit risks associated with the creation and assessment of public returns come from threats to the creation of the returns and to their detection. And there appear to be two main sources of those threats: one is what we will call “theory failure” and the other is “exogenous factors.” In theory failure, the underlying assumptions or theory on which the project is based are flawed or simply wrong. One such theory failure caused the US Department of Education to abandon a multi-million dollar pilot project for online college student financial aid administration. Developed without significant par ticipation from college financial aid officers, the system did not attract supporters and generated much stakeholder resistance. 14 It is important to use both sources and types of benefits threats in a full risk analysis, as suggested by the two-by-two array in Table 6. The risk analysis process can then use the threats identified in this way to estimate the potential loss or cost that each threat represents and the likelihood of the threat materializing. These estimates should be based to the extent possible on input from stakeholders, analysts, users, developers,
managers, and policy makers. This will provide a basis for accurate estimates and concentrating attention on the threats with the highest combined loss-likelihood estimates. In some cases, the value, cost, and loss-likelihood estimates can be quantified to yield decision tools for moving ahead with an investment. The displays in Figure 13 and Figure 14 on the next page, show such a result from the US Federal CIO Council’s Value Measurement Method, which employs voting and other methods to prioritize and estimate quantitative values for cost and their related risks. The decision boundaries shown in the figures come from management decisions or some other deliberative process.15 The threats due to difficulty in detecting indirect or second-order effects deserve some added discussion. Public value outcomes can extend beyond those involved in the direct interaction with government. For example, drivers are the direct beneficiaries of an online license renewal system. However, the scope of possible indirect value outcomes and beneficiaries can be very broad. Residents in the neighborhoods of the license renewal agencies will benefit
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
27
Figure 13. Example of a Value-Risk Boundary Analysis
Figure 14. Example of a Cost-Risk Boundary Analysis
28
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
to some degree from reduced traffic congestion and pollution. Shopkeepers in that neighborhood may lose business for the same reasons. Second order effects may be even more diffused and difficult to detect. Learning to trust the online process for license renewal, for example, can result in more use and greater benefits from using other online services. Since risks are tied both to stakeholders and possible value outcome variables, pursuing this line of reasoning can lead to a very large, and likely infeasible, list of risk analysis factors and tasks. To work within resource constraints will require limiting the risk analysis to the most important value variables and stakeholders. Setting priorities for this kind of analysis will therefore be essential, and must be based on the goals of the project and stakeholder value estimates.
one step can easily lead to revisions in conclusions or data collected at preceding steps. This is par ticularly so for the risk analysis steps in the process. It is not feasible to complete the risk analysis for achieving any particular public value return until the details about that return and how to assess it are known. Loops can occur in the process when a step produces unexpected or challenging results. For example, identifying all related business processes for a large, complex project may produce too many results to carry forward in the analysis. That would require looping back to reduce the scope. So the potential user should view this process summary as a rough guide rather than a lock-step procedure. The process also represents a comprehensive approach to public value assessment. We argue that all the steps are needed to ensure that the full range of public value returns will be included in any assessment and report. Some small or very narrow projects, of course, may have limited public value potential and not require such an extensive review. But the research and analysis leading to this framework have clearly demonstrated that the scope of public value possible from government IT investments can far exceed what an internally focused, financial assessment will reveal. Unless assessments use a much broader view, as represented in this framework and other models cited above, the full value of government IT investments will not come to light. It also seems likely that this same kind of analytical framework would be useful for other forms of government investments. We did not, however, explore that possibility for this report.
I. Overview of Using the Framework.
The framework is a combination of individual analysis steps combined into an overall process or plan for public ROI assessment. A summary of that overall process is shown in Figure 15 on pages 30-31. As shown there, the process follows the arrows, starting with a clear understanding of the larger context of public value generation. The investment goals that emerge from the investment plans and understanding of the context link to the business processes and public value impact mechanisms. Analysis of the impact mechanisms and public return types leads to specific impacts listed in the stakeholder interest analysis. Identifying the interests and impacts will then guide choosing public value variables, analysis methods, risk analysis, and ultimately, reporting tools and techniques. For simplicity, this process diagram is somewhat idealized. It shows only the basic linear path through the various steps. In practice, the actual path is rather more likely to have backtracking and loops. Learning at
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
29
Figure 15. The Public Value Framework Overview
Public Cost Internal Agency Returns
*
IT Investment
Secondary Performance Gain
*
Business Goal or Problem Private Cost (?) Vendor/ Industry Benefit
* *
Public-atLarge Benefit
*
*
Direct Citizen Benefit
* * *
Policy/ Electoral Benefit
*
*
*
General Economic Development
* *
Development Risk Benefit Risk Direct Benefit Flow Indirect Benefit Flow Direct Benefit Accumulation Indirect Benefit Accumulation
Business Reference Model
Value Impact Types
Business Component Law enforcement Financial management Supply chain management Planning & resource a l l o c a ti o n D i r e c t s e r v i c e t o c l i e n ts Financial management
Business Process conduct auction budget for procurement qualify vendors for auction deter mine communication needs conduct auction pay vendors
Public Value Mechanism citizen - type 3 citizen - type 3 citizen - type 3 vendor- type 1 citizen - type 3 gov't staf f - type 1 vendor - type 1 vendor - type 1
Public Return Types F = Financial P = Political So = Social I = Ideological St = Strategic Sw = Stewardship
(Continued on next page)
30
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Figure 15. The Public Value Framework Overview (continued)
Identify Risk Mitigation Strategies Results & Reporting Value Analysis Variable and Analysis Selection Risk Analysis
Stakeholder Impact Variables Value Type Financial Political Citizens at Large local economic development Vendors increased revenue Elected Officials
suppor t from vendors positive media coverage enhanced reputation in agencies receiving fur niture increase level of market competition increase in citizen access to infor mation about procurement: greater transparency enhanced reputation for promoting market competition
Social Ideological
Strategic
increased market share increased knowledge about competition
enhanced reputation being moder n & progressive
Stewardship
increased trust in gover nment
reputation for frugality & good management
Stakeholders Business Process Budget for procurement Deter mine needs Qualify vendors Conduct auction Pay vendors Value Generator intrinsic intrinsic, ef fectiveness ef ficiency, intrinsic ef ficiency, intrinsic, ef fectiveness ef ficiency, intrinsic Impact Type 3 3 1,3 1,3 1,3 Citizens at Large stewardship ideology, stewardship ideology, stewardship ideology, stewardship stewardship political, strategic political, strategic political, strategic, stewardship political, stewardship Vendors Elected Officials
(Continued from previous page)
financial, strategic financial, strategic financial, strategic
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
31
J. How to Summarize and Present Results?
This framework for public value assessment presents both problems and opportunities for summarizing and repor ting. The problems arise from the large number and types of results that the assessment can produce. For presentation to policy makers and non-technical audiences, the results should be as simple and accessible as possible. Simple char ts and summary tables are best for this purpose. For multiple stakeholders and value variables, the number and complexity of charts may become a problem. This section discusses some of the specific issues and alternative methods available. For qualitative variables, the presentation of results can show the presence of a value result, and information about magnitude and direction, if relevant and available. Using the information in Table 3 (page 21), a summary display similar to Table 7 below can present the types and direction of results, with the estimated relative
magnitudes as well. The down arrow in the stewardship row for vendors indicates a potential loss to vendors due to transparency increases that diminish opportunities for some forms of influencing procurement. Whether this is a positive or negative result overall may not be clear. For the more quantitative results, where the public value variables lend themselves to calculation and statistical analysis, many presentation and summary methods are available. Clear and simple visuals are generally preferable to tables of quantitative data for non-technical audiences. For cross sectional data, a column char t, such as in Figure 16, can be used to present the same comparative public value data converted to a ten-point index or scale. Such a multidimensional chart can be difficult to interpret, however, and some ROI methods use the so-called radar char t for the same data, shown in Figure 17. This kind of char t provides a clearer image of the pattern of results for each stakeholder type and value dimension. However, this kind of display becomes much more difficult to
Table 7. Qualitative Display of Stakeholder Impacts
Stakeholder Impact Variables Value Type Financial Political Social Ideological Strategic Stewardship Citizens at Large Vendors Elected Officials
32
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Figure 16. Stakeholder Impact Column Chart
Public Value Index by Stakeholder Groups
10 9 Public Value Index 8 7 6 5 4 3 2 1 0 citizens at large vendors elected officials Financial Political Social Ideological Strategic Stewardship
Figure 17. Stakeholder Impact Radar Chart
Financial 10
Stew ardship
5
Political
0
Strategic
Social
Ideological Citizens at Large Vendors Elected Officials
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
33
Figure 18. Portfolio Risk and Value Comparisons - DVAM
interpret if the number of axes or stakeholders is large. The use of indexes can also be used to compare the relative risk and value of alternative investments on a value scale if the variables can be combined into a single index. The Demand and Value Assessment Model (Australia) provides guidance on how to produce such a public value index and use it for comparison purposes. An example of that kind of result is shown in Figure 18 above. For cross sectional assessments, more elaborate presentations are available, par ticularly if the value variables are indexed or based on quantitative data. The Accenture Public Service Value Model (PSV) can use historical data about government program effectiveness and costs to show changes in performance over time. 16 The model is based on the principle that public value is created when both outcome results and cost effectiveness increases. An example of this form of results presentation
is shown in Figure 19 for the Arizona Depar tment of Revenue. Overall, the performance shown is improving from 1999 to 2001 and 2002 to 2003 when the organization is creating increased outcomes and doing so more cost effectively. Documentation of results through this kind of chart or visual device should include background material about methods and measurement issues. This framework advocates the use of a wide range of data and analytical styles, many of which are considered controversial or suspect in some environments. Decision makers and analysts often have deep-seated biases about the validity of qualitative data or social statistics, or other non-traditional material for ROI analysis. Those performing a public value assessment must, therefore, be thorough in providing rationale and supporting material for all results. They must be attentive to the issues of interpretation and validity that may affect how key members of their audience respond to the assessment results.
34
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Financial and Social Worth
Figure 19. Historical Performance Change Model (PSV – Accenture)
PSV Agency Performance Matrix (FY 1996 – FY 2003)
Higher
10% 8% 6% 4%
1997
Outcomes
1996
2%
2001 2003 2002 5%
10% 15% 20%
-20%
-15%
-10%
1998 -5% 2000
AVERAGE 0% SCORE 0% -2%
-4% -6% -8% -10%
1999
Lower
Lower
Cost Effectiveness
Higher
The principles for SROI analysis used by the REDF organization provide valuable guidance for the conduct of the kind of assessment described here. These principles can be applied to traditional ROI analysis as well, but seem particularly well suited to the public value issues involved with this framework and related methods. The principles make an appropriate bridge from the general ideas and methods presented here to the difficult work of carrying out public value and ROI assessment in practice. SROI Design Principles: 17
4.
Replicable – The framework should result in similar conclusions when applied by different practitioners who use similar parameters (such as the scope and options). Thus, results should also be comparable over time and among organizations, at least among analyses that use similar options and where the options are clearly noted. Transparent – The process by which the analysis was prepared, and the context in which results should be seen, should be transparent. Credible – The results should be credible to investors, purchasers, mangers, and other users. Integrative – The framework should relate to, and where possible integrate with, other approaches to understand social value. Avoids misuse – Proper application of the framework should reduce the risk of misuse of, or misleading, SROI numbers or analyses.
5.
6. 1. Feasible – A basic SROI Analysis should be something any organization can afford to prepare itself. 7. 2. Accessible – The process should be understandable and relevant to organizations at various stages of development. 8. 3. Rigorous – The method should be substantive and well-executed, and based upon premises that are validated by informed practitioners.
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
35
9.
Open source – The framework should continuously be informed and improved by the collective wisdom of practitioners in an inclusive, iterative process.
10. Useful – Applying the framework should result in information that enables users to make decisions or take actions that further their goals.
36
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
IV. Reflections on the Framework and the Value of Public Value
T
his framework grows from the rather simple principle: that the value of a government’s investment in IT should be assessed from the point of view of the public it serves. That principle leads us to identify two distinct but equally important types of public value: delivering benefits directly to citizens and enhancing the value of government itself as a public asset. From this seemingly simple beginning grows the rather high level of complexity involved in working through this framework and its potential application. That complexity derives in part from the way these ideas radically expand the possible scope of inquiry needed to identify and document public value creation. That expanded scope brings with it a host of measurement problems that emerge when the many social and political outcomes come into play. The cost of this expanded and more complex assessment can be quite high. So government executives and IT planners can rightly question whether assessing public value is wor th the effort. Part of the answer may be that they will have no choice. As we noted earlier, elected officials have begun insisting on more comprehensive cost and return analyses for IT investment proposals. Conventional approaches to ROI analysis may simply be inadequate for these increased demands. More importantly, however, the desire for a more comprehensive and robust justification for new IT investments reflects their greater complexity and ambition. The low-hanging fruit available from earlier IT investments, such as establishing Web presence and automating simple service transactions, has typically been harvested. More substantial improvements in government are now possible by exploiting the integrative and transformative potential of IT, but they require much larger investments. These projects require a way
of assessing public value that matches their greater scope and complexity, a way that can build the needed public support and guide development. In this light, the cost of using a framework such as this should be easily justified. Moreover, the new knowledge about public value possible from such an assessment can help guide other forms of investment and contribute to long-term government improvement.
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
37
V. Appendix: The Research Basis for the Public Value Framework
A. Consultative Workshop
On July 13, 2005, the Center for Technology in Government, in collaboration with SAP , hosted a consultative workshop on assessing public return on government investments in IT. A select international group of 22 experts from government, academia, and the private sector were invited to meet with researchers from CTG to discuss the core issues and themes that make assessing public ROI for IT investments such a complex and difficult problem. The workshop results were used to inform CTG’s next steps for the project. The workshop results were used to help develop a preliminary framework for conducting ROI analysis in the government sector and design the case studies research approach. This approach was employed to identify and select the five case studies, which were used to refine the preliminary framework and develop the white paper. The lists below summarize the key issues that emerged from the workshop that characterize the complexity of the issue. Workshop par ticipants provided their recommendations in the form of key themes for CTG to explore in the follow-on research. what should be done, but not on what has already been done. Governments have trouble harvesting savings, which often get moved around the budget. There is no straightforward quantitative bottom line value measure for ROI in public sector. Government is multidimensional. Nonlinear, complex interactions among benefits—hard to measure results and link to specific programs or technologies, i.e., proving the causal relationship between the two. ROI requires advanced project management and portfolio management skills that are often lacking. Comprehensive ROI analysis can greatly increase transparency of government decisions and investment results. This level of transparency is a “risk” in and of itself, increasing the probability for embarrassment and criticism leading to loss of support. ROI is done in a vacuum. Not focused on the strategic investment. Lack of longer-term tracking and assessment makes it difficult to build a measurement model. It is hard to evaluate IT ROI elsewhere in the government enterprise because the outcome frameworks (intersectors) aren’t established.
1. Key Issues Characterizing the Complexity of Assessing Public ROI for Government IT Investments
Lack of incentives to assess public ROI. There may be no consequences for absence of ROI or other demonstration of results. Lack of historical perspective and data. Government tends to be prospective (not retrospective), so it tends to focus on
38
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
2. Themes to Explore in the Research
Value and impact measurements should take into consideration the cost impacts on other business processes, by elimination or changes in the way we work together. Look for efficiency and cost-reduction in non-IT areas. IT is an enabler, and the purpose of IT is to enable other things, including changing the culture of organization, strategies, etc. Thus, it is important to see how IT is enabling returns in other areas. Assessment should include attention to how particular IT investment enables IT elsewhere. ROI analysis must get beyond evaluating things on a more traditional basis to include questions of outcomes for broader range of beneficiaries or stakeholders. Social cost-benefit analysis and political considerations involve different people making different decisions. It is important to understand how this may impact the IT aspect of ROI calculations. Need more attention to risks as well as benefits. Assessment should include attention to par ticular government sensitivity to risk factors. Method development should include system analysis that engages a broad scope of operations. Analysis should include questions about leadership, feasibility, political support as well as results/outcomes, how constituents and opponents will react. Focus assessment on question of getting ROI on the programs, not the IT; IT doesn’t deliver the outcomes.
Por tfolio management is a big theme in current discussion of IT management. As applied to public ROI, implies looking at the overall picture, not just a slice of the project. ROI methods must accommodate or provide for shared outcomes that cross over existing stovepipes. While benefits are measured in terms of traditional (financial), political and social factors, risks are measured based on other factors: (1) technical, (2) organizational, (3) time, and (4) political. Find relationships between inputs and outcomes. Once indicators are developed cause and effect relationships can be explored using econometric methods. Value of IT investment may be enabling change. Find ways to ensure follow-through in delivery and assessment of government programs. Explore what are the attributes of a good public ROI model: measurable in different ways, creates expectations of performance and assessment at both IT and program levels, is integrated with the budget process, and is included in a yearly review.
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
39
B. Case Study Summaries18
1. The Government of Israel’s Merkava Project
Some government IT initiatives focus narrowly on a specific technical problem, like enabling mobile data communication or Web-based transactions. Others are driven by a broader, more ambitious goal, to use IT as a tool to transform government. The Merkava Project in the Government of Israel (GOI) is most cer tainly one of these transformative effor ts. It is transformative in its own right, as an effort to restructure the financial, logistics, and human resource components of governmentwide administration into an integrated ERP system. Merkava is also part of a comprehensive e-government initiative that includes the ERP system as one of five layers of new technologies and operational systems for enhanced internal operations and improved benefits and services to citizens. These layers are part of a multiyear strategic plan, described in 2002 by Prime Minister Sharon: “The e-government project is an expression of the government goal to provide state-of-the-art strategic systems, while affecting social and economic national targets. We believe that this infrastructure constitutes a better tomorrow and that tomorrow is already here.” From its beginnings in 1999 to the present, the Merkava ERP project has been a very large undertaking. It is described by the Ministry of Finance as the largest single IT project in Israeli history. It has been implemented or is in the roll-out process in 30 government units (out of 100), and currently supports approximately 2,300 users. The 2005-2007 IT master plan calls for the Merkava system to be implemented in 90% of the government’s offices during the planning period. Work on the other four layers in the overall eGov initiative has been
progressing in parallel with the Merkava implementation. The integration provided by the Merkava ERP provides an important par t of the infrastructure for operations in the three upper layers of the initiative and contributes to the returns that they deliver as well. The success of implementation effor ts to date and the growing record of returns, both internal to the government and the public, suggest that the remaining roll-out will continue as planned and the accumulation of returns will continue to grow.
2. The Austrian Federal Budgeting and Bookkeeping System
How a government obtains value from its citizens and external public environment and how it returns value to them are linked directly to its financial management systems. These systems are crucial links in the flows of revenues to the government and the flow of expenditures and services back to the public. Improving financial management, therefore, has the potential to produce significant returns in terms of both greater internal efficiencies and enhanced public returns. These were the goals of the Austrian Federal Budgeting and Bookkeeping System project initiated in 1997 by the Minister of Finance and supported by the Chancellor (Prime Minister). The goal of the project was to redesign and integrate the electronic workflow of the federal government’s budget and bookkeeping processes. The strategy they chose was to implement a single ERP software standard throughout the federal government, along with the adoption of necessary legal authority. The result was that, by 2005, the Ministry of Finance successfully consolidated 85 bookkeeping units across the federal government into one federally owned, but privately operated, agency. The consolidation and integration produced immediate and tangible benefits in terms of
40
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
internal efficiencies. These resulted from the implementation of a new standardized work process for accounting and budgeting throughout the federal government, with reduced work process steps and processing time. As of 2005, the legal consolidation of the numerous bookkeeping depar tments into one agency, along with the technical and organizational implementation of the ERP , has resulted in annual savings of approximately €30 million. In terms of these returns to the government itself, the project is clearly a resounding success. In terms of broader public returns, the project goals went beyond internal financial management efficiencies. The aim of restructuring the Austrian Federal Accounting and Bookkeeping system was much more than simply an “IT investment” from the very beginning. The bookkeeping system was part of a larger effort to implement the SAP ERP technology throughout the Austrian Federal Government as part of a governmentwide public management reform effort. The ERP technology was one part of a comprehensive strategy that included legislative reform, governmentwide reorganization and consolidation, and implementing a modern accounting and budgeting standard across the government. Though the specific ERP technology was just one element of a larger approach, involving legal and organizational strategies, the ERP technology played a very important enabling role vis-à-vis the legal and organizational strategies.
record keeping in a democratic society. That is also the foundation of the public value proposition guiding the Digital Archives program: that the state has the constitutional and statutory mandate to preserve and provide access to records of enduring legal and historical significance. The growth of electronic records in government agencies in the 1990’s presented a challenge to the State Archives’ ability to fulfill its mission, since it lacked an effective program and technology to deal with records in these new formats. The Washington State Digital Archives (WSDA), a program within the Office of Secretary of State, is the response to that challenge. It was initiated by the Office of the Secretary of State, with initial planning begun by the then State Archivist in March of 2000. The initiative was taken up in 2001 as a priority by the newly elected Secretary of State Sam Reed, and included in the Secretary of State’s 2001-2007 Strategic Plan. The project was subsequently supported by the state legislature and included in the State of Washington’s 2001-2003 Capital Budget. Construction of the physical hub of the WSDA in Cheney, Washington, began in January 2003. Beginning in mid-2001, the WSDA team began exploring a wide range of technologies and techniques for collection, access, and preservation. The results led to the custom development of a Web interface and database design that blended the latest technologies with traditional archival theory to create a first-of-its-kind digital records repository for state government. The goal of the program was to make the historical electronic records of Washington’s state and local governments easily accessible, from anywhere, at anytime. The initial vision and value proposition were carried through a complex political and technical process to a functioning digital archiving program and facility delivering the promised public value. The WSDA project team began with a clear vision of the expected value of the Digital Archives to both the government and citizens. In addition, it successfully identified
3. The Washington State Digital Archives
Washington State’s investment in digital archiving for government records provides a highly focused and successful example of pursuing public value through information technology. The job of collecting, preserving, and providing access to the records of government is central to the mission of Washington’s Office of Secretary of State. That mission recognizes the fundamental importance of government
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
41
the benefits that it needed to communicate to the state and local government agencies that were keepers of public recorders in order to mobilize their support and par ticipation. The WSDA project demonstrates a strong connection between the initial high-level public value proposition that motivated the project and its realization in the performance of WSDA itself.
recently taken steps to move in this direction by implementing the necessary institutional structures and policies to take fuller advantage of this enterprise-level asset.
5. Service New Brunswick
Service New Brunswick (SNB) is well-known internationally for its expertise in providing multi-channel “single window” citizen access to government services as well as for developing and maintaining geographic information databases. SNB’s awardwinning approach provides one-stopshopping for different government services on behalf of provincial and municipal government agencies, but also provides a linkage to the Canadian Federal Government in a “joined up” government model. SNB was launched in a time of high pressure from citizens for improved service delivery; today it serves the province through its award winning service deliver y model, and also and maybe more importantly in the long run, through its innovations in economic development. The numbers behind SNB speak for themselves; customer satisfaction numbers are the highest in Canada going from about 50% in late 1980’s to 92% in 2005. These numbers require that responsible examinations of public value include SNB. But there is more to this story than customer satisfaction ratings. The essential par t of the public value story in this case is Service New Brunswick (SNB) as a quasigovernmental organization and its evolution as an integrated service provider and economic development innovation. SNB is a public corporation with a single shareholder – the government of New Brunswick. This shareholder has guided SNB with this model throughout its existence: make public service good business.
4. The Commonwealth of Pennsylvania’s Integrated Enterprise System
From the beginning, the Commonwealth of Pennsylvania’s Integrated Enterprise System was much more than a technology project. The ERP implementation continued through three gubernatorial administrations with consistent top level executive support; eventually putting in place the technical infrastructure and enterprise standards for core administrative functions with improved public value. Between early 2001 and mid-2004, the ERP implementation for the five business functions was completed for 53 Commonwealth agencies including all 49 of the agencies under the governor’s jurisdiction. Immediate returns in the form of improved government operations were realized soon after implementation and continue today. However, this infrastructure also provides the Commonwealth with capability that can be fur ther leveraged to support additional improvements in government operations that go well beyond direct improvements in core administrative functions. The Commonwealth has begun such efforts. The IES infrastructure provides public returns in the form of direct improvements in the efficiency and effectiveness of core administrative functions. This infrastructure also provides the basis for improvements in the back office operations of other service areas which, in turn, offer improved services to the public. The Commonwealth has
42
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Endnotes
Expenditure data from: DG Information Society, European Commission; http://ec.europa.eu/idabc/en/ document/4336/5860; China: CCID Advisory (2005, May) http://www.ccidadvisory.com/ report_details.php?r_serial=229; India: Public Sector Technology and Management (2004, Jul 12) http:// www.pstm.net/ar ticle/index.php?articleid=236; U.S. Office of Management and Budget Web site: http:// www.whitehouse.gov/omb/egov/g-9-budget_highglights.html; Datamonitor. (2005, Jun). IT opportunities in state and local government: Five-year outlook for vendors. New York: Datamonitor.
1
http://www.datamonitor.com/~3d016a3bb9db4c6cb608be617baf36f5~/Products/Free/Report/DMTC1150/ 010DMTC1150.pdf; also http://zdnetasia.com/news/business/0,39044229,39371070,00.htm http://www.publictechnology.net/modules.php?op=modload&name=News&file=ar ticle&sid=1920
2
Matthew Weigelt, “Senate committee questions e-gov ROI.” FCW (July 17, 2006), http://www.fcw.com/ ar ticle95291-07-17-06-Web&newsletter%3Dyes View and download copies of the white paper and case study reports at www.ctg.albany.edu/projects/proi.
3 4
This terminology is consistent with recent work on assessing government IT investment. See for example: Center for Digital Government, Prove IT: The Disciplines of Harvesting Value from Public Sector Information Technology , 2006. (www.centerdigitalgovernment.com); and eGovernment Economics Project (eGEP): Measurement Framework Final Version . Prepared for the eGovernment Unit, DG Information Society and Media, European Commission, by Christiano Codagnone (RSO SPA) and Paulo Boccardelli (Luiss Management), May, 2006.
5 The Toxic Release Inventory is a publicly available data base maintained by the US Environmental Protection Agency that contains information on toxic chemical releases and other waste management activities reported annually by certain covered industry groups as well as federal facilities (http:// www.epa.gov/tri/).
These are taken from the Business Reference Model document, which contains considerable detail on the definitions and details of these components (http://www.whitehouse.gov/omb/egov/documents/CRM.PDF).
6
See the European Interoperability Framework for Pan-European eGovernment Services, http:// europa.eu.int/scadplus/leg/en/lvb/l24226b.htm; The UK eGovernment Interoperability Framework, http:// www.govtalk.gov.uk/documents/eGIF%20v6_1(1).pdf; Malaysian Government Interoperability Framework (MyGIF), http://unpan1.un.org/intradoc/groups/public/documents/APCITY/UNPAN016393.pdf;
7
Hong Kong Special Administrative Region (HKSARG) Interoperability Framework, http:// www.bilgitoplumu.gov.tr/kdep/34/hongkongif_v1_0.pdf.
8 The NASCIO Enterprise Architecture materials are listed at http://www.nascio.org/publications/index.cfm. The Athena Project repor ts are available at http://www.athena-ip.org. 9 10
The Zachman Institute for Framework Advancement. http://www.zifa.com/
See, for example, polling results at Zogby International (http://www.zogby.com/index.cfm), or The Pew Research Center for People and the Press (http://people-press.org/). Additional information and suggestions for conducting stakeholder analyses can be found in these sources:
11
http://www.ittoolkit.com/workbooks/qt_stake.pdf; http://www.stsc.hill.af.mil/crosstalk/2000/12/smith.html; http://www.boxesandarrows.com/view/understanding_organizational_stakeholders_for_design_success; http://www.ctg.albany.edu/publications/guides/smar tit2?chapter=11§ion=5; http://www.ctg.albany.edu/ publications/guides/and_justice_for_all?chapter=8§ion=2 Overseas Development Administration, Guidance Note on How to Do Stakeholder Analysis of Aid Projects and Programmes , Social Development Department, July 1995.
12
More information on these data types and analysis methods can be found in social science research method texts. See, for example, Earl Babbie, The Practice of Social Research, Tenth Ed. Belmont, CA: Wadswor th, 2004.
13 14
Stephen Burd. “Education Department Pulls the Plug on Novel Way to Deliver Financial Aid. Chronicle of Higher Education . Vol 46, No. 29 (March 24, 2000). See US Federal CIO Council Best Practices Committee. VMM How-To-Guide . http://www.cio.gov/documents/ValueMeasuring_Methodology_HowToGuide_Oct_2002.pdf
15 16
Accenture Public Service Value (PSV) Model, http://www.accenture.com/Global/Services/By_Industry/ Government/PublicGovernment.htm
17 18
A Framework for Approaches to SROI Analysis. http://www.redf.org/ View and download copies of the white paper and case study reports at www.ctg.albany.edu/projects/proi.
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
43
Center for Technology in Government 187 Wolf Road, Suite 301 Albany, NY 12205 Phone: (518) 442-3892 Fax: (518) 442-3886 E-mail: [email protected] www.ctg.albany.edu
doc_957140032.pdf
The study of how an investment is likely to perform and how suitable it is for a given investor. Investment analysis is key to any sound portfolio-management strategy.
Advancing Return on Investment Analysis for Government IT
A Public Value Framework
Anthony M. Cresswell G. Brian Burke Theresa A. Pardo
Center for Technology in Government University at Albany, SUNY 187 Wolf Road, Suite 301 Albany, NY 12205 Phone: (518) 442-3892 Fax: (518) 442-3886 E-mail: [email protected] www.ctg.albany.edu
September 2006
This material is based upon work supported by SAP . Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of SAP . ©2006 Center for Technology in Government The Center grants permission to reprint this document provided this cover page is included.
THIS PAGE INTENTIONALLY LEFT BLANK
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Acknowledgments
his paper and the project it is based on—”Public ROI: Advancing Return on Investment Analysis for Government IT”—was made possible with funding and guidance from SAP ,a provider of enterprise software for the public sector. In particular, we would like to thank Russ LeFevre and Peg Kates, who have been partners from the very beginning of this work, and special thanks to Tom Shirk, Ingo Hoffman, Rod Massey, Ian Swann, Mor Sagmon, Carsten Friedland, Alex Bratzler, and Bonnie Rothenstein. Several members of the Center for Technology in Government staff made many important contributions to this white paper and the overall project: Sharon Dawes, Jane KrummSchwan, Alison Heaphy, Paula Hauser, and Lucy Dadayan. Finally, this white paper and the case study reports depended on the enthusiastic participation of many government officials from the five case study sites. In particular, we would like to thank Christian Ihle from the Ministry of Finance in Austria, Adam Jansen from the Washington State Digital Archives, Ronny Jacobowitz and Yitshak Cohen from the Merkava Project in Israel, Andy McIntyre from the Integrated Enterprise System in the Commonwealth of Pennsylvania, and Darrell Fowler from Service New Brunswick in Canada. We thank them for their generous hospitality and commitment in helping us conduct our site visits and creating this resource for improving government IT investments.
T
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
THIS PAGE INTENTIONALLY LEFT BLANK
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Table of Contents
Executive Summary Introduction Section I Section II Section III 1 4 5 Why Assess Public ROI for Government IT 6 Research Summary of Public Value Case Studies 8 A Public Value Framework for Government IT Assessment 10 A. The Framework Strategy. 11 B. How Does Government IT Investment Link to the Public? 13 C. What Kinds of Impacts Matter for Public Value. 14 D. What is the Investment? Linking IT to Goals and Business Processes. 16 E. What Kinds of Public Value are Produced? 19 F. Who Receives Value? The Stakeholder Analysis. 20 G. How to Demonstrate the Value? Identifying Variables and Methods. 26 H. What Can Interfere With or Prevent Public Returns? The Risk Analysis. 29 I. Overview of Using the Framework. 32 J. How to Summarize and Present Results? Section IV Appendix 37 Reflections on the Framework and the Value of Public Value 38 The Research Basis for the New Framework 38 A. Consultative Workshop 40 B. Case Study Summaries E ndnotes 43
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
THIS PAGE INTENTIONALLY LEFT BLANK
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Executive Summary
he range and complexity of government information technology (IT) investments makes assessing their returns a daunting prospect. Projects can range from systemwide transformations, to improving financial transparency, to more efficient dog licensing. The returns may be large or small, obvious or obscure, and can run from a few minutes saved in a routine transaction to improving the trust and legitimacy of an entire government. In spite of the difficulty in assessment, however, knowledge about public returns can be vital to fully informed and justified IT investment decisions. Assessing these returns remains a core problem in IT planning and decision making. That problem results from shortcomings in the available methods and models for assessing public returns, what we call public return on investment (ROI). In looking over these existing methods and models we saw three significant shortcomings: Incomplete analysis of public value, resulting in too narrow a scope of what can be considered returns to the public. Lack of systematic attention to how government IT investments generate results of value from the point of view of the public.
T
Weak or absent methods for tailoring a public ROI assessment to the specific context and goals of a government IT investment. Existing methods also deal with risks to government IT investment, but primarily from the point of view of technology development and implementation. The risks that involve the public beneficiaries of the investments merit more attention. This white paper presents a public ROI assessment framework that addresses these shortcomings. We call it a public value framework to emphasize the point of view of the public, not the government, as the basis for the assessment. The framework describes how to identify and assess public value through the kinds of activities shown below in Figure 1. The framework’s strategy is simple in concept, but complex in application: connect what happens in the government (on the left) with the impacts on stakeholders in the public domain (to the right), then report and apply the results. The activities on the left identify the potential value mechanisms and outcome goals. Those are linked to stakeholder interests, impacts, and risks to the right. The curved arrows indicate that in
Figure 1. Public Value Framework
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
1
practice the process would seldom be linear, requiring reflection and backtracking to adjust for learning and new information. The full paper presents a detailed version of this schematic, showing the links among these activities. The public value proposition takes center stage. This value proposition must be broadly conceived to do justice to the scope of government and how it affects individuals, groups, and both public and private organizations. This framework presents a new and more comprehensive way of describing public value, based on six kinds of impacts government IT can have on the interests of public stakeholders: Financial – impacts on current or anticipated income, asset values, liabilities, entitlements, and other aspects of wealth or risks to any of the above. Political – impacts on personal or corporate influence on government actions or policy, role in political affairs, or influence in political parties or prospects for current of future public office. Social – impacts on family or community relationships, social mobility, status, and identity. Strategic – impacts on economic or political advantage or opportunities, goals, resources for innovation or planning.
Ideological – impacts on beliefs, moral or ethical commitments, alignment of government actions or policies or social outcomes with beliefs, or moral or ethical positions. Stewardship – impacts on the public’s view of government officials as faithful stewards or guardians of the value of the government itself in terms of public trust, integrity, and legitimacy. Expanding the view of stakeholder interests in this way brings into focus two distinct but equally impor tant types of public value: the delivery of benefits directly to citizens and enhancing the value of government itself as a public asset. An IT investment that makes government more transparent, more just, and a better steward has added public value, a non-financial but nonetheless important return. This framework describes how to include both in public value assessments. The framework also identifies the basic ways government IT investments link to public value. The simplest link results from an IT investment embedded directly in a service delivery process (Figure 2:A) in a way that enhances its value to citizens or officials. The total value may be a composite of several separate measures: cost savings, quality, enhanced trust. An indirect link (Figure 2:B) can result from infrastructure improvements to business processes. This link may be more difficult to trace and can involve more risk. Enterprise Resource
Figure 2. Direct and Indirect Links to Public Value
Citizen/ Official
V1 V2
government with old IT government with new IT
Citizen/ Official
V1 V2
old government business process new government business process
old IT infrastructure new IT infrastructure
Return = V2 – V1
Return = V2 – V1
A - Direct
B - Indirect
2
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Figure 3. Mix of Direct and Indirect Links to Public Value
environment Citizen/ Official Vi Citizen/ Official Vd government with new IT
Return = Vd + Vi Vi = indirect value Vd = direct value
Planning (ERP) software implementations in our case studies are excellent examples of this kind of investment. Value can also result from a mix of direct and indirect links (Figure 3) when new IT changes the environment. Identifying these links is just part of the story. Each link can involve several valuegenerating mechanisms. The framework describes how to trace these mechanisms working through specific business processes to produce different kinds of value. The value-generating mechanisms are: Increases in efficiency – obtaining increased outputs or goal attainment with the same resources, or obtaining the same outputs or goals with lower resource consumption. In our Austrian and Pennsylvania case studies, for example, new ERP systems helped achieve substantial efficiencies in financial management. Increases in effectiveness – increasing the quality and/or quantity of the desirable thing. Our case study of Service New Brunswick, for example, repor ts how an online registry for land data can contribute to improvements in proper ty tax administration. Enablement – providing means or allowing otherwise infeasible or prohibited desirable activity, or preventing or reducing undesirable events or outcomes. In our Washington
State Digital Archives case study, for example, putting birth and marriage records online enabled research by local historians and genealogists. Intrinsic enhancements – changing the environment or circumstances of a stakeholder in ways that are valued for their own sake. For example, our Israel case showed how enhanced financial accounting and reporting in the Merkava ERP system opened government financial decision making to greater transparency. An IT investment project can deliver public value through any or all of these mechanisms. The framework is deliberately presented at a moderate level of generality to make it most widely useful. Every government IT project will have its own unique goals, value propositions, and stakeholders. So this framework can be used to plan and guide a public value assessment, in combination with measurements, analysis tools, and repor ting techniques chosen for the specific situation. To aid in this regard, the paper presents an overview of more detailed and highly specified assessment methods that can be used in conjunction with this framework. This includes a more detailed flow char t for the assessment process, a summary of several ROI models and methods, and suggestions for analysis and repor ting of results.
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
3
Introduction
very investment decision requires a leap of faith—sometimes a large one— into an uncer tain future. When much of that uncertainty involves the value of the expected returns, risk increases and the investment decision is all the more difficult. This is an even bigger problem for government IT projects, which are notoriously risky and aimed at public value returns that are often very difficult to define and even harder to measure. This white paper takes on one important part of that problem: the question of describing and measuring public value. We chose to focus on public value because it is both the most poorly understood and the most complex par t of the overall problem. We can divide the overall analysis of return on investment (ROI) in government IT expenditure into four parts: costs, internal returns (agency efficiencies, cost avoidance, etc.), public returns, and risks. Of these, the public returns element receives the least amount of attention, though interest in this topic is growing. The key question about public returns, of course, is what do we mean by public value and how can we observe, measure, and document its creation. The goal of this white paper is to help answer that question.
E
The approach we take to answering that question takes a point of view based on our experience, analysis, and the background research for this paper. From this point of view we see two sources of public returns: (1) value to the public that results from improving the government itself, and (2) value that results from delivering specific benefits directly to persons or groups. We also see potential value creation that goes far beyond the traditional financial and service evaluation data. Value creation can come as much from increasing the integrity and transparency of government as from reducing costs through online tax payments. This expanded scope of value includes an often wide range of stakeholders, each with their special interests and expectations from government. This point of view dictates an expansive way of seeking public value. We call this a public value framework, meaning that it is less a specific measurement tool and more a way of identifying and as sessing as wide a range of public value as possible. This is not a small task, and cannot be completed in one such effor t. But this approach can advance the search for a way to effectively measure public value.
4
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Section I: Why Assess Public ROI for Government IT?
he scope of government investment in IT and the associated problems certainly deserve serious attention. Both the levels and growth rates in government IT spending are substantial. The most recent data available shows that the level of this investment in both the industrialized countries and the developing world has grown to a ver y large scale. European government IT spending is expected to increase from US$110 billion in 2005 to US$119 billion by 2007, with US$26 billion in the UK alone, which is about 40% above Germany or France. IT spending by Asia-Pacific region governments, excluding Japan, is expected to reach US$31.7 billion by 2010, from US$22.7 billion in 2006. The Chinese government expenditures of USD$5 billion in 2004 are expected to grow 16% in the following five years. India spent US$943 million on e-government in 2002, and this figure is predicted to increase by 15% a year to US$3.3 billion by 2009. The U.S. federal budget for fiscal year 2007 provides $64 billion in funding for IT investments, approximately a 3% increase from the 2006 enacted level of $62 billion. Total state and local government IT spending was about $55 billion in FY2004 and is expected to grow to $62.4 billion by FY2009. Though what is included in the IT category may vary across these examples, the overall scale of expenditure remains substantial.1 This level of expenditure is receiving increasing scrutiny. The reason for this scrutiny is clear, according to Andrea DiMaio, a Gartner Vice President focusing
T
on the public sector: “If governments do not accurately measure the full value of their IT investments, they risk a serious political backlash. They will be accused of wasting billions of pounds of taxpayers’ money on unnecessary technology.” This sentiment has reached the US Congress. In July of this year the Senate Appropriations Committee recommended no funding for the administration’s 2007 e-government initiatives, reporting that “… the committee has no confidence that the amounts being assessed have any relationship to the benefits anticipated to be returned.” And in the House of Representatives, the current appropriations bill calls for a cost-benefit analysis of all e-government initiatives. According to Mike Hettinger, staff director at the House Government Reform Committee’s Government Management, Finance and Accountability Subcommittee, “the language speaks for itself and reiterates what the subcommittee has been saying for the past year, that in order for this initiative to be successful, we need to have a better understanding of the costs and benefits and clearer guidance for the agencies to follow.” 2 Certainly this enhanced attention to ROI for IT must include a comprehensive and effective way to deal with the public value side of the problem.
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
5
Section II: Research Summary of Public Value Case Studies
his framework is based in part on the results of five case studies that examined how government IT investment projects came to deliver value to the public. 3 The projects were the Integrated Enterprise System in the Commonwealth of Pennsylvania and the Washington State Digital Archives in the US, the Merkava Project in Israel, the Austrian Federal Budgeting and Bookkeeping System in Austria, and Service New Brunswick in Canada. All five case study reports and the white paper are available on CTG’s Web site.2 In the case studies we examined how the IT investments were conceived and developed, with particular attention to the role of public value in the process. We took the view that government IT investment generates public value in two ways: By improving the value of the government itself from the perspective of the citizens, and By delivering specific benefits directly to persons, groups, or the public at large. The first is based on the idea that, assuming a benign government, the better it functions overall, the better off its citizens will be. The government is an asset to the community or nation that delivers a wide range of values. Internal improvements make it a more valuable asset to the public. The second way of generating value has three forms: financial, political, and social. Financial value results from lowering the cost or increasing the efficiency of government or delivering direct financial benefits to the citizens. Political value consists of increasing participation, fairness, transparency, legitimacy, or conferring political capital to elected officials or citizens. Social returns include increased social status, stronger relationships, or opportunities; increased safety, trust in government, and economic advantage.
T
These general understandings of public value guided the data collection and presentation of the case results. The case studies examined public returns to the overall IT investment, including returns generated by a government IT investment and the possible mechanisms to produce them. We did this by considering the links between goals, implemented systems, government performance, and public returns in terms of where they represent value in the chain shown in Figure 4 (next page). The public returns from the investment can flow from the internal improvements in the agency resulting in returns to individual citizens and the public at large (the main flow through the center). Other returns can flow to the political system and the economic environment (below the center), or through effects on other agencies (secondary performance gains). This general view of public returns informed the case studies and helped summarize the results.
6
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Figure 4. Public ROI Value Propositions
Public Cost Internal Agency Returns
*
IT Investment
Secondary Performance Gain
*
Business Goal or Problem Private Cost (?) Vendor/ Industry Benefit
* *
Public-atLarge Benefit
*
*
Direct Citizen Benefit
* * *
Policy/ Electoral Benefit
*
*
*
General Economic Development
* *
Development Risk Benefit Risk Direct Benefit Flow Indirect Benefit Flow Direct Benefit Accumulation Indirect Benefit Accumulation
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
7
Section III: A Public Value Framework for Government IT Assessment
T
his framework is designed to assist government IT executives and analysts in understanding and measuring the value to the public of government IT investments. The main goal is to produce an assessment of public returns that is credible, persuasive, and highly relevant to the investment decisions faced by the government. We use value here, rather than return to emphasize the broad scope of the framework. Most methods for assessing return on investment focus on financial or economic metrics; this approach includes a much broader view of how IT investments can produce results of value to citizens or to the society as a whole.4 This concept of value includes more than the usual financial or economic metrics common in ROI analysis. It is a new and expanded way of understanding the results of government IT expenditures. We call this set of ideas a framework to indicate that it is more than a particular method for public value assessment. It is broad in scope so that it can be applied to virtually any government IT investment, from simple Web sites to governmentwide information systems and architectures. It is broad in scope because this range of investments requires a comparable range of assessment methods. Our framework, therefore is a way of thinking about and organizing the analysis of a family of problems that can encompass many methods. A spreadsheet, for example, is a framework for working on a broad class of problems or analytical tasks. Any par ticular spreadsheet may include specific methods, such as scenario analysis or a net present value calculation. Instead of guiding the assessment process in terms of calculations in a matrix, however, this framework provides an analysis process that starts with a high level view of the IT investment and then drills down through successive steps to identify the specific measures and methods that will reveal and document public value.
In this way, the framework offers both a systematic way of thinking about public value and a way to apply that thinking to par ticular IT investment decisions. The drilling down process is necessary to tailor a specific public value assessment to the nature of a par ticular investment decision. The framework shows how to take into account how public value can change across the many interests of citizens and groups in interacting with governments. In the morning, for example, an executive doing business with the government may think about how IT speeds payments on her government contracts, in the evening while helping with homework she may observe how computers improve the quality of schools, and while watching the news on TV at night she might learn how a new crime mapping system makes the neighborhood safer. These ways of thinking about public returns include both easily measured value, like improved financial flows, as well as highly subjective ones like public safety, service quality, or government integrity. A framework for public value assessment must provide a way to deal with these many perspectives and possible measures of public value. Not every aspect of public value is relevant for a par ticular IT investment. The Washington State Digital Archives project, for example, had no particular public safety related goals, but is of considerable value to genealogists and local historians. Our framework begins the process of narrowing and focusing by starting with the three basic elements of analyzing public value: the investment, the government operations affected, and the stakeholders. At the beginning of a public value assessment, an analyst often knows that there are many connections among theses three elements, but those connections are poorly specified and understood. The situation might be described by the overlapping parts of
8
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Figure 5 below. The public value is to be found by unpacking, so to speak, the area of overlap. The task of the assessment, therefore, is to identify the connections and gather data about how the IT investment produces value for the relevant stakeholders. Close attention to all three elements is essential. The connections among these elements are the keys to a fully informed public value assessment. The framework provides a way of describing these connections to show how public value is generated and the risks involved therein. Focusing on one or two elements alone cannot reveal the necessary scope of public values involved in an IT investment or how they can be assessed. The risks of slighting one or another of these elements in an assessment can be substantial. In addition to missing significant public returns, such limited thinking can lead to stakeholder resistance, flawed technology decisions, or poor integration with or disruption of business processes.
Analyzing those connections can, in principle, start in any part of the problem. Ultimately, of course, these three elements must be considered together in order to ensure that the value assessment for any par ticular investment project or system is tailored to the specific value context, investment situation, and the beneficiaries involved. In practice, however, the assessment must begin somewhere. The framework presented below describes the logic and methods that guide the assessment process. The sequence of activities shown for an assessment is not designed or intended, however, to suggest how IT investment decisions should be made.
Figure 5. The Basic Elements in the Public Value Framework
The Stakeholders Public Value The Technology Government Investment Programs & Operations
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
9
A. The Framework Strategy
The framework describes public value assessment as a series of steps to gather specific data and use it to answer questions that lead to public ROI results. The results can include measures and documentation of public returns, risk analysis of the threats to achieving those results, and suggestions for presenting results to decision makers. Each component of the framework deals with a different set of questions and possible data to use in subsequent steps, and so for th. By working with the ideas and methods presented in the framework, the analysis can produce a public ROI assessment that is broad in scope, yet tailored to the par ticular investment of interest. The steps are not intended to be used in a lock-step manner. The questions and inquiry involved in one part of the framework may be obvious and easily answered in some cases and quite complex and difficult in others. It may be necessary to explore later steps before preceding ones are finished, and then cycle back. In
general, however, the sequence follows the path shown in Figure 6 below. The steps illustrate how the process of value identification and measurement carries through from the goals of an IT investment, through the value generating mechanisms of the business processes, connection with stakeholders, to specific data and reporting. These are the main components of the framework. By starting with a deliberately broad scope, then narrowing to specific questions for each project, this approach addresses one of the shortcomings of previous work on ROI for government IT, namely the narrow scope of possible value questions considered. By expanding the initial scope of analysis to include a broad view of public returns, the framework can lead to a more comprehensive result. The framework thus does not replace or supersede the existing methods developed by others for assessing returns to government IT investments. Rather the framework shows how these methods can be used in assessment, what additional value issues should be addressed, and how additional assessment methods may be used.
Figure 6. Steps in Applying the Public Value Framework
10
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
B. How Does Government IT Investment Link to the Public?
The question of how an IT investment can impact public value can be answered initially in terms of three impact mechanisms. These three, alone or in combination, describe the general ways IT investments can connect with the public, as revealed in our research. Identifying these linking mechanisms is an important step in the analysis process because it leads to a more detailed examination of impact pathways than might otherwise occur. IT investment can enhance public value in many ways, some of which are only indirectly related to the performance of a particular IT system. Attention to these indirect and more complex mechanisms can help ensure a full picture of public value results is developed.
Figure 7. Type 1: Direct Service Impact
government with old IT government with new IT
Citizen/ Official
V1 V2
Return = V2 – V1
value propositions for these investments go beyond the important but obvious cost and time savings to include attention to service quality, access, equity, and the full range of values described below.
2. Indirect Service Impacts
Indirect service impacts (Figure 8) occur when back office or infrastructure investments produces changes in a government business process. As in Type 1 mechanisms, the value to the public is reflected in the changed interaction or transaction with a government business process. However, the process change results from an IT investment at least one step removed from interaction with the public. Because of this indirect route, it may be more difficult in Type 2 cases to trace the links from the IT investment clearly and unambiguously to the public. Risks to the effectiveness of the investment are higher as well, due to dependence on business process changes that are independent of the IT itself. The IT investment may have potential results spread over many business process and may interact with other technologies, further obscuring the impact linkages. The ERP system implementations in Austria, Israel, and Pennsylvania that were examined in our case studies are excellent examples of this kind of investment. The primary impact of these systems is on the internal, back office operations of the government. How to identify and describe
1. Direct Service Impacts
Direct service impacts (Figure 7) occur when IT is embedded directly in a service delivery process, generating service changes that enhance value to the citizen or official involved in the service process. The net public value is just the difference between the value of the new and the old. Value in this sense, and in the other mechanisms as well, is assessed from the point of view of the public participant in the transaction, not the government. The total value may be a composite of several separate measures, such as lower cost to the citizen, plus increased satisfaction with quality of service, plus enhanced trust in government. The specific kinds of value involved are described in more detail in a later section. The e-government investments in our case studies and much of the literature on e-government provide many examples of this kind of investment. These include systems for fee and tax payments, license application and renewals, obtaining information, filing forms, etc. The public
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
11
Figure 8. Type 2: Indirect Service or Impact
Citizen/ Official
V1 V2
old government business process new government business process
old IT infrastructure new IT infrastructure
Return = V2 – V1
these linked back office operations is discussed in a later section of the framework in terms of business reference models (see Figure 10 on page 15). The impacts on these internal operations may be very positive and extensive, and still remain hard to trace to many kinds of value gains for individual citizens or organizations. This is an impor tant part of the public value problem dealt with in detail in the value types discussion to follow. ERP systems also contribute to public value generation through the Type 3 mechanism discussed below.
3. Mixed Service & Environmental Value Impact
In this third type of impact mechanism (Figure 9 below), the links to public value
are more complex. The direct value mechanism shown here as Vd, is the same as a Type 1 or Type 2 mechanism, producing a value increment for a citizen, official, or group (shown in a simplified way). In this type of system, the new IT is also linked to changes in the environment and relationships between the direct beneficiary and other entities (persons, groups, organizations) in the public arena. This indirect mechanism can result in additional public value flowing from interactions outside the government, between other persons or organizations. The changes in the environment may also produce direct value beyond the direct transactions. There are many examples of this type of value mechanism in our cases and in other settings. In the Washington State Digital Archives case at least two such impacts
Figure 9. Type 3: Mixed Direct, Indirect Service & Environmental Impact
environment Citizen/ Official Vi Citizen/ Official Vd government with new IT
Return = Vd + Vi Vi = indirect value Vd = direct value
12
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
occurred. The ability of the county auditors to shift responsibility for preserving digital records to the state archive produced savings in cost and workload at the county level (Vd). This allowed county auditors to improve services to their citizens in other areas (Vi). In addition, the accessibility of the government records online increases the overall environment of transparency for the government, benefiting all citizens. In the Service New Brunswick case, Web access to company registrations provided direct value to citizens such as accountants, lawyers, financial institutions, and the general public searching for such business information (Vd). In addition, the decision by CGI to locate its Global E-Government Headquarters in Fredericton, New Brunswick, due in large part to its strong partnership with Service New Brunswick, benefits the economic development of the city of Fredericton and surrounding areas in the province (Vi). In a different way, putting some forms of information online, such as the Toxic Release Inventory in the US, enable citizens to obtain benefits from other transactions, such as lawsuits against polluters (Vi).5
the same outputs or goals with lower resource consumption. In our Austria and Pennsylvania case studies, for example, new ERP systems helped achieve substantial efficiencies in financial management and other core administrative functions of government. Increases in effectiveness – increasing the quality and/or quantity of the desirable thing. Our case study of Service New Brunswick, for example, repor ts how an online registry for land data can contribute to improvements in proper ty tax administration. Enablement – providing means or allowing otherwise infeasible or prohibited desirable activity, or preventing or reducing undesirable events or outcomes. In our Washington State Digital Archives case study, for example, putting birth and marriage records online enabled research by local historians and genealogists. Intrinsic enhancements – changing the environment or circumstances of a stakeholder in ways that are valued for their own sake. For example, our Israel case showed how enhanced financial accounting and reporting in the Merkava ERP opened government financial decision making to greater transparency. The examples above illustrate public returns that accrue in addition to, and largely independently of, internal efficiency gains that may accompany the IT investment. These value generators can also operate together, increasing the overall return. Research by the local historians mentioned above was enabled by access to previously inaccessible records; online access to these records also made research much more efficient compared to paper records. Access to crime mapping information by citizens could also improve the efficiency of choosing a place to live or locate a business.
C. What Kinds of Impacts Matter for Public Value?
Just identifying these operational mechanisms in general terms, however, does not tell the full value story. Each mechanism can involve more than one kind of value generator. These mechanisms show how IT investments can link to increased public value, but not how that increase is produced. Each mechanism has the potential to generate more than one kind of public value increase, depending on the details of the situation. The framework recognizes four basic kinds of public value generators, listed below, each with a different range of measurements and implications for assessment: Increases in efficiency – obtaining increased outputs or goal attainment with the same resources, or obtaining
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
13
D. What is the Investment? Linking IT to Goals and Business Processes.
1. Links to the Business of Government
This analysis of public value generation requires identifying how the IT investment project links to government goals and activities. IT investments generate value in relation to the policy and administrative setting in which they operate. The goals and business processes provide links between how the technology operates and the interactions with stakeholders that generate value. The analysis, therefore, includes linking the investment to the relevant government goals, operations, and business processes. This linking process is more complex than it may appear, requiring a comprehensive and reasonably detailed picture of government goals and operations. Such a picture will ensure that all the relevant links between the technology and business processes will be identified. Many government IT investments have potential links across many agencies and processes. The Washington State Digital Archives, for example, collects records from hundreds of state and local agencies. The Merkava project in Israel will eventually involve all government agencies. The Pennsylvania IES supports human resources management, budgeting, and other administrative functions for all executive agencies in the Commonwealth. The relevant business process setting of an IT investment, therefore, can be quite extensive. To deal with this contingency, this part of the framework is based on just such a comprehensive picture. This is a useful analysis strategy because it helps draw attention to value generating aspects of the investment that might be missed because they result from indirect effects or complex interactions across many agencies or processes.
The first part of the linking takes advantage of the comprehensive integrated descriptions of government goals and processes that are found in the enterprise architecture work currently underway for US and other governments. For our purposes here, the most useful comprehensive description of government processes is in the US Federal Enterprise Architecture (FEA) Business Reference Model (seen in Figure 10).6 With some minor modifications, the components in this FEA model can be used to identify the business operations and general goals of an IT investment in most government agencies. Local governments, for example, do not typically have defense and national security operations. The supporting documents that form part of the Business Reference Model contain descriptions of each of the components. These descriptions can be used to clarify meaning and help identify which components are linked to a particular IT investment. While comprehensive, this FEA model is not the only enterprise architecture model that may be used. There are a number of comparable models or business process frameworks that could work as well, such as those developed for the EU, the UK, Hong Kong, and Malaysia.7 The National Association of State Chief Information Officers (NASCIO) has an enterprise architecture model, adapted for US state governments, that may also be used. The European Commission also suppor ts additional framework activity through the Athena Project, which has published similar documents.8 Using this model, the analysis proceedsby identifying where the IT investment links to government business. This linking can be based on the new technology’s stated goals, organizational location, and intended operational profile. The detailed descriptions of each Business Reference Model component are useful in identifying these links. If needed, those responsible for the analysis can enlist a team of IT and operational experts to help ensure all relevant links are identified.
14
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Figure 10. Elements in the FEA Business Reference Model
Services for Citizens Community and social services Defense and national security Economic development Energy Science and innovation Income security Law enforcement Natural resources Workforce management Information& technology management
Purpose of Government
*
Correctional activities Disaster management Education Environmental management Health International affairs & commerce Judicial activities Transportation Financial management
*
Mechanisms used to achieve purpose
Mode of Delivery Government Service Delivery Direct Services to Clients Knowledge creation & Management Public Goods Creation Regulatory compliance & enforcement
*
Financial Vehicles Financial assistance Credit & insurance Transfers to local governments
Government Operations Support Functions
Support Service Delivery Legislative relations Regulatory development Planning & resource allocation Internal risk management & mitigation
*
Public affairs Controls & oversight Revenue collection General government
Resource Management Functions
Management of Government Resources Supply chain management Human resource management Administrative management Financial management Information& technology management
*
*
*
To illustrate how to use the model, we marked several of the components in Figure 10 with asterisks. The asterisks mark an example of the business links we identified for an IT investment project in the Merkava case study used for this framework: an online reverse auction system for procurement. The agency conducting the online auction sets the terms of a procurement, publishes them, and qualifies vendors as potential bidders. At a predetermined time, a Web site is opened for bidding. All qualified vendors can then submit authenticated price bids to a public space on the site, visible to all other vendors. The bidding continues until the low bid remains unchanged for a predetermined time (e.g., five minutes), closing the auction.
The low price bidder wins the contract. The entire auction takes place in public view and is recorded in detail. For an illustrative example, consider the procurement of new police radios for a law enforcement agency. Based on the nature of the e-procurement system and the component descriptions, we see it as linked to the five marked components in Figure 10: law enforcement, financial management, supply chain management, direct service to clients (vendors), and planning and resource allocation.
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
15
2. Links to Business Processes
Making further links to business processes means connecting the larger goal or function from the Business Reference Model to specific activities identifiable as business processes. For this step, picking the appropriate level of detail is important. Choosing too general or too fine-grained a process analysis will tend to obscure stakeholder interactions that are necessary to identify public value. For this step in the analysis, we recommend another enterprise architecture tool: the Zachman Framework. 9 The descriptions in the first two rows of the Zachman Framework (Scope and Business Model) are a good guide for the most useful level of detail. This method identifies several business processes for the e-procurement example. They include: determining communication needs for the law enforcement agencies budgeting for the procurement recruiting and qualifying vendors conducting the auction paying the vendors
The list could be extended for fur ther analysis by adding deployment of the radios, training for users (human resource management), and evaluation of the impacts of the new radios use in the agencies and business processes where installed. To keep this example manageable, however, we are limiting the analysis to the processes listed below. The components identified from Figure 10 are linked to the business processes in the rows of Table 1 below. Identifying the business processes leads to two key questions: (1) How does a business process generate increased public value? and (2) For whom? A way of answering the first question is recorded in the third column of Table 1. Our analysis of public value generating mechanisms is shown in the discussion of value impact mechanisms below.
E. What Kinds of Public Value are Produced?
Identifying the linkages between business processes and public value generators described above carries the analysis a critical step farther: from how the IT changes a business process to the impact of those changes for a par ticular public or
Table 1. Business Component, Process, and Value Mechanism
Business Component Law enforcement Financial management Supply chain management Planning & resource allocation Direct ser vice to clients Financial management Business Process conduct auction budget for procurement qualify vendors for auction deter mine communication needs conduct auction pay vendors Public Value Mechanism citizen - type 3 citizen - type 3 citizen - type 3 vendor - type 1 citizen - type 3 gov't staf f - type 1 vendor - type 1 vendor - type 1
16
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
stakeholders. That step is accomplished by looking to the value of the impact in terms of the interests of one or another stakeholder. So the framework includes an analysis scheme for taking into account interests that can apply to the full range of stakeholders. The framework employs these six basic kinds of interests: Financial – impacts on current or anticipated income, asset values, liabilities, entitlements, and other aspects of wealth or risks to any of the above Political – impacts on personal or corporate influence on government actions or policy, role in political affairs, or influence in political parties or prospects for current or future public office Social – impacts on family or community relationships, social mobility, status, and identity. Strategic – impacts on economic or political advantage or oppor tunities, goals, and resources for innovation or planning Ideological – impacts on beliefs, moral or ethical commitments, alignment of government actions or policies or social outcomes with beliefs, or moral or ethical positions Stewardship – impacts on the public’s view of government officials as faithful stewards or guardians of the value of the government itself in terms of public trust, integrity, and legitimacy This last interest—stewardship—is deliberately not included in the ideology categor y, though it could logically fit there. Setting it off separately emphasizes its importance in the overall public value framework. Just as corporate managers and directors are responsible for stewardship of
a corporation’s integrity and assets on behalf of stockholders, public managers and elected officials have a parallel responsibility for government on behalf of the public. Recent financial scandals in the US have demonstrated how lack of stewardship in major corporations (e.g., Enron, WorldCom) can destroy their financial value. The government officials in our cases expressed a similar stewardship responsibility in terms of, for example, using IT to increase transparency and accountability. Public surveys of trust in government show that citizens have similar stewardship expectations of government.10 Clearly this list goes well beyond the internal efficiency or service quality impacts of an IT investment. Of course, attending to this expanded concept of value raises many problems of definition and measurement. However, our case research and results from many other kinds of inquiry show the importance of this more comprehensive treatment of value and interests. Many of the measurements and data implied by these interest types are found in existing ROI methods as well. However, none of the methods we reviewed for this white paper cover the full range represented here. Working with this expanded range of value types represents one impor tant contribution of this framework. It is also clear from this way of describing public value that it supplements rather than replaces methods that assess the efficiency gains or savings returns captured by government agencies internally. For example, it seems safe to say that a Department of Motor Vehicle’s internal cost savings from putting license renewals online are invisible and largely irrelevant to drivers. These savings would have no detectable impact on overall tax burdens or benefits resulting from a shift of government resources to some other service. The information about the savings, however, is another matter. If government officials make some political use of the cost savings information, that would represent strategic value to those officials or to their political
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
17
allies. The public learning about the savings might also improve their opinion of the government’s stewardship, fairness, or other ideological value.
thinking and include multiple opinions and information sources Identifying multiple stakeholder roles, internal and external to the organization setting (e.g., internal user, customer, vendor, developer, manager, oversight, politician, taxpayer, analyst, advocate, etc.) Identifying stakeholder expectations, influence potential, past and future par ticipation possibilities, and level of interest In this kind of analysis, continuing reference to the nature of the investment and impact mechanisms will help produce the needed focus and detail.11 The results of a stakeholder analysis have implications for value assessment and for understanding their potential to influence the investment project. These results can be presented in ways that show the multiple dimensions used in the analysis. An example of results from a hypothetical stakeholder analysis is shown in Figure 11. This figure combines data for 14 stakeholders’ roles (SA = Stakeholder A, etc.) and times of possible influence in a project’s lifecycle. This kind of display illustrates some of the complexities of stakeholder analysis and the possibilities for interactions among different stakeholder groups. A more complete analysis would include estimates of the stakeholder’s influence capabilities and specific interests in the project development and outcomes. These issues are discussed in more detail in the section on risk analysis (section III.H). To focus on stakeholder interests, the illustration in Table 2 below shows how the results of a simple stakeholder analysis can be linked to the business process, value types, and mechanisms. The business processes identified in Table 2 on page 20 link to a partial list of stakeholder types in the columns: citizens at
F. Who Receives Value? The Stakeholder Analysis.
Completing the analysis of value impacts requires identifying those with an interest in the value generating process: i.e., the stakeholders. The kind of stakeholder analysis required for this task has three main parts: (1) identifying the persons or groups (including organizations) whose interests are potentially affected, (2) identifying what their specific interests may be, and (3) assessing their role and potential influence in the deliver y of public value. The first two par ts are necessary for the analysis of value impacts and possible assessment strategies and so are discussed here. The third part is more relevant to the risk analysis discussed in more detail in section III.H below. Identifying stakeholders for the framework will depend on in-depth knowledge of the context of the investment and the agencies involved in its development and use. Typically, stakeholder analysis engages a group of participants with extensive knowledge of the political and organizational setting of the investment and, hopefully, with experience in this kind of analysis. Because the analysis is so context-dependent, there are no standard processes to follow. There are consistencies, however, among the many versions of stakeholder analysis methods found in the management, planning, and assessment literature. These include: Involving multiple participants with wide knowledge of the stakeholder environment Looking widely to identify all relevant stakeholders through brainstorming and related methods to stimulate divergent
18
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Figure 11. Stakeholder Analysis Matrix12
Type of Participation
Inform Initiation Identification Consult Partnership Control
SA SH SK
SC SB SF SM SN
SI SJ
SE SD
Stage in Lifecycle
Planning
Execution Implementation Controlling Monitoring and Evolution
SG SL SO SD SA
Sx = Stakeholder X
Closing
large, vendors, and elected officials. An actual assessment would include a more detailed list of stakeholders. The value types most likely to map onto each stakeholder type are entered in the cells. The basis for this mapping is linking the nature of the value as understood in that context with detailed knowledge of the stakeholder groups. This part of the assessment can include data collection about the interests and expectations of stakeholders through interviews, surveys, or focus groups if needed. The entries in the column for the citizens at large, for example, are based on the assumption that their interests in government stewardship would be served by the e-procurement system. While a reasonable assumption, this could be confirmed by collecting opinion data from a sample of citizens, or relying on previous opinion research. Similarly, the entries in the column for government IT staff are based on
assumptions about what advances the interests of those staff members. A successful procurement system would provide political value and some strategic advantage to elected officials, as well as enhance their reputation as good stewards of public resources.
G. How to Demonstrate the Value? Identifying Variables and Methods.
Using the framework up to this point has identified many kinds of data relevant to assessing public value. The next step is to choose the specific variables or points of observation for collecting the assessment data. This is as much a practical problem as an analytical one. Many variables may be relevant for each combination of
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
19
Table 2. Stakeholder Value Matrix
Stakeholders Business Process Value Generator Impact Type Citizens at Large Elected Officials
Vendors
Budget for procurement
intrinsic
3
stewardship
Deter mine needs
intrinsic, ef fectiveness
3
ideology, stewardship
political, strategic
Qualify vendors
ef ficiency, intrinsic ef ficiency, intrinsic, ef fectiveness ef ficiency, intrinsic
1,3
ideology, stewardship
financial, strategic
political, strategic political, strategic, stewardship political, stewardship
Conduct auction
1,3
ideology, stewardship
financial, strategic
Pay vendors
1,3
stewardship
financial, strategic
stakeholder and value type. A few variables that would be relevant to this example are shown in Table 3. This table illustrates two important aspects of its role in the framework and assessment process. First, it is highly unlikely that applying the framework to any IT investment would lead to variables in all, or even most of the cells of this matrix. This matrix can be thought of as an heuristic device, prompting questions about what might be useful and available variables for each row and column without expecting to fill them. The second is that identifying a specific variable relies on combining information about stakeholder interests, the value type, the impact mechanisms, and the context. This is a complex and demanding process. This section offers additional guidance on choosing the best public return variables for a given assessment. In addition, the other ROI methods described below include many
variables and additional methods that can be helpful in that task. However, the more the choice of variables can be tailored to the specific public value context, the more valid and persuasive the assessment is likely to be. An actual public value assessment should be based, of course, on the best available information. But actual assessments take place in practical settings of limited resources and access to data, plus being par t of the additional work needed for internal returns and costs. The priority setting described in the risk analysis section can narrow the field to only the most important public value outcomes. The section below describes additional strategies for choosing the appropriate variables and analysis methods.
20
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Table 3. Stakeholder Variables by Type
Stakeholder Impact Variables
Value Type Financial
Citizens at Large local economic development
Vendors increased revenue
Elected Officials
Political
suppor t from vendors
positive media coverage Social enhanced reputation in agencies receiving radios increase level of market competition increase in citizen access to infor mation about procurement: greater transparency Strategic increased market share enhanced reputation for being moder n & progressive enhanced reputation for promoting market competition
Ideological
increased knowledge about competition Stewardship increased trust in gover nment reputation for frugality & good management
1. Variables and Analysis Methods
The choices of variables and analysis methods for the empirical par ts of a public value assessment should be considered together. In terms of basic measurement and analytical methodology, what constitutes an appropriate analysis depends, in part, on the types of data and variables involved.
The scope of this framework, as applied to a particular IT project, could encompass a very wide range of data types. Many kinds of quantitative data from financial sources, operations research, and surveys are appropriate for statistical analysis, modeling, and simulations. Many of the social, political, and intrinsic value returns can be expressed in normalized scales, or may best be revealed in qualitative terms or in simple dimensions that are not suitable
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
21
for much quantitative manipulation. To help guide the assessment, given the very large number of possible public return data types and variables, the framework provides two kinds of resources. The first is a general scheme suggesting variable types and sources for different value types (see Table 4). The second consists of summaries of the approaches and variables available in a range of existing ROI methods that can be of value in completing an assessment (see Table 5). A wide range of possible public value data can be identified by the methods presented here. The framework approach is based on the assumption that virtually any kind of data can be useful in describing public value creation, from the most precise quantitative figures available from financial or physical measurements, to material as diverse as the content of blogs or observation of user or customer behavior. A conclusion about public value creation
requires an inference, since value does not stalk about wearing a label. Valid inferences about value can be formed from qualitative as well as quantitative data, content analysis as well as statistics. Taking these four principles into account, choices made about how a specific analysis proceeds should be based on three criteria: 1) What constitutes the best data? 2) What kind of analysis is appropriate to that type? 3) Who will be the audience for the conclusions reached? The best kind of data available will be specific to the operational and stakeholder context. The kinds of analysis appropriate to various data types are shown in summary form in Table 4.13 Beyond these general considerations, the choice of variables and analyses for an assessment can draw on a volume of existing work on ROI methods for guidance. These methods, summarized in Table 5, vary widely in the number and type of variables used, the scope of public value considered,
Table 4. Variable Type for Public Value Data
Variable Type Examples Analysis Methods averages, totals, dif ferences, percents, parametric & nonparametric statistics (cor relation, regression, etc.), simulation, time series, optimization averages (with assumptions), frequencies, scale analysis, nonparametric statistics (x2, etc.), cluster analysis cor relation, frequencies, scaling analysis frequencies, clustering, network analysis content analysis, word frequency, semantic analysis process maps, network analysis, behavior patter ns
ratio scale
financial data, time, age, distance, frequency, transaction volume
inter val scale
opinion scales, rating scales, grades, Liker t scales
ordinal scales nominal variables text/discourse data obser vational data
rankings, rating scales gender, ethnicity, location, relationships, user behaviors inter views, blogs, news repor ts user studies, field obser vation notes, photos, film/video
22
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Table 5. Summary of Selected ROI Methods and Models
Methods/Models Method Descriptors Time Orientation Time Frame Institutional Focus Gover nment Level Project Orientation Value Measuring Methodology, US Federal CIO Council prospective cross sectional public national/federal Social Retur n on Investment, REDF (US) retrospective longitudinal private n /a Best Value Perfor mance Indicators (ODPM, UK) retrospective mixed public local UK local gover nment programs predominantly quantitative +qualitative Gar tner Gover nment Perfor mance Framework retrospective cross sectional* public general general perfor mance orientation predominantly quantitative +qualitative highy quantitative, employing indexes mixed inter nal & exter nal
IT mixed, predominantly quantitative general categories + examples & suggestions
social ser vices
Variable Types
predominantly quantitative
Variable Specification
highly specific
highly specific
PROI Orientation
primarily inter nal, some public
mixed inter nal/exter nal with exter nal emphasis broad method fully specified par tial suppor t not included suppor t materials examples provided
mixed; many inter nal and exter nal variables large & diverse limited to variable definition not included not included none none
PROI Variable Set Analytical Methods Analytical Process Suppor t Risk Analysis Decision Suppor t Presentation Tools
nar row, limited some method suppor t modest suppor t included decision tool included minor suggestions
limited
method specified
unknown limited inclusion suppor t materials results matrices
*Includes some longitudinal variables
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
23
Table 5. Summary of Selected ROI Methods and Models (continued)
Methods/Models Perfor mance Reference Model (US Federal Enter prise Architecture) retrospective mixed public national-US program perfor mance mixed predominantly variable categories mixed ver y large & diverse not included not included n/a material included par tial
Method Descriptors
Demand and Value Assessment Model (Australia)
Public Ser vice Value Method, Accenture
MAREVA (Gover nment of France)
Time Orientation
prospective cross sectional public national/general general mixed, largely quantitative mixed categories and some specified mixed inter nal and exter nal; few political large, diverse highly specified & detailed highly specified & detailed included suppor t materials and guidelines included
combined longitudinal public general general predominantly quantitative mostly specific perfor mance & cost metrics primarily inter nal, some public nar row, limited analysis method specified included not included suppor t materials standard displays
retrospective cross sectional* public national-France IT mixed, largely quantitative
Time Frame Institutional Focus Gover nment Level Project Orientation
Variable Types
Variable Specification
largely specified
PROI Orientation
mixed, largely inter nal limited included included included material included par tial
PROI Variable Set Analytical Methods Analytical Process Suppor t Risk Analysis
Decision Suppor t Presentation Tools
*Includes some longitudinal variables
24
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
and the level of analytical detail and technique included. Some are intended for use prospectively, in planning for and justifying an investment. Others are aimed primarily at showing impacts of investments after the fact. They also vary in terms of the degree to which they deal with both internal and public value results of the investment, and whether they are designed specifically for IT or government investments generally. The summary of these methods or models in Table 5 can help in the selection of variables and analysis to fit the IT project. The SROI (Social Return on Investment) model, the only private sector oriented one in the summary, is included for its special features. Its private orientation refers not to the commercial sector but to a private philanthropic orientation. The method was developed by a San Francisco foundation to assess its social and economic development programs. It illustrates not so much how to assess an IT investment but rather how to deal with personal and community impacts in a systematic and comprehensive way.
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
25
H. What Can Interfere With or Prevent Public Returns? The Risk Analysis.
Government IT innovation is risky business. Evidence from many surveys and cases suggests that the threats and other challenges of IT projects often overwhelm the capabilities of the developers and implementers. Data from the private sector tells a similar story. For this paper, we focus our attention not on the full range of risks to government IT project success, but concentrate on the public return aspects of that problem. The distinction between the public return and more generic aspects of risk analysis is not, however, a simple one. The approach is the same: identify and evaluate threats, develop and evaluate response methods, and produce a summary analysis and
mitigation strategy. To identify and evaluate threats, we return to the overall value proposition schematic presented earlier (Figure 12). In this figure, we identified two kinds of risk: 1) development risk and 2) benefit risk. Development risk, simply put, is risk that the development and implementation of the IT will fail outright or will not perform as designed and intended. Benefit risk applies to whether the IT investment will fail to produce the envisioned benefits in spite of being successfully developed and implemented. For the public value framework we focus on benefit risk. Of course any threat to the development of the IT is an indirect threat to public returns. However, several of the ROI methods described below include adequate analysis methods for development risk. These methods do not deal adequately, however, with the additional threats and issues in benefit risk.
Figure 12. Public Return on Investment Value Chain
Secondary Performance Gain
Public Cost Internal Agency Returns
*
IT Investment
*
Business Goal or Problem Private Cost (?) Vendor/ Industry Benefit
* *
Public-atLarge Benefit
*
*
Direct Citizen Benefit
* * *
Policy/ Electoral Benefit
*
*
*
General Economic Development
* *
Development Risk Benefit Risk Direct Benefit Flow Indirect Benefit Flow Direct Benefit Accumulation Indirect Benefit Accumulation
26
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Table 6. Examples of Threats to Public Value
Sources of Threats Threat Type Theory Failure overestimate of public demand leads to non-use of ser vice; flawed understanding of impact mechanisms; misunderstanding of stakeholder interests Exogenous Factors confounding or cancelling ef fects: down-tur n in economy cancels financial advantages; changes in gover nment counter stewardship outcomes; political scandal cancels positive media repor ts indirect and second-order ef fects undetectable; data inaccessible due to privacy or sensitivity
To creation of retur ns
To detection of retur ns
lack of or er roneous baseline data; mis-specification of outcome variables; inappropriate analysis methods or models; accounting systems based on inappropriate data definitions and models
The benefit risks associated with the creation and assessment of public returns come from threats to the creation of the returns and to their detection. And there appear to be two main sources of those threats: one is what we will call “theory failure” and the other is “exogenous factors.” In theory failure, the underlying assumptions or theory on which the project is based are flawed or simply wrong. One such theory failure caused the US Department of Education to abandon a multi-million dollar pilot project for online college student financial aid administration. Developed without significant par ticipation from college financial aid officers, the system did not attract supporters and generated much stakeholder resistance. 14 It is important to use both sources and types of benefits threats in a full risk analysis, as suggested by the two-by-two array in Table 6. The risk analysis process can then use the threats identified in this way to estimate the potential loss or cost that each threat represents and the likelihood of the threat materializing. These estimates should be based to the extent possible on input from stakeholders, analysts, users, developers,
managers, and policy makers. This will provide a basis for accurate estimates and concentrating attention on the threats with the highest combined loss-likelihood estimates. In some cases, the value, cost, and loss-likelihood estimates can be quantified to yield decision tools for moving ahead with an investment. The displays in Figure 13 and Figure 14 on the next page, show such a result from the US Federal CIO Council’s Value Measurement Method, which employs voting and other methods to prioritize and estimate quantitative values for cost and their related risks. The decision boundaries shown in the figures come from management decisions or some other deliberative process.15 The threats due to difficulty in detecting indirect or second-order effects deserve some added discussion. Public value outcomes can extend beyond those involved in the direct interaction with government. For example, drivers are the direct beneficiaries of an online license renewal system. However, the scope of possible indirect value outcomes and beneficiaries can be very broad. Residents in the neighborhoods of the license renewal agencies will benefit
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
27
Figure 13. Example of a Value-Risk Boundary Analysis
Figure 14. Example of a Cost-Risk Boundary Analysis
28
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
to some degree from reduced traffic congestion and pollution. Shopkeepers in that neighborhood may lose business for the same reasons. Second order effects may be even more diffused and difficult to detect. Learning to trust the online process for license renewal, for example, can result in more use and greater benefits from using other online services. Since risks are tied both to stakeholders and possible value outcome variables, pursuing this line of reasoning can lead to a very large, and likely infeasible, list of risk analysis factors and tasks. To work within resource constraints will require limiting the risk analysis to the most important value variables and stakeholders. Setting priorities for this kind of analysis will therefore be essential, and must be based on the goals of the project and stakeholder value estimates.
one step can easily lead to revisions in conclusions or data collected at preceding steps. This is par ticularly so for the risk analysis steps in the process. It is not feasible to complete the risk analysis for achieving any particular public value return until the details about that return and how to assess it are known. Loops can occur in the process when a step produces unexpected or challenging results. For example, identifying all related business processes for a large, complex project may produce too many results to carry forward in the analysis. That would require looping back to reduce the scope. So the potential user should view this process summary as a rough guide rather than a lock-step procedure. The process also represents a comprehensive approach to public value assessment. We argue that all the steps are needed to ensure that the full range of public value returns will be included in any assessment and report. Some small or very narrow projects, of course, may have limited public value potential and not require such an extensive review. But the research and analysis leading to this framework have clearly demonstrated that the scope of public value possible from government IT investments can far exceed what an internally focused, financial assessment will reveal. Unless assessments use a much broader view, as represented in this framework and other models cited above, the full value of government IT investments will not come to light. It also seems likely that this same kind of analytical framework would be useful for other forms of government investments. We did not, however, explore that possibility for this report.
I. Overview of Using the Framework.
The framework is a combination of individual analysis steps combined into an overall process or plan for public ROI assessment. A summary of that overall process is shown in Figure 15 on pages 30-31. As shown there, the process follows the arrows, starting with a clear understanding of the larger context of public value generation. The investment goals that emerge from the investment plans and understanding of the context link to the business processes and public value impact mechanisms. Analysis of the impact mechanisms and public return types leads to specific impacts listed in the stakeholder interest analysis. Identifying the interests and impacts will then guide choosing public value variables, analysis methods, risk analysis, and ultimately, reporting tools and techniques. For simplicity, this process diagram is somewhat idealized. It shows only the basic linear path through the various steps. In practice, the actual path is rather more likely to have backtracking and loops. Learning at
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
29
Figure 15. The Public Value Framework Overview
Public Cost Internal Agency Returns
*
IT Investment
Secondary Performance Gain
*
Business Goal or Problem Private Cost (?) Vendor/ Industry Benefit
* *
Public-atLarge Benefit
*
*
Direct Citizen Benefit
* * *
Policy/ Electoral Benefit
*
*
*
General Economic Development
* *
Development Risk Benefit Risk Direct Benefit Flow Indirect Benefit Flow Direct Benefit Accumulation Indirect Benefit Accumulation
Business Reference Model
Value Impact Types
Business Component Law enforcement Financial management Supply chain management Planning & resource a l l o c a ti o n D i r e c t s e r v i c e t o c l i e n ts Financial management
Business Process conduct auction budget for procurement qualify vendors for auction deter mine communication needs conduct auction pay vendors
Public Value Mechanism citizen - type 3 citizen - type 3 citizen - type 3 vendor- type 1 citizen - type 3 gov't staf f - type 1 vendor - type 1 vendor - type 1
Public Return Types F = Financial P = Political So = Social I = Ideological St = Strategic Sw = Stewardship
(Continued on next page)
30
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Figure 15. The Public Value Framework Overview (continued)
Identify Risk Mitigation Strategies Results & Reporting Value Analysis Variable and Analysis Selection Risk Analysis
Stakeholder Impact Variables Value Type Financial Political Citizens at Large local economic development Vendors increased revenue Elected Officials
suppor t from vendors positive media coverage enhanced reputation in agencies receiving fur niture increase level of market competition increase in citizen access to infor mation about procurement: greater transparency enhanced reputation for promoting market competition
Social Ideological
Strategic
increased market share increased knowledge about competition
enhanced reputation being moder n & progressive
Stewardship
increased trust in gover nment
reputation for frugality & good management
Stakeholders Business Process Budget for procurement Deter mine needs Qualify vendors Conduct auction Pay vendors Value Generator intrinsic intrinsic, ef fectiveness ef ficiency, intrinsic ef ficiency, intrinsic, ef fectiveness ef ficiency, intrinsic Impact Type 3 3 1,3 1,3 1,3 Citizens at Large stewardship ideology, stewardship ideology, stewardship ideology, stewardship stewardship political, strategic political, strategic political, strategic, stewardship political, stewardship Vendors Elected Officials
(Continued from previous page)
financial, strategic financial, strategic financial, strategic
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
31
J. How to Summarize and Present Results?
This framework for public value assessment presents both problems and opportunities for summarizing and repor ting. The problems arise from the large number and types of results that the assessment can produce. For presentation to policy makers and non-technical audiences, the results should be as simple and accessible as possible. Simple char ts and summary tables are best for this purpose. For multiple stakeholders and value variables, the number and complexity of charts may become a problem. This section discusses some of the specific issues and alternative methods available. For qualitative variables, the presentation of results can show the presence of a value result, and information about magnitude and direction, if relevant and available. Using the information in Table 3 (page 21), a summary display similar to Table 7 below can present the types and direction of results, with the estimated relative
magnitudes as well. The down arrow in the stewardship row for vendors indicates a potential loss to vendors due to transparency increases that diminish opportunities for some forms of influencing procurement. Whether this is a positive or negative result overall may not be clear. For the more quantitative results, where the public value variables lend themselves to calculation and statistical analysis, many presentation and summary methods are available. Clear and simple visuals are generally preferable to tables of quantitative data for non-technical audiences. For cross sectional data, a column char t, such as in Figure 16, can be used to present the same comparative public value data converted to a ten-point index or scale. Such a multidimensional chart can be difficult to interpret, however, and some ROI methods use the so-called radar char t for the same data, shown in Figure 17. This kind of char t provides a clearer image of the pattern of results for each stakeholder type and value dimension. However, this kind of display becomes much more difficult to
Table 7. Qualitative Display of Stakeholder Impacts
Stakeholder Impact Variables Value Type Financial Political Social Ideological Strategic Stewardship Citizens at Large Vendors Elected Officials
32
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Figure 16. Stakeholder Impact Column Chart
Public Value Index by Stakeholder Groups
10 9 Public Value Index 8 7 6 5 4 3 2 1 0 citizens at large vendors elected officials Financial Political Social Ideological Strategic Stewardship
Figure 17. Stakeholder Impact Radar Chart
Financial 10
Stew ardship
5
Political
0
Strategic
Social
Ideological Citizens at Large Vendors Elected Officials
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
33
Figure 18. Portfolio Risk and Value Comparisons - DVAM
interpret if the number of axes or stakeholders is large. The use of indexes can also be used to compare the relative risk and value of alternative investments on a value scale if the variables can be combined into a single index. The Demand and Value Assessment Model (Australia) provides guidance on how to produce such a public value index and use it for comparison purposes. An example of that kind of result is shown in Figure 18 above. For cross sectional assessments, more elaborate presentations are available, par ticularly if the value variables are indexed or based on quantitative data. The Accenture Public Service Value Model (PSV) can use historical data about government program effectiveness and costs to show changes in performance over time. 16 The model is based on the principle that public value is created when both outcome results and cost effectiveness increases. An example of this form of results presentation
is shown in Figure 19 for the Arizona Depar tment of Revenue. Overall, the performance shown is improving from 1999 to 2001 and 2002 to 2003 when the organization is creating increased outcomes and doing so more cost effectively. Documentation of results through this kind of chart or visual device should include background material about methods and measurement issues. This framework advocates the use of a wide range of data and analytical styles, many of which are considered controversial or suspect in some environments. Decision makers and analysts often have deep-seated biases about the validity of qualitative data or social statistics, or other non-traditional material for ROI analysis. Those performing a public value assessment must, therefore, be thorough in providing rationale and supporting material for all results. They must be attentive to the issues of interpretation and validity that may affect how key members of their audience respond to the assessment results.
34
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Financial and Social Worth
Figure 19. Historical Performance Change Model (PSV – Accenture)
PSV Agency Performance Matrix (FY 1996 – FY 2003)
Higher
10% 8% 6% 4%
1997
Outcomes
1996
2%
2001 2003 2002 5%
10% 15% 20%
-20%
-15%
-10%
1998 -5% 2000
AVERAGE 0% SCORE 0% -2%
-4% -6% -8% -10%
1999
Lower
Lower
Cost Effectiveness
Higher
The principles for SROI analysis used by the REDF organization provide valuable guidance for the conduct of the kind of assessment described here. These principles can be applied to traditional ROI analysis as well, but seem particularly well suited to the public value issues involved with this framework and related methods. The principles make an appropriate bridge from the general ideas and methods presented here to the difficult work of carrying out public value and ROI assessment in practice. SROI Design Principles: 17
4.
Replicable – The framework should result in similar conclusions when applied by different practitioners who use similar parameters (such as the scope and options). Thus, results should also be comparable over time and among organizations, at least among analyses that use similar options and where the options are clearly noted. Transparent – The process by which the analysis was prepared, and the context in which results should be seen, should be transparent. Credible – The results should be credible to investors, purchasers, mangers, and other users. Integrative – The framework should relate to, and where possible integrate with, other approaches to understand social value. Avoids misuse – Proper application of the framework should reduce the risk of misuse of, or misleading, SROI numbers or analyses.
5.
6. 1. Feasible – A basic SROI Analysis should be something any organization can afford to prepare itself. 7. 2. Accessible – The process should be understandable and relevant to organizations at various stages of development. 8. 3. Rigorous – The method should be substantive and well-executed, and based upon premises that are validated by informed practitioners.
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
35
9.
Open source – The framework should continuously be informed and improved by the collective wisdom of practitioners in an inclusive, iterative process.
10. Useful – Applying the framework should result in information that enables users to make decisions or take actions that further their goals.
36
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
IV. Reflections on the Framework and the Value of Public Value
T
his framework grows from the rather simple principle: that the value of a government’s investment in IT should be assessed from the point of view of the public it serves. That principle leads us to identify two distinct but equally important types of public value: delivering benefits directly to citizens and enhancing the value of government itself as a public asset. From this seemingly simple beginning grows the rather high level of complexity involved in working through this framework and its potential application. That complexity derives in part from the way these ideas radically expand the possible scope of inquiry needed to identify and document public value creation. That expanded scope brings with it a host of measurement problems that emerge when the many social and political outcomes come into play. The cost of this expanded and more complex assessment can be quite high. So government executives and IT planners can rightly question whether assessing public value is wor th the effort. Part of the answer may be that they will have no choice. As we noted earlier, elected officials have begun insisting on more comprehensive cost and return analyses for IT investment proposals. Conventional approaches to ROI analysis may simply be inadequate for these increased demands. More importantly, however, the desire for a more comprehensive and robust justification for new IT investments reflects their greater complexity and ambition. The low-hanging fruit available from earlier IT investments, such as establishing Web presence and automating simple service transactions, has typically been harvested. More substantial improvements in government are now possible by exploiting the integrative and transformative potential of IT, but they require much larger investments. These projects require a way
of assessing public value that matches their greater scope and complexity, a way that can build the needed public support and guide development. In this light, the cost of using a framework such as this should be easily justified. Moreover, the new knowledge about public value possible from such an assessment can help guide other forms of investment and contribute to long-term government improvement.
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
37
V. Appendix: The Research Basis for the Public Value Framework
A. Consultative Workshop
On July 13, 2005, the Center for Technology in Government, in collaboration with SAP , hosted a consultative workshop on assessing public return on government investments in IT. A select international group of 22 experts from government, academia, and the private sector were invited to meet with researchers from CTG to discuss the core issues and themes that make assessing public ROI for IT investments such a complex and difficult problem. The workshop results were used to inform CTG’s next steps for the project. The workshop results were used to help develop a preliminary framework for conducting ROI analysis in the government sector and design the case studies research approach. This approach was employed to identify and select the five case studies, which were used to refine the preliminary framework and develop the white paper. The lists below summarize the key issues that emerged from the workshop that characterize the complexity of the issue. Workshop par ticipants provided their recommendations in the form of key themes for CTG to explore in the follow-on research. what should be done, but not on what has already been done. Governments have trouble harvesting savings, which often get moved around the budget. There is no straightforward quantitative bottom line value measure for ROI in public sector. Government is multidimensional. Nonlinear, complex interactions among benefits—hard to measure results and link to specific programs or technologies, i.e., proving the causal relationship between the two. ROI requires advanced project management and portfolio management skills that are often lacking. Comprehensive ROI analysis can greatly increase transparency of government decisions and investment results. This level of transparency is a “risk” in and of itself, increasing the probability for embarrassment and criticism leading to loss of support. ROI is done in a vacuum. Not focused on the strategic investment. Lack of longer-term tracking and assessment makes it difficult to build a measurement model. It is hard to evaluate IT ROI elsewhere in the government enterprise because the outcome frameworks (intersectors) aren’t established.
1. Key Issues Characterizing the Complexity of Assessing Public ROI for Government IT Investments
Lack of incentives to assess public ROI. There may be no consequences for absence of ROI or other demonstration of results. Lack of historical perspective and data. Government tends to be prospective (not retrospective), so it tends to focus on
38
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
2. Themes to Explore in the Research
Value and impact measurements should take into consideration the cost impacts on other business processes, by elimination or changes in the way we work together. Look for efficiency and cost-reduction in non-IT areas. IT is an enabler, and the purpose of IT is to enable other things, including changing the culture of organization, strategies, etc. Thus, it is important to see how IT is enabling returns in other areas. Assessment should include attention to how particular IT investment enables IT elsewhere. ROI analysis must get beyond evaluating things on a more traditional basis to include questions of outcomes for broader range of beneficiaries or stakeholders. Social cost-benefit analysis and political considerations involve different people making different decisions. It is important to understand how this may impact the IT aspect of ROI calculations. Need more attention to risks as well as benefits. Assessment should include attention to par ticular government sensitivity to risk factors. Method development should include system analysis that engages a broad scope of operations. Analysis should include questions about leadership, feasibility, political support as well as results/outcomes, how constituents and opponents will react. Focus assessment on question of getting ROI on the programs, not the IT; IT doesn’t deliver the outcomes.
Por tfolio management is a big theme in current discussion of IT management. As applied to public ROI, implies looking at the overall picture, not just a slice of the project. ROI methods must accommodate or provide for shared outcomes that cross over existing stovepipes. While benefits are measured in terms of traditional (financial), political and social factors, risks are measured based on other factors: (1) technical, (2) organizational, (3) time, and (4) political. Find relationships between inputs and outcomes. Once indicators are developed cause and effect relationships can be explored using econometric methods. Value of IT investment may be enabling change. Find ways to ensure follow-through in delivery and assessment of government programs. Explore what are the attributes of a good public ROI model: measurable in different ways, creates expectations of performance and assessment at both IT and program levels, is integrated with the budget process, and is included in a yearly review.
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
39
B. Case Study Summaries18
1. The Government of Israel’s Merkava Project
Some government IT initiatives focus narrowly on a specific technical problem, like enabling mobile data communication or Web-based transactions. Others are driven by a broader, more ambitious goal, to use IT as a tool to transform government. The Merkava Project in the Government of Israel (GOI) is most cer tainly one of these transformative effor ts. It is transformative in its own right, as an effort to restructure the financial, logistics, and human resource components of governmentwide administration into an integrated ERP system. Merkava is also part of a comprehensive e-government initiative that includes the ERP system as one of five layers of new technologies and operational systems for enhanced internal operations and improved benefits and services to citizens. These layers are part of a multiyear strategic plan, described in 2002 by Prime Minister Sharon: “The e-government project is an expression of the government goal to provide state-of-the-art strategic systems, while affecting social and economic national targets. We believe that this infrastructure constitutes a better tomorrow and that tomorrow is already here.” From its beginnings in 1999 to the present, the Merkava ERP project has been a very large undertaking. It is described by the Ministry of Finance as the largest single IT project in Israeli history. It has been implemented or is in the roll-out process in 30 government units (out of 100), and currently supports approximately 2,300 users. The 2005-2007 IT master plan calls for the Merkava system to be implemented in 90% of the government’s offices during the planning period. Work on the other four layers in the overall eGov initiative has been
progressing in parallel with the Merkava implementation. The integration provided by the Merkava ERP provides an important par t of the infrastructure for operations in the three upper layers of the initiative and contributes to the returns that they deliver as well. The success of implementation effor ts to date and the growing record of returns, both internal to the government and the public, suggest that the remaining roll-out will continue as planned and the accumulation of returns will continue to grow.
2. The Austrian Federal Budgeting and Bookkeeping System
How a government obtains value from its citizens and external public environment and how it returns value to them are linked directly to its financial management systems. These systems are crucial links in the flows of revenues to the government and the flow of expenditures and services back to the public. Improving financial management, therefore, has the potential to produce significant returns in terms of both greater internal efficiencies and enhanced public returns. These were the goals of the Austrian Federal Budgeting and Bookkeeping System project initiated in 1997 by the Minister of Finance and supported by the Chancellor (Prime Minister). The goal of the project was to redesign and integrate the electronic workflow of the federal government’s budget and bookkeeping processes. The strategy they chose was to implement a single ERP software standard throughout the federal government, along with the adoption of necessary legal authority. The result was that, by 2005, the Ministry of Finance successfully consolidated 85 bookkeeping units across the federal government into one federally owned, but privately operated, agency. The consolidation and integration produced immediate and tangible benefits in terms of
40
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
internal efficiencies. These resulted from the implementation of a new standardized work process for accounting and budgeting throughout the federal government, with reduced work process steps and processing time. As of 2005, the legal consolidation of the numerous bookkeeping depar tments into one agency, along with the technical and organizational implementation of the ERP , has resulted in annual savings of approximately €30 million. In terms of these returns to the government itself, the project is clearly a resounding success. In terms of broader public returns, the project goals went beyond internal financial management efficiencies. The aim of restructuring the Austrian Federal Accounting and Bookkeeping system was much more than simply an “IT investment” from the very beginning. The bookkeeping system was part of a larger effort to implement the SAP ERP technology throughout the Austrian Federal Government as part of a governmentwide public management reform effort. The ERP technology was one part of a comprehensive strategy that included legislative reform, governmentwide reorganization and consolidation, and implementing a modern accounting and budgeting standard across the government. Though the specific ERP technology was just one element of a larger approach, involving legal and organizational strategies, the ERP technology played a very important enabling role vis-à-vis the legal and organizational strategies.
record keeping in a democratic society. That is also the foundation of the public value proposition guiding the Digital Archives program: that the state has the constitutional and statutory mandate to preserve and provide access to records of enduring legal and historical significance. The growth of electronic records in government agencies in the 1990’s presented a challenge to the State Archives’ ability to fulfill its mission, since it lacked an effective program and technology to deal with records in these new formats. The Washington State Digital Archives (WSDA), a program within the Office of Secretary of State, is the response to that challenge. It was initiated by the Office of the Secretary of State, with initial planning begun by the then State Archivist in March of 2000. The initiative was taken up in 2001 as a priority by the newly elected Secretary of State Sam Reed, and included in the Secretary of State’s 2001-2007 Strategic Plan. The project was subsequently supported by the state legislature and included in the State of Washington’s 2001-2003 Capital Budget. Construction of the physical hub of the WSDA in Cheney, Washington, began in January 2003. Beginning in mid-2001, the WSDA team began exploring a wide range of technologies and techniques for collection, access, and preservation. The results led to the custom development of a Web interface and database design that blended the latest technologies with traditional archival theory to create a first-of-its-kind digital records repository for state government. The goal of the program was to make the historical electronic records of Washington’s state and local governments easily accessible, from anywhere, at anytime. The initial vision and value proposition were carried through a complex political and technical process to a functioning digital archiving program and facility delivering the promised public value. The WSDA project team began with a clear vision of the expected value of the Digital Archives to both the government and citizens. In addition, it successfully identified
3. The Washington State Digital Archives
Washington State’s investment in digital archiving for government records provides a highly focused and successful example of pursuing public value through information technology. The job of collecting, preserving, and providing access to the records of government is central to the mission of Washington’s Office of Secretary of State. That mission recognizes the fundamental importance of government
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
41
the benefits that it needed to communicate to the state and local government agencies that were keepers of public recorders in order to mobilize their support and par ticipation. The WSDA project demonstrates a strong connection between the initial high-level public value proposition that motivated the project and its realization in the performance of WSDA itself.
recently taken steps to move in this direction by implementing the necessary institutional structures and policies to take fuller advantage of this enterprise-level asset.
5. Service New Brunswick
Service New Brunswick (SNB) is well-known internationally for its expertise in providing multi-channel “single window” citizen access to government services as well as for developing and maintaining geographic information databases. SNB’s awardwinning approach provides one-stopshopping for different government services on behalf of provincial and municipal government agencies, but also provides a linkage to the Canadian Federal Government in a “joined up” government model. SNB was launched in a time of high pressure from citizens for improved service delivery; today it serves the province through its award winning service deliver y model, and also and maybe more importantly in the long run, through its innovations in economic development. The numbers behind SNB speak for themselves; customer satisfaction numbers are the highest in Canada going from about 50% in late 1980’s to 92% in 2005. These numbers require that responsible examinations of public value include SNB. But there is more to this story than customer satisfaction ratings. The essential par t of the public value story in this case is Service New Brunswick (SNB) as a quasigovernmental organization and its evolution as an integrated service provider and economic development innovation. SNB is a public corporation with a single shareholder – the government of New Brunswick. This shareholder has guided SNB with this model throughout its existence: make public service good business.
4. The Commonwealth of Pennsylvania’s Integrated Enterprise System
From the beginning, the Commonwealth of Pennsylvania’s Integrated Enterprise System was much more than a technology project. The ERP implementation continued through three gubernatorial administrations with consistent top level executive support; eventually putting in place the technical infrastructure and enterprise standards for core administrative functions with improved public value. Between early 2001 and mid-2004, the ERP implementation for the five business functions was completed for 53 Commonwealth agencies including all 49 of the agencies under the governor’s jurisdiction. Immediate returns in the form of improved government operations were realized soon after implementation and continue today. However, this infrastructure also provides the Commonwealth with capability that can be fur ther leveraged to support additional improvements in government operations that go well beyond direct improvements in core administrative functions. The Commonwealth has begun such efforts. The IES infrastructure provides public returns in the form of direct improvements in the efficiency and effectiveness of core administrative functions. This infrastructure also provides the basis for improvements in the back office operations of other service areas which, in turn, offer improved services to the public. The Commonwealth has
42
CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
Endnotes
Expenditure data from: DG Information Society, European Commission; http://ec.europa.eu/idabc/en/ document/4336/5860; China: CCID Advisory (2005, May) http://www.ccidadvisory.com/ report_details.php?r_serial=229; India: Public Sector Technology and Management (2004, Jul 12) http:// www.pstm.net/ar ticle/index.php?articleid=236; U.S. Office of Management and Budget Web site: http:// www.whitehouse.gov/omb/egov/g-9-budget_highglights.html; Datamonitor. (2005, Jun). IT opportunities in state and local government: Five-year outlook for vendors. New York: Datamonitor.
1
http://www.datamonitor.com/~3d016a3bb9db4c6cb608be617baf36f5~/Products/Free/Report/DMTC1150/ 010DMTC1150.pdf; also http://zdnetasia.com/news/business/0,39044229,39371070,00.htm http://www.publictechnology.net/modules.php?op=modload&name=News&file=ar ticle&sid=1920
2
Matthew Weigelt, “Senate committee questions e-gov ROI.” FCW (July 17, 2006), http://www.fcw.com/ ar ticle95291-07-17-06-Web&newsletter%3Dyes View and download copies of the white paper and case study reports at www.ctg.albany.edu/projects/proi.
3 4
This terminology is consistent with recent work on assessing government IT investment. See for example: Center for Digital Government, Prove IT: The Disciplines of Harvesting Value from Public Sector Information Technology , 2006. (www.centerdigitalgovernment.com); and eGovernment Economics Project (eGEP): Measurement Framework Final Version . Prepared for the eGovernment Unit, DG Information Society and Media, European Commission, by Christiano Codagnone (RSO SPA) and Paulo Boccardelli (Luiss Management), May, 2006.
5 The Toxic Release Inventory is a publicly available data base maintained by the US Environmental Protection Agency that contains information on toxic chemical releases and other waste management activities reported annually by certain covered industry groups as well as federal facilities (http:// www.epa.gov/tri/).
These are taken from the Business Reference Model document, which contains considerable detail on the definitions and details of these components (http://www.whitehouse.gov/omb/egov/documents/CRM.PDF).
6
See the European Interoperability Framework for Pan-European eGovernment Services, http:// europa.eu.int/scadplus/leg/en/lvb/l24226b.htm; The UK eGovernment Interoperability Framework, http:// www.govtalk.gov.uk/documents/eGIF%20v6_1(1).pdf; Malaysian Government Interoperability Framework (MyGIF), http://unpan1.un.org/intradoc/groups/public/documents/APCITY/UNPAN016393.pdf;
7
Hong Kong Special Administrative Region (HKSARG) Interoperability Framework, http:// www.bilgitoplumu.gov.tr/kdep/34/hongkongif_v1_0.pdf.
8 The NASCIO Enterprise Architecture materials are listed at http://www.nascio.org/publications/index.cfm. The Athena Project repor ts are available at http://www.athena-ip.org. 9 10
The Zachman Institute for Framework Advancement. http://www.zifa.com/
See, for example, polling results at Zogby International (http://www.zogby.com/index.cfm), or The Pew Research Center for People and the Press (http://people-press.org/). Additional information and suggestions for conducting stakeholder analyses can be found in these sources:
11
http://www.ittoolkit.com/workbooks/qt_stake.pdf; http://www.stsc.hill.af.mil/crosstalk/2000/12/smith.html; http://www.boxesandarrows.com/view/understanding_organizational_stakeholders_for_design_success; http://www.ctg.albany.edu/publications/guides/smar tit2?chapter=11§ion=5; http://www.ctg.albany.edu/ publications/guides/and_justice_for_all?chapter=8§ion=2 Overseas Development Administration, Guidance Note on How to Do Stakeholder Analysis of Aid Projects and Programmes , Social Development Department, July 1995.
12
More information on these data types and analysis methods can be found in social science research method texts. See, for example, Earl Babbie, The Practice of Social Research, Tenth Ed. Belmont, CA: Wadswor th, 2004.
13 14
Stephen Burd. “Education Department Pulls the Plug on Novel Way to Deliver Financial Aid. Chronicle of Higher Education . Vol 46, No. 29 (March 24, 2000). See US Federal CIO Council Best Practices Committee. VMM How-To-Guide . http://www.cio.gov/documents/ValueMeasuring_Methodology_HowToGuide_Oct_2002.pdf
15 16
Accenture Public Service Value (PSV) Model, http://www.accenture.com/Global/Services/By_Industry/ Government/PublicGovernment.htm
17 18
A Framework for Approaches to SROI Analysis. http://www.redf.org/ View and download copies of the white paper and case study reports at www.ctg.albany.edu/projects/proi.
CENTER FOR TECHNOLOGY IN GOVERNMENT— ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK
43
Center for Technology in Government 187 Wolf Road, Suite 301 Albany, NY 12205 Phone: (518) 442-3892 Fax: (518) 442-3886 E-mail: [email protected] www.ctg.albany.edu
doc_957140032.pdf