Description
With this particular brief file concerning creating an entrepreneurial appalachian region findings and lessons from an evaluation.
Creating an Entrepreneurial Appalachian Region:
Findings and Lessons from an Evaluation
of the Appalachian Regional Commission’s
Entrepreneurship Initiative 1997 – 2005
Evaluation Team:
Deborah Markley, RUPRI Center for Rural Entrepreneurship
Erik Pages, EntreWorks Consulting
Brian Dabson, RUPRI
Thomas Johnson, RUPRI
Sara Lawrence, RTI
Sara Yanosy, RTI
Karen Dabson, RUPRI Center for Rural Entrepreneurship
April 2008
i
ACKNOWLEDGEMENTS
The Evaluation Team – Deborah Markley, Erik Pages, Brian Dabson, Thomas
Johnson, Sara Lawrence, Sara Yanosy, and Karen Dabson – would like to
acknowledge the important role played by Catherine Renault on this project.
Cathy was involved in developing the proposal and early project design, before
returning to her roots in Maine as the Director of the Office of Innovation,
Department of Economic and Community Development. She also provided input
on the final report. The team would also like to acknowledge the input of the
advisory group – Thomas Lyons, Edward Malecki, and Jonathan Potter. Their
keen insights and input throughout the project helped to strengthen project
design and the final report. The team also benefited from discussions held with
ARC staff throughout the project, particularly Greg Bischak and Ray Daffner, as
well as a group of advisors who discussed a summary report on key findings.
The senior members of the evaluation team would also like to acknowledge the
heroic efforts of two members of the team, Sara Lawrence and Sara Yanosy, to
develop and complete the database for the sample projects. They diligently
tracked down project leaders to discuss projects that, in some cases, were
completed almost 10 years earlier. They shared data and insights that greatly
enhanced this evaluation.
Finally, this evaluation would not have been possible without the many
individuals throughout the ARC region who shared project data, recollections and
insights into the importance of the ARC Entrepreneurship Initiative to their
individual rural places and to the broader region. We acknowledge their important
contribution and greatly appreciate the time they took to talk with team members
about their past and ongoing efforts to build a more entrepreneurial Appalachian
region.
ii
EXECUTIVE SUMMARY
From 1997 through 2005, the Appalachian Regional Commission (ARC) invested
nearly $43 million in a ground-breaking program to stimulate and support
entrepreneurship across Appalachia. The Entrepreneurship Initiative (EI) was
the first large scale attempt to give greater focus to homegrown business
development as a regional economic development strategy.
The Rural Policy Research Institute (RUPRI), the RUPRI Center for Rural
Entrepreneurship, EntreWorks Consulting, and RTI were commissioned by ARC
in 2006 to conduct an evaluation of EI in terms both of outcomes achieved by a
sample of funded projects and of broader policy impacts across the region. The
evaluation team undertook literature reviews, reviewed project files for a sample
of 114 projects, conducted phone interviews with 36 stakeholders and experts,
developed a metrics framework, completed interviews with project staff
associated with 88 projects, made four site visits, and conducted a meta-analysis
of the outcomes and impacts. The team’s work was informed by a three-person
advisory committee of leading academic experts on entrepreneurship – Dr.
Thomas Lyons, Dr. Edward Malecki, and Dr. Jonathan Potter.
A review of entrepreneurship trends in the region during the EI provided the
context and backdrop for ARC’s investments. Over the period of the EI, trends in
nonfarm proprietor and microenterprise employment in Appalachia showed
increases in line with the nation as a whole, but trends in nonfarm proprietor
income showed the region lagging behind the nation and slipping further behind
by 2005. Data on the impact of entrepreneurship on the local economy showed
that only 15 percent of Appalachian counties saw income increases associated
with entrepreneurial activity that were higher than the national rate. It appears
that entrepreneurship had greater impact in terms of both employment and
income in the southern tier of Appalachian states. This evaluation was not
designed to discern cause and effect between EI investments and these trends;
however, the context is important for interpreting evaluation findings and
understanding the resulting recommendations.
Also of contextual importance is the rapid growth in interest in and adoption of
entrepreneurship development policies and programs since the EI began in
1997. A review of current literature provides many insights on the linkages
between entrepreneurship and regional development and on the efficacy and
impact of different types of entrepreneurship programs. This body of research
work was not available to the designers of the EI but was particularly helpful in
conducting the evaluation and determining appropriate performance metrics.
The evaluation team identified three goals that were at the core of the EI – to
increase the number of entrepreneurs establishing businesses in the region, to
increase the survival rate of such ventures, and to increase the proportion that
develop into high growth businesses that create jobs and wealth in Appalachia.
iii
These goals were operationalized through five program categories –
entrepreneurship education, access to capital, business incubators, sector
interventions, and technical assistance and training. There was also a sixth
cross-cutting category of community capacity-building.
As identified through the final reports submitted to ARC, the EI led to the creation
of at least 9,156 jobs, the retention of a further 3,022 jobs, the formation of 1,787
new businesses, and the provision of services to 8,242 businesses. The cost per
job created was $4,693, which compares favorably with other economic
development efforts. ARC investments were made in 340 unique projects across
the region at an average investment per state of $3.3 million and investment per
capita of $1.82. The total ARC investment has leveraged an additional $72.8
million in private investment for those projects that have been closed, a figure
that is projected to rise to $109.9 million when all projects in the portfolio have
been completed.
Other metrics identified through in-depth investigation of outcomes from the
sample of projects expand on this picture. In the 88 projects included in the
sample, over 11,500 students and teachers participated in or received training in
entrepreneurship education projects, 1,500 entrepreneurs took part in sector-
focused activities and another 1,620 received training and technical assistance.
The evaluation team’s assessment of qualitative impacts were drawn from
interviews with project leaders most familiar with the investments and regional
stakeholders and entrepreneurship experts with deep experience both in the
region and with entrepreneurship development – key informants. Common
themes identified were that ARC investments:
Raised the profile of entrepreneurship as a development strategy, helping
to change the mindset within the region
Represented “but for” money in the region, providing start-up funding for
innovative projects
Leveraged additional resources that helped some projects achieve scale
and impact
Facilitated networking and collaboration among practitioners
Helped to change people’s attitudes, particularly among youth and their
teachers.
There were also a number of lessons gleaned from the many interviews
conducted across the region, some of which will benefit those who are actively
engaged in implementing entrepreneurship programs – the practitioners – and
others which will guide future programs either of ARC or agencies across the
country pursuing similar efforts. For the practitioners, the lessons were of three
kinds:
iv
Lessons for Program Leadership
o Successful entrepreneurship initiatives had sparkplugs or local
champions that provided leadership for these efforts.
o Local capacity was a key to success.
Lessons for Program Management
o Program self-sufficiency (sustainability) and success went hand in
hand.
o Entrepreneurship development was recognized to be a long-term
process.
o Successful projects altered their goals and approaches as
conditions warranted.
Lessons for Program Outreach
o Partnerships and collaborations were important to success.
o Successful projects celebrated and shared the story of their
success.
For program designers and implementers, again the lessons were of three kinds:
Lessons for Program Design
o Practitioners and entrepreneurs have unique local knowledge that
can be applied to program design and subsequent program
refinements.
o Successful initiatives brought together related investments, in this
case, other regional economic development or entrepreneurship-
related investments.
Lessons for Program Implementation
o Getting EI funds to local partners was dependent upon state
leaders, such as governors and program managers, and varied
based on the importance assigned to the initiative.
o The size of ARC grants placed limits on regional impacts.
Lessons for Program Impacts
o Building a broader base of support for entrepreneurship
investments requires continued efforts to “make the case” to local
leaders.
o Programs can be improved by embracing long-term and locally-
driven evaluation of program outcomes and impacts.
Finally, the evaluation team offered three sets of recommendations to ARC.
Regarding investments in entrepreneurship development:
v
Entrepreneurship development initiatives should include assessment of
existing capacity and capacity-building activities as part of the project
design.
Entrepreneurship development initiatives should be made with a focus on
the long term.
Entrepreneurship development initiatives receiving investments should be
market-driven and practice continuous improvement.
Emphasis should be placed on investing in initiatives that demonstrate the
ability to form regional partnerships and collaborations.
The second set of recommendations is for creating a “best in class” metrics
system:
“Job creation” is an overused metric, paints an incomplete picture of the
outcomes of entrepreneurship development investments, and should be
replaced by an “entrepreneurship development metrics portfolio.” In
addition to jobs created/retained and new business starts, this system
should include outcome measures such as:
o Change in business profitability (performance) following a capital
investment
o Number of youth considering business creation as a career option
after participation in an education program
o Percent of incubator tenants who graduate and remain in the region
o Change in total sector sales over time as a result of investment to
encourage sector development
o Number of customers still in business after receiving technical
assistance
o Positive change in perceived community support for
entrepreneurship as measured by community pre- and post-
surveys.
A “best in class” metrics system requires investment in a “best in class”
evaluation system.
The third set of recommendations focus on program design and management:
ARC’s initiative process should be regularized so that state program
managers can more effectively plan for and promote the use of the
resources.
ARC’s proven experience can be applied to developing and delivering
effective, region-wide education programs that help make the case for
entrepreneurship as a core economic development strategy for the
Appalachian region.
To build on the momentum created by the EI, ARC should create a Next
Generation Entrepreneurship Innovation Initiative that will be
groundbreaking in its design. A long-term investment is recommended
vi
that incorporates all the learning from the EI and the emerging
entrepreneurship development field.
vii
Table of Contents
PAGE
CHAPTER 1 INTRODUCTION 1
CHAPTER 2 REVIEW OF THE ENTREPRENEURSHIP
DEVELOPMENT FIELD 5
CHAPTER 3 CONCEPTUAL FRAMEWORK AND REVIEW
OF THE LITERATURE 14
CHAPTER 4 APPROACH TO THE EVALUATION 31
CHAPTER 5 THE WIDER REGIONAL ECONOMY –
ENTREPRENEURSHIP TRENDS IN APPALACHIA 44
CHAPTER 6 ARC INVESTMENT PORTFOLIO – PROJECT
IMPLEMENTATION AND REPORTED METRICS 58
CHAPTER 7 SAMPLE PROJECT OUTCOMES AND
BROADER POLICY IMPACTS 75
CHAPTER 8 LESSONS LEARNED FROM THE ARC
ENTREPRENEURSHIP INITIATIVE EXPERIENCE 88
CHAPTER 9 CONCLUSIONS AND RECOMMENDATIONS 106
BIBLIOGRAPHY 118
APPENDIX A LITERATURE REVIEW 130
APPENDIX B NON-PROJECT STAKEHOLDER INTERVIEWS 159
APPENDIX C PROTOCOL FOR PROJECT LEADERS 160
APPENDIX D PROTOCOL FOR NON-PROJECT STAKEHOLDERS 167
1
CHAPTER 1
INTRODUCTION
In 1997, the Appalachian Regional Commission (ARC) began a multi-year
initiative to invest in projects designed to build entrepreneurial economies across
the region – the Entrepreneurship Initiative (EI). Since that time, ARC has
invested almost $43 million (composed of EI funds and dollars from other ARC
accounts) in various entrepreneurship development projects. Over 10 years,
these projects have created jobs and businesses, supported partnerships and
collaborations, and helped leaders at the community and state levels recognize
the value of entrepreneurship as an economic development strategy. While ARC
has collected select data to describe the outcomes of these investments, these
data do not begin to tell the story about the extent to which and how the EI has
had an impact on the region. This evaluation was designed to provide a more
detailed and nuanced description of the impact of the EI, both in terms of the
outcomes achieved by the portfolio of projects and the broader policy impacts
across the region. This evaluation, however, must first be placed within the
context of the initiative’s history.
HISTORY OF THE PROGRAM
The EI emerged in 1997 as a special initiative under the leadership of then-
Federal Co-Chairman Jesse White. White had long pushed for economic
development strategies focused more on home-grown business development as
opposed to business recruitment and attraction. As he told a Federal Reserve
Bank of Kansas City symposium in 2000, ”[W]e’ve got to re-instill in rural America
the idea, particularly in Appalachia and the Rural South, that job creation,
business creation, and, most importantly, wealth creation, occurs as a result of
local indigenous business creation.”
1
The EI was designed as a test of public
policy approaches that sought to achieve this objective.
The genesis for the EI was in the belief that entrepreneurial activity could be
encouraged through strategic investments in education, business assistance,
and capacity building projects. Areas for strategic investment identified prior to
the launch of the EI included:
Access to capital and financial assistance
Technical and managerial assistance
Technology transfer
Entrepreneurial education and training
Entrepreneurial networks.
1
Jesse White, “Overview Panel Comments,” Proceedings of Federal Reserve Bank of Kansas
City Conference on Beyond Agriculture: New Policies for Rural America (April 2000):193.
2
The EI was originally funded with $15 million over three years, with additional
investments being provided beyond this original total. Through 2000, ARC had
invested $17.6 million in 169 projects.
2
Through 2003, ARC had invested $31.4
million in 368 projects. An additional amount of approximately $11 million was
invested in subsequent years through the EI or via the use of Area Development
funds. Table 1.1 shows ARC investment in entrepreneurship-related activities by
source of funds for the period of this evaluation. While all ARC investments in
entrepreneurship were not specifically drawn from EI funds, the rationale for
making these investments was clearly driven by the goals associated with the EI.
Today, ARC continues to fund entrepreneurship development-related projects
under the Asset-Based Development Initiative launched in 2005, although the EI
is no longer operating as a stand alone initiative. The Asset-Based Development
Initiative is building on the foundation laid by the EI, leveraging the new
businesses and additional capacity created by EI investments.
Table 1.1 ARC Funds Invested in Entrepreneurship-Related Projects, 1997-
2005, by Source of Funds
Source of funds $ invested
Entrepreneurship Initiative 26,546,366
Area Development 6,698,724
Commission’s EI 3,206,803
Distressed Counties 3,052,109
CoChair Fund 2,432,440
Regional Initiatives 868,673
Goal Fund 114,000
New Markets Fund 52,574
TOTAL 42,971,688
ORGANIZATION OF THE EI
ARC’s special initiatives are traditionally designed to support innovative local
projects and also to serve as a demonstration of new regional development
strategies and approaches. ARC officials were very explicit about the importance
of the EI as a means to demonstrate the viability of entrepreneurship as an
economic development strategy. Interviewed experts also emphasized this
aspect of the EI.
The EI’s organization reflects this dual focus on educating local leaders and on
maximizing the effect of local project investments. In pursuing its wider
educational goals, ARC recognized the need to form partnerships and engage
other institutions in order to achieve sustainable impacts in the region. Through
the formation of four advisory committees with significant private sector
2
For an early review of these investments, see Regional Technology Strategies, Inc., Evaluation
of the Early Stages of the Appalachian Regional Commission’s Entrepreneurship Initiative, A
Report to the Appalachian Regional Commission, December 2001.
3
participation, ARC was able to tap into national expertise and “best practices” to
guide the initiative. Each advisory committee was charged with providing input to
ARC in a particular program area – entrepreneurship education, technical
assistance, capital access, and sectorally targeted strategies. In particular, these
advisory committees helped to organize a region-wide educational effort that
augmented the specific project investments made by ARC. Elements of this
educational effort included, for example:
Scholarships to the 16
th
Annual Entrepreneurship Education Forum and
support for three regional entrepreneurship conferences; creation of the
Springboard Youth Entrepreneurship Education Awards; creation of the
Entrepreneurship Everywhere web-based resource guide
3
Support for conferences on sector-based development; a competitive
grant program for “strategic sectoral interventions” in the region
Sponsorship of a workshop on community development venture capital;
follow-up regional workshops on equity capital in rural communities;
publications on developmental venture capital in the region; creation of an
opportunity fund to leverage private investment using the New Markets
Tax Credit program
Funding for four workshops on business incubation “best practices” in the
region; a survey on business incubators in Appalachia; creation of a
business incubation mentor program.
These region-wide educational efforts were, for the most part, funded outside the
EI, using other ARC dollars, such as Commission or CoChair funds. While using
distinct funding sources, these projects also contributed to the mission of the EI –
to encourage the development of an entrepreneurial economy in the region. In
fact, it is not possible to separate the impact of broader ARC investments in
entrepreneurship from those specifically identified with the EI. Most of ARC’s
investments through the EI resulted in project-specific outcomes; the outcomes
of their educational efforts were much broader, serving to raise the overall level
of regional awareness about specific aspects of entrepreneurship development –
education, sector approaches, capital and incubation. The impacts, in most
cases, were not confined to particular communities or even states. Instead, the
impacts could be measured in terms of increased understanding of several
aspects of entrepreneurship development and enhanced capacity to design and
implement entrepreneurship activities within the region.
While this evaluation did not seek to quantify ARC’s capacity building and
education impacts related to entrepreneurship development, this effect was cited
in nearly all of the interviews with outside experts and state level officials.
During interviews, the evaluation team regularly heard comments such as the EI
“opened people’s eyes to other possibilities” or “the program got people talking
about entrepreneurship.” Several other programs, such as the New Markets
3
Available at .
4
Venture Capital program, built on momentum generated by the EI. In fact, two of
the first six New Markets Venture Capital companies operate in Appalachia.
While the EI’s impact in terms of building better awareness of entrepreneurship
cannot be quantified, it should be included in any full accounting of the EI’s
outcomes. The EI was designed as a demonstration, i.e., to test new policy
models and to encourage Appalachian communities to focus within on nurturing
home-grown businesses. Today, nearly all of the Appalachian states are involved
in some form of organized efforts to promote entrepreneurial development.
While the ARC cannot claim sole credit for this shift in thinking, based on
feedback from key stakeholders it is clear that the EI played a role in convincing
policy makers that supporting entrepreneurs is good economic development
policy.
STRUCTURE OF THIS REPORT
This report presents the findings from an evaluation of ARC’s investments in
entrepreneurship development from 1997 through 2005. Based on analysis of
data collected by ARC, in-depth research into the impacts associated with a
sample of ARC-funded projects, and interviews with a broad range of
stakeholders, this evaluation provides key insights into the value of the EI to the
region. Chapter 2 provides an overview of entrepreneurship development,
particularly in terms of the state of the field in 1997, the beginning of the EI. This
context is important for understanding how innovative ARC was in launching the
EI, plowing new ground in terms of economic development. Chapter 3 lays out
the conceptual framework developed to guide this evaluation process and
reviews relevant evaluation literature related to entrepreneurship development.
This review of the literature served as the basis for developing the metrics
framework that would guide in-depth evaluation of sample projects. Chapter 4
describes this metrics framework and the overall research approach used in this
study.
Chapter 5 describes key aspects of the Appalachian region’s business and
economic environment that created the context for the EI. It is this context that
created the need for and gave rise to the projects that became part of the EI
portfolio. Changes over the ten year time horizon of the EI are also presented.
Chapters 6 and 7 provide the key findings from this evaluation. Chapter 6
describes the types of projects in which ARC invested and presents data on key
metrics collected by ARC for all projects. Chapter 7 provides a detailed
discussion of the impacts, quantitative and qualitative, associated with the
sample of projects considered for this evaluation. Chapter 8 presents the
lessons learned from the EI experience for both practitioners of entrepreneurship
development and policy makers who may be considering similar types of
investment programs in the future. Finally, Chapter 9 provides recommendations,
based on evaluation findings, for the field and for ARC going forward.
5
CHAPTER 2
REVIEW OF THE ENTREPRENEURSHIP DEVELOPMENT FIELD
While entrepreneurship has always been a feature of the American (and
Appalachian) landscape, the history of explicit public support for regional
entrepreneurial development is quite brief. Indeed, ARC’s Entrepreneurship
Initiative serves as one of the first such large-scale efforts, and the absolute first
such effort investing Federal dollars in regional entrepreneurship strategies.
At the time of the Entrepreneurship Initiative’s unveiling in 1997, ARC’s
leadership had grown increasingly concerned about Appalachia’s future vitality.
The region was especially hard hit by cutbacks in declining sectors such as
timber, textiles, tobacco, and was seeking new approaches to jump start
development. Entrepreneurship-focused economic development strategies were
viewed by ARC as “. . . a critical element in the establishment of self-sustaining
communities that create jobs, build local wealth, and contribute broadly to
economic and community development.”
4
By investing in the Entrepreneurship Initiative in 1997, the ARC was something of
an “early adopter.” Up to that stage, economic developers had focused almost
exclusively on industrial recruitment as a core strategy. The industrial
restructuring of the 1970s and the 1980s had stimulated some interest in
business retention and technology development strategies, but entrepreneurial
development strategies represented something of a “new thing” in 1997.
5
Nevertheless, ARC was not promoting entrepreneurship without some evidence
that it would make a significant contribution to economic opportunity in
Appalachia. In the late 1980s, Eisinger discussed the limited returns to traditional
economic development activities and suggested a more important role for
entrepreneurial, growth from within, economic development strategies.
6
There
was some research that questioned the efficacy of traditional “smokestack
chasing” and some that provided insights into the impact of specific programs
such as business incubation, youth entrepreneurship, and targeted technical
assistance. What was missing at that time was the evidence to link
entrepreneurship to regional and community development, and to identify which
policy interventions, and in what combination, would lead to increased
entrepreneurial activity. Over the past decade, there has been rapidly growing
interest among researchers and policy analysts in entrepreneurship as a core
4
Appalachian Regional Commission, Entrepreneurship Initiative: Program Summary and
Approved Projects, September 2003, 9 August 2007
.
5
For background, see Erik R. Pages, Doris Freedman, and Patrick Von Bargen ,
“Entrepreneurship as a State and Local Economic Development Strategy,” The Emergence of
Entrepreneurship Policy: Governance, Start-Ups and Growth in the U.S. Knowledge Economy,
Ed. David Hart, Cambridge, MA: Cambridge University Press, 2003.
6
Peter K. Eisinger, The Rise of the Entrepreneurial State: State and Local Economic
Development Policy in the United States, Madison, WI: University of Wisconsin Press, 1988.
6
economic development strategy, and more evidence has been found to support
this approach. It was important to the evaluation to understand the implications of
this expanded understanding in order to appropriately assess the value and
impact of the EI investment.
This chapter provides a brief overview of research, policy and practice that
addresses two primary questions:
What makes a region (or community) entrepreneurial?
What policy or community interventions, if any, can help support
entrepreneurial development?
WHAT MAKES A REGION ENTREPRENEURIAL?
Researchers have examined a host of both macroeconomic and microeconomic
factors that help explain the innovation or entrepreneurial capacities of a region.
At the broadest level, recent research has reviewed how leading demographic
trends are correlated with new firm births. For example, in a 1994 review article,
Reynolds, Storey and Westhead describe key factors that are associated with
higher levels of new firm starts.
7
These include net population growth, increases
in personal or household income or regional gross product, high population
density, high educational levels among population, and a high percentage of
population between the ages of 25 and 44. Acs and Armington similarly find that
higher relative education levels strongly affect new firm formation rates,
especially among service businesses.
8
These same factors have been
highlighted in the cross-national research studies conducted under the auspices
of the Global Entrepreneurship Monitor (GEM) research consortium.
9
While this literature contains something of a consensus about several key factors
(e.g., population growth, higher education levels) that are correlated with higher
new firm formation rates, some researchers caution that the causal chain still
remains unclear. For example, Feldman notes that many key factors associated
with entrepreneurship, such as the presence of local venture capital firms, may
actually lag instead of lead entrepreneurial growth.
10
In other words, these
7
Paul Reynolds, D.J. Storey, and P. Westhead. “Crossnational Comparisons of the Variation in
New Firm Formation Rates, Regional Studies 28.4 (1994):443-456.
8
Zoltan Acs and Catherine Armington, “Using Census BITS to Explore, Entrepreneurship,
Geography, and Economic Growth,” Research Summary No. 248, Small Business Administration
Office of Advocacy, February 2005. Similar findings that emphasize the importance of local
educational levels as well as local employment growth and productivity rates can be found in
Advanced Research Technologies, “The Innovation-Entrepreneurship NEXUS,” Research
Summary No. 256, Small Business Administration Office of Advocacy, April 2005.
9
Begun in 1999, the Global Entrepreneurship Monitor project is a cross-national effort to assess
national rates of entrepreneurship. Begun with ten countries, the research program now tracks
entrepreneurial activity in forty-two countries. The project’s research reports can be accessed at
.
10
Maryann Feldman, “The Entrepreneurial Event Revisited: Firm Formation in a Regional
Context,” Industrial and Corporate Change 10.4 (2001):861-891.
7
regional assets emerge as a result of strong local entrepreneurial activity. They
are a by-product, as opposed to a trigger, for high firm formation rates. Feldman
concludes by cautioning that each region’s entrepreneurial development activities
generally emerge from a unique and idiosyncratic mix of historical factors, local
resources, and business conditions.
PUBLIC POLICY AND ENTREPRENEURSHIP
While many economists and researchers continue to assess how various
demographic and macroeconomic factors help drive regional entrepreneurial
activity, policy makers, including the ARC’s leadership, are more concerned with
questions of how (and whether) policy interventions can affect a region’s
entrepreneurial propensities and development patterns. A large and growing
literature examines this issue,
11
and much of this work has helped guide the
strategic direction of ARC’s own Entrepreneurship Initiative.
While individual analysts may differ on the relative importance of certain regional
factors, there is relatively strong consensus that five factors are especially
important:
Access to Capital
Enabling Culture
Local Networks
Supportive Infrastructure
Supportive Government Policies.
A more detailed description of each factor and the public policy approaches used
to address them is provided below.
Access to Capital
Successful entrepreneurial regions tend to enjoy a wide range of options for
financing businesses at different stages of the business life cycle. Successful
regions host a variety of financial institutions that can provide businesses with a
range of both equity and debt financing options. Rural communities may be
especially challenged on this front. Recent US Department of Agriculture-
sponsored research has found that rural areas tend to have fewer lenders and
less diverse markets. While rural areas do have less bank competition, the study
found few rural-urban differences between the cost and availability of debt
11
For an excellent comprehensive literature review, see Jill S. Taylor, “What Makes a Region
Entrepreneurial? A Review of the Literature,” Monograph, Cleveland State University, Center for
Economic Development, September 2006. Other good sources include Brian Dabson, et al.,
Mapping Rural Entrepreneurship, Washington, D.C.: CFED, August 2003; Deborah Markley, Don
Macke and Vicki Luther, Energizing Entrepreneurs: Charting a Course for Rural Communities,
Lincoln, NE: Heartland Center for Leadership Development, 2005, and OECD, Entrepreneurship
and Local Economic Development, Paris: OECD, 2003.
8
financing.
12
In fact, a recent study of capital access in the Appalachian region
found that banks in the region had higher small business loan to deposit ratios as
compared to national figures.
13
However, businesses in rural regions experience capital access problems when
their needs fall outside the type of loans traditionally made by banks – e.g.,
microloans (to fund very small enterprises with limited collateral) and equity
finance. Microenterprise development was just emerging in the early to mid-
1990s. A recent study of the industry noted that only about one-quarter of
microenterprise programs listed in a 2001 directory existed before 1991.
14
The
importance of providing both training and microfinancing to entrepreneurs was
just being viewed as a development strategy as ARC invested in the EI.
Appalachia also suffers in terms of access to equity finance. For example, a 2000
ARC study found that only 1/3 of 1% of all venture capital ($117 million) was
invested in rural regions of Appalachia.
15
A more recent study of capital access in
the Appalachian region shows improved access to equity capital.
16
Through
2004, the study shows that ARC had invested in 11 funds in seven states and
these funds collectively had invested $13.6 million in regional businesses. If all of
this investment is considered new to the region, it represents a 12% increase in
equity capital as compared to that found in 2000. However, the later study also
identified a need to continue to expand the capacity of community development
financial institutions in the region by broadening sources of funds, increasing self-
sufficiency, and expanding products available to businesses.
As a result of these market gaps, policy makers in Appalachia and elsewhere
have supported a host of initiatives to develop new sources of microcredit and
equity and equity-like capital.
17
Within the region, ARC funded microenterprise
development initiatives and revolving loan funds, as well as others that created a
venture capital industry. At the Federal level, these initiatives include the Small
Business Administration’s (SBA) Microenterprise Loan and PRIME (Program for
12
Ray Collender, et al., “Financial Markets Serve Rural Areas Fairly Well,” Rural Development
Perspectives 14.1 (May 1999):28-35.
13
National Community Reinvestment Coalition, Access to Capital and Credit for Small
Businesses in Appalachia, Washington, D.C.: National Community Reinvestment Coalition, April
2007.
14
Elaine L. Edgcomb and Joyce A. Klein, Opening Doors, Building Ownership: Fulfilling the
Promise of Microenterprise in the United States, Washington, D.C.: FIELD, A Program of the
Aspen Institute, 2005 28 January
2008.
15
Appalachian Regional Commission, Capitalizing on Rural Communities, Washington, DC:
Appalachian Regional Commission, 2000, 8.
16
National Community Reinvestment Coalition, April 2007.
17
Deborah Markley, et al. Rural Equity Capital Initiative Study of Nontraditional Venture Capital
Institutions, RUPRI PB2001-11A-D, 2001,
. An excellent summary of
these efforts can be found in a special issue of the Federal Reserve Bank of San Francisco’s
Community Development Review 3.2 (2006).
9
Investment in Micro-Entrepreneurs) programs, the Small Business Investment
Company (SBIC) program, the New Markets Tax Credit initiative, the CDFI Fund,
and other efforts. State and local initiatives have been more far-reaching and
comprehensive, including Nebraska’s Microenterprise Partnership Fund, the
Oklahoma Center for the Advancement of Science and Technology (OCAST)
and the Pappajohn Entrepreneurship Center at North Iowa Area Community
College (NIACC). Nebraska provides state support for microenterprise
development through a public-private partnership that channels investment to
microenterprise development programs across the state through a competitive
process. OCAST has developed a comprehensive set of financing tools for
technology businesses in the state that includes pre-seed financing of up to
$100,000 for tech start-up companies, a seed capital program with equity
investments up to $750,000, and a number of technical assistance and sector-
specific programs.
18
NIACC’s capital programs are designed to work hand-in-
hand with technical assistance and education programs available at the
community college. Capital programs include a revolving loan fund providing debt
capital, a nanoloan program providing debt capital for microenterprises, and
access to both formal venture capital and angel investors.
19
Finally, private
programs, such as the creation of local angel investor networks, are also being
introduced across the US. Angel investor groups have grown from 50 formal
groups in 1997 to an estimated 170 formal and informal groups in 2002;
20
there
were an estimated 200,000 individual angel investors active in 2002 and 234,000
in 2006.
21
Enabling Culture
The need for an “enabling culture” is widely recognized in the literature, but the
details of what constitutes such a culture are expressed in different ways by
researchers and practitioners. At the most basic level, an enabling culture is
one that understands, recognizes, and honors the importance of local
entrepreneurs. These three terms – understanding, recognizing, and honoring –
also connote three different sets of potential policy interventions.
18
More information is available about OCAST on their website, .
19
More information about NIACC can be found on their website,
and in Deborah Markley and Karen Dabson, Innovative
Approaches to Entrepreneurial Development: Cases from the Northwest Region, RUPRI Center
for Rural Entrepreneurship, 2006, 32-41.
20
“Business Angel Investing Groups Growing in North America,” Ewing Marion Kauffman
Foundation, October 2002, 4 December 2007
;
21
“Full Year 2002 Angel Market Analysis Report,” Center for Venture Research, Whittemore
School of Business and Economics, University of New Hampshire, 4 December 2007
; Full Year
2006 Angel Market Analysis Report,” Center for Venture Research, Whittemore School of
Business and Economics, University of New Hampshire, 4 December 2007
.
10
To promote understanding, analysts recommend the introduction of
entrepreneurship education to all parts of the population. The introduction of
entrepreneurship education at the college and university level has been a
resounding success. In 1979, only 127 schools offered courses in small
business and entrepreneurship. Today, more than 1600 schools offer this
training.
22
Community colleges have also witnessed major growth in
entrepreneurship training. The introduction of entrepreneurship education at the
primary and secondary school levels has been less smooth, but innovative
programs and curricula are widely available across the US.
23
To promote recognition, analysts have advocated for several ideas, including
more active entrepreneur involvement in the policy-making process. Given the
time demands of running a business, few entrepreneurs have stepped up to this
challenge. However, several states have created advisory bodies, such as New
York’s Small Business Advisory Board, to provide opportunities for this input.
The creation of regional business plan competitions is another widely used tool
that helps publicize local entrepreneurs. These competitions exist across the
US, with a scale that can range from major national/international competitions to
smaller local efforts focused on youth or specific market segments. Several of
the ARC’s Entrepreneurship Initiative grants funded efforts of this sort.
To honor entrepreneurs, many analysts recommend the creation of awards
programs such as local Entrepreneur of the Year Awards.
24
ARC’s own
Springboard Awards, designed to honor innovations by entrepreneurship
educators, was a particularly effective design for a regional awards program.
These education and recognition efforts are critical to improving the local climate
for entrepreneurs, but existing cultural attitudes also come into play. Extensive
research indicates that cultural factors play an important role in explaining
differing entrepreneurship rates across countries and even within countries. For
example, Giannetti and Simonov find that the presence of local entrepreneurial
role models helps explain differences in regional entrepreneurship rates.
25
A
recent US study also stresses the importance of role models.
26
Research
sponsored by the Global Entrepreneurship Monitor project notes that differences
22
Jerome Katz, And Another Thing. . .,”2006 Coleman Foundation White Paper on
Entrepreneurship Education, 2006 Annual Meeting of US Association of Small Business and
Entrepreneurship, January 13, 2006, 9 August 2007
.
23
For background, see National Governors Association, A Governor’s Guide to Strengthening
State Entrepreneurship Policy, Washington, DC: National Governors Association, 2004.
24
National Governors Association, 2004.
25
Mariassunta Gianetti and Andrei Simonov, On the Determinants of Entrepreneurial Activity:
Individual Characteristics, Economic Environment, and Social Norms, White Paper, Stockholm
School of Economics, June 2004, 9 August 2007,
.
26
Edward J. Malecki, “Geographical Environments for Entrepreneurship,” International Journal of
Entrepreneurship and Small Business, Forthcoming 2008.
11
in cross-national entrepreneurship rates are influenced by local cultural attitudes
toward risk-taking and fear of failure.
27
Like many rural regions, some
Appalachian communities perform poorly on various measures of cultural
attitudes toward entrepreneurship. Indeed, the ARC’s own materials note that
“the culture of entrepreneurship is neither wide nor deep throughout
Appalachia.”
28
From the perspective of ARC’s Entrepreneurship Initiative, recognition of the
importance of an enabling culture is reflected in the number of program
investments that can be described as “community capacity building” – facilitating
visioning, leadership development, asset-mapping and community engagement
activities intended to make the community more supportive of and attractive to
entrepreneurs. In many cases, initial ARC investments were designed to build
capacity for supporting entrepreneurs, with follow on investments supporting
program implementation. These capacity building investments were designed, in
essence, to enhance community social capital. An extensive literature measuring
the importance of social capital for economic development, and the impact of
enhanced social capital on community development outcomes, has been
developed by researchers, particularly Cornelia Flora.
29
Local Networks
Networks refer to local locations (both virtual and physical locations) where
entrepreneurs can gain access to peers and others with expertise or knowledge
about the processes of starting and growing a business. Entrepreneurs
regularly report that such networks are a critical component in helping them learn
the ins and outs of business and gain easier access to needed support
services.
30
Such networks are commonplace in major urban areas, especially in
technology hot spots such as Silicon Valley or Boston. They are less commonly
found in rural areas as they typically depend on a critical mass of local business
owners with an interest in networking. Because these dense concentrations of
business owners and service providers do not exist in many rural regions,
networking opportunities are often lacking.
27
See, for example, Maria Minniti with William Bygrave and Erkko Autio, Global Entrepreneurship
Monitor 2005 Executive Report,
.
28
Appalachian Regional Commission, Entrepreneurship Initiative: Program Summary and
Approved Projects, September 2003, 9 August 2007
.
29
See for example, Mary Emery and Cornelia Flora, “Spiraling Up: Mapping Community
Transformation with Community Capitals Framework,” Journal of the Community Development
Society 37.1 (Spring 2006):19-35.
30
See, for example, Erik R. Pages and Shari Garmise, “The Power of Entrepreneurial Networks,"
The Economic Development Journal 2.3 (Summer 2003):20-30. For a local example of network
building efforts, see North Carolina Rural Economic Development Center, Hello, My Business
Name is : A Guide to Building Entrepreneurial Networks in North Carolina, Raleigh, NC: North
Carolina Rural Economic Development Center, 2007.
12
Many regional organizations – both public and private – have jumped in to help
seed new networks. By providing staff support and limited funding, economic
developers hope to jump start local networks that can then operate on their own.
The networks can operate in a manner that is open to all entrepreneurs, such as
a regional network, or they can focus on the issues and challenges facing a
specific industry sector or cluster, such as ceramics and related industries.
Supportive Infrastructure
Entrepreneurs are no different from other kinds of business owners in their need
for a supportive and reliable physical infrastructure including transportation, water
and sewer. Building such infrastructure has been a traditional focus of ARC,
through its highway program and other investments. More recently, ARC’s
broadband initiative has invested in providing critical IT-related infrastructure for
the region’s businesses.
ARC-sponsored research clearly indicates the powerful impacts of investments in
traditional infrastructure such as highways, water and sewer facilities, and
industrial parks.
31
Investments in broadband infrastructure can also have
beneficial economic development impacts. Research from Minnesota indicates
that the presence of high speed Internet access was a major factor in explaining
the presence of local gazelle (fast-growing) businesses in regions across the
state.
32
Other research indicates that communities with extensive broadband
access outperform comparable regions without such amenities in terms of job
growth and the number of businesses.
33
Supportive Government Policies
While entrepreneurs may succeed anywhere, they are more likely to flourish in a
region where government and community leaders are “entrepreneur-friendly.”
This can take two forms:
The creation and nurturing of a supportive tax and regulatory climate
The provision of a wide range of private and publicly-funded business
services, including technical assistance, incubators, and other forms of
specialized support.
31
The Brandow Company and Economic Development Research Group, Evaluation of the
Appalachian Regional Commission’s Infrastructure and Public Works Program Projects,
Washington, D.C.: Appalachian Regional Commission, 2000.
32
Thu-Mai Ho-Kim and Ernesto Venegas, “Starting Up Economic Engines: Key Factors for
Growing Successful Start-Ups,” Minnesota Department of Employment and Economic
Development Issue Brief, October 2003.
33
Sharon E. Gillett, et al, Measuring the Economic Impact of Broadband Deployment,
Washington, D.C.: Economic Development Administration, February 28, 2006.
13
An early study by the National Commission on Entrepreneurship documented
federal policy in support of entrepreneurship from 1958-1998.
34
Since that time,
much of the policy innovation surrounding entrepreneurship has occurred at the
state level. For example, in 2004, the Kansas legislature passed a
comprehensive package of legislative support for entrepreneurship development
including the creation of an entrepreneurship center and implementation of a tax
credit program to spur investment in new ventures.
35
As the focus has shifted
from federal to state policy in recent years, the impact analysis of these policies
remains to be done.
THE STATE OF THE FIELD AND THE EI
Much of the research and policy innovation described above has happened in the
years since ARC’s EI was launched. For example, the Global Entrepreneurship
Monitor project was initiated first in 1999 with only 10 countries; GEM 2007
research is being conducted in 42 countries. Most state innovations have
occurred since 2000. ARC was clearly leading the way in encouraging the
practice of entrepreneurship as an economic development strategy. As such,
ARC investments were designed to demonstrate the potential associated with a
wide range of strategies, from youth education to venture capital development.
Given that the entrepreneurship development field was in its early innovation
stage, such an approach, referred to by one stakeholder as “let a thousand
flowers bloom,” was strategic in its design and appropriate for the time.
34
National Commission on Entrepreneurship, American Formula For Growth, Washington, D.C.:
National Commission on Entrepreneurship, October 2002.
35
The Kansas Economic Growth Act of 2004 established the Kansas Center for Entrepreneurship
and introduced several capital initiatives including StartUp Kansas – a seed capital fund,
authorization for regional foundations, and angel networks. A review of other state efforts can be
found in National Governors Association, 2004.
14
CHAPTER 3
CONCEPTUAL FRAMEWORK AND REVIEW OF THE
LITERATURE
Given the breadth and diversity of investment activity undertaken by ARC as part
of the EI, the evaluation team began by reviewing the actual operating realities of
the initiative. The ARC set general guidelines for programs, but each state
responded to these guidelines in a unique manner. Similarly, each project was
quite unique. It is easy to understand that a grant for high school
entrepreneurship summer camps will have different metrics than those of a
revolving loan fund. Yet, great diversity also existed within single program
categories. For example, investments in business incubation might fund a
feasibility study, build a new incubator, or provide technical assistance to new
sets of customers. As will be seen in Chapter 4, building a set of consensus
metrics for such a diverse set of program interventions proved quite challenging.
At the outset of the EI, ARC convened four advisory committees charged with
identifying best practices and key insights into four areas ARC had identified as
critical for building an entrepreneurial region – access to capital,
entrepreneurship education, technical assistance, and sectoral strategies. As an
outgrowth of the work undertaken by these committees, ARC eventually
identified five strategic areas for investment - access to capital and financial
assistance, technical and managerial assistance, technology transfer,
entrepreneurial education and training, and entrepreneurial networks. However,
after an initial review of the portfolio of projects receiving ARC investments from
1997-2005, the evaluation team identified five program categories that captured
the actual range of projects implemented:
Capital Access – provision of services and technical assistance to
connect entrepreneurs and businesses to appropriate levels and types of
debt and equity capital
Entrepreneurship Education – structured experiential opportunities in
and out of school for young people (K-16) to learn entrepreneurial skills
and attributes and to understand business basics with the aim of
encouraging young people to consider entrepreneurship as a career
sooner or later
Incubators – provision of opportunities for acquiring information, skills,
resources within (often subsidized) workspace settings
Technical Assistance and Training – provision of expert and/or peer
one-on-one mentoring and consulting services on technical and
managerial matters and provision of information and skills in formal
classroom or laboratory settings to adult entrepreneurs
Sectors – packaging of some combination of training, incubating,
technical assistance, and capital access services targeted at a single,
specific business sector.
15
Another category, Community Capacity, facilitating visioning, leadership
development, asset-mapping and community engagement activities intended to
make the community more supportive of and attractive to entrepreneurs, was
also identified. This category actually cuts across the other programmatic areas
as initial ARC investments were often designed to build capacity for supporting
entrepreneurs, with follow on investments supporting program implementation.
The program categories defined by the evaluation team differ somewhat from the
five strategic areas originally defined by the Appalachian Regional Commission.
After reviewing actual investments made by the EI, the team determined that
investments had not been made in the technology transfer area, while actual
investments in entrepreneurial networks were more accurately described as
investments in either sector-specific activities or incubation activities, such as
investment in Virginia’s sustainable wood products industry or Ohio’s food sector.
As an example, ARC's investments in commercial kitchen incubators, while
focused on a specific sector, were coded as incubators. Table 3.1 lists ARC’s
strategic areas and describes how and why they were modified and repackaged
into the five program categories used as the basis for this evaluation. All of these
changes were made after a review of the actual projects in which ARC invested
and represent the adaptation of the ARC investment process to the needs of
communities as reflected by their proposed project activities.
Table 3.1: Comparison of ARC Strategic Areas and Evaluation Program
Categories
ARC Strategic Areas Comments
Access to capital and financial assistance Capital Access – no change
Technical and managerial assistance Split into Technical Assistance and Training
and Incubators to reflect different modes of
entrepreneurial support
Technology transfer Not implemented
Entrepreneurial education and training Split between Entrepreneurship Education
for youth and Training (in Technical
Assistance) for adults
Entrepreneurial networks Actually implemented as Sector or Incubator
initiatives
Community Capacity – added to capture the
place-based and cultural dimensions of the
initiative; a category that cuts across program
categories and strategic areas
DEVELOPMENT OF A CONCEPTUAL FRAMEWORK
The origins of the Appalachian Regional Commission’s Entrepreneurship
Initiative were rooted in the challenge for states and communities throughout the
region – a challenge that is shared across rural America – of how to foster the
economic and cultural conditions that give birth to entrepreneurs, support
innovation, and assist in the development and expansion of successful
enterprises. These desired outcomes were, and still are, considered critical to
16
Appalachia’s future economic vitality. Moreover, they were seen as essential to
reducing the region’s dependence upon extractive industries and branch plant
manufacturing, to replacing the long-term practice of asset-stripping with asset
accumulation and value-addition, and to creating a more entrepreneurial and
outward-looking culture.
Given these expectations, the evaluation team chose to approach its charge by
establishing at the outset a conceptual framework that required a sharp focus on
the essence of the initiative and reduced the temptation to explore interesting but
tangential avenues that would perhaps make the evaluation richer but less useful
as a guide to future policy and action. Three fundamental goals were identified
which best reflect the primary purposes of the EI. While the following goals were
not articulated specifically by ARC, they reflect the mission and purpose of the EI
as understood through ARC publications and interviews with key leaders:
More entrepreneurs – To increase the number of entrepreneurs
establishing businesses in Appalachia
Stronger entrepreneurs – To increase the survival rate of entrepreneurial
ventures in Appalachia
More high growth entrepreneurs – To increase the proportion of
entrepreneurial ventures that achieve rapid growth rates, thus providing
jobs and wealth within and increasing the competitiveness of Appalachia.
These fundamental goals then were operationalized by the evaluation team into
six programmatic goals:
More entrepreneurs in the pipeline – increasing the number of people,
youth and adults, who are actively considering setting up their own
businesses
More entrepreneurs staying – creating the conditions in which
entrepreneurs wish to stay and grow their businesses in their community
Better informed entrepreneurs – providing entrepreneurs with the
information and tools they need to establish and grow their businesses
Better skilled entrepreneurs – providing entrepreneurs with the technical
and managerial skills they need to sustain and grow their businesses
More job creating businesses – providing the tools and resources to
encourage entrepreneurs to expand and employ others
Greater business productivity – providing the tools and resources to
enable entrepreneurs to operate efficient and competitive businesses.
While this conceptual framework proved useful as an organizing framework for
the evaluation, the team needed to align this framework with the stated goals and
operational practice of the EI. The program categories that grew out of the
evaluation team’s review of ARC’s investment portfolio map well onto the set of
programmatic goals articulated in the conceptual model. Each category was
identified as contributing to the attainment of at least one of these programmatic
17
goals. Table 3.2 shows how the program categories and programmatic goals
intersect. It is certainly possible to argue for additional Xs in this table. Indeed,
one might argue that all project types could have a long-term impact on every
programmatic goal. However, we have chosen to focus on the primary goal(s) of
each program type. For example, while entrepreneurship education programs
ideally, in the long run, lead to more youth creating businesses and/or staying in
the region, the primary goal of these educational efforts is to expose young
people to entrepreneurial concepts and possibilities, with the ultimate goal of
having more youth pursue entrepreneurship as a career path, leading to more
entrepreneurs in the pipeline.
The conceptual framework developed to provide context for the ARC
Entrepreneurship Initiative and to ground the evaluation was also used to inform
the review of the literature related to entrepreneurship development.
REVIEW OF THE LITERATURE
Entrepreneurship and efforts to promote entrepreneurship as an economic
development strategy have emerged as important topics for both research and
policy discussion. The literature is vast and seems to be growing larger every
day. The majority of this research focuses on questions directly related to
entrepreneurs and their companies by examining questions such as appropriate
business development strategies, the qualities of successful entrepreneurs, or
the role of various management practices in producing business success or
failure. Very few studies address the measurement of program impacts or
community outcomes. This section presents a review of both current trends in
economic development performance measurement and of the limited evaluation
literature.
Evaluation Research by Program Category and Objective
The following sections summarize recent research and thinking about
performance measurement in the five ARC program evaluation categories:
Capital Access, Sectors, Entrepreneurship Education, Business Incubation, and
Technical Assistance and Training. Appendix A contains a more complete review
of the literature. As Table 3.2 indicates, each of the five program categories
seeks to achieve multiple goals. Yet, many of the efforts can be categorized
according to a predominant objective. For example, entrepreneurship education,
with its heavy emphasis on youth empowerment, is most concerned with the
objective of creating more entrepreneurs. Meanwhile, capital access programs,
especially those equity programs emphasized by the ARC, are largely focused
on creating more high-growth entrepreneurs.
1
8
T
a
b
l
e
3
.
2
:
I
n
t
e
r
s
e
c
t
i
o
n
o
f
P
r
o
g
r
a
m
C
a
t
e
g
o
r
i
e
s
a
n
d
P
r
o
g
r
a
m
m
a
t
i
c
G
o
a
l
s
C
a
p
i
t
a
l
A
c
c
e
s
s
E
n
t
r
e
p
r
e
n
e
u
r
i
a
l
E
d
u
c
a
t
i
o
n
I
n
c
u
b
a
t
o
r
s
S
e
c
t
o
r
s
T
e
c
h
n
i
c
a
l
A
s
s
i
s
t
a
n
c
e
a
n
d
T
r
a
i
n
i
n
g
C
o
m
m
u
n
i
t
y
C
a
p
a
c
i
t
y
M
o
r
e
e
n
t
r
e
p
r
e
n
e
u
r
s
i
n
p
i
p
e
l
i
n
e
X
X
X
M
o
r
e
e
n
t
r
e
p
r
e
n
e
u
r
s
s
t
a
y
i
n
g
X
X
X
X
B
e
t
t
e
r
i
n
f
o
r
m
e
d
e
n
t
r
e
p
r
e
n
e
u
r
s
X
X
X
X
B
e
t
t
e
r
s
k
i
l
l
e
d
e
n
t
r
e
p
r
e
n
e
u
r
s
X
X
X
X
M
o
r
e
j
o
b
c
r
e
a
t
i
n
g
b
u
s
i
n
e
s
s
e
s
X
X
X
X
G
r
e
a
t
e
r
b
u
s
i
n
e
s
s
p
r
o
d
u
c
t
i
v
i
t
y
X
X
X
X
X
19
Creating More Entrepreneurs
Within ARC’s program categories, entrepreneurship education is most concerned
with feeding the pipeline with new entrepreneurs – exposing youth to
entrepreneurship as a potential career path and to entrepreneurial ways of
thinking that can be applied to working for oneself or for someone else. An
ultimate or long-term goal is to have these new entrepreneurs create new
enterprises that remain in the region and produce jobs and wealth.
Entrepreneurship Education. Evaluations of entrepreneurship education
efforts vary across different levels of the educational system. For programs
targeting youth, a host of different evaluation methodologies have been
deployed. In general, most of these measures stress student performance and
outcomes as opposed to community outcomes. The programs operate on an
implicit assumption that by empowering and providing skills to young people
(especially at-risk youth), positive community outcomes will emerge over the
long-term.
Studies of youth entrepreneurship programs indicate that these efforts have a
strong impact on program participants. While some programs, including ARC-
funded efforts, provide training for primary school students, most existing
programs targeted middle and high school students. Studies sponsored by the
National Foundation for Teaching Entrepreneurship (NFTE) find that NFTE
program participants had more interest in attending college and had more
ambitious career aspirations.
36
A study of the EnterprisePrep curriculum used in
Philadelphia found that program participants had lower drop out rates and
improved performance in science, math and English. Studies of the Junior
Achievement curriculum identify similar positive outcomes in terms of youth
attitudes toward entrepreneurship.
37
Evaluations of entrepreneurship training at the college and graduate level place
more emphasis on outcomes in terms of new business formation. A detailed
study of the University of Arizona’s Berger Entrepreneurship program found that
program graduates were three times more likely to start a business than their
student counterparts. Program graduates also enjoyed higher average
incomes.
38
A whole host of other studies provide similar results.
39
36
Michael Nakkula, Expanded Explorations into the Psychology of Entrepreneurship: Findings
form the 2001-2002 Study of NFTE in two Boston Public High Schools, Working Paper, Harvard
University Graduate School of Education, 2003.
37
Junior Achievement, The Impact on Students of Participation in Junior Achievement: Selected
Cumulative and Longitudinal Findings, Monograph, January 26, 2004.
38
Alberta Charney and Gary Libecap, The Impact of Entrepreneurship Education: An Evaluation
of the Berger Entrepreneurship Program at the University of Arizona, 1985-1999, Final Report to
the Kauffman Center for Entrepreneurial Leadership, 2000.
39
Lena Lee, Entrepreneurship Education: A Compendium of Related Issues, Working Paper,
NUS Entrepreneurship Centre, National University of Singapore, 2005.
20
In general, entrepreneurship education programs are not expected to provide
direct community impacts in terms of new job or wealth creation. Instead, they
help produce intermediate outcomes, which may then translate into positive
economic development outcomes. Analysts contend that entrepreneurship
education efforts change attitudes toward entrepreneurship, increase awareness
of key business concepts, and build necessary skills for starting and operating a
business. These claims are generally confirmed in surveys of participants in
major programs. These attitudinal and skill changes may lead to new business
starts, which are then expected to generate positive community outcomes.
Based on this logic chain, most entrepreneurship education programs emphasize
program output and student performance measures. While most programs do
track new business starts by students, they place greater emphasis on more
short-term measures of student performance and attitudinal change. In contrast,
many technical assistance programs, such as trainings sponsored by Small
Business Development Center programs, are evaluated according to their
capacity to generate new business starts.
Creating Stronger Entrepreneurs
Three of the ARC’s program interventions – technical assistance and training,
business incubation, and sectors/networks – were primarily concerned with
supporting the mission of stronger entrepreneurs, i.e. to increase the survival rate
of local entrepreneurial ventures. These three initiatives operate according to
similar rationales. They seek to create more skilled and better informed
entrepreneurs through the provision of:
New information (via trainings and workshops)
Access to mentors and peer support (via networks)
Access to subsidized facilities or equipment (via incubators)
Access to potential new markets (via networks or training).
The research literature for each of the program categories emphasizes different
evaluation methodologies, but they all share some common characteristics.
First, they seek to assess general customer satisfaction with the provided
assistance. Second, they seek to assess whether this support led to improved
knowledge or skills for the entrepreneur. Finally, they seek to assess whether
this new knowledge has led to changes in behavior; specifically, has the
intervention led to improved performance by the company or its management
team?
Technical and Managerial Assistance. Since business incubators often
provide technical and managerial assistance as part of their program operations,
incubators and technical assistance often use similar measurement tools and
methodologies. Efforts to track and measure the effects of business assistance
programs have been underway for a long time. Many Federal programs, such as
21
the Small Business Development Center program and the Manufacturing
Extension Partnerships, have invested significant resources into evaluation
efforts.
Most technical assistance evaluations begin with calculations of traditional
economic development impacts such as job creation or the leveraging of outside
investments. As noted earlier, many state-level programs have become quite
sophisticated in their evaluation efforts. For example, Pennsylvania’s Ben
Franklin Technology Partners (BFTP) tracks its customers by company size and
sector; they also compare their client companies to state averages in terms of
average wages. This latter metric offers a useful measure of job quality. BFTP
tracks the public return on investment and impact on state gross product via
calculations of job creation, new investment, and newly generated tax
revenues.
40
BFTP is also one of the few programs to utilize a control group
methodology in its assessments. This costly but effective method compares the
performance of program clients to comparable firms who did not utilize BFTP
services. This comparison indicates that BFTP-supported firms employed three
more people in each year after the program investment.
In addition to basic measures of job creation and customer satisfaction, the
Oklahoma Center for the Advancement of Science and Technology (OCAST)
tracks outside research investments, Small Business Innovation Research
(SBIR) funds per capita (compared to national benchmarks), and company
financing by stage. OCAST’s partner, I2E, utilizes some other specialized
measures such as growth in number of companies positioned for financing,
creation of commercialization road maps for customers, and a host of activity
measures, such as creation of new partnerships, number of events, and so on.
Both OCAST and I2E also provide breakouts (via pie charts) of customers by
industrial sector and by region.
41
The Maine Technology Institute (MTI) utilizes university researchers to assess its
program operations and the impact of its grants.
42
The MTI’s performance has
been tracked using four categories of measures: economic impact, effects on
company finances, intellectual property development, and relationships. Within
these categories, several unique measures are used. These include sources of
material and service inputs (used to assess in-state purchasing) and
relationships. This latter measure tracks usage of other service providers, such
as SBDCs, and measures related to customer satisfaction and impact.
40
Nexus Associates, A Continuing Record of Achievement: The Economic Impact of the Ben
Franklin Technology Partners, Final Report, March 2003.
41
Oklahoma Center for the Advancement of Science and Technology, Impact Report 2006
(Oklahoma City, Oklahoma: OCAST, January 2006) 10 November 2006
.
42
Charles S. Colgan and Bruce Andrews, Evaluation of Maine Technology Institute Programs,
University of Southern Maine Center for Business and Economic Research, December 2004, 10
November 2006,
.
22
Business Incubation. Because business incubation has been a core economic
development strategy for several decades, both practitioners and analysts have
developed an array of metrics and tools for measuring the effectiveness of
business incubation programs. The National Business Incubation Association
(NBIA) has led many of these efforts and was an important ARC partner during
this initiative. In addition, ARC has funded incubator programs throughout its
existence and has funded several useful studies of the field.
43
Most studies of business incubation provide reviews of best practices that
typically cover the details of program management and facility operations.
44
Many of these reports also include suggestions for assessing the impact of
business incubation programs. Lichtenstein and Lyons place a heavy emphasis
on effective evaluations that capture both process and outcome measures.
45
Most analysts recommend that traditional economic development metrics, such
as job creation, be supplemented with other measures that capture unique
aspects of the business incubation process. For example, most business
incubators regularly track graduated companies, i.e., firms that have moved from
subsidized incubator space to owning or leasing space at market rates.
Customer satisfaction surveys are also frequently used. Business performance
measures – both during and after residence in the incubator – also offer useful
data on an incubator’s regional impacts. For example, a firm’s post-graduation
ability to continue growth, to access outside financing, and to enter new markets
are all important measures of community economic impacts from business
incubation.
Sectors/Networks. Economic developers’ thinking about sector/network
strategies has undergone an interesting evolution over the past twenty years.
Beginning in the 1980s and 1990s, a number of fledgling programs sought to
stimulate the development of sector-based networks in industries such as wood
products and manufacturing. Much of this initial work was based on successful
experiments in Denmark and Italy, with overseas lessons applied to US
experience. Several of the first such US-based programs were located in
Appalachia. The North Carolina Rural Economic Development Center supported
a networks initiative in Western North Carolina, and ACEnet sponsored a similar
effort in Appalachian Ohio.
43
For example, Greenwood Consulting Group, A Survey of Business Incubators in Appalachia
(Washington, DC: ARC, July 2005) 10 November 2006
.
44
See, for example, Louis G. Tornatzky et al., Incubating Technology Businesses: A National
Benchmarking Study (Athens, Ohio: National Business Incubation Association, 2003).
45
Gregg A. Lichtenstein and Thomas S. Lyons, Incubating New Enterprises: A Guide to
Successful Practice (Washington, DC: Aspen Institute, 1996).
23
These initial network building efforts remained limited to networks based on an
industrial cluster or sector.
46
More recently, policy makers have sought to
support the creation of broader entrepreneurial networks that include participants
from a variety of sectors and disciplines. ARC has invested in both types of
networks via the Entrepreneurship Initiative. For example, the Team
Pennsylvania Foundation sought to use ARC funds to stimulate the creation of
several regional entrepreneurship networks across the state. Other ARC
investments sought to build sector-based networks in ceramics (New York), arts
(Ohio), and aquaculture (Georgia).
While the operations of cluster-based and entrepreneurial networks may differ
slightly, both types are evaluated using a similar methodology. Evaluators
regularly seek hard quantitative data on the impact of networks, but qualitative
measures are also necessary as much of the network impact is qualitative.
Relationship building, trust, and increased knowledge are all important outcomes
of network activities. In general, research strongly indicates that more networked
firms tend to perform better than firms with weak networks and limited strategic
alliances.
47
Networks are typically evaluated using customer surveys, interviews, and case
studies.
48
Typical qualitative outcomes would be high rates of customer
satisfaction, better awareness of resources and networks, and more openness
toward collaboration. Quantitative measures examine the network’s ability to
help produce changes in business outcomes, such as entry into new markets,
revenue growth, job creation, and the ability to access outside financing.
As networking becomes a more important part of a typical economic
development portfolio, analysts are searching for new tools that can better
assess the role of business support providers as network builders. Social
network analysis seeks to map and evaluate the power of networking activities.
In the 1980s and 1990s, researchers sought to simply count the number of
outside alliances and assess their strength. Today, new software tools allow
users to map social networks and assess them according to their diversity,
strength, and influence. These maps can be used to diagnose network
46
For background, see Stuart A. Rosenfeld, “Networks and Clusters: The Yin and Yang of Rural
Development,” Proceedings of Federal Reserve Bank of Kansas City Conference on Exploring
Policy Options for a New Rural America, (September 2001):103-120.
47
An excellent review of this literature can be found in Luke Pittaway, Maxine Robertson, Kamal
Munir, and David Denyer, Networking and Innovation: A Systematic Review of the Evidence,
University of Lancaster Institute for Entrepreneurship and Enterprise Development, Working
Paper 016, 2004, 10 November 2006 . See also Christopher T. Street
and Ann-Frances Cameron, “External Relationships and Small Business: A Review of Small
Business Alliance and Network Research,” Journal of Small Business Management 45.2
(2007):239-266.
48
For example, see Peter Witt, “Entrepreneurs’ Networks and the Success of Start-Ups,”
Entrepreneurship and Regional Development 16 (2004):391-412; Philip Shapira, The Evaluation
of USNet: Overview of Methods, Results and Implications – Final Report, Georgia Tech Policy
Project on Industrial Modernization Working Paper 9805, 1998.
24
weaknesses and design interventions to strengthen the network and the quality
of company network ties.
49
Social network analysis has also been used as a
means to better describe and understand company and industry supply chains.
Creating More High-Growth Entrepreneurs
While all program interventions share the goal of creating high-growth
entrepreneurs, very few of the Entrepreneurship Initiative’s grantees focused
exclusively on this objective. However, a number of grant recipients in the
capital category did embrace a mission of supporting gazelle businesses.
Because its metrics rely on quantifiable financial measures, evaluations of capital
access programs tend to be much more rigorous than other types of
entrepreneurship-related policy interventions.
Capital. ARC’s capital projects tend to fall into two broad categories – support for
microenterprise initiatives and investments in more specialized types of financial
assistance, such as specialized loan funds or new sources of equity capital.
Many of these investment vehicles promote what they refer to as a “double
bottom line.” In other words, the funds seek to build strong businesses but also
promote other social goals, such as community development or environmental
sustainability. In addition, microenterprise programs measure progress along
two fronts – traditional business outcomes and measures that capture the
empowerment of individual clients. These latter metrics could include movement
off of welfare, increases in family income, and length of self-employment periods.
Since ARC’s Entrepreneurship Initiative investments have focused on regional
economic development, this literature review is concerned primarily with how to
measure the economic impacts of investments in capital programs. Since the EI
first began in 1997, capacity to measure these impacts has greatly improved. At
that time, policy makers had very limited experience in creating and operating
publicly-sponsored seed or equity capital programs. And, many early programs
were shut down due to political controversies.
50
The intervening decade has been one of great experimentation and innovation in
the development of new capital access initiatives and associated evaluation
tools. Since 1997, the New Markets Venture Capital Program, the Community
Development Financial Institutions program, and revisions to the Small Business
Investment Company (SBIC) program have all been put into place. In addition,
several new trade associations, including the Community Development Venture
Capital Association (CDVCA), the National Association of Seed and Venture
49
For example, see .
50
An example is Mississippi’s Magnolia Fund which operated for only 2 ½ years before being
closed due to misappropriation of funds. David L. Barkley, et al., Establishing Nontraditional
Venture Capital Institutions: Lessons Learned, Rural Equity Capital Initiative Study of
Nontraditional Venture Capital Institutions, RUPRI PB2001-11A, 2001,
.
25
Funds, and the Angel Capital Association, have all been established to help
professionalize the field.
Meanwhile, more established disciplines, like microenterprise and community
development corporations, have become much more rigorous in strengthening
professional development programs and creating common performance
measurement systems. In the microenterprise field, the Aspen Institute’s FIELD
(Microenterprise Fund for Innovation, Effectiveness, Learning and Dissemination)
program has played a critical role in collecting, developing and disseminating
new program ideas and new performance measures and tools.
51
Its MicroTest
program provides a useful framework for assessing both program performance
and client outcomes. Similar comprehensive efforts are underway at leading
organizations in the field. These include the CDFI Data Project, the Opportunity
Finance Network’s CDFI Assessment and Rating System (CARS), and the
CDVCA’s Return on Investment Project.
52
All of these efforts, and other outside analysis, reach a similar conclusion –
publicly-sponsored investment programs need to be managed and measured just
like private investments. While programs may pursue multiple goals, the
programs must be managed and assessed on their capacity to make good
business decisions. As a 2000 National Governor’s Association guide put it,
“The best programs are not afraid to make money.”
53
This same study noted: “In
the best cases, state leaders take the initiative in getting programs launched and
setting long-term direction. They rely on experienced, private-sector managers to
make the day-to-day investment decisions.”
54
As seed and equity capital programs have moved in this direction, issues of
performance measurement have become more straightforward. At the most
basic level, most funds should be expected to produce a reasonable rate of
return. A reasonable rate will differ depending on the types of companies in a
fund’s portfolio. For example, institutional venture capital funds have averaged a
20.3% annual return over the past ten years, but only a 1% return over the past
five years. A 2002 study of venture capital rates of return in five countries
identified average rates of return that fell anywhere between 26% and 45%.
55
Meanwhile, the CDVCA estimates that its members have enjoyed an average
51
An excellent review of effects of microenterprise programs can be found in Signe-Mary
McKernan and Henry Chen, Small Business and Microenterprise as an Opportunity and Asset-
Building Strategy, Urban Institute Issue Brief No. 3, June 2005.
52
Community Development Venture Capital Association, Measuring Impacts Toolkit (New York:
Community Development Venture Capital Alliance, 2005).
53
Robert Heard and John Sibert, Growing New Businesses with Seed and Venture Capital: State
Experiences and Options (Washington, DC: National Governors Association, 2000) 18.
54
Heard and Sibert, 17.
55
Sophie Manigart, et al, “Determinants of Required Return in Venture Capital Investments: A
Five Country Study,” Journal of Business Venturing 16.6 (July 2002):291-312.
26
annual return of 15.5% over a twenty-five year period.
56
Programs that invest in
microenterprises or slower growth businesses should be expected to post rates
of return much lower than these benchmarks.
In addition to rates of return, funds can also be assessed in terms of program
outputs and economic development outcomes. Program outputs refer directly to
the activities of the fund or its related programs. They are activity measures that
track data such as the number and amount of loans, average loan size or
number/amount of financing provided to certain target customer sets.
Outcome measures seek to assess a fund’s business and community impacts.
In this case, most analysts recognize that many traditional economic
development metrics do a good job of measuring community outcomes. Thus,
most studies continue to recommend tracking job creation and retention,
business performance of portfolio companies, and the leveraging of outside
investments.
IMPLICATIONS FOR PERFORMANCE MEASUREMENT
The conceptual framework presented above seeks to portray entrepreneurial
development as an array of programs that serve entrepreneurs at various points
in the lifecycle of their business. This “pipeline” model of entrepreneurial
development, first described by Lichtenstein and Lyons,
57
recommends multiple
policy interventions that help achieve the three broad purposes of creating more
entrepreneurs, stronger entrepreneurs, and more high-growth entrepreneurs.
In practice, few program managers have the scope to manage programs that
cover the whole pipeline of entrepreneurial development. Instead, they typically
manage a single program or a single type of policy intervention, such as training
or business financing. This restricted span of control over business outcomes
makes it difficult for individual programs to introduce more sophisticated tools for
measuring program performance.
In the field, entrepreneurship program managers are beginning to consider
alternative performance measurement systems, but they still feel pressured by
funders and elected officials to utilize traditional measures of job and firm
creation outcomes.
58
By using traditional economic development metrics to
56
Amy Simpkins, “Community Development Venture Capital: Producing Results for
Entrepreneurs, Investors, and Communities,” Bridges Summer 2006, 10 November 2006
.
57
Gregg A. Lichtenstein and Thomas S. Lyons, “The Entrepreneurial Development System:
Transforming Business Talent and Community Economies,” Economic Development Quarterly
15.1 (2001):3-20.
58
Laura Czohara and Julia Melkers, Performance Measurement in State Economic Development
Agencies: Lessons and Next Steps for GDITT, Georgia State University Fiscal Research Center
Report No. 92, February 2004. See also, Erik R. Pages and Kenneth A. Poole,
“Entrepreneurship Promotion as an Economic Development Strategy: Next Steps in
27
assess entrepreneurial and innovation-based development, program managers
are employing inappropriate performance measures. Building businesses takes
patience and resilience. It is unrealistic to expect quick results in terms of
traditional economic development outcomes. Program managers need a more
thoughtful approach to tracking the performance of their efforts. Moreover, new
measures for entrepreneurial development need to be devised that recognize a
stream of benefits over an extended period of time. Measuring entrepreneurial
development using annual job creation impacts alone is like measuring the
success of a loan program solely by the ability of the borrower to repay on a
monthly basis. Short-term job creation is simply not the purpose of these
programs.
In their quest to “tell a better story,” program managers are considering a host of
other metrics, which tend to fall into the following categories:
Activity/Output Measures (e.g., number of customers served)
Customer Satisfaction Surveys
Input Measures (e.g., increase in budget)
Outcome Measures (e.g., increase in business starts)
Cost Efficiency (e.g., return on investment).
While some programs use a full range of measures, most economic developers,
including ARC grant recipients, use a more limited menu of metrics that is
generally limited to job creation and retention and the leveraging of outside
investments. These limited metrics can provide a much skewed picture of the
impact of entrepreneurial development efforts.
Economic development programs have responded to this challenge in a number
of ways. The metrics used by Small Business Development Centers (SBDCs)
offer one model. For example, the North Carolina Small Business and
Technology Development Centers track outside financing (loans, equity, and
SBIR/Small Business Technology Transfer Program funds), state and federal
contract awards, customer satisfaction, job creation, firm sales growth, and
incremental taxes generated.
59
Florida combines capital formation, business
start-ups, jobs created/retained, sales growth, and contract awards to calculate
the return on investment (ROI) for its SBDC system.
60
A slightly more
sophisticated set of metrics is used by various statewide technology development
organizations such as Pennsylvania’s Ben Franklin Partners, the Oklahoma
Center for the Advancement of Science and Technology, and the Maine
Technology Institute.
Institutionalizing the Field,” Applied Research in Economic Development 3.2 (December
2006):.10-27.
59
North Carolina Small Business and Technology Development Center, 2005 Annual Report,
.
60
See .
28
Useful benchmarks for metrics can also be found in the work of various trade
associations working in the fields of microenterprise and community
development. These groups are seeking to develop industry-wide metrics that
can help improve management practices and provide industry benchmarks for
effective practice.
Local groups are also experimenting with new methods. For example, Maine’s
Coastal Enterprises, Inc. has developed a very rigorous and comprehensive set
of program measures.
61
Its Social Information System combines a host of
measures that provide internal feedback to management and employees, permit
assessment of program outcomes, and also generate data and case studies that
can be communicated to an outside audience.
These varied initiatives seek similar types of information on client companies.
They ask for information on firm growth, performance, and local economic
impact. In keeping with their social missions, the organizations also tend to track
information on employee benefits, wages, and community and environmental
impacts.
At the federal level, the US Department of Agriculture’s Rural Business-
Cooperative Services (RBS) is implementing a Socio-Economic Benefits
Assessment System (SEBAS) developed at the University of Missouri-
Columbia.
62
SEBAS provides a means of evaluating performance and
effectiveness of RBS’ loan and grant programs by measuring the economic and
social impacts that these have on rural community environments. Using a multi-
regional social accounting matrix model, SEBAS is able to measure both direct
and indirect effects of the loans and grants, such as business sales, income,
indirect business taxes, employment, household income, public revenues, and
distribution of household income and occupations.
Finally, many organizations are seeking new ways to measure “innovation
impacts.” This work remains relevant to the existing literature on
entrepreneurship as many entrepreneurial development programs seek to
stimulate innovation-related outcomes, such as improved productivity rates,
wider diffusion of new technologies, and improvements in human capital.
63
The
US Department of Commerce has created a new “Measuring Innovation in the
21
st
Century Economy Advisory Committee.” Commerce’s Manufacturing
Extension Partnerships and the Advanced Technology Program have probably
faced the most rigorous scrutiny of any publicly-funded technology program in
the country. The National Institutes of Standards and Technology (NIST) have a
61
Coastal Enterprises, Inc., Measuring Impact in Practice (Wiscasset, ME: Coastal Enterprises
Inc., February 2006).
62
Robinson, Dennis and Zuoming Liu, A User Guide to the Socio-Economic Benefits Assessment
System: A Rural Business-Cooperative Services Assessment Tool for Economic Development.
Community Policy Analysis Center, University of Missouri-Columbia, December 2004.
63
B.K. Atrostic, “Measuring U.S. Innovative Activity,” US Census Bureau Center for Economic
Studies Working Paper (07-11), March 2007.
29
huge library of resources on this work.
64
Studies of the MEP have generally
used industry surveys to assess whether client firms have introduced new
processes (such as Total Quality Management or a reconfigured plant layout) as
a result of MEP’s technical assistance.
65
Overseas, the European Union, the OECD, and many national and regional
programs are developing interesting new ways to measure progress in innovation
policy. They have also made major efforts to go beyond studies and to get
practitioners to use these tools in the field. For example, the OECD has
produced the very detailed Oslo Manual for collecting and interpreting innovation
data. The European Union has its own PAXIS Manual that profiles hundreds of
effective measurement tools and practices. Many national governments are also
doing good work in this area. Britain’s Department of Trade and Industry has
recently published a useful study of UK innovation indicators.
66
These efforts all
share a commitment to regular comprehensive performance measurements that
capture both program outputs as well as community outcomes.
These efforts generally propose the use of detailed company surveys to capture
data on a wider range of variables related to innovation. In addition to traditional
measures such as new patents or licenses, the new survey tools ask questions
about education backgrounds of new hires and existing personnel, development
of new R&D projects, joint ventures, training expenditures, and the deployment of
“high-impact” human resource practices (e.g., equity sharing, team building).
The UK study captures some interesting innovation information via firm
surveys.
67
These data are organized around three categories, with several sub-
measures under each category:
Product-Orientated Effects: increased range of goods or services; entry
into new markets; improved quality of goods and services
Process-Orientated Effects: improved flexibility of production or service
provision; increased capacity for production or service provision; reduced
unit costs
Other Effects: reduced environmental impact or improved health/safety;
met regulatory requirements; improved value added.
64
These reports are available at .
65
See, for example, Eric Oldsman, “Do Manufacturing Extension Programs Matter?” Research
Policy 25.2 (March 1996):215-232.
66
European Commission, Directorate General, Enterprise and Industry, The PAXIS Manual for
Innovation Policy Makers and Practitioners (Brussels: European Commission, 2006);
Organization for Economic Cooperation and Development, Oslo Manual: Guidelines for Collecting
and Interpreting Innovation Data Third Edition (Paris: Organization for Economic Cooperation and
Development, 2005); United Kingdom Department of Trade and Industry, Innovation in the UK:
Indicators and Insights, DTI Occasional Paper No. 6, July 2006.
67
United Kingdom Department of Trade and Industry (2006).
30
These surveys also ask companies to comment on innovation constraints they
face in their region or industry. These data help program managers anticipate
future needs or service requirements.
Drawing on this review of literature and insights into performance measurement
gained, the evaluation team developed a metrics framework to guide the
evaluation, particularly the data collection for the sample of ARC projects.
Chapter 4 describes the metrics framework and the overall approach to the
evaluation.
31
CHAPTER 4
APPROACH TO THE EVALUATION
In designing the approach to this evaluation, the team was conscious of the
charge articulated by ARC in the Request for Proposals for the evaluation.
ARC was seeking a “policy impact and program evaluation”
68
of the
Entrepreneurship Initiative that would include:
An examination of the project outcomes of a sample of projects closed
since 1997
An assessment of wider policy impacts informed by input from
stakeholders within the region.
Through its Entrepreneurship Initiative, the ARC has invested in a diverse set of
projects in a diverse set of communities. Thus, a “one-size-fits-all” approach
would not produce an effective and comprehensive evaluation. The evaluation
team’s approach to this project was to combine a detailed and rigorous review of
a sample set of projects with extensive interviews of key players on the project
teams and in the targeted communities. This approach provided us with a strong
set of collective program metrics as well as rich detailed case-study-like
information on key projects and their outcomes.
The ideal evaluation would measure project outcomes relative to the sample
programs’ ultimate goals.
69
The ARC evaluation is complicated by the fact that
individual projects (those receiving ARC investments) had a set of specific goals
each was trying to achieve, i.e., “local” impacts, such as increasing access to
equity capital or incorporating entrepreneurship education into high school
classrooms. In addition, through investments made in the totality of EI projects,
ARC was trying to provide communities with the tools they needed to support
homegrown entrepreneurs because they “play an important role in creating self-
sustaining local economies and improving the quality of life in Appalachia”
70
– a
broader set of regional impacts. While “local” and regional impacts were
complementary in most cases, individual projects were evaluated based on their
ability to achieve locally articulated goals and not these broader regional impacts.
To distinguish between these two levels of impact, the evaluation team chose to
couple an assessment of sample projects’ outcomes, as measured by data
reported by project leaders, with identification of wider policy impacts as reported
68
Request for Proposals for A Program Evaluation of ARC’s Entrepreneurship Initiative,
Appalachian Regional Commission, July 28, 2006.
69
David J. Storey, Six Steps to Heaven: Evaluating the Impact of Public Policies to Support Small
Businesses in Developed Economies, Working paper No. 59, Small and Medium Sized Enterprise
Centre, Warwick Business School, University of Warwick, September 1998, 3 December 2007
.
70
Appalachian Regional Commission, Entrepreneurship Initiative, 9 August 2006
.
32
by key informants – both project leaders and regional stakeholders. A list of non-
project stakeholders is included as Appendix B.
Program outcomes for the EI investments can be measured in two ways. One is
to compare the levels of outcome measures before and after the ARC
investment. To be accurate, one must be sure that the metric or outcome
measure would not have changed without the ARC-funded program (i.e., the “but
for” criterion). Alternatively, one could conduct a with and without comparison,
estimating what the outcome would be without ARC investment and comparing
this measure to the outcome with ARC investment. To be accurate using this
strategy, one would need to employ a control group of communities, identical in
every way to the communities that housed an ARC-funded program. While
employing a control group may make sense when looking at firm-level impacts of
public policy, as suggested by Storey, it is more difficult to consider identifying
control groups for public policies aimed at bringing about changes in firms,
students, educators and even communities. As a result, the with and without
approach was deemed impractical for this evaluation as it would require
identifying at least 88 “control” places – some individual communities, some
multi-county regions, some multi-state regions – and collecting comparable data
through interviews and secondary data.
Given the scope of the evaluation as articulated by ARC, this latter approach was
not used in this evaluation. While neither approach is without criticism (see
caveats at the end of this chapter), the evaluation team chose the before and
after strategy and relied on those administering the programs and respondents to
stakeholder interviews to indicate if the “but for” criterion was satisfied.
RESEARCH QUESTIONS
A number of research questions guided this evaluation effort.
Have sample projects achieved their stated project objectives?
How does the performance of ARC projects compare to the performance
of similar types of projects in other regions or countries?
How well do existing performance metrics capture the impact of ARC
projects and what additional metrics would improve the usefulness and
integrity of evaluation results?
What broader policy impacts are associated with ARC project
investments?
What innovative practices or lessons learned have relevance for other
national and international economic development efforts?
ARC’s Entrepreneurship Initiative is unique in that investments have been made
in a diverse project portfolio, varying widely by geography, program type (i.e.,
access to capital, technical assistance and training, incubators, entrepreneurship
education, and sectors), type of lead institution and identified output and outcome
33
measures. As such, no single performance measurement or metric would
adequately capture the impact of ARC entrepreneurship investments on the
region. For example, the performance measures associated with a youth
entrepreneurship education project are likely to vary significantly from those
associated with a community development venture capital project. The evaluation
methodology recognized this uniqueness and created a rigorous and broad set of
project metrics that would demonstrate the impact of a wide range of project
types on the region.
METRICS FRAMEWORK
The review of the literature on the state of the art in evaluating entrepreneurial
development programs yields one important conclusion: the field has a long way
to go in terms of creating rigorous, compelling, and effective techniques and
strategies for evaluating programs and communicating their effectiveness to
policy makers and community stakeholders. Indeed, much of the literature
reflects what Storey would describe as “monitoring” as opposed to rigorous
evaluation of performance as compared to the objectives established for a
program or investment.
71
Drawing on the lessons from this review and based on
the conceptual framework developed to guide the evaluation process (described
in Chapter 3), performance measures were defined for each of the program
categories that, as far as is practical from the available data, reflect the
appropriate programmatic goals. The evaluation team developed a list of metrics
that captures key outputs and outcomes for each of the five program categories.
This framework is depicted in Table 4.1.
The development of this metrics framework was guided by the literature review
and a critical discussion of the need to link outcomes to program goals. For
example, while the literature on entrepreneurship education offers a range of
metrics related to student outcomes (e.g., higher test scores), the evaluation
team chose to focus on those metrics that relate directly to the programmatic
goals of the ARC Entrepreneurship Initiative – in the case of entrepreneurship
education, the goal is to create more entrepreneurs in the pipeline.
These metrics allow researchers to obtain a good picture of the progress of the
various projects funded through the ARC’s Entrepreneurship Initiative. They
allow an understanding of the impacts of each program type, and also help
identify exemplary practices and programs. They do not yet give a good picture
of the overall effects of varied entrepreneurship initiatives across a specific
region. Achieving this objective requires that researchers assess programs
operated by numerous different organizations (many outside of ARC’s purview)
across all types of policy interventions along the business life cycle. This
important effort, which falls outside scope of this evaluation, presents an
interesting challenge for future research.
71
Storey (1998):12.
34
TABLE 4.1. PROPOSED METRICS FRAMEWORK
TYPE OF PROJECT OUTPUT INDICATORS OUTCOME INDICATORS
Capital Access Number of loans/year
Amount ($)/year
Number of funds created
Fund size ($)
Average loan size
Percent sectoral distribution
of loans ($)
Amount ($) funds leveraged
(public, private or other)
Number of jobs (FTEs)
created/retained
Percent of funded firms still in
business
Annual income and benefits/job or
average wage/job
Sectors Number of participants in
networking meetings
Number of members
(change over time)
Increase in inter-firm
collaborations
Number of partnerships created
Amount ($) of increased sales
from network participation
Number of jobs (FTEs)
created/retained
Change in total sector sales
Number of business starts in
targeted sector
Incubators
Number of current clients
Number of clients served
Number of graduated firms
Number of clients still in
business
Amount ($) leveraged by
incubator (other
public/private money)
% businesses retained in service
area
# of jobs (FTEs) created/retained
while in incubator
# of jobs (FTEs) created/retained
post-graduation
Amount ($) of capital raised by
tenants
Entrepreneurship
Education
Number of participants
enrolled in the program
% of local schools offering
(pre and post investment)
% of participants completing
the program
Increase in awareness of
business concepts (pre vs. post)
Increase in number of participants
considering business creation as
a career option (pre vs. post)
Change in student performance
before and after program
Number of students starting
businesses
Number of students that stay
within the service area
Technical
Assistance
And Training
Number of business starts
Number of business
expansions
Number of clients
Number of clients still in business
Number of jobs (FTEs)
created/retained
Private $ raised by client firm
Annual income and benefits/job or
average wage/job
Jobs (FTE) = wage earners and proprietors
35
While this broad set of performance metrics provided a useful framework for the
evaluation, a “best in class” metrics system as proposed later in this report would
require refinement of this list down to those outcome measures that best capture
the impacts of entrepreneurship development investments across a range of
project types. Recommendations in Chapter 9 include the identification of a set of
outcome measures that could result in an operationalized “best in class” metrics
system to guide future investments by ARC.
DATA COLLECTION
To gather the data identified in the metrics framework, the evaluation team used
a four-part approach:
1. Data for each sample project were gathered from the ARC project file –
the program category, funding level, goals (or need addressed), and how
the program was implemented.
2. Additional data on project performance were gathered through phone
interviews with project directors – outcomes produced (both quantitative
and qualitative), value attributed to the project, success in achieving
objectives.
3. Data on broader capacity and policy impacts were gathered through
phone interviews with both project and non-project stakeholders – policy
impacts in the region, other qualitative and quantitative impacts on
capacity in the region.
4. Data on place-based and broad policy impacts were gathered through
selected site visits in geographic areas where investments in a number of
program categories had been made, determined after the previous parts
of the data collection process were completed.
To facilitate the collection of data in part two above, a draft protocol for the follow-
up phone interviews was developed (see Appendix C).
DESCRIPTION OF EVALUATION TASKS
A number of specific evaluation tasks were identified and completed. To provide
context for the evaluation and to guide the development of the metrics
framework, the team completed a literature review of entrepreneurship
development project evaluations conducted by organizations such as
Organisation for Economic Co-operation and Development (OECD), Small
Business Administration (SBA), Economic Development Administration (EDA),
US Department of Agriculture (USDA), and National Institute of Standards and
Technology (NIST) – Manufacturing Extension Program (MEP) and Advanced
Technology Program (ATP), as well as evaluations by private organizations such
36
as Community Development Venture Capital Alliance (CDVCA), National
Business Incubation Association (NBIA) and other local and state organizations
(see Chapter 3 and Appendix A). In addition to guiding development of the
metrics framework, this review helped the evaluation team determine whether a
comparative analysis of ARC performance relative to other similar types of
initiatives was feasible.
Members of the evaluation team conducted phone interviews with non-project
stakeholders to identify the broader impacts of ARC Entrepreneurship Initiative
projects. These individuals tended to be national or regional experts on
economic development and the specific issues and challenges facing the
Appalachian region or specific states and localities within the ARC service area
(see Appendix B for a list of stakeholders interviewed and Appendix D for the
protocol used.) Interviews were also conducted with state program managers to
develop an understanding of how the EI was implemented in each state and with
other key leaders who had deep experience in the region. Recurring themes from
these key informant interviews were identified and reported as qualitative impacts
of ARC program investments.
The most significant evaluation task was the team’s review of project files
provided by ARC for a sample of 114 projects to understand the purpose, goals,
objectives, and identified outputs and outcomes. ARC’s performance
measurement system tracked a number of key performance metrics across
projects that were relevant to this evaluation:
Businesses served
Jobs created
Jobs retained
Project participants
New businesses created
Private investment leveraged.
While this common set of metrics facilitates assessment of the impacts of ARC’s
project portfolio, the evaluation team also used the metrics framework developed
from the literature review to collect additional data on the sample projects. The
team conducted phone interviews with key staff of each project’s lead institution
to review and/or collect information and data on both the performance metrics
reported to ARC and the broader set of performance metrics identified by the
project team in consultation with the advisory group. It is important to note that
few project leaders were able to report data for this broad set of metrics. Since
these metrics were not necessarily identified as outcomes by the grantees, it was
not expected that they would be able to report on all of these metrics. However,
the evaluation team wanted to be able to capture as broad a set of performance
metrics as possible during these interviews, so the metrics framework was used
as a guide.
37
Using data obtained from final project reports submitted to ARC and data
collected during the phone interviews, the team prepared a descriptive statistical
analysis of performance metrics. This analysis allowed the team to evaluate how
sample projects had achieved their stated objectives and how additional metrics
contribute to understanding the outcomes and impacts of project investment.
These interviews provided useful data on project outcomes. As part of the effort
to gain a sense of impact beyond individual projects, the project team also
conducted site visits to four locations with the region. These site visits were used
to conduct personal interviews and/or focus groups with project teams and other
community stakeholders. The insights gained through these site visits helped to
confirm what the team learned through other interviews and to provide case
examples to illustrate specific lessons learned associated with this evaluation.
A final task was to conduct a meta-analysis of the outcomes and impacts
associated with ARC projects as compared to any benchmarks identified through
the literature review. The purpose of this analysis was to assess the impacts of
ARC programs relative to similar investments in economic development
activities. As described in Chapter 3, the literature review identified few other
evaluations that sought to measure the outcomes of entrepreneurship
development investments like ARC – initiatives that are designed to change the
culture and economic development direction of a region. Assessments were most
often completed for particular programs, such as SBDCs, with a focus on
measuring impacts at the individual entrepreneur (customer) level. Most of these
assessments have measured outputs from program activities, e.g., number of
clients served, dollars invested, as opposed to outcomes on either individual
businesses or the overall economy.
It was difficult to use information from these previous assessments for a
comparative analysis of the ARC EI since ARC investments were strategically
made to demonstrate the potential for entrepreneurship development and to
change attitudes about economic development within communities and the
broader region. Specific projects were not designed simply to create jobs, but
had a broader set of goals including, for example, to:
Expand use of e-commerce
Create an angel network
Double the number of high schools teaching entrepreneurship
Train teachers to use entrepreneurship curriculum
Prepare 100 new business plans as part of a competition
Complete a business incubator feasibility study and strategic plan
Attract 10,000 visitors in first year of a heritage tourism development.
Even detailed metrics for these types of broader goals tell only part of the story. It
is not possible to talk about entrepreneurship development by reporting on any
single metric. However, in lieu of a broader meta analysis, it was possible to
calculate for the sample projects and for the entire portfolio an estimate of public
cost per job created that could be compared to estimates for other similar
programs. These comparisons are provided in Chapter 6.
38
As a result of these challenges, the evaluation team chose to recognize the
embryonic nature of entrepreneurship development evaluation research and to
use what had been learned through both the literature review and in-depth
assessment of sample projects to suggest what performance metrics would best
capture the essence of initiatives like ARC’s EI – a “best in class” metrics system.
This system is discussed in detail in Chapter 9.
SAMPLE SELECTION PROCESS
ARC made 448 grants to entrepreneurship-related projects from 1997 through
2005. This evaluation focused on projects whose commitment from ARC had
been completed, i.e., “closed” projects between 1997 and 2005; a total of 354
grants were closed at the time of the evaluation. However, since a number of
grants were for follow-on investments, the evaluation team further narrowed the
universe of eligible projects to a set of 229 unique, closed projects. While most
projects were designed to address the specific issues or needs within a
community, region or state, some of these investments were in region-wide
projects, primarily designed to raise the level of awareness of entrepreneurship
generally or to provide an opportunity to explore a specific issue with applicability
across the region, such as the role of business incubators or issues related to
access to capital for entrepreneurs.
This group of 229 projects represented investments in all types of projects (Table
4.2) and in all states in the region (Table 4.4). It was from this universe of
projects that the evaluation sample was drawn. All grants related to a specific
project were reviewed as part of the sample analysis.
Each of the closed, unique projects was assigned to one of five categories by
reviewing the title of the project and the description of the project contained in the
September 2003 ARC publication, Entrepreneurship Initiative: Program Summary
and Approved Projects. Some projects in the database were not in this
publication, notably those coded as “Commission” rather than coded by a specific
state. These Commission projects were coded according to the titles where
possible. The five categories used were:
Capital Access – any project where loans, grants or equity investments
were made in companies
Entrepreneurship Education– all projects for youth
Incubator – any project to study the feasibility of, plan, or operate an
incubator; virtual incubators included under technical assistance
Sectors – any project whose aim was to support entrepreneurs in a single
sector or type of business; excluding incubator projects
Technical assistance and training – any project where assistance was
given directly to individual entrepreneurs.
39
Since community capacity was a cross-cutting theme, the evaluation team
determined that all projects in the sample would be evaluated for community
capacity building. A total of 28 projects were not coded because they did not
have a description, the title was vague, or the project was for a conference or
similar activity. The distribution of closed, unique projects is shown in Table 4.2.
Based on this distribution, the team randomly selected a sample of 114 projects.
Then, three projects were discarded and three were added to make the state
distribution more even.
Table 4.2. Distribution by Program Category – Universe of Unique Projects
Category Number Percent
Capital Access 18 7.9
Entrepreneurship Education 42 18.3
Incubators 30 13.1
Sectors 40 17.5
Technical Assistance and Training 71 31.0
Other/Not Coded 28 12.2
Total 229 100
The objective was to choose a representative sample of projects, defined as
having the same distribution of projects by category and by state as the universe
of unique projects. In addition, the team sought to have a sample that was
representative in terms of size of investment by ARC. The team concluded that
the sample was representative, but not strictly random, as described below.
Table 4.3 shows the number and percent of projects in each program category in
the original sample of 114 and in the final sample of 88.
72
Based on the
distribution of the sample projects by program category, the final evaluation
sample appears to be representative of ARC’s EI projects as a whole.
The geographic distribution of the sample projects as compared to the
distribution by state for the universe of unique projects is shown in Table 4.4 and
in Figure 4.1.
73
Note that three states, Georgia, North Carolina and South
Carolina, appear in the sample at a rate slightly higher than their actual number
72
Although 114 projects were included in the original sample, completed interviews were
obtained for 88 projects. Project interviews were not completed for a variety of reasons including
loss of institutional memory of the project because project leaders had left the organization,
inability to schedule interviews after repeated attempts, and the closure of lead organizations.
73
The map in Figure 4.1 indicates that some grants went to organizations outside the region. This
apparent anomaly occurs because, in some cases, the lead organization was located outside the
ARC region, although the project was implemented within the region.
40
of projects. These three states had fewer total projects and, in order to include
more than one project from each of these states in the sample, the team chose to
over sample to insure more representative findings from these states. The
sample appears to be representative of the geographic diversity of EI projects.
In terms of ARC investment, the sample again appears to be representative of
the project universe. Average ARC investment per project was $126,387, while
the average amount of ARC investment per project in the sample was $145,997.
The range of ARC investments in the universe was $2,000 to $2.2 million; within
the sample, the range was $3,500 to $2.2 million. The total investment in the
ARC projects was $42,971,688; the total in the sample was $12,847,733.
Table 4.3. Distribution by Program Category – Original and Final Sample of
Projects
CATEGORY ORIGINAL ORIGINAL
%
FINAL FINAL
%
Capital Access 10 8.8 8 9.1
Entrepreneurship
Education
23 20.2 17 19.3
Incubators 17 14.9 12 13.6
Sectors 23 20.2 17 19.3
Technical assistance and
Training
41 36.0 34 38.6
Other/not coded --- --- --- ---
Total 114 100% 88 100%
CAVEATS TO THE EVALUATION APPROACH
Any evaluation must address the potential caveats associated with the proposed
methodology. While it is important to acknowledge and, to the extent possible,
mitigate these shortcomings, the methodology used in this project was as
rigorous as possible given the diverse set of individual projects, each with its own
set of self-defined performance measures that characterize the ARC
Entrepreneurship Initiative.
The first caveat to the selected methodology was its reliance on self-reporting of
outcomes and other performance metrics. In many circumstances, this would be
a severe constraint on the integrity and objectivity of the data and conclusions. In
this case, however, the grantee organizations were required to submit final
reports as part of their contractual agreement with ARC and these reports could
be, and often were, subject to audit. In addition, ARC’s Regional Planning and
Research division made quasi-random validation field visits to project leaders.
41
The goal of these visits was to validate that the activities proposed under the
grant were actually taking place. The evaluation team took the view that the
project reports could legitimately be regarded as binding and accurate depictions
of the impacts associated with each project. It was beyond the scope of this
evaluation to independently verify the data reported by individual projects as part
of their reporting to ARC or as part of the data collection process associated with
this evaluation.
To develop insights into the broader policy impacts associated with ARC
investments, we chose to rely on interviews with key stakeholders, an adaptation
of the key informant research approach. Since these broader impacts related to
such things as changes in attitudes toward entrepreneurship and economic
development in the region, renewed hope for the future, and the elevation of a
new set of leaders, the best means of gaining insight into these changes in
regional or community capacity was to question those who had deep
appreciation for the culture of Appalachia, extensive experience working in
economic development in the region, and expertise in particular project areas in
which ARC invested. Recognizing the inherent bias in relying on interviews with
individuals to assess overall change in a region, the evaluation team chose to
identify a broad group of stakeholders and to report recurring themes and
observations that were widely held within this group.
Another caveat relates to what might be called a “bias toward success.” If the
universe of closed projects does not include both “successful” and “failed”
projects, the evaluation results will be biased toward success, i.e., the evaluation
will not capture the insights and outcomes (or lack thereof) associated with
projects that were not successful. Indeed, failed projects may provide insights
that are helpful in addressing some of the research questions articulated in this
proposal. Based on the outcome of the project interviews, it was clear that the
sample included projects that were successful and sustainable, as well as those
that were not.
The final caveat relates to the rigor of statistical analysis permitted as part of this
evaluation. The small sample size and the diverse types of projects included in
the sample prevent the use of sophisticated statistical models or methods.
However, the descriptive statistical methods used allowed us to address the
critical research questions outlined above and to provide recommendations for
ARC’s consideration in guiding the entrepreneurship investments in the future.
4
2
T
a
b
l
e
4
.
4
:
D
i
s
t
r
i
b
u
t
i
o
n
o
f
A
R
C
P
r
o
j
e
c
t
s
b
y
S
t
a
t
e
–
U
n
i
v
e
r
s
e
o
f
U
n
i
q
u
e
P
r
o
j
e
c
t
s
,
O
r
i
g
i
n
a
l
S
a
m
p
l
e
a
n
d
F
i
n
a
l
S
a
m
p
l
e
*
#
U
N
I
V
E
R
S
E
#
O
R
I
G
I
N
A
L
S
A
M
P
L
E
#
F
I
N
A
L
S
A
M
P
L
E
%
U
N
I
V
E
R
S
E
%
O
R
I
G
I
N
A
L
S
A
M
P
L
E
%
F
I
N
A
L
S
A
M
P
L
E
A
l
a
b
a
m
a
2
0
1
1
8
1
0
1
0
9
G
e
o
r
g
i
a
7
5
4
3
4
4
K
e
n
t
u
c
k
y
1
4
8
6
7
7
7
M
a
r
y
l
a
n
d
1
1
5
5
5
4
6
M
i
s
s
i
s
s
i
p
p
i
1
4
8
5
7
7
6
N
e
w
Y
o
r
k
1
6
9
7
8
8
8
N
o
r
t
h
C
a
r
o
l
i
n
a
3
3
3
1
3
3
O
h
i
o
2
6
1
4
1
2
1
3
1
2
1
4
P
e
n
n
s
y
l
v
a
n
i
a
1
2
7
6
6
6
7
S
o
u
t
h
C
a
r
o
l
i
n
a
3
3
3
1
3
3
T
e
n
n
e
s
s
e
e
1
0
6
4
5
5
4
V
i
r
g
i
n
i
a
3
8
2
1
1
4
1
9
1
8
1
6
W
e
s
t
V
i
r
g
i
n
i
a
2
1
1
1
1
0
1
0
1
0
1
1
A
R
C
C
o
m
m
i
s
s
i
o
n
6
3
1
3
3
1
2
0
1
1
1
4
8
8
1
0
0
1
0
0
1
0
0
*
A
t
o
t
a
l
o
f
2
8
p
r
o
j
e
c
t
s
w
e
r
e
n
o
t
c
o
d
e
d
a
s
d
e
s
c
r
i
b
e
d
i
n
t
h
e
t
e
x
t
.
43
Figure 4.1. Map of Sample and Total EI Projects, by State
44
CHAPTER 5
THE WIDER REGIONAL ECONOMY – ENTREPRENEURSHIP
TRENDS IN APPALACHIA
INTRODUCTION
The Entrepreneurship Initiative was a bold attempt to bring about an economic
and attitudinal transformation across the Appalachian region. It is therefore
appropriate to step back from the specifics of the initiative – the individual
projects and programs funded and the communities in which they are located –
to consider at a more general level entrepreneurship trends across the region. In
particular, there is value in looking at the state of entrepreneurship in Appalachia
at the beginning of the initiative, in comparison with the nation as a whole, and
whether there has been any discernible change in the distribution of
entrepreneurial activity within the region. The initiative was launched in 1997 and
this evaluation considered projects closed through 2005, so these are the two
comparison dates used to assemble data in the following analysis. No attempt is
being made to establish cause and effect; the analysis is intended to
demonstrate the context within which the initiative was implemented.
THE CHALLENGE OF MEASUREMENT
Entrepreneurship is a difficult concept to measure, and many aspects of
entrepreneurship success are intangible.
74
However, several data sources allow
an examination of employment and income indicators at the county level to
assess general entrepreneurship trends. This analysis relies upon three data
sets:
Bureau of Economic Analysis’ Regional Economic Information System
U.S. Census Bureau’s Nonemployer Statistics
U.S. Census Bureau’s County Business Patterns.
The Center for the Study of Rural America at the Federal Reserve Bank of
Kansas City developed a series of indicators on the breadth and depth of
entrepreneurship at a regional level.
75
These measures use data from the
Bureau of Economic Analysis’ Regional Economic Information System and the
U.S. Census Bureau’s Nonemployer Statistics. The “breadth” of
entrepreneurship is measured by nonfarm proprietors as a proportion of total
nonfarm employment, and the “depth” of entrepreneurship is a measure of
74
Stephan J. Goetz and David Freshwater, “State-Level Determinants of Entrepreneurship and a
Preliminary Measure of Entrepreneurial Culture,” Economic Development Quarterly 15.1
(February 2001):58-70.
75
Sarah Low, “Regional Asset Indicators: Entrepreneurship Breadth and Depth,” The Main Street
Economist (September 2004):1-4.
45
nonfarm proprietors’ income per total nonfarm proprietors. These indicators have
been applied to all the counties in the Appalachian region for 1997 and 2005.
The second set of indicators follows from work of the Association for Enterprise
Opportunity (AEO). This organization uses the U.S. Census Bureau’s
Nonemployer Statistics and County Business Patterns data to estimate the
number of microenterprise businesses and the number of microenterprise
employees.
76
These datasets have been used in this analysis to provide
estimates for the number of microenterprises and the amount of microenterprise
employment for Appalachian counties for 1997 and 2005.
77
Additionally, a measure was developed to provide a sense of how important
entrepreneurship is to the local economy. This measure presents both
proprietors’ income as a proportion of total personal income in each county in
Appalachia and – in order to distinguish the effects of entrepreneurship from
broader trends – shows how these two components have themselves changed
over time.
It should be noted that ARC used a broad definition for the EI that includes small-
and medium-sized enterprises with fewer than 200 employees. This has
particular relevance for the capital access programs and for some of the
technical assistance activities, but for the purposes of this analysis data on
proprietors and microenterprises have been used as they serve as better (but far
from perfect) proxies for entrepreneurial activity. To provide a context, Table 5.1
shows growth rates in the number of new businesses in different size categories
between 1997 and 2005.
Table 5.1: Change in the Number of Small- and Medium-Sized Enterprises
410 ARC COUNTIES UNITED STATES
Size
# Employees
1997 2005 %
Change
1997 2005 %
Change
1-4 274,877 208,856 2.2 3,757,627 4,119,363 9.6
5-9 106,893 110,852 3.7 1,354,488 1,411,199 4.2
10-19 64,319 70,180 9.1 856,118 937,617 9.5
20-49 41,708 46,164 10.7 572,437 636,625 11.2
50-99 13,773 15,258 10.8 194,068 219,324 13.0
100-249 8,413 8,750 4.0 113,832 125,027 9.8
Total 509,983 532,060 4.3 6,848,570 7,449,155 8.8
Source: U.S. Census Bureau, County Business Patterns – Establishments by Size Class
76
For more information, see .
77
The number of microenterprise businesses is the number of nonemployer establishments plus
the number of establishments with 1-4 employees. The estimate of microenterprise employment
is the number of nonemployers plus an average of 2.5 employees per establishment
(establishments with 1-4 employees). This methodology differs from the AEO methodology.
46
Overall, the growth rate of business establishments with fewer than 250
employees in the ARC region was less than half of that of the nation as a whole,
with the greatest divergence in those establishments with fewer than five
employees and those in the range of 100-249 employees. In no size category
does the ARC region outperform the national rate. Given the comparatively low
growth rates for small- and medium-sized enterprises in the region, the EI’s
inclusion of these firms as a target for support appears to be strategic.
UNDERSTANDING THE REGIONAL CONTEXT
Three main observations derive from the following analysis of regional
entrepreneurship data and trends:
Trends in nonfarm proprietor and microenterprise employment in
Appalachia showed an increase over the 1997-2005 period consistent with
the nation as a whole. The counties benefiting most from above national
average increases were in the southern tier of Appalachia.
Trends in nonfarm proprietor income showed the Appalachian region
lagging behind U.S. levels in 1997 and slipping further behind by 2005.
Forty-two percent of Appalachian counties saw a decrease in income
levels, primarily in the northern and central tiers, whereas 25% saw
increases in nonfarm proprietor income levels higher than the national
rate, the majority of which are located in the southern tier.
Trends in the impact of entrepreneurship on the local economy, as might
be expected, generally echo the trends in nonfarm proprietor income, with
similar levels in Appalachia and the U.S. in 1997, but with Appalachia
substantially falling behind by 2005. This decline occurred in spite of the
fact that almost all Appalachian counties experienced increases in total
personal income. Only 15% of counties benefited from levels higher than
the national rate in 2005, and these were evenly distributed across the
region.
A detailed presentation of the data underlying these observations follows.
Entrepreneurship Employment
As there are many ways to determine or interpret what constitutes
entrepreneurship employment, two methods are presented here, each of which
gives a slightly different picture of entrepreneurship employment in the region.
Table 5.2 provides a summary of data for the U.S. and the 410 counties that
comprise the Appalachian Region.
As the table shows, trends in entrepreneurship employment in Appalachia are
broadly similar to those for the nation as a whole. In 1997, nonfarm proprietor
employment comprised 15.5% of total nonfarm employment in the U.S. and
47
15.6% in Appalachia. By 2005, these values had increased to 18.2% in the U.S.
and 18.0% in Appalachia.
Table 5.2. Indicators of Entrepreneurship Employment
INDICATOR / MEASURE 1997 VALUE 2005 VALUE
Nonfarm Proprietors Employment
U.S.
Appalachia
23,648,200
1,744,246
31,147,600
2,202,141
Nonfarm Proprietors Employment/Total Nonfarm Employment
U.S.
Appalachia
15.5%
15.6%
18.2%
18.0%
Number of Microenterprise Businesses
U.S.
Appalachia
19,197,236
1,432,733
24,511,431
1,757,105
Estimate of Microenterprise Employment
U.S.
Appalachia
24,833,677
1,845,049
31,690,476
2,178,389
Microenterprise Employment Estimate/Total Nonfarm
Employment
U.S.
Appalachia
16.3%
16.5%
17.9%
17.8%
Sources: U.S. Census Bureau Nonemployer Statistics and County Business Patterns;
Bureau of Economic Analysis, Regional Economic Information System
The following maps show nonfarm proprietors’ employment as a proportion of
total nonfarm employment in the Appalachian region. In 1997, there were 39
counties in Appalachia in which nonfarm proprietors accounted for 30 percent or
more of total nonfarm employment (Figure 5.1); this number almost doubled to
78 counties by 2005 (Figure 5.2).
Figure 5.1
Entrepreneurship Employment in
Appalachia:
Nonfarm Proprietors as a Percent of Total
Nonfarm Employment, 1997
Data Source: Bureau of Economic Analysis, Regional
Economic Information System
Map Prepared by RUPRI
October 2007
7.96% to 19.99% (243)
20% to 29.99% (128)
30% to 60.15% (39)
48
Figure 5.2
Entrepreneurship Employment in
Appalachia:
Nonfarm Proprietors as a Percent of Total
Nonfarm Employment, 2005
4.46% to 19.99% (187)
20% to 29.99% (145)
30% to 66.16% (78)
Data Source: Bureau of Economic Analysis, Regional
Economic Information System
Map Prepared by RUPRI
October 2007
Nonfarm proprietor employment increased 31.7% from 1997 to 2005 in the U.S.
and 26.3% in Appalachia. Levels increased in all but 70 Appalachian counties
from 1997 through 2005, and grew at a rate exceeding the national average in
141 Appalachian counties – more than doubling in 21 counties (Figure 5.3).
Figure 5.3
Entrepreneurship Employment
Change in Appalachia:
Percent Change in Nonfarm Proprietor
Employment, 1997-2005
Data Source: Bureau of Economic Analysis, Regional
Economic Information System
Map Prepared by RUPRI
October 2007
Decrease or No Change (70)
Increase 0.1%-31.7% (199)
Increase 31.8%-271.7% (141)
49
As Figure 5.3 shows, although the distribution of growth in entrepreneurship
employment extends to all parts of the region, there is a concentration of
counties with growth above the national average in eastern Tennessee, western
North Carolina, northern Georgia and northern Alabama.
Examining the number of microenterprise employees shows a similar picture of
Appalachia tracking the national levels. In 1997, microenterprise employment as
a percent of total nonfarm employment was 16.3% in the U.S. and 16.5% in
Appalachia. By 2005, these levels had increased to 17.9% and 17.8%
respectively. In Appalachia in 1997, there were 25 counties in which
microenterprise employment accounted for 30% or more of total nonfarm
employment (Figure 5.4); by 2005 there were 48 such counties (Figure 5.5).
Figure 5.4
Microenterprise Employment in
Appalachia:
Microbusiness Employment* as a Percent of
Total Nonfarm Employment, 1997
10.0% to 19.9% (215)
20% to 29.9% (170)
30% to 42.9% (25)
*Microbusiness employment equals the number of
nonemployers plus the estimated number of employees in
establishments of employment size class 1-4 (2.5
employees per establishment).
Data Source: U.S. Census Bureau County Business
Patterns and Nonemployer Statistics; Bureau of
Economic Analysis, Regional Economic Information
System
Map Prepared by RUPRI
October 2007
From 1997 to 2005, the number of microenterprise businesses increased 27.7%
in the U.S. and 22.6% in Appalachia. Microenterprise employment increased
23.6% in the U.S. and 18.1% in Appalachia over this same period. Only 24
counties in Appalachia had a decrease or no change in the number of
microenterprise businesses (Figure 5.6), and only 44 counties saw a decrease or
no change in microenterprise employment from 1997 to 2005 (Figure 5.7).
50
Figure 5.5
Microenterprise Employment in
Appalachia:
Microbusiness Employment* as a Percent of
Total Nonfarm Employment, 2005
8.27% to 19.9% (196)
20% to 29.9% (166)
30% to 45.7% (48)
Data Source: U.S. Census Bureau County Business
Patterns and Nonemployer Statistics; Bureau of
Economic Analysis, Regional Economic Information
System
*Microbusiness employment equals the number of
nonemployers plus the estimated number of employees
in establishments of employment size class 1-4 (2.5
employees per establishment).
Map Prepared by RUPRI
October 2007
Figure 5.6
Microenterprise in Appalachia:
Percent Change in the Number of
Microenterprise Businesses*, 1997-2005
*Microbusiness business are the number of
nonemployers plus the number establishments of
employment size class 1-4.
Data Source: U.S. Census Bureau County Business
Patterns and Nonemployer Statistics; Bureau of
Economic Analysis, Regional Economic Information
System
Decrease or No Change (24)
Increase 0.1%-27.7% (272)
Increase 27.8%-124.7% (114)
Map Prepared by RUPRI
October 2007
51
Figure 5.7
Microenterprise in Appalachia:
Percent Change in the Number of
Microenterprise Employees*, 1997-2005
*Microbusiness employment equals the number of
nonemployers plus the estimated number of employees
in establishments of employment size class 1-4 (2.5
employees per establishment).
Data Source: U.S. Census Bureau County Business
Patterns and Nonemployer Statistics; Bureau of
Economic Analysis, Regional Economic Information
System
Decrease or No Change (44)
Increase 0.1%-23.7% (258)
Increase 23.8%-118.8% (108)
Map Prepared by RUPRI
October 2007
Entrepreneurship Income
A measure of entrepreneurship income is nonfarm proprietors’ income per
nonfarm proprietor employment. Table 5.3 shows this indicator for 1997 and
2005, along with other indicators of income and earnings as a basis for
comparison.
Table 5.3. Indicators of Entrepreneurship Income
INDICATOR / MEASURE 1997 VALUE
(ADJUSTED
TO $05)
2005 VALUE
Nonfarm Proprietors’ Income / Nonfarm Proprietor
Employment
U.S.
Appalachia
Appalachia as a % of U.S. Value
$27,881
$23,094
82.8%
$30,193
$23,218
76.9%
Per Capita Income
U.S.
Appalachia
Appalachia as a % of U.S. Value
$30,827
$25,687
83.3%
$34,471
$28,336
82.2%
Wage & Salary Disbursements / Wage & Salary Employment
U.S.
Appalachia
Appalachia as a % of U.S. Value
$36,331
$30,706
84.5%
$40,146
$32,779
81.6%
Source: Bureau of Economic Analysis, Regional Economic Information System
On all measures of income, Appalachia lags somewhat behind the national
figures and fell further behind over the period 1997-2005. In 1997, the three
52
income indicators for Appalachia were between 82% and 85% of the U.S. value.
By 2005, nonfarm proprietor income per nonfarm proprietor had fallen to 76.9%
of the U.S. value and wage and salary disbursements per wage and salary job
had slipped to 81.6% of the U.S. level.
In 1997, nonfarm proprietor income per nonfarm proprietor (adjusted to 2005
dollars) ranged from $5,263 to $54,117 in Appalachia (Figure 5.8). In 26
counties in Appalachia, the value exceeded the U.S. value of $27,881 for this
indicator. In 2005, nonfarm proprietor income per nonfarm proprietor in
Appalachia ranged from $7,190 to $64,689, and 28 counties had values
exceeding the U.S. figure of $30,193 (Figure 5.9).
Figure 5.8
$5,263 - $16,572 (135)
$16,573 - $27,881 (249)
$27,882 - $54,117 (26)
Entrepreneurship Income in
Appalachia:
Nonfarm Proprietors Income per Nonfarm
Proprietor, 1997
(adjusted to 2005$)
Data Source: Bureau of Economic Analysis, Regional
Economic Information System
Map Prepared by RUPRI
October 2007
Total nonfarm proprietors’ income (real $2005) increased by 42.6% in the U.S.
and 26.9% in Appalachia from 1997 to 2005 (Figure 5.10). Within Appalachia,
total nonfarm proprietors’ income decreased in 172 counties, noticeably in New
York, Pennsylvania, eastern Kentucky, western West Virginia and southwestern
Virginia. Increases were experienced in 238 counties – 102 counties in
Appalachia had increases in nonfarm proprietors’ income exceeding the national
rate, and in 22 counties income more than doubled during this time period. The
greatest gains were to be found in eastern West Virginia, eastern Tennessee,
western North Carolina, and northern Alabama.
53
Figure 5.9
Entrepreneurship Income in
Appalachia:
Nonfarm Proprietors Income per Nonfarm
Proprietor, 2005
Data Source: Bureau of Economic Analysis,
Regional Economic Information System
$7,189 - $18,641 (284)
$18,642 – $30,193 (98)
$30,194 - $64,686 (28)
Map Prepared by RUPRI
October 2007
Figure 5.10
Entrepreneurship Income in
Appalachia:
Percent Change in Total Nonfarm
Proprietors’ Income, 1997-2005
(1997 adjusted to 2005$)
Decrease (172)
Increase 0.1%-42.6% (136)
Increase 42.7%-188.7% (102)
Data Source: Bureau of Economic Analysis,
Regional Economic Information System
Map Prepared by RUPRI
October 2007
54
Entrepreneurship’s Contribution to the Local Economy
One method of assessing the importance of entrepreneurship to the local
economy is to examine the share of nonfarm proprietors’ income to total county
personal income in 1997 and 2005. Table 5.4 summarizes this indicator for the
U.S. and Appalachia.
Table 5.4. Indicators of Entrepreneurial Contribution
INDICATOR/MEASURE 1997 VALUE 2005 VALUE
Nonfarm Proprietor Income/Total Personal Income
U.S.
Appalachia
7.8%
7.0%
9.2%
7.6%
As Table 5.4 shows, Appalachia has fallen substantially behind the U.S. on this
measure of entrepreneurial contribution. Within Appalachia in 1997, 90 counties
had shares greater than the national average of 7.8% (Figure 5.11), but by 2005,
there were only 63 counties in Appalachia with a share greater than the national
average of 9.2% (Figure 5.12). From 1997 to 2005, the nonfarm proprietors’
income share of total county personal income increased in 170 Appalachian
counties, and decreased in 240 Appalachian counties (Figure 5.13).
Before drawing any conclusions about these increasing or decreasing shares, it
is first necessary to look at the trends in actual nonfarm proprietors’ income and
total personal income. In some cases, proprietors’ income share may be
increasing as a share because total county income is on the decline, resulting in
a higher share even if actual nonfarm proprietors’ income did not rise. The map
in Figure 5.14 compares nonfarm proprietors’ income change to total county
personal income change.
In 238 Appalachian counties, nonfarm proprietors’ income (in real $2005)
increased from 1997 to 2005, and total county personal income decreased in
only 4 of these counties. Nonfarm proprietors’ income decreased in 172
Appalachian counties; in 15 of these counties, total county personal income also
decreased. The conclusion to be drawn here is that whether or not counties
have an increasing or decreasing share of nonfarm proprietor income does not
generally appear to be impacted by increases or decreases in total personal
income.
55
Figure 5.11
Entrepreneurship and the Local
Economy:
Nonfarm Proprietors’ Income as a Share of
Total Personal Income (TPI), 1997
At or below U.S. Share (320)
Above U.S. Share (90)
Data Source: Bureau of Economic Analysis, Regional
Economic Information System
Map Prepared by RUPRI
October 2007
U.S. Share = 7.84%
Figure 5.12
Entrepreneurship and the Local
Economy:
Nonfarm Proprietors’ Income as a Share of
Total Personal Income (TPI), 2005
At or below U.S. Share (347)
Above U.S. Share (63)
Data Source: Bureau of Economic Analysis, Regional
Economic Information System
Map Prepared by RUPRI
October 2007
U.S. Share = 9.20%
56
Figure 5.13
Entrepreneurship and the Local
Economy:
Change in the Share of Nonfarm
Proprietors’ Income to Total Personal
Income (TPI), 1997-2005
Share Increased (170)
Share Decreased (240)
Data Source: Bureau of Economic Analysis, Regional
Economic Information System
Map Prepared by RUPRI
October 2007
EI PERFORMANCE WITHIN THE REGIONAL CONTEXT
What do these findings suggest about the impact of ARC’s EI on the region?
While it is difficult to draw any causal connections between ARC investments and
these regional data, several trends are evident. Entrepreneurship, when defined
broadly as nonfarm proprietor and microenterprise employment, is an
increasingly important component of the Appalachian regional economy. For
some parts of the region – the southern tier in particular – rates of employment
associated with entrepreneurship are greater than for the country as a whole.
This observation confirms the importance of investments in entrepreneurship
development as a key piece of a regional economic development strategy. In
essence, entrepreneurship represents an asset upon which the region can build.
However, in terms of income associated with entrepreneurship, the picture
unfortunately mirrors the past – a region continuing to lag behind the rest of the
nation in terms of nonfarm proprietors’ income. This observation suggests that in
spite of ARC’s investment, the income generated through entrepreneurial activity
is not growing in such a way as to allow the region to catch up with the rest of the
country. This analysis also suggests that ARC’s investments have had an impact
on getting more entrepreneurs into the pipeline, but have had a lesser impact on
creating stronger and higher growth entrepreneurs, at least as measured by
income trends. To better understand the reason behind this disparity, a more
detailed investigation of entrepreneurial income and wage and salary levels
associated with entrepreneurship in the region should be undertaken. It was not
possible in the current evaluation to gain insights into the income and wealth
creating potential of entrepreneurial ventures or into the quality of jobs created
for others by these entrepreneurs.
57
Have ARC investments contributed to creating a more entrepreneurial region? It
is difficult to say from analysis of secondary data. Entrepreneurial activity has
increased over the period of investment and some parts of the region have
performed even better than the nation. It is possible that the Appalachian region
would have fallen even further behind the nation in terms of income from
entrepreneurship without EI investments. Insights from more detailed analysis of
sample projects and key stakeholders help answer this question in a way that
secondary data analysis cannot.
Figure 5.14
Entrepreneurship and the Local
Economy:
Change in Nonfarm Proprietors’ Income and
Total Personal Income (TPI), 1997-2005
(Values adjusted to $05)
TPI Increased (234)
TPI Decreased (4)
Nonfarm Proprietors Income Increased
Nonfarm Proprietors Income Decreased
TPI Increased (157)
TPI Decreased (15)
Data Source: Bureau of Economic Analysis, Regional
Economic Information System
Map Prepared by RUPRI
October 2007
58
CHAPTER 6
ARC INVESTMENT PORTFOLIO –
PROJECT IMPLEMENTATION AND REPORTED METRICS
Because ARC is built around a federal-state-local partnership, few of its
programs operate according to a single cookie-cutter model. Instead, each state
and each local program tend to take a distinctive approach to ARC funding
programs and opportunities. Differences in program management from state to
state and differences in the types of projects funded make generalizing impacts
across the EI portfolio difficult. This chapter is designed to provide insights into
project implementation so that later discussions of lessons learned and
recommendations are understood within this overall context.
78
ARC required the reporting of several metrics for all project investments,
including job creation and retention, business creation, businesses served, and
private investment leveraged. Data from the final project reports submitted to
ARC by all grant recipients provide useful insights into the size and scope of the
initiative, as well as these specific, but limited, impacts of project investments.
Deeper understanding of project impacts and the broader policy implications of
ARC investments gained through analysis of the sample projects is presented in
Chapter 7.
PROJECT IMPLEMENTATION
As with most ARC initiatives, the approach to using the EI funds varied widely
across the states in the region. To better understand these differences, and what
they might mean in terms of program outcomes, the evaluation team conducted
phone interviews with state program managers and with the director of the
Entrepreneurship Initiative in Washington, D.C. The purpose of these interviews
was to develop a sense of how each state approached the EI and to get a view of
the initiative from the state level.
Each governor and state program manager approached the Entrepreneurship
Initiative with a different level of commitment to entrepreneurship as an economic
development strategy and a different strategy for distributing what in many cases
was described as a relatively small resource pool. Table 6.1 describes the
strategy or approach used to distribute funds in each state while Table 6.2 shows
the distribution of investments across the region over the 1997 to 2005 period. In
one state, Pennsylvania, a large portion of resources was used to support a
specific intervention – the creation of entrepreneurial networks. In other states,
local development districts were responsible for promoting the initiative and
encouraging local organizations to submit grant proposals. In most cases, states
78
Insights and observations presented in the first section of this chapter are drawn from
interviews with regional stakeholders and from project leaders. The evaluation team reported
insights that were widely held and recurring across the range of interviews conducted.
59
Table 6.1. Description of State Approaches to EI Investments*
STATE APPROACH
Alabama Entrepreneurship included in Appalachian Development Plan; provided impetus
for the state to support diverse homegrown initiatives, including entrepreneurship
education and incubators
Georgia State program manager’s office identified local and regional projects to be
funded; relied less on Local Development Districts (LDDs) to take the lead on
developing projects
Kentucky State created a government-sponsored commission whose chair was the state
alternate; Chair lived in region and selected both local and regional projects; also
funded other state organizations to expand activities into region; identified
technical assistance gap and tried to select projects to fill it
Maryland Initially tried to start new programs in the region, providing several years of
funding; partnered with WV on projects; efforts were not able to create
sustainable new institutions; later stages, plugged EI into ongoing activities in the
counties, like Main Street program
Mississippi State program manager and alternate active in identifying projects; tried to work
with local philanthropy organizations
New York Worked at the state level to fund some large programs to support
entrepreneurship education – some local and some regional; supported some
other local initiatives
North
Carolina
Rolled money over until state had a pool of money; contracted with state
Department of Commerce for multi-year program; efforts were not able to create
sustainability; locally driven projects were also funded, with some more success
Ohio Leadership came from Governor’s Office of Appalachia; active in identifying
projects with both Local Development Districts (LDDs) and nonprofits; joint
funding of projects with the Foundation for Appalachian Ohio; open process for
soliciting proposals; multi-year funding of regional institutions and one time
projects
Pennsylvania Most of funds were used to support local networking activities; also put money
into region-wide youth entrepreneurship program; state program manager and
alternate worked out of the Office of Appalachian Development in state
Department of Community and Economic Development
South
Carolina
Most projects identified by state program managers; relied less on Local
Development Districts (LDDs)
Tennessee Worked with some Local Development Districts (LDDs); major investment made
in the development of one organization, leading to the creation of a sustainable
organization with a broad reach
Virginia High priority for the state; spread grants around to as many communities as
possible; saw EI funds as being part of a set of resources, including ARC
telecommunications and Main Street programs, that communities could tap; used
a competitive RFP process with an open workshop planning process leading to a
diverse set of project proposals; funded multi-year projects; drew in other funding
resources, particularly from foundations
West Virginia State highlighted entrepreneurship funds as one of many tools available; non-
profit organizations stepped up and provided most of the impetus for developing
and implementing grant projects; used a competitive RFP process with an open
workshop planning process leading to a diverse set of project proposals; funded
multi-year projects; drew in other funding resources, particularly other state funds
*Interviews were attempted but not completed with state program managers in Mississippi and
Ohio due to illness and staff turnover, respectively, at the time the interviews were undertaken.
Information about their approach to the EI was obtained through an interview with the director of
the EI.
60
used their annual allocations to fund a range of projects; in one case, North
Carolina, funds were rolled over until the state had a pool of resources to support
a larger project.
Several themes were identified through the program manager interviews. From
the perspective of local communities, ARC Entrepreneurship Initiative resources
were described as “catalytic” and important to demonstrating new approaches to
entrepreneurship development. In many cases, these projects would not have
gotten started without ARC investment. As one manager described it, these were
“but for” resources. At the same time, program managers admitted that the
relatively small pool of resources attached to the initiative made it harder to make
entrepreneurship investments a priority, particularly if local development districts
were not focusing in this area and encouraging proposals from their regions.
It was also clear that the state leaders, including governors and program
managers, played a “gate keeping” role in terms of ARC initiatives, including
entrepreneurship. Strong support for entrepreneurship from state leaders often
translated into more aggressive promotion of the program within the state, such
as was seen in Virginia. In addition, since the innovation in entrepreneurship
programs was primarily coming from the local rather than the state level, the
degree to which the state program manager was networked with local economic
development organizations, especially non-profits, could impact the ability to
effectively implement the initiative. Much of the investment in West Virginia, for
example, was driven by community non-profit organizations.
In addition to the diverse approaches used by the states in allocating resources,
ARC investments were spread across a portfolio of program categories. Prior to
selecting the evaluation sample, Entrepreneurship Initiative projects were
classified into program categories based on a review of project descriptions,
including the goals of each project, and follow-up interviews with project
leadership. Project goals differed even within program categories. Metrics used
to report on project outcomes varied by program category, and it was clear from
the interviews with project leaders that job creation and business creation were
not the only metrics considered in articulating project goals and reporting
outcome measures.
To put the findings of this evaluation into better perspective, a description of the
range of projects found within these program types is provided here. These
descriptions should provide readers with a sense of the diversity within each
project type and the breadth of investments undertaken by ARC in support of
entrepreneurship development in the region.
Entrepreneurship Education Projects
Entrepreneurship education projects were designed to introduce
entrepreneurship concepts and curricula into the schools, kindergarten through
61
community college and university. In the language of the programmatic goals set
out in Chapter 3, these programs sought to help create more entrepreneurs in
the pipeline, and were targeted primarily to students as opposed to adult
learners. (Programs for adult learners were included in the technical assistance
and training category.) A profile of the sample entrepreneurship education
projects appears below.
Sample Entrepreneurship Education Projects
Number of projects = 17
Total ARC investment = $2,199,207
Average ARC investment = $129,365
Minimum ARC investment = $10,000
Maximum ARC investment = $784,517
Range of years for project investment = 1 – 4
The range of entrepreneurship education projects supported by the EI was quite
diverse. For example, funds were used to:
Support outreach into rural communities for an existing business plan
competition
Create an internship program and place youth in entrepreneurial
companies
Organize summer youth entrepreneurship camps, using REAL (Rural
Entrepreneurship through Action Learning) and other curriculum
Train teachers in the REAL curriculum and bring it into the classroom
Train middle school students using Junior Achievement, with follow-up
training using a virtual business simulation model
Create student run, school-based enterprises.
Some projects were designed as pilots to test an entrepreneurship education
approach in a community or region. Others were designed to build
entrepreneurship education capacity by training teachers and incorporating new
curriculum into school systems, or by investing in the creation of school-based
enterprises that could be passed down to future classes of students. About half
the sample projects were defined by project leaders as being sustainable, i.e.,
they continued beyond the ARC grant with support from other funders or, in
some cases, with an additional ARC grant.
With the exception of those projects that created school-based enterprises, most
were not designed to create or retain jobs or to build new businesses. Rather,
they were designed to expose youth to entrepreneurship with an expectation that
these entrepreneurial skills would serve to prepare them to become better
employees in the future and/or to motivate them to create their own economic
futures through enterprise development. Given their goals, traditional economic
development metrics provide an incomplete story of the impact of these projects.
6
2
T
a
b
l
e
6
.
2
.
A
n
n
u
a
l
a
n
d
T
o
t
a
l
A
R
C
E
n
t
r
e
p
r
e
n
e
u
r
s
h
i
p
I
n
v
e
s
t
m
e
n
t
s
,
1
9
9
7
-
2
0
0
5
,
b
y
S
t
a
t
e
(
$
)
*
S
T
A
T
E
1
9
9
7
1
9
9
8
1
9
9
9
2
0
0
0
2
0
0
1
2
0
0
2
2
0
0
3
2
0
0
4
2
0
0
5
T
O
T
A
L
A
l
a
b
a
m
a
0
1
5
9
,
0
0
0
3
8
1
,
6
6
3
5
3
7
,
1
8
7
1
9
8
,
8
5
0
4
0
5
,
0
0
0
9
2
,
0
0
0
5
1
8
,
8
9
2
1
1
2
,
6
7
5
2
,
4
0
5
,
2
6
7
G
e
o
r
g
i
a
1
1
0
,
0
0
0
0
4
4
1
,
8
2
6
0
5
8
1
,
5
2
9
3
4
9
,
4
4
2
8
3
,
0
0
0
7
2
3
,
7
4
7
3
3
4
,
1
1
8
2
,
6
2
3
,
6
6
2
K
e
n
t
u
c
k
y
9
2
,
7
6
9
2
2
6
,
0
0
0
7
8
7
,
5
0
0
2
9
0
,
5
3
2
5
1
0
,
5
0
0
2
2
5
,
3
6
2
6
5
7
,
8
8
0
3
8
3
,
7
9
4
5
8
7
,
1
9
8
3
,
7
6
1
,
5
3
5
M
a
r
y
l
a
n
d
0
2
0
7
,
0
0
0
4
5
5
,
2
0
0
5
4
8
,
9
1
8
3
1
2
,
3
7
9
1
3
9
,
0
0
0
9
9
,
0
0
0
1
3
0
,
0
0
0
4
5
,
0
0
0
1
,
9
3
6
,
4
9
7
M
i
s
s
i
s
s
i
p
p
i
0
2
2
0
,
5
7
7
3
0
,
0
0
0
1
6
4
,
9
7
0
5
9
4
,
8
3
5
1
5
7
,
0
0
0
0
2
0
0
,
0
0
0
2
2
6
,
0
0
0
1
,
5
9
3
,
3
8
2
N
e
w
Y
o
r
k
0
2
4
3
,
1
4
0
1
8
7
,
6
5
4
9
2
0
,
0
0
0
7
5
,
0
0
0
2
5
,
0
0
0
1
4
5
,
0
0
0
2
6
5
,
0
0
0
3
5
,
0
0
0
1
,
8
9
5
,
7
9
4
N
o
r
t
h
C
a
r
o
l
i
n
a
0
0
0
1
,
0
5
0
,
0
0
0
0
4
4
4
,
9
0
8
2
0
0
,
0
0
0
1
5
7
,
5
0
0
6
7
3
,
6
7
3
2
,
5
2
6
,
0
8
1
O
h
i
o
0
1
5
3
,
9
4
0
4
0
6
,
2
1
3
4
5
1
,
9
2
1
6
1
3
,
7
0
0
2
4
4
,
5
0
0
1
,
6
7
3
,
3
3
0
1
5
,
0
0
0
5
0
,
0
0
0
3
,
6
0
8
,
6
0
4
P
e
n
n
s
y
l
v
a
n
i
a
3
1
3
,
3
9
7
1
,
0
7
2
,
8
0
8
1
,
0
9
9
,
0
0
0
1
9
9
,
9
0
0
6
5
9
,
6
3
6
3
8
3
,
7
5
0
3
6
2
,
0
0
0
4
6
0
,
0
0
0
1
2
5
,
0
0
0
4
,
6
7
5
,
4
9
1
S
o
u
t
h
C
a
r
o
l
i
n
a
0
5
0
0
,
0
0
0
0
7
5
0
,
0
0
0
1
1
9
,
1
0
5
7
5
2
,
0
0
0
5
0
0
,
0
0
0
8
9
1
,
3
8
2
5
8
,
7
5
0
3
,
5
7
1
,
2
3
7
T
e
n
n
e
s
s
e
e
0
4
5
1
,
7
4
8
3
2
5
,
0
0
0
3
9
0
,
0
0
0
4
1
0
,
1
8
5
1
8
5
,
0
0
0
9
3
,
0
0
0
2
0
1
,
1
3
3
3
5
,
0
0
0
2
,
0
9
1
,
0
6
6
V
i
r
g
i
n
i
a
3
7
4
,
5
0
0
0
4
9
5
,
3
8
0
4
1
6
,
5
2
7
4
7
6
,
6
3
0
4
3
3
,
0
0
0
1
8
1
,
2
0
0
2
8
0
,
3
9
1
9
0
,
0
0
0
2
,
7
4
7
,
6
2
8
W
e
s
t
V
i
r
g
i
n
i
a
1
4
2
,
1
5
0
7
5
2
,
9
4
8
1
,
0
7
5
,
9
1
6
3
0
0
,
5
3
7
1
,
6
7
3
,
7
4
8
1
,
0
4
2
,
8
3
5
5
8
2
,
0
0
0
4
5
8
,
4
0
0
1
,
1
0
5
,
4
7
4
7
,
1
3
4
,
0
0
8
R
e
g
i
o
n
a
l
1
4
,
0
0
0
3
3
2
,
5
0
3
1
2
4
,
6
2
3
2
6
4
,
9
0
1
1
6
9
,
1
6
2
3
3
9
,
6
1
0
9
7
3
,
5
7
4
1
0
8
,
0
6
3
7
5
,
0
0
0
2
,
4
0
1
,
4
3
6
T
o
t
a
l
A
R
C
1
,
0
4
6
,
8
1
6
4
,
3
1
9
,
6
6
4
5
,
8
0
9
,
9
7
5
6
,
2
8
5
,
3
9
3
6
,
3
9
5
,
2
5
9
5
,
1
2
6
,
4
0
7
5
,
6
4
1
,
9
8
4
4
,
7
9
3
,
3
0
2
3
,
5
5
2
,
8
8
8
4
2
,
9
7
1
,
6
8
8
*
F
i
g
u
r
e
s
i
n
c
l
u
d
e
a
l
l
p
r
o
j
e
c
t
s
(
c
l
o
s
e
d
a
n
d
o
p
e
n
)
a
n
d
i
n
v
e
s
t
m
e
n
t
s
f
r
o
m
a
l
l
f
u
n
d
i
n
g
s
o
u
r
c
e
s
w
i
t
h
i
n
A
R
C
.
63
Several examples help to demonstrate the range of entrepreneurship education
projects funded by ARC and the type of outcomes they achieved.
In 1999, the Kentucky Science and Technology Council (KSTC) received funding
from ARC to establish next generation entrepreneurial schools in two pilot
districts. The goal of the project was to create learning communities where
students could embrace an entrepreneurial frame of mind and, in turn, to help all
students generate ideas for and start new ventures in their home communities.
Using initial and follow-on ARC grants, KSTC supported the creation of 99
EdVentures – school-based enterprises – including 31 in high schools, 25 in
middle schools, 37 in elementary schools, and 6 district-wide. KSTC has
documented the outcomes of these activities in two ways – through metrics
including the number of school-based enterprises started (99) and the number of
participants in these ventures (27,000) as well as through stories of each
individual EdVenture that describe the start-up process, funds raised, students
involved, etc.
In West Virginia, ARC funds were used to bring an entrepreneurship focus to an
existing summer camp program, using the REAL curriculum. The Ohio-West
Virginia YMCA provided leadership for this initiative which secured four years of
ARC funding. In addition to serving the 465 student participants, the initiative has
generated additional interest in youth entrepreneurship in the state, attracting
funding to make the program sustainable and engaging youth in several ways.
The statewide economic development planning process, Vision Shared,
partnered with the Ohio-West Virginia YMCA to provide a teen perspective on the
future of the state. The YMCA held a Teen Forum on West Virginia’s Future in
2002 to develop this perspective. The Department of Education created a full-
time entrepreneurship coordinator position and hired a certified REAL instructor
as the coordinator.
ACEnet in Ohio used ARC funding to extend into three high schools a computer
opportunities program designed to provide training to students in setting up
computer-based enterprises. The program created three new businesses and 14
new jobs. In addition, however, it expanded the computer technology available in
these schools and exposed students to entrepreneurship concepts. Another
outcome was the attraction of additional federal support to continue the program
beyond ARC funding.
Technical Assistance and Training Projects
Technical assistance and training projects were designed to build the skill sets of
individual entrepreneurs by offering them access to one-on-one counseling and
workshops related to starting and growing their businesses. The programmatic
goals of these programs were to create more entrepreneurs in the pipeline, and
better informed and skilled entrepreneurs whose businesses were more
productive and created more jobs in the region. Most of these programs were
64
targeted at adult entrepreneurs as distinguished from the entrepreneurship
education projects that primarily targeted youth. A profile of sample technical
assistance and training projects appears below.
Sample Technical Assistance and Training Projects
Number of projects = 34
Total ARC investment = $4,933,997
Average ARC investment = $145,118
Minimum ARC investment = $10,000
Maximum ARC investment = $2,088,961
Range of years for project investment = 1 – 6
One large grant to Pennsylvania to support a statewide technical
assistance/networking initiative skews average investment. If the
Pennsylvania project is removed from the sample, the range of
investments is $10,000 to $650,615 and average investment
per technical assistance project is $86,213.
As with education projects, the range of technical assistance projects was quite
wide. Investments were used to fund such activities as:
Grants to individual entrepreneurs to purchase private sector services,
such as accounting or legal assistance
Funding for “how to start a business” and other workshops for
entrepreneurs and potential entrepreneurs
Providing technical assistance in conjunction with a microenterprise
program
Developing export assistance programs for entrepreneurs
Offering one-on-one assistance to entrepreneurs in accounting, marketing,
and other business management areas
Delivering marketing assistance to clients of a food incubator
Developing computer-based entrepreneurship training programs
Providing managerial and technical assistance to high-tech start ups
Creating a virtual business assistance center.
In general, these projects provided entrepreneurs with access to business
support resources, both customized and in workshop settings. Often, these
services represented additions or expansions to other programs offered by the
local organizations. However, in some cases, these investments were used to
create new capacity in the region, such as through the creation of a virtual
assistance center. About three quarters of projects were defined as sustainable
by project leaders. Several examples provide insights into these technical
assistance and training projects.
Kentucky Highlands Investment Corporation (KHIC) received funding from ARC
for its business support center, The Launching Pad. These funds were used to
provide technical assistance to entrepreneurs, some of whom were customers
65
(or future customers) who had received capital through KHIC’s range of financial
instruments. The Launching Pad provided a place for entrepreneurs to access
computers and other technology, and to obtain services from KHIC’s skilled staff.
This assistance served to formalize and build the capacity of KHIC to support the
non-financial needs of its business customers. Previous assessment of KHIC’s
programs found that this technical assistance or managerial support was as
important to entrepreneurs as the capital that KHIC provided.
79
The Ohio Valley Regional Development Commission used ARC funds to provide
small grants (up to $5,000) for entrepreneurs to purchase business services from
private sector providers. This program was designed in response to the
expressed needs of the Commission’s revolving loan customers who sought
assistance early in the life of their businesses, before they had the sales and
resources to pay for services. The project was successful in meeting the needs
of these entrepreneurs and connecting them to appropriate service providers.
The Commission was able to help almost twice as many entrepreneurs as initially
projected because entrepreneurs often required very small grants to get the help
they needed.
Mississippi State University received support to develop training programs to
make entrepreneurs aware of the power of marketing via the Internet. In 1998,
when the ARC investment was made, the concept and value of e-commerce was
not universally recognized. Mississippi State’s programs exposed home-based
business owners, farmers and youth to Internet marketing concepts. While the
program helped individual business owners, it also launched regional and
national efforts to develop e-commerce as an effective business strategy. The
curriculum developed at Mississippi State is used throughout the country and a
national e-commerce education effort through Cooperative Extension continues
to support education and training in this area.
Sector Projects
Projects funded in the sectors category were focused on supporting
entrepreneurs in a particular sector, such as food, wood products, ceramics,
often by building networks among entrepreneurs. In terms of programmatic
goals, these projects were designed to create more informed and skilled
entrepreneurs, operating more productive businesses that create jobs in the
region. The projects included in the sample, however, were as diverse as the
sectors that make up the economy of the Appalachian region, including:
Creation of a consignment gift shop to support local artisans
Development of an aquaculture program including entrepreneurship
training
79
David Barkley and Deborah Markley, The Development of an Entrepreneurial Support
Organization: The Case of the Kentucky Highlands Investment Corporation, RUPRI Center for
Rural Entrepreneurship, Research Case Study Series Number 1, March 2003.
66
Support for a ceramics corridor cluster in collaboration with a university
Operational support for the Ralph Stanley museum and music center
Development of a driving tour to cultural and heritage attractions and
artisan businesses
Funding for a market study of the cut flowers industry in Mississippi
Support for workshops on development of a craft woodworking industry.
A profile of sample sector projects appears below. Less than half of project
leaders defined their efforts as being sustainable over time. Several examples
serve to illustrate the high degree of diversity in these sector projects.
Sample Sector Projects
Number of projects = 17
Total ARC investment = $1,496,585
Average ARC investment = $90,976
Minimum ARC investment = $1,400
Maximum ARC investment = $393,711
Range of years for project investment = 1 – 4
The Kentucky Artisan Trails Project was designed to create a gateway attraction
on I-75 that would promote a driving tour of regional cultural and heritage sites as
well as artisan businesses along the route. Through development of an
information kiosk on the interstate, a website and hard copy maps, the project
helped to encourage tourism development by networking individual sites into a
destination tour. The website features products produced by regional
entrepreneurs, places for tourists to visit, and events that are planned throughout
the region. The project focused on the heritage tourism sector, including
entrepreneurs, and developed a network of people, places and organizations that
worked to build the sector in the region.
Advantage West, a regional economic development organization in western
North Carolina, used ARC investment to help introduce and develop a biotech
sector in the region. There was much interest in biotech in the region, but limited
staff capacity to move the concept forward. ARC funds were used to hire a
biotech coordinator and to develop wet lab space at the community college – the
first in the region. Advantage West also established a steering committee to
consider strategic opportunities for advancing the sector. These initial
investments have helped create a biotech sector focused on the natural products
available in the region. The region has attracted additional local government and
regional university investments to continue to build this sector.
Appalachian by Design, a non-profit rural economic development organization,
wanted to help create a knitting industry in rural Appalachia by connecting female
artisans with broader markets. ARC funding was used to help individual rural
women develop a cottage knitting industry. The project was initially quite
67
successful, producing high quality products and tapping new markets. Since its
initial success, the project has scaled back as resources became constrained,
forcing the organization to focus on a more limited market.
Incubator Projects
Incubators projects in the region were designed to provide a supportive
environment for entrepreneurs to hatch and grow new enterprises. Incubator
programmatic goals usually related to creating more informed and better skilled
entrepreneurs, helping them create more productive businesses and ultimately
encouraging more entrepreneurs to stay in the region. ARC investments in
incubator projects, however, were generally of three types:
Investments to determine the feasibility of an incubator in a particular
community or region
Investments to support the building and/or operation of an incubator
facility
Investments to develop programs offered through the incubator.
The sample of 12 incubator investments was equally split across these three
types. A profile of sample incubator projects appears below.
Sample Incubator Projects
Number of projects = 12
Total ARC investment = $913,291
Average ARC investment = $76,108
Minimum ARC investment = $10,000
Maximum ARC investment = $388,084
Range of years for project investment = .5 – 1
Projects tended to be much more short term than the other program types,
ranging from six months to one year. Nine of the 12 incubator projects were
defined as being sustainable. Examples of incubator projects follow.
ARC funds were used to enhance the services provided by the Clinch Powell
Community Kitchen in east Tennessee. While the community kitchen already
provided shared space for food entrepreneurs in the area to develop and
produce their goods, ARC funds allowed the incubator to purchase new
equipment and expand the training support offered to entrepreneurs, particularly
in the areas of product development, marketing and general business training.
Incubator staff was able to devote more time to exploring marketing channels
that would add value to products, for example, by combining gift baskets with
specialty food products produced in the region. The incubator was successful in
expanding the number of clients served and was able to leverage additional
funds to become sustainable.
68
In western North Carolina, ARC investment was used to create Blue Ridge Food
Ventures, a kitchen incubator set up to help farmers and others develop value-
added agricultural products. ARC funding provided most of their operating
budget, with other income coming from users of the facility. This relatively new
incubator (at the time of the interview) developed its capacity to serve its clients
by partnering with service providers in the area, such as a microenterprise
development organization, to become a “one stop shop.” The incubator staff
spent most of their time working with clients on product development, specifically
how to put a new twist on a food product.
The city of Gadsden, Alabama received ARC funds to conduct an incubator
feasibility study. The city lost a major employer, a steel mill, in 2000 – a
devastating blow to the economy. The city’s community development director
was looking for alternatives and the ARC funds provided an opportunity to
determine whether it would be feasible for the city to develop an incubator. The
study concluded that an incubator was feasible and that there were residents in
the community who had an interest in starting their own businesses. What the
study did, according to the development director, was to give the city information
and options – they could begin to think about what kind of incubator facility might
be most appropriate, where it would be located and what funding they would
need to develop the incubator. The attitude in the city changed when they saw
the possibilities associated with an incubator strategy.
Capital Access Projects
Projects in this category, in general, were designed to enhance access to capital
for entrepreneurs in the region. The programmatic goals were to create more
productive and job creating businesses and entrepreneurs who could remain in
the region. Based on ARC investment in this area, access to capital was
considered to be critical to helping entrepreneurs start and grow businesses in
the region. In addition to the EI investments in capital projects, ARC invested an
additional almost $4 million from area development, regional initiatives,
commission and co-chair funds. However, the projects were as diverse as the
sources of capital. A profile of capital access projects appears below.
Sample Capital Access Projects
Number of projects = 8
Total ARC investment = $901,340
Average ARC investment = $112,668
Minimum ARC investment = $30,000
Maximum ARC investment = $447,440
Range of years for project investment = 1 – 6
69
The geographic regions served by these projects ranged from individual
communities to multi-county and even multi-state service areas. While some
projects were one-time investments by ARC, others represented multi-year
commitments, with additional investments phased in as the project progressed.
All of the capital projects in the sample were judged to be sustainable by project
leaders, i.e., the projects continued beyond the period of the ARC grant.
ARC investments were used in diverse ways, including:
Support for research and planning to explore the creation of alternative
financial institutions, such as New Market Venture funds, Community
Development Financial Institutions (CDFIs), and angel capital networks
Support for technical assistance and outreach associated with revolving
loan funds
Investment in capital programs ranging from revolving loan funds and
microenterprise programs to equity funds.
Several examples serve to illustrate the ways in which many of these capital
projects were supported by ARC. Appalachian Community Enterprises (ACE), a
non-profit microenterprise program started in northeast Georgia in 1997, received
initial funding from ARC in 1999 to support a loan fund, including funds for
technical assistance and training. Similar follow-on grants were made to ACE in
subsequent years. Building on this base of activity and their track record of
lending, in 2004, ACE received CDFI certification and significant funding from
ARC’s Area Development program for their microenterprise loan program.
In 1998, the South Carolina Appalachian Council of Governments received a
modest initial investment from the EI to support the development of a loan fund
and technical assistance and outreach to rural entrepreneurs. Complementing
this EI investment was significant funding in 1998 under ARC’s Regional
Initiatives to recapitalize the loan fund. Additional support for the revolving loan
fund through ARC’s Area Development program was received in 2004.
In 2003, Advantage West, a regional economic development organization in
western North Carolina, received funding from ARC’s EI to develop a regional
entrepreneurship network. Funding was used to put on workshops throughout the
region, to develop a website, and to complete a market assessment for a
regional investment fund. During this time, Advantage West assisted a number of
entrepreneurs in acquiring capital from angel investors. A second grant was used
to develop a regional angel investor network that continues to operate in the
region.
REPORTED METRICS
From its inception in 1997 through 2005, ARC made almost $43 million in
entrepreneurship-related investments in the region, including investments made
70
specifically by the Entrepreneurship Initiative. As Table 6.2 shows, annual
investments increased from inception in 1997 to a peak in 2001, with gradual
reductions in annual investments since then. This pattern of investment
represents the phasing in and out of the EI and, in more recent years, the
inclusion of entrepreneurship investments in ARC’s Asset-Based Initiative. Total
investments were made from a number of sources within ARC, with the most
significant investment coming through the EI (Table 6.3). Total ARC investment,
in turn, leveraged an additional $72.8 million in private investment (for those
projects that have closed) with projected total private leveraged investment of
$109.9 million once all projects in the portfolio have closed.
Table 6.3. ARC Investments from All Sources – Universe of Projects, 1997-
2005
SOURCE OF FUNDS ARC $
Entrepreneurship Initiative $22,519,996
Area Development 11,603,420
Distressed Counties 3,722,992
Commission EI 2,126,380
CoChair Fund 1,553,887
Regional Initiatives 1,368,439
New Markets 52,574
Goal Fund 24,000
Total $42,971,688
ARC investments were made in 340 unique projects across the region (Table
6.4). The distribution of both dollars and projects varied across the region,
ranging from a high of 48 projects funded in Virginia to a low of nine projects
funded in South Carolina. On average, investment per state was $3.3 million and
investment per capita was $1.82 from 1997 through 2005.
In addition to projects funded in individual states, ARC made investments in a
number of projects that had a region-wide focus. For example, regional projects
included support for:
A microenterprise conference
The development of regional community development venture capital
funds, including support for technical assistance
A survey of Appalachian business incubators
The creation of entrepreneurship education materials, including working
with REAL and Junior Achievement
71
A regional conference on entrepreneurship education and training
A regional technology commercialization initiative
The development of sector-based entrepreneurship initiatives.
Table 6.4. Distribution of Entrepreneurship Projects Funded from All ARC
Sources, 1997-2005
*
STATE # CLOSED PROJECTS # OPEN PROJECTS # TOTAL
PROJECTS
Alabama 23 5 28
Georgia 7 3 10
Kentucky 22 9 31
Maryland 11 3 14
Mississippi 15 2 17
New York 17 4 21
North Carolina 5 6 11
Ohio 37 3 40
Pennsylvania 15 6 21
South Carolina 4 5 9
Tennessee 11 2 13
Virginia 42 6 48
West Virginia 24 13 37
Region 30 10 40
Total 263 77 340
*These totals include unique projects only.
Table 6.5 shows total and average investment in the universe of ARC projects
(both open and closed projects) as well as in the evaluation sample. Average
investment per project for the universe was $126,387, and projects ranged in
size from $2,000 to almost $2.2 million.
80
ARC invested almost $13 million in the
specific projects included in the evaluation sample, with average investment per
project of $145,997. Sample projects were drawn from those projects that were
primarily funded with Entrepreneurship Initiative dollars, as opposed to
80
The $2.2 million project investment was made in Pennsylvania, where the state chose to invest
in building regional assistance networks across the state. While this project was treated as a
single unique project, the funds flowed to regions across the state.
72
Commission EI or CoChair funds. As a result, some of the smaller projects, such
as Springboard Awards for Youth Entrepreneurship, were excluded from sample
selection providing a slight upward bias on project size. However, the range of
projects shows that the sample includes both small and large projects.
Table 6.5. Dollars Invested in Entrepreneurship Initiative Projects, from all
sources – Universe and Sample
ARC $
Universe of Projects
Total 42,971,688
Mean 126,387
Min 2,000
Max 2,177,326
Sample Projects
Total 12,847,733
Mean 145,997
Min 10,000
Max 2,177,326
ARC collected outcome data as part of the final reporting requirements for each
project. Table 6.6 shows actual and projected jobs created, jobs retained, new
businesses created, and businesses served for the universe of unique projects,
as reported in the close out documents submitted to ARC and included in the
database provided by ARC. In total, 9,156 jobs were created, 3,022 jobs
retained, 1,787 new businesses created, and 8,242 businesses served across
the region between 1997 and 2005. On average, projects created almost 27
jobs, retained almost 9 jobs, created 5 new businesses, and served 24
businesses. The 9,156 jobs were created at a cost, in terms of ARC funds
invested, of $4,693. As discussed in more detail in Chapter 7, this figure
compares favorably to other economic development efforts.
Table 6.7 shows these same metrics for those projects included in the evaluation
sample. Collectively, sample projects created 4,332 jobs, retained 1,351, created
1,083 new businesses, and served 2,957 businesses. On average, sample
projects achieved greater outcomes than the universe of projects in terms of job
creation (49), job retention (15), new business creation (12), and businesses
served (almost 34). This variation can be explained by two factors. First, the
universe of projects includes some of the smaller projects funded from ARC
sources other than the EI, likely reducing the overall impact numbers for the
73
universe. Second, the evaluation team was able to capture through follow-up
interviews those ongoing impacts achieved by projects that were sustainable
beyond the ARC investments. These job and business creation numbers,
therefore, represent a more accurate view of the impact of ARC investment than
those developed strictly from close out reports submitted to ARC.
CONCLUSIONS ABOUT OVERALL EI PORTFOLIO PERFORMANCE
The outcome data described in this chapter suggest that ARC investments have
been successful in generating jobs and businesses within the region. Actual jobs
created and retained and number of businesses served exceeded the projections
or goals established by the projects in their funding applications to ARC for the
sample of projects; only new business creation numbers fell short of
projections.
81
Sample data provide the most accurate view of the impact of ARC
investments since they reflect ongoing impacts associated with the projects.
Even considering all closed projects (where data are reported at project close out
only), ARC investments have created/retained more jobs and served more
businesses than projected.
81
It is not possible within the scope of this evaluation to determine whether the initial business
creation goals for the universe of projects were, in fact, realistic. It is possible that projects
identified business creation as a goal because it was a reporting metric for ARC, and not because
it was a realistic outcome of program investment. A deeper understanding of specific projects
would be required to address this issue. However, insights based on our analysis of the sample
projects are provided in Chapter 7.
7
4
T
a
b
l
e
6
.
6
J
o
b
C
r
e
a
t
i
o
n
a
n
d
R
e
t
e
n
t
i
o
n
,
N
e
w
B
u
s
i
n
e
s
s
C
r
e
a
t
i
o
n
,
a
n
d
B
u
s
i
n
e
s
s
e
s
S
e
r
v
e
d
f
o
r
t
h
e
U
n
i
v
e
r
s
e
o
f
P
r
o
j
e
c
t
s
*
J
O
B
S
C
R
E
A
T
E
D
A
C
T
U
A
L
J
O
B
S
C
R
E
A
T
E
D
P
R
O
J
E
C
T
E
D
J
O
B
S
R
E
T
A
I
N
E
D
A
C
T
U
A
L
J
O
B
S
R
E
T
A
I
N
E
D
P
R
O
J
E
C
T
E
D
N
E
W
B
U
S
I
N
E
S
S
E
S
C
R
E
A
T
E
D
A
C
T
U
A
L
N
E
W
B
U
S
I
N
E
S
S
E
S
C
R
E
A
T
E
D
P
R
O
J
E
C
T
E
D
B
U
S
I
N
E
S
S
E
S
S
E
R
V
E
D
A
C
T
U
A
L
B
U
S
I
N
E
S
S
E
S
S
E
R
V
E
D
P
R
O
J
E
C
T
E
D
U
n
i
v
e
r
s
e
(
3
4
0
)
T
o
t
a
l
9
,
1
5
6
1
6
,
1
9
6
3
,
0
2
2
2
,
3
8
5
1
,
7
8
7
3
,
2
0
7
8
,
2
4
2
1
2
,
0
2
5
M
e
a
n
2
6
.
9
4
7
.
6
8
.
9
7
.
0
5
.
3
9
.
4
2
4
.
2
3
5
.
4
M
i
n
0
0
0
0
0
0
0
0
M
a
x
5
2
8
1
,
0
0
0
4
3
0
3
7
2
2
6
4
1
,
0
0
0
2
,
7
1
7
2
,
5
0
0
*
T
h
e
s
e
d
a
t
a
a
r
e
d
r
a
w
n
f
r
o
m
t
h
e
A
R
C
d
a
t
a
b
a
s
e
a
n
d
d
o
n
o
t
r
e
f
l
e
c
t
a
d
d
i
t
i
o
n
a
l
j
o
b
c
r
e
a
t
i
o
n
/
r
e
t
e
n
t
i
o
n
,
b
u
s
i
n
e
s
s
c
r
e
a
t
i
o
n
a
n
d
b
u
s
i
n
e
s
s
e
s
s
e
r
v
e
d
t
h
a
t
m
a
y
h
a
v
e
o
c
c
u
r
r
e
d
a
s
p
r
o
j
e
c
t
s
c
o
n
t
i
n
u
e
d
b
e
y
o
n
d
t
h
e
p
e
r
i
o
d
o
f
t
h
e
A
R
C
g
r
a
n
t
.
T
a
b
l
e
6
.
7
.
J
o
b
C
r
e
a
t
i
o
n
a
n
d
R
e
t
e
n
t
i
o
n
,
N
e
w
B
u
s
i
n
e
s
s
C
r
e
a
t
i
o
n
,
a
n
d
B
u
s
i
n
e
s
s
e
s
S
e
r
v
e
d
f
o
r
t
h
e
S
a
m
p
l
e
o
f
P
r
o
j
e
c
t
s
*
J
O
B
S
C
R
E
A
T
E
D
A
C
T
U
A
L
J
O
B
S
C
R
E
A
T
E
D
P
R
O
J
E
C
T
E
D
J
O
B
S
R
E
T
A
I
N
E
D
A
C
T
U
A
L
J
O
B
S
R
E
T
A
I
N
E
D
P
R
O
J
E
C
T
E
D
N
E
W
B
U
S
I
N
E
S
S
E
S
C
R
E
A
T
E
D
A
C
T
U
A
L
N
E
W
B
U
S
I
N
E
S
S
E
S
C
R
E
A
T
E
D
P
R
O
J
E
C
T
E
D
B
U
S
I
N
E
S
S
E
S
S
E
R
V
E
D
A
C
T
U
A
L
B
U
S
I
N
E
S
S
E
S
S
E
R
V
E
D
P
R
O
J
E
C
T
E
D
S
a
m
p
l
e
(
8
8
)
T
o
t
a
l
4
,
3
3
2
2
,
9
3
7
1
,
3
5
1
4
4
3
1
,
0
8
3
9
8
1
2
,
9
5
7
2
,
7
5
7
M
e
a
n
4
9
.
2
3
3
.
4
1
5
.
4
5
.
0
1
2
.
3
1
1
.
2
3
3
.
6
3
1
.
3
M
i
n
0
0
0
0
0
0
0
0
M
a
x
7
5
8
4
3
4
4
1
0
0
3
1
6
2
5
1
6
4
3
6
9
5
*
T
h
e
s
e
d
a
t
a
a
r
e
d
r
a
w
n
f
r
o
m
f
o
l
l
o
w
-
u
p
i
n
t
e
r
v
i
e
w
s
w
i
t
h
p
r
o
j
e
c
t
l
e
a
d
e
r
s
a
n
d
d
o
r
e
f
l
e
c
t
a
d
d
i
t
i
o
n
a
l
j
o
b
c
r
e
a
t
i
o
n
/
r
e
t
e
n
t
i
o
n
,
b
u
s
i
n
e
s
s
c
r
e
a
t
i
o
n
a
n
d
b
u
s
i
n
e
s
s
e
s
s
e
r
v
e
d
t
h
a
t
o
c
c
u
r
r
e
d
a
s
p
r
o
j
e
c
t
s
c
o
n
t
i
n
u
e
d
b
e
y
o
n
d
t
h
e
p
e
r
i
o
d
o
f
t
h
e
A
R
C
g
r
a
n
t
.
75
CHAPTER 7
SAMPLE PROJECT OUTCOMES AND
BROADER POLICY IMPACTS
Chapter 6 reported on select metrics for the entire portfolio of projects in which
ARC invested; these data describe one level of impact the EI had on the region.
However, entrepreneurship development strategies, including those funded by
ARC, are often designed to achieve goals that extend beyond job and business
creation. In Chapter 6, the evaluation team identified an underlying set of
programmatic goals related to the conceptual model that were attributed to each
project type. In reality, each project was designed to address a unique set of
goals that were relevant to a particular rural place. To understand the full range
of project impacts, and to assess whether these projects were achieving their
goals, it was necessary to do a more in-depth evaluation of a sample of projects.
This part of the evaluation of ARC’s EI has generated a series of findings that fall
into two main categories – those related to the outcomes achieved by the
representative sample of projects in which ARC invested and those related to the
broader policy impacts on the Appalachian region associated with EI’s entire
portfolio of investments.
IMPACTS OF EI FUNDED PROJECTS
These findings are based on detailed analysis of project outcomes associated
with a sample of 88 projects that reflects the diversity, geographic reach, and
scope of the EI. As described in Chapter 4, the sample was representative of the
universe of projects receiving ARC investments between 1997 and 2005 in terms
of both program type and state.
82
Quantitative Impacts
To understand the quantitative impacts of ARC investments, the evaluation team
began by identifying three classes of metrics that were common to most projects
within each program category (Table 7.1). One class included common goals that
were articulated in project proposals but for which no outcome measures were
provided by project leaders. These are metrics that were believed to be important
during the design of the project but, for some reason, no outcome data were
collected for most projects in the category. In some cases, it may have been a
matter of definition – data collected as part of the project were not defined using
the same terminology as project goals. For example, technical assistance
providers often stated a goal of “number of businesses served” but actually
82
While the sample was judged to be representative, the evaluation team is not suggesting that
the quantitative results of this evaluation should be extrapolated to provide an estimate of overall
quantitative impacts of the EI. Rather, outcomes associated with these specific projects should be
viewed as reflective of what other projects might produce given similar capacity, assets,
leadership, etc.
76
reported “number of clients”. However, in some cases, it appears that data
considered relevant when the project was designed were not collected. For
example, entrepreneurship education providers often stated a goal of “number of
jobs created” but did not report such data as an outcome measure. While these
metrics do not provide insight into the overall performance of the EI, they were
useful in considering the development of the “best in class” metrics system
described in Chapter 9.
Table 7.1. Metrics Common within Program Categories
PROGRAM
CATEGORY
PROJECT
METRICS
INCLUDED AS
STATED
PROJECT GOALS
ONLY
PROJECT METRICS
INCLUDED AS
MEASURED
PROJECT
OUTCOMES ONLY
PROJECT METRICS
INCLUDED AS BOTH
STATED PROJECT
GOALS AND
MEASURED PROJECT
OUTCOMES
Capital Access - Businesses
served
- Businesses
created
- Businesses
created/expanded
- Jobs created/retained
Entrepreneurship
Education
- Jobs created - Number of
participants/trainees (both
students and teachers)
- Number of business
starts (student or adult)
Sectors - Businesses
served
- Number of
participants/trainees
- Businesses
created/expanded
- Jobs created/retained
Incubators - Businesses
created
- Businesses
served
- Number of current
clients
- Total clients served
- Number of graduated
firms
- Jobs created/retained
- Complete incubator
feasibility study
Technical
Assistance and
Training
- Businesses
served
- Number of
business plans
created/assisted
- Number of
trainings, seminars,
conferences
- Businesses expanded
- Number of clients
- Number of clients
retained in service area
- Business created
- Jobs created/retained
A second class of metrics included measured and reported project outcomes that
did not appear among the goals articulated for most projects in the category.
These are metrics that likely appeared relevant only after a project was
implemented or that were measured in a way that was different from the original
project goals. For example, most capital projects reported the number of
“businesses created and expanded” but had set a goal that related only to
“businesses created”. And, incubator projects generally reported “graduated
firms” as an outcome measure, but did not always include this metric as a project
goal. These measures help describe the impact of ARC investments, but are not
useful in evaluating the success of projects in achieving stated goals.
77
The third class of metrics included those that were both stated project goals and
measured and reported project outcomes. These metrics provide a means of
measuring the “success” of the portfolio of projects included within each program
category as will be described in more detail below.
Table 7.2
83
provides initial quantitative results for metrics that were common
within program categories. These results provide insight into the range of
measurable impacts associated with EI investments. Entrepreneurship education
projects, targeted to youth, exposed 11,634 students and teachers to
entrepreneurship principles. Almost 1,500 entrepreneurs participated in sector-
specific activities. Incubators served 475 clients and graduated at least 15 firms.
Training and technical assistance were provided to 1,620 entrepreneurs in the
region. While these data suggest positive outcomes for EI investments, it is
difficult to quantify these results for the entire EI portfolio of investments since
metrics varied across project types.
Some measures were found to be common across categories – jobs
created/retained and businesses created/expanded.
84
Table 7.3 provides initial
quantitative results for these metrics. These results show that EI investments did
produce positive quantifiable results in terms of both job and business creation –
metrics most often reported in evaluations of economic development projects. EI
investments in sample projects created or retained 5,339 jobs, created 248,
expanded 39, and created or expanded 324 businesses.
While the evaluation team cautions against using single performance measures
to gauge the success and impact of ARC’s EI, there is value in placing these
entrepreneurship development investments within the context of more traditional
economic development metrics – specifically, cost per job created/retained and
cost per business created/expanded. Table 7.4 presents estimates of these costs
for the ARC sample projects as well as for similar types of business development
programs. ARC public costs per job or business created compare favorably with
program investments made in a variety of similar types of programs. With the
83
Job creation/retention and business creation numbers included in Tables 7.2 and 7.3 in this
chapter differ slightly from those reported in Chapter 6, Table 6.7. Data in this chapter are more
conservative because some individual projects were excluded from this analysis if they did not
report both measured and projected outcomes. In Chapter 6, totals were calculated across the
entire sample as a collective and, therefore, no observations were dropped.
84
As with any data collection effort, there are caveats that must be stated. In conducting the
follow-up interviews with project leaders, most reported jobs created and retained as a single
category rather than separate categories. The evaluation team has chosen to report this
combined category since it was not possible for most respondents to distinguish between new
and retained jobs. In addition, one can consider the impact of a job retained within a community
or region as being equivalent, from an economic development perspective, to a new job created.
A similar caveat applies to businesses created and businesses expanded. Depending upon
program type, some projects reported combined metrics – businesses created and/or expanded –
while others, such as incubators, reported only business creation numbers. We have chosen to
report these categories as they were reported during interviews with project leaders.
78
exception of incubator costs, ARC investments produced jobs and businesses at
lower public cost than other types of investments. However, it is important to
recognize that these ARC figures reflect the cost in terms of ARC dollars and not
total dollars invested in these projects. Given that sample projects leveraged
$1.20 in private investments per $1 of ARC investments, these costs are
understated by about half. However, even inflating these cost figures to reflect
total investments shows that ARC investments compare well against similar
investments. As importantly, these job creation cost figures are relatively small
compared to the average cost per job created via industrial recruitment strategies
which can range anywhere from $7,000-$15,000 per job on an annual basis.
85
While positive, these metrics alone cannot answer the question of whether EI
projects have achieved their objectives. The evaluation team considered two
factors in evaluating the overall success of the initiative:
Whether project leaders considered the project to be successful
Whether measured outcomes exceeded stated goals (for quantitative
measures).
To assess the first factor, project leaders were asked whether they considered
the project a success, whether the project had met stated objectives, and
whether the project was sustainable over time, i.e., beyond the ARC grant period.
The evaluation team felt justified in using this key informant information since
there was no compelling reason why project leaders would not be objective in
evaluating success. For most projects, ARC funding had ended and project
leaders had nothing to gain by being overly optimistic about project results. In
addition, ARC has not continued to fund the EI as a separate initiative, so project
leaders had no incentive to overestimate impacts. Finally, project leaders were
informed that the evaluation was about the overall impacts of EI projects and not
about the individual performance of any single project. In addition, during the
interviews, project leaders shared information about both the successes and
failures associated with their activities.
Table 7.5 shows that, overwhelmingly, project leaders considered projects to be
successful and to have achieved program objectives, as determined through
follow-up interviews. Project leaders generally defined success in terms of
accomplishing the goals set out at the beginning of the project. However, they
were also likely to consider a project that had achieved some, but not all, of its
objectives as being successful. And, they were less concerned about the extent
to which a goal was met, as long as they saw some accomplishment toward the
goal. For example, oftentimes a project manager would consider a goal of
“creating 50 jobs” as being met if they had created some but not all of those jobs.
85
Peter Fisher, “The Fiscal Consequences of Competition for Capital,” Reining in the Competition
for Capital, Ed. Ann Markusen (Kalamazoo, MI: WE Upjohn Institute, 2007).
79
If project goals are used as a benchmark for assessing project success, it
becomes important to assess whether project goals, as initially defined, were
attainable. That is, were project goals reasonable and realistic? Unfortunately, it
is not possible through this ex post evaluation to determine on a project by
project basis whether goals were attainable. However, interviews with project
leaders suggest that project goals did change over time, often being revised as
the project was implemented and the overall direction refined. For instance,
some project leaders indicated that they learned over time that their goals had
been too ambitious and that they did not realize how difficult implementing the
project would be. In addition, there were qualitative results that contributed to the
project leader’s view that the project was successful, but which did not relate to
the original goals of the project. In both cases, project leaders had legitimate
reasons for declaring the project a success, based on their personal experience
and understanding of the broad set of impacts achieved.
Project leaders were generally objective in identifying the sustainability of their
projects – defined simply as whether or not the project continued beyond the
period of the ARC grant. It is important to recognize that sustainability was
defined in terms of the project itself and not in terms of the businesses or jobs
that were created as a result. Again, there appeared to be no incentive for these
leaders to overestimate sustainability and the responses to these questions
suggest that project leaders evaluated sustainability more critically than they did
success. Even when some part of a project proved to be sustainable, project
leaders most often defined the project as “not sustainable” if the most substantive
parts of the project did not continue.
86
While the perception of project leaders is an important factor to consider in
evaluating the success of the EI, follow-up interviews also generated data on
both stated goals and measured outcomes for a set of metrics. These metrics
provide quantitative information to assess the performance of the initiative. While
individual projects experienced varying levels of success in achieving stated
goals, the evaluation team chose to view the success of the EI from a portfolio
perspective, i.e., data are reported for each program category rather than for
each individual project within that category. From this portfolio perspective, it is
clear that the initiative was successful in achieving most of the common goals
identified for each program category.
Table 7.6 shows that for six of nine metrics, the stated goal was exceeded or
met. For the other three variables, the stated goal was not met. However, it is
useful to consider each of these variables in more detail. For entrepreneurship
education projects, a stated goal was new business starts. Given our original
conceptual model, the primary programmatic goal for these types of projects was
to get more entrepreneurs into the pipeline – to build entrepreneurial skills in
86
It is important to note that most of the projects that were sustainable beyond the ARC grant
continued to rely on local, state, foundation and other support. Very few of these projects were
sustainable defined as producing income sufficient to cover the operating costs of a program.
80
young people and expose them to entrepreneurship as a potential career path.
The literature is quite clear that entrepreneurship education for youth is not about
creating businesses in the short run. It is about inspiring young people to develop
skills that can lead them toward entrepreneurship in the future. For ARC’s EI,
metrics were reported for entrepreneurship education projects that lasted from
one to four years, as described in Chapter 6. Creating fewer new business starts
than projected may have more to do with a misalignment of goals and metrics
than with lack of success for this set of programs. Following students who
participated in these ARC funded projects as they advance through school and
into a career path would likely generate more accurate metrics on business
creation than could be expected from this initial, short-term glimpse of project
impacts. Unfortunately, none of these projects provided that long-term follow up.
In terms of incubator performance, one can argue that incubators are established
to help create more informed and more highly skilled entrepreneurs whose
businesses, as a result are more productive and more likely to remain in the
region. These goals could very well be achieved with no impact on job creation,
at least in the short run. In addition, the range of activities included in the
incubator program category – from grants for feasibility studies to implementation
grants – may skew job creation figures. Capacity building projects such as an
incubator feasibility study are unlikely to have any measurable impact on job and
business creation.
Similarly, one can argue that technical assistance and training projects serve to
create better skilled and informed entrepreneurs who, in turn, improve the
performance of existing businesses. While some aspiring entrepreneurs may
create businesses as a result of the technical assistance and training they
receive, new business creation is not the primary goal of many technical
assistance providers, such as some Small Business Development Centers that
work mostly with existing business owners.
The leveraging of private investment is a final quantitative measure that tracks
local support of entrepreneurship. ARC investments were described by key
stakeholders as catalytic. The ability to attract private investment, i.e., leverage,
as a result of ARC investment is one measure of the impact that these funded
projects have had on the region. Table 7.7 presents actual and projected total
private leverage for all ARC projects and sample projects. Projected leverage
includes the private investment associated with both closed and open projects,
suggesting the longer term impact associated with ARC investments. For the
ARC portfolio as a whole, private leverage rates range from 1.7:1 (actual) to
2.6:1 (projected). For the sample, leverage rates range from 1.2:1 (actual) to
1.4:1 (projected).
87
Collectively, ARC and leveraged private investments have
87
The discrepancy in leveraging rates between the universe of projects and our sample results
from four revolving loan fund projects that were not included in the sample but leveraged almost
$32 million in private investment.
81
had an impact by creating the quantitative and qualitative impacts described in
this chapter.
Conclusions Regarding Quantitative Impacts
Based on analysis of quantitative metrics for the evaluation sample, ARC’s
Entrepreneurship Initiative has had an impact on the region. Collectively, sample
projects created or retained over 5,300 jobs and created or expanded over 600
businesses. While it is not possible to determine definitively whether these jobs
would have been created without ARC investment, project leaders reported these
outcomes and indicated that ARC investment was critical to achieving these
impacts.
Business and job creation numbers tell only part of the story. Over 11,500
students and teachers participated in or received training from the sample
entrepreneurship education projects in which ARC invested. Almost 1,500
entrepreneurs participated in sector-focused activities. Another 1,620 received
training and technical assistance, while 475 were served by incubators. Every
dollar invested by ARC in these sample projects leveraged $1.20 in actual private
investment and is projected to leverage $1.40 private dollars for every ARC dollar
invested when project investments are complete. And, project leaders and others
in the region identified a number of qualitative impacts from these investments
that are having far reaching consequences for the Appalachian region. In
addition to supporting the conclusion that the EI has had an important positive
impact on the region, these observations also suggest that a portfolio of
programs like ARC’s EI requires a “portfolio of metrics” to accurately tell the story
of program impact.
8
2
T
a
b
l
e
7
.
2
.
Q
u
a
n
t
i
t
a
t
i
v
e
R
e
s
u
l
t
s
f
o
r
M
e
t
r
i
c
s
C
o
m
m
o
n
w
i
t
h
i
n
P
r
o
g
r
a
m
C
a
t
e
g
o
r
i
e
s
8
8
C
A
P
I
T
A
L
A
C
C
E
S
S
E
N
T
R
E
P
R
E
N
E
U
R
S
H
I
P
E
D
U
C
A
T
I
O
N
S
E
C
T
O
R
S
I
N
C
U
B
A
T
O
R
S
T
E
C
H
N
I
C
A
L
A
S
S
I
S
T
A
N
C
E
A
N
D
T
R
A
I
N
I
N
G
M
e
t
r
i
c
#
M
e
t
r
i
c
#
M
e
t
r
i
c
#
M
e
t
r
i
c
#
M
e
t
r
i
c
#
-
B
u
s
i
n
e
s
s
e
s
c
r
e
a
t
e
d
/
e
x
p
a
n
d
e
d
2
2
3
-
J
o
b
s
c
r
e
a
t
e
d
/
r
e
t
a
i
n
e
d
1
,
2
2
9
-
#
o
f
p
a
r
t
i
c
i
p
a
n
t
s
/
t
r
a
i
n
e
e
s
(
s
t
u
d
e
n
t
s
o
r
t
e
a
c
h
e
r
s
)
1
1
,
6
3
4
-
N
e
w
b
u
s
i
n
e
s
s
s
t
a
r
t
s
(
s
t
u
d
e
n
t
o
r
a
d
u
l
t
)
8
5
-
#
o
f
p
a
r
t
i
c
i
p
a
n
t
s
/
t
r
a
i
n
e
e
s
1
,
4
9
7
-
B
u
s
i
n
e
s
s
c
r
e
a
t
e
d
/
e
x
p
a
n
d
e
d
1
0
1
-
J
o
b
s
c
r
e
a
t
e
d
/
r
e
t
a
i
n
e
d
9
9
4
-
#
c
u
r
r
e
n
t
c
l
i
e
n
t
s
1
1
4
-
T
o
t
a
l
c
l
i
e
n
t
s
s
e
r
v
e
d
4
7
5
-
#
g
r
a
d
u
a
t
e
d
f
i
r
m
s
1
5
-
J
o
b
s
c
r
e
a
t
e
d
/
r
e
t
a
i
n
e
d
1
3
0
-
C
o
m
p
l
e
t
e
i
n
c
u
b
a
t
o
r
f
e
a
s
i
b
i
l
i
t
y
s
t
u
d
y
3
-
#
b
u
s
i
n
e
s
s
e
x
p
a
n
d
e
d
3
9
-
#
o
f
c
l
i
e
n
t
s
1
,
6
2
0
-
#
o
f
c
l
i
e
n
t
s
r
e
m
a
i
n
i
n
g
i
n
s
e
r
v
i
c
e
a
r
e
a
1
1
4
-
B
u
s
i
n
e
s
s
e
s
c
r
e
a
t
e
d
1
6
3
-
J
o
b
s
c
r
e
a
t
e
d
/
r
e
t
a
i
n
e
d
2
,
9
8
6
T
a
b
l
e
7
.
3
.
Q
u
a
n
t
i
t
a
t
i
v
e
R
e
s
u
l
t
s
f
o
r
M
e
t
r
i
c
s
C
o
m
m
o
n
a
c
r
o
s
s
P
r
o
g
r
a
m
C
a
t
e
g
o
r
i
e
s
P
R
O
G
R
A
M
C
A
T
E
G
O
R
Y
J
O
B
S
C
R
E
A
T
E
D
/
R
E
T
A
I
N
E
D
B
U
S
I
N
E
S
S
E
S
C
R
E
A
T
E
D
B
U
S
I
N
E
S
S
E
S
E
X
P
A
N
D
E
D
B
U
S
I
N
E
S
S
E
S
C
R
E
A
T
E
D
/
E
X
P
A
N
D
E
D
C
a
p
i
t
a
l
A
c
c
e
s
s
1
2
2
9
-
-
-
-
-
-
2
2
3
E
n
t
r
e
p
r
e
n
e
u
r
s
h
i
p
E
d
u
c
a
t
i
o
n
-
-
-
8
5
-
-
-
-
-
-
S
e
c
t
o
r
s
9
9
4
-
-
-
-
-
-
1
0
1
I
n
c
u
b
a
t
o
r
s
1
3
0
-
-
-
-
-
-
-
-
-
T
e
c
h
n
i
c
a
l
A
s
s
i
s
t
a
n
c
e
a
n
d
T
r
a
i
n
i
n
g
2
9
8
6
1
6
3
3
9
-
-
-
T
O
T
A
L
5
3
3
9
2
4
8
3
9
3
2
4
8
8
O
n
l
y
t
h
o
s
e
m
e
t
r
i
c
s
t
h
a
t
w
e
r
e
r
e
p
o
r
t
e
d
a
s
p
r
o
j
e
c
t
o
u
t
c
o
m
e
s
a
r
e
i
n
c
l
u
d
e
d
i
n
t
h
i
s
t
a
b
l
e
.
8
3
T
A
B
L
E
7
.
4
.
C
O
M
P
A
R
I
S
O
N
O
F
C
O
S
T
P
E
R
J
O
B
C
R
E
A
T
E
D
/
R
E
T
A
I
N
E
D
A
N
D
B
U
S
I
N
E
S
S
C
R
E
A
T
E
D
/
E
X
P
A
N
D
E
D
–
A
R
C
S
A
M
P
L
E
P
R
O
J
E
C
T
S
A
N
D
O
T
H
E
R
R
E
P
O
R
T
E
D
P
R
O
J
E
C
T
S
A
R
C
S
A
M
P
L
E
P
R
O
J
E
C
T
S
A
E
D
A
P
R
O
J
E
C
T
S
P
E
N
N
S
Y
L
V
A
N
I
A
S
M
A
L
L
B
U
S
I
N
E
S
S
D
E
V
E
L
O
P
M
E
N
T
C
E
N
T
E
R
S
G
I
N
C
U
B
A
T
O
R
S
H
E
N
T
E
R
P
R
I
S
E
F
A
C
I
L
I
T
A
T
I
O
N
I
N
K
A
N
S
A
S
I
S
B
A
M
I
C
R
O
L
O
A
N
P
R
O
G
R
A
M
J
$
p
e
r
j
o
b
c
r
e
a
t
e
d
/
r
e
t
a
i
n
e
d
:
C
a
p
i
t
a
l
A
c
c
e
s
s
-
6
2
4
S
e
c
t
o
r
s
-
7
3
2
T
e
c
h
n
i
c
a
l
A
s
s
i
s
t
a
n
c
e
-
5
7
9
I
n
c
u
b
a
t
o
r
s
-
3
,
9
9
4
$
p
e
r
b
u
s
i
n
e
s
s
c
r
e
a
t
e
d
/
e
x
p
a
n
d
e
d
:
C
a
p
i
t
a
l
A
c
c
e
s
s
-
2
,
9
8
8
S
e
c
t
o
r
s
-
3
,
7
5
9
T
e
c
h
n
i
c
a
l
A
s
s
i
s
t
a
n
c
e
-
7
,
8
1
8
$
p
e
r
j
o
b
c
r
e
a
t
e
d
:
R
e
v
o
l
v
i
n
g
L
o
a
n
F
u
n
d
s
b
-
9
3
6
R
u
r
a
l
P
r
o
j
e
c
t
s
c
-
6
,
1
5
7
U
r
b
a
n
P
r
o
j
e
c
t
s
d
-
2
,
9
8
2
P
u
b
l
i
c
W
o
r
k
s
e
-
4
,
8
5
7
D
e
f
e
n
s
e
A
d
j
u
s
t
m
e
n
t
f
-
1
2
,
0
0
0
$
p
e
r
j
o
b
c
r
e
a
t
e
d
-
9
2
9
$
p
e
r
b
u
s
i
n
e
s
s
c
r
e
a
t
e
d
-
3
,
3
0
0
$
p
e
r
j
o
b
c
r
e
a
t
e
d
-
1
,
1
0
0
$
p
e
r
j
o
b
c
r
e
a
t
e
d
-
2
,
4
0
0
–
7
,
8
0
0
$
p
e
r
j
o
b
c
r
e
a
t
e
d
-
4
,
5
6
8
a
F
i
g
u
r
e
s
a
r
e
p
u
b
l
i
c
c
o
s
t
p
e
r
j
o
b
,
c
o
u
n
t
i
n
g
A
R
C
p
r
o
j
e
c
t
i
n
v
e
s
t
m
e
n
t
s
o
n
l
y
,
n
o
t
t
o
t
a
l
p
r
o
j
e
c
t
i
n
v
e
s
t
m
e
n
t
i
n
c
l
u
d
i
n
g
l
e
v
e
r
a
g
e
d
p
r
i
v
a
t
e
f
u
n
d
s
.
b
F
i
g
u
r
e
s
a
r
e
f
o
r
E
D
A
p
r
o
j
e
c
t
i
n
v
e
s
t
m
e
n
t
s
o
n
l
y
,
n
o
t
t
o
t
a
l
p
r
o
j
e
c
t
i
n
v
e
s
t
m
e
n
t
.
S
o
u
r
c
e
:
R
o
b
e
r
t
W
.
B
u
r
c
h
e
l
l
,
E
D
A
R
L
F
s
:
P
e
r
f
o
r
m
a
n
c
e
E
v
a
l
u
a
t
i
o
n
(
W
a
s
h
i
n
g
t
o
n
,
D
C
:
U
.
S
.
E
c
o
n
o
m
i
c
D
e
v
e
l
o
p
m
e
n
t
A
d
m
i
n
i
s
t
r
a
t
i
o
n
,
2
0
0
2
)
.
c
S
o
u
r
c
e
:
A
m
y
K
.
G
l
a
s
m
e
i
e
r
,
C
o
s
t
p
e
r
J
o
b
A
s
s
o
c
i
a
t
e
d
w
i
t
h
E
D
A
I
n
v
e
s
t
m
e
n
t
s
i
n
U
r
b
a
n
a
n
d
R
u
r
a
l
A
r
e
a
s
(
W
a
s
h
i
n
g
t
o
n
,
D
C
:
U
.
S
.
E
c
o
n
o
m
i
c
D
e
v
e
l
o
p
m
e
n
t
A
d
m
i
n
i
s
t
r
a
t
i
o
n
,
2
0
0
2
)
.
d
S
o
u
r
c
e
:
G
l
a
s
m
e
i
e
r
(
2
0
0
2
)
.
e
S
o
u
r
c
e
:
R
o
b
e
r
t
W
.
B
u
r
c
h
e
l
l
,
E
D
A
P
u
b
l
i
c
W
o
r
k
s
P
r
o
g
r
a
m
:
P
e
r
f
o
r
m
a
n
c
e
E
v
a
l
u
a
t
i
o
n
(
W
a
s
h
i
n
g
t
o
n
,
D
C
:
U
.
S
.
E
c
o
n
o
m
i
c
D
e
v
e
l
o
p
m
e
n
t
A
d
m
i
n
i
s
t
r
a
t
i
o
n
,
1
9
9
7
)
.
f
S
o
u
r
c
e
:
R
o
b
e
r
t
W
.
B
u
r
c
h
e
l
l
,
E
D
A
D
e
f
e
n
s
e
A
d
j
u
s
t
m
e
n
t
P
r
o
g
r
a
m
:
P
e
r
f
o
r
m
a
n
c
e
E
v
a
l
u
a
t
i
o
n
(
W
a
s
h
i
n
g
t
o
n
,
D
C
:
U
.
S
.
E
c
o
n
o
m
i
c
D
e
v
e
l
o
p
m
e
n
t
A
d
m
i
n
i
s
t
r
a
t
i
o
n
,
1
9
9
7
)
.
g
S
o
u
r
c
e
:
P
e
n
n
s
y
l
v
a
n
i
a
S
m
a
l
l
B
u
s
i
n
e
s
s
D
e
v
e
l
o
p
m
e
n
t
C
e
n
t
e
r
s
,
“
R
e
t
u
r
n
o
n
I
n
v
e
s
t
m
e
n
t
:
2
0
0
5
”
1
3
D
e
c
e
m
b
e
r
2
0
0
7
<
h
t
t
p
:
/
/
w
w
w
.
p
a
s
b
d
c
.
o
r
g
/
d
o
w
n
l
o
a
d
s
/
p
d
f
/
i
m
p
a
c
t
.
p
d
f
>
.
h
S
o
u
r
c
e
:
L
a
w
r
e
n
c
e
M
o
l
n
a
r
,
e
t
a
l
.
,
I
m
p
a
c
t
o
f
I
n
c
u
b
a
t
o
r
I
n
v
e
s
t
m
e
n
t
s
(
A
t
h
e
n
s
,
O
H
:
N
a
t
i
o
n
a
l
B
u
s
i
n
e
s
s
I
n
c
u
b
a
t
i
o
n
A
s
s
o
c
i
a
t
i
o
n
,
1
9
9
7
)
.
i
S
o
u
r
c
e
:
D
o
n
M
a
c
k
e
a
n
d
D
e
b
o
r
a
h
M
a
r
k
l
e
y
a
n
d
E
r
i
k
P
a
g
e
s
,
“
E
n
t
e
r
p
r
i
s
e
F
a
c
i
l
i
a
t
i
o
n
®
i
n
K
a
n
s
a
s
:
L
e
s
s
o
n
s
a
n
d
R
e
c
o
m
m
e
n
d
a
t
i
o
n
s
,
”
R
U
P
R
I
C
e
n
t
e
r
f
o
r
R
u
r
a
l
E
n
t
r
e
p
r
e
n
e
u
r
s
h
i
p
,
A
u
g
u
s
t
2
0
0
5
.
j
F
i
g
u
r
e
s
i
n
c
l
u
d
e
a
c
t
u
a
l
d
o
l
l
a
r
s
l
o
a
n
e
d
p
l
u
s
c
o
s
t
o
f
t
e
c
h
n
i
c
a
l
a
s
s
i
s
t
a
n
c
e
p
e
r
j
o
b
c
r
e
a
t
e
d
.
S
o
u
r
c
e
:
A
s
s
o
c
i
a
t
i
o
n
f
o
r
E
n
t
e
r
p
r
i
s
e
O
p
p
o
r
t
u
n
i
t
y
,
T
e
s
t
i
m
o
n
y
b
y
K
e
v
i
n
K
e
l
l
y
,
M
a
n
a
g
i
n
g
D
i
r
e
c
t
o
r
f
o
r
P
o
l
i
c
y
a
n
d
A
d
v
o
c
a
c
y
,
b
e
f
o
r
e
t
h
e
H
o
u
s
e
C
o
m
m
i
t
t
e
e
o
n
S
m
a
l
l
B
u
s
i
n
e
s
s
,
S
e
p
t
e
m
b
e
r
2
7
,
2
0
0
7
.
84
Table 7.5. Project Leaders’ Perceptions of Project Success and
Sustainability
QUESTION # RESPONDING
“YES”
% RESPONDING
“YES”
Did you think the project was a success? 87 89.7
Do you feel you achieved the objectives set
forth for this project?
85
92.9
Has the project continued after ARC funding
ended?
86
79.1
Table 7.6. Measured Success of ARC’s Entrepreneurship Initiative Portfolio
PROGRAM
CATEGORY
STATED GOAL MEASURED
OUTCOME
PERFORMANCE
(=/-/+)
Capital Access 79 jobs created/
retained
1,229 jobs
created/retained
+
Entrepreneurship
Education
4,483
participants/trainees
(students or teachers)
141 new business
starts
11,634
participants/trainees
(students or teachers)
85 new business
starts
+
-
Sectors 38 businesses
created/expanded
438 jobs
created/retained
101 businesses
created/expanded
994 jobs
created/retained
+
+
Incubators 162 jobs
created/retained
3 completed incubator
feasibility studies
130 jobs
created/retained
3 completed incubator
feasibility studies
-
=
Technical
assistance and
training
177 businesses
created
1,295 jobs
created/retained
163 businesses
created
2,986 jobs
created/retained
-
+
Table 7.7. Actual and Projected Leveraged Private Investment – Universe
and Sample Projects
ARC $ ACTUAL
LEVERAGED
PRIVATE $
PROJECTED
LEVERAGED
PRIVATE $
Universe 42,971,688 72,802,868 109,879,064
Sample 12,847,733 15,856,275 18,596,174
85
QUALITATIVE IMPACTS
There is an emerging consensus among practitioners, particularly in rural places,
that entrepreneurship development is a long-term economic development
strategy. It requires a cultural shift from a mindset of dependency to one of self-
sufficiency. In other words, this requires a change in the view that growth will only
come from decisions and investments that are controlled from outside the
community to one that growth will come from encouraging homegrown
entrepreneurs and the businesses they create. ARC’s Entrepreneurship Initiative
was designed to encourage this culture change and to build capacity within the
region to become more entrepreneurial. One might expect, therefore, that the
outcomes achieved through EI investments would extend beyond job and
business creation. Interviews with project leaders and stakeholders across the
region identified a set of cross-cutting qualitative outcomes, described below, that
suggest a much broader impact on the region than that described through the
analysis of quantitative metrics.
ARC investments raised the profile of entrepreneurship as a
development strategy, helping to change the mindset within the
region. The investments made by the EI helped to raise awareness of
entrepreneurship and to give credibility to entrepreneurship as an
economic development approach. The EI projects “opened people’s eyes
to other possibilities” and showed that there was “more to economic
development than infrastructure”. According to one interviewee, the “most
effective part was the fact that ARC recognized entrepreneurship as
economic development. That, in itself, was a major step.”
ARC investments represented “but for” money in the region,
providing start-up funding for innovative projects. The EI projects
often represented “outside the box” thinking or demonstration projects that
would not have gotten off the ground “but for” the ARC investment. In
some places, these projects served to prove a concept or approach that
then attracted additional investment. ARC investments were variously
described as being “catalytic” or “foundational” to the efforts to encourage
entrepreneurship in the region.
ARC investments leveraged additional resources that helped some
projects achieve scale and impact. As demonstrated in Table 7.7,
ARC’s EI portfolio leveraged significant private investment. To the extent
that these private funds represented resources new to the region, the
impact would indeed be positive. It is possible, however, that some of
these investments represent a reallocation of capital from one use to
another in the region. However, in the case of capital access, ARC
investment leveraged additional resources that helped to create an
industry, in this case, the venture capital industry in the region.
ARC investments facilitated networking and collaboration among
practitioners. The increased focus on entrepreneurship helped to
reinforce that practitioners were doing “good work” and served to connect
86
both people and organizations in broader regional partnerships. The
investment provided support and encouragement to what one interviewee
described as a “fragile community” of practitioners engaged in
entrepreneurship development. Practitioners realized that they had assets,
skills and opportunities to combine that would help them achieve self-
sufficiency. As one interviewee noted, “It’s amazing what we can get done
here.”
ARC investments helped to change people’s attitudes, particularly
among youth and their teachers. Project leaders noted increased
enthusiasm and a change in attitude among young people in particular.
These changes were described in a number of ways including increased
self esteem, improved performance in school, and a new “entrepreneurial
mindset” for students; increased enthusiasm and interest in pursuing their
own entrepreneurial aspirations among teachers.
These qualitative insights are drawn from interviews with key stakeholders in the
region. It was beyond the scope of this evaluation project to conduct the in-depth
field work required to accurately verify these outcomes. However, these recurring
themes were heard across our interviews, from people representing different
states and organizations – non-profit and governmental – and engaged in
different aspects of entrepreneurship development – from providing capital to
educating youth. A more rigorous assessment of these qualitative impacts would
be possible by designing a participatory evaluation approach as part of project
design. The challenges and opportunities associated with this approach are
discussed in Chapter 9 as part of the “best in class” metric system.
Conclusions Regarding Qualitative Impacts
By identifying the qualitative outcomes associated with ARC’s EI investments, a
more nuanced, in-depth picture of the impact on the region is obtained. The EI
has served to “change the conversation” – to elevate entrepreneurship as a key
component of economic development in the region. The beginning of a culture
shift is evident at the community level, where EI projects have been having
demonstrable quantitative and qualitative impacts. There is less evidence that
these impacts are translating into policy change at the state level, suggesting the
need to explore the lessons learned from the EI experience for both
entrepreneurship development, generally, and the design and management of
future ARC entrepreneurship investment activities.
What do these qualitative impacts suggest about the metrics needed to measure
the broad set of outcomes entrepreneurship investments may have? The
observation of these impacts supports the need for using a broad set of metrics
to accurately depict project outcomes. These qualitative outcome measures
could include:
87
Public investment ($) in entrepreneurship development activities (pre- vs.
post project investment)
Private investment leveraged ($) as a result of project investment in
entrepreneurship development
Perceived change in community/regional support for entrepreneurship
development (as measured through pre- and post-investment community
surveys)
Increased collaboration among support providers (as measured by the
number of partners contributing resources to entrepreneurship
development).
88
CHAPTER 8
LESSONS LEARNED FROM THE ARC ENTREPRENEURSHIP INITIATIVE
EXPERIENCE
When ARC first announced the Entrepreneurship Initiative in 1997, ARC staff
and its state and local partners had a very limited base of experience and
effective practices upon which they could build. Yet the EI investments were not
a completely new thing. In fact, many EI projects had similarities with earlier
ARC investments in capacity building or sector development strategies. In
addition, other small business development programs (such as the national Small
Business Development Center initiative) or state business support programs
(such as Pennsylvania’s Ben Franklin Partners effort) offered useful ideas and
lessons learned.
While these earlier programs provided some useful lessons, none of these
predecessors sought to combine the objectives of supporting traditional
economic development goals, such as new business starts, with a wider mission
of knitting entrepreneurial development into the mainstream of economic
development thinking and practice in Appalachia. As a result, the ARC team
was, in some sense, building the road as they traveled it.
Through the course of the EI, ARC and its program partners learned a great deal
about how to do entrepreneurial development right. These lessons were
sometimes learned through the school of hard knocks, as once promising
initiatives failed to pay the expected dividends. In other cases, successful pilot
projects were replicated throughout Appalachia and throughout the US. For
example, ARC’s early investments in developing alternative equity capital
sources were one stimulus for creation of the New Markets Venture Capital
initiative and New Markets Tax Credit program which now supports more than
$19 billion in investments in low-income communities.
Throughout the evaluation process, the research team has focused on gathering
these “lessons learned” as means to capture established best practices,
exemplary program models, as well as informal and tacit learning that has
occurred through the life of the EI program. These lessons learned should inform
future ARC investments (in entrepreneurial development and elsewhere) as well
as other federal, state, and local efforts to promote entrepreneurship.
The compiled “lessons learned” generally fall into two broad categories. One set
of lessons applies to those actively engaged in the practice of entrepreneurship
development – people who are implementing entrepreneurship education,
training and technical assistance, capital, incubator, and networking or sector
specific initiatives in their communities, regions or states. The other set of
lessons applies more directly to the design and implementation of ARC’s
Entrepreneurship Initiative and would be most useful to those seeking to create
similar or additional region-wide initiatives.
89
LESSONS FOR PRACTITIONERS – WHAT WORKS IN ENTREPRENEURIAL
DEVELOPMENT
Nearly all of the interviewees offered thoughtful lessons learned based on their
participation in EI. Because EI was something of a “new thing” for many state
and local partners, it forced many project leaders to think differently and move
out of their traditional comfort zones. As the projects progressed, the project
teams evolved in their thinking and became more sophisticated in understanding
the key ingredients for a successful regional entrepreneurship strategy.
Project leaders and other regional stakeholders emphasized several key lessons
learned. If one were concocting a recipe for successful regional
entrepreneurship projects, the following ingredients would be required.
Organizations with these attributes tended to be more successful than their
counterparts who lacked some or all of the necessary key ingredients. These
exemplary practices related to program and community leadership, a program’s
management, goals and objectives, and a program’s outreach efforts (Table 8.1).
Additional information on each of these factors is provided below.
Table 8.1. Lessons Learned for Practitioners from Evaluation of ARC’s
Entrepreneurship Initiative
LESSONS FOR PROGRAM
LEADERSHIP
LESSONS FOR PROGRAM
MANAGEMENT
LESSONS FOR PROGRAM
OUTREACH
Successful entrepreneurship
initiatives had sparkplugs or
local champions that provided
leadership for these efforts.
Local capacity was a key to
success.
Program self-sufficiency
(sustainability) and success
went hand in hand.
Entrepreneurship
development was recognized
to be a long-term process.
Successful projects altered
their goals and approaches as
conditions warranted.
Partnerships and
collaborations were important
to success.
Successful projects
celebrated and shared the
story of their success.
Lessons for Program Leadership
Successful entrepreneurship initiatives had sparkplugs or local champions
that provided leadership for these efforts.
The need for committed local leadership is a critical requirement for success with
regional entrepreneurship efforts – just as it is with other forms of economic
development. The role of this local “sparkplug” became especially important in
the EI since it was, in part, designed as a response to the decreasing dividends
generated by traditional economic development strategies based on business
recruitment and attraction. Changing thirty years of economic development
90
practice would not occur overnight. In many communities, the concepts of
entrepreneurial development were not well understood or were resisted by those
who were comfortable with the status quo as the way things had always been
done.
In the midst of this environment where the new concepts were poorly understood
or discounted, strong leadership was required. Entrepreneurial development
needed a strong “brand” and a compelling vision that would capture the
imagination of local leaders and residents. In most cases, this vision was
generated by a local champion who came to embody the new approaches and
the new vision for the region.
These leaders came from non-profit organizations, community colleges, schools,
economic development organizations, and other institutions. They included
people who were visionaries and saw the potential for entrepreneurship to be a
force for change in their communities. They embodied the characteristics of
entrepreneurs themselves – they saw opportunities, marshaled resources, were
flexible, and determined and committed to creating a new economic development
path in their part of Appalachia.
Abingdon, Virginia’s Appalachian Sustainable Development (ASD) and its leader,
Anthony Flaccavento, exemplify this pattern.
89
Founded in 1995, ASD seeks to
promote sustainable farming and forestry in Appalachian regions of Virginia and
Tennessee. Southwest Virginia had always been a center of agriculture and
forestry, but these industries, especially forestry, had never previously focused
on sustainable practices. And, for many local leaders, environmentalism and
economic development were opposing forces. In their view, environmentalists
were opposed to business and insufficiently concerned about strengthening the
local economy.
Flaccavento and ASD sought to redefine the debate by highlighting the
tremendous entrepreneurial opportunities presented by value-added agriculture
and forestry. They educated farmers about sustainable forestry practices and
enlisted their support, so that instead of simply harvesting logs, local
entrepreneurs could sustainably gather the wood, process the lumber close to
home and use it in value-added products like flooring and wood trim. The
Appalachian Sustainable Woods Processing Center, created with EI funds,
became a local symbol for home-grown initiatives. It combined exciting
entrepreneurial opportunities with a respect for home-grown traditions and
industries, and fostered a growing belief throughout the region that sustainable
business practices can and do work. In 2001, ASD created and is actively
building its own Sustainable Woods product line
90
to sell wood products that are
environmentally friendly.
89
An interview with Anthony Flaccavento can be found at PBS NOW Enterprising Ideas website,
.
90
To learn more, visit .
91
Local capacity was a key to success.
ARC has a long history of investing in local community capacity building,
91
and a
number of the EI investments sought to develop local capacity for supporting
entrepreneurs. For example, TEAM Pennsylvania sought to seed
entrepreneurial assistance networks across the state. These networks would be
affiliated with the seven economic development districts located within ARC’s
jurisdiction. This effort ultimately generated mixed results. Several of the
networks jelled, and one effort (the Northeast Pennsylvania Entrepreneurial
Alliance)
92
received a national award. Yet, few of these networks are now in
operation and their role in supporting local entrepreneurs was limited.
The TEAM Pennsylvania experience was fairly typical for EI. Because of limited
resources, EI faced significant challenges in creating new community capacity
where none had previously existed. However, one case where EI contributed to
building capacity is Tech 2020 in Tennessee. Tech 2020 was initiated in 1993 to
build on the unique regional assets in eastern Tennessee and create a high tech
industry. ARC made a series of investments totaling $1.2 million over five years
to build the capacity of Tech 2020, primarily in the area of venture investing.
Tech 2020 has grown over time into a significant economic development
organization in the state and the region, playing a major role in establishing the
Southern Appalachian Fund, one of the first New Markets Venture Capital
Companies making investments in Tennessee, Kentucky and Appalachian
Georgia and Mississippi.
When strong organizations with existing capacity were already in place, EI
investments had a catalytic effect. In most of the successful cases, a community
was home to an organization with a strong track record in other related fields of
activity and this capacity was leveraged in support of entrepreneurship
development. For example, the Kentucky Highlands Investment Corporation
(KHIC) had been supporting community development efforts in Eastern Kentucky
since 1968.
93
KHIC began as a traditional community development organization
but its mission has evolved over time. Over the years, it has become an
intermediary for many Federal lending programs, such as SBA and USDA, and
had thus developed extensive in-house financial expertise and capacity. At the
time of the EI’s introduction, KHIC was seeking to increase the supply of seed-
91
For an evaluation of these projects, see Brian Kleiner, et al., Evaluation of The Appalachian
Regional Commission’s Community Capacity-Building Projects, Final Report to the Appalachian
Regional Commission, July 2004, xi.
92
National Association of Development Organizations (NADO) Research Foundation, Business
Not as Usual: Regional Development Organizations Promote Rural Entrepreneurship
(Washington, DC: NADO, 2002), .
93
To learn more, see Deborah Markley and David Barkley, Development of an Entrepreneur
Support Organization: The Case of Kentucky Highlands Investment Corporation, RUPRI Center
for Rural Entrepreneurship, Research Case Studies Series No. 1, March 2003,
.
92
stage equity in the region. Thanks to EI investments, KHIC was able to branch
out into these new related markets. Today, the Southern Appalachian Fund,
backed in part with EI dollars, manages $12.5 million that can be invested in local
firms seeking early stage equity capital. The success with which the Southern
Appalachian Fund emerged as a developmental venture capital force in the
region was due, in part, to the strong KHIC base upon which it was built.
Lessons for Program Management
Program self-sufficiency (sustainability) and success went hand in hand.
While all ARC grantees seek to be self-sustaining, effective EI projects viewed
ARC funds as start-up investments that were not an end in themselves. Instead,
ARC dollars were used to jump-start programs that would rise or fall based on
how they performed in the marketplace – just like any other entrepreneurial
venture. Projects that had a goal of becoming self-sufficient appeared to create
better outcomes and stronger sustained efforts. Leaders of these projects
pursued sustainability by creating partnerships and finding resources to continue
to build the program beyond the ARC grant. Creating sustainable economic
development programs takes time; this lesson suggests that making self-
sufficiency an explicit project goal may result in greater priority being placed on
its achievement.
As a group, the EI’s Capital Access projects performed best in terms of
leveraging outside investments to have a sustainable impact on the region. As
described above, a developmental venture capital industry was spawned through
the efforts of organizations funded, in part, by ARC. KHIC and Tech 2020 were
able to launch the Southern Appalachian Fund (SAF) as one of six New Markets
Venture Capital companies in the country through initial support provided by
ARC. SAF received $2 million in New Market Tax Credits in 2002, and was able
to raise a total of $12.5 million from investors, including the Tennessee Valley
Authority, foundations and a number of banking institutions. The principals
behind SAF credit ARC with its creation. One noted that if EI’s director “had not
been creative, Southern Appalachian Fund would not have been created.” ARC
was also an early investor in Meritus Ventures, a $36 million Rural Business
Investment Company established in 2002. As with SAF, Meritus brings a much
needed source of equity capital to support expanding companies in the region.
Among Meritus’ investors are regional entities, such as TVA and the University of
Kentucky, private financial institutions, and some high net worth individuals. Both
SAF and Meritus are combining access to capital with business support services,
funded in part by the U.S. Small Business Administration and U.S. Department of
Agriculture.
Yet attracting new money is not the only measure of an effective and sustainable
program. Sustainability can also be generated when programs succeed in
attracting new partners, building local collaboration, and generating energy and
93
buzz about local community-building efforts. The development of Athens, Ohio’s
Dairy Barn Arts Center reflects this pattern. Thanks to a $50,000 EI grant, the
Dairy Barn Southeast Ohio Arts Center was able to build a shop to sell artwork
and crafts produced by local craftspeople. The shop soon turned a profit,
becoming self-sustaining and providing valuable income to approximately 100
local artisans. The shop has also stimulated new partnerships with local schools,
and has generated buzz about the Arts Center’s other projects. From an
entrepreneurship standpoint, the project has also helped educate local people
about the potential for “self-sufficiency through art.”
The Clinch-Powell Community Kitchens (in Treadway, TN) pursued a similar path
to sustainability. ARC’s investments aided the kitchen in purchasing some new
equipment, but more importantly, program managers invested these dollars to
improve marketing capabilities for incubator clients. As the Community Kitchen
has grown, it has developed close partnerships with the Clinch Appalachian
Artists Cooperative. As artists and food producers have built partnerships, they
have entered new markets with new combined products such as gift baskets.
Many of these products are now sold via the Appalachian Spring Cooperative, a
joint marketing effort designed to promote family farms and local artisans.
Entrepreneurship development was recognized to be a long-term process.
It takes years to produce the culture change that is a desired outcome of many
entrepreneurship development efforts. Successful project leaders recognized the
long-term nature of their endeavors and concentrated on developing the staying
power – resources, leadership, organizational capacity, community support –
needed. As one interviewee noted, “It usually takes longer to reach the critical
mass (and resultant job and business creation) than you would expect.” And,
another noted the need to “be patient, education projects don’t have immediate
results.”
By definition, the EI’s entire portfolio of youth entrepreneurship education
projects reflects this perspective. These projects rarely assessed their
performance based on traditional economic development measures of job
creation or leveraging of new investments. Appropriately, they measured
progress according to unique measures, such as the number of schools offering
entrepreneurship training or students’ increased awareness of business
concepts.
ARC sought to publicize the best regional youth entrepreneurship initiatives
through its sponsorship of the 2002 and 2003 Appalachian Youth
Entrepreneurship Springboard Awards (Table 8.2).
94
Twelve different programs
were honored for their ability to teach youth about the key components of
entrepreneurship, to develop clear and measurable outcomes that provided value
94
See Appalachian Regional Commission, Appalachian Youth Entrepreneurship Springboard
Award: 2002 and 2003 Winners, 2004, .
94
to the local community, and to create models that could be sustained at home
and replicated elsewhere.
The Springboard Award winners were located across the region, and were based
at a variety of institutions including public schools, non-profits, and vocational
training centers. Walhalla High School (in Walhalla, SC) was one of the first
Springboard awardees. The Walhalla School District requires all ninth graders to
be exposed to entrepreneurship training in their social studies class. When these
students enter high school, they can take two separate entrepreneurship-related
courses where they are introduced to basic business concepts and move on to
start their own businesses. Each year, Walhalla High School students start
dozens of new school-based companies. In 2007, a local student won the
prestigious International Young Entrepreneur of the Year Award from the
National Foundation for Teaching Entrepreneurship.
Other Springboard winners sought to use entrepreneurship education as an effort
to slow the out-migration of area youth. Virginia’s Lonesome Pine Office of Youth
used ARC funds to support the Stay for Life Project at Bush Mill. This effort
linked local youth with area volunteers who rebuilt and refurbished a local stone
mill that is now used to produce flour and corn meal. Area youth continue to
operate the mill and sell its products, and point to the effort as a great means to
learn about entrepreneurship and about the history of their community. More
importantly, they were exposed to the idea that they can enjoy a successful
economic future without having to leave their hometowns.
Table 8.2. Springboard Award Winners – 2002 and 2003
2002 WINNERS 2003 WINNERS
Estill County High School,
Irvine, Kentucky
Tupelo Middle School, Tupelo, Mississippi
ACEnet, Athens, Ohio
Walhalla High School, Walhalla, South
Carolina
Lonesome Pine Office on Youth, Big Stone
Gap, Virginia
Randolph County Vocational Technical
Center, Elkins, West Virginia
Hale County Technology Center,
Greensboro, Alabama
Monroe County High School,
Tompkinsville, Kentucky
Ripley Union Lewis Huntington High
School, Ripley, Ohio
East Stroudsburg High School-North,
Dingmans Ferry, Pennsylvania
Carroll County Public Schools, Hillsville,
Virginia
United Technical Center, Clarksburg, West
Virginia
Additional information about the Springboard Awards can be found on the ARC website athttp://www.arc.gov/index.do?nodeId=1994.
95
In addition to the Springboard Awards, ARC made investments in other youth
education programs that have proven to be sustainable. ARC invested in a
program run by the University of Alabama to bring entrepreneurship education to
a 10-county region in the state. The program was successful in using the REAL
curriculum to train students and to build school-based enterprises. The program
continued beyond ARC funding, expanding to 14 counties. Many of the students
were at risk and according to the project leader some businesses were started as
a result of the training. In another case, ARC funded the Ohio-West Virginia
YMCA to use REAL training in a summer camp for youth. Two of the important
outcomes identified were the engagement of schools, communities and an
economic development district in the project and the change in attitudes seen in
the young people for participated in the camps.
While ARC’s investment in youth entrepreneurship programs and approaches
reflects a concern for changing the culture of the region, it was less apparent that
portfolio investments were made with explicit attention to the development of
entrepreneurs over time. Many of the incubation and TA projects measured
outputs such as businesses served rather than measuring the transformation of
entrepreneurial skills or the outcomes associated with the projects, such as
improved business performance. Insights gained from the implementation of
Lyons and Lichtenstein’s Entrepreneurial League System
®
approach suggest that
the long-term transformative impact of entrepreneurship investments depends, in
part, on the development of entrepreneurs over time.
95
Successful projects altered their goals and approaches as conditions
warranted.
The EI was the first regional effort to encourage entrepreneurship development
and investments were made when the field was relatively new. Project leaders
were plowing new economic development territory and many projects were
considered demonstrations. Effective project leaders adjusted their approaches
to reflect changing demand and to overcome unexpected obstacles. As a result,
these projects achieved outcomes that were positive but, in many cases, different
from what was originally intended.
Georgia’s Appalachian Community Enterprises’ (ACE) experience during the EI
may be instructive. ACE was established to help address a pressing capital gap
facing many small businesses in North Georgia. As ACE developed its microloan
program, it soon discovered that simply making new loan funds available was an
insufficient response to the challenges facing the region’s small firms and
aspiring entrepreneurs. Many local residents lacked money skills or suffered from
poor credit ratings that made it nearly impossible for them to do business with
local banks. As a result, ACE quickly geared up its financial literacy education
efforts. Programs such as ACE’s Money Camp for Grown-Ups are now used to
help residents learn personal money management skills.
95
Gregg A. Lichtenstein and Thomas S. Lyons (2001).
96
As ACE has introduced new programs, it has also begun to enter new markets.
North Georgia is in the midst of a massive influx of Latino immigrants who are
changing the face of many of the area’s small towns. These new residents are
also very interested in starting new businesses, and ACE has now begun an
aggressive effort to provide services to the region’s Hispanic entrepreneurs.
In Coeburn, Virginia, the initial intent behind an EI grant was to provide training to
low-income people to become crafters. However, according to the project
designer, “We learned that you can’t teach people to be artists and crafters
unless they want to be.” But through the exploration associated with the ARC
grant, they learned that a significant number of musicians were seeking ways to
expand their music businesses. The project leader credits the EI with helping the
local area spawn a popular spot for bluegrass and country music. Now people of
all ages are participating in the Friday night concerts and Thursday evening jam
sessions. Crowds are coming from miles around to dance, eat, and enjoy
affordable, excellent entertainment. The community center originally started
through the ARC grant is a vintage downtown building that formerly housed Lay’s
Hardware. It is being conserved through income generated and utilized as a
mecca for music, the arts, and various art-related training sessions. They are a
site for, and one of the progenitors of, The Crooked Road, a 13-county effort to
highlight and market country music venues throughout Southwestern Virginia.
Lessons for Program Outreach
Partnerships and collaborations were important to success.
Successful projects marshaled resources by forming partnerships and
collaborating with other organizations to share resources and build capacity.
They leveraged assets and avoided duplication of efforts. These partnerships
also facilitated networking among service providers, creating a better
environment for entrepreneurs.
Nearly all of the EI projects were built on partnerships of some sort. In fact,
partnerships – at least in the form of matching dollars from state or local sources
– were a requirement of all EI grants. But, several EI projects developed
exemplary collaborations that still exist today. For example, ARC invested
$100,000 to support a regional biotechnology initiative in western North Carolina.
This project linked Advantage West (the regional development agency), NC Bio,
Buncombe County, Western Carolina University, and several other key players.
These funds allowed Advantage West to hire a full-time regional biotech
coordinator, and to also develop biotech incubator space at Ashe-Buncombe
Technical Community College. They also developed a region-wide biotech
steering committee that still operates today. This effort is now an integral part of
a statewide North BioNetwork, with Ashe-Buncombe College serving as
headquarters for the state’s BioNetwork BioBusiness Center.
97
Another example of regional collaboration can be found in the Start Smart
initiative funded by the EI in a nine-county region of Appalachian Tennessee.
Funding was provided to the Southeast Local Development Corporation to
provide technical assistance and training to entrepreneurs in these rural counties
– a region of the state that did not have a critical mass of business resources.
This initiative built on prior successful work by the Southeast Women’s Business
Center and project leaders were able to leverage this reputation to get buy-in for
the ARC-funded project. However, collaboration was not achieved through
reputation alone. The Women’s Center program director identified all the
business resource providers who served the region and personally met with each
of them. By understanding who the potential partners were – their strengths and
potential weaknesses – the director leveraged significant resources in support of
entrepreneurs throughout the region and designed a value-added program.
Successful projects celebrated and shared the story of their success.
Many projects engaged the media to help build community support as well as to
publicize their activities as part of a broader marketing campaign. Some
communities held up their successful entrepreneurs as role models. ARC was
also instrumental in sharing success through their region-wide education efforts,
e.g., conferences, and programs like the Appalachian Youth Entrepreneurship
Education Springboard award.
Kentucky’s Artisan Heritage Trails Program, headquartered in Richmond,
Kentucky, was especially savvy in its media outreach strategies. Program
leaders faced two challenges in terms of media outreach. First, they needed to
engage local artisans to participate in the program. Second, they needed to
market outside of the region to attract tourists to use the trails and visit local
crafts people. In terms of engaging local artisans, project leaders focused first on
using the Internet as a marketing tool. As local crafts people began profiting from
online sales, word of mouth did the rest of the work. More artisans signed up and
the project was off and running. The Trails program began with a small base of
82 artisans. Today, more than 600 artisans sell their wares through the program.
The Trails Programs has also been very savvy and fortunate in its efforts to
attract tourists to the region. The most recent innovation is an online trail system
that allows visitors to map out their travel plans (along 17 different trails) and also
learn more about sites and shops long the trail. But, a more important partnership
has been developed with National Geographic. The site was featured in National
Geographic Traveler magazine and its online maps can be accessed at the
National Geographic Traveler website.
98
LESSONS RELATED TO EI PROGRAM – WHAT WORKS IN PROGRAM
DESIGN AND IMPLEMENTATION
While this evaluation uncovered significant praise for ARC’s leadership in
entrepreneurship development and the value of the investments made from
1997-2005, challenges experienced with the program provide some lessons that
may contribute to the continuous improvement of this or other initiatives in the
future. Interviewees cited several key challenges that relate to the program’s
initial design, its structure for implementation, and efforts to track and
communicate its impacts (Table 8.3).
Table 8.3. Lessons Learned for Program Design and Implementation from
Evaluation of ARC’s Entrepreneurship Initiative
LESSONS FOR PROGRAM
DESIGN
LESSONS FOR PROGRAM
IMPLEMENTATION
LESSONS FOR PROGRAM
IMPACTS
Practitioners and
entrepreneurs have unique
local knowledge that can be
applied to program design and
subsequent program
refinements.
Successful initiatives brought
together related investments,
in this case, other regional
economic development or
entrepreneurship-related
investments.
Getting EI funds to local
partners was dependent upon
state program managers and
varied based on the
importance assigned to the
initiative.
The size of the ARC EI grants
placed limits on regional
impacts.
Building a broader base of
support for entrepreneurship
investments requires
continued efforts to “make the
case” to local leaders.
Programs can be improved by
embracing long-term and
locally-driven evaluation of
program outcomes and
impacts.
Lessons for Program Design
Practitioners and entrepreneurs have unique local knowledge that can be
applied to program design and subsequent program refinements.
Nearly every interviewee felt that the EI was an idea whose time had come. The
initiative was appropriately tailored for local needs, and helped introduce new
economic development concepts into the region. Nonetheless, two primary
issues emerged when program managers and other stakeholders commented on
the initial design of the EI. First, some practitioners felt they could have been
more engaged in the design of the Entrepreneurship Initiative. People in the field
felt they had limited input on how the EI was designed or on helping ARC
understand what was needed in the region. And, they felt there was no explicit
approach to engaging entrepreneurs – the customers – in program design.
Second, they also suggested that ARC encourage a more holistic approach to
entrepreneurship development. Successful programs weaved together the
various ARC initiatives into a more holistic, systems approach to development,
combining EI investment with broadband, etc. This lesson could guide ARC in
designing future programs.
99
Critics recognized that the EI, and other ARC special initiatives, do not simply
emerge out of thin air. ARC staff and state program managers are responding to
trends in the field and to their own analyses of local economic development
needs. Indeed, ARC engaged advisory groups with good representation from the
private and non-profit sectors to inform the design and implementation of the EI.
However, as ARC continues to develop future special initiatives or other
entrepreneurship-related investments, it might consider supplementing these
approaches with other methods that capture local input in a broader and more
systematic manner. Multiple methods could be employed. These could include
online surveys, public forums (e.g., listening tours or town hall meetings) that
discuss potential new program ideas, or even a process of formal petitions or
suggestions provided by state or local program partners. Of paramount
importance in any approach taken by ARC, however, is to engage the
entrepreneurs in decision making about entrepreneurship development program
design.
These techniques are all designed to create a more transparent process of
communication between local partner organizations and ARC staff. This more
open process will not only help to improve local satisfaction with ARC programs,
it will also improve their effectiveness as they become more responsive to local
markets and more cognizant of local economic development capacities.
Successful initiatives brought together related investments, in this case,
other regional economic development or entrepreneurship-related
investments.
A second challenge concerns the relationship between EI and other regional
economic development investments from ARC and other local, state or federal
partners. For some communities, EI investments served as a one-time
investment to support a youth entrepreneurship summer camp or to fund the
start-up costs of an incubator. When these projects lacked a sustainability plan
or were poorly coordinated with other regional development efforts, they tended
to fizzle out once ARC funding ceased.
Virginia’s use of EI funds presents a compelling contrast, offering a model of how
to link ARC funding into a broader regional regeneration strategy. Southwest
Virginia’s economy had long depended on mining and manufacturing as regional
drivers. At the time of EI’s inception, state and local leaders were focused on
developing new strategies for regional reconstruction. They viewed EI funds as
seed investments to support strategies around heritage tourism, sustainable
agriculture and forestry, and Main Street development. As such, they sought to
tie EI projects to the Virginia Main Street program, and also used EI dollars to
start-up marquee projects like The Crooked Road heritage music trail and the
Round the Mountain artisans’ network. In fact, six of the eight venues on The
Crooked Road tour have received ARC funds. Virginia has invested dollars from
subsequent ARC special initiatives (in broadband and asset-based development)
100
to further support this larger region-wide strategy. As one statewide observer put
it, “We see ARC as part of a mosaic, not as projects. . . . ARC programs are
tools, not ends in themselves.”
A similar process occurred in Western Maryland’s Garrett County where local
leaders linked the EI to an ongoing effort to revitalize the county’s downtown
areas. EI investments were coordinated with Main Street programs. Main Street
programs focused on issues of streetscapes, beautification, and supporting local
retail; EI dollars supported expanded business technical assistance, a new
microloan program, and creation of a business incubator. Together, these efforts
have helped revitalize downtown Oakland and the county’s other town centers.
Lessons for Program Implementation
Getting EI funds to local partners was dependent upon state leadership and
varied based on the importance assigned to the initiative.
State governments are the ARC’s key regional partners, so it is not surprising
that several dimensions of state government helped shape the EI process. The
differences between state approaches to the EI are quite striking. As we saw in
the case of Virginia, several states used EI funds to support components of a
wider regional development strategy. Other states, such as Pennsylvania,
steered investments toward certain types of programs or strategies (e.g.,
entrepreneurial assistance through networking). Finally, other states supported
grass-roots innovation and funded a host of local pilot projects.
In addition to pursuing different strategies, states also differed in terms of their
commitment to promoting entrepreneurship as a regional development strategy.
Some governors and state program managers were cool to the new approach,
while others viewed it as a critical component for community transformation. Not
surprisingly, the level of “buy-in” by state leaders had a large impact on how the
deployment, outcomes and effectiveness of the program varied by state. If the EI
was not a priority for state leaders, it was not likely to be promoted within their
state. Statewide performance of EI projects also seemed to improve when state
program managers had close ties with local community practitioners.
When it comes to state program management, there is no one best approach.
Each strategy has its pros and cons, and each strategy may also exclude some
programs from potential consideration for ARC funding. These programs may be
located in regions or may be pursuing approaches that fall outside of the state’s
primary strategic focus. State leaders will retain a central role in future ARC
project decisions, but, in the case of special initiatives, ARC might consider
setting aside some portion of funds that are open to direct application from
localities or non-profits. Further details of this proposed Innovation Fund are
presented in Chapter 9.
101
The size of ARC grants placed limits on regional impacts.
Practitioners, and state program managers, were challenged by the relatively
small pool of money allocated to the EI. State program managers noted that it
was difficult to figure out what to do with small amounts of money and it was hard
to get recognition for the EI as a result. Most of those interviewed recognized the
political necessity for ARC to spread resources throughout the region but noted
that this often resulted in too little money to achieve significant impacts. As one
stakeholder noted, the initiative sowed many seeds but the ground was not fertile
enough to grow sustainability for most of these efforts.
Table 8.4 provides a summary of state-level EI investments on an annual basis.
They indicate that ARC faced serious obstacles when seeking to transform
regions or even a region’s thinking about entrepreneurial development. ARC’s
investments were generally dwarfed by other economic development spending in
the region. For example, Kentucky spent $808 million on economic development
in 2004,
96
dwarfing ARC’s 2004 Kentucky EI investments of nearly $384,000.
Because of these disparities, ARC investments must be tightly focused on
programs that can build scale, be sustainable, and have major impact.
One interviewee referred to the “sprinkling effect” of EI investments. ARC spent
enough dollars to meet one of the EI’s key objectives – to educate state and local
leaders about the potential of entrepreneurial development as a regional
development strategy. Unfortunately, the investments were not sufficiently large
to meet ARC’s other objectives of fostering “systemic change in the economic
development landscape of the region.”
97
With average per project investments of
$126,387, the EI served mainly as seed stage funding for regional
transformation. There are some noteworthy exceptions that prove the value of
more sustained investments in program development. ARC made a series of
grants to a number of programs throughout the region that have demonstrated
sustainability after the EI investments were completed. The Shoals
Entrepreneurial Center incubator in Alabama and the Natural Capital Investment
Fund (NCIF) in West Virginia are two examples of multi-year grant making that
helped build institutional capacity that ultimately led to sustainable organizations
that continue to have an impact on their states and, in the case of NCIF, the
region.
What this lesson learned suggests for the structure of any future
entrepreneurship investments is that “next level” investments for local initiatives
that the EI has spawned are needed. Interviewed project managers remained
strongly committed to building on the momentum generated by EI. They noted
96
Mountain Association for Community Economic Development, Accounting for Impact:
Economic Development Spending in Kentucky (Berea, KY: Mountain Association for Community
Economic Development, 2005).
97
Appalachian Regional Commission, Entrepreneurship Initiative Program Summary, September
30, 2003,http://www.arc.gov/index.do?nodeId=1970.
102
that the EI had “kick-started the conversation” and had “got people thinking”
about new ways to build community prosperity. The cultural shift sought by EI’s
designers is starting to take hold. One community leader put it well, “I don’t hear
the phrase, ‘this isn’t gonna work’ anymore.” The region’s residents – and its
economic development organizations – are increasingly aware of local economic
assets and their ability to support home-grown economic development
approaches.
ARC has sought to build on this momentum through its regular programs and
ongoing Chairman’s initiatives in broadband, energy, and asset-based
development. But, it should not abandon its previous focus on local
entrepreneurial development. Indeed, as demonstrated in Chapter 5,
entrepreneurship is a key asset in the region and can be a driving force within
ARC’s asset-based development initiative. ARC was instrumental in seeding
entrepreneurship activity in the region, and it should consider some way to
continue these investments. Emerging successful programs would benefit from
“next level” funding that would allow project leaders to move these initiatives to a
level that has the potential for transformative impact.
Lessons for Program Impacts
Building a broader base of support for entrepreneurship investments
requires continued efforts to “make the case” to local leaders.
While the EI has been successful in beginning to change attitudes toward
entrepreneurship, this cultural shift is by no means universal throughout the
region. Particularly among local elected officials, capacity building and efforts to
make the case are needed.
Interviewees were quite positive about ARC’s ongoing education efforts. Under
EI, the agency sponsored several well-attended conferences on entrepreneurial
development in general, and on specific topics such as business incubation and
support for creative industries. All of these events were well attended and
received high marks for attendees. Although ARC issued specific invitations to
local elected officials in an attempt to get them to these events, for the most part,
attendees were already engaged with ARC as grantees and partners in the EI
program. ARC should consider additional outreach efforts, spread across the
region to encourage participation, as well as investments to provide workshops
or training to the region’s elected officials or those who work in government
agencies or cooperatives focused on other rural development issues (e.g., rural
telecommunications cooperatives, USDA Cooperative Extension officials,
Conservation Districts, etc.) These programs need not be funded solely by ARC
but could be developed in cooperation with relevant trade associations. Through
these partnerships, ARC could not only spread the message about the
importance of entrepreneurial development. It would also reinforce an important
message that effective entrepreneurial development requires a holistic and
103
collaborative approach to economic development. Entrepreneurial development
is not the sole province of economic developers. It requires partnerships with
elected officials, educators, local business owners, and other key stakeholders in
the region.
Programs can be improved by embracing long-term and locally-driven
evaluation of program outcomes and impacts.
Practitioners would have benefited from ongoing assessment of project
outcomes and follow up from ARC to share lessons learned and support mid-
course changes as needed. It was clear that a better metrics system, both in
terms of defining relevant outcomes and collecting and reporting data, was
needed.
This finding echoes the results of Westat’s 2004 evaluation of ARC’s community
capacity-building projects which found that community program managers were
ill-equipped to track, measure, and publicize the outcomes of their programs and
ARC investments.
98
In EI’s case, few projects tracked metrics beyond those
required as part of the ARC grant. This measurement challenge was identified as
part of the initial evaluation of ARC’s EI investments, completed early in the life of
the initiative.
99
This early study observed that the internal evaluation and
monitoring systems used in the sample projects lacked “specific outcome
measures.” While most projects they studied had created an evaluation system,
“43 percent of all projects cited ‘monitoring outcomes’ as a problem.” The
persistence of this measurement challenge suggests the need for corrective
action for future entrepreneurship investments.
As noted in Chapters 3 and 4, these shortcomings are not unique to ARC
projects or to the general field of entrepreneurial development. In fact, because
of their short life spans, few entrepreneurial development programs have been
subjected to rigorous performance measurement and assessment.
100
Moreover,
most ARC grantees had limited budgets and limited staff who were primarily
focused on delivering programs and serving customers. As a result, few
programs could devote sufficient attention to performance measurement.
Given the resource and time pressures facing program managers, ARC cannot
expect them to embrace rigorous performance measurement and should instead
seek to create incentives for more effective program assessment. ARC could set
aside some portion of all grant funds (1-2%) to support program measurement.
ARC could also develop a program measurement “toolkit” that helps walk all
98
Brian Kleiner, et al., Evaluation of The Appalachian Regional Commission’s Community
Capacity-Building Projects, Report to the Appalachian Regional Commission, July 2004.
99
Regional Technology Strategies, Inc., Evaluation of the Early Stages of the Appalachian
Regional Commission’s Entrepreneurship Initiative, Report to the Appalachian Regional
Commission, December 2001.
100
Organization for Economic Cooperation and Development, OECD Framework for the
Evaluation of SME and Entrepreneurship Policies and Programmes (Paris: OECD, forthcoming).
104
grantees through the evaluation process. Finally, ARC could designate resource
experts among its own staff (or partner organizations) who can provide technical
assistance (when needed) on pressing performance measurement issues. These
recommendations are further discussed in the next chapter.
1
0
5
T
a
b
l
e
8
.
4
.
A
n
n
u
a
l
a
n
d
T
o
t
a
l
A
R
C
E
n
t
r
e
p
r
e
n
e
u
r
s
h
i
p
I
n
v
e
s
t
m
e
n
t
s
,
1
9
9
7
-
2
0
0
5
,
b
y
S
t
a
t
e
(
$
)
S
T
A
T
E
1
9
9
7
1
9
9
8
1
9
9
9
2
0
0
0
2
0
0
1
2
0
0
2
2
0
0
3
2
0
0
4
2
0
0
5
T
O
T
A
L
A
l
a
b
a
m
a
0
1
5
9
,
0
0
0
3
8
1
,
6
6
3
5
3
7
,
1
8
7
1
9
8
,
8
5
0
4
0
5
,
0
0
0
9
2
,
0
0
0
5
1
8
,
8
9
2
1
1
2
,
6
7
5
2
,
4
0
5
,
2
6
7
G
e
o
r
g
i
a
1
1
0
,
0
0
0
0
4
4
1
,
8
2
6
0
5
8
1
,
5
2
9
3
4
9
,
4
4
2
8
3
,
0
0
0
7
2
3
,
7
4
7
3
3
4
,
1
1
8
2
,
6
2
3
,
6
6
2
K
e
n
t
u
c
k
y
9
2
,
7
6
9
2
2
6
,
0
0
0
7
8
7
,
5
0
0
2
9
0
,
5
3
2
5
1
0
,
5
0
0
2
2
5
,
3
6
2
6
5
7
,
8
8
0
3
8
3
,
7
9
4
5
8
7
,
1
9
8
3
,
7
6
1
,
5
3
6
M
a
r
y
l
a
n
d
0
2
0
7
,
0
0
0
4
5
5
,
2
0
0
5
4
8
,
9
1
8
3
1
2
,
3
7
9
1
3
9
,
0
0
0
9
9
,
0
0
0
1
3
0
,
0
0
0
4
5
,
0
0
0
1
,
9
3
6
,
4
9
7
M
i
s
s
i
s
s
i
p
p
i
0
2
2
0
,
5
7
7
3
0
,
0
0
0
1
6
4
,
9
7
0
5
9
4
,
8
3
5
1
5
7
,
0
0
0
0
2
0
0
,
0
0
0
2
2
6
,
0
0
0
1
,
5
9
3
,
3
8
2
N
e
w
Y
o
r
k
0
2
4
3
,
1
4
0
1
8
7
,
6
5
4
9
2
0
,
0
0
0
7
5
,
0
0
0
2
5
,
0
0
0
1
4
5
,
0
0
0
2
6
5
,
0
0
0
3
5
,
0
0
0
1
,
8
9
5
,
7
9
4
N
o
r
t
h
C
a
r
o
l
i
n
a
0
0
0
1
,
0
5
0
,
0
0
0
0
4
4
4
,
9
0
8
2
0
0
,
0
0
0
1
5
7
,
5
0
0
6
7
3
,
6
7
3
2
,
5
2
6
,
0
8
1
O
h
i
o
0
1
5
3
,
9
4
0
4
0
6
,
2
1
3
4
5
1
,
9
2
1
6
1
3
,
7
0
0
2
4
4
,
5
0
0
1
,
6
7
3
,
3
3
0
1
5
,
0
0
0
5
0
,
0
0
0
3
,
6
0
8
,
6
0
4
P
e
n
n
s
y
l
v
a
n
i
a
3
1
3
,
3
9
7
1
,
0
7
2
,
8
0
8
1
,
0
9
9
,
0
0
0
1
9
9
,
9
0
0
6
5
9
,
6
3
6
3
8
3
,
7
5
0
3
6
2
,
0
0
0
4
6
0
,
0
0
0
1
2
5
,
0
0
0
4
,
6
7
5
,
4
9
1
S
o
u
t
h
C
a
r
o
l
i
n
a
0
5
0
0
,
0
0
0
0
7
5
0
,
0
0
0
1
1
9
,
1
0
5
7
5
2
,
0
0
0
5
0
0
,
0
0
0
8
9
1
,
3
8
2
5
8
,
7
5
0
3
,
5
7
1
,
2
3
7
T
e
n
n
e
s
s
e
e
0
4
5
1
,
7
4
8
3
2
5
,
0
0
0
3
9
0
,
0
0
0
4
1
0
,
1
8
5
1
8
5
,
0
0
0
9
3
,
0
0
0
2
0
1
,
1
3
3
3
5
,
0
0
0
2
,
0
9
1
,
0
6
6
V
i
r
g
i
n
i
a
3
7
4
,
5
0
0
0
4
9
5
,
3
8
0
4
1
6
,
5
2
7
4
7
6
,
6
3
0
4
3
3
,
0
0
0
1
8
1
,
2
0
0
2
8
0
,
3
9
1
9
0
,
0
0
0
2
,
7
4
7
,
6
2
8
W
e
s
t
V
i
r
g
i
n
i
a
1
4
2
,
1
5
0
7
5
2
,
9
4
8
1
,
0
7
5
,
9
1
6
3
0
0
,
5
3
7
1
,
6
7
3
,
7
4
8
1
,
0
4
2
,
8
3
5
5
8
2
,
0
0
0
4
5
8
,
4
0
0
1
,
1
0
5
,
4
7
4
7
,
1
3
4
,
0
0
8
R
e
g
i
o
n
a
l
1
4
,
0
0
0
3
3
2
,
5
0
3
1
2
4
,
6
2
3
2
6
4
,
9
0
1
1
6
9
,
1
6
2
7
9
,
6
1
0
9
7
3
,
5
7
4
1
0
8
,
0
6
3
7
5
,
0
0
0
2
,
4
0
1
,
4
3
6
T
o
t
a
l
A
R
C
1
,
0
4
6
,
8
1
6
4
,
3
1
9
,
6
6
4
5
,
8
0
9
,
9
7
5
6
,
2
8
5
,
3
9
3
6
,
3
9
5
,
2
5
9
5
,
1
2
6
,
4
0
7
5
,
6
4
1
,
9
8
4
4
,
7
9
3
,
3
0
2
3
,
5
5
2
,
8
8
8
4
2
,
9
7
1
,
6
8
8
*
F
i
g
u
r
e
s
i
n
c
l
u
d
e
a
l
l
p
r
o
j
e
c
t
s
(
c
l
o
s
e
d
a
n
d
o
p
e
n
)
a
n
d
i
n
v
e
s
t
m
e
n
t
s
f
r
o
m
a
l
l
f
u
n
d
i
n
g
s
o
u
r
c
e
s
w
i
t
h
i
n
A
R
C
.
106
CHAPTER 9
CONCLUSIONS AND RECOMMENDATIONS
CONCLUSIONS
This evaluation was designed to assess the outcomes achieved by the portfolio
of projects that received investments from ARC’s Entrepreneurship Initiative from
1997-2005, as well as the broader policy impacts that accrued to the region as a
result of this initiative. By considering a limited set of performance measures
collected for the universe of closed projects and a more detailed set of measures
and insights gained from in-depth study of a sample of projects, the evaluation
team was able to develop a thorough understanding of what EI investments have
meant to individual projects and the region as a whole. Job and business
creation have been important and significant outcomes of EI investments, as
summarized in Table 9.1 below. Over 12,000 jobs have been created or retained
and over 1,700 new businesses created, at a cost that suggests an efficient
allocation of resources relative to other similar types of economic development
programs. However, these measures tell only part of the story of impacts. EI
investments have helped to train teachers and expose students to
entrepreneurship concepts, almost 12,000 throughout the region. These
investments have been instrumental in attracting almost $73 million in private
investment to support entrepreneurship development in the region. And, through
their educational investments and the demonstration effect of the projects
funded, ARC has made entrepreneurship a legitimate and desirable economic
development activity in local communities and raised awareness about the
importance of the entrepreneurial and small business sector to the region’s
economy.
Based on the results of this evaluation, the evaluation team concludes that the
ARC Entrepreneurship Initiative has had an impact in the region by creating more
entrepreneurs in the pipeline (through its entrepreneurship education
investments), better informed entrepreneurs and better skilled entrepreneurs
(through its technical assistance and training, incubator and sector investments).
These investments have resulted in more job creating businesses. EI
investments have also helped to create and enhance capacity to support
entrepreneurship development in the region, most prominently through
investments in equity capital funds that seeded and facilitated the creation of a
developmental venture capital industry in the region. In addition, ARC
investments have created the beginning of culture change in the region –
increased recognition of the importance of entrepreneurship as an economic
development strategy and increased support for those people and organizations
that are committing their talents and resources to pursuing an entrepreneurial
path. ARC investments have energized and empowered a new set of actors in
the region, especially non-profit organizations, who continue to provide
innovative, entrepreneurial leadership in the region.
107
Table 9.1. Summary of Quantitative and Qualitative Evaluation Findings
QUANTITATIVE OUTCOMES FOR THE UNIVERSE OF PROJECTS
Jobs created 9,156
Jobs retained 3,022
New businesses created 1,787
Businesses served 8,242
Actual private $ leveraged $72,802,868
Public cost / job created or retained $579 - $3,994
Public cost / business created or expanded $2,988 - $7,818
QUANTITATIVE OUTCOMES FOR THE SAMPLE PROJECTS
Jobs created 4,332
Jobs retained 1,351
New businesses created 1,083
Businesses served 2,957
Incubator clients served 475
Students or teachers trained 11,634
Actual private $ leveraged $15,856,275
QUALITATIVE OUTCOMES OF ARC EI INVESTMENTS
Raised the profile of entrepreneurship as a development strategy and helped change the mindset
within the region
Provided start-up funding for innovative projects that would not have happened “but for” ARC
investment
Leveraged additional resources that helped some projects achieve scale and impact
Facilitated networking and collaboration among practitioners
Helped change people’s attitudes, particularly among youth and their teachers
RECOMMENDATIONS
ARC’s Entrepreneurship Initiative was plowing new ground when first conceived
in 1997. At that time, only a handful of states and localities were experimenting
with approaches that placed entrepreneurs and their companies at the center of
economic development thinking and strategy. Today, the economic development
landscape is quite different. Entrepreneurship has become a mainstream
component of local economic development strategies and has been making
strong inroads in education at the primary, secondary, and post-secondary
levels.
In the decade since EI’s inception, the field of entrepreneurship development has
been created. An important body of knowledge about effective strategies and
programs now exists. Yet, the field must still be defined as “emerging.” Lists of
effective practices and programs have been developed,
101
but these program
ideas have not been accompanied with a rigorous approach to program design,
management, and evaluation.
102
101
For example, see Markley, Macke and Luther (2005); Dabson, et al. (2003); OECD (2003).
102
OECD (Forthcoming).
108
Because it was a forerunner of today’s programmatic innovations, ARC’s EI
experience can and should be tapped for useful ideas on how to improve the field
of entrepreneurial development. With a decade of experience under its belt,
ARC staff and partners can offer invaluable guidance to other federal, state, and
local policy makers. For example, the Department of Labor’s new WIRED
(Workforce Innovation in Regional Economic Development) initiative has
invested more than $300 million in regional projects designed in part to stimulate
local entrepreneurial activity. These program managers could learn a great deal
from the EI experience.
In an effort to strengthen future entrepreneurial development programs, the team
has developed a set of recommendations that fall into three categories:
Recommendations for entrepreneurship development investments
Recommendations for creating a “best in class” metrics system
Recommendations for program design and management.
Recommendations for Entrepreneurship Development Investments
Entrepreneurship development initiatives should include assessment of
existing capacity and capacity building activities as part of project design.
The evaluation of ARC’s EI investments highlighted the value of capacity building
– visioning, leadership development (youth and adult), asset mapping,
community engagement, and strategy development. Successful entrepreneurship
development initiatives build on existing capabilities, as was seen with Kentucky
Highlands Investment Corporation, Appalachian Community Enterprise in
Georgia, and the Shoals Entrepreneurial Center in Alabama, or create new
capacity , as evidenced by the PACERS youth entrepreneurship program in
Alabama and the Tech 2020 program in Tennessee. Therefore, including
capacity building as an integral part of entrepreneurship investments should
result in stronger, more effective initiatives.
ARC (and other federal, state or local program managers) could address this
finding in several ways. At the most basic level, it could require applicants to
provide an assessment of existing community capacity during the grant
application process. Such an assessment would be more than a simple listing of
assets. Rather, project leaders should identify their leadership team, the partner
organizations, their level of resource commitment to the initiative, and how this
existing capacity will be used in substantive ways to support regional
entrepreneurship initiatives. They could also be required to outline a plan for
enhancing resource or leadership capacity if necessary.
A second approach would create closer links between existing community
capacity building programs and entrepreneurial development investments.
Many of the goals pursued through ARC’s community capacity investments, such
109
as increased organizational capacity, enhanced skills for individuals, and
improved economic development outcomes,
103
are similar or identical to those
pursued via an entrepreneurial development strategy. Many program
interventions, such as enhanced technical assistance to business, also share
similarities. Building these linkages will be eased thanks to these close
connections. Indeed, ARC has already experimented with this approach under its
2005 Asset-Based Development Initiative which explicitly identified promotion of
civic entrepreneurship as a primary program goal. ARC might consider more
direct linkages between entrepreneurial development and other existing
programs.
Finally, ARC might consider future programs that operate according to a staged
process where initial investments are made in a host of community capacity
building efforts. Entrepreneurship development would be considered a “second-
stage” investment for communities and programs that had relatively robust local
capacity in place. This staging of investments would likely produce better
community outcomes. Not surprisingly, the EI experience indicates that the
presence of strong local capacity has a significant effect in contributing to better
economic development outcomes.
Entrepreneurship development investments should be made with a focus
on the long term.
The long-term nature of entrepreneurship development requires a long-term
approach to investment. If a goal of these initiatives is to transform the culture of
Appalachia, or another region, then a 10-year or longer time horizon is more
appropriate than the more typical one to three year grant cycle. Throughout the
interviews, local program managers commented that ARC investments were too
short-lived to generate sustainable local impacts. Programs could be initiated and
local momentum could be generated. Yet, in most cases, ARC funds ran out at
this critical point of impact, two to three years after program initiation.
This pattern creates a dilemma for program managers. Like a start-up
entrepreneur, they are intensely focused on scaling up their new programs.
Because entrepreneurial companies take time to generate outcomes (in terms of
new products or job creation), support programs are unlikely to be able to boast
of major economic development impacts during the start-up phase. They may
have only a few isolated success stories after the first year or two of operations.
Without these large (and quick) community impacts, program managers are then
handicapped in their ability to identify other sources for program funds.
ARC should consider giving preference to multi-year funding commitments that
would provide a flow of resources over a longer time horizon, assuming
performance on the part of the local partner and availability of federal funding on
the part of ARC. These dynamics do not imply that projects should have an
103
For a review, see Brian Kleiner, et al., July 2004, xi.
110
indefinite period of funding. It is appropriate that project leaders articulate a plan
for sustainability and show progress toward reaching that goal as a condition of
long-term investment. But, multi-year funding commitments would provide a
stable base from which these local projects could develop.
Entrepreneurship development initiatives receiving investments should be
market driven and practice continuous improvement.
The needs of the customer – defined as youth, entrepreneurs or communities –
must be the key drivers of entrepreneurship development. Initiatives should be
designed to include mechanisms that obtain performance feedback, allow for
mid-course corrections, and in some cases, redefine project goals, based on
what project leaders are learning from their markets. While the field of
entrepreneurship development has come a long way since the EI began in 1997,
it is still in an experimentation stage. Organizations throughout Appalachia and
the country are continually evolving new ways of working with entrepreneurs and
communities that can inform both the design of new and the improvement of
existing efforts to encourage entrepreneurship as a core economic development
strategy.
ARC should consider requiring project leaders to conduct annual performance
reviews. These reviews could be conducted as part of the ongoing evaluation
that is recommended below. And, they should include some assessment of the
project’s market – for example, assessing the experience of entrepreneurs
participating in a particular training or technical assistance program through focus
groups or customer surveys. It is recommended that ARC participate in these
reviews, providing constructive feedback and suggesting resources that local
partners might use to improve program performance. ARC could use its network
of regional and national partners to link a local project with a more experienced
practitioner elsewhere to address a specific performance issue or concern.
Emphasis should be placed on investing in initiatives that demonstrate the
ability to form regional partnerships and collaborations.
With capacity and resource limitations a reality for most organizations, the ability
to form dynamic and effective partnerships that share resources becomes
paramount to success. These cross-organizational and cross-regional
collaborations should be emphasized in the design of entrepreneurship
initiatives, and effective partnerships should be rewarded as part of the
investment process. ARC should require extensive community partnerships for
all of its future entrepreneurial development investments.
However, ARC must recognize that it take resources to facilitate collaboration.
While local projects should be required to demonstrate true collaboration across
geography (e.g., multi-county projects) and organizations (e.g., public, private
and non-profit partners), ARC should consider requiring that project leaders
111
demonstrate budgetary commitment to cover the costs of collaboration. This
resource commitment could come through allocating part of ARC funds to
support collaborative activities or through the allocation of local matching dollars
to this process. ARC can help further true collaboration by making it a priority in
grant making and by educating local partners to the resource realities associated
with collaboration.
Recommendations for Creating a “Best in Class” Metrics System
Job creation is an overused metric, paints an incomplete picture of the
outcomes of entrepreneurship development investments, and should be
replaced by an “entrepreneurship development metrics portfolio.”
ARC’s existing performance measurement system requires all projects to report
the following relevant measures as part of its final project close-out process:
Businesses Served
Jobs Created
Jobs Retained
Project Participants
New Businesses Created.
These metrics provide some useful insights, but most project managers and
outside experts felt they were poorly suited to providing a complete picture of the
EI’s impact. They were better tailored to measure the impact of more traditional
economic development investments in physical infrastructure. In recognition of
these shortcomings, nearly all local projects devised their own performance
indicators which ranged from simple additions such as use of customer
satisfaction surveys to more extensive systems that tracked the financial
performance of assisted companies.
While job creation is reported as a result of ARC’s EI investments, a much richer
understanding of the initiative’s impact has come through efforts to define and
capture outcomes as measured by a broad set of performance metrics. These
metrics include both quantitative output measures, e.g., students trained, as well
as more qualitative measures, such as enhanced leadership capacity and
resources leveraged through partnerships. What is clearly needed is the creation
of a portfolio of metrics linked to particular types of entrepreneurship
development programs. These metrics would be outcome measures that are
clearly linked to the goals of particular projects. For example, to evaluate the
impacts of entrepreneurship education efforts, metrics would focus on measuring
the outcomes of training – whether young people are more likely to consider
entrepreneurship as a career path, whether they go on to create businesses in
the future, whether they return home to the region, etc.
112
Earlier in this report, Table 4.1 provided a portfolio of performance or outcome
metrics that can be used to measure the impacts of entrepreneurship
development investments across the program types that are part of ARC’s EI
portfolio. These outcome measures were drawn from a review of the literature
and formed a framework for this evaluation. While this framework could be used
as a basis for discussion by ARC and its grantees as part of the evaluation
partnership approach described below, Table 9.2 provides a more limited set of
metrics that might be operationalized into a “best in class” metrics system. The
proposed metrics include quantitative measures that would be collected from
project leaders, who in turn would use survey and other tools to gather data from
their customers. In addition, there are a number of metrics that will capture the
qualitative impacts that were observed through this evaluation of the EI
investments. Outcome measures to capture these qualitative changes in cultural
attitudes would need to become part of the evaluation system adopted by
grantees, using tools and techniques developed with support of ARC or another
funding organization.
Table 9.2. Proposed “Best in Class” Outcome Measures
Capital Access – Projects designed to provide
access to a range of capital resources to help
businesses start and grow and, in the process,
become stronger competitors in local, regional,
national and/or international markets.
Number of new businesses financed
(measure of business starts)
Number of jobs (FTEs) created/retained
(measure of business growth)
Percent of funded firms still in business
(measure of business performance)
Average wage/job created (measure of
business performance)
Percent change profitability (measure of
business performance)
Sectors – Projects focused on improving the
start up, growth, and performance of
businesses in a particular sector and on
growing particular sectors of the local or
regional economy. Projects included
networking activities designed to improve
business performance.
Amount of increased sales ($) attributed to
network or sector participation (measure
of business performance)
Number of jobs (FTEs) created/retained
(measure of business growth)
Increase in number of business starts in
targeted sector (measure of business start
up)
Change in total sector sales over time
(measure of growth in the sector)
Incubators – Projects focused on creating a
physical space for businesses to start up and
grow, with the goal of graduating these firms
into the local or regional economy.
Number jobs (FTEs) created/retained
while in the incubator (measure of
business starts/growth)
Number jobs (FTEs) created/retained after
graduation (measure of business growth)
Amount of capital ($) raised by tenants
(measure of business growth)
Percent of business tenants retained in
the service area (measure of local/regional
economic impact)
113
Entrepreneurship Education* – Projects
focused on exposing young people to
entrepreneurial concepts and experiences to
enhance their understanding of
entrepreneurship as a career option and to
encourage youth retention through
entrepreneurship.
Increase in awareness of business
concepts (measure of exposure to
entrepreneurship concepts)
Increase in number of participants
considering business creation as a career
option (measure of exposure to
entrepreneurship concepts)
Number of students that stay or return to
the service area (measure of impact on
youth retention)
Technical Assistance and Training –
Projects designed to build the skills of
individual entrepreneurs so that they can start
and grow their businesses and create stronger
enterprises in the local and regional economy.
Number jobs (FTEs) created/retained
(measure of business growth)
Number of clients still in business
(measure of business performance)
Private capital ($) raised by clients
(measure of business growth)
Average wage/job created (measure of
business performance)
Percent change in profitability (measure of
business performance)
Culture Change – Projects often achieve
qualitative impacts (both intended and
unintended) that relate to changes in people’s
attitudes toward entrepreneurship, their view of
the importance of entrepreneurs to the
local/regional economy, and the value placed
on collaborative decision making and
partnerships to create a more supportive
environment for entrepreneurs.
Public investment ($) in entrepreneurship
development activities (pre vs. post project
investment)
Private investment leveraged ($) as a
result of project investment in
entrepreneurship development
Perceived change in community/regional
support for entrepreneurship development
(as measure through pre and post-
investment community surveys)
Increased collaboration among support
providers (as measured by the number of
partners contributing resources to
entrepreneurship development)
* The metrics developed for entrepreneurship education projects refer to potential outcomes of
these projects as economic development initiatives. Therefore, metrics focus on outcomes that
have potential impacts on the community and not just the individual young person. These
individual outcomes have been measured through metrics related to changes in student
performance, e.g., increased test scores, increased applications to college, improved reading,
and increased leadership activities in school/community.
A “best in class” metrics system requires investment in a “best in class”
evaluation system.
Performance measurement should be viewed as an integral part of program
development – from the perspective of funding agencies like ARC and project
leaders. One of the first steps in developing any initiative needs to be an
articulation of program goals – what are you trying to achieve – followed by
identification of how success or performance will be measured. These inputs
form a performance measurement system that can be used by local project
leaders to report on success, broaden support, and attract additional resources
and partners to the effort. From ARC’s perspective, developing the evaluation
114
framework before investment will help to insure that individual projects contribute
to the overall goals set forth by ARC and that the agency will have appropriate
metrics to use to report on the performance of the overall initiative.
Ex post evaluations of major investments like ARC’s EI face serious challenges
in terms of the collection and integrity of data; recalling accurately the impacts of
a project that ended 7-10 years ago was a challenge for most project leaders.
Greater investment must be made in establishing criteria and providing funding
for an ongoing evaluation of entrepreneurship development initiatives as they
unfold. As noted above, local grantees regularly developed their own in-house
metrics to supplement those required as part of ARC’s grants process. While
these in-house efforts generated a fair amount of useful data, they provided no
means to aggregate data across program types, across regions, or across the
entire EI spectrum.
ARC should support the creation of a performance measurement system for
future investments by developing a participatory evaluation system in partnership
with grantees. This measurement system would be developed by grantees and
their customers, i.e., the entrepreneurs, with support from ARC, and would be
designed to provide project leaders with useful information that can be used to
adapt programs to changing circumstances as well as to report to ARC on project
performance. The evaluation framework should be built into the program from the
beginning, and project leaders would be expected to sign off on that evaluation
system as part of a grant agreement. By taking this partnership approach to
evaluation, ARC would be in a stronger position to hold project leaders
accountable for generating the outcome metrics identified for the project, and to
provide them with feedback on performance.
The evaluation team also recommends that ARC consider two sets of outcome
metrics: a base set of metrics for all programs, and a tailored set of metrics for
each specific type of program intervention, as laid out in Table 9.2. For example,
an entrepreneurship training program and a new business incubator might both
be assessed according to traditional metrics of job creation or new business
starts. Beyond these base measures, incubators might be assessed according to
the number of firms graduated from the facility, the incubator’s annual revenue,
and a range of financial metrics for businesses served by the incubator. Training
programs might use a common customer satisfaction survey or other tools that
assess whether participants gained new skills or knowledge, supplemented by
follow-up surveys to collect financial performance metrics for business
customers. Youth entrepreneurship education programs might develop metrics
related to measuring the entrepreneurial skills acquired by young people who
participate in these programs.
ARC has an opportunity to take the lead, among any newly authorized regional
authorities and other federal agencies, in developing and standardizing
assessment criteria and methods that can have an impact on how
115
entrepreneurship development investments are evaluated and, more broadly,
how the impacts of economic development programs are measured.
Recommendations for Program Design and Management
ARC’s initiative process should be regularized so that state program
managers can more effectively plan for and promote use of the resources.
Interviews indicate that state program managers and local grantees like the ARC
initiative process. They like the focus on specialized issues and concerns, and
they like the learning opportunities provided by ARC through various sponsored
conferences and reports. They are less enamored with the episodic nature of the
initiative process. They would like more input into the discussions about new
initiative topics, and more information on initiatives being considered in the
future.
Given the gate keeping role of state leaders, ARC should provide structure and
consistency to the initiatives to encourage the active buy-in of people at the state
level. A more transparent and open process for community input on project
design and implementation would also help ARC create more effective initiatives
and empower local people and organizations to actively participate in these
efforts.
As noted in Chapter 8, ARC went through an extensive process of getting input
into the design of the EI. Advisory groups of state leaders and private/non-profit
sector practitioners and others with experience in entrepreneurship development
helped to inform the decision making behind the EI. Meetings and educational
events throughout the region provided opportunities for state and local leaders to
provide input to the process. In spite of these significant steps, there remains a
sense that local practitioners had limited input into the design and
implementation of the EI. One way to address this perception might work as
follows. As ARC’s leadership considers new topics for potential initiatives, it can
open a process for outside input. Ideally, suggestions should be provided in
multiple formats from a formal request for comment in the Federal Register to the
use of blogs as a means to generate online discussion. In-person sessions, such
as town hall meetings held in widely dispersed locations throughout the region,
should also be considered. ARC should consider tapping into the ever growing
infrastructure of entrepreneurship service providers in the region, using online
surveys to get their input on what is needed in the region. The US Department of
Agriculture’s series of sponsored Farm Bill Forums, to discuss key sections of the
2007 Farm Bill, offers one excellent model for organizing public outreach and
discussion.
104
104
Summaries of the Farm Bill Forums, held throughout 2006, can be found at
.
116
ARC’s proven experience can be applied to developing and delivering
effective, region-wide education programs that help make the case for
entrepreneurship as a core economic development strategy for the
Appalachian region.
There continues to be a strong need to make the case for entrepreneurship,
particularly among local elected officials and traditional economic developers.
Throughout the EI, ARC has organized educational opportunities to share
information about topics such as capital access, business incubation, and
entrepreneurship education. What is needed now, however, is a broader effort to
provide community leaders, elected and others, with the understanding and tools
they need to embrace entrepreneurship as part of an economic development
strategy.
ARC has developed a reputation as a trusted authority in this field and also has
the lessons learned from ten years of entrepreneurship development investments
to share throughout the region. In addition, ARC’s partnership approach in the
beginning of the EI process can be brought to bear on this educational effort,
drawing on resources and experiences of other organizations working on
entrepreneurship development throughout the country.
A Next Generation Entrepreneurship Initiative
This evaluation has generated a host of new ideas and lessons learned, but one
prevailing idea has emerged throughout the evaluation process – additional
investments in entrepreneurship development throughout Appalachia are still in
significant demand. Given the success and capacity that the ARC EI has already
been building, the evidence suggests that continued ARC investment in
entrepreneurship development in the region is a compellingly logical and vital
next step.
To build on this momentum, ARC should create a Next Generation
Entrepreneurship Innovation Initiative that will be groundbreaking in its
design. A long-term investment is recommended that incorporates all the
learning from the EI and the emerging entrepreneurship development field.
It will include four critical elements:
The Entrepreneurship Innovation Fund would provide selective,
competitive, strategic investments in “next level” entrepreneurship
development activities throughout the region. The Entrepreneurship
Innovation Fund would not be tied to individual states, but would be
competitively awarded across the region. Investments would be made in
initiatives that demonstrate a holistic, systems approach to
entrepreneurship development, with an emphasis on those initiatives that
have the potential to be transformational and sustainable. It is
recommended that ARC take a portfolio approach to these investments –
117
investing in more proven innovations as well as those that offer promise
but are still early stage innovations.
The second element would be a pool of funds distributed to the states for
investments in first tier entrepreneurship projects at the ground level.
Similar to the EI, these projects would build capacity and fulfill distinct
entrepreneurship metrics that are developed by the communities and ARC
working together.
In the interest of capacity building, ARC should fund the development of
“Entrepreneurship Innovation – Guidelines for the Future” – a framework
for communities to use based on what ARC has learned from 10 years of
investment in this field and what its partner organizations across the
country have learned through their various activities.
The fourth element would be a built-in evaluation system that is initiated
from the beginning of the Entrepreneurship Innovation Initiative. It would
incorporate the “best in class” metrics derived from this evaluation,
discussions with ARC, and input from the field. This evaluation system will
be essential to making the case as well as measuring and ensuring
impact.
This capstone recommendation is based on the recognition that, while the
Entrepreneurship Initiative has achieved important impacts at the community
level, the region has not seen widespread or significant policy change at the state
level. The entrepreneurship context assessment in Chapter 5 suggests that many
parts of the region continue to lag the nation, particularly in terms of income
generated by entrepreneurial activities. The Entrepreneurship Innovation
Initiative would give ARC an opportunity to make investments that are deeper
and more transformational, generating impacts that are influential in achieving
policy change at the state, as well as local, level throughout the region. This
initiative would also provide an opportunity to implement a participatory
evaluation system that can generate the data and insights that will provide a
deeper understanding of how ARC investments help to change the economic
outlook and performance of the region.
118
BIBLIOGRAPHY
Acs, Zoltan J. “The State of the Literature on Small to Medium Sized Enterprises
and Entrepreneurship in Low-Income Communities.” Federal Reserve
Bank of Kansas City and Ewing Marion Kauffman Foundation, 2006.
Acs, Zoltan J. and Catherine Armington. Using Census BITS to Explore,
Entrepreneurship, Geography, and Economic Growth. Washington, DC:
US Small Business Administration Office of Advocacy, 2005.
Advanced Research Technologies. The Innovation-Entrepreneurship Nexus.
Washington, DC: US Small Business Administration Office of Advocacy,
2005.
Aernoudt, Rudy. “Incubators: Tool for Entrepreneurship?” Small Business
Economics. Vol. 23, No. 2, September 2004. 127-135.
Alliance for Regional Stewardship. Regional Indicators: Telling Stories,
Measuring Trends, Inspiring Actions. ARS Monograph Series No. 10.
November 2005.
Appalachian Regional Commission. Appalachian Youth Entrepreneurship
Springboard Award: 2002 and 2003 Winners. Washington, DC:
Appalachian Regional Commission, 2004.
Appalachian Regional Commission. Capitalizing on Rural Communities.
Washington, DC: Appalachian Regional Commission, 2000.
Appalachian Regional Commission. Entrepreneurship Initiative: Program
Summary and Approved Projects. Washington, DC: Appalachian
Regional Commission, 2003.
Atkinson, Robert D. The Past and Future of America’s Economy.
Northhampton, MA: Edward Elgar, 2004.
Atrostic, B.K. “Measuring U.S. Innovative Activity.” US Census Bureau Center
for Economic Studies Working Paper (07-11). March 2007.
Association of University Technology Managers. AUTM U.S. Licensing Survey,
FY 2004: A Survey Summary of Technology Licensing (and Related)
Performance for U.S. Academic and Nonprofit Institutions, and
Technology Investment Firms. Washington, DC: AUTM, 2004.
Audretsch, David, et al. “The Knowledge Filter and Economic Growth: The Role
of Scientist Entrepreneurship.” Ewing Marion Kauffman Foundation,
2006.
119
Barkley, David L. et al. Establishing Nontraditional Venture Capital Institutions:
Lessons Learned. Rural Equity Capital Initiative Study of Nontraditional
Venture Capital Institutions. Columbia, MO: Rural Policy Research
Institute, 2001.
Barkley, David L. and Deborah Markley. The Development of an Entrepreneurial
Support Organization: The Case of the Kentucky Highlands Investment
Corporation. RUPRI Center for Rural Entrepreneurship Research Case
Study Series Number 1. March 2003.
Bartik, Timothy J. Evaluating the Impacts of Economic Development Programs
on Local Economic Outcomes: What Has Been Done and What is
Doable? Upjohn Institute Staff Working Paper (No. 03-89). Kalamazoo,
MI: Upjohn Institute, 2002.
Bartik, Timothy J. and Richard D. Bingham. "Can Economic Development
Programs be Evaluated?" Dilemmas of Urban Economic Development.
Eds. R.D. Bingham and Robert Mier. Thousand Oaks, CA: Sage, 1997.
246-277.
Block, Z. and S. A. Stumpf. “Entrepreneurship Education Research: Experience
and Challenge.” The State of the Art of Entrepreneurship. Eds. D. L.
Sexton and J. D. Kasarda. Boston, MA: PWS-Kent Publishing, 1992. 17-
45.
Bollingtoft, Anne and John P. Ulhoi. “The Networked Business Incubator –
Leveraging Entrepreneurial Agency?” Journal of Business Venturing Vol.
20, 2005: 265-290.
Brandow Company and Economic Development Research Group. Evaluation of
the Appalachian Regional Commission’s Infrastructure and Public Works
Program Projects, Final Report to the Appalachian Regional Commission.
Washington, DC: Appalachian Regional Commission, 2000.
Brown, J. David, John S. Earle, and Dana Lup. “What Makes Small Firms Grow?
Finance, Human Capital, Technical Assistance, and the Business
Environment in Romania.” Economic Development and Cultural Change
Vol. 54, No. 1, 2005: 33-70.
Burchell, Robert W. EDA RLFs: Performance Evaluation. Washington, DC: U.S.
Economic Development Administration, 2002.
Burchell, Robert W. EDA Defense Adjustment Program: Performance
Evaluation. Washington, DC: U.S. Economic Development
Administration, 1997.
120
Burchell, Robert W. EDA Public Works Program: Performance Evaluation.
Washington, DC: U.S. Economic Development Administration, 1997.
Carter, Sara, et al. “Beyond Portfolio Entrepreneurship: Multiple Income Sources
in Small Firms.” Entrepreneurship and Regional Development Vol. 16,
2004: 481-499.
Charney, Alberta and Gary Libecap. The Impact of Entrepreneurship Education:
An Evaluation of the Berger Entrepreneurship Program at the University of
Arizona, 1985-1999. Kauffman Center for Entrepreneurial Leadership,
2000.
Chukumba, Celestine and Richard Jensen. Invention, Entrepreneurship, and
Start-Ups. National Bureau of Economic Research Working Paper
(#11475). July 2005.
Coalition of Community Development Financial Institutions. Providing Capital
Building Communities Creating Impact. Washington, DC: CDFI Coalition,
2004.
Coastal Enterprises, Inc. Measuring Impact in Practice. Wiscasset, ME: Coastal
Enterprises Inc., February 2006.
Colgan, Charles S. and Bruce Andrews. Evaluation of Maine Technology
Institute Programs. University of Southern Maine Center for Business and
Economic Research. December 2004.
Collender, Ray et al. “Financial Markets Serve Rural Areas Fairly Well.” Rural
Development Perspectives Vol. 14, No. 1, May 1999: 28-35.
Community Development Venture Capital Association. Measuring Impacts
Toolkit. New York: Community Development Venture Capital Alliance,
2005.
Cooke, Phil. Regional Innovation Systems as Public Goods.” Working Paper.
United Nations Industrial Development Organization, 2005.
Council on Competitiveness. Measuring Regional Innovation. Washington, DC:
Council on Competitiveness, 2006.
Council on Competitiveness. Innovate America. Washington, DC: Council on
Competitiveness, December 2004
Czohara, Laura and Julia Melkers. Performance Measurement in State Economic
Development Agencies: Lessons and Next Steps for GDITT. Georgia
State University Fiscal Research Center Report No. 92. February 2004.
121
Dabson, Brian, et al. Mapping Rural Entrepreneurship. Washington, D.C.: CFED,
August 2003.
Drabenstott, Mark. A Review of the Federal Role in Regional Economic
Development. Working Paper. Center for the Study of Rural America,
Federal Reserve Bank of Kansas City, May 2005.
Eberts, Randall W., George A. Erickcek, and Jack Kleinhenz. “Development of
Regional Economic Dashboard.” Employment Research July 2006: 3-5.
Edgcomb, Elaine L. and Joyce A. Klein. Opening Doors, Building Ownership:
Fulfilling the Promise of Microenterprise in the United States. Washington,
D.C.: FIELD, A Program of the Aspen Institute, 2005.
Eisinger, Peter K. The Rise of the Entrepreneurial State: State and Local
Economic Development Policy in the United States. Madison, WI:
University of Wisconsin Press, 1988.
Elfring, Tom and Willem Hulsink. “Networks in Entrepreneurship: The Case of
High-technology Firms.” Small Business Economics Vol. 21, No. 4, 2003:
409-422.
Emery, Mary and Cornelia Flora. “Spiraling Up: Mapping Community
Transformation with Community Capitals Framework.” Journal of the
Community Development Society Vol. 37, No. 1, Spring 2006: 19-35.
Enterprise Corporation for the Delta. Enterprise Corporation for the Delta
Program Monitoring Report – Business Technical Assistance 2003.
Jackson, MS: Enterprise Corporation for the Delta, 2003.
European Commission, Directorate General, Enterprise and Industry. The
PAXIS Manual for Innovation Policy Makers and Practitioners. Brussels:
European Commission, 2006.
Fairlie, Robert W. Kaufffman Index of Entrepreneurial Activity, State Report 2005.
Kansas City: Ewing Marion Kauffman Foundation, 2006.
Fayolle, Alain. “Evaluation of Entrepreneurship Education: Behavior Performing
or Intention Increasing?” International Journal of Entrepreneurship and
Small Business Vol. 2, No.1, 2005: 89-98.
Feldman, Maryann. “The Entrepreneurial Event Revisited: Firm Formation in a
Regional Context.” Industrial and Corporate Change Vol. 10, No. 4, 2001:
861-891.
122
Felsenstein, Daniel and Aliza Fleischer. “Small-Scale Entrepreneurship and
Access to Capital in Peripheral Locations: An Empirical Analysis.” Growth
and Change Vol. 33, 2002: 196-215.
Fisher, Peter. “The Fiscal Consequences of Competition for Capital.” Reining in
the Competition for Capital. Ed. Ann Markusen. Kalamazoo, MI: WE
Upjohn Institute, 2007.
Foster, Lucia. Reallocation and Productivity Dynamics in the Appalachian
Region. US Census Bureau Center for Economic Studies Working Paper
(06-03). January 2006.
Giannetti, Mariassunta and Andrei Simonov. On the Determinants of
Entrepreneurial Activity: Individual Characteristics, Economic
Environment, and Social Norms. White Paper. Stockholm School of
Economics, June 2004.
Gilbert, Brett Anitra, David B. Audretsch, and Patricia P. McDougall. “The
Emergence of Entrepreneurship Policy.” Small Business Economics Vol.
22, 2004: 313-323.
Gillett, Sharon E., et al. Measuring the Economic Impact of Broadband
Deployment, Final Report to the Economic Development Administration.
Washington, DC: Economic Development Administration, February 2006.
Glasmeier, Amy K. Cost per Job Associated with EDA Investments in Urban and
Rural Areas. Washington, DC: U.S. Economic Development
Administration, 2002.
Goetz, Stephan J. and David Freshwater. “State-Level Determinants of
Entrepreneurship and a Preliminary Measure of Entrepreneurial Culture.”
Economic Development Quarterly Vol. 15, No. 1, February 2001: 58-70.
Greenburg, Elizabeth and Richard Reeder. Who Benefits from Business
Assistance Programs? Results of the ERS Rural Manufacturing Survey.
United States Department of Agriculture Information Bulletin Number 736-
04. 1998.
Greene, F.J. and D.J. Storey. “An Assessment of a Venture Capital Creation
Programme: the Case of Shell LiveWIRE.” Entrepreneurship and
Regional Development Vol. 16, No. 2, March 2004: 145-159.
Greenwood Consulting Group. A Survey of Business Incubators in Appalachia.
Washington, DC: Appalachian Regional Commission, July 2005.
123
Grimaldi, Rosa and Alessandro Grandi. “Business Incubators and New Venture
Creation: An Assessment of Incubating Models.” Technovation Vol. 25,
2005: 111-121.
Heard, Robert and John Sibert. Growing New Businesses with Seed and Venture
Capital: State Experiences and Options. Washington, DC: National
Governors Association, 2000.
Ho-Kim, Thu-Mai and Ernesto Venegas. Starting Up Economic Engines: Key
Factors for Growing Successful Start-Ups. Minnesota Department of
Employment and Economic Development Issue Brief. October 2003.
Huggins, R. “The Success and Failure of Policy-Implanted Inter-firm Network
Initiatives: Motivations, Processes and Structure.” Entrepreneurship and
Regional Development Vol. 12, 2000: 111-135.
Innovation Associates Inc. Accelerating Economic Development Through
University Technology Transfer. February 2005.
Institute for Competitiveness and Prosperity, Measuring Ontario’s Prosperity:
Developing an Economic Indicator System. Working Paper No. 2.
Ontario, Canada: Institute for Competitiveness and Prosperity, 2002.
Junior Achievement. The Impact on Students of Participation in Junior
Achievement: Selected Cumulative and Longitudinal Findings.
Monograph. January 26, 2004.
Katz, Jerome. And Another Thing. . . 2006 Coleman Foundation White Paper on
Entrepreneurship Education. 2006 Annual Meeting of US Association of
Small Business and Entrepreneurship, January 13, 2006.
Katz, Jerome A. “The Chronology and Intellectual Trajectory of American
Entrepreneurship Education, 1876-1999.” Journal of Business Venturing
Vol. 18, 2003: 283-300.
Kingsley, Gordon and Edward J. Malecki. “Networking for Competitiveness.”
Small Business Economics Vol. 23, 2004: 71–84.
Kleiner, Brian, et al. Evaluation of The Appalachian Regional Commission’s
Community Capacity-Building Projects. Washington, D.C.: Appalachian
Regional Commission, July 2004.
Kourilsky, Marilyn, and William B. Walstad. The E Generation. Dubuque, IA:
Kendall-Hunt, 2000.
124
Lambrecht, Johan and Fabrice Pirnay. “An Evaluation of Public Support
Measures for Private External Consultancies to SMEs in the Walloon
Region of Belgium.” Entrepreneurship and Regional Development Vol.
17, 2005: 89-108.
Lee, Lena. Entrepreneurship Education: A Compendium of Related Issues.
Working Paper. National University of Singapore Entrepreneurship Centre,
2005.
Lichtenstein, Gregg A. and Thomas S. Lyons. “The Entrepreneurial Development
System: Transforming Business Talent and Community Economies.”
Economic Development Quarterly Vol. 15, No. 1, 2001: 3-20.
Lichtenstein, Gregg A. and Thomas S. Lyons, Incubating New Enterprises: A
Guide to Successful Practice. Washington, DC: Aspen Institute, 1996.
Low, Sarah. “Regional Asset Indicators: Entrepreneurship Breadth and Depth.”
The Main Street Economist September 2004:1-4.
Low, Sarah, Jason Henderson, and Stephan Weiler. “Gauging a Region’s
Entrepreneurial Potential. “Economic Review Third Quarter, 2005.
Edward Lowe Foundation, et al. Michigan: Toward an Entrepreneurial Economy,
2005-2006. Cassopolis, MI; Edward Lowe Foundation, 2006.
Malecki, Edward J. “Geographical Environments for Entrepreneurship.”
International Journal of Entrepreneurship and Small Business
Forthcoming 2008.
Manigart, Sophie, et al. “Determinants of Required Return in Venture Capital
Investments: A Five Country Study.” Journal of Business Venturing Vol.
16, No. 6, July 2002: 291-312.
Maunula, Mari. The Perceived Value-Added of Venture Capital Investors.
Discussion Paper No. 1030. The Research Institute of the Finnish
Economy, March 2006.
Markley, Deborah, et al. Rural Equity Capital Initiative Study of Nontraditional
Venture Capital Institutions. Working Paper PB2001-11A-D. Rural Policy
Research Institute, 2001.
Markley, Deborah and Karen Dabson. Innovative Approaches to Entrepreneurial
Development: Cases from the Northwest Region. RUPRI Center for Rural
Entrepreneurship, 2006. 32-41.
125
Markley, Deborah, Don Macke and Vicki Luther. Energizing Entrepreneurs:
Charting a Course for Rural Communities. Lincoln, NE: Heartland Center
for Leadership Development, 2005.
Markman, Gideon, et al. “Entrepreneurship and University-based Technology
Transfer.” Journal of Business Venturing Vol. 20, No. 2, 2005: 241-263.
McKernan, Signe-Mary and Henry Chen. Small Business and Microenterprise as
an Opportunity and Asset-Building Strategy. Urban Institute Issue Brief
No. 3. June 2005.
McMullan, W. E. and W.A. Long. “Entrepreneurship Education in the Nineties.”
Journal of Business Venturing Vol. 18, No. 2, 1987: 261-275.
Microenterprise Fund for Innovation, Effectiveness, Learning and Dissemination.
Improving Microenterprise Training and Technical Assistance: Findings for
Program Managers. Washington, DC: Aspen Institute, 2002.
Microenterprise Fund for Innovation, Effectiveness, Learning and Dissemination.
Assessing the Effectiveness of Training and Technical Assistance. Field
Forum Issue 1. Washington, DC: Aspen Institute, 1999.
Minniti, Maria, with William Bygrave and Erkko Autio. Global Entrepreneurship
Monitor 2005 Executive Report. London: London Business School, 2005.
Molnar, Lawrence, et al. Impact of Incubator Investments. Athens, OH: National
Business Incubation Association, 1997.
Mountain Association for Community Economic Development. Accounting for
Impact: Economic Development Spending in Kentucky. Berea, KY:
Mountain Association for Community Economic Development, 2005.
Nakkula, Michael. Expanded Explorations into the Psychology of
Entrepreneurship: Findings form the 2001-2002 Study of NFTE in two
Boston Public High Schools. Working Paper. Harvard University Graduate
School of Education, 2003.
National Association of Development Organizations Research Foundation.
Business Not as Usual: Regional Development Organizations Promote
Rural Entrepreneurship. Washington, DC: NADO, 2002.
The National Association of Seed and Venture Funds. Seed and Venture
Capital: State Experiences and Options. NASVF White Paper. May 2006.
126
National Commission on Entrepreneurship. American Formula For Growth.
Washington, D.C.: National Commission on Entrepreneurship, October
2002.
National Community Reinvestment Coalition. Access to Capital and Credit for
Small Businesses in Appalachia. Washington, DC: National Community
Reinvestment Coalition, 2007.
National Governors Association. A Governor’s Guide to Strengthening State
Entrepreneurship Policy. Washington, DC: National Governors
Association, 2004.
Nexus Associates. A Continuing Record of Achievement: The Economic Impact
of the Ben Franklin Technology Partners, Final Report. March 2003.
North Carolina Rural Economic Development Center. Hello, My Business Name
is : A Guide to Building Entrepreneurial Networks in North Carolina.
Raleigh, NC: North Carolina Rural Economic Development Center, 2007.
Oklahoma Center for the Advancement of Science and Technology, Impact
Report 2006. Oklahoma City, Oklahoma: OCAST, January 2006.
Oldsman, Eric. Evaluation as an Effective Management Tool. Nexus Associates
Inc., 2003.
Oldsman, Eric. “Do Manufacturing Extension Programs Matter?” Research Policy
Vol. 25, No. 2, March 1996: 215-232.
Organization for Economic Cooperation and Development. OECD Framework for
the Evaluation of SME and Entrepreneurship Policies and Programmes.
Paris: Organization for Economic Cooperation and Development,
Forthcoming.
Organization for Economic Cooperation and Development. Oslo Manual:
Guidelines for Collecting and Interpreting Innovation Data Third Edition.
Paris: Organization for Economic Cooperation and Development, 2005.
Organization for Economic Cooperation and Development. Entrepreneurship and
Local Economic Development. Paris: Organization for Economic
Cooperation and Development, 2003.
Pages, Erik R., Doris Freedman, and Patrick Von Bargen. “Entrepreneurship as a
State and Local Economic Development Strategy.” The Emergence of
Entrepreneurship Policy: Governance, Start-Ups and Growth in the U.S.
Knowledge Economy. Ed. David Hart. Cambridge, MA: Cambridge
University Press, 2003.
127
Pages, Erik R. and Shari Garmise. “The Power of Entrepreneurial Networks." The
Economic Development Journal Vol. 2, No. 3, Summer 2003: 20-30.
Pages, Erik R. and Kenneth A. Poole. “Entrepreneurship Promotion as an
Economic Development Strategy: Next Steps in Institutionalizing the
Field.” Applied Research in Economic Development Vol. 3, No. 2,
December 2006: 10-27.
Phan, Phillip and Siegel, Donald. The Effectiveness of University Technology
Transfer: Lessons Learned from Quantitative and Qualitative Research in
the U.S. and the U.K. Rensselaer Working Papers in Economics. April
2006.
Pittaway, Luke, Maxine Robertson, Kamal Munir, and David Denyer. Networking
and Innovation: A Systematic Review of the Evidence. University of
Lancaster Institute for Entrepreneurship and Enterprise Development
Working Paper 016. 2004.
Reamer, Andrew, Larry Icerman, and Jan Youtie. Technology Transfer and
Commercialization: Their Role in Economic Development. Washington,
D.C.: US Economic Development Administration, August 2003.
Reese, Laura and David Fasenfast. "Critical Perspectives on Local Development
Policy Evaluations." Economic Development Quarterly Vol. 13, No. 1,
1999: 3-7.
Regional Technology Strategies, Inc. Evaluation of the Early Stages of the
Appalachian Regional Commission’s Entrepreneurship Initiative.
Washington, D.C.: Appalachian Regional Commission, December 2001.
Reynolds, Paul, D.J. Storey, and P. Westhead. “Crossnational Comparisons of
the Variation in New Firm Formation Rates.” Regional Studies Vol. 28, No.
4, 1994: 443-456.
Rice, Mark P. “Co-production of Business Assistance in Business Incubators: An
Exploratory Study.” Journal of Business Venturing Vol. 17, No. 2, 2002:
163-187.
Robinson, Dennis and Zuoming Liu. A User Guide to the Socio-Economic
Benefits Assessment System: A Rural Business-Cooperative Services
Assessment Tool for Economic Development. Community Policy Analysis
Center, University of Missouri-Columbia, December 2004.
Romanelli, Elaine and C.B. Schoonhoven, eds. The Entrepreneurship Dynamic.
Stanford, CA: Stanford University Press, 2001.
128
Rosenfeld, Stuart A. “Networks and Clusters: The Yin and Yang of Rural
Development.” Proceedings of Federal Reserve Bank of Kansas City
Conference on Exploring Policy Options for a New Rural America.
September 2001: 103-120.
Sandler, David. The Effective Use of Tax Credits in State Venture Capital
Programs. Ottawa: Canadian Tax Foundation, 2004.
Schlough, Charles and Streeter, Deborah. Cornell University’s Entrepreneurship
Education and Outreach Program: An Evaluation and Proposal.
Department of Agricultural, Resource, and Managerial Economics
Working Paper. Cornell University, 1999.
Shane, Scott. Academic Entrepreneurship: University Spin-offs and Wealth
Creation. Cheltenham: Edward Elgar, 2004.
Shapira, Philip. The Evaluation of USNet: Overview of Methods, Results and
Implications Final Report submitted to USNet Partners. August 1998.
Shapira, Philip. The Evaluation of USNet: Overview of Methods, Results and
Implications – Final Report. Georgia Tech Policy Project on Industrial
Modernization Working Paper 9805. 1998.
Simpkins, Amy. “Community Development Venture Capital: Producing Results
for Entrepreneurs, Investors, and Communities.” Bridges Vol. 10,
November 2006.
Smits, Ruud and Stefan Kuhlmann. “The Rise of Systemic Instruments in
Innovation Policy.” International Journal of Foresight and Innovation
Policy. Vol. 1, Nos. 1/ 2, 2004.
Solomon, George G., S. Duffy and A. Tarabishy. “The State of Entrepreneurship
Education in the United States: A Nationwide Survey and Analysis.”
International Journal of Entrepreneurship Education Vol. 1, No. 1, 2002:
65-86.
Storey, David J. Six Steps to Heaven: Evaluating the Impact of Public Policies to
Support Small Businesses in Developed Economies. Warwick Business
School Small and Medium Sized Enterprise Centre Working Paper No. 59.
University of Warwick, September 1998.
Street, Christopher T. and Ann-Frances Cameron. “External Relationships and
Small Business: A Review of Small Business Alliance and Network
Research.” Journal of Small Business Management Vol. 45, No. 2, 2007:
239-266.
129
Taylor, Jill S. What Makes a Region Entrepreneurial? A Review of the Literature.
Cleveland State University Center for Economic Development Monograph.
September 2006.
Tornatzky, Louis G. et al. Incubating Technology Businesses: A National
Benchmarking Study. Athens, Ohio: National Business Incubation
Association, 2003.
Tornatzky, Louis G. et al. Innovation U.: New University Roles in a Knowledge
Economy. Durham, NC: Southern Growth Policies Board, 2002.
United Kingdom Department of Trade and Industry. Innovation in the UK:
Indicators and Insights. DTI Occasional Paper No. 6. July 2006.
U.S. Small Business Administration Office of Advocacy and the Ewing Marion
Kauffman Foundation. Entrepreneurship in the 21
st
Century. Conference
Proceedings. March 26, 2004.
Vesper, Karl and William Gartner. “Measuring Progress in Entrepreneurship
Education.” Journal of Business Venturing Vol. 12, No. 5, May 1997: 403-
421.
Welch, Doug, et al. Net Benefits: An Assessment of a Set of Manufacturing
Business Networks and their Impacts on Member Companies. Report
Prepared for USNet and Regional Technology Strategies, Inc. 1997.
Westat. Evaluation of The Appalachian Regional Commission’s Community
Capacity-Building Projects. Washington, D.C.: Appalachian Regional
Commission, July 2004.
White, Jesse. “Overview Panel Comments.” Proceedings of Federal Reserve
Bank of Kansas City Conference on Beyond Agriculture: New Policies for
Rural America. April 2000.
Witt, Peter. “Entrepreneurs’ Networks and the Success of Start-Ups.”
Entrepreneurship and Regional Development Vol. 16, 2004: 391-412.
Zider, B. “How Venture Capital Works.” Harvard Business Review Vol. 76, No. 6,
1998: 131-139.
130
APPENDIX A
LITERATURE REVIEW
A review of literature was completed for works involving evaluation of
entrepreneurship initiatives in five areas – access to capital, entrepreneurial
networks, entrepreneurship education, technical assistance (including incubator
projects), and technology transfer. These five program areas correspond to the
original five areas identified by ARC as requiring investment in order to create a
more entrepreneurial region. The literature review included primarily works that
reported on more comprehensive evaluations of entrepreneurship efforts. Not all
of the publications cited in this report were considered as candidates for the
literature review. A citation and brief summary are provided for each publication
reviewed and the metrics used or suggested are highlighted. Works appear
alphabetically within each program section.
Access to Capital
Colgan, Charles D. and Bruce H. Andrews. Evaluation of Maine Technology
Institute Programs. Maine Institute for Technology and the University of
Southern Maine Center for Business and Economic Research. 2004.
Summary
This presentation evaluates the states investments in technology by looking at
the impacts for institutional grant recipients, the economic impact, effects on
company finances, intellectual property development, relationships cultivated,
and quality of assistance programs.
The evaluation finds that assistance recipients have had significant success in
developing new products leading to intellectual property protection and that they
are likely to have an economic impact on the state. Grant assistance has served
as a much needed catalyst for external financing, and relationships have been
cultivated to enhance the economic cluster networks in the state.
Metrics Suggested
Employment growth at assisted companies
Matching funds received (federal grants, external debt, external equity and
grantee match)
Number of firms indicating new product for sale
Sources of firm revenue
Distribution of company sales by geography and technology sector
Expected raw materials purchased within the state
Sources and amount of debt and equity capital
Patent activity
Other intellectual protection activity
Number of company respondents that received assistance from the
identified program (compared to other assistance programs).
131
Mean rating of organizations consulted and percent of respondents who
claimed relationship was critical to success
Client assessments of interactions with program in relation to percent
satisfaction to all state R&D assistance
Community Development Financial Institutions Data Project. Providing
Capital, Building Communities, Creating Impact. Fourth Edition.
Washington, D.C. 2004. 10 November 2006
.
Summary
Data from 517 Community Development Financial Institutions was analyzed to
demonstrate the impact of CDFIs on emerging domestic markets throughout the
U.S. Some of the results include that almost 7,000 businesses received
assistance from CDFIs resulting in 28,330 jobs. The financial assistance served
niche domestic markets that are underserved by traditional lending institutions
and the transactions were prudent and effective. CDFIs were also shown to be
flexible and timely enough to grow and change with the dynamics of the market
and respond effectively to large-scale disasters such as hurricane Katrina.
Metrics Suggested
Number of CDFIs
Total Assets
Average Assets
Total FTEs
Total Direct Financing Outstanding
Average Direct Financing Outstanding
Percent of Direct Financing Outstanding to specific sectors
Net Charge-Off Ratio
Delinquency Rate > 90 days
Delinquency Rate > 2 months
Average Capital
Percent of Debt from different types of institutions
Markets served: rural vs. urban and regional markets
Types of community betterment programs assisted, such as childcare,
affordable housing, service organizations, educational slots, payday loan
alternatives
Felsenstein, Daniel and Aliza Fleischer. “Small-Scale Entrepreneurship
and Access to Capital in Peripheral Locations: An Empirical Analysis.”
Growth and Change 33 (2002):196-215.
Summary
This paper analyses public assistance programs for small-scale entrepreneurship
programs in rural areas. The authors use data from Israel to establish that
lending institutions perceive a high risk when lending to areas where there is little
132
information. The study has three findings: 1. Location matters in determining risk
profile, 2. Location-oriented programs can improve information asymmetries that
can be a risk factor and 3. These programs can create positive welfare effects.
The study also asserts that there is speculation about the ability of information
technology to increase visibility of small rural firms and enhancing information
flow.
Metrics Suggested
An estimation framework using
Net employment
Total employment
Monthly wage for an individual income groups
Revenue
Loan value
Guarantee Value
Greene, F.J. and D.J. Storey. “An Assessment of Venture Capital Creation
Programme: The Case of Shell LiveWIRE.” Entrepreneurship and Regional
Development 16.2 (March 2004):145-159.
Summary
This paper suggests there are two areas when considering the problems inherent
to assessing venture capital creation programmes: assessment is contingent
upon the evaluation context and an input-output analysis is inadequate. The
researchers create a new instrument to assess the value of these programs.
They find that “soft” forms of support were of little value and that the more likely
an individual was to engage in entrepreneurial activity the less likely they would
be to seek the venture capital program’s services.
Points to Consider for appropriate metrics
What is the purpose of the evaluation? Evaluate the efficacy of the
program or the internal efficiencies?
Be cognizant of the operational issues involved with a program.
Manigart, Sophie et al. “Determinants of Required Return in Venture
Capital Investments: A Five Country Study.” Journal of Business
Venturing 16.6 (July 2002):291-312.
Summary
The authors use two theoretical perspectives (resource theory and financial
theory) to develop hypotheses about the determinants of the return required by
venture capitalists. They test them on over 200 companies in 5 countries. They
find that acquisition and buyout specialists require a significantly lower return
than other venture capital companies (VCCs). Also, highly stage-diversified
VCCs, independent VCCs, and VCCs providing more intensity of involvement all
expect higher return rates.
133
Metrics Suggested for Average Rate of Return for Venture Capital Companies
36%-45% for early stage investments
26%-30% for expansion investments, acquisitions, buy-outs and other
later stage categories
42% for early stage, stage specific rates
33% for later stage, stage specific
National Association of Seed and Venture Funds. Seed and Venture
Capital: State Experiences and Options. May 2006. 10 November 2006
.
Summary
The report includes a survey of the 50 states and their status in seed and venture
capital. The report concludes there are 10 lessons learned. One of these
lessons in that each state should develop a “System of Evaluation.” It claims that
the best programs establish outcome measures from the beginning, keep track of
the program results and evolve according to changing conditions.
The report mentions that state funds can have different objectives. It notes that if
funds have a more economic or social development target than on should not
expect the same rate of return as private funds.
The report establishes a starting point for more in-depth evaluation and design
prospects for state venture capital funds.
Metrics Suggested
Number of jobs created or retained
Geographic reach
Industry reach
Sandler, Daniel. “The Effective Use of Tax Credits in State Venture Capital
Programs.” Tax Paper 108, Canadian Tax Foundation. 2004.
Summary
In his paper, Sandler states how the Venture Capital Industry in the U.S. is highly
localized and for many states to encourage the geographic dispersal of venture
funds, they are using tax credits. He offers several suggestions on how to
evaluate these tax programs.
Metrics Suggested
Economic growth in the state generated by the SMEs that are funded.
The amount of capital raised through the tax credits
134
All other capital the business raised
Number of new employees
Wage rates
Capital expenditures
135
Entrepreneurial Networks
Elfring, Tom and Willem Hulsink. “Networks in Entrepreneurship: The Case
of High-technology Firms.” Small Business Economics 21.4 (2003):409-422.
Summary
The value of networks as an integral part of the explanation of entrepreneurial
success is widely acknowledged. However, the network perspective does not
specify the role of networks in the emergence and early growth of a venture. We
have distinguished three entrepreneurial processes in new venture development,
i.e. discovery of opportunities, securing resources, and obtaining legitimacy,
which are of importance for survival and performance. This paper examines how
these processes are influenced by strong and/or weak ties and whether the
degree of innovation (incremental versus radical) acts as a contingency factor in
the way network ties support entrepreneurial processes. In this explorative study
three cases on high technology firms in The Netherlands provide empirical
material enabling us to develop a number of propositions on the network effect,
in particular the mix of strong and weak ties, on the three entrepreneurial
processes.
Metrics Suggested
Network effects
Types of network ties
Motives for getting involved in networks
Kingsley, Gordon and Edward J. Malecki. “Networking for
Competitiveness.” Small Business Economics 23 (2004):71–84.
Summary
A policy innovation that has achieved widespread diffusion across national and
sub national governments in industrialized countries is the promotion of networks
among small manufacturers as a means of promoting competitiveness.
However, research and evaluations of formal networks formed in response to
policy initiatives tend not account for the informal networks that small
manufacturers routinely use in gathering information and business resources.
This study examines the use of informal networks by 50 small manufacturing
firms in rural and urban regions of northern Florida. The analysis is inductive and
designed to provide a point of comparison to the growing literature on formal
small manufacturing networks. Unlike formal networks, the links that comprise
informal networks tend to be geographically and socially mixed. Small firms use
informal networks to gather information on a mix of issues. Urban and rural firms
have similar patterns of network use on issues affecting product development
and competitiveness. But they have different patterns of network usage for
issues associated with exporting and labor problems.
136
Informal networks draw upon local and non-local information resources that do
not require significant amounts of interpersonal contact across actors. Proximity
is not a factor in the effective use of informal networks for information purposes.
Rather the emphasis is upon locating “tried-and-true” solutions that solve the
business needs. There is little evidence suggesting that the informal networks
that these small manufacturers use are gravitating towards or seeking the
development of formal networks. Thus, policy prescriptions identifying barriers to
networking among small manufacturers are borne out in this study.
Metrics Suggested
Business relationships between participating firms
Needs of firms that prompted them to seek out networks (were those
needs met?)
Information sources used (did network participation change these?)
Pages, Erik and Shari Garmise. “The Power of Entrepreneurial
Networking.” Economic Development Journal 2.3 (Summer 2003):20-30.
Summary
In 2000 the National Commission of Entrepreneurship helped convene a national
series of focus groups to find out what makes a community entrepreneurial.
They found that strong universities, access to venture capital, and good physical
infrastructure matter, but so do soft, people-based assets. The opportunity to
network with other entrepreneurs was an important factor in the success of
entrepreneurs in regions.
This article was designed to offer strategies for creating and nurturing networks.
Metrics Suggested
Learning between individuals (rather than exclusively inter-firm)
Linkages created between entrepreneurs (brokering)
Creation of common perspectives.
Cultural change
Creation of civic leaders
Branding of a region
Enhancing regional competitiveness
Networks spun off from parent network
Regional Technology Strategies, Inc. Evaluation of the Early Stages of the
Appalachian Regional Commission’s Entrepreneurship Initiative. A Report
to the Appalachian Regional Commission. December 2001.
Summary
This is the evaluation of the Appalachian Regional Commission’s $17.6 million
effort, begun in 1997, to bolster the entrepreneurial infrastructure of Appalachia.
137
ARC awarded Regional Technology Strategies, Inc. (RTS) the contract to
undertake the assessment and analysis based on a sample of 23 - 25 projects.
To collect the necessary data, the research team (1) conducted a careful review
of information and documentation supplied to the ARC by grant recipients, (2)
conducted a survey of the sample of projects supplemented by selected
telephone interviews with project staff, (3) conducted telephone interviews with
randomly selected clients and partner organizations from among names
submitted by projects, and (4) visited two project sites for a more in-depth
perspective.
The evaluation uses a variety of measures to assess outcomes and impacts on
individuals and the local economy. The first set of measures assesses relative
degrees of satisfaction of clients with the intervention. The second important set
evaluates the economic results of the intervention on the client such as new
enterprises started, new markets, or new products lines. The third captures the
more general impacts on the economy in terms of jobs created by the new
enterprises, growth potential of the enterprises, and potential sustainability of the
program.
Clients of the programs generally expressed high levels of satisfaction with the
support they received. Three-quarters of projects reported businesses developed
new products, 55 percent indicated that firms upgraded technologies or
management methods, and half reported starting new businesses.
About 52 percent of the sample projects reported creating jobs in existing firms
and 39 percent reported saving jobs that would have otherwise been lost. Adults
created 214 new firms—33 new firms with1 and 181 without employees. Based
on those projects that were able to report hard numbers in the survey, 356 new
jobs were created—54 in new firms, 121 in existing firms, and 181 jobs through
self-employment. Another 85 jobs were saved from extinction in existing firms. In
addition, surveyed projects reported 46 new businesses created by youth or
students as class projects, some of which could become self-sustaining
businesses after graduation.
Metrics Suggested
Client levels of satisfaction
New product development in existing businesses
Technology or management method improvement in existing businesses
New businesses created
Jobs created
Jobs retained that would have been lost without the project
Businesses started by youths
Income of businesses created
138
Rosenfeld, Stuart. “Networks and Clusters: The Yin and Yang of Rural
Development.” Proceedings of Federal Reserve Bank of Kansas City
Conference on Exploring Policy Options for a New Rural America.
(September 2001):103-120.
Summary
Beginning in the mid-1980s, policymakers, particularly in rural areas, realized
that they could no longer rely only on attracting large branch plants to sustain
their economies. Increasing competition from newly developed and less
developed low-wage nations was erasing their cost advantages. The more
creative development agencies began to rechannel efforts towards stimulating
entrepreneurial activity and strengthening indigenous businesses. Among the
many policies discovered and promulgated by various experts and advocates in
the late 1980s was interfirm collaboration, i.e., the widespread formation of
formal and informal alliances, or networks, among groups of companies for
mutual competitive advantage.
The policy levers to convert networks from a practice considered by many to be
uniquely linked to northern Italy’s social and business culture into a more
universal practice were first formulated in Denmark in 1989. This approach to
interfirm cooperation, designed in Denmark with advice from American consultant
Richard Hatch, became the U.S. and international standard. With an allocation of
$25 million from the Ministry of Trade and Industry, the scheme consisted mainly
of training people, called brokers, to create networks and then offering groups of
three or more companies sequentially phased grants for conceptualization,
planning, and implementation. Eligible network activities included joint marketing,
production, problem solving, research and development, and purchasing.
The first and only, national effort to move the numbers of networks in the U.S. to
a scale that could have impact began in 1993 under a grant from the National
Institute for Standards and Technology to Regional Technology Strategies, Inc.
The project, called USNet, was based on the documented value of networks to
industrial modernization and technology adoption in Europe and involved 15
state partners working together to build statewide network programs for SMEs. It
relied heavily on the Danish model of training brokers and multipliers and on a
process for informing companies and development organizations. But it
depended on the individual partner states to provide the financial incentives.
While not a rural development initiative per se, many of the early adopting areas
were rural.
Metrics Suggested
Improved quality of products
New customers
New suppliers outside network
Increased sales
Increased profitability
139
Improved existing process
Improved relationships with customers
Adopted new technologies
Improved supplier quality
Savings by group purchasing or shared resources
Developed new product
Increased exports
Established new company
**Also suggests that learning is a sufficient outcome for businesses to network.
Shapira, Philip. The Evaluation of USNet: Overview of Methods, Results
and Implications. Final Report to USNet Partners. August 1998.
Summary
This report summarizes the aims, methods, and principal findings from the
evaluation of USNet. The USNet project was a pilot initiative to build the capacity
of its partners to promote inter-firm collaboration, with the ultimate aim of
enhancing the competitiveness of small and mid-sized manufacturing
enterprises.
The evaluation discusses the findings of the major USNet evaluation studies
undertaken by members of the evaluation team. The principal findings of the
evaluation are:
Firms who collaborate in inter-firm networks report positive net benefits,
while greatest private impacts are associated with strong industry
leadership of networks
USNet’s original network promotion goals were too ambitious, given the
resources available; judged against more realistic expectations, USNet
has performed well
USNet’s training programs have generated widespread awareness about
inter-firm collaboration
USNet special projects demonstrate the value of explicit follow-on
initiatives to promote inter-firm collaboration
USNet policy and organizational impacts at the state level were modest
Federal support can strengthen efforts to promote inter-firm collaboration
at the state level and aid shared learning
Metrics Suggested
Impacts of networks on states-number of firms involved in networks,
attitudes of organizations toward inter-firm collaboration, learning new
information, making new contacts, learned about other states’ practices,
getting ideas to resolve problems, becoming aware of tools.
Net economic benefits accruing to participating firms
Generating awareness of inter-firm collaboration
140
Welch, Doug, et al. “Net benefits: An Assessment of a Set of Manufacturing
Business Networks and their Impacts on Member Companies.” USNet
Evaluation Working Paper 9701. 1997.
Summary
During the 1990s, a growing number of US companies became involved in
collaborative interfirm partnerships, flexible business networks and other
organized collaborative efforts to aid business performance. Drawing on survey
data from industrial companies and network brokers in five US states, this study
identifies and measures a range of hard and soft impacts on firms resulting from
their participation in interfirm networks. The measures include effects on firms’
activities, business strategy, relationships, trust, confidence, technology use,
know-how, employment and economic benefits and costs.
USNet, a federal-state initiative to strengthen interfirm collaboration, sponsored
this study. In order to assess the impacts of network participation on member
companies, two types of surveys were administered at the beginning of 1997, the
“1997 National Benchmark Survey of Industrial Network Companies” and the
“Survey of Network Presidents/Coordinators.” This study presents findings from
descriptive and comparative analyses of data from these surveys.
A total of ninety-nine members of 13 separate business networks responded to
the survey. Principal findings from this dual survey effort include the following:
Most of the thirteen surveyed networks are young, urban organizations
with limited staff resources.
With a few exceptions, the network members responding to the survey
were small manufacturers.
The networks’ most common primary objective is information sharing.
Companies of different sizes report different network activities.
Overall, most respondents are satisfied with the networks. Thirty-one
Most companies report positive effects to date and expect even larger
future effects.
The average total net benefits per firm of network participation are
positive.
On average, companies experienced a net increase in their employment
levels as a result of network participation.
A few more ‘intensive’ network activities are associated with stronger
overall impacts.
Companies that share sales leads report higher net benefits of network
participation.
Companies that have been in networks longer are more likely to report
sharing technical capabilities with other network members.
Metrics Suggested
Network objectives
Differential network activities depending on client characteristics
141
Satisfaction levels
Client experiences of positive effects on strength of positive effects
Total net benefit per firm ($)
Change in employment levels of participating companies
Intensity of network activities
Improvement in employee and management skills due to network
participation
Knowledge sharing with other companies
142
Entrepreneurship Education
Charney, Alberta and Gary Libecap. The Impact of Entrepreneurship
Education: An Evaluation of the Berger Entrepreneurship Program at the
University of Arizona, 1985-1999. Final Report to Kauffman Center for
Entrepreneurial Leadership. May 2000.
Summary
The report details the results of a series of surveys on the effect that the Berger
Entrepreneurship Program at the University of Arizona has had on its graduates,
the university’s technology transfer program, financial contributions to the college
of business, and the degree to which entrepreneurship has been incorporated
into curriculums in other departments. The Berger Program was a good
candidate for such an analysis because it had been in existence for sixteen years
at the time of the report. A sample of program alumni and other university
graduates were surveyed to assess the program’s effect on students. According
to the results, program graduates were three times more likely to start a new
business than non-entrepreneurship graduates. Additionally, program graduates’
average annual income was found to be twenty-seven percent higher than the
average annual income of general university graduates. However, there was
little evidence that program participation led to higher levels of job satisfaction.
With regard to technology transfer, the report concludes that entrepreneurship
education promotes technology transfer as graduates were more likely to be
involved with firms that use licensed technologies or with firms that license
technologies to others. A separate survey of University of Arizona administrators
revealed that entrepreneurship education had reportedly led to increased
financial contributions to the university and respondents felt that other
curriculums had been enriched due to pedagogical innovations in the
entrepreneurship program.
Metrics Suggested
Number of business ventures started
Number of workers employed in venture
Amount of sales/revenue
Graduate job satisfaction
Type of venture started
Whether or not venture is “high-tech”
Fayolle, Alain. “Evaluation of Entrepreneurship Education: Behavior
Performing or Intention Increasing?” International Journal of
Entrepreneurship and Small Business 2.1 (2005):89-98.
Summary
Fayolle says there is a growing interest around the question of how to evaluate
entrepreneurship education programs. He then suggests that the standard
metrics such as direct and indirect job creation are insufficient measuring sticks.
143
Fayolle points out that there are significant timing issues associated with using
job creation as the primary criteria, i.e. he says the entrepreneurial process is a
nonlinear one and there is no definitive way to know at what point employment
opportunities may be created. Accordingly, he advocates for the inclusion of
some more intangible metrics. Fayolle believes that entrepreneurship programs
should also be evaluated on the attitudes, mindsets, and intentions of students.
He also discusses how different teaching strategies can influence student
behavior. Fayolle notes that entrepreneurship training may enable students to
start their own enterprise, but it can at the same time dent their desire to do so –
consequently, teaching style matters. In essence, Fayolle says that because
entrepreneurship is so connected to personal desire and motivation that it is very
important to evaluate the student psyche throughout the educational process.
Bottom line being that evaluating programs on job creation alone is difficult and
incomplete; therefore, that analysis should be supplemented with some
assessment of how the program nurtures the entrepreneurial spirit.
Metrics Suggested
Teaching strategies
Student attitudes and intentions throughout the process
An evaluation of overall program goal compared to student goals
Knowledge acquisition
Nakkula, Michael. “Expanded Explorations into the Psychology of
Entrepreneurship: Findings from the 2001-2002 Study of the NFTE in two
Boston Public High Schools.” Working Paper, Harvard University Graduate
School of Education. 2003.
Summary
This reports details the results of a National Foundation for Teaching
Entrepreneurship (NFTE) program that was taught in two Boston public schools
during the 2001-2002 school year. In the Boston public school system every
student is required to take a school to career “pathway” class that forces them to
think about their plans after graduation. In concert with that, a study was set up
where some students were enrolled in a special “pathway” class sponsored by
NFTE. Students were assessed at the beginning of their “pathway” classes and
then again upon completion regarding questions like desire to go to college.
Students taking NFTE’s entrepreneurship “pathway” course were found to have
experienced considerably higher increases in college interest and occupational
aspirations over the duration of the class than students enrolled in non-NFTE
“pathway” courses.
Metrics Suggested
Level of college interest, pre and posttest
Occupational interest, pre and posttest
Overall school engagement
Amount of independent reading
144
Schlough, Charles and Deborah Streeter. “Cornell University’s
Entrepreneurship Education and Outreach Program: An Evaluation and
Proposal.” Working Paper, Department of Agricultural, Resource, and
Managerial Economics, Cornell University. 1999.
Summary
The first part of the report evaluates Cornell’s Entrepreneurship Education and
Outreach (EEO) program. The authors discuss the challenges that the program
has encountered during its first two years including a lack of widespread
cooperative support from state and local-level stakeholders. Consequently, the
second part of the report is a proposal to develop a statewide network of
supportive partnerships in order to improve Cornell’s entrepreneurship education
mission. Included in that plan is a set of “criteria for success” and a
corresponding list of metrics. The authors suggest evaluating the proposed
program by measuring the number of small businesses started by graduates, the
number of jobs created within those ventures, the number of decisions by
graduates to not proceed with a business concept, and annual sustained
enrollment levels in program courses.
Metrics Suggested
Number of businesses started
Number of people employed within those businesses
Number of decisions by graduates to not pursue a business concept
Annual sustained enrollment in program courses
Soloman, G.T., et al. “The State of Entrepreneurship Education in the
United States: A Nationwide Survey and Analysis.” International Journal of
Entrepreneurship Education 1.1 (2002):65-86.
Summary
This paper offers an assessment of entrepreneurship education both
domestically and abroad as of 2000. Included in the article is a brief literature
review on entrepreneurship education. The following is a summary of two
articles discussed in that literature review for which full-text versions were not
readily available. Citations for both works are listed below.
Block and Stumpf (1992) and McMullan and Long (1987) both assert that
traditional measures of program effectiveness such as number of graduates are
insufficient indicators of success. Instead they agree that programs should be
evaluated according to their socioeconomic impacts. Accordingly, Block and
Stumpf (1992) suggest using number of jobs created by graduates and overall
job satisfaction as relevant measures. Similarly, McMullan and Long (1987)
advocate assessing the number, type, and growth of companies created by
entrepreneurship graduates as a better gauge of success.
145
Metrics Suggested
Number of jobs created
Level of job satisfaction
Number of businesses created
Types of companies created
Growth of companies created
Sources
Block, Z. and S. A. Stumpf. “Entrepreneurship Education Research: Experience
and Challenge.” The State of the Art of Entrepreneurship. Eds. D. L. Sexton and
J. D. Kasarda. Boston: PWS-Kent Publishing S.A., 1999. 17-45.
McMullan, W. E. and W.A. Long. “Entrepreneurship Education in the Nineties.”
Journal of Business Venturing 2 (1987):261-275.
Vesper, Karl and William Gartner. “Measuring Progress in
Entrepreneurship Education.” Journal of Business Venturing 12
(September 1997): 403-421.
Summary
In this article, Vesper and Gartner report on the results of a 1994 survey which
attempted to rank university-level entrepreneurship education programs. The
survey was mailed to 941 business school deans both domestically and abroad.
Respondents were asked to rank the top programs and to list the most important
criteria considered when doing so. Among the possible 18 criterion listed,
courses offered was ranked number one followed by faculty publications, impact
on community, exploits of alumni, and innovations. Faculty start-ups and location
were listed as the bottom two. However, Vesper and Gartner quickly caution the
reader about the results of the survey and similar rankings in general. The
authors state that the survey did not tie the program rank to specific criteria nor
did it evaluate respondent knowledge of the other programs in question. With
respect to other popular rankings, for example, program ratings published by
Business Week, Vesper and Gartner argue that traditional metrics used in those
rankings – GMAT scores, computers per capita, etc. – may not be appropriate
predictors of entrepreneurial success. Instead, they suggest employing the set of
criteria used in awarding the Malcolm Baldrige National Quality Award (MBNQA),
a well-established quality improvement program. Using the MBNQA framework,
Vesper and Gartner stress the importance of evaluating entrepreneurship
programs based on the following seven factors: leadership, information and
analysis, strategic and operational planning, human resource development and
management, educational and business process management, school
performance results, and student focus and student and stakeholder satisfaction.
Metrics Suggested
Student performance (in specific classes or as demonstrated through a
portfolio)
146
Student improvement throughout program
Student satisfaction
Impact on the community (number of start-ups, students employed in new
firms, students working in positions assisting new firms)
147
Technical Assistance
Aernoudt, Rudy. “Incubators: Tool for Entrepreneurship?” Small Business
Economics 23.2 (2004):127-135.
Summary
This paper examines U.S. and European experiences with business incubators
and stresses the need for accurate evaluations of their impact relative to their
different types. The author finds that lack of entrepreneurship and the
underdevelopment of seed financing and business angel networks are some of
the biggest barriers to success. He asserts that seed financing, links with
business angels and business angel networks as well as involvement in second
round financing and IPO assistance should be integrated into the business
incubation concept.
Metrics Suggested
Survival rate
Tenants by incubator
Employment by tenants
Employment created by graduates
Graduates remaining in the community
Brown, J. David, John S. Earle and Dana Lup. “What Makes Small Firms
Grow? Finance, Human Capital, Technical Assistance, and the Business
Environment in Romania.” Economic Development and Cultural Change 54
(October 2005):33-70.
Summary
This paper sought to explore new ground by looking at the policy-relevant factors
that may stimulate or hinder small start-up companies, and fill the need for
quantitative studies using panel data to analyze statistical relationships between
firm growth and objective measures of factors related to policies. They found
that availability of loans is a factor while internal finance and trade credit tend to
be unimportant. They also found that high school education raises growth but
university education and worker training are not necessarily a factor. Technical
assistance was also considered a weak factor for small business growth
success.
Metrics Suggested
Retained earnings,
conventional bank lending,
informal credit markets,
tax credits offered by the state
Membership in business association
Membership in consultancy programs
148
Rate of usefulness of assistance programs
Community Development Financial Institutions Data Project. Providing
Capital, Building Communities, Creating Impact. Fourth Edition.
Washington, D.C. 2004. 10 November 2006
.
Summary
Data from 517 Community Development Financial Institutions was analyzed to
demonstrate the impact of CDFIs on emerging domestic markets throughout the
U.S. Some of the results include that almost 7,000 businesses received
assistance from CDFIs resulting in 28,330 jobs. The financial assistance served
niche domestic markets that are underserved by traditional lending institutions
and the transactions were prudent and effective. CDFIs were also shown to be
flexible and timely enough to grow and change with the dynamics of the market
and respond effectively to large-scale disasters such as hurricane Katrina.
Metrics Suggested
Number of people receiving group-based training
Number of people receiving one on one technical assistance
Number of organizations receiving training
Number of jobs created overall from the project
Number of business receiving training
Enterprise Corporation for the Delta. Enterprise Corporation for the Delta
Program Monitoring Report—Business Technical Assistance. 2003. 10
November 2006
.
Summary
This report provides baseline information about the Enterprise Corporation for the
Delta’s technical Assistance activities associated with their Community
Development Financial Institution. This baseline is established to measure the
progress of their efforts to address the non-financial needs of potential and
existing customers and lay the groundwork for more in-depth analysis in the
future.
The ECD has three technical assistance activities: FastTrac entrepreneurial
training, brokered TA and Business LINC mentor/protégé program. ECD learned
that very few graduates from FastTrac attained loan financing, but many
graduates demonstrated more sound decision making. The Brokered TA mostly
focused on establishing the foundation for future analysis. The mentoring
program of the ECD found that creating a mentoring environment for small start-
ups works, but the conditions for its success is very different than many urban
programs mostly because there are not large established companies to become
a cadre of mentors. They also found that protégés were more successful if they
149
had several years of experience and if they were in close proximity to their
mentor.
Metrics Suggested
Number of mentor relationships established
Finance seeking intentions of graduates, before and after course
completion
Number of technical assistance engagements
Types of TA services requested
Loan ratings of TA graduates and loan ratings of control group
Satisfaction level with mentor programs or other TA initiatives.
Greenburg, Elizabeth and Richard Reeder. “Who Benefits from Business
Assistance Programs? Results of the ERS Rural Manufacturing Survey.”
Agriculture Information Bulletin Number 736-04. United States Department
of Agriculture. 1998.
Summary
The authors sought to find out how much government programs helped rural
manufacturers and who exactly is benefiting from these programs. They
discovered that over 60 percent of manufacturing establishments benefited from
the programs and 28 percent of these firms found these programs to be very
important to their operations over the last three years. They discovered that
manufacturers using advanced technologies benefited more than other
manufacturers. Also, large businesses were more likely to benefit than small
ones, although small firms seemed to have more problems and benefit the most
once assistance was administered.
Business assistance programs were identified as: tax incentives, loans (direct,
indirect/guaranteed, and revolving), industrial parks and enterprise zones, and
training and technical assistance.
Manufacturing establishments were characterized by the following: metro/non
metro, geographic region in the U.S., employment size, type (branch plant and
high tech), and distressed (high poverty rates, high unemployment rates and
population loss).
The study further analyzed which types of firms benefited from the specific types
of assistance programs. State and local tax breaks benefited the largest
proportion of nonmetro establishments (46 percent), training and technical
assistance (29 percent), industrial parks/enterprise zones (21 percent), direct
loans (15 percent), guaranteed loans (13 percent), revolving loan funds (9
percent).
Metrics Suggested
150
Survey of how important each assistance program was to their business.
This metric can be cross-analyzed by
o Geographic location
o Size
o Type of business
o Type of distressed area
o Rural/Urban
Lambrecht, Johan and Fabrice Pinray. “An Evaluation of Public Support
Measures for Private External Consultancies to SMEs in the Walloon
Region of Belgium.” Entrepreneurship and Regional Development. 17.2
(March 2005):89-108.
Summary
This paper evaluated public support measures for private consultancies to SMEs
in the Walloon region of Belgium. It presents an analysis of the supply and
demand, an evaluation of the efficiency and the effectiveness of policy measures,
and real policy recommendations. The paper recommends that the real needs of
the entrepreneur and of the SME determine the publicly financed advisory
process. It also recommends a “one stop shop” for private external consultants
that help SMEs.
Metrics Suggested
Profit
Sales
Market Share
Employment
Microenterprise Fund for Innovation, Effectiveness, Learning and
Dissemination (FIELD). Improving Microenterprise Training and Technical
Assistance: Findings for Program Managers. 2002. 10 November 2006
.
One of the first tasks FIELD set for itself and for the organizations who received
awards from their RFPs was to answer “What makes for effective training and
technical assistance?” In support of this question they also asked “What are the
appropriate indicators, which intermediate measures are better indicators of
financial impact on clients, and what practical approaches can programs used to
document and track outcomes?
They found that many entrepreneurs sustained and grew their businesses, while
some other businesses floundered. This was mostly due to family or personal
reasons. The analysis found that effective training programs acknowledge the
importance of client readiness for business and offer a range of services to help
clients meet these needs. It also found that adult learning theory is an effective
method for designing and delivering training. Training must include soft skills
151
and basic competencies as well as key financial and marketing skills. Finally,
FIELD learned TA should be offered within a structure that keeps clients
connected to a larger program, but places the initiative on the client to receive
services.
Metrics Suggested
Completion of Training
Development of Business Plan
Progress in Seeking Business Financing
Business Starts
Business Expansions
Business Stabilizations
Is there an improvement in income reporting
Increase in business assets
Increase in personal assets, such as cars, savings, homeownership.
Microenterprise Fund for Innovation, effectiveness, Learning and
Dissemination. “Assessing the Effectiveness of Training and Technical
Assistance.” FIELD Forum Issue 1. 1999. 10 November 2006
.
Summary
FIELD recognizes that training and technical assistance to small businesses has
a significant impact on their success, but there is little data to substantiate this to
policymakers and funders. Therefore they sought to identify models and other
metrics to show what makes for effective training and how that equates to lower
costs. FIELD asked practioners to propose strategies and identify indicators that
could establish a link between service and outcomes. Some intermediate and
final outcomes were suggested.
Suggested Metrics
Course completion and graduation
Pre and post knowledge testing
Business plan completion
Satisfaction
Hours of Training and TA
Sequence of Training and TA
Business Skills acquired
Personal effectiveness skills
Established networks/relationships
Business start-ups/survivals
Sales and profit/loss
Number of employees
Employee wages and benefits
Household income assets and net worth
Change in public assistance
152
Oldsman, Eric. “Evaluation as an Effective Management Tool.” Nexus
Associates, Inc. 2003. 10 November 2006 .
Summary
This paper critiques the Performance Measurement Framework established by
the Committee of Donor Agencies for Small Enterprise Development to learn
more about the performance of business development services. The paper
suggests that the PMF framework can be useful as a tool for managers to
improve their performance but considerable caution should be taken when using
common performance standards for such diverse programs. The author
recommends that evaluations be grounded in explicit theories of the particular
initiatives.
Things to consider when developing metrics
Characterize conditions within markets to learn more about their structure
and performance
Determine specific needs within a target population of firms in order to
design new programs
Establish whether existing programs are being implemented as intended
Find out whether existing programs are achieving their objectives
Compare existing programs to judge the relative merits of different
approaches to addressing specific needs.
Examine operations in great detail (aggregate data can mask a lot).
Oldsman, Eric. “Do Manufacturing Extension Programs Matter?”
Research Policy 25.2 (March 1996):215-232.
Summary
Based on his evaluation of the New York based Industrial Technology Extension
Service Program, Oldsman finds that manufacturing extension programs can
have a favorable impact on participating companies. Because of the expertise of
field agents, firms have been able to reduce costs, particularly with respect to
direct and indirect labor, and in some instances increase revenue.
He found that MEP programs should be designed to focus on adding value rather
than cutting costs; pay attention to direct, long-term assistance; and foster
cooperation to compensate for the lack of internal economies of scale.
Metrics Suggested
Refining layout of operations
Purchased or developed new software
Ask entrepreneurs if they had not received assistance would they have
stayed in operation and in the state?
Cost savings
153
Direct labor productivity
Reduction in inventory
Reductions in manufacturing lead time
Reductions in direct labor costs per unit
Reductions in material costs per unit
Reductions in energy costs per unit
Reductions in indirect labor costs per unit
Reductions in other overhead costs
154
Technology Transfer
Association of University Technology Managers. AUTM U.S. Licensing
Survey: FY 2004 Survey Summary. 2004. 10 November 2006
.
Summary
This year’s Licensing Survey shows a continued steady growth in the 6 percent
range for most of the performance measures that are considered meaningful
indicators within the profession:
Products available to the public
Invention disclosures received
Licenses and options executed
Licenses and options active
Licenses and options generating income
Licenses and options generating running royalties
Net income
One or two important performance measures, specifically U.S. patents issued,
though down from fiscal year 2003, appeared to be consistent with long-term
growth trends. However, the most dramatic results were the clear evidence of a
recovery from the very difficult market conditions for new company startups
reported in the fiscal years 2002 and 2003 Licensing Surveys. Institutions
launched 23.5 percent more new startups in fiscal year 2004 than in fiscal year
2003, and the number of existing startup companies that went out of business
declined more than 30 percent.
The new startup company activity reflects the changed circumstances in capital
markets. The second half of 2003 will be remembered as the end of the venture
industry’s hemorrhaging that followed the nearly simultaneous collapse of the e-
commerce, telecommunications and biotechnology markets.
The second half of 2003 also saw the first revitalization of the initial public
offering market since 2000. The number of venture-backed IPOs began
increasing in the second half of 2003, with 20 of the 22 venture backed IPOs for
2003 occurring in the third and fourth quarters of the year. The first two quarters
of 2004 saw 34 venture-backed IPOs, and the year ended with 67 venture-
backed IPOs raising $4.98 billion vs. the $1.4 billion raised in 2003.
Metrics Suggested
Products available to the public
Invention disclosures received
Licenses and options executed
Licenses and options active
Licenses and options generating income
Licenses and options generating running royalties
155
Net income
New Startups
Number of startups going out of business
University equity interests in their startups
IPOs
Audretsch, David B., Taylor Aldridge, and Alexander Oettl. The Knowledge
Filter and Economic Growth: The Role of Scientist Entrepreneurship.
Ewing Marion Kauffman Foundation. 2006. 10 November 2006
.
Summary
This study examines the prevalence and determinants of the commercialization
of research by the top twenty percent of university scientists funded by grants
from the National Cancer Institute (NCI). Because the two publicly available
modes of scientist commercialization – patents and Small Business Innovation
Research (SBIR) grants – do not cover the full spectrum of commercializing
activities undertaken by university scientists, the study also includes two
additional measures obtained from detailed scientist interviews: licensing of
intellectual property and starting a new firm. These measures are used to assess
both the prevalence and determinants of scientist commercialization of research.
In particular, two distinct routes for commercializing scientist research are
identified, the Technology Transfer Office (TTO) route and the entrepreneurial
route, which does not involve assigning a patent to the university. This study in
no way provides an assessment or judgment about the efficacy of the TTO.
Rather, this study highlights the extent to which additional commercialization of
research takes place, suggesting that the contribution of universities to U.S.
innovation and ultimately economic growth may be greater than had previously
been believed. Relevant findings include:
Two paths for commercialization of scientist research are identified - the
TTO route and the entrepreneurial route. Scientists who select the TTO
route by commercializing their research through assigning all patents to
their university TTO account for 70 percent of NCI patenting scientists.
Scientists who choose the entrepreneurial route to commercialize their
research, in that they do not assign patents to their university TTO,
comprise 30 percent of patenting NCI scientists.
Social capital enhances the propensity for scientists to commercialize their
research. The impact of social capital is particularly high for the
commercialization mode of scientist entrepreneurship.
Scientists choosing the entrepreneurial route to commercialize their
research, by not assigning patents to their university to commercialize
research, tend to rely on the commercialization mode of entrepreneurship.
By contrast, scientists who select the TTO route by assigning their patents
to the university tend to rely on the commercialization mode of licensing.
Metrics suggested
156
Research grants to university researchers (i.e., SBIR)
Patents
Licenses of intellectual property
University researchers starting new firms
Route by which commercialization happens- entrepreneurial or through
tech transfer office
Chukumba, Celestine and Richard Jensen. “University Invention,
Entrepreneurship, and Start-Ups.” National Bureau of Economic Research
Working Paper No. 11475. July 2005.
Summary
This study examines the commercialization of university inventions in licensing to
both start-up firms and established firms, and seek to determine when licensing
to start-ups is more likely. They construct a theoretical model that predicts start-
ups are more likely if their opportunity cost of development and
commercialization is lower or if the technology transfer officer’s (TTO) opportunity
cost of searching for a partner among established firms is higher. Using data
from the Association of University Technology Managers, the National Venture
Capital Association Yearbook, and the National Research Council, the study
finds that inventor quality and measures of past TTO success (age, the number
of disclosures, gross royalties) are all positively and significantly related to the
number of licenses to both start-ups and established firms. However, it also finds
that start-up activity is positively and significantly related to the S&P 500, but
negatively and significantly related to the interest rate and rate of return to
venture capital.
Metrics suggested
Licensing to start-ups and existing firms
Start-ups created
Partnering with outside entities by tech transfer offices
Venture capital secured by start-ups
Markman, Gideon, et al. “Entrepreneurship and University-based
Technology Transfer.” Journal of Business Venturing 20.2 (2005):241-263.
Summary
The success of business incubators and technology parks in university settings is
often determined by how well technology is transferred from the labs to their
startup firms. University technology transfer offices (UTTOs) function as
‘‘technology intermediaries’’ in fulfilling this role. This article builds a framework to
address two questions: (a) Which UTTOs’ structures and licensing strategies are
most conducive to new venture formation; and (b) how are the various UTTOs’
structures and licensing strategies correlated with each other. The findings reveal
a complex set of relationships between UTTO structure and strategies, new
venture formation, and business incubation.
157
Based on interviews with 128 UTTO directors, findings show that whereas for-
profit UTTO structures are positively related to new venture formation, traditional
university and nonprofit UTTO structures are more likely to correlate with the
presence of university-based business incubators. Licensing-for equity strategy is
positively related to new venture formation while sponsored research licensing
strategy is negatively related. The licensing-for-cash strategy, the most prevalent
transfer strategy, is least correlated to new venture formation. A content analysis
of UTTO mission statements also revealed an overemphasis on royalty income
and an under emphasis on entrepreneurship.
Metrics Suggested
Structure of tech transfer office (for- or non-profit)
Incubators
Technology parks
Startup firms
Business incubation
Licensing
University equity shares in startups
Applied v. basic research at university where tech transfer office is located
Oklahoma Center for the Advancement of Science and Technology. Impact
Report 2006. Oklahoma City, Oklahoma. January 2006. 10 November 2006
.
Summary
The report summarized OCAST’s program impacts in terms of award amounts
and leveraged private and federal funds. It also gives some detail about each
program, including the Oklahoma Technology Commercialization Center, whose
progress is measured in terms of facilitation of capital acquisition, jobs created,
and companies served.
Metrics suggested
Amount of capital acquisition facilitated
Jobs created
Number of technology companies served
Palmintera, Diane. Accelerating Economic Development Through
University Technology Transfer. Reston, VA: Innovation Associates Inc.,
February 2005.
Summary
This report highlights models of university tech transfer and commercialization,
related efforts like entrepreneurship programs, and the infrastructure and
environment needed to support commercialization efforts. It includes case
158
studies of university-based tech transfer and related economic development
initiatives that lay the groundwork for state, university, and corporate actions to
leverage university resources.
Practices in tech transfer at 10 universities were examined, along with related
entrepreneurship programs and other programs. These case studies were
analyzed to extract best practices and recommendations.
Suggested Metrics
Corporate sponsored research
Levels of government funding
Seed capital and source (i.e., university-created funds or private funds)
Innovation centers
Number of start-ups created
Number of start-ups assisted
Incubators
Research Parks
Employment
Phan, Phillip and Donald Siegel. “The Effectiveness of University
Technology Transfer: Lessons Learned from Quantitative and Qualitative
Research in the U.S. and the U.K.” Rensselaer Working Papers in
Economics Number 0609. Rensselaer Polytechnic Institute. April 2006.
Summary
In recent years, there have been numerous studies of the effectiveness of
university technology transfer. Such technology transfer mechanisms include
licensing agreements between the university and private firms, science parks,
incubators, and university-based startups. This study reviews and synthesizes
these papers and presents some recommendations on how to enhance
effectiveness. Implementation of these recommendations will depend on the
mechanisms that universities choose to stress, based on their technology
transfer “strategy.” For example, institutions that emphasize the entrepreneurial
dimension of technology transfer must address skill deficiencies in technology
transfer offices, reward systems that are inconsistent with enhanced
entrepreneurial activity and the lack of training for faculty members, post-docs,
and graduate students in starting new ventures or interacting with entrepreneurs.
Metrics Suggested
Licensing
Science parks
Incubators
University-based startups
Training provided to employees in working with entrepreneurs/start ups
159
APPENDIX B
NON-PROJECT* STAKEHOLDER INTERVIEWS
REGIONAL
STAKEHOLDERs
REGIONAL STAKEHOLDER ORGANIZATIONs
Dinah Adkins President, National Business Incubation Association (OH)
Cathy Ashmore Executive Director, Consortium for Entrepreneurship Education
Bill Campbell Director, Alabama Small Business Development Centers
Caroline Carpenter Program Director, W.K. Kellogg Foundation
Dale Carroll President, Advantage West (NC)
Pam Curry Executive Director, Center for Economic Options
Eleanor Herndon Executive Director, North Carolina REAL
June Holley Consultant, Network Weaving (OH)
Mary Hunt-Lieving Program Officer, Benedum Foundation
Lisa Ison President, New Century Venture Center (VA)
Kris Kimmel President, Kentucky Science and Technology Corporation
Bill Loope New River Community and Technical College (WV)
Justin Maxson President, MACED
Ray Moncrief Vice President, Kentucky Highlands Investment Corporation
Welthy Soni Myers Managing Director for Special Initiatives, Association for Enterprise
Opportunity (VT)
Becky Naugle Director, Kentucky Small Business Development Centers
Kim Pate Vice President, CFED
Stuart Rosenfeld Principal, Regional Technology Strategies (NC)
Greg Rutherford President, York Technical College (SC)
Jeff Spencer Executive Director, Ohio Valley Regional Development Commission
Kerwin Tesdell President, Community Development Venture Capital Alliance (NY)
Jesse White Director, Office of Economic and Business Development, University of
North Carolina
PROGRAM
LEADER
STATE
Denise Ambrose Program Manager, Virginia Department of Housing and Community
Development
Bonnie Ammons Senior Program Manager, Office of Community Grant Programs (SC)
Todd Christiansen Associate Director, Virginia Department of Housing and Community
Development
Olivia Collier ARC Program Manager, North Carolina
Bonnie Durham ARC Program Manager, Alabama
Al Feldstein ARC Program Manager, Maryland
Neil Fowler ARC Program Manager, Pennsylvania
Ralph Goolsby ARC Program Manager, West Virginia
Elisabeth Kovacs ARC Program Manager, South Carolina
Rick Meredith Assistant Commissioner, Tennessee Department of Economic and
Community Development
Peggy Satterly ARC Program Manager, Kentucky
Sara Stuckey Retired ARC Program Manager, North Carolina
James Thompson ARC Program Manager, Georgia
Kyle Wilbur ARC Program Manager, New York
*Some stakeholders were from organizations that received ARC funding for entrepreneurship
projects but were included because they (1) had broad and unique knowledge of some program
area and/or entrepreneurship, (2) the organization’s project was not included in the sample,
and/or (3) the individual was not interviewed as follow up with grantees included in the sample.
160
Appendix C
PROTOCOL FOR PROJECT LEADERS
Enter the following data from the project folder:
Project Number: __________________ Project Status (circle one): I
C1 C2
Project Title: _______________________ Project Type (circle one): C N I
E TA
Grantee: _________________________
Year Project Initiated:_____________ Number of Years of ARC Funding:
_____
Name of Person Interviewed: _______________________Phone/Email:
___________
Organization of Person Interviewed:______________________
ARC Funds Invested:_____________________ Leveraged
Funds:_______________
Project Summary:
Stated goals of the project:
Number of Businesses Served:
Number of jobs created
Jobs retained:
Amount of leveraged private investment:
For all calls:
1. Were you involved with this project and/or are you knowledgeable about
it? (If no, get referral.) If yes, in what capacity did you work with the
project?
2. What is your background? (Try to understand the importance of their
leadership.) Alternative: Tell me a little about your background and
experience with projects like this in the past. How did you get involved in
this project? How long have you been involved with the relevant
community? If not long have you had similar experiences in your
previous communities?
161
3. What was the problem that you were trying to address with this project?
How were you trying to address this problem?
4. Did you think the project was a success? Why or why not? What were the
elements of success/failure?
5. What specific results or outcomes were achieved through this project
during the period of ARC investment?
a. Are there specific results that you believe were particularly
important? If so, why?
b. Are there specific results that were unintended or unexpected?
c. Do you feel you achieved the objectives set forth for this project?
6. Has the project continued after ARC funding ended?
a. If yes, how was the project funded after ARC?
b. What outcomes have you experienced post-ARC?
7. What value did this project create in your service area/community?
a. Are there specific quantifiable changes that you have seen in the
community because of this project?
b. Are there specific qualitative changes that you have seen in the
community because of this project?
8. Is there anything you learned from this project that would be useful to
others who are attempting to do something similar in their communities?
9. Is there anything else you would like to add about the project and its
implementation in your service area/community?
10. Is there anyone else who is familiar with this project and its broad impacts
on the community and/or the region who might provide us with useful
insights? (If yes, collect contact information.)
GO TO SPECFIC QUESTIONS FOR EACH TYPE OF PROJECT
162
Capital
At ARC Project End After ARC Funding
Total number of loans
Number of years
operated
Total $ amount of loans
Number of funds created
Size of Fund(s)
Distribution of loans by
sector
$ funds leveraged
Jobs created
Jobs retained
Percent of portfolio
companies still in
business
*Wages, income per job
or total
Comments:
163
Sectors
At ARC Project End After ARC Funding
Number of participants
Number of members
Jobs created
Jobs retained
*Increase in interfirm
collaboration
*Change in total sector
sales
*Number of business
start-ups in targeted
sector
*Number of participants
retained in service area
*Number of participants
still in business
Comments:
164
Incubators
At ARC Project End After ARC Funding
Number of current clients
Number of clients served
Number of graduated
firms
Number of clients still in
business
Amount $ leveraged by
incubator
*Number of graduates
retained in service area
Jobs created while in
incubator
Jobs created after
graduation
Jobs retained while in
incubator
Jobs retained post-
graduation
$ capital raised by
tenants
Comments:
165
Education
At ARC Project End After ARC Funding
Number of participants
enrolled
Number of participants
completing program
Number of schools
offering entrepreneurship
education pre and post
Number of schools in
service area, pre and
post
Change in student
performance pre and
post
Number of student
businesses started
Number of students that
stay within the service
area
*Increase in awareness
of business concepts
*Increase in number
considering business
creation as a career
option
* Change in community
attitudes toward
entrepreneurship
Comments:
166
Technical Assistance
At ARC Project End After ARC Funding
Number of business
starts
Number of business
expansions
Number of clients
*Number of discouraged
clients
Number of clients still in
business
Number of jobs created
Number of jobs retained
*Firm performance ($
capital raised)
*Number of clients
retained in service area
*Number of clients still in
business
Comments:
167
APPENDIX D
PROTOCOL FOR NON-PROJECT STAKEHOLDERS
1. Background information on the stakeholder
Name:
Title:
Organization:
2. In what ways are you (or have you been) involved with entrepreneurship
and/or economic development in the Appalachian Region?
3. Were you directly involved in any specific projects funded by the ARC
Entrepreneurship Initiative? If yes, please identify the project(s):
4. [For State Alternates Only] What was your state’s strategy for engaging
with ARC’s Entrepreneurship Initiative? (Was this initiative a priority in
your state? Was there a single statewide project approach or did individual
communities/organizations propose projects?)
5. [For State Alternative Only] Did your state’s participation in ARC’s
Entrepreneurship Initiative lead to any change in the policy environment
for supporting entrepreneurship? If so, please explain.
6. What do you think are the most significant region-wide impacts associated
with ARC Entrepreneurship Initiative projects generally? Were there
specific community or project impacts that you can identify?
7. How have these broader impacts been measured?
8. Would you provide some concrete examples of these broader impacts?
9. In your view, what has limited the broader impacts associated with the
ARC projects?
10. In your view, what has contributed to the broader impacts associated with
the ARC projects?
11. Do you think the ARC Entrepreneurship Initiative projects have had an
impact on creating a more supportive climate for entrepreneurs in the
region? Why or why not?
168
12. Considering the broad impacts associated with the ARC Entrepreneurship
Initiative you have identified above, what do you think are some of the
most valuable or effective performance metrics for capturing these
impacts?
doc_537046311.pdf
With this particular brief file concerning creating an entrepreneurial appalachian region findings and lessons from an evaluation.
Creating an Entrepreneurial Appalachian Region:
Findings and Lessons from an Evaluation
of the Appalachian Regional Commission’s
Entrepreneurship Initiative 1997 – 2005
Evaluation Team:
Deborah Markley, RUPRI Center for Rural Entrepreneurship
Erik Pages, EntreWorks Consulting
Brian Dabson, RUPRI
Thomas Johnson, RUPRI
Sara Lawrence, RTI
Sara Yanosy, RTI
Karen Dabson, RUPRI Center for Rural Entrepreneurship
April 2008
i
ACKNOWLEDGEMENTS
The Evaluation Team – Deborah Markley, Erik Pages, Brian Dabson, Thomas
Johnson, Sara Lawrence, Sara Yanosy, and Karen Dabson – would like to
acknowledge the important role played by Catherine Renault on this project.
Cathy was involved in developing the proposal and early project design, before
returning to her roots in Maine as the Director of the Office of Innovation,
Department of Economic and Community Development. She also provided input
on the final report. The team would also like to acknowledge the input of the
advisory group – Thomas Lyons, Edward Malecki, and Jonathan Potter. Their
keen insights and input throughout the project helped to strengthen project
design and the final report. The team also benefited from discussions held with
ARC staff throughout the project, particularly Greg Bischak and Ray Daffner, as
well as a group of advisors who discussed a summary report on key findings.
The senior members of the evaluation team would also like to acknowledge the
heroic efforts of two members of the team, Sara Lawrence and Sara Yanosy, to
develop and complete the database for the sample projects. They diligently
tracked down project leaders to discuss projects that, in some cases, were
completed almost 10 years earlier. They shared data and insights that greatly
enhanced this evaluation.
Finally, this evaluation would not have been possible without the many
individuals throughout the ARC region who shared project data, recollections and
insights into the importance of the ARC Entrepreneurship Initiative to their
individual rural places and to the broader region. We acknowledge their important
contribution and greatly appreciate the time they took to talk with team members
about their past and ongoing efforts to build a more entrepreneurial Appalachian
region.
ii
EXECUTIVE SUMMARY
From 1997 through 2005, the Appalachian Regional Commission (ARC) invested
nearly $43 million in a ground-breaking program to stimulate and support
entrepreneurship across Appalachia. The Entrepreneurship Initiative (EI) was
the first large scale attempt to give greater focus to homegrown business
development as a regional economic development strategy.
The Rural Policy Research Institute (RUPRI), the RUPRI Center for Rural
Entrepreneurship, EntreWorks Consulting, and RTI were commissioned by ARC
in 2006 to conduct an evaluation of EI in terms both of outcomes achieved by a
sample of funded projects and of broader policy impacts across the region. The
evaluation team undertook literature reviews, reviewed project files for a sample
of 114 projects, conducted phone interviews with 36 stakeholders and experts,
developed a metrics framework, completed interviews with project staff
associated with 88 projects, made four site visits, and conducted a meta-analysis
of the outcomes and impacts. The team’s work was informed by a three-person
advisory committee of leading academic experts on entrepreneurship – Dr.
Thomas Lyons, Dr. Edward Malecki, and Dr. Jonathan Potter.
A review of entrepreneurship trends in the region during the EI provided the
context and backdrop for ARC’s investments. Over the period of the EI, trends in
nonfarm proprietor and microenterprise employment in Appalachia showed
increases in line with the nation as a whole, but trends in nonfarm proprietor
income showed the region lagging behind the nation and slipping further behind
by 2005. Data on the impact of entrepreneurship on the local economy showed
that only 15 percent of Appalachian counties saw income increases associated
with entrepreneurial activity that were higher than the national rate. It appears
that entrepreneurship had greater impact in terms of both employment and
income in the southern tier of Appalachian states. This evaluation was not
designed to discern cause and effect between EI investments and these trends;
however, the context is important for interpreting evaluation findings and
understanding the resulting recommendations.
Also of contextual importance is the rapid growth in interest in and adoption of
entrepreneurship development policies and programs since the EI began in
1997. A review of current literature provides many insights on the linkages
between entrepreneurship and regional development and on the efficacy and
impact of different types of entrepreneurship programs. This body of research
work was not available to the designers of the EI but was particularly helpful in
conducting the evaluation and determining appropriate performance metrics.
The evaluation team identified three goals that were at the core of the EI – to
increase the number of entrepreneurs establishing businesses in the region, to
increase the survival rate of such ventures, and to increase the proportion that
develop into high growth businesses that create jobs and wealth in Appalachia.
iii
These goals were operationalized through five program categories –
entrepreneurship education, access to capital, business incubators, sector
interventions, and technical assistance and training. There was also a sixth
cross-cutting category of community capacity-building.
As identified through the final reports submitted to ARC, the EI led to the creation
of at least 9,156 jobs, the retention of a further 3,022 jobs, the formation of 1,787
new businesses, and the provision of services to 8,242 businesses. The cost per
job created was $4,693, which compares favorably with other economic
development efforts. ARC investments were made in 340 unique projects across
the region at an average investment per state of $3.3 million and investment per
capita of $1.82. The total ARC investment has leveraged an additional $72.8
million in private investment for those projects that have been closed, a figure
that is projected to rise to $109.9 million when all projects in the portfolio have
been completed.
Other metrics identified through in-depth investigation of outcomes from the
sample of projects expand on this picture. In the 88 projects included in the
sample, over 11,500 students and teachers participated in or received training in
entrepreneurship education projects, 1,500 entrepreneurs took part in sector-
focused activities and another 1,620 received training and technical assistance.
The evaluation team’s assessment of qualitative impacts were drawn from
interviews with project leaders most familiar with the investments and regional
stakeholders and entrepreneurship experts with deep experience both in the
region and with entrepreneurship development – key informants. Common
themes identified were that ARC investments:
Raised the profile of entrepreneurship as a development strategy, helping
to change the mindset within the region
Represented “but for” money in the region, providing start-up funding for
innovative projects
Leveraged additional resources that helped some projects achieve scale
and impact
Facilitated networking and collaboration among practitioners
Helped to change people’s attitudes, particularly among youth and their
teachers.
There were also a number of lessons gleaned from the many interviews
conducted across the region, some of which will benefit those who are actively
engaged in implementing entrepreneurship programs – the practitioners – and
others which will guide future programs either of ARC or agencies across the
country pursuing similar efforts. For the practitioners, the lessons were of three
kinds:
iv
Lessons for Program Leadership
o Successful entrepreneurship initiatives had sparkplugs or local
champions that provided leadership for these efforts.
o Local capacity was a key to success.
Lessons for Program Management
o Program self-sufficiency (sustainability) and success went hand in
hand.
o Entrepreneurship development was recognized to be a long-term
process.
o Successful projects altered their goals and approaches as
conditions warranted.
Lessons for Program Outreach
o Partnerships and collaborations were important to success.
o Successful projects celebrated and shared the story of their
success.
For program designers and implementers, again the lessons were of three kinds:
Lessons for Program Design
o Practitioners and entrepreneurs have unique local knowledge that
can be applied to program design and subsequent program
refinements.
o Successful initiatives brought together related investments, in this
case, other regional economic development or entrepreneurship-
related investments.
Lessons for Program Implementation
o Getting EI funds to local partners was dependent upon state
leaders, such as governors and program managers, and varied
based on the importance assigned to the initiative.
o The size of ARC grants placed limits on regional impacts.
Lessons for Program Impacts
o Building a broader base of support for entrepreneurship
investments requires continued efforts to “make the case” to local
leaders.
o Programs can be improved by embracing long-term and locally-
driven evaluation of program outcomes and impacts.
Finally, the evaluation team offered three sets of recommendations to ARC.
Regarding investments in entrepreneurship development:
v
Entrepreneurship development initiatives should include assessment of
existing capacity and capacity-building activities as part of the project
design.
Entrepreneurship development initiatives should be made with a focus on
the long term.
Entrepreneurship development initiatives receiving investments should be
market-driven and practice continuous improvement.
Emphasis should be placed on investing in initiatives that demonstrate the
ability to form regional partnerships and collaborations.
The second set of recommendations is for creating a “best in class” metrics
system:
“Job creation” is an overused metric, paints an incomplete picture of the
outcomes of entrepreneurship development investments, and should be
replaced by an “entrepreneurship development metrics portfolio.” In
addition to jobs created/retained and new business starts, this system
should include outcome measures such as:
o Change in business profitability (performance) following a capital
investment
o Number of youth considering business creation as a career option
after participation in an education program
o Percent of incubator tenants who graduate and remain in the region
o Change in total sector sales over time as a result of investment to
encourage sector development
o Number of customers still in business after receiving technical
assistance
o Positive change in perceived community support for
entrepreneurship as measured by community pre- and post-
surveys.
A “best in class” metrics system requires investment in a “best in class”
evaluation system.
The third set of recommendations focus on program design and management:
ARC’s initiative process should be regularized so that state program
managers can more effectively plan for and promote the use of the
resources.
ARC’s proven experience can be applied to developing and delivering
effective, region-wide education programs that help make the case for
entrepreneurship as a core economic development strategy for the
Appalachian region.
To build on the momentum created by the EI, ARC should create a Next
Generation Entrepreneurship Innovation Initiative that will be
groundbreaking in its design. A long-term investment is recommended
vi
that incorporates all the learning from the EI and the emerging
entrepreneurship development field.
vii
Table of Contents
PAGE
CHAPTER 1 INTRODUCTION 1
CHAPTER 2 REVIEW OF THE ENTREPRENEURSHIP
DEVELOPMENT FIELD 5
CHAPTER 3 CONCEPTUAL FRAMEWORK AND REVIEW
OF THE LITERATURE 14
CHAPTER 4 APPROACH TO THE EVALUATION 31
CHAPTER 5 THE WIDER REGIONAL ECONOMY –
ENTREPRENEURSHIP TRENDS IN APPALACHIA 44
CHAPTER 6 ARC INVESTMENT PORTFOLIO – PROJECT
IMPLEMENTATION AND REPORTED METRICS 58
CHAPTER 7 SAMPLE PROJECT OUTCOMES AND
BROADER POLICY IMPACTS 75
CHAPTER 8 LESSONS LEARNED FROM THE ARC
ENTREPRENEURSHIP INITIATIVE EXPERIENCE 88
CHAPTER 9 CONCLUSIONS AND RECOMMENDATIONS 106
BIBLIOGRAPHY 118
APPENDIX A LITERATURE REVIEW 130
APPENDIX B NON-PROJECT STAKEHOLDER INTERVIEWS 159
APPENDIX C PROTOCOL FOR PROJECT LEADERS 160
APPENDIX D PROTOCOL FOR NON-PROJECT STAKEHOLDERS 167
1
CHAPTER 1
INTRODUCTION
In 1997, the Appalachian Regional Commission (ARC) began a multi-year
initiative to invest in projects designed to build entrepreneurial economies across
the region – the Entrepreneurship Initiative (EI). Since that time, ARC has
invested almost $43 million (composed of EI funds and dollars from other ARC
accounts) in various entrepreneurship development projects. Over 10 years,
these projects have created jobs and businesses, supported partnerships and
collaborations, and helped leaders at the community and state levels recognize
the value of entrepreneurship as an economic development strategy. While ARC
has collected select data to describe the outcomes of these investments, these
data do not begin to tell the story about the extent to which and how the EI has
had an impact on the region. This evaluation was designed to provide a more
detailed and nuanced description of the impact of the EI, both in terms of the
outcomes achieved by the portfolio of projects and the broader policy impacts
across the region. This evaluation, however, must first be placed within the
context of the initiative’s history.
HISTORY OF THE PROGRAM
The EI emerged in 1997 as a special initiative under the leadership of then-
Federal Co-Chairman Jesse White. White had long pushed for economic
development strategies focused more on home-grown business development as
opposed to business recruitment and attraction. As he told a Federal Reserve
Bank of Kansas City symposium in 2000, ”[W]e’ve got to re-instill in rural America
the idea, particularly in Appalachia and the Rural South, that job creation,
business creation, and, most importantly, wealth creation, occurs as a result of
local indigenous business creation.”
1
The EI was designed as a test of public
policy approaches that sought to achieve this objective.
The genesis for the EI was in the belief that entrepreneurial activity could be
encouraged through strategic investments in education, business assistance,
and capacity building projects. Areas for strategic investment identified prior to
the launch of the EI included:
Access to capital and financial assistance
Technical and managerial assistance
Technology transfer
Entrepreneurial education and training
Entrepreneurial networks.
1
Jesse White, “Overview Panel Comments,” Proceedings of Federal Reserve Bank of Kansas
City Conference on Beyond Agriculture: New Policies for Rural America (April 2000):193.
2
The EI was originally funded with $15 million over three years, with additional
investments being provided beyond this original total. Through 2000, ARC had
invested $17.6 million in 169 projects.
2
Through 2003, ARC had invested $31.4
million in 368 projects. An additional amount of approximately $11 million was
invested in subsequent years through the EI or via the use of Area Development
funds. Table 1.1 shows ARC investment in entrepreneurship-related activities by
source of funds for the period of this evaluation. While all ARC investments in
entrepreneurship were not specifically drawn from EI funds, the rationale for
making these investments was clearly driven by the goals associated with the EI.
Today, ARC continues to fund entrepreneurship development-related projects
under the Asset-Based Development Initiative launched in 2005, although the EI
is no longer operating as a stand alone initiative. The Asset-Based Development
Initiative is building on the foundation laid by the EI, leveraging the new
businesses and additional capacity created by EI investments.
Table 1.1 ARC Funds Invested in Entrepreneurship-Related Projects, 1997-
2005, by Source of Funds
Source of funds $ invested
Entrepreneurship Initiative 26,546,366
Area Development 6,698,724
Commission’s EI 3,206,803
Distressed Counties 3,052,109
CoChair Fund 2,432,440
Regional Initiatives 868,673
Goal Fund 114,000
New Markets Fund 52,574
TOTAL 42,971,688
ORGANIZATION OF THE EI
ARC’s special initiatives are traditionally designed to support innovative local
projects and also to serve as a demonstration of new regional development
strategies and approaches. ARC officials were very explicit about the importance
of the EI as a means to demonstrate the viability of entrepreneurship as an
economic development strategy. Interviewed experts also emphasized this
aspect of the EI.
The EI’s organization reflects this dual focus on educating local leaders and on
maximizing the effect of local project investments. In pursuing its wider
educational goals, ARC recognized the need to form partnerships and engage
other institutions in order to achieve sustainable impacts in the region. Through
the formation of four advisory committees with significant private sector
2
For an early review of these investments, see Regional Technology Strategies, Inc., Evaluation
of the Early Stages of the Appalachian Regional Commission’s Entrepreneurship Initiative, A
Report to the Appalachian Regional Commission, December 2001.
3
participation, ARC was able to tap into national expertise and “best practices” to
guide the initiative. Each advisory committee was charged with providing input to
ARC in a particular program area – entrepreneurship education, technical
assistance, capital access, and sectorally targeted strategies. In particular, these
advisory committees helped to organize a region-wide educational effort that
augmented the specific project investments made by ARC. Elements of this
educational effort included, for example:
Scholarships to the 16
th
Annual Entrepreneurship Education Forum and
support for three regional entrepreneurship conferences; creation of the
Springboard Youth Entrepreneurship Education Awards; creation of the
Entrepreneurship Everywhere web-based resource guide
3
Support for conferences on sector-based development; a competitive
grant program for “strategic sectoral interventions” in the region
Sponsorship of a workshop on community development venture capital;
follow-up regional workshops on equity capital in rural communities;
publications on developmental venture capital in the region; creation of an
opportunity fund to leverage private investment using the New Markets
Tax Credit program
Funding for four workshops on business incubation “best practices” in the
region; a survey on business incubators in Appalachia; creation of a
business incubation mentor program.
These region-wide educational efforts were, for the most part, funded outside the
EI, using other ARC dollars, such as Commission or CoChair funds. While using
distinct funding sources, these projects also contributed to the mission of the EI –
to encourage the development of an entrepreneurial economy in the region. In
fact, it is not possible to separate the impact of broader ARC investments in
entrepreneurship from those specifically identified with the EI. Most of ARC’s
investments through the EI resulted in project-specific outcomes; the outcomes
of their educational efforts were much broader, serving to raise the overall level
of regional awareness about specific aspects of entrepreneurship development –
education, sector approaches, capital and incubation. The impacts, in most
cases, were not confined to particular communities or even states. Instead, the
impacts could be measured in terms of increased understanding of several
aspects of entrepreneurship development and enhanced capacity to design and
implement entrepreneurship activities within the region.
While this evaluation did not seek to quantify ARC’s capacity building and
education impacts related to entrepreneurship development, this effect was cited
in nearly all of the interviews with outside experts and state level officials.
During interviews, the evaluation team regularly heard comments such as the EI
“opened people’s eyes to other possibilities” or “the program got people talking
about entrepreneurship.” Several other programs, such as the New Markets
3
Available at .
4
Venture Capital program, built on momentum generated by the EI. In fact, two of
the first six New Markets Venture Capital companies operate in Appalachia.
While the EI’s impact in terms of building better awareness of entrepreneurship
cannot be quantified, it should be included in any full accounting of the EI’s
outcomes. The EI was designed as a demonstration, i.e., to test new policy
models and to encourage Appalachian communities to focus within on nurturing
home-grown businesses. Today, nearly all of the Appalachian states are involved
in some form of organized efforts to promote entrepreneurial development.
While the ARC cannot claim sole credit for this shift in thinking, based on
feedback from key stakeholders it is clear that the EI played a role in convincing
policy makers that supporting entrepreneurs is good economic development
policy.
STRUCTURE OF THIS REPORT
This report presents the findings from an evaluation of ARC’s investments in
entrepreneurship development from 1997 through 2005. Based on analysis of
data collected by ARC, in-depth research into the impacts associated with a
sample of ARC-funded projects, and interviews with a broad range of
stakeholders, this evaluation provides key insights into the value of the EI to the
region. Chapter 2 provides an overview of entrepreneurship development,
particularly in terms of the state of the field in 1997, the beginning of the EI. This
context is important for understanding how innovative ARC was in launching the
EI, plowing new ground in terms of economic development. Chapter 3 lays out
the conceptual framework developed to guide this evaluation process and
reviews relevant evaluation literature related to entrepreneurship development.
This review of the literature served as the basis for developing the metrics
framework that would guide in-depth evaluation of sample projects. Chapter 4
describes this metrics framework and the overall research approach used in this
study.
Chapter 5 describes key aspects of the Appalachian region’s business and
economic environment that created the context for the EI. It is this context that
created the need for and gave rise to the projects that became part of the EI
portfolio. Changes over the ten year time horizon of the EI are also presented.
Chapters 6 and 7 provide the key findings from this evaluation. Chapter 6
describes the types of projects in which ARC invested and presents data on key
metrics collected by ARC for all projects. Chapter 7 provides a detailed
discussion of the impacts, quantitative and qualitative, associated with the
sample of projects considered for this evaluation. Chapter 8 presents the
lessons learned from the EI experience for both practitioners of entrepreneurship
development and policy makers who may be considering similar types of
investment programs in the future. Finally, Chapter 9 provides recommendations,
based on evaluation findings, for the field and for ARC going forward.
5
CHAPTER 2
REVIEW OF THE ENTREPRENEURSHIP DEVELOPMENT FIELD
While entrepreneurship has always been a feature of the American (and
Appalachian) landscape, the history of explicit public support for regional
entrepreneurial development is quite brief. Indeed, ARC’s Entrepreneurship
Initiative serves as one of the first such large-scale efforts, and the absolute first
such effort investing Federal dollars in regional entrepreneurship strategies.
At the time of the Entrepreneurship Initiative’s unveiling in 1997, ARC’s
leadership had grown increasingly concerned about Appalachia’s future vitality.
The region was especially hard hit by cutbacks in declining sectors such as
timber, textiles, tobacco, and was seeking new approaches to jump start
development. Entrepreneurship-focused economic development strategies were
viewed by ARC as “. . . a critical element in the establishment of self-sustaining
communities that create jobs, build local wealth, and contribute broadly to
economic and community development.”
4
By investing in the Entrepreneurship Initiative in 1997, the ARC was something of
an “early adopter.” Up to that stage, economic developers had focused almost
exclusively on industrial recruitment as a core strategy. The industrial
restructuring of the 1970s and the 1980s had stimulated some interest in
business retention and technology development strategies, but entrepreneurial
development strategies represented something of a “new thing” in 1997.
5
Nevertheless, ARC was not promoting entrepreneurship without some evidence
that it would make a significant contribution to economic opportunity in
Appalachia. In the late 1980s, Eisinger discussed the limited returns to traditional
economic development activities and suggested a more important role for
entrepreneurial, growth from within, economic development strategies.
6
There
was some research that questioned the efficacy of traditional “smokestack
chasing” and some that provided insights into the impact of specific programs
such as business incubation, youth entrepreneurship, and targeted technical
assistance. What was missing at that time was the evidence to link
entrepreneurship to regional and community development, and to identify which
policy interventions, and in what combination, would lead to increased
entrepreneurial activity. Over the past decade, there has been rapidly growing
interest among researchers and policy analysts in entrepreneurship as a core
4
Appalachian Regional Commission, Entrepreneurship Initiative: Program Summary and
Approved Projects, September 2003, 9 August 2007
.
5
For background, see Erik R. Pages, Doris Freedman, and Patrick Von Bargen ,
“Entrepreneurship as a State and Local Economic Development Strategy,” The Emergence of
Entrepreneurship Policy: Governance, Start-Ups and Growth in the U.S. Knowledge Economy,
Ed. David Hart, Cambridge, MA: Cambridge University Press, 2003.
6
Peter K. Eisinger, The Rise of the Entrepreneurial State: State and Local Economic
Development Policy in the United States, Madison, WI: University of Wisconsin Press, 1988.
6
economic development strategy, and more evidence has been found to support
this approach. It was important to the evaluation to understand the implications of
this expanded understanding in order to appropriately assess the value and
impact of the EI investment.
This chapter provides a brief overview of research, policy and practice that
addresses two primary questions:
What makes a region (or community) entrepreneurial?
What policy or community interventions, if any, can help support
entrepreneurial development?
WHAT MAKES A REGION ENTREPRENEURIAL?
Researchers have examined a host of both macroeconomic and microeconomic
factors that help explain the innovation or entrepreneurial capacities of a region.
At the broadest level, recent research has reviewed how leading demographic
trends are correlated with new firm births. For example, in a 1994 review article,
Reynolds, Storey and Westhead describe key factors that are associated with
higher levels of new firm starts.
7
These include net population growth, increases
in personal or household income or regional gross product, high population
density, high educational levels among population, and a high percentage of
population between the ages of 25 and 44. Acs and Armington similarly find that
higher relative education levels strongly affect new firm formation rates,
especially among service businesses.
8
These same factors have been
highlighted in the cross-national research studies conducted under the auspices
of the Global Entrepreneurship Monitor (GEM) research consortium.
9
While this literature contains something of a consensus about several key factors
(e.g., population growth, higher education levels) that are correlated with higher
new firm formation rates, some researchers caution that the causal chain still
remains unclear. For example, Feldman notes that many key factors associated
with entrepreneurship, such as the presence of local venture capital firms, may
actually lag instead of lead entrepreneurial growth.
10
In other words, these
7
Paul Reynolds, D.J. Storey, and P. Westhead. “Crossnational Comparisons of the Variation in
New Firm Formation Rates, Regional Studies 28.4 (1994):443-456.
8
Zoltan Acs and Catherine Armington, “Using Census BITS to Explore, Entrepreneurship,
Geography, and Economic Growth,” Research Summary No. 248, Small Business Administration
Office of Advocacy, February 2005. Similar findings that emphasize the importance of local
educational levels as well as local employment growth and productivity rates can be found in
Advanced Research Technologies, “The Innovation-Entrepreneurship NEXUS,” Research
Summary No. 256, Small Business Administration Office of Advocacy, April 2005.
9
Begun in 1999, the Global Entrepreneurship Monitor project is a cross-national effort to assess
national rates of entrepreneurship. Begun with ten countries, the research program now tracks
entrepreneurial activity in forty-two countries. The project’s research reports can be accessed at
.
10
Maryann Feldman, “The Entrepreneurial Event Revisited: Firm Formation in a Regional
Context,” Industrial and Corporate Change 10.4 (2001):861-891.
7
regional assets emerge as a result of strong local entrepreneurial activity. They
are a by-product, as opposed to a trigger, for high firm formation rates. Feldman
concludes by cautioning that each region’s entrepreneurial development activities
generally emerge from a unique and idiosyncratic mix of historical factors, local
resources, and business conditions.
PUBLIC POLICY AND ENTREPRENEURSHIP
While many economists and researchers continue to assess how various
demographic and macroeconomic factors help drive regional entrepreneurial
activity, policy makers, including the ARC’s leadership, are more concerned with
questions of how (and whether) policy interventions can affect a region’s
entrepreneurial propensities and development patterns. A large and growing
literature examines this issue,
11
and much of this work has helped guide the
strategic direction of ARC’s own Entrepreneurship Initiative.
While individual analysts may differ on the relative importance of certain regional
factors, there is relatively strong consensus that five factors are especially
important:
Access to Capital
Enabling Culture
Local Networks
Supportive Infrastructure
Supportive Government Policies.
A more detailed description of each factor and the public policy approaches used
to address them is provided below.
Access to Capital
Successful entrepreneurial regions tend to enjoy a wide range of options for
financing businesses at different stages of the business life cycle. Successful
regions host a variety of financial institutions that can provide businesses with a
range of both equity and debt financing options. Rural communities may be
especially challenged on this front. Recent US Department of Agriculture-
sponsored research has found that rural areas tend to have fewer lenders and
less diverse markets. While rural areas do have less bank competition, the study
found few rural-urban differences between the cost and availability of debt
11
For an excellent comprehensive literature review, see Jill S. Taylor, “What Makes a Region
Entrepreneurial? A Review of the Literature,” Monograph, Cleveland State University, Center for
Economic Development, September 2006. Other good sources include Brian Dabson, et al.,
Mapping Rural Entrepreneurship, Washington, D.C.: CFED, August 2003; Deborah Markley, Don
Macke and Vicki Luther, Energizing Entrepreneurs: Charting a Course for Rural Communities,
Lincoln, NE: Heartland Center for Leadership Development, 2005, and OECD, Entrepreneurship
and Local Economic Development, Paris: OECD, 2003.
8
financing.
12
In fact, a recent study of capital access in the Appalachian region
found that banks in the region had higher small business loan to deposit ratios as
compared to national figures.
13
However, businesses in rural regions experience capital access problems when
their needs fall outside the type of loans traditionally made by banks – e.g.,
microloans (to fund very small enterprises with limited collateral) and equity
finance. Microenterprise development was just emerging in the early to mid-
1990s. A recent study of the industry noted that only about one-quarter of
microenterprise programs listed in a 2001 directory existed before 1991.
14
The
importance of providing both training and microfinancing to entrepreneurs was
just being viewed as a development strategy as ARC invested in the EI.
Appalachia also suffers in terms of access to equity finance. For example, a 2000
ARC study found that only 1/3 of 1% of all venture capital ($117 million) was
invested in rural regions of Appalachia.
15
A more recent study of capital access in
the Appalachian region shows improved access to equity capital.
16
Through
2004, the study shows that ARC had invested in 11 funds in seven states and
these funds collectively had invested $13.6 million in regional businesses. If all of
this investment is considered new to the region, it represents a 12% increase in
equity capital as compared to that found in 2000. However, the later study also
identified a need to continue to expand the capacity of community development
financial institutions in the region by broadening sources of funds, increasing self-
sufficiency, and expanding products available to businesses.
As a result of these market gaps, policy makers in Appalachia and elsewhere
have supported a host of initiatives to develop new sources of microcredit and
equity and equity-like capital.
17
Within the region, ARC funded microenterprise
development initiatives and revolving loan funds, as well as others that created a
venture capital industry. At the Federal level, these initiatives include the Small
Business Administration’s (SBA) Microenterprise Loan and PRIME (Program for
12
Ray Collender, et al., “Financial Markets Serve Rural Areas Fairly Well,” Rural Development
Perspectives 14.1 (May 1999):28-35.
13
National Community Reinvestment Coalition, Access to Capital and Credit for Small
Businesses in Appalachia, Washington, D.C.: National Community Reinvestment Coalition, April
2007.
14
Elaine L. Edgcomb and Joyce A. Klein, Opening Doors, Building Ownership: Fulfilling the
Promise of Microenterprise in the United States, Washington, D.C.: FIELD, A Program of the
Aspen Institute, 2005 28 January
2008.
15
Appalachian Regional Commission, Capitalizing on Rural Communities, Washington, DC:
Appalachian Regional Commission, 2000, 8.
16
National Community Reinvestment Coalition, April 2007.
17
Deborah Markley, et al. Rural Equity Capital Initiative Study of Nontraditional Venture Capital
Institutions, RUPRI PB2001-11A-D, 2001,
. An excellent summary of
these efforts can be found in a special issue of the Federal Reserve Bank of San Francisco’s
Community Development Review 3.2 (2006).
9
Investment in Micro-Entrepreneurs) programs, the Small Business Investment
Company (SBIC) program, the New Markets Tax Credit initiative, the CDFI Fund,
and other efforts. State and local initiatives have been more far-reaching and
comprehensive, including Nebraska’s Microenterprise Partnership Fund, the
Oklahoma Center for the Advancement of Science and Technology (OCAST)
and the Pappajohn Entrepreneurship Center at North Iowa Area Community
College (NIACC). Nebraska provides state support for microenterprise
development through a public-private partnership that channels investment to
microenterprise development programs across the state through a competitive
process. OCAST has developed a comprehensive set of financing tools for
technology businesses in the state that includes pre-seed financing of up to
$100,000 for tech start-up companies, a seed capital program with equity
investments up to $750,000, and a number of technical assistance and sector-
specific programs.
18
NIACC’s capital programs are designed to work hand-in-
hand with technical assistance and education programs available at the
community college. Capital programs include a revolving loan fund providing debt
capital, a nanoloan program providing debt capital for microenterprises, and
access to both formal venture capital and angel investors.
19
Finally, private
programs, such as the creation of local angel investor networks, are also being
introduced across the US. Angel investor groups have grown from 50 formal
groups in 1997 to an estimated 170 formal and informal groups in 2002;
20
there
were an estimated 200,000 individual angel investors active in 2002 and 234,000
in 2006.
21
Enabling Culture
The need for an “enabling culture” is widely recognized in the literature, but the
details of what constitutes such a culture are expressed in different ways by
researchers and practitioners. At the most basic level, an enabling culture is
one that understands, recognizes, and honors the importance of local
entrepreneurs. These three terms – understanding, recognizing, and honoring –
also connote three different sets of potential policy interventions.
18
More information is available about OCAST on their website, .
19
More information about NIACC can be found on their website,
and in Deborah Markley and Karen Dabson, Innovative
Approaches to Entrepreneurial Development: Cases from the Northwest Region, RUPRI Center
for Rural Entrepreneurship, 2006, 32-41.
20
“Business Angel Investing Groups Growing in North America,” Ewing Marion Kauffman
Foundation, October 2002, 4 December 2007
;
21
“Full Year 2002 Angel Market Analysis Report,” Center for Venture Research, Whittemore
School of Business and Economics, University of New Hampshire, 4 December 2007
; Full Year
2006 Angel Market Analysis Report,” Center for Venture Research, Whittemore School of
Business and Economics, University of New Hampshire, 4 December 2007
.
10
To promote understanding, analysts recommend the introduction of
entrepreneurship education to all parts of the population. The introduction of
entrepreneurship education at the college and university level has been a
resounding success. In 1979, only 127 schools offered courses in small
business and entrepreneurship. Today, more than 1600 schools offer this
training.
22
Community colleges have also witnessed major growth in
entrepreneurship training. The introduction of entrepreneurship education at the
primary and secondary school levels has been less smooth, but innovative
programs and curricula are widely available across the US.
23
To promote recognition, analysts have advocated for several ideas, including
more active entrepreneur involvement in the policy-making process. Given the
time demands of running a business, few entrepreneurs have stepped up to this
challenge. However, several states have created advisory bodies, such as New
York’s Small Business Advisory Board, to provide opportunities for this input.
The creation of regional business plan competitions is another widely used tool
that helps publicize local entrepreneurs. These competitions exist across the
US, with a scale that can range from major national/international competitions to
smaller local efforts focused on youth or specific market segments. Several of
the ARC’s Entrepreneurship Initiative grants funded efforts of this sort.
To honor entrepreneurs, many analysts recommend the creation of awards
programs such as local Entrepreneur of the Year Awards.
24
ARC’s own
Springboard Awards, designed to honor innovations by entrepreneurship
educators, was a particularly effective design for a regional awards program.
These education and recognition efforts are critical to improving the local climate
for entrepreneurs, but existing cultural attitudes also come into play. Extensive
research indicates that cultural factors play an important role in explaining
differing entrepreneurship rates across countries and even within countries. For
example, Giannetti and Simonov find that the presence of local entrepreneurial
role models helps explain differences in regional entrepreneurship rates.
25
A
recent US study also stresses the importance of role models.
26
Research
sponsored by the Global Entrepreneurship Monitor project notes that differences
22
Jerome Katz, And Another Thing. . .,”2006 Coleman Foundation White Paper on
Entrepreneurship Education, 2006 Annual Meeting of US Association of Small Business and
Entrepreneurship, January 13, 2006, 9 August 2007
.
23
For background, see National Governors Association, A Governor’s Guide to Strengthening
State Entrepreneurship Policy, Washington, DC: National Governors Association, 2004.
24
National Governors Association, 2004.
25
Mariassunta Gianetti and Andrei Simonov, On the Determinants of Entrepreneurial Activity:
Individual Characteristics, Economic Environment, and Social Norms, White Paper, Stockholm
School of Economics, June 2004, 9 August 2007,
.
26
Edward J. Malecki, “Geographical Environments for Entrepreneurship,” International Journal of
Entrepreneurship and Small Business, Forthcoming 2008.
11
in cross-national entrepreneurship rates are influenced by local cultural attitudes
toward risk-taking and fear of failure.
27
Like many rural regions, some
Appalachian communities perform poorly on various measures of cultural
attitudes toward entrepreneurship. Indeed, the ARC’s own materials note that
“the culture of entrepreneurship is neither wide nor deep throughout
Appalachia.”
28
From the perspective of ARC’s Entrepreneurship Initiative, recognition of the
importance of an enabling culture is reflected in the number of program
investments that can be described as “community capacity building” – facilitating
visioning, leadership development, asset-mapping and community engagement
activities intended to make the community more supportive of and attractive to
entrepreneurs. In many cases, initial ARC investments were designed to build
capacity for supporting entrepreneurs, with follow on investments supporting
program implementation. These capacity building investments were designed, in
essence, to enhance community social capital. An extensive literature measuring
the importance of social capital for economic development, and the impact of
enhanced social capital on community development outcomes, has been
developed by researchers, particularly Cornelia Flora.
29
Local Networks
Networks refer to local locations (both virtual and physical locations) where
entrepreneurs can gain access to peers and others with expertise or knowledge
about the processes of starting and growing a business. Entrepreneurs
regularly report that such networks are a critical component in helping them learn
the ins and outs of business and gain easier access to needed support
services.
30
Such networks are commonplace in major urban areas, especially in
technology hot spots such as Silicon Valley or Boston. They are less commonly
found in rural areas as they typically depend on a critical mass of local business
owners with an interest in networking. Because these dense concentrations of
business owners and service providers do not exist in many rural regions,
networking opportunities are often lacking.
27
See, for example, Maria Minniti with William Bygrave and Erkko Autio, Global Entrepreneurship
Monitor 2005 Executive Report,
.
28
Appalachian Regional Commission, Entrepreneurship Initiative: Program Summary and
Approved Projects, September 2003, 9 August 2007
.
29
See for example, Mary Emery and Cornelia Flora, “Spiraling Up: Mapping Community
Transformation with Community Capitals Framework,” Journal of the Community Development
Society 37.1 (Spring 2006):19-35.
30
See, for example, Erik R. Pages and Shari Garmise, “The Power of Entrepreneurial Networks,"
The Economic Development Journal 2.3 (Summer 2003):20-30. For a local example of network
building efforts, see North Carolina Rural Economic Development Center, Hello, My Business
Name is : A Guide to Building Entrepreneurial Networks in North Carolina, Raleigh, NC: North
Carolina Rural Economic Development Center, 2007.
12
Many regional organizations – both public and private – have jumped in to help
seed new networks. By providing staff support and limited funding, economic
developers hope to jump start local networks that can then operate on their own.
The networks can operate in a manner that is open to all entrepreneurs, such as
a regional network, or they can focus on the issues and challenges facing a
specific industry sector or cluster, such as ceramics and related industries.
Supportive Infrastructure
Entrepreneurs are no different from other kinds of business owners in their need
for a supportive and reliable physical infrastructure including transportation, water
and sewer. Building such infrastructure has been a traditional focus of ARC,
through its highway program and other investments. More recently, ARC’s
broadband initiative has invested in providing critical IT-related infrastructure for
the region’s businesses.
ARC-sponsored research clearly indicates the powerful impacts of investments in
traditional infrastructure such as highways, water and sewer facilities, and
industrial parks.
31
Investments in broadband infrastructure can also have
beneficial economic development impacts. Research from Minnesota indicates
that the presence of high speed Internet access was a major factor in explaining
the presence of local gazelle (fast-growing) businesses in regions across the
state.
32
Other research indicates that communities with extensive broadband
access outperform comparable regions without such amenities in terms of job
growth and the number of businesses.
33
Supportive Government Policies
While entrepreneurs may succeed anywhere, they are more likely to flourish in a
region where government and community leaders are “entrepreneur-friendly.”
This can take two forms:
The creation and nurturing of a supportive tax and regulatory climate
The provision of a wide range of private and publicly-funded business
services, including technical assistance, incubators, and other forms of
specialized support.
31
The Brandow Company and Economic Development Research Group, Evaluation of the
Appalachian Regional Commission’s Infrastructure and Public Works Program Projects,
Washington, D.C.: Appalachian Regional Commission, 2000.
32
Thu-Mai Ho-Kim and Ernesto Venegas, “Starting Up Economic Engines: Key Factors for
Growing Successful Start-Ups,” Minnesota Department of Employment and Economic
Development Issue Brief, October 2003.
33
Sharon E. Gillett, et al, Measuring the Economic Impact of Broadband Deployment,
Washington, D.C.: Economic Development Administration, February 28, 2006.
13
An early study by the National Commission on Entrepreneurship documented
federal policy in support of entrepreneurship from 1958-1998.
34
Since that time,
much of the policy innovation surrounding entrepreneurship has occurred at the
state level. For example, in 2004, the Kansas legislature passed a
comprehensive package of legislative support for entrepreneurship development
including the creation of an entrepreneurship center and implementation of a tax
credit program to spur investment in new ventures.
35
As the focus has shifted
from federal to state policy in recent years, the impact analysis of these policies
remains to be done.
THE STATE OF THE FIELD AND THE EI
Much of the research and policy innovation described above has happened in the
years since ARC’s EI was launched. For example, the Global Entrepreneurship
Monitor project was initiated first in 1999 with only 10 countries; GEM 2007
research is being conducted in 42 countries. Most state innovations have
occurred since 2000. ARC was clearly leading the way in encouraging the
practice of entrepreneurship as an economic development strategy. As such,
ARC investments were designed to demonstrate the potential associated with a
wide range of strategies, from youth education to venture capital development.
Given that the entrepreneurship development field was in its early innovation
stage, such an approach, referred to by one stakeholder as “let a thousand
flowers bloom,” was strategic in its design and appropriate for the time.
34
National Commission on Entrepreneurship, American Formula For Growth, Washington, D.C.:
National Commission on Entrepreneurship, October 2002.
35
The Kansas Economic Growth Act of 2004 established the Kansas Center for Entrepreneurship
and introduced several capital initiatives including StartUp Kansas – a seed capital fund,
authorization for regional foundations, and angel networks. A review of other state efforts can be
found in National Governors Association, 2004.
14
CHAPTER 3
CONCEPTUAL FRAMEWORK AND REVIEW OF THE
LITERATURE
Given the breadth and diversity of investment activity undertaken by ARC as part
of the EI, the evaluation team began by reviewing the actual operating realities of
the initiative. The ARC set general guidelines for programs, but each state
responded to these guidelines in a unique manner. Similarly, each project was
quite unique. It is easy to understand that a grant for high school
entrepreneurship summer camps will have different metrics than those of a
revolving loan fund. Yet, great diversity also existed within single program
categories. For example, investments in business incubation might fund a
feasibility study, build a new incubator, or provide technical assistance to new
sets of customers. As will be seen in Chapter 4, building a set of consensus
metrics for such a diverse set of program interventions proved quite challenging.
At the outset of the EI, ARC convened four advisory committees charged with
identifying best practices and key insights into four areas ARC had identified as
critical for building an entrepreneurial region – access to capital,
entrepreneurship education, technical assistance, and sectoral strategies. As an
outgrowth of the work undertaken by these committees, ARC eventually
identified five strategic areas for investment - access to capital and financial
assistance, technical and managerial assistance, technology transfer,
entrepreneurial education and training, and entrepreneurial networks. However,
after an initial review of the portfolio of projects receiving ARC investments from
1997-2005, the evaluation team identified five program categories that captured
the actual range of projects implemented:
Capital Access – provision of services and technical assistance to
connect entrepreneurs and businesses to appropriate levels and types of
debt and equity capital
Entrepreneurship Education – structured experiential opportunities in
and out of school for young people (K-16) to learn entrepreneurial skills
and attributes and to understand business basics with the aim of
encouraging young people to consider entrepreneurship as a career
sooner or later
Incubators – provision of opportunities for acquiring information, skills,
resources within (often subsidized) workspace settings
Technical Assistance and Training – provision of expert and/or peer
one-on-one mentoring and consulting services on technical and
managerial matters and provision of information and skills in formal
classroom or laboratory settings to adult entrepreneurs
Sectors – packaging of some combination of training, incubating,
technical assistance, and capital access services targeted at a single,
specific business sector.
15
Another category, Community Capacity, facilitating visioning, leadership
development, asset-mapping and community engagement activities intended to
make the community more supportive of and attractive to entrepreneurs, was
also identified. This category actually cuts across the other programmatic areas
as initial ARC investments were often designed to build capacity for supporting
entrepreneurs, with follow on investments supporting program implementation.
The program categories defined by the evaluation team differ somewhat from the
five strategic areas originally defined by the Appalachian Regional Commission.
After reviewing actual investments made by the EI, the team determined that
investments had not been made in the technology transfer area, while actual
investments in entrepreneurial networks were more accurately described as
investments in either sector-specific activities or incubation activities, such as
investment in Virginia’s sustainable wood products industry or Ohio’s food sector.
As an example, ARC's investments in commercial kitchen incubators, while
focused on a specific sector, were coded as incubators. Table 3.1 lists ARC’s
strategic areas and describes how and why they were modified and repackaged
into the five program categories used as the basis for this evaluation. All of these
changes were made after a review of the actual projects in which ARC invested
and represent the adaptation of the ARC investment process to the needs of
communities as reflected by their proposed project activities.
Table 3.1: Comparison of ARC Strategic Areas and Evaluation Program
Categories
ARC Strategic Areas Comments
Access to capital and financial assistance Capital Access – no change
Technical and managerial assistance Split into Technical Assistance and Training
and Incubators to reflect different modes of
entrepreneurial support
Technology transfer Not implemented
Entrepreneurial education and training Split between Entrepreneurship Education
for youth and Training (in Technical
Assistance) for adults
Entrepreneurial networks Actually implemented as Sector or Incubator
initiatives
Community Capacity – added to capture the
place-based and cultural dimensions of the
initiative; a category that cuts across program
categories and strategic areas
DEVELOPMENT OF A CONCEPTUAL FRAMEWORK
The origins of the Appalachian Regional Commission’s Entrepreneurship
Initiative were rooted in the challenge for states and communities throughout the
region – a challenge that is shared across rural America – of how to foster the
economic and cultural conditions that give birth to entrepreneurs, support
innovation, and assist in the development and expansion of successful
enterprises. These desired outcomes were, and still are, considered critical to
16
Appalachia’s future economic vitality. Moreover, they were seen as essential to
reducing the region’s dependence upon extractive industries and branch plant
manufacturing, to replacing the long-term practice of asset-stripping with asset
accumulation and value-addition, and to creating a more entrepreneurial and
outward-looking culture.
Given these expectations, the evaluation team chose to approach its charge by
establishing at the outset a conceptual framework that required a sharp focus on
the essence of the initiative and reduced the temptation to explore interesting but
tangential avenues that would perhaps make the evaluation richer but less useful
as a guide to future policy and action. Three fundamental goals were identified
which best reflect the primary purposes of the EI. While the following goals were
not articulated specifically by ARC, they reflect the mission and purpose of the EI
as understood through ARC publications and interviews with key leaders:
More entrepreneurs – To increase the number of entrepreneurs
establishing businesses in Appalachia
Stronger entrepreneurs – To increase the survival rate of entrepreneurial
ventures in Appalachia
More high growth entrepreneurs – To increase the proportion of
entrepreneurial ventures that achieve rapid growth rates, thus providing
jobs and wealth within and increasing the competitiveness of Appalachia.
These fundamental goals then were operationalized by the evaluation team into
six programmatic goals:
More entrepreneurs in the pipeline – increasing the number of people,
youth and adults, who are actively considering setting up their own
businesses
More entrepreneurs staying – creating the conditions in which
entrepreneurs wish to stay and grow their businesses in their community
Better informed entrepreneurs – providing entrepreneurs with the
information and tools they need to establish and grow their businesses
Better skilled entrepreneurs – providing entrepreneurs with the technical
and managerial skills they need to sustain and grow their businesses
More job creating businesses – providing the tools and resources to
encourage entrepreneurs to expand and employ others
Greater business productivity – providing the tools and resources to
enable entrepreneurs to operate efficient and competitive businesses.
While this conceptual framework proved useful as an organizing framework for
the evaluation, the team needed to align this framework with the stated goals and
operational practice of the EI. The program categories that grew out of the
evaluation team’s review of ARC’s investment portfolio map well onto the set of
programmatic goals articulated in the conceptual model. Each category was
identified as contributing to the attainment of at least one of these programmatic
17
goals. Table 3.2 shows how the program categories and programmatic goals
intersect. It is certainly possible to argue for additional Xs in this table. Indeed,
one might argue that all project types could have a long-term impact on every
programmatic goal. However, we have chosen to focus on the primary goal(s) of
each program type. For example, while entrepreneurship education programs
ideally, in the long run, lead to more youth creating businesses and/or staying in
the region, the primary goal of these educational efforts is to expose young
people to entrepreneurial concepts and possibilities, with the ultimate goal of
having more youth pursue entrepreneurship as a career path, leading to more
entrepreneurs in the pipeline.
The conceptual framework developed to provide context for the ARC
Entrepreneurship Initiative and to ground the evaluation was also used to inform
the review of the literature related to entrepreneurship development.
REVIEW OF THE LITERATURE
Entrepreneurship and efforts to promote entrepreneurship as an economic
development strategy have emerged as important topics for both research and
policy discussion. The literature is vast and seems to be growing larger every
day. The majority of this research focuses on questions directly related to
entrepreneurs and their companies by examining questions such as appropriate
business development strategies, the qualities of successful entrepreneurs, or
the role of various management practices in producing business success or
failure. Very few studies address the measurement of program impacts or
community outcomes. This section presents a review of both current trends in
economic development performance measurement and of the limited evaluation
literature.
Evaluation Research by Program Category and Objective
The following sections summarize recent research and thinking about
performance measurement in the five ARC program evaluation categories:
Capital Access, Sectors, Entrepreneurship Education, Business Incubation, and
Technical Assistance and Training. Appendix A contains a more complete review
of the literature. As Table 3.2 indicates, each of the five program categories
seeks to achieve multiple goals. Yet, many of the efforts can be categorized
according to a predominant objective. For example, entrepreneurship education,
with its heavy emphasis on youth empowerment, is most concerned with the
objective of creating more entrepreneurs. Meanwhile, capital access programs,
especially those equity programs emphasized by the ARC, are largely focused
on creating more high-growth entrepreneurs.
1
8
T
a
b
l
e
3
.
2
:
I
n
t
e
r
s
e
c
t
i
o
n
o
f
P
r
o
g
r
a
m
C
a
t
e
g
o
r
i
e
s
a
n
d
P
r
o
g
r
a
m
m
a
t
i
c
G
o
a
l
s
C
a
p
i
t
a
l
A
c
c
e
s
s
E
n
t
r
e
p
r
e
n
e
u
r
i
a
l
E
d
u
c
a
t
i
o
n
I
n
c
u
b
a
t
o
r
s
S
e
c
t
o
r
s
T
e
c
h
n
i
c
a
l
A
s
s
i
s
t
a
n
c
e
a
n
d
T
r
a
i
n
i
n
g
C
o
m
m
u
n
i
t
y
C
a
p
a
c
i
t
y
M
o
r
e
e
n
t
r
e
p
r
e
n
e
u
r
s
i
n
p
i
p
e
l
i
n
e
X
X
X
M
o
r
e
e
n
t
r
e
p
r
e
n
e
u
r
s
s
t
a
y
i
n
g
X
X
X
X
B
e
t
t
e
r
i
n
f
o
r
m
e
d
e
n
t
r
e
p
r
e
n
e
u
r
s
X
X
X
X
B
e
t
t
e
r
s
k
i
l
l
e
d
e
n
t
r
e
p
r
e
n
e
u
r
s
X
X
X
X
M
o
r
e
j
o
b
c
r
e
a
t
i
n
g
b
u
s
i
n
e
s
s
e
s
X
X
X
X
G
r
e
a
t
e
r
b
u
s
i
n
e
s
s
p
r
o
d
u
c
t
i
v
i
t
y
X
X
X
X
X
19
Creating More Entrepreneurs
Within ARC’s program categories, entrepreneurship education is most concerned
with feeding the pipeline with new entrepreneurs – exposing youth to
entrepreneurship as a potential career path and to entrepreneurial ways of
thinking that can be applied to working for oneself or for someone else. An
ultimate or long-term goal is to have these new entrepreneurs create new
enterprises that remain in the region and produce jobs and wealth.
Entrepreneurship Education. Evaluations of entrepreneurship education
efforts vary across different levels of the educational system. For programs
targeting youth, a host of different evaluation methodologies have been
deployed. In general, most of these measures stress student performance and
outcomes as opposed to community outcomes. The programs operate on an
implicit assumption that by empowering and providing skills to young people
(especially at-risk youth), positive community outcomes will emerge over the
long-term.
Studies of youth entrepreneurship programs indicate that these efforts have a
strong impact on program participants. While some programs, including ARC-
funded efforts, provide training for primary school students, most existing
programs targeted middle and high school students. Studies sponsored by the
National Foundation for Teaching Entrepreneurship (NFTE) find that NFTE
program participants had more interest in attending college and had more
ambitious career aspirations.
36
A study of the EnterprisePrep curriculum used in
Philadelphia found that program participants had lower drop out rates and
improved performance in science, math and English. Studies of the Junior
Achievement curriculum identify similar positive outcomes in terms of youth
attitudes toward entrepreneurship.
37
Evaluations of entrepreneurship training at the college and graduate level place
more emphasis on outcomes in terms of new business formation. A detailed
study of the University of Arizona’s Berger Entrepreneurship program found that
program graduates were three times more likely to start a business than their
student counterparts. Program graduates also enjoyed higher average
incomes.
38
A whole host of other studies provide similar results.
39
36
Michael Nakkula, Expanded Explorations into the Psychology of Entrepreneurship: Findings
form the 2001-2002 Study of NFTE in two Boston Public High Schools, Working Paper, Harvard
University Graduate School of Education, 2003.
37
Junior Achievement, The Impact on Students of Participation in Junior Achievement: Selected
Cumulative and Longitudinal Findings, Monograph, January 26, 2004.
38
Alberta Charney and Gary Libecap, The Impact of Entrepreneurship Education: An Evaluation
of the Berger Entrepreneurship Program at the University of Arizona, 1985-1999, Final Report to
the Kauffman Center for Entrepreneurial Leadership, 2000.
39
Lena Lee, Entrepreneurship Education: A Compendium of Related Issues, Working Paper,
NUS Entrepreneurship Centre, National University of Singapore, 2005.
20
In general, entrepreneurship education programs are not expected to provide
direct community impacts in terms of new job or wealth creation. Instead, they
help produce intermediate outcomes, which may then translate into positive
economic development outcomes. Analysts contend that entrepreneurship
education efforts change attitudes toward entrepreneurship, increase awareness
of key business concepts, and build necessary skills for starting and operating a
business. These claims are generally confirmed in surveys of participants in
major programs. These attitudinal and skill changes may lead to new business
starts, which are then expected to generate positive community outcomes.
Based on this logic chain, most entrepreneurship education programs emphasize
program output and student performance measures. While most programs do
track new business starts by students, they place greater emphasis on more
short-term measures of student performance and attitudinal change. In contrast,
many technical assistance programs, such as trainings sponsored by Small
Business Development Center programs, are evaluated according to their
capacity to generate new business starts.
Creating Stronger Entrepreneurs
Three of the ARC’s program interventions – technical assistance and training,
business incubation, and sectors/networks – were primarily concerned with
supporting the mission of stronger entrepreneurs, i.e. to increase the survival rate
of local entrepreneurial ventures. These three initiatives operate according to
similar rationales. They seek to create more skilled and better informed
entrepreneurs through the provision of:
New information (via trainings and workshops)
Access to mentors and peer support (via networks)
Access to subsidized facilities or equipment (via incubators)
Access to potential new markets (via networks or training).
The research literature for each of the program categories emphasizes different
evaluation methodologies, but they all share some common characteristics.
First, they seek to assess general customer satisfaction with the provided
assistance. Second, they seek to assess whether this support led to improved
knowledge or skills for the entrepreneur. Finally, they seek to assess whether
this new knowledge has led to changes in behavior; specifically, has the
intervention led to improved performance by the company or its management
team?
Technical and Managerial Assistance. Since business incubators often
provide technical and managerial assistance as part of their program operations,
incubators and technical assistance often use similar measurement tools and
methodologies. Efforts to track and measure the effects of business assistance
programs have been underway for a long time. Many Federal programs, such as
21
the Small Business Development Center program and the Manufacturing
Extension Partnerships, have invested significant resources into evaluation
efforts.
Most technical assistance evaluations begin with calculations of traditional
economic development impacts such as job creation or the leveraging of outside
investments. As noted earlier, many state-level programs have become quite
sophisticated in their evaluation efforts. For example, Pennsylvania’s Ben
Franklin Technology Partners (BFTP) tracks its customers by company size and
sector; they also compare their client companies to state averages in terms of
average wages. This latter metric offers a useful measure of job quality. BFTP
tracks the public return on investment and impact on state gross product via
calculations of job creation, new investment, and newly generated tax
revenues.
40
BFTP is also one of the few programs to utilize a control group
methodology in its assessments. This costly but effective method compares the
performance of program clients to comparable firms who did not utilize BFTP
services. This comparison indicates that BFTP-supported firms employed three
more people in each year after the program investment.
In addition to basic measures of job creation and customer satisfaction, the
Oklahoma Center for the Advancement of Science and Technology (OCAST)
tracks outside research investments, Small Business Innovation Research
(SBIR) funds per capita (compared to national benchmarks), and company
financing by stage. OCAST’s partner, I2E, utilizes some other specialized
measures such as growth in number of companies positioned for financing,
creation of commercialization road maps for customers, and a host of activity
measures, such as creation of new partnerships, number of events, and so on.
Both OCAST and I2E also provide breakouts (via pie charts) of customers by
industrial sector and by region.
41
The Maine Technology Institute (MTI) utilizes university researchers to assess its
program operations and the impact of its grants.
42
The MTI’s performance has
been tracked using four categories of measures: economic impact, effects on
company finances, intellectual property development, and relationships. Within
these categories, several unique measures are used. These include sources of
material and service inputs (used to assess in-state purchasing) and
relationships. This latter measure tracks usage of other service providers, such
as SBDCs, and measures related to customer satisfaction and impact.
40
Nexus Associates, A Continuing Record of Achievement: The Economic Impact of the Ben
Franklin Technology Partners, Final Report, March 2003.
41
Oklahoma Center for the Advancement of Science and Technology, Impact Report 2006
(Oklahoma City, Oklahoma: OCAST, January 2006) 10 November 2006
.
42
Charles S. Colgan and Bruce Andrews, Evaluation of Maine Technology Institute Programs,
University of Southern Maine Center for Business and Economic Research, December 2004, 10
November 2006,
.
22
Business Incubation. Because business incubation has been a core economic
development strategy for several decades, both practitioners and analysts have
developed an array of metrics and tools for measuring the effectiveness of
business incubation programs. The National Business Incubation Association
(NBIA) has led many of these efforts and was an important ARC partner during
this initiative. In addition, ARC has funded incubator programs throughout its
existence and has funded several useful studies of the field.
43
Most studies of business incubation provide reviews of best practices that
typically cover the details of program management and facility operations.
44
Many of these reports also include suggestions for assessing the impact of
business incubation programs. Lichtenstein and Lyons place a heavy emphasis
on effective evaluations that capture both process and outcome measures.
45
Most analysts recommend that traditional economic development metrics, such
as job creation, be supplemented with other measures that capture unique
aspects of the business incubation process. For example, most business
incubators regularly track graduated companies, i.e., firms that have moved from
subsidized incubator space to owning or leasing space at market rates.
Customer satisfaction surveys are also frequently used. Business performance
measures – both during and after residence in the incubator – also offer useful
data on an incubator’s regional impacts. For example, a firm’s post-graduation
ability to continue growth, to access outside financing, and to enter new markets
are all important measures of community economic impacts from business
incubation.
Sectors/Networks. Economic developers’ thinking about sector/network
strategies has undergone an interesting evolution over the past twenty years.
Beginning in the 1980s and 1990s, a number of fledgling programs sought to
stimulate the development of sector-based networks in industries such as wood
products and manufacturing. Much of this initial work was based on successful
experiments in Denmark and Italy, with overseas lessons applied to US
experience. Several of the first such US-based programs were located in
Appalachia. The North Carolina Rural Economic Development Center supported
a networks initiative in Western North Carolina, and ACEnet sponsored a similar
effort in Appalachian Ohio.
43
For example, Greenwood Consulting Group, A Survey of Business Incubators in Appalachia
(Washington, DC: ARC, July 2005) 10 November 2006
.
44
See, for example, Louis G. Tornatzky et al., Incubating Technology Businesses: A National
Benchmarking Study (Athens, Ohio: National Business Incubation Association, 2003).
45
Gregg A. Lichtenstein and Thomas S. Lyons, Incubating New Enterprises: A Guide to
Successful Practice (Washington, DC: Aspen Institute, 1996).
23
These initial network building efforts remained limited to networks based on an
industrial cluster or sector.
46
More recently, policy makers have sought to
support the creation of broader entrepreneurial networks that include participants
from a variety of sectors and disciplines. ARC has invested in both types of
networks via the Entrepreneurship Initiative. For example, the Team
Pennsylvania Foundation sought to use ARC funds to stimulate the creation of
several regional entrepreneurship networks across the state. Other ARC
investments sought to build sector-based networks in ceramics (New York), arts
(Ohio), and aquaculture (Georgia).
While the operations of cluster-based and entrepreneurial networks may differ
slightly, both types are evaluated using a similar methodology. Evaluators
regularly seek hard quantitative data on the impact of networks, but qualitative
measures are also necessary as much of the network impact is qualitative.
Relationship building, trust, and increased knowledge are all important outcomes
of network activities. In general, research strongly indicates that more networked
firms tend to perform better than firms with weak networks and limited strategic
alliances.
47
Networks are typically evaluated using customer surveys, interviews, and case
studies.
48
Typical qualitative outcomes would be high rates of customer
satisfaction, better awareness of resources and networks, and more openness
toward collaboration. Quantitative measures examine the network’s ability to
help produce changes in business outcomes, such as entry into new markets,
revenue growth, job creation, and the ability to access outside financing.
As networking becomes a more important part of a typical economic
development portfolio, analysts are searching for new tools that can better
assess the role of business support providers as network builders. Social
network analysis seeks to map and evaluate the power of networking activities.
In the 1980s and 1990s, researchers sought to simply count the number of
outside alliances and assess their strength. Today, new software tools allow
users to map social networks and assess them according to their diversity,
strength, and influence. These maps can be used to diagnose network
46
For background, see Stuart A. Rosenfeld, “Networks and Clusters: The Yin and Yang of Rural
Development,” Proceedings of Federal Reserve Bank of Kansas City Conference on Exploring
Policy Options for a New Rural America, (September 2001):103-120.
47
An excellent review of this literature can be found in Luke Pittaway, Maxine Robertson, Kamal
Munir, and David Denyer, Networking and Innovation: A Systematic Review of the Evidence,
University of Lancaster Institute for Entrepreneurship and Enterprise Development, Working
Paper 016, 2004, 10 November 2006 . See also Christopher T. Street
and Ann-Frances Cameron, “External Relationships and Small Business: A Review of Small
Business Alliance and Network Research,” Journal of Small Business Management 45.2
(2007):239-266.
48
For example, see Peter Witt, “Entrepreneurs’ Networks and the Success of Start-Ups,”
Entrepreneurship and Regional Development 16 (2004):391-412; Philip Shapira, The Evaluation
of USNet: Overview of Methods, Results and Implications – Final Report, Georgia Tech Policy
Project on Industrial Modernization Working Paper 9805, 1998.
24
weaknesses and design interventions to strengthen the network and the quality
of company network ties.
49
Social network analysis has also been used as a
means to better describe and understand company and industry supply chains.
Creating More High-Growth Entrepreneurs
While all program interventions share the goal of creating high-growth
entrepreneurs, very few of the Entrepreneurship Initiative’s grantees focused
exclusively on this objective. However, a number of grant recipients in the
capital category did embrace a mission of supporting gazelle businesses.
Because its metrics rely on quantifiable financial measures, evaluations of capital
access programs tend to be much more rigorous than other types of
entrepreneurship-related policy interventions.
Capital. ARC’s capital projects tend to fall into two broad categories – support for
microenterprise initiatives and investments in more specialized types of financial
assistance, such as specialized loan funds or new sources of equity capital.
Many of these investment vehicles promote what they refer to as a “double
bottom line.” In other words, the funds seek to build strong businesses but also
promote other social goals, such as community development or environmental
sustainability. In addition, microenterprise programs measure progress along
two fronts – traditional business outcomes and measures that capture the
empowerment of individual clients. These latter metrics could include movement
off of welfare, increases in family income, and length of self-employment periods.
Since ARC’s Entrepreneurship Initiative investments have focused on regional
economic development, this literature review is concerned primarily with how to
measure the economic impacts of investments in capital programs. Since the EI
first began in 1997, capacity to measure these impacts has greatly improved. At
that time, policy makers had very limited experience in creating and operating
publicly-sponsored seed or equity capital programs. And, many early programs
were shut down due to political controversies.
50
The intervening decade has been one of great experimentation and innovation in
the development of new capital access initiatives and associated evaluation
tools. Since 1997, the New Markets Venture Capital Program, the Community
Development Financial Institutions program, and revisions to the Small Business
Investment Company (SBIC) program have all been put into place. In addition,
several new trade associations, including the Community Development Venture
Capital Association (CDVCA), the National Association of Seed and Venture
49
For example, see .
50
An example is Mississippi’s Magnolia Fund which operated for only 2 ½ years before being
closed due to misappropriation of funds. David L. Barkley, et al., Establishing Nontraditional
Venture Capital Institutions: Lessons Learned, Rural Equity Capital Initiative Study of
Nontraditional Venture Capital Institutions, RUPRI PB2001-11A, 2001,
.
25
Funds, and the Angel Capital Association, have all been established to help
professionalize the field.
Meanwhile, more established disciplines, like microenterprise and community
development corporations, have become much more rigorous in strengthening
professional development programs and creating common performance
measurement systems. In the microenterprise field, the Aspen Institute’s FIELD
(Microenterprise Fund for Innovation, Effectiveness, Learning and Dissemination)
program has played a critical role in collecting, developing and disseminating
new program ideas and new performance measures and tools.
51
Its MicroTest
program provides a useful framework for assessing both program performance
and client outcomes. Similar comprehensive efforts are underway at leading
organizations in the field. These include the CDFI Data Project, the Opportunity
Finance Network’s CDFI Assessment and Rating System (CARS), and the
CDVCA’s Return on Investment Project.
52
All of these efforts, and other outside analysis, reach a similar conclusion –
publicly-sponsored investment programs need to be managed and measured just
like private investments. While programs may pursue multiple goals, the
programs must be managed and assessed on their capacity to make good
business decisions. As a 2000 National Governor’s Association guide put it,
“The best programs are not afraid to make money.”
53
This same study noted: “In
the best cases, state leaders take the initiative in getting programs launched and
setting long-term direction. They rely on experienced, private-sector managers to
make the day-to-day investment decisions.”
54
As seed and equity capital programs have moved in this direction, issues of
performance measurement have become more straightforward. At the most
basic level, most funds should be expected to produce a reasonable rate of
return. A reasonable rate will differ depending on the types of companies in a
fund’s portfolio. For example, institutional venture capital funds have averaged a
20.3% annual return over the past ten years, but only a 1% return over the past
five years. A 2002 study of venture capital rates of return in five countries
identified average rates of return that fell anywhere between 26% and 45%.
55
Meanwhile, the CDVCA estimates that its members have enjoyed an average
51
An excellent review of effects of microenterprise programs can be found in Signe-Mary
McKernan and Henry Chen, Small Business and Microenterprise as an Opportunity and Asset-
Building Strategy, Urban Institute Issue Brief No. 3, June 2005.
52
Community Development Venture Capital Association, Measuring Impacts Toolkit (New York:
Community Development Venture Capital Alliance, 2005).
53
Robert Heard and John Sibert, Growing New Businesses with Seed and Venture Capital: State
Experiences and Options (Washington, DC: National Governors Association, 2000) 18.
54
Heard and Sibert, 17.
55
Sophie Manigart, et al, “Determinants of Required Return in Venture Capital Investments: A
Five Country Study,” Journal of Business Venturing 16.6 (July 2002):291-312.
26
annual return of 15.5% over a twenty-five year period.
56
Programs that invest in
microenterprises or slower growth businesses should be expected to post rates
of return much lower than these benchmarks.
In addition to rates of return, funds can also be assessed in terms of program
outputs and economic development outcomes. Program outputs refer directly to
the activities of the fund or its related programs. They are activity measures that
track data such as the number and amount of loans, average loan size or
number/amount of financing provided to certain target customer sets.
Outcome measures seek to assess a fund’s business and community impacts.
In this case, most analysts recognize that many traditional economic
development metrics do a good job of measuring community outcomes. Thus,
most studies continue to recommend tracking job creation and retention,
business performance of portfolio companies, and the leveraging of outside
investments.
IMPLICATIONS FOR PERFORMANCE MEASUREMENT
The conceptual framework presented above seeks to portray entrepreneurial
development as an array of programs that serve entrepreneurs at various points
in the lifecycle of their business. This “pipeline” model of entrepreneurial
development, first described by Lichtenstein and Lyons,
57
recommends multiple
policy interventions that help achieve the three broad purposes of creating more
entrepreneurs, stronger entrepreneurs, and more high-growth entrepreneurs.
In practice, few program managers have the scope to manage programs that
cover the whole pipeline of entrepreneurial development. Instead, they typically
manage a single program or a single type of policy intervention, such as training
or business financing. This restricted span of control over business outcomes
makes it difficult for individual programs to introduce more sophisticated tools for
measuring program performance.
In the field, entrepreneurship program managers are beginning to consider
alternative performance measurement systems, but they still feel pressured by
funders and elected officials to utilize traditional measures of job and firm
creation outcomes.
58
By using traditional economic development metrics to
56
Amy Simpkins, “Community Development Venture Capital: Producing Results for
Entrepreneurs, Investors, and Communities,” Bridges Summer 2006, 10 November 2006
.
57
Gregg A. Lichtenstein and Thomas S. Lyons, “The Entrepreneurial Development System:
Transforming Business Talent and Community Economies,” Economic Development Quarterly
15.1 (2001):3-20.
58
Laura Czohara and Julia Melkers, Performance Measurement in State Economic Development
Agencies: Lessons and Next Steps for GDITT, Georgia State University Fiscal Research Center
Report No. 92, February 2004. See also, Erik R. Pages and Kenneth A. Poole,
“Entrepreneurship Promotion as an Economic Development Strategy: Next Steps in
27
assess entrepreneurial and innovation-based development, program managers
are employing inappropriate performance measures. Building businesses takes
patience and resilience. It is unrealistic to expect quick results in terms of
traditional economic development outcomes. Program managers need a more
thoughtful approach to tracking the performance of their efforts. Moreover, new
measures for entrepreneurial development need to be devised that recognize a
stream of benefits over an extended period of time. Measuring entrepreneurial
development using annual job creation impacts alone is like measuring the
success of a loan program solely by the ability of the borrower to repay on a
monthly basis. Short-term job creation is simply not the purpose of these
programs.
In their quest to “tell a better story,” program managers are considering a host of
other metrics, which tend to fall into the following categories:
Activity/Output Measures (e.g., number of customers served)
Customer Satisfaction Surveys
Input Measures (e.g., increase in budget)
Outcome Measures (e.g., increase in business starts)
Cost Efficiency (e.g., return on investment).
While some programs use a full range of measures, most economic developers,
including ARC grant recipients, use a more limited menu of metrics that is
generally limited to job creation and retention and the leveraging of outside
investments. These limited metrics can provide a much skewed picture of the
impact of entrepreneurial development efforts.
Economic development programs have responded to this challenge in a number
of ways. The metrics used by Small Business Development Centers (SBDCs)
offer one model. For example, the North Carolina Small Business and
Technology Development Centers track outside financing (loans, equity, and
SBIR/Small Business Technology Transfer Program funds), state and federal
contract awards, customer satisfaction, job creation, firm sales growth, and
incremental taxes generated.
59
Florida combines capital formation, business
start-ups, jobs created/retained, sales growth, and contract awards to calculate
the return on investment (ROI) for its SBDC system.
60
A slightly more
sophisticated set of metrics is used by various statewide technology development
organizations such as Pennsylvania’s Ben Franklin Partners, the Oklahoma
Center for the Advancement of Science and Technology, and the Maine
Technology Institute.
Institutionalizing the Field,” Applied Research in Economic Development 3.2 (December
2006):.10-27.
59
North Carolina Small Business and Technology Development Center, 2005 Annual Report,
.
60
See .
28
Useful benchmarks for metrics can also be found in the work of various trade
associations working in the fields of microenterprise and community
development. These groups are seeking to develop industry-wide metrics that
can help improve management practices and provide industry benchmarks for
effective practice.
Local groups are also experimenting with new methods. For example, Maine’s
Coastal Enterprises, Inc. has developed a very rigorous and comprehensive set
of program measures.
61
Its Social Information System combines a host of
measures that provide internal feedback to management and employees, permit
assessment of program outcomes, and also generate data and case studies that
can be communicated to an outside audience.
These varied initiatives seek similar types of information on client companies.
They ask for information on firm growth, performance, and local economic
impact. In keeping with their social missions, the organizations also tend to track
information on employee benefits, wages, and community and environmental
impacts.
At the federal level, the US Department of Agriculture’s Rural Business-
Cooperative Services (RBS) is implementing a Socio-Economic Benefits
Assessment System (SEBAS) developed at the University of Missouri-
Columbia.
62
SEBAS provides a means of evaluating performance and
effectiveness of RBS’ loan and grant programs by measuring the economic and
social impacts that these have on rural community environments. Using a multi-
regional social accounting matrix model, SEBAS is able to measure both direct
and indirect effects of the loans and grants, such as business sales, income,
indirect business taxes, employment, household income, public revenues, and
distribution of household income and occupations.
Finally, many organizations are seeking new ways to measure “innovation
impacts.” This work remains relevant to the existing literature on
entrepreneurship as many entrepreneurial development programs seek to
stimulate innovation-related outcomes, such as improved productivity rates,
wider diffusion of new technologies, and improvements in human capital.
63
The
US Department of Commerce has created a new “Measuring Innovation in the
21
st
Century Economy Advisory Committee.” Commerce’s Manufacturing
Extension Partnerships and the Advanced Technology Program have probably
faced the most rigorous scrutiny of any publicly-funded technology program in
the country. The National Institutes of Standards and Technology (NIST) have a
61
Coastal Enterprises, Inc., Measuring Impact in Practice (Wiscasset, ME: Coastal Enterprises
Inc., February 2006).
62
Robinson, Dennis and Zuoming Liu, A User Guide to the Socio-Economic Benefits Assessment
System: A Rural Business-Cooperative Services Assessment Tool for Economic Development.
Community Policy Analysis Center, University of Missouri-Columbia, December 2004.
63
B.K. Atrostic, “Measuring U.S. Innovative Activity,” US Census Bureau Center for Economic
Studies Working Paper (07-11), March 2007.
29
huge library of resources on this work.
64
Studies of the MEP have generally
used industry surveys to assess whether client firms have introduced new
processes (such as Total Quality Management or a reconfigured plant layout) as
a result of MEP’s technical assistance.
65
Overseas, the European Union, the OECD, and many national and regional
programs are developing interesting new ways to measure progress in innovation
policy. They have also made major efforts to go beyond studies and to get
practitioners to use these tools in the field. For example, the OECD has
produced the very detailed Oslo Manual for collecting and interpreting innovation
data. The European Union has its own PAXIS Manual that profiles hundreds of
effective measurement tools and practices. Many national governments are also
doing good work in this area. Britain’s Department of Trade and Industry has
recently published a useful study of UK innovation indicators.
66
These efforts all
share a commitment to regular comprehensive performance measurements that
capture both program outputs as well as community outcomes.
These efforts generally propose the use of detailed company surveys to capture
data on a wider range of variables related to innovation. In addition to traditional
measures such as new patents or licenses, the new survey tools ask questions
about education backgrounds of new hires and existing personnel, development
of new R&D projects, joint ventures, training expenditures, and the deployment of
“high-impact” human resource practices (e.g., equity sharing, team building).
The UK study captures some interesting innovation information via firm
surveys.
67
These data are organized around three categories, with several sub-
measures under each category:
Product-Orientated Effects: increased range of goods or services; entry
into new markets; improved quality of goods and services
Process-Orientated Effects: improved flexibility of production or service
provision; increased capacity for production or service provision; reduced
unit costs
Other Effects: reduced environmental impact or improved health/safety;
met regulatory requirements; improved value added.
64
These reports are available at .
65
See, for example, Eric Oldsman, “Do Manufacturing Extension Programs Matter?” Research
Policy 25.2 (March 1996):215-232.
66
European Commission, Directorate General, Enterprise and Industry, The PAXIS Manual for
Innovation Policy Makers and Practitioners (Brussels: European Commission, 2006);
Organization for Economic Cooperation and Development, Oslo Manual: Guidelines for Collecting
and Interpreting Innovation Data Third Edition (Paris: Organization for Economic Cooperation and
Development, 2005); United Kingdom Department of Trade and Industry, Innovation in the UK:
Indicators and Insights, DTI Occasional Paper No. 6, July 2006.
67
United Kingdom Department of Trade and Industry (2006).
30
These surveys also ask companies to comment on innovation constraints they
face in their region or industry. These data help program managers anticipate
future needs or service requirements.
Drawing on this review of literature and insights into performance measurement
gained, the evaluation team developed a metrics framework to guide the
evaluation, particularly the data collection for the sample of ARC projects.
Chapter 4 describes the metrics framework and the overall approach to the
evaluation.
31
CHAPTER 4
APPROACH TO THE EVALUATION
In designing the approach to this evaluation, the team was conscious of the
charge articulated by ARC in the Request for Proposals for the evaluation.
ARC was seeking a “policy impact and program evaluation”
68
of the
Entrepreneurship Initiative that would include:
An examination of the project outcomes of a sample of projects closed
since 1997
An assessment of wider policy impacts informed by input from
stakeholders within the region.
Through its Entrepreneurship Initiative, the ARC has invested in a diverse set of
projects in a diverse set of communities. Thus, a “one-size-fits-all” approach
would not produce an effective and comprehensive evaluation. The evaluation
team’s approach to this project was to combine a detailed and rigorous review of
a sample set of projects with extensive interviews of key players on the project
teams and in the targeted communities. This approach provided us with a strong
set of collective program metrics as well as rich detailed case-study-like
information on key projects and their outcomes.
The ideal evaluation would measure project outcomes relative to the sample
programs’ ultimate goals.
69
The ARC evaluation is complicated by the fact that
individual projects (those receiving ARC investments) had a set of specific goals
each was trying to achieve, i.e., “local” impacts, such as increasing access to
equity capital or incorporating entrepreneurship education into high school
classrooms. In addition, through investments made in the totality of EI projects,
ARC was trying to provide communities with the tools they needed to support
homegrown entrepreneurs because they “play an important role in creating self-
sustaining local economies and improving the quality of life in Appalachia”
70
– a
broader set of regional impacts. While “local” and regional impacts were
complementary in most cases, individual projects were evaluated based on their
ability to achieve locally articulated goals and not these broader regional impacts.
To distinguish between these two levels of impact, the evaluation team chose to
couple an assessment of sample projects’ outcomes, as measured by data
reported by project leaders, with identification of wider policy impacts as reported
68
Request for Proposals for A Program Evaluation of ARC’s Entrepreneurship Initiative,
Appalachian Regional Commission, July 28, 2006.
69
David J. Storey, Six Steps to Heaven: Evaluating the Impact of Public Policies to Support Small
Businesses in Developed Economies, Working paper No. 59, Small and Medium Sized Enterprise
Centre, Warwick Business School, University of Warwick, September 1998, 3 December 2007
.
70
Appalachian Regional Commission, Entrepreneurship Initiative, 9 August 2006
.
32
by key informants – both project leaders and regional stakeholders. A list of non-
project stakeholders is included as Appendix B.
Program outcomes for the EI investments can be measured in two ways. One is
to compare the levels of outcome measures before and after the ARC
investment. To be accurate, one must be sure that the metric or outcome
measure would not have changed without the ARC-funded program (i.e., the “but
for” criterion). Alternatively, one could conduct a with and without comparison,
estimating what the outcome would be without ARC investment and comparing
this measure to the outcome with ARC investment. To be accurate using this
strategy, one would need to employ a control group of communities, identical in
every way to the communities that housed an ARC-funded program. While
employing a control group may make sense when looking at firm-level impacts of
public policy, as suggested by Storey, it is more difficult to consider identifying
control groups for public policies aimed at bringing about changes in firms,
students, educators and even communities. As a result, the with and without
approach was deemed impractical for this evaluation as it would require
identifying at least 88 “control” places – some individual communities, some
multi-county regions, some multi-state regions – and collecting comparable data
through interviews and secondary data.
Given the scope of the evaluation as articulated by ARC, this latter approach was
not used in this evaluation. While neither approach is without criticism (see
caveats at the end of this chapter), the evaluation team chose the before and
after strategy and relied on those administering the programs and respondents to
stakeholder interviews to indicate if the “but for” criterion was satisfied.
RESEARCH QUESTIONS
A number of research questions guided this evaluation effort.
Have sample projects achieved their stated project objectives?
How does the performance of ARC projects compare to the performance
of similar types of projects in other regions or countries?
How well do existing performance metrics capture the impact of ARC
projects and what additional metrics would improve the usefulness and
integrity of evaluation results?
What broader policy impacts are associated with ARC project
investments?
What innovative practices or lessons learned have relevance for other
national and international economic development efforts?
ARC’s Entrepreneurship Initiative is unique in that investments have been made
in a diverse project portfolio, varying widely by geography, program type (i.e.,
access to capital, technical assistance and training, incubators, entrepreneurship
education, and sectors), type of lead institution and identified output and outcome
33
measures. As such, no single performance measurement or metric would
adequately capture the impact of ARC entrepreneurship investments on the
region. For example, the performance measures associated with a youth
entrepreneurship education project are likely to vary significantly from those
associated with a community development venture capital project. The evaluation
methodology recognized this uniqueness and created a rigorous and broad set of
project metrics that would demonstrate the impact of a wide range of project
types on the region.
METRICS FRAMEWORK
The review of the literature on the state of the art in evaluating entrepreneurial
development programs yields one important conclusion: the field has a long way
to go in terms of creating rigorous, compelling, and effective techniques and
strategies for evaluating programs and communicating their effectiveness to
policy makers and community stakeholders. Indeed, much of the literature
reflects what Storey would describe as “monitoring” as opposed to rigorous
evaluation of performance as compared to the objectives established for a
program or investment.
71
Drawing on the lessons from this review and based on
the conceptual framework developed to guide the evaluation process (described
in Chapter 3), performance measures were defined for each of the program
categories that, as far as is practical from the available data, reflect the
appropriate programmatic goals. The evaluation team developed a list of metrics
that captures key outputs and outcomes for each of the five program categories.
This framework is depicted in Table 4.1.
The development of this metrics framework was guided by the literature review
and a critical discussion of the need to link outcomes to program goals. For
example, while the literature on entrepreneurship education offers a range of
metrics related to student outcomes (e.g., higher test scores), the evaluation
team chose to focus on those metrics that relate directly to the programmatic
goals of the ARC Entrepreneurship Initiative – in the case of entrepreneurship
education, the goal is to create more entrepreneurs in the pipeline.
These metrics allow researchers to obtain a good picture of the progress of the
various projects funded through the ARC’s Entrepreneurship Initiative. They
allow an understanding of the impacts of each program type, and also help
identify exemplary practices and programs. They do not yet give a good picture
of the overall effects of varied entrepreneurship initiatives across a specific
region. Achieving this objective requires that researchers assess programs
operated by numerous different organizations (many outside of ARC’s purview)
across all types of policy interventions along the business life cycle. This
important effort, which falls outside scope of this evaluation, presents an
interesting challenge for future research.
71
Storey (1998):12.
34
TABLE 4.1. PROPOSED METRICS FRAMEWORK
TYPE OF PROJECT OUTPUT INDICATORS OUTCOME INDICATORS
Capital Access Number of loans/year
Amount ($)/year
Number of funds created
Fund size ($)
Average loan size
Percent sectoral distribution
of loans ($)
Amount ($) funds leveraged
(public, private or other)
Number of jobs (FTEs)
created/retained
Percent of funded firms still in
business
Annual income and benefits/job or
average wage/job
Sectors Number of participants in
networking meetings
Number of members
(change over time)
Increase in inter-firm
collaborations
Number of partnerships created
Amount ($) of increased sales
from network participation
Number of jobs (FTEs)
created/retained
Change in total sector sales
Number of business starts in
targeted sector
Incubators
Number of current clients
Number of clients served
Number of graduated firms
Number of clients still in
business
Amount ($) leveraged by
incubator (other
public/private money)
% businesses retained in service
area
# of jobs (FTEs) created/retained
while in incubator
# of jobs (FTEs) created/retained
post-graduation
Amount ($) of capital raised by
tenants
Entrepreneurship
Education
Number of participants
enrolled in the program
% of local schools offering
(pre and post investment)
% of participants completing
the program
Increase in awareness of
business concepts (pre vs. post)
Increase in number of participants
considering business creation as
a career option (pre vs. post)
Change in student performance
before and after program
Number of students starting
businesses
Number of students that stay
within the service area
Technical
Assistance
And Training
Number of business starts
Number of business
expansions
Number of clients
Number of clients still in business
Number of jobs (FTEs)
created/retained
Private $ raised by client firm
Annual income and benefits/job or
average wage/job
Jobs (FTE) = wage earners and proprietors
35
While this broad set of performance metrics provided a useful framework for the
evaluation, a “best in class” metrics system as proposed later in this report would
require refinement of this list down to those outcome measures that best capture
the impacts of entrepreneurship development investments across a range of
project types. Recommendations in Chapter 9 include the identification of a set of
outcome measures that could result in an operationalized “best in class” metrics
system to guide future investments by ARC.
DATA COLLECTION
To gather the data identified in the metrics framework, the evaluation team used
a four-part approach:
1. Data for each sample project were gathered from the ARC project file –
the program category, funding level, goals (or need addressed), and how
the program was implemented.
2. Additional data on project performance were gathered through phone
interviews with project directors – outcomes produced (both quantitative
and qualitative), value attributed to the project, success in achieving
objectives.
3. Data on broader capacity and policy impacts were gathered through
phone interviews with both project and non-project stakeholders – policy
impacts in the region, other qualitative and quantitative impacts on
capacity in the region.
4. Data on place-based and broad policy impacts were gathered through
selected site visits in geographic areas where investments in a number of
program categories had been made, determined after the previous parts
of the data collection process were completed.
To facilitate the collection of data in part two above, a draft protocol for the follow-
up phone interviews was developed (see Appendix C).
DESCRIPTION OF EVALUATION TASKS
A number of specific evaluation tasks were identified and completed. To provide
context for the evaluation and to guide the development of the metrics
framework, the team completed a literature review of entrepreneurship
development project evaluations conducted by organizations such as
Organisation for Economic Co-operation and Development (OECD), Small
Business Administration (SBA), Economic Development Administration (EDA),
US Department of Agriculture (USDA), and National Institute of Standards and
Technology (NIST) – Manufacturing Extension Program (MEP) and Advanced
Technology Program (ATP), as well as evaluations by private organizations such
36
as Community Development Venture Capital Alliance (CDVCA), National
Business Incubation Association (NBIA) and other local and state organizations
(see Chapter 3 and Appendix A). In addition to guiding development of the
metrics framework, this review helped the evaluation team determine whether a
comparative analysis of ARC performance relative to other similar types of
initiatives was feasible.
Members of the evaluation team conducted phone interviews with non-project
stakeholders to identify the broader impacts of ARC Entrepreneurship Initiative
projects. These individuals tended to be national or regional experts on
economic development and the specific issues and challenges facing the
Appalachian region or specific states and localities within the ARC service area
(see Appendix B for a list of stakeholders interviewed and Appendix D for the
protocol used.) Interviews were also conducted with state program managers to
develop an understanding of how the EI was implemented in each state and with
other key leaders who had deep experience in the region. Recurring themes from
these key informant interviews were identified and reported as qualitative impacts
of ARC program investments.
The most significant evaluation task was the team’s review of project files
provided by ARC for a sample of 114 projects to understand the purpose, goals,
objectives, and identified outputs and outcomes. ARC’s performance
measurement system tracked a number of key performance metrics across
projects that were relevant to this evaluation:
Businesses served
Jobs created
Jobs retained
Project participants
New businesses created
Private investment leveraged.
While this common set of metrics facilitates assessment of the impacts of ARC’s
project portfolio, the evaluation team also used the metrics framework developed
from the literature review to collect additional data on the sample projects. The
team conducted phone interviews with key staff of each project’s lead institution
to review and/or collect information and data on both the performance metrics
reported to ARC and the broader set of performance metrics identified by the
project team in consultation with the advisory group. It is important to note that
few project leaders were able to report data for this broad set of metrics. Since
these metrics were not necessarily identified as outcomes by the grantees, it was
not expected that they would be able to report on all of these metrics. However,
the evaluation team wanted to be able to capture as broad a set of performance
metrics as possible during these interviews, so the metrics framework was used
as a guide.
37
Using data obtained from final project reports submitted to ARC and data
collected during the phone interviews, the team prepared a descriptive statistical
analysis of performance metrics. This analysis allowed the team to evaluate how
sample projects had achieved their stated objectives and how additional metrics
contribute to understanding the outcomes and impacts of project investment.
These interviews provided useful data on project outcomes. As part of the effort
to gain a sense of impact beyond individual projects, the project team also
conducted site visits to four locations with the region. These site visits were used
to conduct personal interviews and/or focus groups with project teams and other
community stakeholders. The insights gained through these site visits helped to
confirm what the team learned through other interviews and to provide case
examples to illustrate specific lessons learned associated with this evaluation.
A final task was to conduct a meta-analysis of the outcomes and impacts
associated with ARC projects as compared to any benchmarks identified through
the literature review. The purpose of this analysis was to assess the impacts of
ARC programs relative to similar investments in economic development
activities. As described in Chapter 3, the literature review identified few other
evaluations that sought to measure the outcomes of entrepreneurship
development investments like ARC – initiatives that are designed to change the
culture and economic development direction of a region. Assessments were most
often completed for particular programs, such as SBDCs, with a focus on
measuring impacts at the individual entrepreneur (customer) level. Most of these
assessments have measured outputs from program activities, e.g., number of
clients served, dollars invested, as opposed to outcomes on either individual
businesses or the overall economy.
It was difficult to use information from these previous assessments for a
comparative analysis of the ARC EI since ARC investments were strategically
made to demonstrate the potential for entrepreneurship development and to
change attitudes about economic development within communities and the
broader region. Specific projects were not designed simply to create jobs, but
had a broader set of goals including, for example, to:
Expand use of e-commerce
Create an angel network
Double the number of high schools teaching entrepreneurship
Train teachers to use entrepreneurship curriculum
Prepare 100 new business plans as part of a competition
Complete a business incubator feasibility study and strategic plan
Attract 10,000 visitors in first year of a heritage tourism development.
Even detailed metrics for these types of broader goals tell only part of the story. It
is not possible to talk about entrepreneurship development by reporting on any
single metric. However, in lieu of a broader meta analysis, it was possible to
calculate for the sample projects and for the entire portfolio an estimate of public
cost per job created that could be compared to estimates for other similar
programs. These comparisons are provided in Chapter 6.
38
As a result of these challenges, the evaluation team chose to recognize the
embryonic nature of entrepreneurship development evaluation research and to
use what had been learned through both the literature review and in-depth
assessment of sample projects to suggest what performance metrics would best
capture the essence of initiatives like ARC’s EI – a “best in class” metrics system.
This system is discussed in detail in Chapter 9.
SAMPLE SELECTION PROCESS
ARC made 448 grants to entrepreneurship-related projects from 1997 through
2005. This evaluation focused on projects whose commitment from ARC had
been completed, i.e., “closed” projects between 1997 and 2005; a total of 354
grants were closed at the time of the evaluation. However, since a number of
grants were for follow-on investments, the evaluation team further narrowed the
universe of eligible projects to a set of 229 unique, closed projects. While most
projects were designed to address the specific issues or needs within a
community, region or state, some of these investments were in region-wide
projects, primarily designed to raise the level of awareness of entrepreneurship
generally or to provide an opportunity to explore a specific issue with applicability
across the region, such as the role of business incubators or issues related to
access to capital for entrepreneurs.
This group of 229 projects represented investments in all types of projects (Table
4.2) and in all states in the region (Table 4.4). It was from this universe of
projects that the evaluation sample was drawn. All grants related to a specific
project were reviewed as part of the sample analysis.
Each of the closed, unique projects was assigned to one of five categories by
reviewing the title of the project and the description of the project contained in the
September 2003 ARC publication, Entrepreneurship Initiative: Program Summary
and Approved Projects. Some projects in the database were not in this
publication, notably those coded as “Commission” rather than coded by a specific
state. These Commission projects were coded according to the titles where
possible. The five categories used were:
Capital Access – any project where loans, grants or equity investments
were made in companies
Entrepreneurship Education– all projects for youth
Incubator – any project to study the feasibility of, plan, or operate an
incubator; virtual incubators included under technical assistance
Sectors – any project whose aim was to support entrepreneurs in a single
sector or type of business; excluding incubator projects
Technical assistance and training – any project where assistance was
given directly to individual entrepreneurs.
39
Since community capacity was a cross-cutting theme, the evaluation team
determined that all projects in the sample would be evaluated for community
capacity building. A total of 28 projects were not coded because they did not
have a description, the title was vague, or the project was for a conference or
similar activity. The distribution of closed, unique projects is shown in Table 4.2.
Based on this distribution, the team randomly selected a sample of 114 projects.
Then, three projects were discarded and three were added to make the state
distribution more even.
Table 4.2. Distribution by Program Category – Universe of Unique Projects
Category Number Percent
Capital Access 18 7.9
Entrepreneurship Education 42 18.3
Incubators 30 13.1
Sectors 40 17.5
Technical Assistance and Training 71 31.0
Other/Not Coded 28 12.2
Total 229 100
The objective was to choose a representative sample of projects, defined as
having the same distribution of projects by category and by state as the universe
of unique projects. In addition, the team sought to have a sample that was
representative in terms of size of investment by ARC. The team concluded that
the sample was representative, but not strictly random, as described below.
Table 4.3 shows the number and percent of projects in each program category in
the original sample of 114 and in the final sample of 88.
72
Based on the
distribution of the sample projects by program category, the final evaluation
sample appears to be representative of ARC’s EI projects as a whole.
The geographic distribution of the sample projects as compared to the
distribution by state for the universe of unique projects is shown in Table 4.4 and
in Figure 4.1.
73
Note that three states, Georgia, North Carolina and South
Carolina, appear in the sample at a rate slightly higher than their actual number
72
Although 114 projects were included in the original sample, completed interviews were
obtained for 88 projects. Project interviews were not completed for a variety of reasons including
loss of institutional memory of the project because project leaders had left the organization,
inability to schedule interviews after repeated attempts, and the closure of lead organizations.
73
The map in Figure 4.1 indicates that some grants went to organizations outside the region. This
apparent anomaly occurs because, in some cases, the lead organization was located outside the
ARC region, although the project was implemented within the region.
40
of projects. These three states had fewer total projects and, in order to include
more than one project from each of these states in the sample, the team chose to
over sample to insure more representative findings from these states. The
sample appears to be representative of the geographic diversity of EI projects.
In terms of ARC investment, the sample again appears to be representative of
the project universe. Average ARC investment per project was $126,387, while
the average amount of ARC investment per project in the sample was $145,997.
The range of ARC investments in the universe was $2,000 to $2.2 million; within
the sample, the range was $3,500 to $2.2 million. The total investment in the
ARC projects was $42,971,688; the total in the sample was $12,847,733.
Table 4.3. Distribution by Program Category – Original and Final Sample of
Projects
CATEGORY ORIGINAL ORIGINAL
%
FINAL FINAL
%
Capital Access 10 8.8 8 9.1
Entrepreneurship
Education
23 20.2 17 19.3
Incubators 17 14.9 12 13.6
Sectors 23 20.2 17 19.3
Technical assistance and
Training
41 36.0 34 38.6
Other/not coded --- --- --- ---
Total 114 100% 88 100%
CAVEATS TO THE EVALUATION APPROACH
Any evaluation must address the potential caveats associated with the proposed
methodology. While it is important to acknowledge and, to the extent possible,
mitigate these shortcomings, the methodology used in this project was as
rigorous as possible given the diverse set of individual projects, each with its own
set of self-defined performance measures that characterize the ARC
Entrepreneurship Initiative.
The first caveat to the selected methodology was its reliance on self-reporting of
outcomes and other performance metrics. In many circumstances, this would be
a severe constraint on the integrity and objectivity of the data and conclusions. In
this case, however, the grantee organizations were required to submit final
reports as part of their contractual agreement with ARC and these reports could
be, and often were, subject to audit. In addition, ARC’s Regional Planning and
Research division made quasi-random validation field visits to project leaders.
41
The goal of these visits was to validate that the activities proposed under the
grant were actually taking place. The evaluation team took the view that the
project reports could legitimately be regarded as binding and accurate depictions
of the impacts associated with each project. It was beyond the scope of this
evaluation to independently verify the data reported by individual projects as part
of their reporting to ARC or as part of the data collection process associated with
this evaluation.
To develop insights into the broader policy impacts associated with ARC
investments, we chose to rely on interviews with key stakeholders, an adaptation
of the key informant research approach. Since these broader impacts related to
such things as changes in attitudes toward entrepreneurship and economic
development in the region, renewed hope for the future, and the elevation of a
new set of leaders, the best means of gaining insight into these changes in
regional or community capacity was to question those who had deep
appreciation for the culture of Appalachia, extensive experience working in
economic development in the region, and expertise in particular project areas in
which ARC invested. Recognizing the inherent bias in relying on interviews with
individuals to assess overall change in a region, the evaluation team chose to
identify a broad group of stakeholders and to report recurring themes and
observations that were widely held within this group.
Another caveat relates to what might be called a “bias toward success.” If the
universe of closed projects does not include both “successful” and “failed”
projects, the evaluation results will be biased toward success, i.e., the evaluation
will not capture the insights and outcomes (or lack thereof) associated with
projects that were not successful. Indeed, failed projects may provide insights
that are helpful in addressing some of the research questions articulated in this
proposal. Based on the outcome of the project interviews, it was clear that the
sample included projects that were successful and sustainable, as well as those
that were not.
The final caveat relates to the rigor of statistical analysis permitted as part of this
evaluation. The small sample size and the diverse types of projects included in
the sample prevent the use of sophisticated statistical models or methods.
However, the descriptive statistical methods used allowed us to address the
critical research questions outlined above and to provide recommendations for
ARC’s consideration in guiding the entrepreneurship investments in the future.
4
2
T
a
b
l
e
4
.
4
:
D
i
s
t
r
i
b
u
t
i
o
n
o
f
A
R
C
P
r
o
j
e
c
t
s
b
y
S
t
a
t
e
–
U
n
i
v
e
r
s
e
o
f
U
n
i
q
u
e
P
r
o
j
e
c
t
s
,
O
r
i
g
i
n
a
l
S
a
m
p
l
e
a
n
d
F
i
n
a
l
S
a
m
p
l
e
*
#
U
N
I
V
E
R
S
E
#
O
R
I
G
I
N
A
L
S
A
M
P
L
E
#
F
I
N
A
L
S
A
M
P
L
E
%
U
N
I
V
E
R
S
E
%
O
R
I
G
I
N
A
L
S
A
M
P
L
E
%
F
I
N
A
L
S
A
M
P
L
E
A
l
a
b
a
m
a
2
0
1
1
8
1
0
1
0
9
G
e
o
r
g
i
a
7
5
4
3
4
4
K
e
n
t
u
c
k
y
1
4
8
6
7
7
7
M
a
r
y
l
a
n
d
1
1
5
5
5
4
6
M
i
s
s
i
s
s
i
p
p
i
1
4
8
5
7
7
6
N
e
w
Y
o
r
k
1
6
9
7
8
8
8
N
o
r
t
h
C
a
r
o
l
i
n
a
3
3
3
1
3
3
O
h
i
o
2
6
1
4
1
2
1
3
1
2
1
4
P
e
n
n
s
y
l
v
a
n
i
a
1
2
7
6
6
6
7
S
o
u
t
h
C
a
r
o
l
i
n
a
3
3
3
1
3
3
T
e
n
n
e
s
s
e
e
1
0
6
4
5
5
4
V
i
r
g
i
n
i
a
3
8
2
1
1
4
1
9
1
8
1
6
W
e
s
t
V
i
r
g
i
n
i
a
2
1
1
1
1
0
1
0
1
0
1
1
A
R
C
C
o
m
m
i
s
s
i
o
n
6
3
1
3
3
1
2
0
1
1
1
4
8
8
1
0
0
1
0
0
1
0
0
*
A
t
o
t
a
l
o
f
2
8
p
r
o
j
e
c
t
s
w
e
r
e
n
o
t
c
o
d
e
d
a
s
d
e
s
c
r
i
b
e
d
i
n
t
h
e
t
e
x
t
.
43
Figure 4.1. Map of Sample and Total EI Projects, by State
44
CHAPTER 5
THE WIDER REGIONAL ECONOMY – ENTREPRENEURSHIP
TRENDS IN APPALACHIA
INTRODUCTION
The Entrepreneurship Initiative was a bold attempt to bring about an economic
and attitudinal transformation across the Appalachian region. It is therefore
appropriate to step back from the specifics of the initiative – the individual
projects and programs funded and the communities in which they are located –
to consider at a more general level entrepreneurship trends across the region. In
particular, there is value in looking at the state of entrepreneurship in Appalachia
at the beginning of the initiative, in comparison with the nation as a whole, and
whether there has been any discernible change in the distribution of
entrepreneurial activity within the region. The initiative was launched in 1997 and
this evaluation considered projects closed through 2005, so these are the two
comparison dates used to assemble data in the following analysis. No attempt is
being made to establish cause and effect; the analysis is intended to
demonstrate the context within which the initiative was implemented.
THE CHALLENGE OF MEASUREMENT
Entrepreneurship is a difficult concept to measure, and many aspects of
entrepreneurship success are intangible.
74
However, several data sources allow
an examination of employment and income indicators at the county level to
assess general entrepreneurship trends. This analysis relies upon three data
sets:
Bureau of Economic Analysis’ Regional Economic Information System
U.S. Census Bureau’s Nonemployer Statistics
U.S. Census Bureau’s County Business Patterns.
The Center for the Study of Rural America at the Federal Reserve Bank of
Kansas City developed a series of indicators on the breadth and depth of
entrepreneurship at a regional level.
75
These measures use data from the
Bureau of Economic Analysis’ Regional Economic Information System and the
U.S. Census Bureau’s Nonemployer Statistics. The “breadth” of
entrepreneurship is measured by nonfarm proprietors as a proportion of total
nonfarm employment, and the “depth” of entrepreneurship is a measure of
74
Stephan J. Goetz and David Freshwater, “State-Level Determinants of Entrepreneurship and a
Preliminary Measure of Entrepreneurial Culture,” Economic Development Quarterly 15.1
(February 2001):58-70.
75
Sarah Low, “Regional Asset Indicators: Entrepreneurship Breadth and Depth,” The Main Street
Economist (September 2004):1-4.
45
nonfarm proprietors’ income per total nonfarm proprietors. These indicators have
been applied to all the counties in the Appalachian region for 1997 and 2005.
The second set of indicators follows from work of the Association for Enterprise
Opportunity (AEO). This organization uses the U.S. Census Bureau’s
Nonemployer Statistics and County Business Patterns data to estimate the
number of microenterprise businesses and the number of microenterprise
employees.
76
These datasets have been used in this analysis to provide
estimates for the number of microenterprises and the amount of microenterprise
employment for Appalachian counties for 1997 and 2005.
77
Additionally, a measure was developed to provide a sense of how important
entrepreneurship is to the local economy. This measure presents both
proprietors’ income as a proportion of total personal income in each county in
Appalachia and – in order to distinguish the effects of entrepreneurship from
broader trends – shows how these two components have themselves changed
over time.
It should be noted that ARC used a broad definition for the EI that includes small-
and medium-sized enterprises with fewer than 200 employees. This has
particular relevance for the capital access programs and for some of the
technical assistance activities, but for the purposes of this analysis data on
proprietors and microenterprises have been used as they serve as better (but far
from perfect) proxies for entrepreneurial activity. To provide a context, Table 5.1
shows growth rates in the number of new businesses in different size categories
between 1997 and 2005.
Table 5.1: Change in the Number of Small- and Medium-Sized Enterprises
410 ARC COUNTIES UNITED STATES
Size
# Employees
1997 2005 %
Change
1997 2005 %
Change
1-4 274,877 208,856 2.2 3,757,627 4,119,363 9.6
5-9 106,893 110,852 3.7 1,354,488 1,411,199 4.2
10-19 64,319 70,180 9.1 856,118 937,617 9.5
20-49 41,708 46,164 10.7 572,437 636,625 11.2
50-99 13,773 15,258 10.8 194,068 219,324 13.0
100-249 8,413 8,750 4.0 113,832 125,027 9.8
Total 509,983 532,060 4.3 6,848,570 7,449,155 8.8
Source: U.S. Census Bureau, County Business Patterns – Establishments by Size Class
76
For more information, see .
77
The number of microenterprise businesses is the number of nonemployer establishments plus
the number of establishments with 1-4 employees. The estimate of microenterprise employment
is the number of nonemployers plus an average of 2.5 employees per establishment
(establishments with 1-4 employees). This methodology differs from the AEO methodology.
46
Overall, the growth rate of business establishments with fewer than 250
employees in the ARC region was less than half of that of the nation as a whole,
with the greatest divergence in those establishments with fewer than five
employees and those in the range of 100-249 employees. In no size category
does the ARC region outperform the national rate. Given the comparatively low
growth rates for small- and medium-sized enterprises in the region, the EI’s
inclusion of these firms as a target for support appears to be strategic.
UNDERSTANDING THE REGIONAL CONTEXT
Three main observations derive from the following analysis of regional
entrepreneurship data and trends:
Trends in nonfarm proprietor and microenterprise employment in
Appalachia showed an increase over the 1997-2005 period consistent with
the nation as a whole. The counties benefiting most from above national
average increases were in the southern tier of Appalachia.
Trends in nonfarm proprietor income showed the Appalachian region
lagging behind U.S. levels in 1997 and slipping further behind by 2005.
Forty-two percent of Appalachian counties saw a decrease in income
levels, primarily in the northern and central tiers, whereas 25% saw
increases in nonfarm proprietor income levels higher than the national
rate, the majority of which are located in the southern tier.
Trends in the impact of entrepreneurship on the local economy, as might
be expected, generally echo the trends in nonfarm proprietor income, with
similar levels in Appalachia and the U.S. in 1997, but with Appalachia
substantially falling behind by 2005. This decline occurred in spite of the
fact that almost all Appalachian counties experienced increases in total
personal income. Only 15% of counties benefited from levels higher than
the national rate in 2005, and these were evenly distributed across the
region.
A detailed presentation of the data underlying these observations follows.
Entrepreneurship Employment
As there are many ways to determine or interpret what constitutes
entrepreneurship employment, two methods are presented here, each of which
gives a slightly different picture of entrepreneurship employment in the region.
Table 5.2 provides a summary of data for the U.S. and the 410 counties that
comprise the Appalachian Region.
As the table shows, trends in entrepreneurship employment in Appalachia are
broadly similar to those for the nation as a whole. In 1997, nonfarm proprietor
employment comprised 15.5% of total nonfarm employment in the U.S. and
47
15.6% in Appalachia. By 2005, these values had increased to 18.2% in the U.S.
and 18.0% in Appalachia.
Table 5.2. Indicators of Entrepreneurship Employment
INDICATOR / MEASURE 1997 VALUE 2005 VALUE
Nonfarm Proprietors Employment
U.S.
Appalachia
23,648,200
1,744,246
31,147,600
2,202,141
Nonfarm Proprietors Employment/Total Nonfarm Employment
U.S.
Appalachia
15.5%
15.6%
18.2%
18.0%
Number of Microenterprise Businesses
U.S.
Appalachia
19,197,236
1,432,733
24,511,431
1,757,105
Estimate of Microenterprise Employment
U.S.
Appalachia
24,833,677
1,845,049
31,690,476
2,178,389
Microenterprise Employment Estimate/Total Nonfarm
Employment
U.S.
Appalachia
16.3%
16.5%
17.9%
17.8%
Sources: U.S. Census Bureau Nonemployer Statistics and County Business Patterns;
Bureau of Economic Analysis, Regional Economic Information System
The following maps show nonfarm proprietors’ employment as a proportion of
total nonfarm employment in the Appalachian region. In 1997, there were 39
counties in Appalachia in which nonfarm proprietors accounted for 30 percent or
more of total nonfarm employment (Figure 5.1); this number almost doubled to
78 counties by 2005 (Figure 5.2).
Figure 5.1
Entrepreneurship Employment in
Appalachia:
Nonfarm Proprietors as a Percent of Total
Nonfarm Employment, 1997
Data Source: Bureau of Economic Analysis, Regional
Economic Information System
Map Prepared by RUPRI
October 2007
7.96% to 19.99% (243)
20% to 29.99% (128)
30% to 60.15% (39)
48
Figure 5.2
Entrepreneurship Employment in
Appalachia:
Nonfarm Proprietors as a Percent of Total
Nonfarm Employment, 2005
4.46% to 19.99% (187)
20% to 29.99% (145)
30% to 66.16% (78)
Data Source: Bureau of Economic Analysis, Regional
Economic Information System
Map Prepared by RUPRI
October 2007
Nonfarm proprietor employment increased 31.7% from 1997 to 2005 in the U.S.
and 26.3% in Appalachia. Levels increased in all but 70 Appalachian counties
from 1997 through 2005, and grew at a rate exceeding the national average in
141 Appalachian counties – more than doubling in 21 counties (Figure 5.3).
Figure 5.3
Entrepreneurship Employment
Change in Appalachia:
Percent Change in Nonfarm Proprietor
Employment, 1997-2005
Data Source: Bureau of Economic Analysis, Regional
Economic Information System
Map Prepared by RUPRI
October 2007
Decrease or No Change (70)
Increase 0.1%-31.7% (199)
Increase 31.8%-271.7% (141)
49
As Figure 5.3 shows, although the distribution of growth in entrepreneurship
employment extends to all parts of the region, there is a concentration of
counties with growth above the national average in eastern Tennessee, western
North Carolina, northern Georgia and northern Alabama.
Examining the number of microenterprise employees shows a similar picture of
Appalachia tracking the national levels. In 1997, microenterprise employment as
a percent of total nonfarm employment was 16.3% in the U.S. and 16.5% in
Appalachia. By 2005, these levels had increased to 17.9% and 17.8%
respectively. In Appalachia in 1997, there were 25 counties in which
microenterprise employment accounted for 30% or more of total nonfarm
employment (Figure 5.4); by 2005 there were 48 such counties (Figure 5.5).
Figure 5.4
Microenterprise Employment in
Appalachia:
Microbusiness Employment* as a Percent of
Total Nonfarm Employment, 1997
10.0% to 19.9% (215)
20% to 29.9% (170)
30% to 42.9% (25)
*Microbusiness employment equals the number of
nonemployers plus the estimated number of employees in
establishments of employment size class 1-4 (2.5
employees per establishment).
Data Source: U.S. Census Bureau County Business
Patterns and Nonemployer Statistics; Bureau of
Economic Analysis, Regional Economic Information
System
Map Prepared by RUPRI
October 2007
From 1997 to 2005, the number of microenterprise businesses increased 27.7%
in the U.S. and 22.6% in Appalachia. Microenterprise employment increased
23.6% in the U.S. and 18.1% in Appalachia over this same period. Only 24
counties in Appalachia had a decrease or no change in the number of
microenterprise businesses (Figure 5.6), and only 44 counties saw a decrease or
no change in microenterprise employment from 1997 to 2005 (Figure 5.7).
50
Figure 5.5
Microenterprise Employment in
Appalachia:
Microbusiness Employment* as a Percent of
Total Nonfarm Employment, 2005
8.27% to 19.9% (196)
20% to 29.9% (166)
30% to 45.7% (48)
Data Source: U.S. Census Bureau County Business
Patterns and Nonemployer Statistics; Bureau of
Economic Analysis, Regional Economic Information
System
*Microbusiness employment equals the number of
nonemployers plus the estimated number of employees
in establishments of employment size class 1-4 (2.5
employees per establishment).
Map Prepared by RUPRI
October 2007
Figure 5.6
Microenterprise in Appalachia:
Percent Change in the Number of
Microenterprise Businesses*, 1997-2005
*Microbusiness business are the number of
nonemployers plus the number establishments of
employment size class 1-4.
Data Source: U.S. Census Bureau County Business
Patterns and Nonemployer Statistics; Bureau of
Economic Analysis, Regional Economic Information
System
Decrease or No Change (24)
Increase 0.1%-27.7% (272)
Increase 27.8%-124.7% (114)
Map Prepared by RUPRI
October 2007
51
Figure 5.7
Microenterprise in Appalachia:
Percent Change in the Number of
Microenterprise Employees*, 1997-2005
*Microbusiness employment equals the number of
nonemployers plus the estimated number of employees
in establishments of employment size class 1-4 (2.5
employees per establishment).
Data Source: U.S. Census Bureau County Business
Patterns and Nonemployer Statistics; Bureau of
Economic Analysis, Regional Economic Information
System
Decrease or No Change (44)
Increase 0.1%-23.7% (258)
Increase 23.8%-118.8% (108)
Map Prepared by RUPRI
October 2007
Entrepreneurship Income
A measure of entrepreneurship income is nonfarm proprietors’ income per
nonfarm proprietor employment. Table 5.3 shows this indicator for 1997 and
2005, along with other indicators of income and earnings as a basis for
comparison.
Table 5.3. Indicators of Entrepreneurship Income
INDICATOR / MEASURE 1997 VALUE
(ADJUSTED
TO $05)
2005 VALUE
Nonfarm Proprietors’ Income / Nonfarm Proprietor
Employment
U.S.
Appalachia
Appalachia as a % of U.S. Value
$27,881
$23,094
82.8%
$30,193
$23,218
76.9%
Per Capita Income
U.S.
Appalachia
Appalachia as a % of U.S. Value
$30,827
$25,687
83.3%
$34,471
$28,336
82.2%
Wage & Salary Disbursements / Wage & Salary Employment
U.S.
Appalachia
Appalachia as a % of U.S. Value
$36,331
$30,706
84.5%
$40,146
$32,779
81.6%
Source: Bureau of Economic Analysis, Regional Economic Information System
On all measures of income, Appalachia lags somewhat behind the national
figures and fell further behind over the period 1997-2005. In 1997, the three
52
income indicators for Appalachia were between 82% and 85% of the U.S. value.
By 2005, nonfarm proprietor income per nonfarm proprietor had fallen to 76.9%
of the U.S. value and wage and salary disbursements per wage and salary job
had slipped to 81.6% of the U.S. level.
In 1997, nonfarm proprietor income per nonfarm proprietor (adjusted to 2005
dollars) ranged from $5,263 to $54,117 in Appalachia (Figure 5.8). In 26
counties in Appalachia, the value exceeded the U.S. value of $27,881 for this
indicator. In 2005, nonfarm proprietor income per nonfarm proprietor in
Appalachia ranged from $7,190 to $64,689, and 28 counties had values
exceeding the U.S. figure of $30,193 (Figure 5.9).
Figure 5.8
$5,263 - $16,572 (135)
$16,573 - $27,881 (249)
$27,882 - $54,117 (26)
Entrepreneurship Income in
Appalachia:
Nonfarm Proprietors Income per Nonfarm
Proprietor, 1997
(adjusted to 2005$)
Data Source: Bureau of Economic Analysis, Regional
Economic Information System
Map Prepared by RUPRI
October 2007
Total nonfarm proprietors’ income (real $2005) increased by 42.6% in the U.S.
and 26.9% in Appalachia from 1997 to 2005 (Figure 5.10). Within Appalachia,
total nonfarm proprietors’ income decreased in 172 counties, noticeably in New
York, Pennsylvania, eastern Kentucky, western West Virginia and southwestern
Virginia. Increases were experienced in 238 counties – 102 counties in
Appalachia had increases in nonfarm proprietors’ income exceeding the national
rate, and in 22 counties income more than doubled during this time period. The
greatest gains were to be found in eastern West Virginia, eastern Tennessee,
western North Carolina, and northern Alabama.
53
Figure 5.9
Entrepreneurship Income in
Appalachia:
Nonfarm Proprietors Income per Nonfarm
Proprietor, 2005
Data Source: Bureau of Economic Analysis,
Regional Economic Information System
$7,189 - $18,641 (284)
$18,642 – $30,193 (98)
$30,194 - $64,686 (28)
Map Prepared by RUPRI
October 2007
Figure 5.10
Entrepreneurship Income in
Appalachia:
Percent Change in Total Nonfarm
Proprietors’ Income, 1997-2005
(1997 adjusted to 2005$)
Decrease (172)
Increase 0.1%-42.6% (136)
Increase 42.7%-188.7% (102)
Data Source: Bureau of Economic Analysis,
Regional Economic Information System
Map Prepared by RUPRI
October 2007
54
Entrepreneurship’s Contribution to the Local Economy
One method of assessing the importance of entrepreneurship to the local
economy is to examine the share of nonfarm proprietors’ income to total county
personal income in 1997 and 2005. Table 5.4 summarizes this indicator for the
U.S. and Appalachia.
Table 5.4. Indicators of Entrepreneurial Contribution
INDICATOR/MEASURE 1997 VALUE 2005 VALUE
Nonfarm Proprietor Income/Total Personal Income
U.S.
Appalachia
7.8%
7.0%
9.2%
7.6%
As Table 5.4 shows, Appalachia has fallen substantially behind the U.S. on this
measure of entrepreneurial contribution. Within Appalachia in 1997, 90 counties
had shares greater than the national average of 7.8% (Figure 5.11), but by 2005,
there were only 63 counties in Appalachia with a share greater than the national
average of 9.2% (Figure 5.12). From 1997 to 2005, the nonfarm proprietors’
income share of total county personal income increased in 170 Appalachian
counties, and decreased in 240 Appalachian counties (Figure 5.13).
Before drawing any conclusions about these increasing or decreasing shares, it
is first necessary to look at the trends in actual nonfarm proprietors’ income and
total personal income. In some cases, proprietors’ income share may be
increasing as a share because total county income is on the decline, resulting in
a higher share even if actual nonfarm proprietors’ income did not rise. The map
in Figure 5.14 compares nonfarm proprietors’ income change to total county
personal income change.
In 238 Appalachian counties, nonfarm proprietors’ income (in real $2005)
increased from 1997 to 2005, and total county personal income decreased in
only 4 of these counties. Nonfarm proprietors’ income decreased in 172
Appalachian counties; in 15 of these counties, total county personal income also
decreased. The conclusion to be drawn here is that whether or not counties
have an increasing or decreasing share of nonfarm proprietor income does not
generally appear to be impacted by increases or decreases in total personal
income.
55
Figure 5.11
Entrepreneurship and the Local
Economy:
Nonfarm Proprietors’ Income as a Share of
Total Personal Income (TPI), 1997
At or below U.S. Share (320)
Above U.S. Share (90)
Data Source: Bureau of Economic Analysis, Regional
Economic Information System
Map Prepared by RUPRI
October 2007
U.S. Share = 7.84%
Figure 5.12
Entrepreneurship and the Local
Economy:
Nonfarm Proprietors’ Income as a Share of
Total Personal Income (TPI), 2005
At or below U.S. Share (347)
Above U.S. Share (63)
Data Source: Bureau of Economic Analysis, Regional
Economic Information System
Map Prepared by RUPRI
October 2007
U.S. Share = 9.20%
56
Figure 5.13
Entrepreneurship and the Local
Economy:
Change in the Share of Nonfarm
Proprietors’ Income to Total Personal
Income (TPI), 1997-2005
Share Increased (170)
Share Decreased (240)
Data Source: Bureau of Economic Analysis, Regional
Economic Information System
Map Prepared by RUPRI
October 2007
EI PERFORMANCE WITHIN THE REGIONAL CONTEXT
What do these findings suggest about the impact of ARC’s EI on the region?
While it is difficult to draw any causal connections between ARC investments and
these regional data, several trends are evident. Entrepreneurship, when defined
broadly as nonfarm proprietor and microenterprise employment, is an
increasingly important component of the Appalachian regional economy. For
some parts of the region – the southern tier in particular – rates of employment
associated with entrepreneurship are greater than for the country as a whole.
This observation confirms the importance of investments in entrepreneurship
development as a key piece of a regional economic development strategy. In
essence, entrepreneurship represents an asset upon which the region can build.
However, in terms of income associated with entrepreneurship, the picture
unfortunately mirrors the past – a region continuing to lag behind the rest of the
nation in terms of nonfarm proprietors’ income. This observation suggests that in
spite of ARC’s investment, the income generated through entrepreneurial activity
is not growing in such a way as to allow the region to catch up with the rest of the
country. This analysis also suggests that ARC’s investments have had an impact
on getting more entrepreneurs into the pipeline, but have had a lesser impact on
creating stronger and higher growth entrepreneurs, at least as measured by
income trends. To better understand the reason behind this disparity, a more
detailed investigation of entrepreneurial income and wage and salary levels
associated with entrepreneurship in the region should be undertaken. It was not
possible in the current evaluation to gain insights into the income and wealth
creating potential of entrepreneurial ventures or into the quality of jobs created
for others by these entrepreneurs.
57
Have ARC investments contributed to creating a more entrepreneurial region? It
is difficult to say from analysis of secondary data. Entrepreneurial activity has
increased over the period of investment and some parts of the region have
performed even better than the nation. It is possible that the Appalachian region
would have fallen even further behind the nation in terms of income from
entrepreneurship without EI investments. Insights from more detailed analysis of
sample projects and key stakeholders help answer this question in a way that
secondary data analysis cannot.
Figure 5.14
Entrepreneurship and the Local
Economy:
Change in Nonfarm Proprietors’ Income and
Total Personal Income (TPI), 1997-2005
(Values adjusted to $05)
TPI Increased (234)
TPI Decreased (4)
Nonfarm Proprietors Income Increased
Nonfarm Proprietors Income Decreased
TPI Increased (157)
TPI Decreased (15)
Data Source: Bureau of Economic Analysis, Regional
Economic Information System
Map Prepared by RUPRI
October 2007
58
CHAPTER 6
ARC INVESTMENT PORTFOLIO –
PROJECT IMPLEMENTATION AND REPORTED METRICS
Because ARC is built around a federal-state-local partnership, few of its
programs operate according to a single cookie-cutter model. Instead, each state
and each local program tend to take a distinctive approach to ARC funding
programs and opportunities. Differences in program management from state to
state and differences in the types of projects funded make generalizing impacts
across the EI portfolio difficult. This chapter is designed to provide insights into
project implementation so that later discussions of lessons learned and
recommendations are understood within this overall context.
78
ARC required the reporting of several metrics for all project investments,
including job creation and retention, business creation, businesses served, and
private investment leveraged. Data from the final project reports submitted to
ARC by all grant recipients provide useful insights into the size and scope of the
initiative, as well as these specific, but limited, impacts of project investments.
Deeper understanding of project impacts and the broader policy implications of
ARC investments gained through analysis of the sample projects is presented in
Chapter 7.
PROJECT IMPLEMENTATION
As with most ARC initiatives, the approach to using the EI funds varied widely
across the states in the region. To better understand these differences, and what
they might mean in terms of program outcomes, the evaluation team conducted
phone interviews with state program managers and with the director of the
Entrepreneurship Initiative in Washington, D.C. The purpose of these interviews
was to develop a sense of how each state approached the EI and to get a view of
the initiative from the state level.
Each governor and state program manager approached the Entrepreneurship
Initiative with a different level of commitment to entrepreneurship as an economic
development strategy and a different strategy for distributing what in many cases
was described as a relatively small resource pool. Table 6.1 describes the
strategy or approach used to distribute funds in each state while Table 6.2 shows
the distribution of investments across the region over the 1997 to 2005 period. In
one state, Pennsylvania, a large portion of resources was used to support a
specific intervention – the creation of entrepreneurial networks. In other states,
local development districts were responsible for promoting the initiative and
encouraging local organizations to submit grant proposals. In most cases, states
78
Insights and observations presented in the first section of this chapter are drawn from
interviews with regional stakeholders and from project leaders. The evaluation team reported
insights that were widely held and recurring across the range of interviews conducted.
59
Table 6.1. Description of State Approaches to EI Investments*
STATE APPROACH
Alabama Entrepreneurship included in Appalachian Development Plan; provided impetus
for the state to support diverse homegrown initiatives, including entrepreneurship
education and incubators
Georgia State program manager’s office identified local and regional projects to be
funded; relied less on Local Development Districts (LDDs) to take the lead on
developing projects
Kentucky State created a government-sponsored commission whose chair was the state
alternate; Chair lived in region and selected both local and regional projects; also
funded other state organizations to expand activities into region; identified
technical assistance gap and tried to select projects to fill it
Maryland Initially tried to start new programs in the region, providing several years of
funding; partnered with WV on projects; efforts were not able to create
sustainable new institutions; later stages, plugged EI into ongoing activities in the
counties, like Main Street program
Mississippi State program manager and alternate active in identifying projects; tried to work
with local philanthropy organizations
New York Worked at the state level to fund some large programs to support
entrepreneurship education – some local and some regional; supported some
other local initiatives
North
Carolina
Rolled money over until state had a pool of money; contracted with state
Department of Commerce for multi-year program; efforts were not able to create
sustainability; locally driven projects were also funded, with some more success
Ohio Leadership came from Governor’s Office of Appalachia; active in identifying
projects with both Local Development Districts (LDDs) and nonprofits; joint
funding of projects with the Foundation for Appalachian Ohio; open process for
soliciting proposals; multi-year funding of regional institutions and one time
projects
Pennsylvania Most of funds were used to support local networking activities; also put money
into region-wide youth entrepreneurship program; state program manager and
alternate worked out of the Office of Appalachian Development in state
Department of Community and Economic Development
South
Carolina
Most projects identified by state program managers; relied less on Local
Development Districts (LDDs)
Tennessee Worked with some Local Development Districts (LDDs); major investment made
in the development of one organization, leading to the creation of a sustainable
organization with a broad reach
Virginia High priority for the state; spread grants around to as many communities as
possible; saw EI funds as being part of a set of resources, including ARC
telecommunications and Main Street programs, that communities could tap; used
a competitive RFP process with an open workshop planning process leading to a
diverse set of project proposals; funded multi-year projects; drew in other funding
resources, particularly from foundations
West Virginia State highlighted entrepreneurship funds as one of many tools available; non-
profit organizations stepped up and provided most of the impetus for developing
and implementing grant projects; used a competitive RFP process with an open
workshop planning process leading to a diverse set of project proposals; funded
multi-year projects; drew in other funding resources, particularly other state funds
*Interviews were attempted but not completed with state program managers in Mississippi and
Ohio due to illness and staff turnover, respectively, at the time the interviews were undertaken.
Information about their approach to the EI was obtained through an interview with the director of
the EI.
60
used their annual allocations to fund a range of projects; in one case, North
Carolina, funds were rolled over until the state had a pool of resources to support
a larger project.
Several themes were identified through the program manager interviews. From
the perspective of local communities, ARC Entrepreneurship Initiative resources
were described as “catalytic” and important to demonstrating new approaches to
entrepreneurship development. In many cases, these projects would not have
gotten started without ARC investment. As one manager described it, these were
“but for” resources. At the same time, program managers admitted that the
relatively small pool of resources attached to the initiative made it harder to make
entrepreneurship investments a priority, particularly if local development districts
were not focusing in this area and encouraging proposals from their regions.
It was also clear that the state leaders, including governors and program
managers, played a “gate keeping” role in terms of ARC initiatives, including
entrepreneurship. Strong support for entrepreneurship from state leaders often
translated into more aggressive promotion of the program within the state, such
as was seen in Virginia. In addition, since the innovation in entrepreneurship
programs was primarily coming from the local rather than the state level, the
degree to which the state program manager was networked with local economic
development organizations, especially non-profits, could impact the ability to
effectively implement the initiative. Much of the investment in West Virginia, for
example, was driven by community non-profit organizations.
In addition to the diverse approaches used by the states in allocating resources,
ARC investments were spread across a portfolio of program categories. Prior to
selecting the evaluation sample, Entrepreneurship Initiative projects were
classified into program categories based on a review of project descriptions,
including the goals of each project, and follow-up interviews with project
leadership. Project goals differed even within program categories. Metrics used
to report on project outcomes varied by program category, and it was clear from
the interviews with project leaders that job creation and business creation were
not the only metrics considered in articulating project goals and reporting
outcome measures.
To put the findings of this evaluation into better perspective, a description of the
range of projects found within these program types is provided here. These
descriptions should provide readers with a sense of the diversity within each
project type and the breadth of investments undertaken by ARC in support of
entrepreneurship development in the region.
Entrepreneurship Education Projects
Entrepreneurship education projects were designed to introduce
entrepreneurship concepts and curricula into the schools, kindergarten through
61
community college and university. In the language of the programmatic goals set
out in Chapter 3, these programs sought to help create more entrepreneurs in
the pipeline, and were targeted primarily to students as opposed to adult
learners. (Programs for adult learners were included in the technical assistance
and training category.) A profile of the sample entrepreneurship education
projects appears below.
Sample Entrepreneurship Education Projects
Number of projects = 17
Total ARC investment = $2,199,207
Average ARC investment = $129,365
Minimum ARC investment = $10,000
Maximum ARC investment = $784,517
Range of years for project investment = 1 – 4
The range of entrepreneurship education projects supported by the EI was quite
diverse. For example, funds were used to:
Support outreach into rural communities for an existing business plan
competition
Create an internship program and place youth in entrepreneurial
companies
Organize summer youth entrepreneurship camps, using REAL (Rural
Entrepreneurship through Action Learning) and other curriculum
Train teachers in the REAL curriculum and bring it into the classroom
Train middle school students using Junior Achievement, with follow-up
training using a virtual business simulation model
Create student run, school-based enterprises.
Some projects were designed as pilots to test an entrepreneurship education
approach in a community or region. Others were designed to build
entrepreneurship education capacity by training teachers and incorporating new
curriculum into school systems, or by investing in the creation of school-based
enterprises that could be passed down to future classes of students. About half
the sample projects were defined by project leaders as being sustainable, i.e.,
they continued beyond the ARC grant with support from other funders or, in
some cases, with an additional ARC grant.
With the exception of those projects that created school-based enterprises, most
were not designed to create or retain jobs or to build new businesses. Rather,
they were designed to expose youth to entrepreneurship with an expectation that
these entrepreneurial skills would serve to prepare them to become better
employees in the future and/or to motivate them to create their own economic
futures through enterprise development. Given their goals, traditional economic
development metrics provide an incomplete story of the impact of these projects.
6
2
T
a
b
l
e
6
.
2
.
A
n
n
u
a
l
a
n
d
T
o
t
a
l
A
R
C
E
n
t
r
e
p
r
e
n
e
u
r
s
h
i
p
I
n
v
e
s
t
m
e
n
t
s
,
1
9
9
7
-
2
0
0
5
,
b
y
S
t
a
t
e
(
$
)
*
S
T
A
T
E
1
9
9
7
1
9
9
8
1
9
9
9
2
0
0
0
2
0
0
1
2
0
0
2
2
0
0
3
2
0
0
4
2
0
0
5
T
O
T
A
L
A
l
a
b
a
m
a
0
1
5
9
,
0
0
0
3
8
1
,
6
6
3
5
3
7
,
1
8
7
1
9
8
,
8
5
0
4
0
5
,
0
0
0
9
2
,
0
0
0
5
1
8
,
8
9
2
1
1
2
,
6
7
5
2
,
4
0
5
,
2
6
7
G
e
o
r
g
i
a
1
1
0
,
0
0
0
0
4
4
1
,
8
2
6
0
5
8
1
,
5
2
9
3
4
9
,
4
4
2
8
3
,
0
0
0
7
2
3
,
7
4
7
3
3
4
,
1
1
8
2
,
6
2
3
,
6
6
2
K
e
n
t
u
c
k
y
9
2
,
7
6
9
2
2
6
,
0
0
0
7
8
7
,
5
0
0
2
9
0
,
5
3
2
5
1
0
,
5
0
0
2
2
5
,
3
6
2
6
5
7
,
8
8
0
3
8
3
,
7
9
4
5
8
7
,
1
9
8
3
,
7
6
1
,
5
3
5
M
a
r
y
l
a
n
d
0
2
0
7
,
0
0
0
4
5
5
,
2
0
0
5
4
8
,
9
1
8
3
1
2
,
3
7
9
1
3
9
,
0
0
0
9
9
,
0
0
0
1
3
0
,
0
0
0
4
5
,
0
0
0
1
,
9
3
6
,
4
9
7
M
i
s
s
i
s
s
i
p
p
i
0
2
2
0
,
5
7
7
3
0
,
0
0
0
1
6
4
,
9
7
0
5
9
4
,
8
3
5
1
5
7
,
0
0
0
0
2
0
0
,
0
0
0
2
2
6
,
0
0
0
1
,
5
9
3
,
3
8
2
N
e
w
Y
o
r
k
0
2
4
3
,
1
4
0
1
8
7
,
6
5
4
9
2
0
,
0
0
0
7
5
,
0
0
0
2
5
,
0
0
0
1
4
5
,
0
0
0
2
6
5
,
0
0
0
3
5
,
0
0
0
1
,
8
9
5
,
7
9
4
N
o
r
t
h
C
a
r
o
l
i
n
a
0
0
0
1
,
0
5
0
,
0
0
0
0
4
4
4
,
9
0
8
2
0
0
,
0
0
0
1
5
7
,
5
0
0
6
7
3
,
6
7
3
2
,
5
2
6
,
0
8
1
O
h
i
o
0
1
5
3
,
9
4
0
4
0
6
,
2
1
3
4
5
1
,
9
2
1
6
1
3
,
7
0
0
2
4
4
,
5
0
0
1
,
6
7
3
,
3
3
0
1
5
,
0
0
0
5
0
,
0
0
0
3
,
6
0
8
,
6
0
4
P
e
n
n
s
y
l
v
a
n
i
a
3
1
3
,
3
9
7
1
,
0
7
2
,
8
0
8
1
,
0
9
9
,
0
0
0
1
9
9
,
9
0
0
6
5
9
,
6
3
6
3
8
3
,
7
5
0
3
6
2
,
0
0
0
4
6
0
,
0
0
0
1
2
5
,
0
0
0
4
,
6
7
5
,
4
9
1
S
o
u
t
h
C
a
r
o
l
i
n
a
0
5
0
0
,
0
0
0
0
7
5
0
,
0
0
0
1
1
9
,
1
0
5
7
5
2
,
0
0
0
5
0
0
,
0
0
0
8
9
1
,
3
8
2
5
8
,
7
5
0
3
,
5
7
1
,
2
3
7
T
e
n
n
e
s
s
e
e
0
4
5
1
,
7
4
8
3
2
5
,
0
0
0
3
9
0
,
0
0
0
4
1
0
,
1
8
5
1
8
5
,
0
0
0
9
3
,
0
0
0
2
0
1
,
1
3
3
3
5
,
0
0
0
2
,
0
9
1
,
0
6
6
V
i
r
g
i
n
i
a
3
7
4
,
5
0
0
0
4
9
5
,
3
8
0
4
1
6
,
5
2
7
4
7
6
,
6
3
0
4
3
3
,
0
0
0
1
8
1
,
2
0
0
2
8
0
,
3
9
1
9
0
,
0
0
0
2
,
7
4
7
,
6
2
8
W
e
s
t
V
i
r
g
i
n
i
a
1
4
2
,
1
5
0
7
5
2
,
9
4
8
1
,
0
7
5
,
9
1
6
3
0
0
,
5
3
7
1
,
6
7
3
,
7
4
8
1
,
0
4
2
,
8
3
5
5
8
2
,
0
0
0
4
5
8
,
4
0
0
1
,
1
0
5
,
4
7
4
7
,
1
3
4
,
0
0
8
R
e
g
i
o
n
a
l
1
4
,
0
0
0
3
3
2
,
5
0
3
1
2
4
,
6
2
3
2
6
4
,
9
0
1
1
6
9
,
1
6
2
3
3
9
,
6
1
0
9
7
3
,
5
7
4
1
0
8
,
0
6
3
7
5
,
0
0
0
2
,
4
0
1
,
4
3
6
T
o
t
a
l
A
R
C
1
,
0
4
6
,
8
1
6
4
,
3
1
9
,
6
6
4
5
,
8
0
9
,
9
7
5
6
,
2
8
5
,
3
9
3
6
,
3
9
5
,
2
5
9
5
,
1
2
6
,
4
0
7
5
,
6
4
1
,
9
8
4
4
,
7
9
3
,
3
0
2
3
,
5
5
2
,
8
8
8
4
2
,
9
7
1
,
6
8
8
*
F
i
g
u
r
e
s
i
n
c
l
u
d
e
a
l
l
p
r
o
j
e
c
t
s
(
c
l
o
s
e
d
a
n
d
o
p
e
n
)
a
n
d
i
n
v
e
s
t
m
e
n
t
s
f
r
o
m
a
l
l
f
u
n
d
i
n
g
s
o
u
r
c
e
s
w
i
t
h
i
n
A
R
C
.
63
Several examples help to demonstrate the range of entrepreneurship education
projects funded by ARC and the type of outcomes they achieved.
In 1999, the Kentucky Science and Technology Council (KSTC) received funding
from ARC to establish next generation entrepreneurial schools in two pilot
districts. The goal of the project was to create learning communities where
students could embrace an entrepreneurial frame of mind and, in turn, to help all
students generate ideas for and start new ventures in their home communities.
Using initial and follow-on ARC grants, KSTC supported the creation of 99
EdVentures – school-based enterprises – including 31 in high schools, 25 in
middle schools, 37 in elementary schools, and 6 district-wide. KSTC has
documented the outcomes of these activities in two ways – through metrics
including the number of school-based enterprises started (99) and the number of
participants in these ventures (27,000) as well as through stories of each
individual EdVenture that describe the start-up process, funds raised, students
involved, etc.
In West Virginia, ARC funds were used to bring an entrepreneurship focus to an
existing summer camp program, using the REAL curriculum. The Ohio-West
Virginia YMCA provided leadership for this initiative which secured four years of
ARC funding. In addition to serving the 465 student participants, the initiative has
generated additional interest in youth entrepreneurship in the state, attracting
funding to make the program sustainable and engaging youth in several ways.
The statewide economic development planning process, Vision Shared,
partnered with the Ohio-West Virginia YMCA to provide a teen perspective on the
future of the state. The YMCA held a Teen Forum on West Virginia’s Future in
2002 to develop this perspective. The Department of Education created a full-
time entrepreneurship coordinator position and hired a certified REAL instructor
as the coordinator.
ACEnet in Ohio used ARC funding to extend into three high schools a computer
opportunities program designed to provide training to students in setting up
computer-based enterprises. The program created three new businesses and 14
new jobs. In addition, however, it expanded the computer technology available in
these schools and exposed students to entrepreneurship concepts. Another
outcome was the attraction of additional federal support to continue the program
beyond ARC funding.
Technical Assistance and Training Projects
Technical assistance and training projects were designed to build the skill sets of
individual entrepreneurs by offering them access to one-on-one counseling and
workshops related to starting and growing their businesses. The programmatic
goals of these programs were to create more entrepreneurs in the pipeline, and
better informed and skilled entrepreneurs whose businesses were more
productive and created more jobs in the region. Most of these programs were
64
targeted at adult entrepreneurs as distinguished from the entrepreneurship
education projects that primarily targeted youth. A profile of sample technical
assistance and training projects appears below.
Sample Technical Assistance and Training Projects
Number of projects = 34
Total ARC investment = $4,933,997
Average ARC investment = $145,118
Minimum ARC investment = $10,000
Maximum ARC investment = $2,088,961
Range of years for project investment = 1 – 6
One large grant to Pennsylvania to support a statewide technical
assistance/networking initiative skews average investment. If the
Pennsylvania project is removed from the sample, the range of
investments is $10,000 to $650,615 and average investment
per technical assistance project is $86,213.
As with education projects, the range of technical assistance projects was quite
wide. Investments were used to fund such activities as:
Grants to individual entrepreneurs to purchase private sector services,
such as accounting or legal assistance
Funding for “how to start a business” and other workshops for
entrepreneurs and potential entrepreneurs
Providing technical assistance in conjunction with a microenterprise
program
Developing export assistance programs for entrepreneurs
Offering one-on-one assistance to entrepreneurs in accounting, marketing,
and other business management areas
Delivering marketing assistance to clients of a food incubator
Developing computer-based entrepreneurship training programs
Providing managerial and technical assistance to high-tech start ups
Creating a virtual business assistance center.
In general, these projects provided entrepreneurs with access to business
support resources, both customized and in workshop settings. Often, these
services represented additions or expansions to other programs offered by the
local organizations. However, in some cases, these investments were used to
create new capacity in the region, such as through the creation of a virtual
assistance center. About three quarters of projects were defined as sustainable
by project leaders. Several examples provide insights into these technical
assistance and training projects.
Kentucky Highlands Investment Corporation (KHIC) received funding from ARC
for its business support center, The Launching Pad. These funds were used to
provide technical assistance to entrepreneurs, some of whom were customers
65
(or future customers) who had received capital through KHIC’s range of financial
instruments. The Launching Pad provided a place for entrepreneurs to access
computers and other technology, and to obtain services from KHIC’s skilled staff.
This assistance served to formalize and build the capacity of KHIC to support the
non-financial needs of its business customers. Previous assessment of KHIC’s
programs found that this technical assistance or managerial support was as
important to entrepreneurs as the capital that KHIC provided.
79
The Ohio Valley Regional Development Commission used ARC funds to provide
small grants (up to $5,000) for entrepreneurs to purchase business services from
private sector providers. This program was designed in response to the
expressed needs of the Commission’s revolving loan customers who sought
assistance early in the life of their businesses, before they had the sales and
resources to pay for services. The project was successful in meeting the needs
of these entrepreneurs and connecting them to appropriate service providers.
The Commission was able to help almost twice as many entrepreneurs as initially
projected because entrepreneurs often required very small grants to get the help
they needed.
Mississippi State University received support to develop training programs to
make entrepreneurs aware of the power of marketing via the Internet. In 1998,
when the ARC investment was made, the concept and value of e-commerce was
not universally recognized. Mississippi State’s programs exposed home-based
business owners, farmers and youth to Internet marketing concepts. While the
program helped individual business owners, it also launched regional and
national efforts to develop e-commerce as an effective business strategy. The
curriculum developed at Mississippi State is used throughout the country and a
national e-commerce education effort through Cooperative Extension continues
to support education and training in this area.
Sector Projects
Projects funded in the sectors category were focused on supporting
entrepreneurs in a particular sector, such as food, wood products, ceramics,
often by building networks among entrepreneurs. In terms of programmatic
goals, these projects were designed to create more informed and skilled
entrepreneurs, operating more productive businesses that create jobs in the
region. The projects included in the sample, however, were as diverse as the
sectors that make up the economy of the Appalachian region, including:
Creation of a consignment gift shop to support local artisans
Development of an aquaculture program including entrepreneurship
training
79
David Barkley and Deborah Markley, The Development of an Entrepreneurial Support
Organization: The Case of the Kentucky Highlands Investment Corporation, RUPRI Center for
Rural Entrepreneurship, Research Case Study Series Number 1, March 2003.
66
Support for a ceramics corridor cluster in collaboration with a university
Operational support for the Ralph Stanley museum and music center
Development of a driving tour to cultural and heritage attractions and
artisan businesses
Funding for a market study of the cut flowers industry in Mississippi
Support for workshops on development of a craft woodworking industry.
A profile of sample sector projects appears below. Less than half of project
leaders defined their efforts as being sustainable over time. Several examples
serve to illustrate the high degree of diversity in these sector projects.
Sample Sector Projects
Number of projects = 17
Total ARC investment = $1,496,585
Average ARC investment = $90,976
Minimum ARC investment = $1,400
Maximum ARC investment = $393,711
Range of years for project investment = 1 – 4
The Kentucky Artisan Trails Project was designed to create a gateway attraction
on I-75 that would promote a driving tour of regional cultural and heritage sites as
well as artisan businesses along the route. Through development of an
information kiosk on the interstate, a website and hard copy maps, the project
helped to encourage tourism development by networking individual sites into a
destination tour. The website features products produced by regional
entrepreneurs, places for tourists to visit, and events that are planned throughout
the region. The project focused on the heritage tourism sector, including
entrepreneurs, and developed a network of people, places and organizations that
worked to build the sector in the region.
Advantage West, a regional economic development organization in western
North Carolina, used ARC investment to help introduce and develop a biotech
sector in the region. There was much interest in biotech in the region, but limited
staff capacity to move the concept forward. ARC funds were used to hire a
biotech coordinator and to develop wet lab space at the community college – the
first in the region. Advantage West also established a steering committee to
consider strategic opportunities for advancing the sector. These initial
investments have helped create a biotech sector focused on the natural products
available in the region. The region has attracted additional local government and
regional university investments to continue to build this sector.
Appalachian by Design, a non-profit rural economic development organization,
wanted to help create a knitting industry in rural Appalachia by connecting female
artisans with broader markets. ARC funding was used to help individual rural
women develop a cottage knitting industry. The project was initially quite
67
successful, producing high quality products and tapping new markets. Since its
initial success, the project has scaled back as resources became constrained,
forcing the organization to focus on a more limited market.
Incubator Projects
Incubators projects in the region were designed to provide a supportive
environment for entrepreneurs to hatch and grow new enterprises. Incubator
programmatic goals usually related to creating more informed and better skilled
entrepreneurs, helping them create more productive businesses and ultimately
encouraging more entrepreneurs to stay in the region. ARC investments in
incubator projects, however, were generally of three types:
Investments to determine the feasibility of an incubator in a particular
community or region
Investments to support the building and/or operation of an incubator
facility
Investments to develop programs offered through the incubator.
The sample of 12 incubator investments was equally split across these three
types. A profile of sample incubator projects appears below.
Sample Incubator Projects
Number of projects = 12
Total ARC investment = $913,291
Average ARC investment = $76,108
Minimum ARC investment = $10,000
Maximum ARC investment = $388,084
Range of years for project investment = .5 – 1
Projects tended to be much more short term than the other program types,
ranging from six months to one year. Nine of the 12 incubator projects were
defined as being sustainable. Examples of incubator projects follow.
ARC funds were used to enhance the services provided by the Clinch Powell
Community Kitchen in east Tennessee. While the community kitchen already
provided shared space for food entrepreneurs in the area to develop and
produce their goods, ARC funds allowed the incubator to purchase new
equipment and expand the training support offered to entrepreneurs, particularly
in the areas of product development, marketing and general business training.
Incubator staff was able to devote more time to exploring marketing channels
that would add value to products, for example, by combining gift baskets with
specialty food products produced in the region. The incubator was successful in
expanding the number of clients served and was able to leverage additional
funds to become sustainable.
68
In western North Carolina, ARC investment was used to create Blue Ridge Food
Ventures, a kitchen incubator set up to help farmers and others develop value-
added agricultural products. ARC funding provided most of their operating
budget, with other income coming from users of the facility. This relatively new
incubator (at the time of the interview) developed its capacity to serve its clients
by partnering with service providers in the area, such as a microenterprise
development organization, to become a “one stop shop.” The incubator staff
spent most of their time working with clients on product development, specifically
how to put a new twist on a food product.
The city of Gadsden, Alabama received ARC funds to conduct an incubator
feasibility study. The city lost a major employer, a steel mill, in 2000 – a
devastating blow to the economy. The city’s community development director
was looking for alternatives and the ARC funds provided an opportunity to
determine whether it would be feasible for the city to develop an incubator. The
study concluded that an incubator was feasible and that there were residents in
the community who had an interest in starting their own businesses. What the
study did, according to the development director, was to give the city information
and options – they could begin to think about what kind of incubator facility might
be most appropriate, where it would be located and what funding they would
need to develop the incubator. The attitude in the city changed when they saw
the possibilities associated with an incubator strategy.
Capital Access Projects
Projects in this category, in general, were designed to enhance access to capital
for entrepreneurs in the region. The programmatic goals were to create more
productive and job creating businesses and entrepreneurs who could remain in
the region. Based on ARC investment in this area, access to capital was
considered to be critical to helping entrepreneurs start and grow businesses in
the region. In addition to the EI investments in capital projects, ARC invested an
additional almost $4 million from area development, regional initiatives,
commission and co-chair funds. However, the projects were as diverse as the
sources of capital. A profile of capital access projects appears below.
Sample Capital Access Projects
Number of projects = 8
Total ARC investment = $901,340
Average ARC investment = $112,668
Minimum ARC investment = $30,000
Maximum ARC investment = $447,440
Range of years for project investment = 1 – 6
69
The geographic regions served by these projects ranged from individual
communities to multi-county and even multi-state service areas. While some
projects were one-time investments by ARC, others represented multi-year
commitments, with additional investments phased in as the project progressed.
All of the capital projects in the sample were judged to be sustainable by project
leaders, i.e., the projects continued beyond the period of the ARC grant.
ARC investments were used in diverse ways, including:
Support for research and planning to explore the creation of alternative
financial institutions, such as New Market Venture funds, Community
Development Financial Institutions (CDFIs), and angel capital networks
Support for technical assistance and outreach associated with revolving
loan funds
Investment in capital programs ranging from revolving loan funds and
microenterprise programs to equity funds.
Several examples serve to illustrate the ways in which many of these capital
projects were supported by ARC. Appalachian Community Enterprises (ACE), a
non-profit microenterprise program started in northeast Georgia in 1997, received
initial funding from ARC in 1999 to support a loan fund, including funds for
technical assistance and training. Similar follow-on grants were made to ACE in
subsequent years. Building on this base of activity and their track record of
lending, in 2004, ACE received CDFI certification and significant funding from
ARC’s Area Development program for their microenterprise loan program.
In 1998, the South Carolina Appalachian Council of Governments received a
modest initial investment from the EI to support the development of a loan fund
and technical assistance and outreach to rural entrepreneurs. Complementing
this EI investment was significant funding in 1998 under ARC’s Regional
Initiatives to recapitalize the loan fund. Additional support for the revolving loan
fund through ARC’s Area Development program was received in 2004.
In 2003, Advantage West, a regional economic development organization in
western North Carolina, received funding from ARC’s EI to develop a regional
entrepreneurship network. Funding was used to put on workshops throughout the
region, to develop a website, and to complete a market assessment for a
regional investment fund. During this time, Advantage West assisted a number of
entrepreneurs in acquiring capital from angel investors. A second grant was used
to develop a regional angel investor network that continues to operate in the
region.
REPORTED METRICS
From its inception in 1997 through 2005, ARC made almost $43 million in
entrepreneurship-related investments in the region, including investments made
70
specifically by the Entrepreneurship Initiative. As Table 6.2 shows, annual
investments increased from inception in 1997 to a peak in 2001, with gradual
reductions in annual investments since then. This pattern of investment
represents the phasing in and out of the EI and, in more recent years, the
inclusion of entrepreneurship investments in ARC’s Asset-Based Initiative. Total
investments were made from a number of sources within ARC, with the most
significant investment coming through the EI (Table 6.3). Total ARC investment,
in turn, leveraged an additional $72.8 million in private investment (for those
projects that have closed) with projected total private leveraged investment of
$109.9 million once all projects in the portfolio have closed.
Table 6.3. ARC Investments from All Sources – Universe of Projects, 1997-
2005
SOURCE OF FUNDS ARC $
Entrepreneurship Initiative $22,519,996
Area Development 11,603,420
Distressed Counties 3,722,992
Commission EI 2,126,380
CoChair Fund 1,553,887
Regional Initiatives 1,368,439
New Markets 52,574
Goal Fund 24,000
Total $42,971,688
ARC investments were made in 340 unique projects across the region (Table
6.4). The distribution of both dollars and projects varied across the region,
ranging from a high of 48 projects funded in Virginia to a low of nine projects
funded in South Carolina. On average, investment per state was $3.3 million and
investment per capita was $1.82 from 1997 through 2005.
In addition to projects funded in individual states, ARC made investments in a
number of projects that had a region-wide focus. For example, regional projects
included support for:
A microenterprise conference
The development of regional community development venture capital
funds, including support for technical assistance
A survey of Appalachian business incubators
The creation of entrepreneurship education materials, including working
with REAL and Junior Achievement
71
A regional conference on entrepreneurship education and training
A regional technology commercialization initiative
The development of sector-based entrepreneurship initiatives.
Table 6.4. Distribution of Entrepreneurship Projects Funded from All ARC
Sources, 1997-2005
*
STATE # CLOSED PROJECTS # OPEN PROJECTS # TOTAL
PROJECTS
Alabama 23 5 28
Georgia 7 3 10
Kentucky 22 9 31
Maryland 11 3 14
Mississippi 15 2 17
New York 17 4 21
North Carolina 5 6 11
Ohio 37 3 40
Pennsylvania 15 6 21
South Carolina 4 5 9
Tennessee 11 2 13
Virginia 42 6 48
West Virginia 24 13 37
Region 30 10 40
Total 263 77 340
*These totals include unique projects only.
Table 6.5 shows total and average investment in the universe of ARC projects
(both open and closed projects) as well as in the evaluation sample. Average
investment per project for the universe was $126,387, and projects ranged in
size from $2,000 to almost $2.2 million.
80
ARC invested almost $13 million in the
specific projects included in the evaluation sample, with average investment per
project of $145,997. Sample projects were drawn from those projects that were
primarily funded with Entrepreneurship Initiative dollars, as opposed to
80
The $2.2 million project investment was made in Pennsylvania, where the state chose to invest
in building regional assistance networks across the state. While this project was treated as a
single unique project, the funds flowed to regions across the state.
72
Commission EI or CoChair funds. As a result, some of the smaller projects, such
as Springboard Awards for Youth Entrepreneurship, were excluded from sample
selection providing a slight upward bias on project size. However, the range of
projects shows that the sample includes both small and large projects.
Table 6.5. Dollars Invested in Entrepreneurship Initiative Projects, from all
sources – Universe and Sample
ARC $
Universe of Projects
Total 42,971,688
Mean 126,387
Min 2,000
Max 2,177,326
Sample Projects
Total 12,847,733
Mean 145,997
Min 10,000
Max 2,177,326
ARC collected outcome data as part of the final reporting requirements for each
project. Table 6.6 shows actual and projected jobs created, jobs retained, new
businesses created, and businesses served for the universe of unique projects,
as reported in the close out documents submitted to ARC and included in the
database provided by ARC. In total, 9,156 jobs were created, 3,022 jobs
retained, 1,787 new businesses created, and 8,242 businesses served across
the region between 1997 and 2005. On average, projects created almost 27
jobs, retained almost 9 jobs, created 5 new businesses, and served 24
businesses. The 9,156 jobs were created at a cost, in terms of ARC funds
invested, of $4,693. As discussed in more detail in Chapter 7, this figure
compares favorably to other economic development efforts.
Table 6.7 shows these same metrics for those projects included in the evaluation
sample. Collectively, sample projects created 4,332 jobs, retained 1,351, created
1,083 new businesses, and served 2,957 businesses. On average, sample
projects achieved greater outcomes than the universe of projects in terms of job
creation (49), job retention (15), new business creation (12), and businesses
served (almost 34). This variation can be explained by two factors. First, the
universe of projects includes some of the smaller projects funded from ARC
sources other than the EI, likely reducing the overall impact numbers for the
73
universe. Second, the evaluation team was able to capture through follow-up
interviews those ongoing impacts achieved by projects that were sustainable
beyond the ARC investments. These job and business creation numbers,
therefore, represent a more accurate view of the impact of ARC investment than
those developed strictly from close out reports submitted to ARC.
CONCLUSIONS ABOUT OVERALL EI PORTFOLIO PERFORMANCE
The outcome data described in this chapter suggest that ARC investments have
been successful in generating jobs and businesses within the region. Actual jobs
created and retained and number of businesses served exceeded the projections
or goals established by the projects in their funding applications to ARC for the
sample of projects; only new business creation numbers fell short of
projections.
81
Sample data provide the most accurate view of the impact of ARC
investments since they reflect ongoing impacts associated with the projects.
Even considering all closed projects (where data are reported at project close out
only), ARC investments have created/retained more jobs and served more
businesses than projected.
81
It is not possible within the scope of this evaluation to determine whether the initial business
creation goals for the universe of projects were, in fact, realistic. It is possible that projects
identified business creation as a goal because it was a reporting metric for ARC, and not because
it was a realistic outcome of program investment. A deeper understanding of specific projects
would be required to address this issue. However, insights based on our analysis of the sample
projects are provided in Chapter 7.
7
4
T
a
b
l
e
6
.
6
J
o
b
C
r
e
a
t
i
o
n
a
n
d
R
e
t
e
n
t
i
o
n
,
N
e
w
B
u
s
i
n
e
s
s
C
r
e
a
t
i
o
n
,
a
n
d
B
u
s
i
n
e
s
s
e
s
S
e
r
v
e
d
f
o
r
t
h
e
U
n
i
v
e
r
s
e
o
f
P
r
o
j
e
c
t
s
*
J
O
B
S
C
R
E
A
T
E
D
A
C
T
U
A
L
J
O
B
S
C
R
E
A
T
E
D
P
R
O
J
E
C
T
E
D
J
O
B
S
R
E
T
A
I
N
E
D
A
C
T
U
A
L
J
O
B
S
R
E
T
A
I
N
E
D
P
R
O
J
E
C
T
E
D
N
E
W
B
U
S
I
N
E
S
S
E
S
C
R
E
A
T
E
D
A
C
T
U
A
L
N
E
W
B
U
S
I
N
E
S
S
E
S
C
R
E
A
T
E
D
P
R
O
J
E
C
T
E
D
B
U
S
I
N
E
S
S
E
S
S
E
R
V
E
D
A
C
T
U
A
L
B
U
S
I
N
E
S
S
E
S
S
E
R
V
E
D
P
R
O
J
E
C
T
E
D
U
n
i
v
e
r
s
e
(
3
4
0
)
T
o
t
a
l
9
,
1
5
6
1
6
,
1
9
6
3
,
0
2
2
2
,
3
8
5
1
,
7
8
7
3
,
2
0
7
8
,
2
4
2
1
2
,
0
2
5
M
e
a
n
2
6
.
9
4
7
.
6
8
.
9
7
.
0
5
.
3
9
.
4
2
4
.
2
3
5
.
4
M
i
n
0
0
0
0
0
0
0
0
M
a
x
5
2
8
1
,
0
0
0
4
3
0
3
7
2
2
6
4
1
,
0
0
0
2
,
7
1
7
2
,
5
0
0
*
T
h
e
s
e
d
a
t
a
a
r
e
d
r
a
w
n
f
r
o
m
t
h
e
A
R
C
d
a
t
a
b
a
s
e
a
n
d
d
o
n
o
t
r
e
f
l
e
c
t
a
d
d
i
t
i
o
n
a
l
j
o
b
c
r
e
a
t
i
o
n
/
r
e
t
e
n
t
i
o
n
,
b
u
s
i
n
e
s
s
c
r
e
a
t
i
o
n
a
n
d
b
u
s
i
n
e
s
s
e
s
s
e
r
v
e
d
t
h
a
t
m
a
y
h
a
v
e
o
c
c
u
r
r
e
d
a
s
p
r
o
j
e
c
t
s
c
o
n
t
i
n
u
e
d
b
e
y
o
n
d
t
h
e
p
e
r
i
o
d
o
f
t
h
e
A
R
C
g
r
a
n
t
.
T
a
b
l
e
6
.
7
.
J
o
b
C
r
e
a
t
i
o
n
a
n
d
R
e
t
e
n
t
i
o
n
,
N
e
w
B
u
s
i
n
e
s
s
C
r
e
a
t
i
o
n
,
a
n
d
B
u
s
i
n
e
s
s
e
s
S
e
r
v
e
d
f
o
r
t
h
e
S
a
m
p
l
e
o
f
P
r
o
j
e
c
t
s
*
J
O
B
S
C
R
E
A
T
E
D
A
C
T
U
A
L
J
O
B
S
C
R
E
A
T
E
D
P
R
O
J
E
C
T
E
D
J
O
B
S
R
E
T
A
I
N
E
D
A
C
T
U
A
L
J
O
B
S
R
E
T
A
I
N
E
D
P
R
O
J
E
C
T
E
D
N
E
W
B
U
S
I
N
E
S
S
E
S
C
R
E
A
T
E
D
A
C
T
U
A
L
N
E
W
B
U
S
I
N
E
S
S
E
S
C
R
E
A
T
E
D
P
R
O
J
E
C
T
E
D
B
U
S
I
N
E
S
S
E
S
S
E
R
V
E
D
A
C
T
U
A
L
B
U
S
I
N
E
S
S
E
S
S
E
R
V
E
D
P
R
O
J
E
C
T
E
D
S
a
m
p
l
e
(
8
8
)
T
o
t
a
l
4
,
3
3
2
2
,
9
3
7
1
,
3
5
1
4
4
3
1
,
0
8
3
9
8
1
2
,
9
5
7
2
,
7
5
7
M
e
a
n
4
9
.
2
3
3
.
4
1
5
.
4
5
.
0
1
2
.
3
1
1
.
2
3
3
.
6
3
1
.
3
M
i
n
0
0
0
0
0
0
0
0
M
a
x
7
5
8
4
3
4
4
1
0
0
3
1
6
2
5
1
6
4
3
6
9
5
*
T
h
e
s
e
d
a
t
a
a
r
e
d
r
a
w
n
f
r
o
m
f
o
l
l
o
w
-
u
p
i
n
t
e
r
v
i
e
w
s
w
i
t
h
p
r
o
j
e
c
t
l
e
a
d
e
r
s
a
n
d
d
o
r
e
f
l
e
c
t
a
d
d
i
t
i
o
n
a
l
j
o
b
c
r
e
a
t
i
o
n
/
r
e
t
e
n
t
i
o
n
,
b
u
s
i
n
e
s
s
c
r
e
a
t
i
o
n
a
n
d
b
u
s
i
n
e
s
s
e
s
s
e
r
v
e
d
t
h
a
t
o
c
c
u
r
r
e
d
a
s
p
r
o
j
e
c
t
s
c
o
n
t
i
n
u
e
d
b
e
y
o
n
d
t
h
e
p
e
r
i
o
d
o
f
t
h
e
A
R
C
g
r
a
n
t
.
75
CHAPTER 7
SAMPLE PROJECT OUTCOMES AND
BROADER POLICY IMPACTS
Chapter 6 reported on select metrics for the entire portfolio of projects in which
ARC invested; these data describe one level of impact the EI had on the region.
However, entrepreneurship development strategies, including those funded by
ARC, are often designed to achieve goals that extend beyond job and business
creation. In Chapter 6, the evaluation team identified an underlying set of
programmatic goals related to the conceptual model that were attributed to each
project type. In reality, each project was designed to address a unique set of
goals that were relevant to a particular rural place. To understand the full range
of project impacts, and to assess whether these projects were achieving their
goals, it was necessary to do a more in-depth evaluation of a sample of projects.
This part of the evaluation of ARC’s EI has generated a series of findings that fall
into two main categories – those related to the outcomes achieved by the
representative sample of projects in which ARC invested and those related to the
broader policy impacts on the Appalachian region associated with EI’s entire
portfolio of investments.
IMPACTS OF EI FUNDED PROJECTS
These findings are based on detailed analysis of project outcomes associated
with a sample of 88 projects that reflects the diversity, geographic reach, and
scope of the EI. As described in Chapter 4, the sample was representative of the
universe of projects receiving ARC investments between 1997 and 2005 in terms
of both program type and state.
82
Quantitative Impacts
To understand the quantitative impacts of ARC investments, the evaluation team
began by identifying three classes of metrics that were common to most projects
within each program category (Table 7.1). One class included common goals that
were articulated in project proposals but for which no outcome measures were
provided by project leaders. These are metrics that were believed to be important
during the design of the project but, for some reason, no outcome data were
collected for most projects in the category. In some cases, it may have been a
matter of definition – data collected as part of the project were not defined using
the same terminology as project goals. For example, technical assistance
providers often stated a goal of “number of businesses served” but actually
82
While the sample was judged to be representative, the evaluation team is not suggesting that
the quantitative results of this evaluation should be extrapolated to provide an estimate of overall
quantitative impacts of the EI. Rather, outcomes associated with these specific projects should be
viewed as reflective of what other projects might produce given similar capacity, assets,
leadership, etc.
76
reported “number of clients”. However, in some cases, it appears that data
considered relevant when the project was designed were not collected. For
example, entrepreneurship education providers often stated a goal of “number of
jobs created” but did not report such data as an outcome measure. While these
metrics do not provide insight into the overall performance of the EI, they were
useful in considering the development of the “best in class” metrics system
described in Chapter 9.
Table 7.1. Metrics Common within Program Categories
PROGRAM
CATEGORY
PROJECT
METRICS
INCLUDED AS
STATED
PROJECT GOALS
ONLY
PROJECT METRICS
INCLUDED AS
MEASURED
PROJECT
OUTCOMES ONLY
PROJECT METRICS
INCLUDED AS BOTH
STATED PROJECT
GOALS AND
MEASURED PROJECT
OUTCOMES
Capital Access - Businesses
served
- Businesses
created
- Businesses
created/expanded
- Jobs created/retained
Entrepreneurship
Education
- Jobs created - Number of
participants/trainees (both
students and teachers)
- Number of business
starts (student or adult)
Sectors - Businesses
served
- Number of
participants/trainees
- Businesses
created/expanded
- Jobs created/retained
Incubators - Businesses
created
- Businesses
served
- Number of current
clients
- Total clients served
- Number of graduated
firms
- Jobs created/retained
- Complete incubator
feasibility study
Technical
Assistance and
Training
- Businesses
served
- Number of
business plans
created/assisted
- Number of
trainings, seminars,
conferences
- Businesses expanded
- Number of clients
- Number of clients
retained in service area
- Business created
- Jobs created/retained
A second class of metrics included measured and reported project outcomes that
did not appear among the goals articulated for most projects in the category.
These are metrics that likely appeared relevant only after a project was
implemented or that were measured in a way that was different from the original
project goals. For example, most capital projects reported the number of
“businesses created and expanded” but had set a goal that related only to
“businesses created”. And, incubator projects generally reported “graduated
firms” as an outcome measure, but did not always include this metric as a project
goal. These measures help describe the impact of ARC investments, but are not
useful in evaluating the success of projects in achieving stated goals.
77
The third class of metrics included those that were both stated project goals and
measured and reported project outcomes. These metrics provide a means of
measuring the “success” of the portfolio of projects included within each program
category as will be described in more detail below.
Table 7.2
83
provides initial quantitative results for metrics that were common
within program categories. These results provide insight into the range of
measurable impacts associated with EI investments. Entrepreneurship education
projects, targeted to youth, exposed 11,634 students and teachers to
entrepreneurship principles. Almost 1,500 entrepreneurs participated in sector-
specific activities. Incubators served 475 clients and graduated at least 15 firms.
Training and technical assistance were provided to 1,620 entrepreneurs in the
region. While these data suggest positive outcomes for EI investments, it is
difficult to quantify these results for the entire EI portfolio of investments since
metrics varied across project types.
Some measures were found to be common across categories – jobs
created/retained and businesses created/expanded.
84
Table 7.3 provides initial
quantitative results for these metrics. These results show that EI investments did
produce positive quantifiable results in terms of both job and business creation –
metrics most often reported in evaluations of economic development projects. EI
investments in sample projects created or retained 5,339 jobs, created 248,
expanded 39, and created or expanded 324 businesses.
While the evaluation team cautions against using single performance measures
to gauge the success and impact of ARC’s EI, there is value in placing these
entrepreneurship development investments within the context of more traditional
economic development metrics – specifically, cost per job created/retained and
cost per business created/expanded. Table 7.4 presents estimates of these costs
for the ARC sample projects as well as for similar types of business development
programs. ARC public costs per job or business created compare favorably with
program investments made in a variety of similar types of programs. With the
83
Job creation/retention and business creation numbers included in Tables 7.2 and 7.3 in this
chapter differ slightly from those reported in Chapter 6, Table 6.7. Data in this chapter are more
conservative because some individual projects were excluded from this analysis if they did not
report both measured and projected outcomes. In Chapter 6, totals were calculated across the
entire sample as a collective and, therefore, no observations were dropped.
84
As with any data collection effort, there are caveats that must be stated. In conducting the
follow-up interviews with project leaders, most reported jobs created and retained as a single
category rather than separate categories. The evaluation team has chosen to report this
combined category since it was not possible for most respondents to distinguish between new
and retained jobs. In addition, one can consider the impact of a job retained within a community
or region as being equivalent, from an economic development perspective, to a new job created.
A similar caveat applies to businesses created and businesses expanded. Depending upon
program type, some projects reported combined metrics – businesses created and/or expanded –
while others, such as incubators, reported only business creation numbers. We have chosen to
report these categories as they were reported during interviews with project leaders.
78
exception of incubator costs, ARC investments produced jobs and businesses at
lower public cost than other types of investments. However, it is important to
recognize that these ARC figures reflect the cost in terms of ARC dollars and not
total dollars invested in these projects. Given that sample projects leveraged
$1.20 in private investments per $1 of ARC investments, these costs are
understated by about half. However, even inflating these cost figures to reflect
total investments shows that ARC investments compare well against similar
investments. As importantly, these job creation cost figures are relatively small
compared to the average cost per job created via industrial recruitment strategies
which can range anywhere from $7,000-$15,000 per job on an annual basis.
85
While positive, these metrics alone cannot answer the question of whether EI
projects have achieved their objectives. The evaluation team considered two
factors in evaluating the overall success of the initiative:
Whether project leaders considered the project to be successful
Whether measured outcomes exceeded stated goals (for quantitative
measures).
To assess the first factor, project leaders were asked whether they considered
the project a success, whether the project had met stated objectives, and
whether the project was sustainable over time, i.e., beyond the ARC grant period.
The evaluation team felt justified in using this key informant information since
there was no compelling reason why project leaders would not be objective in
evaluating success. For most projects, ARC funding had ended and project
leaders had nothing to gain by being overly optimistic about project results. In
addition, ARC has not continued to fund the EI as a separate initiative, so project
leaders had no incentive to overestimate impacts. Finally, project leaders were
informed that the evaluation was about the overall impacts of EI projects and not
about the individual performance of any single project. In addition, during the
interviews, project leaders shared information about both the successes and
failures associated with their activities.
Table 7.5 shows that, overwhelmingly, project leaders considered projects to be
successful and to have achieved program objectives, as determined through
follow-up interviews. Project leaders generally defined success in terms of
accomplishing the goals set out at the beginning of the project. However, they
were also likely to consider a project that had achieved some, but not all, of its
objectives as being successful. And, they were less concerned about the extent
to which a goal was met, as long as they saw some accomplishment toward the
goal. For example, oftentimes a project manager would consider a goal of
“creating 50 jobs” as being met if they had created some but not all of those jobs.
85
Peter Fisher, “The Fiscal Consequences of Competition for Capital,” Reining in the Competition
for Capital, Ed. Ann Markusen (Kalamazoo, MI: WE Upjohn Institute, 2007).
79
If project goals are used as a benchmark for assessing project success, it
becomes important to assess whether project goals, as initially defined, were
attainable. That is, were project goals reasonable and realistic? Unfortunately, it
is not possible through this ex post evaluation to determine on a project by
project basis whether goals were attainable. However, interviews with project
leaders suggest that project goals did change over time, often being revised as
the project was implemented and the overall direction refined. For instance,
some project leaders indicated that they learned over time that their goals had
been too ambitious and that they did not realize how difficult implementing the
project would be. In addition, there were qualitative results that contributed to the
project leader’s view that the project was successful, but which did not relate to
the original goals of the project. In both cases, project leaders had legitimate
reasons for declaring the project a success, based on their personal experience
and understanding of the broad set of impacts achieved.
Project leaders were generally objective in identifying the sustainability of their
projects – defined simply as whether or not the project continued beyond the
period of the ARC grant. It is important to recognize that sustainability was
defined in terms of the project itself and not in terms of the businesses or jobs
that were created as a result. Again, there appeared to be no incentive for these
leaders to overestimate sustainability and the responses to these questions
suggest that project leaders evaluated sustainability more critically than they did
success. Even when some part of a project proved to be sustainable, project
leaders most often defined the project as “not sustainable” if the most substantive
parts of the project did not continue.
86
While the perception of project leaders is an important factor to consider in
evaluating the success of the EI, follow-up interviews also generated data on
both stated goals and measured outcomes for a set of metrics. These metrics
provide quantitative information to assess the performance of the initiative. While
individual projects experienced varying levels of success in achieving stated
goals, the evaluation team chose to view the success of the EI from a portfolio
perspective, i.e., data are reported for each program category rather than for
each individual project within that category. From this portfolio perspective, it is
clear that the initiative was successful in achieving most of the common goals
identified for each program category.
Table 7.6 shows that for six of nine metrics, the stated goal was exceeded or
met. For the other three variables, the stated goal was not met. However, it is
useful to consider each of these variables in more detail. For entrepreneurship
education projects, a stated goal was new business starts. Given our original
conceptual model, the primary programmatic goal for these types of projects was
to get more entrepreneurs into the pipeline – to build entrepreneurial skills in
86
It is important to note that most of the projects that were sustainable beyond the ARC grant
continued to rely on local, state, foundation and other support. Very few of these projects were
sustainable defined as producing income sufficient to cover the operating costs of a program.
80
young people and expose them to entrepreneurship as a potential career path.
The literature is quite clear that entrepreneurship education for youth is not about
creating businesses in the short run. It is about inspiring young people to develop
skills that can lead them toward entrepreneurship in the future. For ARC’s EI,
metrics were reported for entrepreneurship education projects that lasted from
one to four years, as described in Chapter 6. Creating fewer new business starts
than projected may have more to do with a misalignment of goals and metrics
than with lack of success for this set of programs. Following students who
participated in these ARC funded projects as they advance through school and
into a career path would likely generate more accurate metrics on business
creation than could be expected from this initial, short-term glimpse of project
impacts. Unfortunately, none of these projects provided that long-term follow up.
In terms of incubator performance, one can argue that incubators are established
to help create more informed and more highly skilled entrepreneurs whose
businesses, as a result are more productive and more likely to remain in the
region. These goals could very well be achieved with no impact on job creation,
at least in the short run. In addition, the range of activities included in the
incubator program category – from grants for feasibility studies to implementation
grants – may skew job creation figures. Capacity building projects such as an
incubator feasibility study are unlikely to have any measurable impact on job and
business creation.
Similarly, one can argue that technical assistance and training projects serve to
create better skilled and informed entrepreneurs who, in turn, improve the
performance of existing businesses. While some aspiring entrepreneurs may
create businesses as a result of the technical assistance and training they
receive, new business creation is not the primary goal of many technical
assistance providers, such as some Small Business Development Centers that
work mostly with existing business owners.
The leveraging of private investment is a final quantitative measure that tracks
local support of entrepreneurship. ARC investments were described by key
stakeholders as catalytic. The ability to attract private investment, i.e., leverage,
as a result of ARC investment is one measure of the impact that these funded
projects have had on the region. Table 7.7 presents actual and projected total
private leverage for all ARC projects and sample projects. Projected leverage
includes the private investment associated with both closed and open projects,
suggesting the longer term impact associated with ARC investments. For the
ARC portfolio as a whole, private leverage rates range from 1.7:1 (actual) to
2.6:1 (projected). For the sample, leverage rates range from 1.2:1 (actual) to
1.4:1 (projected).
87
Collectively, ARC and leveraged private investments have
87
The discrepancy in leveraging rates between the universe of projects and our sample results
from four revolving loan fund projects that were not included in the sample but leveraged almost
$32 million in private investment.
81
had an impact by creating the quantitative and qualitative impacts described in
this chapter.
Conclusions Regarding Quantitative Impacts
Based on analysis of quantitative metrics for the evaluation sample, ARC’s
Entrepreneurship Initiative has had an impact on the region. Collectively, sample
projects created or retained over 5,300 jobs and created or expanded over 600
businesses. While it is not possible to determine definitively whether these jobs
would have been created without ARC investment, project leaders reported these
outcomes and indicated that ARC investment was critical to achieving these
impacts.
Business and job creation numbers tell only part of the story. Over 11,500
students and teachers participated in or received training from the sample
entrepreneurship education projects in which ARC invested. Almost 1,500
entrepreneurs participated in sector-focused activities. Another 1,620 received
training and technical assistance, while 475 were served by incubators. Every
dollar invested by ARC in these sample projects leveraged $1.20 in actual private
investment and is projected to leverage $1.40 private dollars for every ARC dollar
invested when project investments are complete. And, project leaders and others
in the region identified a number of qualitative impacts from these investments
that are having far reaching consequences for the Appalachian region. In
addition to supporting the conclusion that the EI has had an important positive
impact on the region, these observations also suggest that a portfolio of
programs like ARC’s EI requires a “portfolio of metrics” to accurately tell the story
of program impact.
8
2
T
a
b
l
e
7
.
2
.
Q
u
a
n
t
i
t
a
t
i
v
e
R
e
s
u
l
t
s
f
o
r
M
e
t
r
i
c
s
C
o
m
m
o
n
w
i
t
h
i
n
P
r
o
g
r
a
m
C
a
t
e
g
o
r
i
e
s
8
8
C
A
P
I
T
A
L
A
C
C
E
S
S
E
N
T
R
E
P
R
E
N
E
U
R
S
H
I
P
E
D
U
C
A
T
I
O
N
S
E
C
T
O
R
S
I
N
C
U
B
A
T
O
R
S
T
E
C
H
N
I
C
A
L
A
S
S
I
S
T
A
N
C
E
A
N
D
T
R
A
I
N
I
N
G
M
e
t
r
i
c
#
M
e
t
r
i
c
#
M
e
t
r
i
c
#
M
e
t
r
i
c
#
M
e
t
r
i
c
#
-
B
u
s
i
n
e
s
s
e
s
c
r
e
a
t
e
d
/
e
x
p
a
n
d
e
d
2
2
3
-
J
o
b
s
c
r
e
a
t
e
d
/
r
e
t
a
i
n
e
d
1
,
2
2
9
-
#
o
f
p
a
r
t
i
c
i
p
a
n
t
s
/
t
r
a
i
n
e
e
s
(
s
t
u
d
e
n
t
s
o
r
t
e
a
c
h
e
r
s
)
1
1
,
6
3
4
-
N
e
w
b
u
s
i
n
e
s
s
s
t
a
r
t
s
(
s
t
u
d
e
n
t
o
r
a
d
u
l
t
)
8
5
-
#
o
f
p
a
r
t
i
c
i
p
a
n
t
s
/
t
r
a
i
n
e
e
s
1
,
4
9
7
-
B
u
s
i
n
e
s
s
c
r
e
a
t
e
d
/
e
x
p
a
n
d
e
d
1
0
1
-
J
o
b
s
c
r
e
a
t
e
d
/
r
e
t
a
i
n
e
d
9
9
4
-
#
c
u
r
r
e
n
t
c
l
i
e
n
t
s
1
1
4
-
T
o
t
a
l
c
l
i
e
n
t
s
s
e
r
v
e
d
4
7
5
-
#
g
r
a
d
u
a
t
e
d
f
i
r
m
s
1
5
-
J
o
b
s
c
r
e
a
t
e
d
/
r
e
t
a
i
n
e
d
1
3
0
-
C
o
m
p
l
e
t
e
i
n
c
u
b
a
t
o
r
f
e
a
s
i
b
i
l
i
t
y
s
t
u
d
y
3
-
#
b
u
s
i
n
e
s
s
e
x
p
a
n
d
e
d
3
9
-
#
o
f
c
l
i
e
n
t
s
1
,
6
2
0
-
#
o
f
c
l
i
e
n
t
s
r
e
m
a
i
n
i
n
g
i
n
s
e
r
v
i
c
e
a
r
e
a
1
1
4
-
B
u
s
i
n
e
s
s
e
s
c
r
e
a
t
e
d
1
6
3
-
J
o
b
s
c
r
e
a
t
e
d
/
r
e
t
a
i
n
e
d
2
,
9
8
6
T
a
b
l
e
7
.
3
.
Q
u
a
n
t
i
t
a
t
i
v
e
R
e
s
u
l
t
s
f
o
r
M
e
t
r
i
c
s
C
o
m
m
o
n
a
c
r
o
s
s
P
r
o
g
r
a
m
C
a
t
e
g
o
r
i
e
s
P
R
O
G
R
A
M
C
A
T
E
G
O
R
Y
J
O
B
S
C
R
E
A
T
E
D
/
R
E
T
A
I
N
E
D
B
U
S
I
N
E
S
S
E
S
C
R
E
A
T
E
D
B
U
S
I
N
E
S
S
E
S
E
X
P
A
N
D
E
D
B
U
S
I
N
E
S
S
E
S
C
R
E
A
T
E
D
/
E
X
P
A
N
D
E
D
C
a
p
i
t
a
l
A
c
c
e
s
s
1
2
2
9
-
-
-
-
-
-
2
2
3
E
n
t
r
e
p
r
e
n
e
u
r
s
h
i
p
E
d
u
c
a
t
i
o
n
-
-
-
8
5
-
-
-
-
-
-
S
e
c
t
o
r
s
9
9
4
-
-
-
-
-
-
1
0
1
I
n
c
u
b
a
t
o
r
s
1
3
0
-
-
-
-
-
-
-
-
-
T
e
c
h
n
i
c
a
l
A
s
s
i
s
t
a
n
c
e
a
n
d
T
r
a
i
n
i
n
g
2
9
8
6
1
6
3
3
9
-
-
-
T
O
T
A
L
5
3
3
9
2
4
8
3
9
3
2
4
8
8
O
n
l
y
t
h
o
s
e
m
e
t
r
i
c
s
t
h
a
t
w
e
r
e
r
e
p
o
r
t
e
d
a
s
p
r
o
j
e
c
t
o
u
t
c
o
m
e
s
a
r
e
i
n
c
l
u
d
e
d
i
n
t
h
i
s
t
a
b
l
e
.
8
3
T
A
B
L
E
7
.
4
.
C
O
M
P
A
R
I
S
O
N
O
F
C
O
S
T
P
E
R
J
O
B
C
R
E
A
T
E
D
/
R
E
T
A
I
N
E
D
A
N
D
B
U
S
I
N
E
S
S
C
R
E
A
T
E
D
/
E
X
P
A
N
D
E
D
–
A
R
C
S
A
M
P
L
E
P
R
O
J
E
C
T
S
A
N
D
O
T
H
E
R
R
E
P
O
R
T
E
D
P
R
O
J
E
C
T
S
A
R
C
S
A
M
P
L
E
P
R
O
J
E
C
T
S
A
E
D
A
P
R
O
J
E
C
T
S
P
E
N
N
S
Y
L
V
A
N
I
A
S
M
A
L
L
B
U
S
I
N
E
S
S
D
E
V
E
L
O
P
M
E
N
T
C
E
N
T
E
R
S
G
I
N
C
U
B
A
T
O
R
S
H
E
N
T
E
R
P
R
I
S
E
F
A
C
I
L
I
T
A
T
I
O
N
I
N
K
A
N
S
A
S
I
S
B
A
M
I
C
R
O
L
O
A
N
P
R
O
G
R
A
M
J
$
p
e
r
j
o
b
c
r
e
a
t
e
d
/
r
e
t
a
i
n
e
d
:
C
a
p
i
t
a
l
A
c
c
e
s
s
-
6
2
4
S
e
c
t
o
r
s
-
7
3
2
T
e
c
h
n
i
c
a
l
A
s
s
i
s
t
a
n
c
e
-
5
7
9
I
n
c
u
b
a
t
o
r
s
-
3
,
9
9
4
$
p
e
r
b
u
s
i
n
e
s
s
c
r
e
a
t
e
d
/
e
x
p
a
n
d
e
d
:
C
a
p
i
t
a
l
A
c
c
e
s
s
-
2
,
9
8
8
S
e
c
t
o
r
s
-
3
,
7
5
9
T
e
c
h
n
i
c
a
l
A
s
s
i
s
t
a
n
c
e
-
7
,
8
1
8
$
p
e
r
j
o
b
c
r
e
a
t
e
d
:
R
e
v
o
l
v
i
n
g
L
o
a
n
F
u
n
d
s
b
-
9
3
6
R
u
r
a
l
P
r
o
j
e
c
t
s
c
-
6
,
1
5
7
U
r
b
a
n
P
r
o
j
e
c
t
s
d
-
2
,
9
8
2
P
u
b
l
i
c
W
o
r
k
s
e
-
4
,
8
5
7
D
e
f
e
n
s
e
A
d
j
u
s
t
m
e
n
t
f
-
1
2
,
0
0
0
$
p
e
r
j
o
b
c
r
e
a
t
e
d
-
9
2
9
$
p
e
r
b
u
s
i
n
e
s
s
c
r
e
a
t
e
d
-
3
,
3
0
0
$
p
e
r
j
o
b
c
r
e
a
t
e
d
-
1
,
1
0
0
$
p
e
r
j
o
b
c
r
e
a
t
e
d
-
2
,
4
0
0
–
7
,
8
0
0
$
p
e
r
j
o
b
c
r
e
a
t
e
d
-
4
,
5
6
8
a
F
i
g
u
r
e
s
a
r
e
p
u
b
l
i
c
c
o
s
t
p
e
r
j
o
b
,
c
o
u
n
t
i
n
g
A
R
C
p
r
o
j
e
c
t
i
n
v
e
s
t
m
e
n
t
s
o
n
l
y
,
n
o
t
t
o
t
a
l
p
r
o
j
e
c
t
i
n
v
e
s
t
m
e
n
t
i
n
c
l
u
d
i
n
g
l
e
v
e
r
a
g
e
d
p
r
i
v
a
t
e
f
u
n
d
s
.
b
F
i
g
u
r
e
s
a
r
e
f
o
r
E
D
A
p
r
o
j
e
c
t
i
n
v
e
s
t
m
e
n
t
s
o
n
l
y
,
n
o
t
t
o
t
a
l
p
r
o
j
e
c
t
i
n
v
e
s
t
m
e
n
t
.
S
o
u
r
c
e
:
R
o
b
e
r
t
W
.
B
u
r
c
h
e
l
l
,
E
D
A
R
L
F
s
:
P
e
r
f
o
r
m
a
n
c
e
E
v
a
l
u
a
t
i
o
n
(
W
a
s
h
i
n
g
t
o
n
,
D
C
:
U
.
S
.
E
c
o
n
o
m
i
c
D
e
v
e
l
o
p
m
e
n
t
A
d
m
i
n
i
s
t
r
a
t
i
o
n
,
2
0
0
2
)
.
c
S
o
u
r
c
e
:
A
m
y
K
.
G
l
a
s
m
e
i
e
r
,
C
o
s
t
p
e
r
J
o
b
A
s
s
o
c
i
a
t
e
d
w
i
t
h
E
D
A
I
n
v
e
s
t
m
e
n
t
s
i
n
U
r
b
a
n
a
n
d
R
u
r
a
l
A
r
e
a
s
(
W
a
s
h
i
n
g
t
o
n
,
D
C
:
U
.
S
.
E
c
o
n
o
m
i
c
D
e
v
e
l
o
p
m
e
n
t
A
d
m
i
n
i
s
t
r
a
t
i
o
n
,
2
0
0
2
)
.
d
S
o
u
r
c
e
:
G
l
a
s
m
e
i
e
r
(
2
0
0
2
)
.
e
S
o
u
r
c
e
:
R
o
b
e
r
t
W
.
B
u
r
c
h
e
l
l
,
E
D
A
P
u
b
l
i
c
W
o
r
k
s
P
r
o
g
r
a
m
:
P
e
r
f
o
r
m
a
n
c
e
E
v
a
l
u
a
t
i
o
n
(
W
a
s
h
i
n
g
t
o
n
,
D
C
:
U
.
S
.
E
c
o
n
o
m
i
c
D
e
v
e
l
o
p
m
e
n
t
A
d
m
i
n
i
s
t
r
a
t
i
o
n
,
1
9
9
7
)
.
f
S
o
u
r
c
e
:
R
o
b
e
r
t
W
.
B
u
r
c
h
e
l
l
,
E
D
A
D
e
f
e
n
s
e
A
d
j
u
s
t
m
e
n
t
P
r
o
g
r
a
m
:
P
e
r
f
o
r
m
a
n
c
e
E
v
a
l
u
a
t
i
o
n
(
W
a
s
h
i
n
g
t
o
n
,
D
C
:
U
.
S
.
E
c
o
n
o
m
i
c
D
e
v
e
l
o
p
m
e
n
t
A
d
m
i
n
i
s
t
r
a
t
i
o
n
,
1
9
9
7
)
.
g
S
o
u
r
c
e
:
P
e
n
n
s
y
l
v
a
n
i
a
S
m
a
l
l
B
u
s
i
n
e
s
s
D
e
v
e
l
o
p
m
e
n
t
C
e
n
t
e
r
s
,
“
R
e
t
u
r
n
o
n
I
n
v
e
s
t
m
e
n
t
:
2
0
0
5
”
1
3
D
e
c
e
m
b
e
r
2
0
0
7
<
h
t
t
p
:
/
/
w
w
w
.
p
a
s
b
d
c
.
o
r
g
/
d
o
w
n
l
o
a
d
s
/
p
d
f
/
i
m
p
a
c
t
.
p
d
f
>
.
h
S
o
u
r
c
e
:
L
a
w
r
e
n
c
e
M
o
l
n
a
r
,
e
t
a
l
.
,
I
m
p
a
c
t
o
f
I
n
c
u
b
a
t
o
r
I
n
v
e
s
t
m
e
n
t
s
(
A
t
h
e
n
s
,
O
H
:
N
a
t
i
o
n
a
l
B
u
s
i
n
e
s
s
I
n
c
u
b
a
t
i
o
n
A
s
s
o
c
i
a
t
i
o
n
,
1
9
9
7
)
.
i
S
o
u
r
c
e
:
D
o
n
M
a
c
k
e
a
n
d
D
e
b
o
r
a
h
M
a
r
k
l
e
y
a
n
d
E
r
i
k
P
a
g
e
s
,
“
E
n
t
e
r
p
r
i
s
e
F
a
c
i
l
i
a
t
i
o
n
®
i
n
K
a
n
s
a
s
:
L
e
s
s
o
n
s
a
n
d
R
e
c
o
m
m
e
n
d
a
t
i
o
n
s
,
”
R
U
P
R
I
C
e
n
t
e
r
f
o
r
R
u
r
a
l
E
n
t
r
e
p
r
e
n
e
u
r
s
h
i
p
,
A
u
g
u
s
t
2
0
0
5
.
j
F
i
g
u
r
e
s
i
n
c
l
u
d
e
a
c
t
u
a
l
d
o
l
l
a
r
s
l
o
a
n
e
d
p
l
u
s
c
o
s
t
o
f
t
e
c
h
n
i
c
a
l
a
s
s
i
s
t
a
n
c
e
p
e
r
j
o
b
c
r
e
a
t
e
d
.
S
o
u
r
c
e
:
A
s
s
o
c
i
a
t
i
o
n
f
o
r
E
n
t
e
r
p
r
i
s
e
O
p
p
o
r
t
u
n
i
t
y
,
T
e
s
t
i
m
o
n
y
b
y
K
e
v
i
n
K
e
l
l
y
,
M
a
n
a
g
i
n
g
D
i
r
e
c
t
o
r
f
o
r
P
o
l
i
c
y
a
n
d
A
d
v
o
c
a
c
y
,
b
e
f
o
r
e
t
h
e
H
o
u
s
e
C
o
m
m
i
t
t
e
e
o
n
S
m
a
l
l
B
u
s
i
n
e
s
s
,
S
e
p
t
e
m
b
e
r
2
7
,
2
0
0
7
.
84
Table 7.5. Project Leaders’ Perceptions of Project Success and
Sustainability
QUESTION # RESPONDING
“YES”
% RESPONDING
“YES”
Did you think the project was a success? 87 89.7
Do you feel you achieved the objectives set
forth for this project?
85
92.9
Has the project continued after ARC funding
ended?
86
79.1
Table 7.6. Measured Success of ARC’s Entrepreneurship Initiative Portfolio
PROGRAM
CATEGORY
STATED GOAL MEASURED
OUTCOME
PERFORMANCE
(=/-/+)
Capital Access 79 jobs created/
retained
1,229 jobs
created/retained
+
Entrepreneurship
Education
4,483
participants/trainees
(students or teachers)
141 new business
starts
11,634
participants/trainees
(students or teachers)
85 new business
starts
+
-
Sectors 38 businesses
created/expanded
438 jobs
created/retained
101 businesses
created/expanded
994 jobs
created/retained
+
+
Incubators 162 jobs
created/retained
3 completed incubator
feasibility studies
130 jobs
created/retained
3 completed incubator
feasibility studies
-
=
Technical
assistance and
training
177 businesses
created
1,295 jobs
created/retained
163 businesses
created
2,986 jobs
created/retained
-
+
Table 7.7. Actual and Projected Leveraged Private Investment – Universe
and Sample Projects
ARC $ ACTUAL
LEVERAGED
PRIVATE $
PROJECTED
LEVERAGED
PRIVATE $
Universe 42,971,688 72,802,868 109,879,064
Sample 12,847,733 15,856,275 18,596,174
85
QUALITATIVE IMPACTS
There is an emerging consensus among practitioners, particularly in rural places,
that entrepreneurship development is a long-term economic development
strategy. It requires a cultural shift from a mindset of dependency to one of self-
sufficiency. In other words, this requires a change in the view that growth will only
come from decisions and investments that are controlled from outside the
community to one that growth will come from encouraging homegrown
entrepreneurs and the businesses they create. ARC’s Entrepreneurship Initiative
was designed to encourage this culture change and to build capacity within the
region to become more entrepreneurial. One might expect, therefore, that the
outcomes achieved through EI investments would extend beyond job and
business creation. Interviews with project leaders and stakeholders across the
region identified a set of cross-cutting qualitative outcomes, described below, that
suggest a much broader impact on the region than that described through the
analysis of quantitative metrics.
ARC investments raised the profile of entrepreneurship as a
development strategy, helping to change the mindset within the
region. The investments made by the EI helped to raise awareness of
entrepreneurship and to give credibility to entrepreneurship as an
economic development approach. The EI projects “opened people’s eyes
to other possibilities” and showed that there was “more to economic
development than infrastructure”. According to one interviewee, the “most
effective part was the fact that ARC recognized entrepreneurship as
economic development. That, in itself, was a major step.”
ARC investments represented “but for” money in the region,
providing start-up funding for innovative projects. The EI projects
often represented “outside the box” thinking or demonstration projects that
would not have gotten off the ground “but for” the ARC investment. In
some places, these projects served to prove a concept or approach that
then attracted additional investment. ARC investments were variously
described as being “catalytic” or “foundational” to the efforts to encourage
entrepreneurship in the region.
ARC investments leveraged additional resources that helped some
projects achieve scale and impact. As demonstrated in Table 7.7,
ARC’s EI portfolio leveraged significant private investment. To the extent
that these private funds represented resources new to the region, the
impact would indeed be positive. It is possible, however, that some of
these investments represent a reallocation of capital from one use to
another in the region. However, in the case of capital access, ARC
investment leveraged additional resources that helped to create an
industry, in this case, the venture capital industry in the region.
ARC investments facilitated networking and collaboration among
practitioners. The increased focus on entrepreneurship helped to
reinforce that practitioners were doing “good work” and served to connect
86
both people and organizations in broader regional partnerships. The
investment provided support and encouragement to what one interviewee
described as a “fragile community” of practitioners engaged in
entrepreneurship development. Practitioners realized that they had assets,
skills and opportunities to combine that would help them achieve self-
sufficiency. As one interviewee noted, “It’s amazing what we can get done
here.”
ARC investments helped to change people’s attitudes, particularly
among youth and their teachers. Project leaders noted increased
enthusiasm and a change in attitude among young people in particular.
These changes were described in a number of ways including increased
self esteem, improved performance in school, and a new “entrepreneurial
mindset” for students; increased enthusiasm and interest in pursuing their
own entrepreneurial aspirations among teachers.
These qualitative insights are drawn from interviews with key stakeholders in the
region. It was beyond the scope of this evaluation project to conduct the in-depth
field work required to accurately verify these outcomes. However, these recurring
themes were heard across our interviews, from people representing different
states and organizations – non-profit and governmental – and engaged in
different aspects of entrepreneurship development – from providing capital to
educating youth. A more rigorous assessment of these qualitative impacts would
be possible by designing a participatory evaluation approach as part of project
design. The challenges and opportunities associated with this approach are
discussed in Chapter 9 as part of the “best in class” metric system.
Conclusions Regarding Qualitative Impacts
By identifying the qualitative outcomes associated with ARC’s EI investments, a
more nuanced, in-depth picture of the impact on the region is obtained. The EI
has served to “change the conversation” – to elevate entrepreneurship as a key
component of economic development in the region. The beginning of a culture
shift is evident at the community level, where EI projects have been having
demonstrable quantitative and qualitative impacts. There is less evidence that
these impacts are translating into policy change at the state level, suggesting the
need to explore the lessons learned from the EI experience for both
entrepreneurship development, generally, and the design and management of
future ARC entrepreneurship investment activities.
What do these qualitative impacts suggest about the metrics needed to measure
the broad set of outcomes entrepreneurship investments may have? The
observation of these impacts supports the need for using a broad set of metrics
to accurately depict project outcomes. These qualitative outcome measures
could include:
87
Public investment ($) in entrepreneurship development activities (pre- vs.
post project investment)
Private investment leveraged ($) as a result of project investment in
entrepreneurship development
Perceived change in community/regional support for entrepreneurship
development (as measured through pre- and post-investment community
surveys)
Increased collaboration among support providers (as measured by the
number of partners contributing resources to entrepreneurship
development).
88
CHAPTER 8
LESSONS LEARNED FROM THE ARC ENTREPRENEURSHIP INITIATIVE
EXPERIENCE
When ARC first announced the Entrepreneurship Initiative in 1997, ARC staff
and its state and local partners had a very limited base of experience and
effective practices upon which they could build. Yet the EI investments were not
a completely new thing. In fact, many EI projects had similarities with earlier
ARC investments in capacity building or sector development strategies. In
addition, other small business development programs (such as the national Small
Business Development Center initiative) or state business support programs
(such as Pennsylvania’s Ben Franklin Partners effort) offered useful ideas and
lessons learned.
While these earlier programs provided some useful lessons, none of these
predecessors sought to combine the objectives of supporting traditional
economic development goals, such as new business starts, with a wider mission
of knitting entrepreneurial development into the mainstream of economic
development thinking and practice in Appalachia. As a result, the ARC team
was, in some sense, building the road as they traveled it.
Through the course of the EI, ARC and its program partners learned a great deal
about how to do entrepreneurial development right. These lessons were
sometimes learned through the school of hard knocks, as once promising
initiatives failed to pay the expected dividends. In other cases, successful pilot
projects were replicated throughout Appalachia and throughout the US. For
example, ARC’s early investments in developing alternative equity capital
sources were one stimulus for creation of the New Markets Venture Capital
initiative and New Markets Tax Credit program which now supports more than
$19 billion in investments in low-income communities.
Throughout the evaluation process, the research team has focused on gathering
these “lessons learned” as means to capture established best practices,
exemplary program models, as well as informal and tacit learning that has
occurred through the life of the EI program. These lessons learned should inform
future ARC investments (in entrepreneurial development and elsewhere) as well
as other federal, state, and local efforts to promote entrepreneurship.
The compiled “lessons learned” generally fall into two broad categories. One set
of lessons applies to those actively engaged in the practice of entrepreneurship
development – people who are implementing entrepreneurship education,
training and technical assistance, capital, incubator, and networking or sector
specific initiatives in their communities, regions or states. The other set of
lessons applies more directly to the design and implementation of ARC’s
Entrepreneurship Initiative and would be most useful to those seeking to create
similar or additional region-wide initiatives.
89
LESSONS FOR PRACTITIONERS – WHAT WORKS IN ENTREPRENEURIAL
DEVELOPMENT
Nearly all of the interviewees offered thoughtful lessons learned based on their
participation in EI. Because EI was something of a “new thing” for many state
and local partners, it forced many project leaders to think differently and move
out of their traditional comfort zones. As the projects progressed, the project
teams evolved in their thinking and became more sophisticated in understanding
the key ingredients for a successful regional entrepreneurship strategy.
Project leaders and other regional stakeholders emphasized several key lessons
learned. If one were concocting a recipe for successful regional
entrepreneurship projects, the following ingredients would be required.
Organizations with these attributes tended to be more successful than their
counterparts who lacked some or all of the necessary key ingredients. These
exemplary practices related to program and community leadership, a program’s
management, goals and objectives, and a program’s outreach efforts (Table 8.1).
Additional information on each of these factors is provided below.
Table 8.1. Lessons Learned for Practitioners from Evaluation of ARC’s
Entrepreneurship Initiative
LESSONS FOR PROGRAM
LEADERSHIP
LESSONS FOR PROGRAM
MANAGEMENT
LESSONS FOR PROGRAM
OUTREACH
Successful entrepreneurship
initiatives had sparkplugs or
local champions that provided
leadership for these efforts.
Local capacity was a key to
success.
Program self-sufficiency
(sustainability) and success
went hand in hand.
Entrepreneurship
development was recognized
to be a long-term process.
Successful projects altered
their goals and approaches as
conditions warranted.
Partnerships and
collaborations were important
to success.
Successful projects
celebrated and shared the
story of their success.
Lessons for Program Leadership
Successful entrepreneurship initiatives had sparkplugs or local champions
that provided leadership for these efforts.
The need for committed local leadership is a critical requirement for success with
regional entrepreneurship efforts – just as it is with other forms of economic
development. The role of this local “sparkplug” became especially important in
the EI since it was, in part, designed as a response to the decreasing dividends
generated by traditional economic development strategies based on business
recruitment and attraction. Changing thirty years of economic development
90
practice would not occur overnight. In many communities, the concepts of
entrepreneurial development were not well understood or were resisted by those
who were comfortable with the status quo as the way things had always been
done.
In the midst of this environment where the new concepts were poorly understood
or discounted, strong leadership was required. Entrepreneurial development
needed a strong “brand” and a compelling vision that would capture the
imagination of local leaders and residents. In most cases, this vision was
generated by a local champion who came to embody the new approaches and
the new vision for the region.
These leaders came from non-profit organizations, community colleges, schools,
economic development organizations, and other institutions. They included
people who were visionaries and saw the potential for entrepreneurship to be a
force for change in their communities. They embodied the characteristics of
entrepreneurs themselves – they saw opportunities, marshaled resources, were
flexible, and determined and committed to creating a new economic development
path in their part of Appalachia.
Abingdon, Virginia’s Appalachian Sustainable Development (ASD) and its leader,
Anthony Flaccavento, exemplify this pattern.
89
Founded in 1995, ASD seeks to
promote sustainable farming and forestry in Appalachian regions of Virginia and
Tennessee. Southwest Virginia had always been a center of agriculture and
forestry, but these industries, especially forestry, had never previously focused
on sustainable practices. And, for many local leaders, environmentalism and
economic development were opposing forces. In their view, environmentalists
were opposed to business and insufficiently concerned about strengthening the
local economy.
Flaccavento and ASD sought to redefine the debate by highlighting the
tremendous entrepreneurial opportunities presented by value-added agriculture
and forestry. They educated farmers about sustainable forestry practices and
enlisted their support, so that instead of simply harvesting logs, local
entrepreneurs could sustainably gather the wood, process the lumber close to
home and use it in value-added products like flooring and wood trim. The
Appalachian Sustainable Woods Processing Center, created with EI funds,
became a local symbol for home-grown initiatives. It combined exciting
entrepreneurial opportunities with a respect for home-grown traditions and
industries, and fostered a growing belief throughout the region that sustainable
business practices can and do work. In 2001, ASD created and is actively
building its own Sustainable Woods product line
90
to sell wood products that are
environmentally friendly.
89
An interview with Anthony Flaccavento can be found at PBS NOW Enterprising Ideas website,
.
90
To learn more, visit .
91
Local capacity was a key to success.
ARC has a long history of investing in local community capacity building,
91
and a
number of the EI investments sought to develop local capacity for supporting
entrepreneurs. For example, TEAM Pennsylvania sought to seed
entrepreneurial assistance networks across the state. These networks would be
affiliated with the seven economic development districts located within ARC’s
jurisdiction. This effort ultimately generated mixed results. Several of the
networks jelled, and one effort (the Northeast Pennsylvania Entrepreneurial
Alliance)
92
received a national award. Yet, few of these networks are now in
operation and their role in supporting local entrepreneurs was limited.
The TEAM Pennsylvania experience was fairly typical for EI. Because of limited
resources, EI faced significant challenges in creating new community capacity
where none had previously existed. However, one case where EI contributed to
building capacity is Tech 2020 in Tennessee. Tech 2020 was initiated in 1993 to
build on the unique regional assets in eastern Tennessee and create a high tech
industry. ARC made a series of investments totaling $1.2 million over five years
to build the capacity of Tech 2020, primarily in the area of venture investing.
Tech 2020 has grown over time into a significant economic development
organization in the state and the region, playing a major role in establishing the
Southern Appalachian Fund, one of the first New Markets Venture Capital
Companies making investments in Tennessee, Kentucky and Appalachian
Georgia and Mississippi.
When strong organizations with existing capacity were already in place, EI
investments had a catalytic effect. In most of the successful cases, a community
was home to an organization with a strong track record in other related fields of
activity and this capacity was leveraged in support of entrepreneurship
development. For example, the Kentucky Highlands Investment Corporation
(KHIC) had been supporting community development efforts in Eastern Kentucky
since 1968.
93
KHIC began as a traditional community development organization
but its mission has evolved over time. Over the years, it has become an
intermediary for many Federal lending programs, such as SBA and USDA, and
had thus developed extensive in-house financial expertise and capacity. At the
time of the EI’s introduction, KHIC was seeking to increase the supply of seed-
91
For an evaluation of these projects, see Brian Kleiner, et al., Evaluation of The Appalachian
Regional Commission’s Community Capacity-Building Projects, Final Report to the Appalachian
Regional Commission, July 2004, xi.
92
National Association of Development Organizations (NADO) Research Foundation, Business
Not as Usual: Regional Development Organizations Promote Rural Entrepreneurship
(Washington, DC: NADO, 2002), .
93
To learn more, see Deborah Markley and David Barkley, Development of an Entrepreneur
Support Organization: The Case of Kentucky Highlands Investment Corporation, RUPRI Center
for Rural Entrepreneurship, Research Case Studies Series No. 1, March 2003,
.
92
stage equity in the region. Thanks to EI investments, KHIC was able to branch
out into these new related markets. Today, the Southern Appalachian Fund,
backed in part with EI dollars, manages $12.5 million that can be invested in local
firms seeking early stage equity capital. The success with which the Southern
Appalachian Fund emerged as a developmental venture capital force in the
region was due, in part, to the strong KHIC base upon which it was built.
Lessons for Program Management
Program self-sufficiency (sustainability) and success went hand in hand.
While all ARC grantees seek to be self-sustaining, effective EI projects viewed
ARC funds as start-up investments that were not an end in themselves. Instead,
ARC dollars were used to jump-start programs that would rise or fall based on
how they performed in the marketplace – just like any other entrepreneurial
venture. Projects that had a goal of becoming self-sufficient appeared to create
better outcomes and stronger sustained efforts. Leaders of these projects
pursued sustainability by creating partnerships and finding resources to continue
to build the program beyond the ARC grant. Creating sustainable economic
development programs takes time; this lesson suggests that making self-
sufficiency an explicit project goal may result in greater priority being placed on
its achievement.
As a group, the EI’s Capital Access projects performed best in terms of
leveraging outside investments to have a sustainable impact on the region. As
described above, a developmental venture capital industry was spawned through
the efforts of organizations funded, in part, by ARC. KHIC and Tech 2020 were
able to launch the Southern Appalachian Fund (SAF) as one of six New Markets
Venture Capital companies in the country through initial support provided by
ARC. SAF received $2 million in New Market Tax Credits in 2002, and was able
to raise a total of $12.5 million from investors, including the Tennessee Valley
Authority, foundations and a number of banking institutions. The principals
behind SAF credit ARC with its creation. One noted that if EI’s director “had not
been creative, Southern Appalachian Fund would not have been created.” ARC
was also an early investor in Meritus Ventures, a $36 million Rural Business
Investment Company established in 2002. As with SAF, Meritus brings a much
needed source of equity capital to support expanding companies in the region.
Among Meritus’ investors are regional entities, such as TVA and the University of
Kentucky, private financial institutions, and some high net worth individuals. Both
SAF and Meritus are combining access to capital with business support services,
funded in part by the U.S. Small Business Administration and U.S. Department of
Agriculture.
Yet attracting new money is not the only measure of an effective and sustainable
program. Sustainability can also be generated when programs succeed in
attracting new partners, building local collaboration, and generating energy and
93
buzz about local community-building efforts. The development of Athens, Ohio’s
Dairy Barn Arts Center reflects this pattern. Thanks to a $50,000 EI grant, the
Dairy Barn Southeast Ohio Arts Center was able to build a shop to sell artwork
and crafts produced by local craftspeople. The shop soon turned a profit,
becoming self-sustaining and providing valuable income to approximately 100
local artisans. The shop has also stimulated new partnerships with local schools,
and has generated buzz about the Arts Center’s other projects. From an
entrepreneurship standpoint, the project has also helped educate local people
about the potential for “self-sufficiency through art.”
The Clinch-Powell Community Kitchens (in Treadway, TN) pursued a similar path
to sustainability. ARC’s investments aided the kitchen in purchasing some new
equipment, but more importantly, program managers invested these dollars to
improve marketing capabilities for incubator clients. As the Community Kitchen
has grown, it has developed close partnerships with the Clinch Appalachian
Artists Cooperative. As artists and food producers have built partnerships, they
have entered new markets with new combined products such as gift baskets.
Many of these products are now sold via the Appalachian Spring Cooperative, a
joint marketing effort designed to promote family farms and local artisans.
Entrepreneurship development was recognized to be a long-term process.
It takes years to produce the culture change that is a desired outcome of many
entrepreneurship development efforts. Successful project leaders recognized the
long-term nature of their endeavors and concentrated on developing the staying
power – resources, leadership, organizational capacity, community support –
needed. As one interviewee noted, “It usually takes longer to reach the critical
mass (and resultant job and business creation) than you would expect.” And,
another noted the need to “be patient, education projects don’t have immediate
results.”
By definition, the EI’s entire portfolio of youth entrepreneurship education
projects reflects this perspective. These projects rarely assessed their
performance based on traditional economic development measures of job
creation or leveraging of new investments. Appropriately, they measured
progress according to unique measures, such as the number of schools offering
entrepreneurship training or students’ increased awareness of business
concepts.
ARC sought to publicize the best regional youth entrepreneurship initiatives
through its sponsorship of the 2002 and 2003 Appalachian Youth
Entrepreneurship Springboard Awards (Table 8.2).
94
Twelve different programs
were honored for their ability to teach youth about the key components of
entrepreneurship, to develop clear and measurable outcomes that provided value
94
See Appalachian Regional Commission, Appalachian Youth Entrepreneurship Springboard
Award: 2002 and 2003 Winners, 2004, .
94
to the local community, and to create models that could be sustained at home
and replicated elsewhere.
The Springboard Award winners were located across the region, and were based
at a variety of institutions including public schools, non-profits, and vocational
training centers. Walhalla High School (in Walhalla, SC) was one of the first
Springboard awardees. The Walhalla School District requires all ninth graders to
be exposed to entrepreneurship training in their social studies class. When these
students enter high school, they can take two separate entrepreneurship-related
courses where they are introduced to basic business concepts and move on to
start their own businesses. Each year, Walhalla High School students start
dozens of new school-based companies. In 2007, a local student won the
prestigious International Young Entrepreneur of the Year Award from the
National Foundation for Teaching Entrepreneurship.
Other Springboard winners sought to use entrepreneurship education as an effort
to slow the out-migration of area youth. Virginia’s Lonesome Pine Office of Youth
used ARC funds to support the Stay for Life Project at Bush Mill. This effort
linked local youth with area volunteers who rebuilt and refurbished a local stone
mill that is now used to produce flour and corn meal. Area youth continue to
operate the mill and sell its products, and point to the effort as a great means to
learn about entrepreneurship and about the history of their community. More
importantly, they were exposed to the idea that they can enjoy a successful
economic future without having to leave their hometowns.
Table 8.2. Springboard Award Winners – 2002 and 2003
2002 WINNERS 2003 WINNERS
Estill County High School,
Irvine, Kentucky
Tupelo Middle School, Tupelo, Mississippi
ACEnet, Athens, Ohio
Walhalla High School, Walhalla, South
Carolina
Lonesome Pine Office on Youth, Big Stone
Gap, Virginia
Randolph County Vocational Technical
Center, Elkins, West Virginia
Hale County Technology Center,
Greensboro, Alabama
Monroe County High School,
Tompkinsville, Kentucky
Ripley Union Lewis Huntington High
School, Ripley, Ohio
East Stroudsburg High School-North,
Dingmans Ferry, Pennsylvania
Carroll County Public Schools, Hillsville,
Virginia
United Technical Center, Clarksburg, West
Virginia
Additional information about the Springboard Awards can be found on the ARC website athttp://www.arc.gov/index.do?nodeId=1994.
95
In addition to the Springboard Awards, ARC made investments in other youth
education programs that have proven to be sustainable. ARC invested in a
program run by the University of Alabama to bring entrepreneurship education to
a 10-county region in the state. The program was successful in using the REAL
curriculum to train students and to build school-based enterprises. The program
continued beyond ARC funding, expanding to 14 counties. Many of the students
were at risk and according to the project leader some businesses were started as
a result of the training. In another case, ARC funded the Ohio-West Virginia
YMCA to use REAL training in a summer camp for youth. Two of the important
outcomes identified were the engagement of schools, communities and an
economic development district in the project and the change in attitudes seen in
the young people for participated in the camps.
While ARC’s investment in youth entrepreneurship programs and approaches
reflects a concern for changing the culture of the region, it was less apparent that
portfolio investments were made with explicit attention to the development of
entrepreneurs over time. Many of the incubation and TA projects measured
outputs such as businesses served rather than measuring the transformation of
entrepreneurial skills or the outcomes associated with the projects, such as
improved business performance. Insights gained from the implementation of
Lyons and Lichtenstein’s Entrepreneurial League System
®
approach suggest that
the long-term transformative impact of entrepreneurship investments depends, in
part, on the development of entrepreneurs over time.
95
Successful projects altered their goals and approaches as conditions
warranted.
The EI was the first regional effort to encourage entrepreneurship development
and investments were made when the field was relatively new. Project leaders
were plowing new economic development territory and many projects were
considered demonstrations. Effective project leaders adjusted their approaches
to reflect changing demand and to overcome unexpected obstacles. As a result,
these projects achieved outcomes that were positive but, in many cases, different
from what was originally intended.
Georgia’s Appalachian Community Enterprises’ (ACE) experience during the EI
may be instructive. ACE was established to help address a pressing capital gap
facing many small businesses in North Georgia. As ACE developed its microloan
program, it soon discovered that simply making new loan funds available was an
insufficient response to the challenges facing the region’s small firms and
aspiring entrepreneurs. Many local residents lacked money skills or suffered from
poor credit ratings that made it nearly impossible for them to do business with
local banks. As a result, ACE quickly geared up its financial literacy education
efforts. Programs such as ACE’s Money Camp for Grown-Ups are now used to
help residents learn personal money management skills.
95
Gregg A. Lichtenstein and Thomas S. Lyons (2001).
96
As ACE has introduced new programs, it has also begun to enter new markets.
North Georgia is in the midst of a massive influx of Latino immigrants who are
changing the face of many of the area’s small towns. These new residents are
also very interested in starting new businesses, and ACE has now begun an
aggressive effort to provide services to the region’s Hispanic entrepreneurs.
In Coeburn, Virginia, the initial intent behind an EI grant was to provide training to
low-income people to become crafters. However, according to the project
designer, “We learned that you can’t teach people to be artists and crafters
unless they want to be.” But through the exploration associated with the ARC
grant, they learned that a significant number of musicians were seeking ways to
expand their music businesses. The project leader credits the EI with helping the
local area spawn a popular spot for bluegrass and country music. Now people of
all ages are participating in the Friday night concerts and Thursday evening jam
sessions. Crowds are coming from miles around to dance, eat, and enjoy
affordable, excellent entertainment. The community center originally started
through the ARC grant is a vintage downtown building that formerly housed Lay’s
Hardware. It is being conserved through income generated and utilized as a
mecca for music, the arts, and various art-related training sessions. They are a
site for, and one of the progenitors of, The Crooked Road, a 13-county effort to
highlight and market country music venues throughout Southwestern Virginia.
Lessons for Program Outreach
Partnerships and collaborations were important to success.
Successful projects marshaled resources by forming partnerships and
collaborating with other organizations to share resources and build capacity.
They leveraged assets and avoided duplication of efforts. These partnerships
also facilitated networking among service providers, creating a better
environment for entrepreneurs.
Nearly all of the EI projects were built on partnerships of some sort. In fact,
partnerships – at least in the form of matching dollars from state or local sources
– were a requirement of all EI grants. But, several EI projects developed
exemplary collaborations that still exist today. For example, ARC invested
$100,000 to support a regional biotechnology initiative in western North Carolina.
This project linked Advantage West (the regional development agency), NC Bio,
Buncombe County, Western Carolina University, and several other key players.
These funds allowed Advantage West to hire a full-time regional biotech
coordinator, and to also develop biotech incubator space at Ashe-Buncombe
Technical Community College. They also developed a region-wide biotech
steering committee that still operates today. This effort is now an integral part of
a statewide North BioNetwork, with Ashe-Buncombe College serving as
headquarters for the state’s BioNetwork BioBusiness Center.
97
Another example of regional collaboration can be found in the Start Smart
initiative funded by the EI in a nine-county region of Appalachian Tennessee.
Funding was provided to the Southeast Local Development Corporation to
provide technical assistance and training to entrepreneurs in these rural counties
– a region of the state that did not have a critical mass of business resources.
This initiative built on prior successful work by the Southeast Women’s Business
Center and project leaders were able to leverage this reputation to get buy-in for
the ARC-funded project. However, collaboration was not achieved through
reputation alone. The Women’s Center program director identified all the
business resource providers who served the region and personally met with each
of them. By understanding who the potential partners were – their strengths and
potential weaknesses – the director leveraged significant resources in support of
entrepreneurs throughout the region and designed a value-added program.
Successful projects celebrated and shared the story of their success.
Many projects engaged the media to help build community support as well as to
publicize their activities as part of a broader marketing campaign. Some
communities held up their successful entrepreneurs as role models. ARC was
also instrumental in sharing success through their region-wide education efforts,
e.g., conferences, and programs like the Appalachian Youth Entrepreneurship
Education Springboard award.
Kentucky’s Artisan Heritage Trails Program, headquartered in Richmond,
Kentucky, was especially savvy in its media outreach strategies. Program
leaders faced two challenges in terms of media outreach. First, they needed to
engage local artisans to participate in the program. Second, they needed to
market outside of the region to attract tourists to use the trails and visit local
crafts people. In terms of engaging local artisans, project leaders focused first on
using the Internet as a marketing tool. As local crafts people began profiting from
online sales, word of mouth did the rest of the work. More artisans signed up and
the project was off and running. The Trails program began with a small base of
82 artisans. Today, more than 600 artisans sell their wares through the program.
The Trails Programs has also been very savvy and fortunate in its efforts to
attract tourists to the region. The most recent innovation is an online trail system
that allows visitors to map out their travel plans (along 17 different trails) and also
learn more about sites and shops long the trail. But, a more important partnership
has been developed with National Geographic. The site was featured in National
Geographic Traveler magazine and its online maps can be accessed at the
National Geographic Traveler website.
98
LESSONS RELATED TO EI PROGRAM – WHAT WORKS IN PROGRAM
DESIGN AND IMPLEMENTATION
While this evaluation uncovered significant praise for ARC’s leadership in
entrepreneurship development and the value of the investments made from
1997-2005, challenges experienced with the program provide some lessons that
may contribute to the continuous improvement of this or other initiatives in the
future. Interviewees cited several key challenges that relate to the program’s
initial design, its structure for implementation, and efforts to track and
communicate its impacts (Table 8.3).
Table 8.3. Lessons Learned for Program Design and Implementation from
Evaluation of ARC’s Entrepreneurship Initiative
LESSONS FOR PROGRAM
DESIGN
LESSONS FOR PROGRAM
IMPLEMENTATION
LESSONS FOR PROGRAM
IMPACTS
Practitioners and
entrepreneurs have unique
local knowledge that can be
applied to program design and
subsequent program
refinements.
Successful initiatives brought
together related investments,
in this case, other regional
economic development or
entrepreneurship-related
investments.
Getting EI funds to local
partners was dependent upon
state program managers and
varied based on the
importance assigned to the
initiative.
The size of the ARC EI grants
placed limits on regional
impacts.
Building a broader base of
support for entrepreneurship
investments requires
continued efforts to “make the
case” to local leaders.
Programs can be improved by
embracing long-term and
locally-driven evaluation of
program outcomes and
impacts.
Lessons for Program Design
Practitioners and entrepreneurs have unique local knowledge that can be
applied to program design and subsequent program refinements.
Nearly every interviewee felt that the EI was an idea whose time had come. The
initiative was appropriately tailored for local needs, and helped introduce new
economic development concepts into the region. Nonetheless, two primary
issues emerged when program managers and other stakeholders commented on
the initial design of the EI. First, some practitioners felt they could have been
more engaged in the design of the Entrepreneurship Initiative. People in the field
felt they had limited input on how the EI was designed or on helping ARC
understand what was needed in the region. And, they felt there was no explicit
approach to engaging entrepreneurs – the customers – in program design.
Second, they also suggested that ARC encourage a more holistic approach to
entrepreneurship development. Successful programs weaved together the
various ARC initiatives into a more holistic, systems approach to development,
combining EI investment with broadband, etc. This lesson could guide ARC in
designing future programs.
99
Critics recognized that the EI, and other ARC special initiatives, do not simply
emerge out of thin air. ARC staff and state program managers are responding to
trends in the field and to their own analyses of local economic development
needs. Indeed, ARC engaged advisory groups with good representation from the
private and non-profit sectors to inform the design and implementation of the EI.
However, as ARC continues to develop future special initiatives or other
entrepreneurship-related investments, it might consider supplementing these
approaches with other methods that capture local input in a broader and more
systematic manner. Multiple methods could be employed. These could include
online surveys, public forums (e.g., listening tours or town hall meetings) that
discuss potential new program ideas, or even a process of formal petitions or
suggestions provided by state or local program partners. Of paramount
importance in any approach taken by ARC, however, is to engage the
entrepreneurs in decision making about entrepreneurship development program
design.
These techniques are all designed to create a more transparent process of
communication between local partner organizations and ARC staff. This more
open process will not only help to improve local satisfaction with ARC programs,
it will also improve their effectiveness as they become more responsive to local
markets and more cognizant of local economic development capacities.
Successful initiatives brought together related investments, in this case,
other regional economic development or entrepreneurship-related
investments.
A second challenge concerns the relationship between EI and other regional
economic development investments from ARC and other local, state or federal
partners. For some communities, EI investments served as a one-time
investment to support a youth entrepreneurship summer camp or to fund the
start-up costs of an incubator. When these projects lacked a sustainability plan
or were poorly coordinated with other regional development efforts, they tended
to fizzle out once ARC funding ceased.
Virginia’s use of EI funds presents a compelling contrast, offering a model of how
to link ARC funding into a broader regional regeneration strategy. Southwest
Virginia’s economy had long depended on mining and manufacturing as regional
drivers. At the time of EI’s inception, state and local leaders were focused on
developing new strategies for regional reconstruction. They viewed EI funds as
seed investments to support strategies around heritage tourism, sustainable
agriculture and forestry, and Main Street development. As such, they sought to
tie EI projects to the Virginia Main Street program, and also used EI dollars to
start-up marquee projects like The Crooked Road heritage music trail and the
Round the Mountain artisans’ network. In fact, six of the eight venues on The
Crooked Road tour have received ARC funds. Virginia has invested dollars from
subsequent ARC special initiatives (in broadband and asset-based development)
100
to further support this larger region-wide strategy. As one statewide observer put
it, “We see ARC as part of a mosaic, not as projects. . . . ARC programs are
tools, not ends in themselves.”
A similar process occurred in Western Maryland’s Garrett County where local
leaders linked the EI to an ongoing effort to revitalize the county’s downtown
areas. EI investments were coordinated with Main Street programs. Main Street
programs focused on issues of streetscapes, beautification, and supporting local
retail; EI dollars supported expanded business technical assistance, a new
microloan program, and creation of a business incubator. Together, these efforts
have helped revitalize downtown Oakland and the county’s other town centers.
Lessons for Program Implementation
Getting EI funds to local partners was dependent upon state leadership and
varied based on the importance assigned to the initiative.
State governments are the ARC’s key regional partners, so it is not surprising
that several dimensions of state government helped shape the EI process. The
differences between state approaches to the EI are quite striking. As we saw in
the case of Virginia, several states used EI funds to support components of a
wider regional development strategy. Other states, such as Pennsylvania,
steered investments toward certain types of programs or strategies (e.g.,
entrepreneurial assistance through networking). Finally, other states supported
grass-roots innovation and funded a host of local pilot projects.
In addition to pursuing different strategies, states also differed in terms of their
commitment to promoting entrepreneurship as a regional development strategy.
Some governors and state program managers were cool to the new approach,
while others viewed it as a critical component for community transformation. Not
surprisingly, the level of “buy-in” by state leaders had a large impact on how the
deployment, outcomes and effectiveness of the program varied by state. If the EI
was not a priority for state leaders, it was not likely to be promoted within their
state. Statewide performance of EI projects also seemed to improve when state
program managers had close ties with local community practitioners.
When it comes to state program management, there is no one best approach.
Each strategy has its pros and cons, and each strategy may also exclude some
programs from potential consideration for ARC funding. These programs may be
located in regions or may be pursuing approaches that fall outside of the state’s
primary strategic focus. State leaders will retain a central role in future ARC
project decisions, but, in the case of special initiatives, ARC might consider
setting aside some portion of funds that are open to direct application from
localities or non-profits. Further details of this proposed Innovation Fund are
presented in Chapter 9.
101
The size of ARC grants placed limits on regional impacts.
Practitioners, and state program managers, were challenged by the relatively
small pool of money allocated to the EI. State program managers noted that it
was difficult to figure out what to do with small amounts of money and it was hard
to get recognition for the EI as a result. Most of those interviewed recognized the
political necessity for ARC to spread resources throughout the region but noted
that this often resulted in too little money to achieve significant impacts. As one
stakeholder noted, the initiative sowed many seeds but the ground was not fertile
enough to grow sustainability for most of these efforts.
Table 8.4 provides a summary of state-level EI investments on an annual basis.
They indicate that ARC faced serious obstacles when seeking to transform
regions or even a region’s thinking about entrepreneurial development. ARC’s
investments were generally dwarfed by other economic development spending in
the region. For example, Kentucky spent $808 million on economic development
in 2004,
96
dwarfing ARC’s 2004 Kentucky EI investments of nearly $384,000.
Because of these disparities, ARC investments must be tightly focused on
programs that can build scale, be sustainable, and have major impact.
One interviewee referred to the “sprinkling effect” of EI investments. ARC spent
enough dollars to meet one of the EI’s key objectives – to educate state and local
leaders about the potential of entrepreneurial development as a regional
development strategy. Unfortunately, the investments were not sufficiently large
to meet ARC’s other objectives of fostering “systemic change in the economic
development landscape of the region.”
97
With average per project investments of
$126,387, the EI served mainly as seed stage funding for regional
transformation. There are some noteworthy exceptions that prove the value of
more sustained investments in program development. ARC made a series of
grants to a number of programs throughout the region that have demonstrated
sustainability after the EI investments were completed. The Shoals
Entrepreneurial Center incubator in Alabama and the Natural Capital Investment
Fund (NCIF) in West Virginia are two examples of multi-year grant making that
helped build institutional capacity that ultimately led to sustainable organizations
that continue to have an impact on their states and, in the case of NCIF, the
region.
What this lesson learned suggests for the structure of any future
entrepreneurship investments is that “next level” investments for local initiatives
that the EI has spawned are needed. Interviewed project managers remained
strongly committed to building on the momentum generated by EI. They noted
96
Mountain Association for Community Economic Development, Accounting for Impact:
Economic Development Spending in Kentucky (Berea, KY: Mountain Association for Community
Economic Development, 2005).
97
Appalachian Regional Commission, Entrepreneurship Initiative Program Summary, September
30, 2003,http://www.arc.gov/index.do?nodeId=1970.
102
that the EI had “kick-started the conversation” and had “got people thinking”
about new ways to build community prosperity. The cultural shift sought by EI’s
designers is starting to take hold. One community leader put it well, “I don’t hear
the phrase, ‘this isn’t gonna work’ anymore.” The region’s residents – and its
economic development organizations – are increasingly aware of local economic
assets and their ability to support home-grown economic development
approaches.
ARC has sought to build on this momentum through its regular programs and
ongoing Chairman’s initiatives in broadband, energy, and asset-based
development. But, it should not abandon its previous focus on local
entrepreneurial development. Indeed, as demonstrated in Chapter 5,
entrepreneurship is a key asset in the region and can be a driving force within
ARC’s asset-based development initiative. ARC was instrumental in seeding
entrepreneurship activity in the region, and it should consider some way to
continue these investments. Emerging successful programs would benefit from
“next level” funding that would allow project leaders to move these initiatives to a
level that has the potential for transformative impact.
Lessons for Program Impacts
Building a broader base of support for entrepreneurship investments
requires continued efforts to “make the case” to local leaders.
While the EI has been successful in beginning to change attitudes toward
entrepreneurship, this cultural shift is by no means universal throughout the
region. Particularly among local elected officials, capacity building and efforts to
make the case are needed.
Interviewees were quite positive about ARC’s ongoing education efforts. Under
EI, the agency sponsored several well-attended conferences on entrepreneurial
development in general, and on specific topics such as business incubation and
support for creative industries. All of these events were well attended and
received high marks for attendees. Although ARC issued specific invitations to
local elected officials in an attempt to get them to these events, for the most part,
attendees were already engaged with ARC as grantees and partners in the EI
program. ARC should consider additional outreach efforts, spread across the
region to encourage participation, as well as investments to provide workshops
or training to the region’s elected officials or those who work in government
agencies or cooperatives focused on other rural development issues (e.g., rural
telecommunications cooperatives, USDA Cooperative Extension officials,
Conservation Districts, etc.) These programs need not be funded solely by ARC
but could be developed in cooperation with relevant trade associations. Through
these partnerships, ARC could not only spread the message about the
importance of entrepreneurial development. It would also reinforce an important
message that effective entrepreneurial development requires a holistic and
103
collaborative approach to economic development. Entrepreneurial development
is not the sole province of economic developers. It requires partnerships with
elected officials, educators, local business owners, and other key stakeholders in
the region.
Programs can be improved by embracing long-term and locally-driven
evaluation of program outcomes and impacts.
Practitioners would have benefited from ongoing assessment of project
outcomes and follow up from ARC to share lessons learned and support mid-
course changes as needed. It was clear that a better metrics system, both in
terms of defining relevant outcomes and collecting and reporting data, was
needed.
This finding echoes the results of Westat’s 2004 evaluation of ARC’s community
capacity-building projects which found that community program managers were
ill-equipped to track, measure, and publicize the outcomes of their programs and
ARC investments.
98
In EI’s case, few projects tracked metrics beyond those
required as part of the ARC grant. This measurement challenge was identified as
part of the initial evaluation of ARC’s EI investments, completed early in the life of
the initiative.
99
This early study observed that the internal evaluation and
monitoring systems used in the sample projects lacked “specific outcome
measures.” While most projects they studied had created an evaluation system,
“43 percent of all projects cited ‘monitoring outcomes’ as a problem.” The
persistence of this measurement challenge suggests the need for corrective
action for future entrepreneurship investments.
As noted in Chapters 3 and 4, these shortcomings are not unique to ARC
projects or to the general field of entrepreneurial development. In fact, because
of their short life spans, few entrepreneurial development programs have been
subjected to rigorous performance measurement and assessment.
100
Moreover,
most ARC grantees had limited budgets and limited staff who were primarily
focused on delivering programs and serving customers. As a result, few
programs could devote sufficient attention to performance measurement.
Given the resource and time pressures facing program managers, ARC cannot
expect them to embrace rigorous performance measurement and should instead
seek to create incentives for more effective program assessment. ARC could set
aside some portion of all grant funds (1-2%) to support program measurement.
ARC could also develop a program measurement “toolkit” that helps walk all
98
Brian Kleiner, et al., Evaluation of The Appalachian Regional Commission’s Community
Capacity-Building Projects, Report to the Appalachian Regional Commission, July 2004.
99
Regional Technology Strategies, Inc., Evaluation of the Early Stages of the Appalachian
Regional Commission’s Entrepreneurship Initiative, Report to the Appalachian Regional
Commission, December 2001.
100
Organization for Economic Cooperation and Development, OECD Framework for the
Evaluation of SME and Entrepreneurship Policies and Programmes (Paris: OECD, forthcoming).
104
grantees through the evaluation process. Finally, ARC could designate resource
experts among its own staff (or partner organizations) who can provide technical
assistance (when needed) on pressing performance measurement issues. These
recommendations are further discussed in the next chapter.
1
0
5
T
a
b
l
e
8
.
4
.
A
n
n
u
a
l
a
n
d
T
o
t
a
l
A
R
C
E
n
t
r
e
p
r
e
n
e
u
r
s
h
i
p
I
n
v
e
s
t
m
e
n
t
s
,
1
9
9
7
-
2
0
0
5
,
b
y
S
t
a
t
e
(
$
)
S
T
A
T
E
1
9
9
7
1
9
9
8
1
9
9
9
2
0
0
0
2
0
0
1
2
0
0
2
2
0
0
3
2
0
0
4
2
0
0
5
T
O
T
A
L
A
l
a
b
a
m
a
0
1
5
9
,
0
0
0
3
8
1
,
6
6
3
5
3
7
,
1
8
7
1
9
8
,
8
5
0
4
0
5
,
0
0
0
9
2
,
0
0
0
5
1
8
,
8
9
2
1
1
2
,
6
7
5
2
,
4
0
5
,
2
6
7
G
e
o
r
g
i
a
1
1
0
,
0
0
0
0
4
4
1
,
8
2
6
0
5
8
1
,
5
2
9
3
4
9
,
4
4
2
8
3
,
0
0
0
7
2
3
,
7
4
7
3
3
4
,
1
1
8
2
,
6
2
3
,
6
6
2
K
e
n
t
u
c
k
y
9
2
,
7
6
9
2
2
6
,
0
0
0
7
8
7
,
5
0
0
2
9
0
,
5
3
2
5
1
0
,
5
0
0
2
2
5
,
3
6
2
6
5
7
,
8
8
0
3
8
3
,
7
9
4
5
8
7
,
1
9
8
3
,
7
6
1
,
5
3
6
M
a
r
y
l
a
n
d
0
2
0
7
,
0
0
0
4
5
5
,
2
0
0
5
4
8
,
9
1
8
3
1
2
,
3
7
9
1
3
9
,
0
0
0
9
9
,
0
0
0
1
3
0
,
0
0
0
4
5
,
0
0
0
1
,
9
3
6
,
4
9
7
M
i
s
s
i
s
s
i
p
p
i
0
2
2
0
,
5
7
7
3
0
,
0
0
0
1
6
4
,
9
7
0
5
9
4
,
8
3
5
1
5
7
,
0
0
0
0
2
0
0
,
0
0
0
2
2
6
,
0
0
0
1
,
5
9
3
,
3
8
2
N
e
w
Y
o
r
k
0
2
4
3
,
1
4
0
1
8
7
,
6
5
4
9
2
0
,
0
0
0
7
5
,
0
0
0
2
5
,
0
0
0
1
4
5
,
0
0
0
2
6
5
,
0
0
0
3
5
,
0
0
0
1
,
8
9
5
,
7
9
4
N
o
r
t
h
C
a
r
o
l
i
n
a
0
0
0
1
,
0
5
0
,
0
0
0
0
4
4
4
,
9
0
8
2
0
0
,
0
0
0
1
5
7
,
5
0
0
6
7
3
,
6
7
3
2
,
5
2
6
,
0
8
1
O
h
i
o
0
1
5
3
,
9
4
0
4
0
6
,
2
1
3
4
5
1
,
9
2
1
6
1
3
,
7
0
0
2
4
4
,
5
0
0
1
,
6
7
3
,
3
3
0
1
5
,
0
0
0
5
0
,
0
0
0
3
,
6
0
8
,
6
0
4
P
e
n
n
s
y
l
v
a
n
i
a
3
1
3
,
3
9
7
1
,
0
7
2
,
8
0
8
1
,
0
9
9
,
0
0
0
1
9
9
,
9
0
0
6
5
9
,
6
3
6
3
8
3
,
7
5
0
3
6
2
,
0
0
0
4
6
0
,
0
0
0
1
2
5
,
0
0
0
4
,
6
7
5
,
4
9
1
S
o
u
t
h
C
a
r
o
l
i
n
a
0
5
0
0
,
0
0
0
0
7
5
0
,
0
0
0
1
1
9
,
1
0
5
7
5
2
,
0
0
0
5
0
0
,
0
0
0
8
9
1
,
3
8
2
5
8
,
7
5
0
3
,
5
7
1
,
2
3
7
T
e
n
n
e
s
s
e
e
0
4
5
1
,
7
4
8
3
2
5
,
0
0
0
3
9
0
,
0
0
0
4
1
0
,
1
8
5
1
8
5
,
0
0
0
9
3
,
0
0
0
2
0
1
,
1
3
3
3
5
,
0
0
0
2
,
0
9
1
,
0
6
6
V
i
r
g
i
n
i
a
3
7
4
,
5
0
0
0
4
9
5
,
3
8
0
4
1
6
,
5
2
7
4
7
6
,
6
3
0
4
3
3
,
0
0
0
1
8
1
,
2
0
0
2
8
0
,
3
9
1
9
0
,
0
0
0
2
,
7
4
7
,
6
2
8
W
e
s
t
V
i
r
g
i
n
i
a
1
4
2
,
1
5
0
7
5
2
,
9
4
8
1
,
0
7
5
,
9
1
6
3
0
0
,
5
3
7
1
,
6
7
3
,
7
4
8
1
,
0
4
2
,
8
3
5
5
8
2
,
0
0
0
4
5
8
,
4
0
0
1
,
1
0
5
,
4
7
4
7
,
1
3
4
,
0
0
8
R
e
g
i
o
n
a
l
1
4
,
0
0
0
3
3
2
,
5
0
3
1
2
4
,
6
2
3
2
6
4
,
9
0
1
1
6
9
,
1
6
2
7
9
,
6
1
0
9
7
3
,
5
7
4
1
0
8
,
0
6
3
7
5
,
0
0
0
2
,
4
0
1
,
4
3
6
T
o
t
a
l
A
R
C
1
,
0
4
6
,
8
1
6
4
,
3
1
9
,
6
6
4
5
,
8
0
9
,
9
7
5
6
,
2
8
5
,
3
9
3
6
,
3
9
5
,
2
5
9
5
,
1
2
6
,
4
0
7
5
,
6
4
1
,
9
8
4
4
,
7
9
3
,
3
0
2
3
,
5
5
2
,
8
8
8
4
2
,
9
7
1
,
6
8
8
*
F
i
g
u
r
e
s
i
n
c
l
u
d
e
a
l
l
p
r
o
j
e
c
t
s
(
c
l
o
s
e
d
a
n
d
o
p
e
n
)
a
n
d
i
n
v
e
s
t
m
e
n
t
s
f
r
o
m
a
l
l
f
u
n
d
i
n
g
s
o
u
r
c
e
s
w
i
t
h
i
n
A
R
C
.
106
CHAPTER 9
CONCLUSIONS AND RECOMMENDATIONS
CONCLUSIONS
This evaluation was designed to assess the outcomes achieved by the portfolio
of projects that received investments from ARC’s Entrepreneurship Initiative from
1997-2005, as well as the broader policy impacts that accrued to the region as a
result of this initiative. By considering a limited set of performance measures
collected for the universe of closed projects and a more detailed set of measures
and insights gained from in-depth study of a sample of projects, the evaluation
team was able to develop a thorough understanding of what EI investments have
meant to individual projects and the region as a whole. Job and business
creation have been important and significant outcomes of EI investments, as
summarized in Table 9.1 below. Over 12,000 jobs have been created or retained
and over 1,700 new businesses created, at a cost that suggests an efficient
allocation of resources relative to other similar types of economic development
programs. However, these measures tell only part of the story of impacts. EI
investments have helped to train teachers and expose students to
entrepreneurship concepts, almost 12,000 throughout the region. These
investments have been instrumental in attracting almost $73 million in private
investment to support entrepreneurship development in the region. And, through
their educational investments and the demonstration effect of the projects
funded, ARC has made entrepreneurship a legitimate and desirable economic
development activity in local communities and raised awareness about the
importance of the entrepreneurial and small business sector to the region’s
economy.
Based on the results of this evaluation, the evaluation team concludes that the
ARC Entrepreneurship Initiative has had an impact in the region by creating more
entrepreneurs in the pipeline (through its entrepreneurship education
investments), better informed entrepreneurs and better skilled entrepreneurs
(through its technical assistance and training, incubator and sector investments).
These investments have resulted in more job creating businesses. EI
investments have also helped to create and enhance capacity to support
entrepreneurship development in the region, most prominently through
investments in equity capital funds that seeded and facilitated the creation of a
developmental venture capital industry in the region. In addition, ARC
investments have created the beginning of culture change in the region –
increased recognition of the importance of entrepreneurship as an economic
development strategy and increased support for those people and organizations
that are committing their talents and resources to pursuing an entrepreneurial
path. ARC investments have energized and empowered a new set of actors in
the region, especially non-profit organizations, who continue to provide
innovative, entrepreneurial leadership in the region.
107
Table 9.1. Summary of Quantitative and Qualitative Evaluation Findings
QUANTITATIVE OUTCOMES FOR THE UNIVERSE OF PROJECTS
Jobs created 9,156
Jobs retained 3,022
New businesses created 1,787
Businesses served 8,242
Actual private $ leveraged $72,802,868
Public cost / job created or retained $579 - $3,994
Public cost / business created or expanded $2,988 - $7,818
QUANTITATIVE OUTCOMES FOR THE SAMPLE PROJECTS
Jobs created 4,332
Jobs retained 1,351
New businesses created 1,083
Businesses served 2,957
Incubator clients served 475
Students or teachers trained 11,634
Actual private $ leveraged $15,856,275
QUALITATIVE OUTCOMES OF ARC EI INVESTMENTS
Raised the profile of entrepreneurship as a development strategy and helped change the mindset
within the region
Provided start-up funding for innovative projects that would not have happened “but for” ARC
investment
Leveraged additional resources that helped some projects achieve scale and impact
Facilitated networking and collaboration among practitioners
Helped change people’s attitudes, particularly among youth and their teachers
RECOMMENDATIONS
ARC’s Entrepreneurship Initiative was plowing new ground when first conceived
in 1997. At that time, only a handful of states and localities were experimenting
with approaches that placed entrepreneurs and their companies at the center of
economic development thinking and strategy. Today, the economic development
landscape is quite different. Entrepreneurship has become a mainstream
component of local economic development strategies and has been making
strong inroads in education at the primary, secondary, and post-secondary
levels.
In the decade since EI’s inception, the field of entrepreneurship development has
been created. An important body of knowledge about effective strategies and
programs now exists. Yet, the field must still be defined as “emerging.” Lists of
effective practices and programs have been developed,
101
but these program
ideas have not been accompanied with a rigorous approach to program design,
management, and evaluation.
102
101
For example, see Markley, Macke and Luther (2005); Dabson, et al. (2003); OECD (2003).
102
OECD (Forthcoming).
108
Because it was a forerunner of today’s programmatic innovations, ARC’s EI
experience can and should be tapped for useful ideas on how to improve the field
of entrepreneurial development. With a decade of experience under its belt,
ARC staff and partners can offer invaluable guidance to other federal, state, and
local policy makers. For example, the Department of Labor’s new WIRED
(Workforce Innovation in Regional Economic Development) initiative has
invested more than $300 million in regional projects designed in part to stimulate
local entrepreneurial activity. These program managers could learn a great deal
from the EI experience.
In an effort to strengthen future entrepreneurial development programs, the team
has developed a set of recommendations that fall into three categories:
Recommendations for entrepreneurship development investments
Recommendations for creating a “best in class” metrics system
Recommendations for program design and management.
Recommendations for Entrepreneurship Development Investments
Entrepreneurship development initiatives should include assessment of
existing capacity and capacity building activities as part of project design.
The evaluation of ARC’s EI investments highlighted the value of capacity building
– visioning, leadership development (youth and adult), asset mapping,
community engagement, and strategy development. Successful entrepreneurship
development initiatives build on existing capabilities, as was seen with Kentucky
Highlands Investment Corporation, Appalachian Community Enterprise in
Georgia, and the Shoals Entrepreneurial Center in Alabama, or create new
capacity , as evidenced by the PACERS youth entrepreneurship program in
Alabama and the Tech 2020 program in Tennessee. Therefore, including
capacity building as an integral part of entrepreneurship investments should
result in stronger, more effective initiatives.
ARC (and other federal, state or local program managers) could address this
finding in several ways. At the most basic level, it could require applicants to
provide an assessment of existing community capacity during the grant
application process. Such an assessment would be more than a simple listing of
assets. Rather, project leaders should identify their leadership team, the partner
organizations, their level of resource commitment to the initiative, and how this
existing capacity will be used in substantive ways to support regional
entrepreneurship initiatives. They could also be required to outline a plan for
enhancing resource or leadership capacity if necessary.
A second approach would create closer links between existing community
capacity building programs and entrepreneurial development investments.
Many of the goals pursued through ARC’s community capacity investments, such
109
as increased organizational capacity, enhanced skills for individuals, and
improved economic development outcomes,
103
are similar or identical to those
pursued via an entrepreneurial development strategy. Many program
interventions, such as enhanced technical assistance to business, also share
similarities. Building these linkages will be eased thanks to these close
connections. Indeed, ARC has already experimented with this approach under its
2005 Asset-Based Development Initiative which explicitly identified promotion of
civic entrepreneurship as a primary program goal. ARC might consider more
direct linkages between entrepreneurial development and other existing
programs.
Finally, ARC might consider future programs that operate according to a staged
process where initial investments are made in a host of community capacity
building efforts. Entrepreneurship development would be considered a “second-
stage” investment for communities and programs that had relatively robust local
capacity in place. This staging of investments would likely produce better
community outcomes. Not surprisingly, the EI experience indicates that the
presence of strong local capacity has a significant effect in contributing to better
economic development outcomes.
Entrepreneurship development investments should be made with a focus
on the long term.
The long-term nature of entrepreneurship development requires a long-term
approach to investment. If a goal of these initiatives is to transform the culture of
Appalachia, or another region, then a 10-year or longer time horizon is more
appropriate than the more typical one to three year grant cycle. Throughout the
interviews, local program managers commented that ARC investments were too
short-lived to generate sustainable local impacts. Programs could be initiated and
local momentum could be generated. Yet, in most cases, ARC funds ran out at
this critical point of impact, two to three years after program initiation.
This pattern creates a dilemma for program managers. Like a start-up
entrepreneur, they are intensely focused on scaling up their new programs.
Because entrepreneurial companies take time to generate outcomes (in terms of
new products or job creation), support programs are unlikely to be able to boast
of major economic development impacts during the start-up phase. They may
have only a few isolated success stories after the first year or two of operations.
Without these large (and quick) community impacts, program managers are then
handicapped in their ability to identify other sources for program funds.
ARC should consider giving preference to multi-year funding commitments that
would provide a flow of resources over a longer time horizon, assuming
performance on the part of the local partner and availability of federal funding on
the part of ARC. These dynamics do not imply that projects should have an
103
For a review, see Brian Kleiner, et al., July 2004, xi.
110
indefinite period of funding. It is appropriate that project leaders articulate a plan
for sustainability and show progress toward reaching that goal as a condition of
long-term investment. But, multi-year funding commitments would provide a
stable base from which these local projects could develop.
Entrepreneurship development initiatives receiving investments should be
market driven and practice continuous improvement.
The needs of the customer – defined as youth, entrepreneurs or communities –
must be the key drivers of entrepreneurship development. Initiatives should be
designed to include mechanisms that obtain performance feedback, allow for
mid-course corrections, and in some cases, redefine project goals, based on
what project leaders are learning from their markets. While the field of
entrepreneurship development has come a long way since the EI began in 1997,
it is still in an experimentation stage. Organizations throughout Appalachia and
the country are continually evolving new ways of working with entrepreneurs and
communities that can inform both the design of new and the improvement of
existing efforts to encourage entrepreneurship as a core economic development
strategy.
ARC should consider requiring project leaders to conduct annual performance
reviews. These reviews could be conducted as part of the ongoing evaluation
that is recommended below. And, they should include some assessment of the
project’s market – for example, assessing the experience of entrepreneurs
participating in a particular training or technical assistance program through focus
groups or customer surveys. It is recommended that ARC participate in these
reviews, providing constructive feedback and suggesting resources that local
partners might use to improve program performance. ARC could use its network
of regional and national partners to link a local project with a more experienced
practitioner elsewhere to address a specific performance issue or concern.
Emphasis should be placed on investing in initiatives that demonstrate the
ability to form regional partnerships and collaborations.
With capacity and resource limitations a reality for most organizations, the ability
to form dynamic and effective partnerships that share resources becomes
paramount to success. These cross-organizational and cross-regional
collaborations should be emphasized in the design of entrepreneurship
initiatives, and effective partnerships should be rewarded as part of the
investment process. ARC should require extensive community partnerships for
all of its future entrepreneurial development investments.
However, ARC must recognize that it take resources to facilitate collaboration.
While local projects should be required to demonstrate true collaboration across
geography (e.g., multi-county projects) and organizations (e.g., public, private
and non-profit partners), ARC should consider requiring that project leaders
111
demonstrate budgetary commitment to cover the costs of collaboration. This
resource commitment could come through allocating part of ARC funds to
support collaborative activities or through the allocation of local matching dollars
to this process. ARC can help further true collaboration by making it a priority in
grant making and by educating local partners to the resource realities associated
with collaboration.
Recommendations for Creating a “Best in Class” Metrics System
Job creation is an overused metric, paints an incomplete picture of the
outcomes of entrepreneurship development investments, and should be
replaced by an “entrepreneurship development metrics portfolio.”
ARC’s existing performance measurement system requires all projects to report
the following relevant measures as part of its final project close-out process:
Businesses Served
Jobs Created
Jobs Retained
Project Participants
New Businesses Created.
These metrics provide some useful insights, but most project managers and
outside experts felt they were poorly suited to providing a complete picture of the
EI’s impact. They were better tailored to measure the impact of more traditional
economic development investments in physical infrastructure. In recognition of
these shortcomings, nearly all local projects devised their own performance
indicators which ranged from simple additions such as use of customer
satisfaction surveys to more extensive systems that tracked the financial
performance of assisted companies.
While job creation is reported as a result of ARC’s EI investments, a much richer
understanding of the initiative’s impact has come through efforts to define and
capture outcomes as measured by a broad set of performance metrics. These
metrics include both quantitative output measures, e.g., students trained, as well
as more qualitative measures, such as enhanced leadership capacity and
resources leveraged through partnerships. What is clearly needed is the creation
of a portfolio of metrics linked to particular types of entrepreneurship
development programs. These metrics would be outcome measures that are
clearly linked to the goals of particular projects. For example, to evaluate the
impacts of entrepreneurship education efforts, metrics would focus on measuring
the outcomes of training – whether young people are more likely to consider
entrepreneurship as a career path, whether they go on to create businesses in
the future, whether they return home to the region, etc.
112
Earlier in this report, Table 4.1 provided a portfolio of performance or outcome
metrics that can be used to measure the impacts of entrepreneurship
development investments across the program types that are part of ARC’s EI
portfolio. These outcome measures were drawn from a review of the literature
and formed a framework for this evaluation. While this framework could be used
as a basis for discussion by ARC and its grantees as part of the evaluation
partnership approach described below, Table 9.2 provides a more limited set of
metrics that might be operationalized into a “best in class” metrics system. The
proposed metrics include quantitative measures that would be collected from
project leaders, who in turn would use survey and other tools to gather data from
their customers. In addition, there are a number of metrics that will capture the
qualitative impacts that were observed through this evaluation of the EI
investments. Outcome measures to capture these qualitative changes in cultural
attitudes would need to become part of the evaluation system adopted by
grantees, using tools and techniques developed with support of ARC or another
funding organization.
Table 9.2. Proposed “Best in Class” Outcome Measures
Capital Access – Projects designed to provide
access to a range of capital resources to help
businesses start and grow and, in the process,
become stronger competitors in local, regional,
national and/or international markets.
Number of new businesses financed
(measure of business starts)
Number of jobs (FTEs) created/retained
(measure of business growth)
Percent of funded firms still in business
(measure of business performance)
Average wage/job created (measure of
business performance)
Percent change profitability (measure of
business performance)
Sectors – Projects focused on improving the
start up, growth, and performance of
businesses in a particular sector and on
growing particular sectors of the local or
regional economy. Projects included
networking activities designed to improve
business performance.
Amount of increased sales ($) attributed to
network or sector participation (measure
of business performance)
Number of jobs (FTEs) created/retained
(measure of business growth)
Increase in number of business starts in
targeted sector (measure of business start
up)
Change in total sector sales over time
(measure of growth in the sector)
Incubators – Projects focused on creating a
physical space for businesses to start up and
grow, with the goal of graduating these firms
into the local or regional economy.
Number jobs (FTEs) created/retained
while in the incubator (measure of
business starts/growth)
Number jobs (FTEs) created/retained after
graduation (measure of business growth)
Amount of capital ($) raised by tenants
(measure of business growth)
Percent of business tenants retained in
the service area (measure of local/regional
economic impact)
113
Entrepreneurship Education* – Projects
focused on exposing young people to
entrepreneurial concepts and experiences to
enhance their understanding of
entrepreneurship as a career option and to
encourage youth retention through
entrepreneurship.
Increase in awareness of business
concepts (measure of exposure to
entrepreneurship concepts)
Increase in number of participants
considering business creation as a career
option (measure of exposure to
entrepreneurship concepts)
Number of students that stay or return to
the service area (measure of impact on
youth retention)
Technical Assistance and Training –
Projects designed to build the skills of
individual entrepreneurs so that they can start
and grow their businesses and create stronger
enterprises in the local and regional economy.
Number jobs (FTEs) created/retained
(measure of business growth)
Number of clients still in business
(measure of business performance)
Private capital ($) raised by clients
(measure of business growth)
Average wage/job created (measure of
business performance)
Percent change in profitability (measure of
business performance)
Culture Change – Projects often achieve
qualitative impacts (both intended and
unintended) that relate to changes in people’s
attitudes toward entrepreneurship, their view of
the importance of entrepreneurs to the
local/regional economy, and the value placed
on collaborative decision making and
partnerships to create a more supportive
environment for entrepreneurs.
Public investment ($) in entrepreneurship
development activities (pre vs. post project
investment)
Private investment leveraged ($) as a
result of project investment in
entrepreneurship development
Perceived change in community/regional
support for entrepreneurship development
(as measure through pre and post-
investment community surveys)
Increased collaboration among support
providers (as measured by the number of
partners contributing resources to
entrepreneurship development)
* The metrics developed for entrepreneurship education projects refer to potential outcomes of
these projects as economic development initiatives. Therefore, metrics focus on outcomes that
have potential impacts on the community and not just the individual young person. These
individual outcomes have been measured through metrics related to changes in student
performance, e.g., increased test scores, increased applications to college, improved reading,
and increased leadership activities in school/community.
A “best in class” metrics system requires investment in a “best in class”
evaluation system.
Performance measurement should be viewed as an integral part of program
development – from the perspective of funding agencies like ARC and project
leaders. One of the first steps in developing any initiative needs to be an
articulation of program goals – what are you trying to achieve – followed by
identification of how success or performance will be measured. These inputs
form a performance measurement system that can be used by local project
leaders to report on success, broaden support, and attract additional resources
and partners to the effort. From ARC’s perspective, developing the evaluation
114
framework before investment will help to insure that individual projects contribute
to the overall goals set forth by ARC and that the agency will have appropriate
metrics to use to report on the performance of the overall initiative.
Ex post evaluations of major investments like ARC’s EI face serious challenges
in terms of the collection and integrity of data; recalling accurately the impacts of
a project that ended 7-10 years ago was a challenge for most project leaders.
Greater investment must be made in establishing criteria and providing funding
for an ongoing evaluation of entrepreneurship development initiatives as they
unfold. As noted above, local grantees regularly developed their own in-house
metrics to supplement those required as part of ARC’s grants process. While
these in-house efforts generated a fair amount of useful data, they provided no
means to aggregate data across program types, across regions, or across the
entire EI spectrum.
ARC should support the creation of a performance measurement system for
future investments by developing a participatory evaluation system in partnership
with grantees. This measurement system would be developed by grantees and
their customers, i.e., the entrepreneurs, with support from ARC, and would be
designed to provide project leaders with useful information that can be used to
adapt programs to changing circumstances as well as to report to ARC on project
performance. The evaluation framework should be built into the program from the
beginning, and project leaders would be expected to sign off on that evaluation
system as part of a grant agreement. By taking this partnership approach to
evaluation, ARC would be in a stronger position to hold project leaders
accountable for generating the outcome metrics identified for the project, and to
provide them with feedback on performance.
The evaluation team also recommends that ARC consider two sets of outcome
metrics: a base set of metrics for all programs, and a tailored set of metrics for
each specific type of program intervention, as laid out in Table 9.2. For example,
an entrepreneurship training program and a new business incubator might both
be assessed according to traditional metrics of job creation or new business
starts. Beyond these base measures, incubators might be assessed according to
the number of firms graduated from the facility, the incubator’s annual revenue,
and a range of financial metrics for businesses served by the incubator. Training
programs might use a common customer satisfaction survey or other tools that
assess whether participants gained new skills or knowledge, supplemented by
follow-up surveys to collect financial performance metrics for business
customers. Youth entrepreneurship education programs might develop metrics
related to measuring the entrepreneurial skills acquired by young people who
participate in these programs.
ARC has an opportunity to take the lead, among any newly authorized regional
authorities and other federal agencies, in developing and standardizing
assessment criteria and methods that can have an impact on how
115
entrepreneurship development investments are evaluated and, more broadly,
how the impacts of economic development programs are measured.
Recommendations for Program Design and Management
ARC’s initiative process should be regularized so that state program
managers can more effectively plan for and promote use of the resources.
Interviews indicate that state program managers and local grantees like the ARC
initiative process. They like the focus on specialized issues and concerns, and
they like the learning opportunities provided by ARC through various sponsored
conferences and reports. They are less enamored with the episodic nature of the
initiative process. They would like more input into the discussions about new
initiative topics, and more information on initiatives being considered in the
future.
Given the gate keeping role of state leaders, ARC should provide structure and
consistency to the initiatives to encourage the active buy-in of people at the state
level. A more transparent and open process for community input on project
design and implementation would also help ARC create more effective initiatives
and empower local people and organizations to actively participate in these
efforts.
As noted in Chapter 8, ARC went through an extensive process of getting input
into the design of the EI. Advisory groups of state leaders and private/non-profit
sector practitioners and others with experience in entrepreneurship development
helped to inform the decision making behind the EI. Meetings and educational
events throughout the region provided opportunities for state and local leaders to
provide input to the process. In spite of these significant steps, there remains a
sense that local practitioners had limited input into the design and
implementation of the EI. One way to address this perception might work as
follows. As ARC’s leadership considers new topics for potential initiatives, it can
open a process for outside input. Ideally, suggestions should be provided in
multiple formats from a formal request for comment in the Federal Register to the
use of blogs as a means to generate online discussion. In-person sessions, such
as town hall meetings held in widely dispersed locations throughout the region,
should also be considered. ARC should consider tapping into the ever growing
infrastructure of entrepreneurship service providers in the region, using online
surveys to get their input on what is needed in the region. The US Department of
Agriculture’s series of sponsored Farm Bill Forums, to discuss key sections of the
2007 Farm Bill, offers one excellent model for organizing public outreach and
discussion.
104
104
Summaries of the Farm Bill Forums, held throughout 2006, can be found at
.
116
ARC’s proven experience can be applied to developing and delivering
effective, region-wide education programs that help make the case for
entrepreneurship as a core economic development strategy for the
Appalachian region.
There continues to be a strong need to make the case for entrepreneurship,
particularly among local elected officials and traditional economic developers.
Throughout the EI, ARC has organized educational opportunities to share
information about topics such as capital access, business incubation, and
entrepreneurship education. What is needed now, however, is a broader effort to
provide community leaders, elected and others, with the understanding and tools
they need to embrace entrepreneurship as part of an economic development
strategy.
ARC has developed a reputation as a trusted authority in this field and also has
the lessons learned from ten years of entrepreneurship development investments
to share throughout the region. In addition, ARC’s partnership approach in the
beginning of the EI process can be brought to bear on this educational effort,
drawing on resources and experiences of other organizations working on
entrepreneurship development throughout the country.
A Next Generation Entrepreneurship Initiative
This evaluation has generated a host of new ideas and lessons learned, but one
prevailing idea has emerged throughout the evaluation process – additional
investments in entrepreneurship development throughout Appalachia are still in
significant demand. Given the success and capacity that the ARC EI has already
been building, the evidence suggests that continued ARC investment in
entrepreneurship development in the region is a compellingly logical and vital
next step.
To build on this momentum, ARC should create a Next Generation
Entrepreneurship Innovation Initiative that will be groundbreaking in its
design. A long-term investment is recommended that incorporates all the
learning from the EI and the emerging entrepreneurship development field.
It will include four critical elements:
The Entrepreneurship Innovation Fund would provide selective,
competitive, strategic investments in “next level” entrepreneurship
development activities throughout the region. The Entrepreneurship
Innovation Fund would not be tied to individual states, but would be
competitively awarded across the region. Investments would be made in
initiatives that demonstrate a holistic, systems approach to
entrepreneurship development, with an emphasis on those initiatives that
have the potential to be transformational and sustainable. It is
recommended that ARC take a portfolio approach to these investments –
117
investing in more proven innovations as well as those that offer promise
but are still early stage innovations.
The second element would be a pool of funds distributed to the states for
investments in first tier entrepreneurship projects at the ground level.
Similar to the EI, these projects would build capacity and fulfill distinct
entrepreneurship metrics that are developed by the communities and ARC
working together.
In the interest of capacity building, ARC should fund the development of
“Entrepreneurship Innovation – Guidelines for the Future” – a framework
for communities to use based on what ARC has learned from 10 years of
investment in this field and what its partner organizations across the
country have learned through their various activities.
The fourth element would be a built-in evaluation system that is initiated
from the beginning of the Entrepreneurship Innovation Initiative. It would
incorporate the “best in class” metrics derived from this evaluation,
discussions with ARC, and input from the field. This evaluation system will
be essential to making the case as well as measuring and ensuring
impact.
This capstone recommendation is based on the recognition that, while the
Entrepreneurship Initiative has achieved important impacts at the community
level, the region has not seen widespread or significant policy change at the state
level. The entrepreneurship context assessment in Chapter 5 suggests that many
parts of the region continue to lag the nation, particularly in terms of income
generated by entrepreneurial activities. The Entrepreneurship Innovation
Initiative would give ARC an opportunity to make investments that are deeper
and more transformational, generating impacts that are influential in achieving
policy change at the state, as well as local, level throughout the region. This
initiative would also provide an opportunity to implement a participatory
evaluation system that can generate the data and insights that will provide a
deeper understanding of how ARC investments help to change the economic
outlook and performance of the region.
118
BIBLIOGRAPHY
Acs, Zoltan J. “The State of the Literature on Small to Medium Sized Enterprises
and Entrepreneurship in Low-Income Communities.” Federal Reserve
Bank of Kansas City and Ewing Marion Kauffman Foundation, 2006.
Acs, Zoltan J. and Catherine Armington. Using Census BITS to Explore,
Entrepreneurship, Geography, and Economic Growth. Washington, DC:
US Small Business Administration Office of Advocacy, 2005.
Advanced Research Technologies. The Innovation-Entrepreneurship Nexus.
Washington, DC: US Small Business Administration Office of Advocacy,
2005.
Aernoudt, Rudy. “Incubators: Tool for Entrepreneurship?” Small Business
Economics. Vol. 23, No. 2, September 2004. 127-135.
Alliance for Regional Stewardship. Regional Indicators: Telling Stories,
Measuring Trends, Inspiring Actions. ARS Monograph Series No. 10.
November 2005.
Appalachian Regional Commission. Appalachian Youth Entrepreneurship
Springboard Award: 2002 and 2003 Winners. Washington, DC:
Appalachian Regional Commission, 2004.
Appalachian Regional Commission. Capitalizing on Rural Communities.
Washington, DC: Appalachian Regional Commission, 2000.
Appalachian Regional Commission. Entrepreneurship Initiative: Program
Summary and Approved Projects. Washington, DC: Appalachian
Regional Commission, 2003.
Atkinson, Robert D. The Past and Future of America’s Economy.
Northhampton, MA: Edward Elgar, 2004.
Atrostic, B.K. “Measuring U.S. Innovative Activity.” US Census Bureau Center
for Economic Studies Working Paper (07-11). March 2007.
Association of University Technology Managers. AUTM U.S. Licensing Survey,
FY 2004: A Survey Summary of Technology Licensing (and Related)
Performance for U.S. Academic and Nonprofit Institutions, and
Technology Investment Firms. Washington, DC: AUTM, 2004.
Audretsch, David, et al. “The Knowledge Filter and Economic Growth: The Role
of Scientist Entrepreneurship.” Ewing Marion Kauffman Foundation,
2006.
119
Barkley, David L. et al. Establishing Nontraditional Venture Capital Institutions:
Lessons Learned. Rural Equity Capital Initiative Study of Nontraditional
Venture Capital Institutions. Columbia, MO: Rural Policy Research
Institute, 2001.
Barkley, David L. and Deborah Markley. The Development of an Entrepreneurial
Support Organization: The Case of the Kentucky Highlands Investment
Corporation. RUPRI Center for Rural Entrepreneurship Research Case
Study Series Number 1. March 2003.
Bartik, Timothy J. Evaluating the Impacts of Economic Development Programs
on Local Economic Outcomes: What Has Been Done and What is
Doable? Upjohn Institute Staff Working Paper (No. 03-89). Kalamazoo,
MI: Upjohn Institute, 2002.
Bartik, Timothy J. and Richard D. Bingham. "Can Economic Development
Programs be Evaluated?" Dilemmas of Urban Economic Development.
Eds. R.D. Bingham and Robert Mier. Thousand Oaks, CA: Sage, 1997.
246-277.
Block, Z. and S. A. Stumpf. “Entrepreneurship Education Research: Experience
and Challenge.” The State of the Art of Entrepreneurship. Eds. D. L.
Sexton and J. D. Kasarda. Boston, MA: PWS-Kent Publishing, 1992. 17-
45.
Bollingtoft, Anne and John P. Ulhoi. “The Networked Business Incubator –
Leveraging Entrepreneurial Agency?” Journal of Business Venturing Vol.
20, 2005: 265-290.
Brandow Company and Economic Development Research Group. Evaluation of
the Appalachian Regional Commission’s Infrastructure and Public Works
Program Projects, Final Report to the Appalachian Regional Commission.
Washington, DC: Appalachian Regional Commission, 2000.
Brown, J. David, John S. Earle, and Dana Lup. “What Makes Small Firms Grow?
Finance, Human Capital, Technical Assistance, and the Business
Environment in Romania.” Economic Development and Cultural Change
Vol. 54, No. 1, 2005: 33-70.
Burchell, Robert W. EDA RLFs: Performance Evaluation. Washington, DC: U.S.
Economic Development Administration, 2002.
Burchell, Robert W. EDA Defense Adjustment Program: Performance
Evaluation. Washington, DC: U.S. Economic Development
Administration, 1997.
120
Burchell, Robert W. EDA Public Works Program: Performance Evaluation.
Washington, DC: U.S. Economic Development Administration, 1997.
Carter, Sara, et al. “Beyond Portfolio Entrepreneurship: Multiple Income Sources
in Small Firms.” Entrepreneurship and Regional Development Vol. 16,
2004: 481-499.
Charney, Alberta and Gary Libecap. The Impact of Entrepreneurship Education:
An Evaluation of the Berger Entrepreneurship Program at the University of
Arizona, 1985-1999. Kauffman Center for Entrepreneurial Leadership,
2000.
Chukumba, Celestine and Richard Jensen. Invention, Entrepreneurship, and
Start-Ups. National Bureau of Economic Research Working Paper
(#11475). July 2005.
Coalition of Community Development Financial Institutions. Providing Capital
Building Communities Creating Impact. Washington, DC: CDFI Coalition,
2004.
Coastal Enterprises, Inc. Measuring Impact in Practice. Wiscasset, ME: Coastal
Enterprises Inc., February 2006.
Colgan, Charles S. and Bruce Andrews. Evaluation of Maine Technology
Institute Programs. University of Southern Maine Center for Business and
Economic Research. December 2004.
Collender, Ray et al. “Financial Markets Serve Rural Areas Fairly Well.” Rural
Development Perspectives Vol. 14, No. 1, May 1999: 28-35.
Community Development Venture Capital Association. Measuring Impacts
Toolkit. New York: Community Development Venture Capital Alliance,
2005.
Cooke, Phil. Regional Innovation Systems as Public Goods.” Working Paper.
United Nations Industrial Development Organization, 2005.
Council on Competitiveness. Measuring Regional Innovation. Washington, DC:
Council on Competitiveness, 2006.
Council on Competitiveness. Innovate America. Washington, DC: Council on
Competitiveness, December 2004
Czohara, Laura and Julia Melkers. Performance Measurement in State Economic
Development Agencies: Lessons and Next Steps for GDITT. Georgia
State University Fiscal Research Center Report No. 92. February 2004.
121
Dabson, Brian, et al. Mapping Rural Entrepreneurship. Washington, D.C.: CFED,
August 2003.
Drabenstott, Mark. A Review of the Federal Role in Regional Economic
Development. Working Paper. Center for the Study of Rural America,
Federal Reserve Bank of Kansas City, May 2005.
Eberts, Randall W., George A. Erickcek, and Jack Kleinhenz. “Development of
Regional Economic Dashboard.” Employment Research July 2006: 3-5.
Edgcomb, Elaine L. and Joyce A. Klein. Opening Doors, Building Ownership:
Fulfilling the Promise of Microenterprise in the United States. Washington,
D.C.: FIELD, A Program of the Aspen Institute, 2005.
Eisinger, Peter K. The Rise of the Entrepreneurial State: State and Local
Economic Development Policy in the United States. Madison, WI:
University of Wisconsin Press, 1988.
Elfring, Tom and Willem Hulsink. “Networks in Entrepreneurship: The Case of
High-technology Firms.” Small Business Economics Vol. 21, No. 4, 2003:
409-422.
Emery, Mary and Cornelia Flora. “Spiraling Up: Mapping Community
Transformation with Community Capitals Framework.” Journal of the
Community Development Society Vol. 37, No. 1, Spring 2006: 19-35.
Enterprise Corporation for the Delta. Enterprise Corporation for the Delta
Program Monitoring Report – Business Technical Assistance 2003.
Jackson, MS: Enterprise Corporation for the Delta, 2003.
European Commission, Directorate General, Enterprise and Industry. The
PAXIS Manual for Innovation Policy Makers and Practitioners. Brussels:
European Commission, 2006.
Fairlie, Robert W. Kaufffman Index of Entrepreneurial Activity, State Report 2005.
Kansas City: Ewing Marion Kauffman Foundation, 2006.
Fayolle, Alain. “Evaluation of Entrepreneurship Education: Behavior Performing
or Intention Increasing?” International Journal of Entrepreneurship and
Small Business Vol. 2, No.1, 2005: 89-98.
Feldman, Maryann. “The Entrepreneurial Event Revisited: Firm Formation in a
Regional Context.” Industrial and Corporate Change Vol. 10, No. 4, 2001:
861-891.
122
Felsenstein, Daniel and Aliza Fleischer. “Small-Scale Entrepreneurship and
Access to Capital in Peripheral Locations: An Empirical Analysis.” Growth
and Change Vol. 33, 2002: 196-215.
Fisher, Peter. “The Fiscal Consequences of Competition for Capital.” Reining in
the Competition for Capital. Ed. Ann Markusen. Kalamazoo, MI: WE
Upjohn Institute, 2007.
Foster, Lucia. Reallocation and Productivity Dynamics in the Appalachian
Region. US Census Bureau Center for Economic Studies Working Paper
(06-03). January 2006.
Giannetti, Mariassunta and Andrei Simonov. On the Determinants of
Entrepreneurial Activity: Individual Characteristics, Economic
Environment, and Social Norms. White Paper. Stockholm School of
Economics, June 2004.
Gilbert, Brett Anitra, David B. Audretsch, and Patricia P. McDougall. “The
Emergence of Entrepreneurship Policy.” Small Business Economics Vol.
22, 2004: 313-323.
Gillett, Sharon E., et al. Measuring the Economic Impact of Broadband
Deployment, Final Report to the Economic Development Administration.
Washington, DC: Economic Development Administration, February 2006.
Glasmeier, Amy K. Cost per Job Associated with EDA Investments in Urban and
Rural Areas. Washington, DC: U.S. Economic Development
Administration, 2002.
Goetz, Stephan J. and David Freshwater. “State-Level Determinants of
Entrepreneurship and a Preliminary Measure of Entrepreneurial Culture.”
Economic Development Quarterly Vol. 15, No. 1, February 2001: 58-70.
Greenburg, Elizabeth and Richard Reeder. Who Benefits from Business
Assistance Programs? Results of the ERS Rural Manufacturing Survey.
United States Department of Agriculture Information Bulletin Number 736-
04. 1998.
Greene, F.J. and D.J. Storey. “An Assessment of a Venture Capital Creation
Programme: the Case of Shell LiveWIRE.” Entrepreneurship and
Regional Development Vol. 16, No. 2, March 2004: 145-159.
Greenwood Consulting Group. A Survey of Business Incubators in Appalachia.
Washington, DC: Appalachian Regional Commission, July 2005.
123
Grimaldi, Rosa and Alessandro Grandi. “Business Incubators and New Venture
Creation: An Assessment of Incubating Models.” Technovation Vol. 25,
2005: 111-121.
Heard, Robert and John Sibert. Growing New Businesses with Seed and Venture
Capital: State Experiences and Options. Washington, DC: National
Governors Association, 2000.
Ho-Kim, Thu-Mai and Ernesto Venegas. Starting Up Economic Engines: Key
Factors for Growing Successful Start-Ups. Minnesota Department of
Employment and Economic Development Issue Brief. October 2003.
Huggins, R. “The Success and Failure of Policy-Implanted Inter-firm Network
Initiatives: Motivations, Processes and Structure.” Entrepreneurship and
Regional Development Vol. 12, 2000: 111-135.
Innovation Associates Inc. Accelerating Economic Development Through
University Technology Transfer. February 2005.
Institute for Competitiveness and Prosperity, Measuring Ontario’s Prosperity:
Developing an Economic Indicator System. Working Paper No. 2.
Ontario, Canada: Institute for Competitiveness and Prosperity, 2002.
Junior Achievement. The Impact on Students of Participation in Junior
Achievement: Selected Cumulative and Longitudinal Findings.
Monograph. January 26, 2004.
Katz, Jerome. And Another Thing. . . 2006 Coleman Foundation White Paper on
Entrepreneurship Education. 2006 Annual Meeting of US Association of
Small Business and Entrepreneurship, January 13, 2006.
Katz, Jerome A. “The Chronology and Intellectual Trajectory of American
Entrepreneurship Education, 1876-1999.” Journal of Business Venturing
Vol. 18, 2003: 283-300.
Kingsley, Gordon and Edward J. Malecki. “Networking for Competitiveness.”
Small Business Economics Vol. 23, 2004: 71–84.
Kleiner, Brian, et al. Evaluation of The Appalachian Regional Commission’s
Community Capacity-Building Projects. Washington, D.C.: Appalachian
Regional Commission, July 2004.
Kourilsky, Marilyn, and William B. Walstad. The E Generation. Dubuque, IA:
Kendall-Hunt, 2000.
124
Lambrecht, Johan and Fabrice Pirnay. “An Evaluation of Public Support
Measures for Private External Consultancies to SMEs in the Walloon
Region of Belgium.” Entrepreneurship and Regional Development Vol.
17, 2005: 89-108.
Lee, Lena. Entrepreneurship Education: A Compendium of Related Issues.
Working Paper. National University of Singapore Entrepreneurship Centre,
2005.
Lichtenstein, Gregg A. and Thomas S. Lyons. “The Entrepreneurial Development
System: Transforming Business Talent and Community Economies.”
Economic Development Quarterly Vol. 15, No. 1, 2001: 3-20.
Lichtenstein, Gregg A. and Thomas S. Lyons, Incubating New Enterprises: A
Guide to Successful Practice. Washington, DC: Aspen Institute, 1996.
Low, Sarah. “Regional Asset Indicators: Entrepreneurship Breadth and Depth.”
The Main Street Economist September 2004:1-4.
Low, Sarah, Jason Henderson, and Stephan Weiler. “Gauging a Region’s
Entrepreneurial Potential. “Economic Review Third Quarter, 2005.
Edward Lowe Foundation, et al. Michigan: Toward an Entrepreneurial Economy,
2005-2006. Cassopolis, MI; Edward Lowe Foundation, 2006.
Malecki, Edward J. “Geographical Environments for Entrepreneurship.”
International Journal of Entrepreneurship and Small Business
Forthcoming 2008.
Manigart, Sophie, et al. “Determinants of Required Return in Venture Capital
Investments: A Five Country Study.” Journal of Business Venturing Vol.
16, No. 6, July 2002: 291-312.
Maunula, Mari. The Perceived Value-Added of Venture Capital Investors.
Discussion Paper No. 1030. The Research Institute of the Finnish
Economy, March 2006.
Markley, Deborah, et al. Rural Equity Capital Initiative Study of Nontraditional
Venture Capital Institutions. Working Paper PB2001-11A-D. Rural Policy
Research Institute, 2001.
Markley, Deborah and Karen Dabson. Innovative Approaches to Entrepreneurial
Development: Cases from the Northwest Region. RUPRI Center for Rural
Entrepreneurship, 2006. 32-41.
125
Markley, Deborah, Don Macke and Vicki Luther. Energizing Entrepreneurs:
Charting a Course for Rural Communities. Lincoln, NE: Heartland Center
for Leadership Development, 2005.
Markman, Gideon, et al. “Entrepreneurship and University-based Technology
Transfer.” Journal of Business Venturing Vol. 20, No. 2, 2005: 241-263.
McKernan, Signe-Mary and Henry Chen. Small Business and Microenterprise as
an Opportunity and Asset-Building Strategy. Urban Institute Issue Brief
No. 3. June 2005.
McMullan, W. E. and W.A. Long. “Entrepreneurship Education in the Nineties.”
Journal of Business Venturing Vol. 18, No. 2, 1987: 261-275.
Microenterprise Fund for Innovation, Effectiveness, Learning and Dissemination.
Improving Microenterprise Training and Technical Assistance: Findings for
Program Managers. Washington, DC: Aspen Institute, 2002.
Microenterprise Fund for Innovation, Effectiveness, Learning and Dissemination.
Assessing the Effectiveness of Training and Technical Assistance. Field
Forum Issue 1. Washington, DC: Aspen Institute, 1999.
Minniti, Maria, with William Bygrave and Erkko Autio. Global Entrepreneurship
Monitor 2005 Executive Report. London: London Business School, 2005.
Molnar, Lawrence, et al. Impact of Incubator Investments. Athens, OH: National
Business Incubation Association, 1997.
Mountain Association for Community Economic Development. Accounting for
Impact: Economic Development Spending in Kentucky. Berea, KY:
Mountain Association for Community Economic Development, 2005.
Nakkula, Michael. Expanded Explorations into the Psychology of
Entrepreneurship: Findings form the 2001-2002 Study of NFTE in two
Boston Public High Schools. Working Paper. Harvard University Graduate
School of Education, 2003.
National Association of Development Organizations Research Foundation.
Business Not as Usual: Regional Development Organizations Promote
Rural Entrepreneurship. Washington, DC: NADO, 2002.
The National Association of Seed and Venture Funds. Seed and Venture
Capital: State Experiences and Options. NASVF White Paper. May 2006.
126
National Commission on Entrepreneurship. American Formula For Growth.
Washington, D.C.: National Commission on Entrepreneurship, October
2002.
National Community Reinvestment Coalition. Access to Capital and Credit for
Small Businesses in Appalachia. Washington, DC: National Community
Reinvestment Coalition, 2007.
National Governors Association. A Governor’s Guide to Strengthening State
Entrepreneurship Policy. Washington, DC: National Governors
Association, 2004.
Nexus Associates. A Continuing Record of Achievement: The Economic Impact
of the Ben Franklin Technology Partners, Final Report. March 2003.
North Carolina Rural Economic Development Center. Hello, My Business Name
is : A Guide to Building Entrepreneurial Networks in North Carolina.
Raleigh, NC: North Carolina Rural Economic Development Center, 2007.
Oklahoma Center for the Advancement of Science and Technology, Impact
Report 2006. Oklahoma City, Oklahoma: OCAST, January 2006.
Oldsman, Eric. Evaluation as an Effective Management Tool. Nexus Associates
Inc., 2003.
Oldsman, Eric. “Do Manufacturing Extension Programs Matter?” Research Policy
Vol. 25, No. 2, March 1996: 215-232.
Organization for Economic Cooperation and Development. OECD Framework for
the Evaluation of SME and Entrepreneurship Policies and Programmes.
Paris: Organization for Economic Cooperation and Development,
Forthcoming.
Organization for Economic Cooperation and Development. Oslo Manual:
Guidelines for Collecting and Interpreting Innovation Data Third Edition.
Paris: Organization for Economic Cooperation and Development, 2005.
Organization for Economic Cooperation and Development. Entrepreneurship and
Local Economic Development. Paris: Organization for Economic
Cooperation and Development, 2003.
Pages, Erik R., Doris Freedman, and Patrick Von Bargen. “Entrepreneurship as a
State and Local Economic Development Strategy.” The Emergence of
Entrepreneurship Policy: Governance, Start-Ups and Growth in the U.S.
Knowledge Economy. Ed. David Hart. Cambridge, MA: Cambridge
University Press, 2003.
127
Pages, Erik R. and Shari Garmise. “The Power of Entrepreneurial Networks." The
Economic Development Journal Vol. 2, No. 3, Summer 2003: 20-30.
Pages, Erik R. and Kenneth A. Poole. “Entrepreneurship Promotion as an
Economic Development Strategy: Next Steps in Institutionalizing the
Field.” Applied Research in Economic Development Vol. 3, No. 2,
December 2006: 10-27.
Phan, Phillip and Siegel, Donald. The Effectiveness of University Technology
Transfer: Lessons Learned from Quantitative and Qualitative Research in
the U.S. and the U.K. Rensselaer Working Papers in Economics. April
2006.
Pittaway, Luke, Maxine Robertson, Kamal Munir, and David Denyer. Networking
and Innovation: A Systematic Review of the Evidence. University of
Lancaster Institute for Entrepreneurship and Enterprise Development
Working Paper 016. 2004.
Reamer, Andrew, Larry Icerman, and Jan Youtie. Technology Transfer and
Commercialization: Their Role in Economic Development. Washington,
D.C.: US Economic Development Administration, August 2003.
Reese, Laura and David Fasenfast. "Critical Perspectives on Local Development
Policy Evaluations." Economic Development Quarterly Vol. 13, No. 1,
1999: 3-7.
Regional Technology Strategies, Inc. Evaluation of the Early Stages of the
Appalachian Regional Commission’s Entrepreneurship Initiative.
Washington, D.C.: Appalachian Regional Commission, December 2001.
Reynolds, Paul, D.J. Storey, and P. Westhead. “Crossnational Comparisons of
the Variation in New Firm Formation Rates.” Regional Studies Vol. 28, No.
4, 1994: 443-456.
Rice, Mark P. “Co-production of Business Assistance in Business Incubators: An
Exploratory Study.” Journal of Business Venturing Vol. 17, No. 2, 2002:
163-187.
Robinson, Dennis and Zuoming Liu. A User Guide to the Socio-Economic
Benefits Assessment System: A Rural Business-Cooperative Services
Assessment Tool for Economic Development. Community Policy Analysis
Center, University of Missouri-Columbia, December 2004.
Romanelli, Elaine and C.B. Schoonhoven, eds. The Entrepreneurship Dynamic.
Stanford, CA: Stanford University Press, 2001.
128
Rosenfeld, Stuart A. “Networks and Clusters: The Yin and Yang of Rural
Development.” Proceedings of Federal Reserve Bank of Kansas City
Conference on Exploring Policy Options for a New Rural America.
September 2001: 103-120.
Sandler, David. The Effective Use of Tax Credits in State Venture Capital
Programs. Ottawa: Canadian Tax Foundation, 2004.
Schlough, Charles and Streeter, Deborah. Cornell University’s Entrepreneurship
Education and Outreach Program: An Evaluation and Proposal.
Department of Agricultural, Resource, and Managerial Economics
Working Paper. Cornell University, 1999.
Shane, Scott. Academic Entrepreneurship: University Spin-offs and Wealth
Creation. Cheltenham: Edward Elgar, 2004.
Shapira, Philip. The Evaluation of USNet: Overview of Methods, Results and
Implications Final Report submitted to USNet Partners. August 1998.
Shapira, Philip. The Evaluation of USNet: Overview of Methods, Results and
Implications – Final Report. Georgia Tech Policy Project on Industrial
Modernization Working Paper 9805. 1998.
Simpkins, Amy. “Community Development Venture Capital: Producing Results
for Entrepreneurs, Investors, and Communities.” Bridges Vol. 10,
November 2006.
Smits, Ruud and Stefan Kuhlmann. “The Rise of Systemic Instruments in
Innovation Policy.” International Journal of Foresight and Innovation
Policy. Vol. 1, Nos. 1/ 2, 2004.
Solomon, George G., S. Duffy and A. Tarabishy. “The State of Entrepreneurship
Education in the United States: A Nationwide Survey and Analysis.”
International Journal of Entrepreneurship Education Vol. 1, No. 1, 2002:
65-86.
Storey, David J. Six Steps to Heaven: Evaluating the Impact of Public Policies to
Support Small Businesses in Developed Economies. Warwick Business
School Small and Medium Sized Enterprise Centre Working Paper No. 59.
University of Warwick, September 1998.
Street, Christopher T. and Ann-Frances Cameron. “External Relationships and
Small Business: A Review of Small Business Alliance and Network
Research.” Journal of Small Business Management Vol. 45, No. 2, 2007:
239-266.
129
Taylor, Jill S. What Makes a Region Entrepreneurial? A Review of the Literature.
Cleveland State University Center for Economic Development Monograph.
September 2006.
Tornatzky, Louis G. et al. Incubating Technology Businesses: A National
Benchmarking Study. Athens, Ohio: National Business Incubation
Association, 2003.
Tornatzky, Louis G. et al. Innovation U.: New University Roles in a Knowledge
Economy. Durham, NC: Southern Growth Policies Board, 2002.
United Kingdom Department of Trade and Industry. Innovation in the UK:
Indicators and Insights. DTI Occasional Paper No. 6. July 2006.
U.S. Small Business Administration Office of Advocacy and the Ewing Marion
Kauffman Foundation. Entrepreneurship in the 21
st
Century. Conference
Proceedings. March 26, 2004.
Vesper, Karl and William Gartner. “Measuring Progress in Entrepreneurship
Education.” Journal of Business Venturing Vol. 12, No. 5, May 1997: 403-
421.
Welch, Doug, et al. Net Benefits: An Assessment of a Set of Manufacturing
Business Networks and their Impacts on Member Companies. Report
Prepared for USNet and Regional Technology Strategies, Inc. 1997.
Westat. Evaluation of The Appalachian Regional Commission’s Community
Capacity-Building Projects. Washington, D.C.: Appalachian Regional
Commission, July 2004.
White, Jesse. “Overview Panel Comments.” Proceedings of Federal Reserve
Bank of Kansas City Conference on Beyond Agriculture: New Policies for
Rural America. April 2000.
Witt, Peter. “Entrepreneurs’ Networks and the Success of Start-Ups.”
Entrepreneurship and Regional Development Vol. 16, 2004: 391-412.
Zider, B. “How Venture Capital Works.” Harvard Business Review Vol. 76, No. 6,
1998: 131-139.
130
APPENDIX A
LITERATURE REVIEW
A review of literature was completed for works involving evaluation of
entrepreneurship initiatives in five areas – access to capital, entrepreneurial
networks, entrepreneurship education, technical assistance (including incubator
projects), and technology transfer. These five program areas correspond to the
original five areas identified by ARC as requiring investment in order to create a
more entrepreneurial region. The literature review included primarily works that
reported on more comprehensive evaluations of entrepreneurship efforts. Not all
of the publications cited in this report were considered as candidates for the
literature review. A citation and brief summary are provided for each publication
reviewed and the metrics used or suggested are highlighted. Works appear
alphabetically within each program section.
Access to Capital
Colgan, Charles D. and Bruce H. Andrews. Evaluation of Maine Technology
Institute Programs. Maine Institute for Technology and the University of
Southern Maine Center for Business and Economic Research. 2004.
Summary
This presentation evaluates the states investments in technology by looking at
the impacts for institutional grant recipients, the economic impact, effects on
company finances, intellectual property development, relationships cultivated,
and quality of assistance programs.
The evaluation finds that assistance recipients have had significant success in
developing new products leading to intellectual property protection and that they
are likely to have an economic impact on the state. Grant assistance has served
as a much needed catalyst for external financing, and relationships have been
cultivated to enhance the economic cluster networks in the state.
Metrics Suggested
Employment growth at assisted companies
Matching funds received (federal grants, external debt, external equity and
grantee match)
Number of firms indicating new product for sale
Sources of firm revenue
Distribution of company sales by geography and technology sector
Expected raw materials purchased within the state
Sources and amount of debt and equity capital
Patent activity
Other intellectual protection activity
Number of company respondents that received assistance from the
identified program (compared to other assistance programs).
131
Mean rating of organizations consulted and percent of respondents who
claimed relationship was critical to success
Client assessments of interactions with program in relation to percent
satisfaction to all state R&D assistance
Community Development Financial Institutions Data Project. Providing
Capital, Building Communities, Creating Impact. Fourth Edition.
Washington, D.C. 2004. 10 November 2006
.
Summary
Data from 517 Community Development Financial Institutions was analyzed to
demonstrate the impact of CDFIs on emerging domestic markets throughout the
U.S. Some of the results include that almost 7,000 businesses received
assistance from CDFIs resulting in 28,330 jobs. The financial assistance served
niche domestic markets that are underserved by traditional lending institutions
and the transactions were prudent and effective. CDFIs were also shown to be
flexible and timely enough to grow and change with the dynamics of the market
and respond effectively to large-scale disasters such as hurricane Katrina.
Metrics Suggested
Number of CDFIs
Total Assets
Average Assets
Total FTEs
Total Direct Financing Outstanding
Average Direct Financing Outstanding
Percent of Direct Financing Outstanding to specific sectors
Net Charge-Off Ratio
Delinquency Rate > 90 days
Delinquency Rate > 2 months
Average Capital
Percent of Debt from different types of institutions
Markets served: rural vs. urban and regional markets
Types of community betterment programs assisted, such as childcare,
affordable housing, service organizations, educational slots, payday loan
alternatives
Felsenstein, Daniel and Aliza Fleischer. “Small-Scale Entrepreneurship
and Access to Capital in Peripheral Locations: An Empirical Analysis.”
Growth and Change 33 (2002):196-215.
Summary
This paper analyses public assistance programs for small-scale entrepreneurship
programs in rural areas. The authors use data from Israel to establish that
lending institutions perceive a high risk when lending to areas where there is little
132
information. The study has three findings: 1. Location matters in determining risk
profile, 2. Location-oriented programs can improve information asymmetries that
can be a risk factor and 3. These programs can create positive welfare effects.
The study also asserts that there is speculation about the ability of information
technology to increase visibility of small rural firms and enhancing information
flow.
Metrics Suggested
An estimation framework using
Net employment
Total employment
Monthly wage for an individual income groups
Revenue
Loan value
Guarantee Value
Greene, F.J. and D.J. Storey. “An Assessment of Venture Capital Creation
Programme: The Case of Shell LiveWIRE.” Entrepreneurship and Regional
Development 16.2 (March 2004):145-159.
Summary
This paper suggests there are two areas when considering the problems inherent
to assessing venture capital creation programmes: assessment is contingent
upon the evaluation context and an input-output analysis is inadequate. The
researchers create a new instrument to assess the value of these programs.
They find that “soft” forms of support were of little value and that the more likely
an individual was to engage in entrepreneurial activity the less likely they would
be to seek the venture capital program’s services.
Points to Consider for appropriate metrics
What is the purpose of the evaluation? Evaluate the efficacy of the
program or the internal efficiencies?
Be cognizant of the operational issues involved with a program.
Manigart, Sophie et al. “Determinants of Required Return in Venture
Capital Investments: A Five Country Study.” Journal of Business
Venturing 16.6 (July 2002):291-312.
Summary
The authors use two theoretical perspectives (resource theory and financial
theory) to develop hypotheses about the determinants of the return required by
venture capitalists. They test them on over 200 companies in 5 countries. They
find that acquisition and buyout specialists require a significantly lower return
than other venture capital companies (VCCs). Also, highly stage-diversified
VCCs, independent VCCs, and VCCs providing more intensity of involvement all
expect higher return rates.
133
Metrics Suggested for Average Rate of Return for Venture Capital Companies
36%-45% for early stage investments
26%-30% for expansion investments, acquisitions, buy-outs and other
later stage categories
42% for early stage, stage specific rates
33% for later stage, stage specific
National Association of Seed and Venture Funds. Seed and Venture
Capital: State Experiences and Options. May 2006. 10 November 2006
.
Summary
The report includes a survey of the 50 states and their status in seed and venture
capital. The report concludes there are 10 lessons learned. One of these
lessons in that each state should develop a “System of Evaluation.” It claims that
the best programs establish outcome measures from the beginning, keep track of
the program results and evolve according to changing conditions.
The report mentions that state funds can have different objectives. It notes that if
funds have a more economic or social development target than on should not
expect the same rate of return as private funds.
The report establishes a starting point for more in-depth evaluation and design
prospects for state venture capital funds.
Metrics Suggested
Number of jobs created or retained
Geographic reach
Industry reach
Sandler, Daniel. “The Effective Use of Tax Credits in State Venture Capital
Programs.” Tax Paper 108, Canadian Tax Foundation. 2004.
Summary
In his paper, Sandler states how the Venture Capital Industry in the U.S. is highly
localized and for many states to encourage the geographic dispersal of venture
funds, they are using tax credits. He offers several suggestions on how to
evaluate these tax programs.
Metrics Suggested
Economic growth in the state generated by the SMEs that are funded.
The amount of capital raised through the tax credits
134
All other capital the business raised
Number of new employees
Wage rates
Capital expenditures
135
Entrepreneurial Networks
Elfring, Tom and Willem Hulsink. “Networks in Entrepreneurship: The Case
of High-technology Firms.” Small Business Economics 21.4 (2003):409-422.
Summary
The value of networks as an integral part of the explanation of entrepreneurial
success is widely acknowledged. However, the network perspective does not
specify the role of networks in the emergence and early growth of a venture. We
have distinguished three entrepreneurial processes in new venture development,
i.e. discovery of opportunities, securing resources, and obtaining legitimacy,
which are of importance for survival and performance. This paper examines how
these processes are influenced by strong and/or weak ties and whether the
degree of innovation (incremental versus radical) acts as a contingency factor in
the way network ties support entrepreneurial processes. In this explorative study
three cases on high technology firms in The Netherlands provide empirical
material enabling us to develop a number of propositions on the network effect,
in particular the mix of strong and weak ties, on the three entrepreneurial
processes.
Metrics Suggested
Network effects
Types of network ties
Motives for getting involved in networks
Kingsley, Gordon and Edward J. Malecki. “Networking for
Competitiveness.” Small Business Economics 23 (2004):71–84.
Summary
A policy innovation that has achieved widespread diffusion across national and
sub national governments in industrialized countries is the promotion of networks
among small manufacturers as a means of promoting competitiveness.
However, research and evaluations of formal networks formed in response to
policy initiatives tend not account for the informal networks that small
manufacturers routinely use in gathering information and business resources.
This study examines the use of informal networks by 50 small manufacturing
firms in rural and urban regions of northern Florida. The analysis is inductive and
designed to provide a point of comparison to the growing literature on formal
small manufacturing networks. Unlike formal networks, the links that comprise
informal networks tend to be geographically and socially mixed. Small firms use
informal networks to gather information on a mix of issues. Urban and rural firms
have similar patterns of network use on issues affecting product development
and competitiveness. But they have different patterns of network usage for
issues associated with exporting and labor problems.
136
Informal networks draw upon local and non-local information resources that do
not require significant amounts of interpersonal contact across actors. Proximity
is not a factor in the effective use of informal networks for information purposes.
Rather the emphasis is upon locating “tried-and-true” solutions that solve the
business needs. There is little evidence suggesting that the informal networks
that these small manufacturers use are gravitating towards or seeking the
development of formal networks. Thus, policy prescriptions identifying barriers to
networking among small manufacturers are borne out in this study.
Metrics Suggested
Business relationships between participating firms
Needs of firms that prompted them to seek out networks (were those
needs met?)
Information sources used (did network participation change these?)
Pages, Erik and Shari Garmise. “The Power of Entrepreneurial
Networking.” Economic Development Journal 2.3 (Summer 2003):20-30.
Summary
In 2000 the National Commission of Entrepreneurship helped convene a national
series of focus groups to find out what makes a community entrepreneurial.
They found that strong universities, access to venture capital, and good physical
infrastructure matter, but so do soft, people-based assets. The opportunity to
network with other entrepreneurs was an important factor in the success of
entrepreneurs in regions.
This article was designed to offer strategies for creating and nurturing networks.
Metrics Suggested
Learning between individuals (rather than exclusively inter-firm)
Linkages created between entrepreneurs (brokering)
Creation of common perspectives.
Cultural change
Creation of civic leaders
Branding of a region
Enhancing regional competitiveness
Networks spun off from parent network
Regional Technology Strategies, Inc. Evaluation of the Early Stages of the
Appalachian Regional Commission’s Entrepreneurship Initiative. A Report
to the Appalachian Regional Commission. December 2001.
Summary
This is the evaluation of the Appalachian Regional Commission’s $17.6 million
effort, begun in 1997, to bolster the entrepreneurial infrastructure of Appalachia.
137
ARC awarded Regional Technology Strategies, Inc. (RTS) the contract to
undertake the assessment and analysis based on a sample of 23 - 25 projects.
To collect the necessary data, the research team (1) conducted a careful review
of information and documentation supplied to the ARC by grant recipients, (2)
conducted a survey of the sample of projects supplemented by selected
telephone interviews with project staff, (3) conducted telephone interviews with
randomly selected clients and partner organizations from among names
submitted by projects, and (4) visited two project sites for a more in-depth
perspective.
The evaluation uses a variety of measures to assess outcomes and impacts on
individuals and the local economy. The first set of measures assesses relative
degrees of satisfaction of clients with the intervention. The second important set
evaluates the economic results of the intervention on the client such as new
enterprises started, new markets, or new products lines. The third captures the
more general impacts on the economy in terms of jobs created by the new
enterprises, growth potential of the enterprises, and potential sustainability of the
program.
Clients of the programs generally expressed high levels of satisfaction with the
support they received. Three-quarters of projects reported businesses developed
new products, 55 percent indicated that firms upgraded technologies or
management methods, and half reported starting new businesses.
About 52 percent of the sample projects reported creating jobs in existing firms
and 39 percent reported saving jobs that would have otherwise been lost. Adults
created 214 new firms—33 new firms with1 and 181 without employees. Based
on those projects that were able to report hard numbers in the survey, 356 new
jobs were created—54 in new firms, 121 in existing firms, and 181 jobs through
self-employment. Another 85 jobs were saved from extinction in existing firms. In
addition, surveyed projects reported 46 new businesses created by youth or
students as class projects, some of which could become self-sustaining
businesses after graduation.
Metrics Suggested
Client levels of satisfaction
New product development in existing businesses
Technology or management method improvement in existing businesses
New businesses created
Jobs created
Jobs retained that would have been lost without the project
Businesses started by youths
Income of businesses created
138
Rosenfeld, Stuart. “Networks and Clusters: The Yin and Yang of Rural
Development.” Proceedings of Federal Reserve Bank of Kansas City
Conference on Exploring Policy Options for a New Rural America.
(September 2001):103-120.
Summary
Beginning in the mid-1980s, policymakers, particularly in rural areas, realized
that they could no longer rely only on attracting large branch plants to sustain
their economies. Increasing competition from newly developed and less
developed low-wage nations was erasing their cost advantages. The more
creative development agencies began to rechannel efforts towards stimulating
entrepreneurial activity and strengthening indigenous businesses. Among the
many policies discovered and promulgated by various experts and advocates in
the late 1980s was interfirm collaboration, i.e., the widespread formation of
formal and informal alliances, or networks, among groups of companies for
mutual competitive advantage.
The policy levers to convert networks from a practice considered by many to be
uniquely linked to northern Italy’s social and business culture into a more
universal practice were first formulated in Denmark in 1989. This approach to
interfirm cooperation, designed in Denmark with advice from American consultant
Richard Hatch, became the U.S. and international standard. With an allocation of
$25 million from the Ministry of Trade and Industry, the scheme consisted mainly
of training people, called brokers, to create networks and then offering groups of
three or more companies sequentially phased grants for conceptualization,
planning, and implementation. Eligible network activities included joint marketing,
production, problem solving, research and development, and purchasing.
The first and only, national effort to move the numbers of networks in the U.S. to
a scale that could have impact began in 1993 under a grant from the National
Institute for Standards and Technology to Regional Technology Strategies, Inc.
The project, called USNet, was based on the documented value of networks to
industrial modernization and technology adoption in Europe and involved 15
state partners working together to build statewide network programs for SMEs. It
relied heavily on the Danish model of training brokers and multipliers and on a
process for informing companies and development organizations. But it
depended on the individual partner states to provide the financial incentives.
While not a rural development initiative per se, many of the early adopting areas
were rural.
Metrics Suggested
Improved quality of products
New customers
New suppliers outside network
Increased sales
Increased profitability
139
Improved existing process
Improved relationships with customers
Adopted new technologies
Improved supplier quality
Savings by group purchasing or shared resources
Developed new product
Increased exports
Established new company
**Also suggests that learning is a sufficient outcome for businesses to network.
Shapira, Philip. The Evaluation of USNet: Overview of Methods, Results
and Implications. Final Report to USNet Partners. August 1998.
Summary
This report summarizes the aims, methods, and principal findings from the
evaluation of USNet. The USNet project was a pilot initiative to build the capacity
of its partners to promote inter-firm collaboration, with the ultimate aim of
enhancing the competitiveness of small and mid-sized manufacturing
enterprises.
The evaluation discusses the findings of the major USNet evaluation studies
undertaken by members of the evaluation team. The principal findings of the
evaluation are:
Firms who collaborate in inter-firm networks report positive net benefits,
while greatest private impacts are associated with strong industry
leadership of networks
USNet’s original network promotion goals were too ambitious, given the
resources available; judged against more realistic expectations, USNet
has performed well
USNet’s training programs have generated widespread awareness about
inter-firm collaboration
USNet special projects demonstrate the value of explicit follow-on
initiatives to promote inter-firm collaboration
USNet policy and organizational impacts at the state level were modest
Federal support can strengthen efforts to promote inter-firm collaboration
at the state level and aid shared learning
Metrics Suggested
Impacts of networks on states-number of firms involved in networks,
attitudes of organizations toward inter-firm collaboration, learning new
information, making new contacts, learned about other states’ practices,
getting ideas to resolve problems, becoming aware of tools.
Net economic benefits accruing to participating firms
Generating awareness of inter-firm collaboration
140
Welch, Doug, et al. “Net benefits: An Assessment of a Set of Manufacturing
Business Networks and their Impacts on Member Companies.” USNet
Evaluation Working Paper 9701. 1997.
Summary
During the 1990s, a growing number of US companies became involved in
collaborative interfirm partnerships, flexible business networks and other
organized collaborative efforts to aid business performance. Drawing on survey
data from industrial companies and network brokers in five US states, this study
identifies and measures a range of hard and soft impacts on firms resulting from
their participation in interfirm networks. The measures include effects on firms’
activities, business strategy, relationships, trust, confidence, technology use,
know-how, employment and economic benefits and costs.
USNet, a federal-state initiative to strengthen interfirm collaboration, sponsored
this study. In order to assess the impacts of network participation on member
companies, two types of surveys were administered at the beginning of 1997, the
“1997 National Benchmark Survey of Industrial Network Companies” and the
“Survey of Network Presidents/Coordinators.” This study presents findings from
descriptive and comparative analyses of data from these surveys.
A total of ninety-nine members of 13 separate business networks responded to
the survey. Principal findings from this dual survey effort include the following:
Most of the thirteen surveyed networks are young, urban organizations
with limited staff resources.
With a few exceptions, the network members responding to the survey
were small manufacturers.
The networks’ most common primary objective is information sharing.
Companies of different sizes report different network activities.
Overall, most respondents are satisfied with the networks. Thirty-one
Most companies report positive effects to date and expect even larger
future effects.
The average total net benefits per firm of network participation are
positive.
On average, companies experienced a net increase in their employment
levels as a result of network participation.
A few more ‘intensive’ network activities are associated with stronger
overall impacts.
Companies that share sales leads report higher net benefits of network
participation.
Companies that have been in networks longer are more likely to report
sharing technical capabilities with other network members.
Metrics Suggested
Network objectives
Differential network activities depending on client characteristics
141
Satisfaction levels
Client experiences of positive effects on strength of positive effects
Total net benefit per firm ($)
Change in employment levels of participating companies
Intensity of network activities
Improvement in employee and management skills due to network
participation
Knowledge sharing with other companies
142
Entrepreneurship Education
Charney, Alberta and Gary Libecap. The Impact of Entrepreneurship
Education: An Evaluation of the Berger Entrepreneurship Program at the
University of Arizona, 1985-1999. Final Report to Kauffman Center for
Entrepreneurial Leadership. May 2000.
Summary
The report details the results of a series of surveys on the effect that the Berger
Entrepreneurship Program at the University of Arizona has had on its graduates,
the university’s technology transfer program, financial contributions to the college
of business, and the degree to which entrepreneurship has been incorporated
into curriculums in other departments. The Berger Program was a good
candidate for such an analysis because it had been in existence for sixteen years
at the time of the report. A sample of program alumni and other university
graduates were surveyed to assess the program’s effect on students. According
to the results, program graduates were three times more likely to start a new
business than non-entrepreneurship graduates. Additionally, program graduates’
average annual income was found to be twenty-seven percent higher than the
average annual income of general university graduates. However, there was
little evidence that program participation led to higher levels of job satisfaction.
With regard to technology transfer, the report concludes that entrepreneurship
education promotes technology transfer as graduates were more likely to be
involved with firms that use licensed technologies or with firms that license
technologies to others. A separate survey of University of Arizona administrators
revealed that entrepreneurship education had reportedly led to increased
financial contributions to the university and respondents felt that other
curriculums had been enriched due to pedagogical innovations in the
entrepreneurship program.
Metrics Suggested
Number of business ventures started
Number of workers employed in venture
Amount of sales/revenue
Graduate job satisfaction
Type of venture started
Whether or not venture is “high-tech”
Fayolle, Alain. “Evaluation of Entrepreneurship Education: Behavior
Performing or Intention Increasing?” International Journal of
Entrepreneurship and Small Business 2.1 (2005):89-98.
Summary
Fayolle says there is a growing interest around the question of how to evaluate
entrepreneurship education programs. He then suggests that the standard
metrics such as direct and indirect job creation are insufficient measuring sticks.
143
Fayolle points out that there are significant timing issues associated with using
job creation as the primary criteria, i.e. he says the entrepreneurial process is a
nonlinear one and there is no definitive way to know at what point employment
opportunities may be created. Accordingly, he advocates for the inclusion of
some more intangible metrics. Fayolle believes that entrepreneurship programs
should also be evaluated on the attitudes, mindsets, and intentions of students.
He also discusses how different teaching strategies can influence student
behavior. Fayolle notes that entrepreneurship training may enable students to
start their own enterprise, but it can at the same time dent their desire to do so –
consequently, teaching style matters. In essence, Fayolle says that because
entrepreneurship is so connected to personal desire and motivation that it is very
important to evaluate the student psyche throughout the educational process.
Bottom line being that evaluating programs on job creation alone is difficult and
incomplete; therefore, that analysis should be supplemented with some
assessment of how the program nurtures the entrepreneurial spirit.
Metrics Suggested
Teaching strategies
Student attitudes and intentions throughout the process
An evaluation of overall program goal compared to student goals
Knowledge acquisition
Nakkula, Michael. “Expanded Explorations into the Psychology of
Entrepreneurship: Findings from the 2001-2002 Study of the NFTE in two
Boston Public High Schools.” Working Paper, Harvard University Graduate
School of Education. 2003.
Summary
This reports details the results of a National Foundation for Teaching
Entrepreneurship (NFTE) program that was taught in two Boston public schools
during the 2001-2002 school year. In the Boston public school system every
student is required to take a school to career “pathway” class that forces them to
think about their plans after graduation. In concert with that, a study was set up
where some students were enrolled in a special “pathway” class sponsored by
NFTE. Students were assessed at the beginning of their “pathway” classes and
then again upon completion regarding questions like desire to go to college.
Students taking NFTE’s entrepreneurship “pathway” course were found to have
experienced considerably higher increases in college interest and occupational
aspirations over the duration of the class than students enrolled in non-NFTE
“pathway” courses.
Metrics Suggested
Level of college interest, pre and posttest
Occupational interest, pre and posttest
Overall school engagement
Amount of independent reading
144
Schlough, Charles and Deborah Streeter. “Cornell University’s
Entrepreneurship Education and Outreach Program: An Evaluation and
Proposal.” Working Paper, Department of Agricultural, Resource, and
Managerial Economics, Cornell University. 1999.
Summary
The first part of the report evaluates Cornell’s Entrepreneurship Education and
Outreach (EEO) program. The authors discuss the challenges that the program
has encountered during its first two years including a lack of widespread
cooperative support from state and local-level stakeholders. Consequently, the
second part of the report is a proposal to develop a statewide network of
supportive partnerships in order to improve Cornell’s entrepreneurship education
mission. Included in that plan is a set of “criteria for success” and a
corresponding list of metrics. The authors suggest evaluating the proposed
program by measuring the number of small businesses started by graduates, the
number of jobs created within those ventures, the number of decisions by
graduates to not proceed with a business concept, and annual sustained
enrollment levels in program courses.
Metrics Suggested
Number of businesses started
Number of people employed within those businesses
Number of decisions by graduates to not pursue a business concept
Annual sustained enrollment in program courses
Soloman, G.T., et al. “The State of Entrepreneurship Education in the
United States: A Nationwide Survey and Analysis.” International Journal of
Entrepreneurship Education 1.1 (2002):65-86.
Summary
This paper offers an assessment of entrepreneurship education both
domestically and abroad as of 2000. Included in the article is a brief literature
review on entrepreneurship education. The following is a summary of two
articles discussed in that literature review for which full-text versions were not
readily available. Citations for both works are listed below.
Block and Stumpf (1992) and McMullan and Long (1987) both assert that
traditional measures of program effectiveness such as number of graduates are
insufficient indicators of success. Instead they agree that programs should be
evaluated according to their socioeconomic impacts. Accordingly, Block and
Stumpf (1992) suggest using number of jobs created by graduates and overall
job satisfaction as relevant measures. Similarly, McMullan and Long (1987)
advocate assessing the number, type, and growth of companies created by
entrepreneurship graduates as a better gauge of success.
145
Metrics Suggested
Number of jobs created
Level of job satisfaction
Number of businesses created
Types of companies created
Growth of companies created
Sources
Block, Z. and S. A. Stumpf. “Entrepreneurship Education Research: Experience
and Challenge.” The State of the Art of Entrepreneurship. Eds. D. L. Sexton and
J. D. Kasarda. Boston: PWS-Kent Publishing S.A., 1999. 17-45.
McMullan, W. E. and W.A. Long. “Entrepreneurship Education in the Nineties.”
Journal of Business Venturing 2 (1987):261-275.
Vesper, Karl and William Gartner. “Measuring Progress in
Entrepreneurship Education.” Journal of Business Venturing 12
(September 1997): 403-421.
Summary
In this article, Vesper and Gartner report on the results of a 1994 survey which
attempted to rank university-level entrepreneurship education programs. The
survey was mailed to 941 business school deans both domestically and abroad.
Respondents were asked to rank the top programs and to list the most important
criteria considered when doing so. Among the possible 18 criterion listed,
courses offered was ranked number one followed by faculty publications, impact
on community, exploits of alumni, and innovations. Faculty start-ups and location
were listed as the bottom two. However, Vesper and Gartner quickly caution the
reader about the results of the survey and similar rankings in general. The
authors state that the survey did not tie the program rank to specific criteria nor
did it evaluate respondent knowledge of the other programs in question. With
respect to other popular rankings, for example, program ratings published by
Business Week, Vesper and Gartner argue that traditional metrics used in those
rankings – GMAT scores, computers per capita, etc. – may not be appropriate
predictors of entrepreneurial success. Instead, they suggest employing the set of
criteria used in awarding the Malcolm Baldrige National Quality Award (MBNQA),
a well-established quality improvement program. Using the MBNQA framework,
Vesper and Gartner stress the importance of evaluating entrepreneurship
programs based on the following seven factors: leadership, information and
analysis, strategic and operational planning, human resource development and
management, educational and business process management, school
performance results, and student focus and student and stakeholder satisfaction.
Metrics Suggested
Student performance (in specific classes or as demonstrated through a
portfolio)
146
Student improvement throughout program
Student satisfaction
Impact on the community (number of start-ups, students employed in new
firms, students working in positions assisting new firms)
147
Technical Assistance
Aernoudt, Rudy. “Incubators: Tool for Entrepreneurship?” Small Business
Economics 23.2 (2004):127-135.
Summary
This paper examines U.S. and European experiences with business incubators
and stresses the need for accurate evaluations of their impact relative to their
different types. The author finds that lack of entrepreneurship and the
underdevelopment of seed financing and business angel networks are some of
the biggest barriers to success. He asserts that seed financing, links with
business angels and business angel networks as well as involvement in second
round financing and IPO assistance should be integrated into the business
incubation concept.
Metrics Suggested
Survival rate
Tenants by incubator
Employment by tenants
Employment created by graduates
Graduates remaining in the community
Brown, J. David, John S. Earle and Dana Lup. “What Makes Small Firms
Grow? Finance, Human Capital, Technical Assistance, and the Business
Environment in Romania.” Economic Development and Cultural Change 54
(October 2005):33-70.
Summary
This paper sought to explore new ground by looking at the policy-relevant factors
that may stimulate or hinder small start-up companies, and fill the need for
quantitative studies using panel data to analyze statistical relationships between
firm growth and objective measures of factors related to policies. They found
that availability of loans is a factor while internal finance and trade credit tend to
be unimportant. They also found that high school education raises growth but
university education and worker training are not necessarily a factor. Technical
assistance was also considered a weak factor for small business growth
success.
Metrics Suggested
Retained earnings,
conventional bank lending,
informal credit markets,
tax credits offered by the state
Membership in business association
Membership in consultancy programs
148
Rate of usefulness of assistance programs
Community Development Financial Institutions Data Project. Providing
Capital, Building Communities, Creating Impact. Fourth Edition.
Washington, D.C. 2004. 10 November 2006
.
Summary
Data from 517 Community Development Financial Institutions was analyzed to
demonstrate the impact of CDFIs on emerging domestic markets throughout the
U.S. Some of the results include that almost 7,000 businesses received
assistance from CDFIs resulting in 28,330 jobs. The financial assistance served
niche domestic markets that are underserved by traditional lending institutions
and the transactions were prudent and effective. CDFIs were also shown to be
flexible and timely enough to grow and change with the dynamics of the market
and respond effectively to large-scale disasters such as hurricane Katrina.
Metrics Suggested
Number of people receiving group-based training
Number of people receiving one on one technical assistance
Number of organizations receiving training
Number of jobs created overall from the project
Number of business receiving training
Enterprise Corporation for the Delta. Enterprise Corporation for the Delta
Program Monitoring Report—Business Technical Assistance. 2003. 10
November 2006
.
Summary
This report provides baseline information about the Enterprise Corporation for the
Delta’s technical Assistance activities associated with their Community
Development Financial Institution. This baseline is established to measure the
progress of their efforts to address the non-financial needs of potential and
existing customers and lay the groundwork for more in-depth analysis in the
future.
The ECD has three technical assistance activities: FastTrac entrepreneurial
training, brokered TA and Business LINC mentor/protégé program. ECD learned
that very few graduates from FastTrac attained loan financing, but many
graduates demonstrated more sound decision making. The Brokered TA mostly
focused on establishing the foundation for future analysis. The mentoring
program of the ECD found that creating a mentoring environment for small start-
ups works, but the conditions for its success is very different than many urban
programs mostly because there are not large established companies to become
a cadre of mentors. They also found that protégés were more successful if they
149
had several years of experience and if they were in close proximity to their
mentor.
Metrics Suggested
Number of mentor relationships established
Finance seeking intentions of graduates, before and after course
completion
Number of technical assistance engagements
Types of TA services requested
Loan ratings of TA graduates and loan ratings of control group
Satisfaction level with mentor programs or other TA initiatives.
Greenburg, Elizabeth and Richard Reeder. “Who Benefits from Business
Assistance Programs? Results of the ERS Rural Manufacturing Survey.”
Agriculture Information Bulletin Number 736-04. United States Department
of Agriculture. 1998.
Summary
The authors sought to find out how much government programs helped rural
manufacturers and who exactly is benefiting from these programs. They
discovered that over 60 percent of manufacturing establishments benefited from
the programs and 28 percent of these firms found these programs to be very
important to their operations over the last three years. They discovered that
manufacturers using advanced technologies benefited more than other
manufacturers. Also, large businesses were more likely to benefit than small
ones, although small firms seemed to have more problems and benefit the most
once assistance was administered.
Business assistance programs were identified as: tax incentives, loans (direct,
indirect/guaranteed, and revolving), industrial parks and enterprise zones, and
training and technical assistance.
Manufacturing establishments were characterized by the following: metro/non
metro, geographic region in the U.S., employment size, type (branch plant and
high tech), and distressed (high poverty rates, high unemployment rates and
population loss).
The study further analyzed which types of firms benefited from the specific types
of assistance programs. State and local tax breaks benefited the largest
proportion of nonmetro establishments (46 percent), training and technical
assistance (29 percent), industrial parks/enterprise zones (21 percent), direct
loans (15 percent), guaranteed loans (13 percent), revolving loan funds (9
percent).
Metrics Suggested
150
Survey of how important each assistance program was to their business.
This metric can be cross-analyzed by
o Geographic location
o Size
o Type of business
o Type of distressed area
o Rural/Urban
Lambrecht, Johan and Fabrice Pinray. “An Evaluation of Public Support
Measures for Private External Consultancies to SMEs in the Walloon
Region of Belgium.” Entrepreneurship and Regional Development. 17.2
(March 2005):89-108.
Summary
This paper evaluated public support measures for private consultancies to SMEs
in the Walloon region of Belgium. It presents an analysis of the supply and
demand, an evaluation of the efficiency and the effectiveness of policy measures,
and real policy recommendations. The paper recommends that the real needs of
the entrepreneur and of the SME determine the publicly financed advisory
process. It also recommends a “one stop shop” for private external consultants
that help SMEs.
Metrics Suggested
Profit
Sales
Market Share
Employment
Microenterprise Fund for Innovation, Effectiveness, Learning and
Dissemination (FIELD). Improving Microenterprise Training and Technical
Assistance: Findings for Program Managers. 2002. 10 November 2006
.
One of the first tasks FIELD set for itself and for the organizations who received
awards from their RFPs was to answer “What makes for effective training and
technical assistance?” In support of this question they also asked “What are the
appropriate indicators, which intermediate measures are better indicators of
financial impact on clients, and what practical approaches can programs used to
document and track outcomes?
They found that many entrepreneurs sustained and grew their businesses, while
some other businesses floundered. This was mostly due to family or personal
reasons. The analysis found that effective training programs acknowledge the
importance of client readiness for business and offer a range of services to help
clients meet these needs. It also found that adult learning theory is an effective
method for designing and delivering training. Training must include soft skills
151
and basic competencies as well as key financial and marketing skills. Finally,
FIELD learned TA should be offered within a structure that keeps clients
connected to a larger program, but places the initiative on the client to receive
services.
Metrics Suggested
Completion of Training
Development of Business Plan
Progress in Seeking Business Financing
Business Starts
Business Expansions
Business Stabilizations
Is there an improvement in income reporting
Increase in business assets
Increase in personal assets, such as cars, savings, homeownership.
Microenterprise Fund for Innovation, effectiveness, Learning and
Dissemination. “Assessing the Effectiveness of Training and Technical
Assistance.” FIELD Forum Issue 1. 1999. 10 November 2006
.
Summary
FIELD recognizes that training and technical assistance to small businesses has
a significant impact on their success, but there is little data to substantiate this to
policymakers and funders. Therefore they sought to identify models and other
metrics to show what makes for effective training and how that equates to lower
costs. FIELD asked practioners to propose strategies and identify indicators that
could establish a link between service and outcomes. Some intermediate and
final outcomes were suggested.
Suggested Metrics
Course completion and graduation
Pre and post knowledge testing
Business plan completion
Satisfaction
Hours of Training and TA
Sequence of Training and TA
Business Skills acquired
Personal effectiveness skills
Established networks/relationships
Business start-ups/survivals
Sales and profit/loss
Number of employees
Employee wages and benefits
Household income assets and net worth
Change in public assistance
152
Oldsman, Eric. “Evaluation as an Effective Management Tool.” Nexus
Associates, Inc. 2003. 10 November 2006 .
Summary
This paper critiques the Performance Measurement Framework established by
the Committee of Donor Agencies for Small Enterprise Development to learn
more about the performance of business development services. The paper
suggests that the PMF framework can be useful as a tool for managers to
improve their performance but considerable caution should be taken when using
common performance standards for such diverse programs. The author
recommends that evaluations be grounded in explicit theories of the particular
initiatives.
Things to consider when developing metrics
Characterize conditions within markets to learn more about their structure
and performance
Determine specific needs within a target population of firms in order to
design new programs
Establish whether existing programs are being implemented as intended
Find out whether existing programs are achieving their objectives
Compare existing programs to judge the relative merits of different
approaches to addressing specific needs.
Examine operations in great detail (aggregate data can mask a lot).
Oldsman, Eric. “Do Manufacturing Extension Programs Matter?”
Research Policy 25.2 (March 1996):215-232.
Summary
Based on his evaluation of the New York based Industrial Technology Extension
Service Program, Oldsman finds that manufacturing extension programs can
have a favorable impact on participating companies. Because of the expertise of
field agents, firms have been able to reduce costs, particularly with respect to
direct and indirect labor, and in some instances increase revenue.
He found that MEP programs should be designed to focus on adding value rather
than cutting costs; pay attention to direct, long-term assistance; and foster
cooperation to compensate for the lack of internal economies of scale.
Metrics Suggested
Refining layout of operations
Purchased or developed new software
Ask entrepreneurs if they had not received assistance would they have
stayed in operation and in the state?
Cost savings
153
Direct labor productivity
Reduction in inventory
Reductions in manufacturing lead time
Reductions in direct labor costs per unit
Reductions in material costs per unit
Reductions in energy costs per unit
Reductions in indirect labor costs per unit
Reductions in other overhead costs
154
Technology Transfer
Association of University Technology Managers. AUTM U.S. Licensing
Survey: FY 2004 Survey Summary. 2004. 10 November 2006
.
Summary
This year’s Licensing Survey shows a continued steady growth in the 6 percent
range for most of the performance measures that are considered meaningful
indicators within the profession:
Products available to the public
Invention disclosures received
Licenses and options executed
Licenses and options active
Licenses and options generating income
Licenses and options generating running royalties
Net income
One or two important performance measures, specifically U.S. patents issued,
though down from fiscal year 2003, appeared to be consistent with long-term
growth trends. However, the most dramatic results were the clear evidence of a
recovery from the very difficult market conditions for new company startups
reported in the fiscal years 2002 and 2003 Licensing Surveys. Institutions
launched 23.5 percent more new startups in fiscal year 2004 than in fiscal year
2003, and the number of existing startup companies that went out of business
declined more than 30 percent.
The new startup company activity reflects the changed circumstances in capital
markets. The second half of 2003 will be remembered as the end of the venture
industry’s hemorrhaging that followed the nearly simultaneous collapse of the e-
commerce, telecommunications and biotechnology markets.
The second half of 2003 also saw the first revitalization of the initial public
offering market since 2000. The number of venture-backed IPOs began
increasing in the second half of 2003, with 20 of the 22 venture backed IPOs for
2003 occurring in the third and fourth quarters of the year. The first two quarters
of 2004 saw 34 venture-backed IPOs, and the year ended with 67 venture-
backed IPOs raising $4.98 billion vs. the $1.4 billion raised in 2003.
Metrics Suggested
Products available to the public
Invention disclosures received
Licenses and options executed
Licenses and options active
Licenses and options generating income
Licenses and options generating running royalties
155
Net income
New Startups
Number of startups going out of business
University equity interests in their startups
IPOs
Audretsch, David B., Taylor Aldridge, and Alexander Oettl. The Knowledge
Filter and Economic Growth: The Role of Scientist Entrepreneurship.
Ewing Marion Kauffman Foundation. 2006. 10 November 2006
.
Summary
This study examines the prevalence and determinants of the commercialization
of research by the top twenty percent of university scientists funded by grants
from the National Cancer Institute (NCI). Because the two publicly available
modes of scientist commercialization – patents and Small Business Innovation
Research (SBIR) grants – do not cover the full spectrum of commercializing
activities undertaken by university scientists, the study also includes two
additional measures obtained from detailed scientist interviews: licensing of
intellectual property and starting a new firm. These measures are used to assess
both the prevalence and determinants of scientist commercialization of research.
In particular, two distinct routes for commercializing scientist research are
identified, the Technology Transfer Office (TTO) route and the entrepreneurial
route, which does not involve assigning a patent to the university. This study in
no way provides an assessment or judgment about the efficacy of the TTO.
Rather, this study highlights the extent to which additional commercialization of
research takes place, suggesting that the contribution of universities to U.S.
innovation and ultimately economic growth may be greater than had previously
been believed. Relevant findings include:
Two paths for commercialization of scientist research are identified - the
TTO route and the entrepreneurial route. Scientists who select the TTO
route by commercializing their research through assigning all patents to
their university TTO account for 70 percent of NCI patenting scientists.
Scientists who choose the entrepreneurial route to commercialize their
research, in that they do not assign patents to their university TTO,
comprise 30 percent of patenting NCI scientists.
Social capital enhances the propensity for scientists to commercialize their
research. The impact of social capital is particularly high for the
commercialization mode of scientist entrepreneurship.
Scientists choosing the entrepreneurial route to commercialize their
research, by not assigning patents to their university to commercialize
research, tend to rely on the commercialization mode of entrepreneurship.
By contrast, scientists who select the TTO route by assigning their patents
to the university tend to rely on the commercialization mode of licensing.
Metrics suggested
156
Research grants to university researchers (i.e., SBIR)
Patents
Licenses of intellectual property
University researchers starting new firms
Route by which commercialization happens- entrepreneurial or through
tech transfer office
Chukumba, Celestine and Richard Jensen. “University Invention,
Entrepreneurship, and Start-Ups.” National Bureau of Economic Research
Working Paper No. 11475. July 2005.
Summary
This study examines the commercialization of university inventions in licensing to
both start-up firms and established firms, and seek to determine when licensing
to start-ups is more likely. They construct a theoretical model that predicts start-
ups are more likely if their opportunity cost of development and
commercialization is lower or if the technology transfer officer’s (TTO) opportunity
cost of searching for a partner among established firms is higher. Using data
from the Association of University Technology Managers, the National Venture
Capital Association Yearbook, and the National Research Council, the study
finds that inventor quality and measures of past TTO success (age, the number
of disclosures, gross royalties) are all positively and significantly related to the
number of licenses to both start-ups and established firms. However, it also finds
that start-up activity is positively and significantly related to the S&P 500, but
negatively and significantly related to the interest rate and rate of return to
venture capital.
Metrics suggested
Licensing to start-ups and existing firms
Start-ups created
Partnering with outside entities by tech transfer offices
Venture capital secured by start-ups
Markman, Gideon, et al. “Entrepreneurship and University-based
Technology Transfer.” Journal of Business Venturing 20.2 (2005):241-263.
Summary
The success of business incubators and technology parks in university settings is
often determined by how well technology is transferred from the labs to their
startup firms. University technology transfer offices (UTTOs) function as
‘‘technology intermediaries’’ in fulfilling this role. This article builds a framework to
address two questions: (a) Which UTTOs’ structures and licensing strategies are
most conducive to new venture formation; and (b) how are the various UTTOs’
structures and licensing strategies correlated with each other. The findings reveal
a complex set of relationships between UTTO structure and strategies, new
venture formation, and business incubation.
157
Based on interviews with 128 UTTO directors, findings show that whereas for-
profit UTTO structures are positively related to new venture formation, traditional
university and nonprofit UTTO structures are more likely to correlate with the
presence of university-based business incubators. Licensing-for equity strategy is
positively related to new venture formation while sponsored research licensing
strategy is negatively related. The licensing-for-cash strategy, the most prevalent
transfer strategy, is least correlated to new venture formation. A content analysis
of UTTO mission statements also revealed an overemphasis on royalty income
and an under emphasis on entrepreneurship.
Metrics Suggested
Structure of tech transfer office (for- or non-profit)
Incubators
Technology parks
Startup firms
Business incubation
Licensing
University equity shares in startups
Applied v. basic research at university where tech transfer office is located
Oklahoma Center for the Advancement of Science and Technology. Impact
Report 2006. Oklahoma City, Oklahoma. January 2006. 10 November 2006
.
Summary
The report summarized OCAST’s program impacts in terms of award amounts
and leveraged private and federal funds. It also gives some detail about each
program, including the Oklahoma Technology Commercialization Center, whose
progress is measured in terms of facilitation of capital acquisition, jobs created,
and companies served.
Metrics suggested
Amount of capital acquisition facilitated
Jobs created
Number of technology companies served
Palmintera, Diane. Accelerating Economic Development Through
University Technology Transfer. Reston, VA: Innovation Associates Inc.,
February 2005.
Summary
This report highlights models of university tech transfer and commercialization,
related efforts like entrepreneurship programs, and the infrastructure and
environment needed to support commercialization efforts. It includes case
158
studies of university-based tech transfer and related economic development
initiatives that lay the groundwork for state, university, and corporate actions to
leverage university resources.
Practices in tech transfer at 10 universities were examined, along with related
entrepreneurship programs and other programs. These case studies were
analyzed to extract best practices and recommendations.
Suggested Metrics
Corporate sponsored research
Levels of government funding
Seed capital and source (i.e., university-created funds or private funds)
Innovation centers
Number of start-ups created
Number of start-ups assisted
Incubators
Research Parks
Employment
Phan, Phillip and Donald Siegel. “The Effectiveness of University
Technology Transfer: Lessons Learned from Quantitative and Qualitative
Research in the U.S. and the U.K.” Rensselaer Working Papers in
Economics Number 0609. Rensselaer Polytechnic Institute. April 2006.
Summary
In recent years, there have been numerous studies of the effectiveness of
university technology transfer. Such technology transfer mechanisms include
licensing agreements between the university and private firms, science parks,
incubators, and university-based startups. This study reviews and synthesizes
these papers and presents some recommendations on how to enhance
effectiveness. Implementation of these recommendations will depend on the
mechanisms that universities choose to stress, based on their technology
transfer “strategy.” For example, institutions that emphasize the entrepreneurial
dimension of technology transfer must address skill deficiencies in technology
transfer offices, reward systems that are inconsistent with enhanced
entrepreneurial activity and the lack of training for faculty members, post-docs,
and graduate students in starting new ventures or interacting with entrepreneurs.
Metrics Suggested
Licensing
Science parks
Incubators
University-based startups
Training provided to employees in working with entrepreneurs/start ups
159
APPENDIX B
NON-PROJECT* STAKEHOLDER INTERVIEWS
REGIONAL
STAKEHOLDERs
REGIONAL STAKEHOLDER ORGANIZATIONs
Dinah Adkins President, National Business Incubation Association (OH)
Cathy Ashmore Executive Director, Consortium for Entrepreneurship Education
Bill Campbell Director, Alabama Small Business Development Centers
Caroline Carpenter Program Director, W.K. Kellogg Foundation
Dale Carroll President, Advantage West (NC)
Pam Curry Executive Director, Center for Economic Options
Eleanor Herndon Executive Director, North Carolina REAL
June Holley Consultant, Network Weaving (OH)
Mary Hunt-Lieving Program Officer, Benedum Foundation
Lisa Ison President, New Century Venture Center (VA)
Kris Kimmel President, Kentucky Science and Technology Corporation
Bill Loope New River Community and Technical College (WV)
Justin Maxson President, MACED
Ray Moncrief Vice President, Kentucky Highlands Investment Corporation
Welthy Soni Myers Managing Director for Special Initiatives, Association for Enterprise
Opportunity (VT)
Becky Naugle Director, Kentucky Small Business Development Centers
Kim Pate Vice President, CFED
Stuart Rosenfeld Principal, Regional Technology Strategies (NC)
Greg Rutherford President, York Technical College (SC)
Jeff Spencer Executive Director, Ohio Valley Regional Development Commission
Kerwin Tesdell President, Community Development Venture Capital Alliance (NY)
Jesse White Director, Office of Economic and Business Development, University of
North Carolina
PROGRAM
LEADER
STATE
Denise Ambrose Program Manager, Virginia Department of Housing and Community
Development
Bonnie Ammons Senior Program Manager, Office of Community Grant Programs (SC)
Todd Christiansen Associate Director, Virginia Department of Housing and Community
Development
Olivia Collier ARC Program Manager, North Carolina
Bonnie Durham ARC Program Manager, Alabama
Al Feldstein ARC Program Manager, Maryland
Neil Fowler ARC Program Manager, Pennsylvania
Ralph Goolsby ARC Program Manager, West Virginia
Elisabeth Kovacs ARC Program Manager, South Carolina
Rick Meredith Assistant Commissioner, Tennessee Department of Economic and
Community Development
Peggy Satterly ARC Program Manager, Kentucky
Sara Stuckey Retired ARC Program Manager, North Carolina
James Thompson ARC Program Manager, Georgia
Kyle Wilbur ARC Program Manager, New York
*Some stakeholders were from organizations that received ARC funding for entrepreneurship
projects but were included because they (1) had broad and unique knowledge of some program
area and/or entrepreneurship, (2) the organization’s project was not included in the sample,
and/or (3) the individual was not interviewed as follow up with grantees included in the sample.
160
Appendix C
PROTOCOL FOR PROJECT LEADERS
Enter the following data from the project folder:
Project Number: __________________ Project Status (circle one): I
C1 C2
Project Title: _______________________ Project Type (circle one): C N I
E TA
Grantee: _________________________
Year Project Initiated:_____________ Number of Years of ARC Funding:
_____
Name of Person Interviewed: _______________________Phone/Email:
___________
Organization of Person Interviewed:______________________
ARC Funds Invested:_____________________ Leveraged
Funds:_______________
Project Summary:
Stated goals of the project:
Number of Businesses Served:
Number of jobs created
Jobs retained:
Amount of leveraged private investment:
For all calls:
1. Were you involved with this project and/or are you knowledgeable about
it? (If no, get referral.) If yes, in what capacity did you work with the
project?
2. What is your background? (Try to understand the importance of their
leadership.) Alternative: Tell me a little about your background and
experience with projects like this in the past. How did you get involved in
this project? How long have you been involved with the relevant
community? If not long have you had similar experiences in your
previous communities?
161
3. What was the problem that you were trying to address with this project?
How were you trying to address this problem?
4. Did you think the project was a success? Why or why not? What were the
elements of success/failure?
5. What specific results or outcomes were achieved through this project
during the period of ARC investment?
a. Are there specific results that you believe were particularly
important? If so, why?
b. Are there specific results that were unintended or unexpected?
c. Do you feel you achieved the objectives set forth for this project?
6. Has the project continued after ARC funding ended?
a. If yes, how was the project funded after ARC?
b. What outcomes have you experienced post-ARC?
7. What value did this project create in your service area/community?
a. Are there specific quantifiable changes that you have seen in the
community because of this project?
b. Are there specific qualitative changes that you have seen in the
community because of this project?
8. Is there anything you learned from this project that would be useful to
others who are attempting to do something similar in their communities?
9. Is there anything else you would like to add about the project and its
implementation in your service area/community?
10. Is there anyone else who is familiar with this project and its broad impacts
on the community and/or the region who might provide us with useful
insights? (If yes, collect contact information.)
GO TO SPECFIC QUESTIONS FOR EACH TYPE OF PROJECT
162
Capital
At ARC Project End After ARC Funding
Total number of loans
Number of years
operated
Total $ amount of loans
Number of funds created
Size of Fund(s)
Distribution of loans by
sector
$ funds leveraged
Jobs created
Jobs retained
Percent of portfolio
companies still in
business
*Wages, income per job
or total
Comments:
163
Sectors
At ARC Project End After ARC Funding
Number of participants
Number of members
Jobs created
Jobs retained
*Increase in interfirm
collaboration
*Change in total sector
sales
*Number of business
start-ups in targeted
sector
*Number of participants
retained in service area
*Number of participants
still in business
Comments:
164
Incubators
At ARC Project End After ARC Funding
Number of current clients
Number of clients served
Number of graduated
firms
Number of clients still in
business
Amount $ leveraged by
incubator
*Number of graduates
retained in service area
Jobs created while in
incubator
Jobs created after
graduation
Jobs retained while in
incubator
Jobs retained post-
graduation
$ capital raised by
tenants
Comments:
165
Education
At ARC Project End After ARC Funding
Number of participants
enrolled
Number of participants
completing program
Number of schools
offering entrepreneurship
education pre and post
Number of schools in
service area, pre and
post
Change in student
performance pre and
post
Number of student
businesses started
Number of students that
stay within the service
area
*Increase in awareness
of business concepts
*Increase in number
considering business
creation as a career
option
* Change in community
attitudes toward
entrepreneurship
Comments:
166
Technical Assistance
At ARC Project End After ARC Funding
Number of business
starts
Number of business
expansions
Number of clients
*Number of discouraged
clients
Number of clients still in
business
Number of jobs created
Number of jobs retained
*Firm performance ($
capital raised)
*Number of clients
retained in service area
*Number of clients still in
business
Comments:
167
APPENDIX D
PROTOCOL FOR NON-PROJECT STAKEHOLDERS
1. Background information on the stakeholder
Name:
Title:
Organization:
2. In what ways are you (or have you been) involved with entrepreneurship
and/or economic development in the Appalachian Region?
3. Were you directly involved in any specific projects funded by the ARC
Entrepreneurship Initiative? If yes, please identify the project(s):
4. [For State Alternates Only] What was your state’s strategy for engaging
with ARC’s Entrepreneurship Initiative? (Was this initiative a priority in
your state? Was there a single statewide project approach or did individual
communities/organizations propose projects?)
5. [For State Alternative Only] Did your state’s participation in ARC’s
Entrepreneurship Initiative lead to any change in the policy environment
for supporting entrepreneurship? If so, please explain.
6. What do you think are the most significant region-wide impacts associated
with ARC Entrepreneurship Initiative projects generally? Were there
specific community or project impacts that you can identify?
7. How have these broader impacts been measured?
8. Would you provide some concrete examples of these broader impacts?
9. In your view, what has limited the broader impacts associated with the
ARC projects?
10. In your view, what has contributed to the broader impacts associated with
the ARC projects?
11. Do you think the ARC Entrepreneurship Initiative projects have had an
impact on creating a more supportive climate for entrepreneurs in the
region? Why or why not?
168
12. Considering the broad impacts associated with the ARC Entrepreneurship
Initiative you have identified above, what do you think are some of the
most valuable or effective performance metrics for capturing these
impacts?
doc_537046311.pdf