Description
In commutative ring theory, a branch of mathematics, the radical of an ideal I is an ideal such that an element x is in the radical if some power of x is in I. A radical ideal (or semiprime ideal) is an ideal that is its own radical (this can be phrased as being a fixed point of an operation on ideals called 'radicalization'). The radical of a primary ideal is prime.
Research for Measuring Radical Innovation in Real Time
Abstract: As Leifer et al. have argued, the management of radical innovation projects necessitates different strategies and practices, including different ways of measuring performance. This paper discusses our efforts to explore indicators of progress for radical innovation in basic and applied research and technology development at both the project and programme levels. The paper provides examples of our work with scientific research projects in two research organisations to develop real-time indicators of the research progress. Finally, it is argued that the continued development of similar indicators can play an important role in managing research to achieve radical innovation. Keywords: innovation; basic research; R&D; performance; measurement. Reference to this paper should be made as follows: Mote, J., Jordan, G. and Hage, J. (2007) 'Measuring radical innovation in real time', Int. J. Technology, Policy and Management, Vol. 7, No. 4, pp.355-377. Biographical notes: Jonathon E. Mote is an Assistant Research Science at the University of Maryland in the Center for Innovation. Gretchen B. Jordan is a Principal Member of the Technical Staff at Sandia National Laboratories, New Mexico. Jerald T. Hage is Professor Emeritus and Director of the Center of Innovation at the University of Maryland.
1
Introduction
While radical innovation is widely recognised as an important goal, how does one actually manage and measure progress towards that goal? Everybody wants to increase radical innovation but nobody knows how to do it. Numerous recent studies have emphasised that the pursuit of radical innovation is something that must be managed differently from other research pursuits. Most prominently, Leifer et al. (2000) have argued that the management of radical innovation projects necessitates different strategies and practices, including different ways of measuring performance, than those directed at more incremental innovations. While the measurement of research performance has been an important issue at least since the emergence of 'Big Science' (Blankenship, 1974), it is clear that radical science needs to find more appropriate measures of scientific and technical progress. Given the heightened uncertainty of radical research, not only in terms of outcomes but also the scope of unknowns in the research process, the ability to manage this type of research effectively is hampered by the lack of adequate measures to assess ongoing progress and link progress to achieving higher-level goals. This kind of information provides both focus and inputs to decisions on the level of uncertainty and on research direction and continuation. In this regard, the ability to measure and assess progress in real-time could have potentially wide ranging consequences for improvement in the management of radical innovation. In this paper, we will be discussing just one aspect of performance assessment of research directed towards radical innovation, that is, measures of progress to assess progress towards stated goals. This narrower focus is akin to 'in-progress peer review' for example, and does not address project selection or assessment of societal outcomes. Further, we will not be discussing measures of research outcomes or processes. Rather, we will only be addressing measures of progress that eventually impact on outcomes or transitions to subsequent research. Our scope is further narrowed in that we focus solely on scientific and technological research at the basic, applied and early product development stages. Most importantly, the measures we discuss are in-progress (realtime), not prospective or retrospective. We argue that these real-time measures could be most helpful for managing radical innovation, particularly if they meet three criteria: continuous variables specific to an organisational performance goal; measured within a planning cycle without a time lag; and describe the extent of progress. While we argue that real-time measures would be useful for individual research projects, it is important to recognise that research projects do not exist in isolation. In the context of R&D organisation, particularly a large organisation with extensive portfolios of research projects, it is critical to develop measures that are commensurate across a range of projects. Hence, such measures should have the ability to aggregate to higher organisational levels and goals and allow for comparison across projects. In demonstrating the efficacy of real-time measures of radical innovation, we draw on the efforts of two studies that aimed to develop the types of measures and indicators of research performance discussed above, and, we would argue, these types of measures could play a key role in the measurement and management of radical innovation. The first study was supported by the US Department of Energy (DOE) and focused on a large multi-disciplinary national research laboratory comprised of a broad range of research units, centres and programmes. The second study was supported by the National Oceanic and Atmospheric Administration (NOAA) and sought not only to develop measures of
Measuring radical innovation in real time
357
research performance, but to incorporate these measures into a scheme that allowed for these measures to be applied up, down and across. In both projects, our approach to developing measures was largely inductive in nature, as we worked with scientists, engineers and technicians to define measures that made sense from the standpoint of the researchers. Overall, we achieved enough progress to assess the applicability of these measures to radical innovation and to indicate how we plan to improve upon what we have accomplished so far. While we suggest that the measures we developed are useful to all types of research, we argue that they could be especially useful in the measurement of radical innovation due to the higher levels of risk and uncertainty associated with this type of research. After a discussion of current measures of research performance and the need for real-time measures, we discuss the types of measures that are most appropriate for basic and applied research. Next, we provide a largely descriptive account of the two studies and the progress measures we developed. The paper concludes with a discussion of the results and implications for further research on the measurement of progress of radical innovation in basic and applied research.
2
The need for real-time measures of research progress
Why is measuring real-time progress important? We would argue that real-time measures are necessary for two primary reasons: accountability and programme improvement. As we discuss below, both of these items, which are necessary for the strategic management of research, are affected by the timeliness, or lack, of existing metrics. But it is important to state that we are not arguing for metrics that displace current metrics. Rather, we argue for complementary metrics that allow for appropriate measurement of the progress and advance of research. With regard to accountability and control, it is critical to demonstrate to investors and funders that research funds have been spent well. From the point of view of researchers, the continued flow of funds is more likely if this need for accountability is better met. But it is not sufficient to know that research is relevant, that the 'right research' is being conducted, or that past research was successful. As Feller (2002, p.444) states with regard to federally funded research, "perhaps the most telling limitation of performance measurement as applied to science policy is that whatever its value may be in tracking past performance, it is of limited value for prospective decisions". Some examples of existing measures that focus on outcomes would include: economic outcomes, like the degree of commercial success (Averch, 1994; Pavitt, 1991) or cost-benefit analysis (Jensen and Warren, 2001); bibliometric outcomes, such as publications and citations (Cozzens et al., 1994; Rinia et al., 2001); or technical outcomes, such as patents (Grupp, 1994; Narin et al., 1984; Zoltan et al., 2002) and awards and honours (Schainblatt, 1982). In general, these measures of research performance do not address the progress of the research and are lagging indicators. While lagging indicators can be useful in confirming patterns of past performance, the length of time between the actual research performance and traditional research metrics restricts their usefulness in measuring current research performance and addressing the need for control and accountability. Similarly, Geisler (2000) argues that the existing set of metrics suffers, in part because they miss the temporal dimension between scientific outputs and technology accomplishments.
358
J. Mote, G. Jordan and J. Hage
The issue of programme improvement speaks to the heart of R&D management and the need to improve research performance and take strategic action. In this regard, it is important to recognise the maxim that 'what gets measured, gets done'. Hence, it is essential to institute measures that stimulate performance towards strategic goals rather than hinder performance and result in goal displacement. For example, putting process indicators in place, such as quarterly reports, may be in conflict with the activities needed to produce the very value that comes from research. In this manner, the advance of science becomes secondary to meeting and completing arbitrary benchmarks that satisfy a bureaucratic need. In other words, the short-term criteria for control are often in conflict with the long-term criteria for evaluating value and strategy realisation. But in addition to stimulating progress and providing focus, it is essential that managers have relevant metrics that allow for making appropriate decisions regarding whether to continue or discontinue the research, maintain or change direction, or add resources or not in order to achieve desired goals. Existing methods of gathering information about research for project management, include benchmark or stage-gate methods (Coombs et al., 1998; Cooper and Kleinschmidt, 1995) and process management (Schumann et al., 1995). Arguably the most influential of these methods, the stage gate process, assesses progress at a series of 'gates' between idea generation and development and commercial launch. The widespread acceptance of the stage gate method is indicative of a need and desire to collect better information on current progress and provide guidance on project definition including scope and outputs. However, we would argue that such methods, particularly for the fuzzy front end that we are addressing, suffers from the same problem as other common measures. Progress is mostly defined in measures that are concrete, such as publications or the completion of an experiment or prototype, or in terms specific to a particular programme rather than general dimensions of performance that can be compared across similar projects or programmes. In summary, the existing types of progress measures, while useful for measuring various aspects of the research process, lack the ability to determine the progress or advance within the process of research itself. While the need for more timely measures ofprogress is a general one in research, it is especially critical for research aimed at radical innovation. It is to this topic we turn in the subsequent section.
3
The need for real-time measures of radical innovation in research
Before turning to a discussion of measures, it is important to ask a more basic question: What is the end goal, that is, what is radical innovation? Definitions differ, often times greatly. Typically, radical innovation is conceptualised as dramatic increases in functionality, performance or dramatic decreases in costs and time. But as McDermott and O'Connor (2002) pointed out with regard to decades of research on radical innovation, 'Researchers are far from a consensus regarding a formal definition of radical innovation'. We will not pursue a lengthy review of efforts to characterise radical innovation in this section, as this has been done more ably and exhaustively by others (see for example Garcia and Calantone, 2002; McDermott and O'Connor, 2002; Leifer et al., 2000).
Measuring radical innovation in real time
359
While definitions may differ, it is nonetheless clear that radical research, by which we mean basic and applied research that aims for radical breakthroughs, has several distinctive characteristics. First and foremost, radical innovation typically involves higher levels of uncertainty. A great amount of uncertainty arises from the fact that radical research involves a greater number of unknowns in the research, what Roussel et al. (1991) called technical novelty and Raz et al. (2002) called technological uncertainty. As Deuten and Rip (2000) discuss, the open-ended and non-routine nature of the research process itself makes innovation hard to predict. Because of this, the goals and outcomes of radical research can be uncertain, if not indefinable. Further, radical research must be flexible and opportunistic in order to capitalise on new bits of knowledge and insight, with the recognition that radical research produces intangible outcomes (learning) as well tangible outcomes (widgets). Finally, because radical research is not a straightforward endeavour, progress can be variable. Indeed, radical innovation often comes in the form of a step function, with progress coming in great leaps forward. Given the inchoate nature of radical research, it is understandable that the measurement of progress in radical research is also somewhat indeterminate (Green et al., 1995). Despite the challenges of measuring radical research, it is important to build on what is known and has been already done. Specifically, we can look to what has been done with technology, where radical is often characterised as a new feature or function. Typically, the radicalness of the technological innovation is characterised as some measurable increase in performance or decrease in cost compared to a fixed reference or baseline, such as the current state of art in the field. For example, Leifer et al. (2001, p.103) provide a good rule of thumb: "at least a five-fold improvement in known performance features, and a significant (30% or greater) reduction in cost". When such measures of degrees of radicalness are feasible, it is also generally possible to assess risk and the probability that this technical performance or cost is achieved is increased (Raz et al., 2002). As we discuss in greater detail below, these can also be applied to radical research with slight modifications. This most clearly applies to scientific advances that involve a tool or a technique, such as the speed and cost of genome sequencing, or increases in understanding, such as the properties of phenomena and materials. But in research, there is also a distinct need to apply measurement to 'learning', particularly in terms of hypotheses pursued (or eliminated) and theory building. Because we are talking about radical research, the fuzziest of the fuzzy front end, it is helpful that the measurement of radical research also relates to higher goals, within the research organisation, specifically, and the S&T community more generally. Two examples of organisational goals drawn from the public sector include increasing the lead time and accuracy of winter storm warnings (Department of Commerce, 2003, p.63) and an increase in the measurement of properties and interaction the top quark (Department of Energy, Office of Science). In this manner, direct linkages of radical research to such higher goals provides the same sort of direction and discipline that the market lends to new product development. But given the high degree of uncertainty of radical research, it is critical that performance metrics provide timely information on research progress in order to determine whether research is making sufficient progress towards stated goals, or whether new and unpredicted outcomes have occurred. In this manner, such metrics
360
J. Mote, G. Jordan and J. Hage
positively contribute to a key factor identified by Pavitt et al. (1989) for the successful management of innovation, that is, the flexibility and speed of the decision making. Similarly, Leifer et al. (2000) point out:
"The project manager's job in the radical innovation context is less one of listing tasks, delegating them, and controlling progress, and more one of monitoring where project stands on each uncertainty dimension and making decisions about which uncertainties to attack at any point in time."
Hence, measurement that can capture progress in real time that has these characteristics would provide very useful information to managers.
4
Assessing research and potential measures
4.1 Assessing the radicalness of the research
Before we turn to the different types of research to explore potential measures, it is useful to first address more specifically the challenge of determining whether an innovation is radical or not and the degree of that radicalness. On this topic, we focus on two primary metrics: the degree of change and the scope of research. We suggest that these are the standards by which to assess the degree of radicalness for whatever progress metric is defined. Thus they are essential to keep in mind when making prospective or retrospective determinations about radical innovations. As Garcia and Calantone (2002) suggest for new product development, it is more appropriate to assess innovation along a continuum, with varying degrees of innovativeness. Hence, the degree of change introduced by the innovation is an important element of radicalness. Unlike new product development, however, a baseline or state of the art for basic and applied research and early engineering and development can be difficult to determine quickly or easily or even conclusively. For instance, while it is relatively easy to determine baseline measures for piezo-resistance pressure sensors by assessing what is currently commercially available (Frenkel et al., 2000), it is altogether more difficult to determine a baseline for something like rarified gas dynamics, where research on the topic is being conducted on various aspects of the phenomena in laboratories around the world. Hence, it is even more essential to determine the extent of radicalness in these instances. Because radical innovation is often marked by steep advances, the degree of change can often be readily identified. But in those instances when it is less clear, particularly to non-technical managers, a key tool is the use of expert judgement, or peer review. In our experience, this is seldom done routinely, as when a major breakthrough is reported before it has been confirmed by other experts. For instance, the announcement that University of Utah researchers had discovered a method for cold fusion in 1989, or, more recently, the work on stem cells by South Korean researcher Hwang Woo-suk. Another important consideration is the scope of the research. Although we did not directly address the scope of research in our studies, it is an important aspect of our work going forward. At its most basic, the scope of research involves the number of variables or processes that are being researched at the same time. Some disciplines have a systemic quality, that is, a large number of variables have to be considered at the same time. Further, not all scientific problems can be approached with small research teams; some of them require a large scale focus. Again, consider the example of the Human Genome
Measuring radical innovation in real time
361
Project. While progress in mapping and sequencing the human genome could have proceeded with small research teams, it took a large scale programme to coordinate a range of efforts so that the time needed to complete the entire genome was appreciably lessened. Indeed, the systemic quality of some types of research is frequently overlooked as a critical dimension to the scientific problem. One example of what might be called a systemic problem is research on the weather, which encompasses both oceanic and atmospheric systems. NOAA was created in 1970 to unify and coordinate the government's research efforts on various aspects of the global environmental system, including the National Weather Service (NWS). Altogether, the scope of NOAA's mandate necessitates quite expensive and specialised equipment and teams to collect the relevant data, including satellites, ships, buoys, and planes. Indeed, one could argue that the choice to conduct large scale data collection represents one form of a broad scope of focus, such as global climate studies or the use of hyperspectral data in weather satellites. Two comments that might provide some additional explanation on this topic. First, the scope of the focus is not determined by how fundamental the research is in our theory. For example, Einstein's theory of special relativity is quite fundamental but consists of only three variables (radical because of its centrality to field and because of the broad scope of its impact). Second, as everyone is familiar, there are multiple levels within each discipline and therefore there can be multiple systems. In this respect, a broad scope of focus means that one is studying multiple levels or multiple systems.
4.2 Assessing different aspects of research progress
Some concrete measures of progress are suggested by the differences that one finds in the research process itself. In this vein, we would differentiate between technology research, focused more on products, and scientific research, focused more on discoveries. Within each of these there are differences that suggest different types of measures of progress. We should point out that we are primarily concerned with research at the project level, with the recognition that a critical challenge in utilising any measurements is aggregation across various levels of organisational structure.
4.2.1 Measures for technology research
In this category, research is typically focused on the earliest stages of development of a new product. Hence, the most important measure is the increase in functionality of the product relative to its predecessor or competition, which allows for straightforward measures of performance characteristics. In addition, such measures can be aggregated across products which share similar functional characteristics. For example, such measures might include:
• • • • •
miniaturisation or size reduction efficiency (in operating costs and training) speed of use or employment precision of use flexibility (frequently ignored), i.e., how easy is it to change from one use to the next.
362
J. Mote, G. Jordan and J. Hage
It is clear that measures that assess the extent of improvements in various performances lend themselves quite easily to the assessment of progress and innovation. But measures of progress towards the achievement of such improvements are complicated by how functionality is achieved. In some instances, the functionality simply reflects the replacement of one component with another, for example, the replacement of analog components with digital components in radar systems. But functionality might involve the synthesis of components, such as the combination of software and hardware components in the development of semi-autonomous modular robotic control technologies. Technology research can also be involved in improving the process of manufacturing products as well. Process measures would include the extent of the change in the machines used to produce the product. Similar to the measures suggested above, such improvements would be typically measured indirectly by the improvement in the following kinds of performance:
• • • •
increases in productivity increases in reliability increases in the flexibility of manufacturing (ability to move from one kind of product to the next on the same production line) reductions in the use of various resources (e.g., energy).
In general, the measures for technology research are closely akin to the types of measures that one typically finds in the new product development literature. But it is important to keep in mind that the measures we discussed are not directed towards assessing outcomes, but towards assessing progress towards outcomes.
4.2.2 Measures for scientific research
In scientific research, we would differentiate among three aspects of research, analytical, descriptive and control, and suggest measures for each. The first is analytical, that is, the explanation or characterisation of a particular phenomenon. As would be expected, this is an important focus of much scientific research. The characterisation of phenomena typically proceeds through repeated trials of experimentation and measurements, with the goal being the development of an accurate model of the phenomena and a more complete understanding of the phenomena's properties. Put another way, the goal of characterisation is to reduce the uncertainty of knowledge about a particular phenomena. Interestingly, we note that a great deal of analytical research now relies on the use of simulation as a tool in characterisation, which offers great potential for the use of validation metrics in measuring progress, such as precision of prediction and reduction of uncertainty. In addition, it is possible to include the complexity of the model as a measure, such as the number of unknowns or processes accounted for in the model. The key here is the percent improvement in the precision of prediction and the percent improvement in the processes included in the model. Another aspect of scientific research is largely descriptive, that is, the portrayals of new patterns. The emphasis is on discovery and the initial identification of patterns. For instance, NOAA researchers have recently designated a new type of hurricane, one that is symmetrical in formation and maintains its intensity much longer across
Measuring radical innovation in real time
363
ocean waters than current models predict. In this aspect of research, some suggested measures would be how many new patterns are discovered and how many concepts and hypotheses are developed to explain the pattern. In other words, the progress of research is measured in the movement towards explanation. The extent of radicalness in the description might be the degree of movement from a concept to a hypothesis to a theory to a paradigm. Intermediate between description and explanation of patterns is when it is possible to exert control to obtain some result. The extent of control in the properties of the phenomena reflects a kind of knowledge even if it is not clearly identified. We would argue that this constitutes an important element of much nanotechnology research where they are able to produce materials with fewer impurities and with certain improvements in performance characteristics such as transmission of electrical current. However, the use of the term control does not imply that there is a theory in place to explain the produced result. Apart from measuring progress of the research itself, another potential area of measurement is innovations in the research process. This is often an overlooked aspect of research and reflects the addition of a new instrument or technique for measurement, or improvements in existing ones, and the performance characteristics of that instrument or technique. The measure is the improvement in the measurement of the phenomena or function that is being measured, or the 'quality of the measurement'. For example, the ability to predict when the orbits of polar-orbiting weather satellites overlap could provide a significant improvement in the ability to calibrate all earth orbiting satellites as well as increasing the data quality for long-term climate studies (Cao et al., 2004). As we discuss in greater detail in the next section, the measures we developed for the projects in our studies closely follow the measures suggested above. As we hope to demonstrate, these measures are amenable to measuring progress of radical research as they allow for an assessment of the progress of research towards innovative outcomes. Once progress measures have been defined, however, it is not only objective quantitative measurement of research performance, but also objective, qualitative judgement of peers (Schumann et al., 1995), as to what constitutes a large or radical advance. In the next section, we present some of the measures we developed and discuss how they might be applied to radical innovation.
4.2.3 Measures for learning in research
Before we turn to the next section, however, it is important to address the notion of learning in research. During the process of developing research measures, it became clear that some of the projects we studied had not yet advanced to the stages of research where measurement of progress such as we described above was appropriate or applicable. In several of the projects this learning emerged in the face of new experimental results or difficulties with instruments. Often times, the learning resulted in some amount of reorienting or restructuring of the project. Clearly, however, learning is not simply a result of the 'fruitful error' in science, but is an integral component of all research. We would suggest that the learning process involves, but is not limited to, the identification of challenges or opportunities, learning something about these, and, in the process, closing in on some and perhaps identifying new challenges. In any event, this aspect of research is one that, while difficult to measure, is essential to the progress and advance of research.
364
J. Mote, G. Jordan and J. Hage
We discussed the desirability of attempting to measure learning and many of the researchers we interviewed, although not all, agreed that this might be potentially helpful. Although this type of progress is much harder to quantify, we feel it is important because it represents an important qualitative approach to the problem of technical progress. And this aspect is particularly important for research aimed at radical innovation given the high level of uncertainty and the potential for changes in direction or approach.
5
Our approach to developing measures of research progress
In this section, we report in much greater detail on the real-time progress measures that were developed in each research organisation and how they might be applied to radical research. Because of the unevenness in responses, measures of performance were not fully operationalised for all research projects. By fully operationalised, we mean that technical measures were agreed upon, data was collected on the measures, and progress on the measures was tracked retrospectively to the present. The two organisations we studied represented very different organisational settings. The first organisation was a large-scale, multi-disciplinary national laboratory. The organisation has a billion dollar annual budget and currently employs several thousand researchers. Research is conducted in a diverse range of fields, from physics and engineering to nanotechnology and biology. And the research work is organised in different ways, from small and large projects to long-term to short-term projects, and single discipline projects to multi-disciplinary projects. In addition, researchers typically work on several projects at the same time. The second organisation was a smaller-scale research unit housed within NOAA that focuses primarily on applied research on satellite data. Research is conducted in a narrower range of fields, primarily oceanography and meteorology, and the research work is organised around particular satellite systems, such as NOAA's family of geostationary weather satellites (GOES). Despite the differences in the organisations, data collection proceeded along similar lines at both sites. In our studies, we had two primary objectives for the measures of research performance. First, we were interested in real-time measures, that is, measures that would be useful to researchers and the management of the research work. The measures should be continuous and apply throughout the life of the project. We were motivated to address the limitations that we had identified with current metrics of research performance, primarily the temporal dimension of lagging indicators and to capture full value of the progress. Hence, we focused on developing measures within the process of research itself, with a particular focus on assessing of the health of the research environment and measures of 'outputs' or progress that occur prior to papers and provide more descriptive, rather than event-based, milestones. Following naturally from the first objective, our second objective was to focus on the scientific and technical aspects of the research. In this respect, we were interested in gaining insights from research personnel on their own interpretations of technical progress, and to conceptualise measures of technical progress that were meaningful to the researchers themselves. In short, we were interested in how researchers determined they were making scientific or technical progress or not. The development of metrics (units of measurement for the measures) for each of the projects was based on ongoing interviews with project principals and a thorough assessment of research materials. For each project, we sought to develop three measures
Measuring radical innovation in real time
365
of research performance that could be used to assess progress of the research throughout the life of the project. In many respects, our approach was similar to some approaches found in the technometrics and technological forecasting literature, such as Frenkel et al.'s (2000) analysis of industrial sensors, Grupp and Hohmeyer's (1986) work on technological standards, and Lenz's (1985) work on a heuristic approach to technology measurement. However, these approaches also differed from ours in a number of respects. First, this literature is primarily focused on the macro- or industrial level, rather than at the level of individual projects. Second, this literature tends to focus on research that is much closer to the market, such as new product development, rather than the fuzzy front end of basic and applied research. In many respects, our approach was similar to that of some of the US Department of Energy's Office of Science (DOE-SC) work on improving the performance measurement practices of Federal basic research organisations, particularly the DOE-SC's effort to develop metrics of research that satisfy the requirements of the PART. For example, Figure 1 illustrates the DOE-SC's approach to identifying a roadmap with specific metrics for research focused on advanced scientific computing. As the figure shows, the DOE-SC articulated a long-range goal at the level of scientific discovery and identified a key number of appropriate metrics to assess progress. However, our work differed in one important respect. Our work was focused on the level of the individual research project, not research portfolios or programmes. But as we were mindful of the importance of aggregating measures to higher levels of organisational structure, we sought to develop metrics at a level of generality that might allow for this possibility.
Figure 1 DOE-SC roadmap for scientific discovery needs for advanced
200
Deliver operating systems for scientific computers that incorporate fault tolerance (2007)
200
201
201
201
Complete programming model that enables scientists to use 100 000 processors (2009)
Deliver algorithm that scale to tends of thousands of processors key mathematical libraries (2007)
Deliver mathematics of complex systems that enables simulations of microbes (2013)
Deliver mathematics of complex systems that enables accurate linkage of multiple time and length scales (2011)
Demonstrate progress towards developing the mathematics, algorithms and software that enable effective, scientifically-critical models of complex systems (2015)
5.1 Sandia National Laboratories
We examined 22 research projects at this large, multidisplinary national laboratory. After an initial review of project documents we found that the projects could be sorted into two general categories, science and technology. The first set of eleven represents technological projects whose objective is to build an instrument, for example a sensor or mobile robot or catalytic converter. The second set of 11 represents science projects whose objective is to build the underlying knowledge about some specialty, e.g., the physics and chemistry of ceramics, plasma physics or the detection of proteins. In several cases, the science is designed to provide the basis for a new technology, for example the
366
J. Mote, G. Jordan and J. Hage
modelling membrane transport. Finally, in a few other projects, although the objective is a new technology, the project itself is more concerned with the science involved, e.g., the synthesis and characterisation of nano-scale energetic materials. The distinction between science and technology projects became important for three reasons. First, the measures that were developed were quite distinctive for each category. Most of the science projects were concerned with increased understanding of various phenomena, which can sometimes be characterised as the precision of experimental prediction, while many of the technology projects are involved with measures of reliability. Second, as it became apparent in our review of the results of an organisationwide survey, the science and the technology projects value aspects of their research environments quite differently (Jordan et al., 2005). Finally, the aggregation of measures was much simpler with the technology than with the science projects. In Table 1, we provide examples of measures of progress for technology research from our study. In addition to identifying specific measures for each project, we attempted to derive a set of measures that would allow for aggregation across projects. These aggregate measures were derived from a thorough review of project materials and identifying similarities and patterns. While the specific measures varied considerably, we were nonetheless able to aggregate the measures into several common categories, particularly along the dimension of science and technological projects.
Table 1 Examples of technology measures
Selected technical measures Aggregate Reliability Specific Overcoming dissipation processes Elimination of leakage, solvent resistance Reduction in high temperature corrosion Efficiency Improvement in harvesting isolate Ability to analyse multiple samples simultaneously Increase power output Precision Increase in dynamic frequency range to 10 GHz Increase in sensitivity in parts/million Increase in accuracy of micro injector Flexibility Band width selectability Performance scalability Number of toxins/agents detected Miniaturisation or reduction in size Decreases in mass size of particles (150 atoms) Decrease in receiver size/weight Decrease in size/weight of SARS module
Beyond the five categories in the table, the issues of ease of use and speed of use were also relevant in a handful of technological projects. Each of the 11 technical projects also involved a measure which can be termed functionality (and which could potentially contain multiple indices). Functionality represents progress relative to the basic goal of the project, such as the development of a hand-held device to detect pollutants without
Measuring radical innovation in real time
367
the use of chemical reagents. As we pointed out earlier, ideally, this progress and the degree of radicalness could be reported relative to the basic standards - the state of the art in the appropriate technological field. The most comprehensive technical measure developed was one where the project manager worked with a staff member to combine several of measures into a single figure of merit, including the initial measures we had agreed upon (see Figure 2). This is similar to the measure described by Lenz (1985) of a hierarchy of parameters whose values could be traded-off, but which culminate in one measure of performance. The approach taken by the project manager was to identify several critical variables for the development of a key element of the overall project, a tunable microwave filter. Each of these variables was then normalised and combined in such a way that progress on the filter could be tracked on a scale of 0 to 1, although the goal of the project was 0.3. The project manager then tracked the progress of the filter through a number of design iterations, as illustrated in Figure 2. In this manner, the measure becomes a useful single measure for showing the interactions among several variables, as well as allowing for useful comparisons across projects conducting research on filters using similar variables. On the opposite end of the spectrum were technical measures that were quantifiable, but were not fully illustrative of progress. Typically, this was due to a number of factors, such as limitations of the item chosen to be measured or the unit of measurement,1 or a small number of data points (both in terms of what was being measured and how long it had been measured). The limitations inherent in these technical measures meant we did not fully capture the progress of the projects within the scope of our study.
Figure 2
0.300 0.250 Actual measured FOM 0.200 FOM 0.150 0.100 0.050 0.000 10/1/2001 Ideal FOM for this filter
Filter figure of merit versus time
1/9/2002
4/19/2002 7/28/2002
11/5/2002 2/13/2003 5/24/2003 Date
9/1/2003 12/10/2003
In the 11 science projects, the specific measures varied, but there was remarkable similarity across a number of projects (see Table 2). In this respect, we identified two primary aggregate categories that are common in seven instances. These are the precision of the prediction and the complexity of the model, that is, the number of variables, forces, or processes involved. These categories partly arise from the increasing use of modelling and simulation in combination with experimentation, which allows for the use of validation metrics (Trucano et al., 2001). In general, the science projects typically focused on improving scientific models, and improvements in the predictive accuracy
368
J. Mote, G. Jordan and J. Hage
of the models represented advances in scientific understanding. However, the attempt to develop or utilise common categories of measures was more difficult in science projects that did not involve this approach. For instance, some science projects involved gaining a preliminary understanding of novel phenomena, such as a project that focused on the formation of plasma crystals. In general, this kind of research is largely descriptive as it moves towards the specification of models, and it is difficult to quantify performance on a real-time basis. But such progress can be indicated in terms of descriptive progress and/or learning.
Table 2 Categories Analytic Selected science measures Specific Increased precision of model of problematic horizontal permeability anisotropy Increased precision of model of transient strain on thermophoretic effects and soot formation/oxidation mechanisms Improved measurement of the effect of insufficient oxygen on the process of soot formation Development of stable atomistic simulation of lipid bilayer Development of model of friction at the mesoscopic scale Control Increase optimisation of novel MEMS friction structures Increase number of precursors for ceramic materials Increase ability to tailor properties of precursor-derived ceramic materials
Descriptive
In terms of measuring and managing research performance, the specific and aggregate measures developed at the national laboratory, as well as the reception of the measures by managers and researchers, indicated that this was a useful exploration. In this manner, we were pleased to demonstrate that specific and aggregate measures could be developed from the 'bottom up'. Finally, while above we have indicated how the specific measures can be generalised, and in turn these general measures can be aggregated, this does not solve all the many issues of a monitoring and evaluation system that functions at multiple levels. This was easier for the larger, broad scope projects because the project typically works to develop a set of progress and project goals to communicate and manage across levels of organisation. However, we recognise that in the context of the national laboratory, the practical application of the measures that we developed are limited. First, the 22 projects studied do not even begin to scratch the surface of overwhelming complexity of the laboratory's overall research portfolio. While aggregate measures were conceptualised for the projects we studied, a top down look would be needed to see how these measures could accurately and validly measure the heterogeneity that exists within relevant parts of the laboratory's work. Second, our study did not take into account the particular interests and concerns for measures at the upper levels of management, including national interests or GPRA and PART considerations.2 While this latter concern was not our charge in this study, it is of primary importance for developing relevant measures that can be utilised for performance measurement at various levels.
Measuring radical innovation in real time
369
5.2 NOAA's office of satellite applications and research
Building on our experience at the national laboratory, we subsequently embarked upon a similar exercise with NOAA's Office of Satellite Applications and Research (STAR). However, the aims of our effort at NOAA were much more ambitious (Colton et al., 2004). Specifically, our aim was not only to develop measures of research performance, but to develop a framework that would provide a monitoring and evaluation system for all of the research of about 150 researchers at STAR. We set out three primary objectives for our work: 1 2 3 respect the need for measures of technical progress at the research project level measure accomplishments at the project and organisational level establish linkages between project-level measures and the specific performance targets that have been established by NOAA as part of their planning process.
As with our work at the national laboratory, we set out to inductively derive measures of research performance as the first step in the development of a performance management system. The scope of the paper limits a complete discussion of our efforts, but below we provide examples of measures developed that attempt to meet the objectives above. STAR is housed within NOAA's National Environmental Satellite, Data and Information Service (NESDIS). NESDIS is an operating branch within NOAA that is charged with managing the operational environmental satellites and providing a series of weather-related data and information services. The primary function of STAR is two-fold: 1 2 researching, developing and using data products from new and existing weather satellites maintaining the quality and integrity of satellite-derived data.
STAR consists of three research and applications divisions that encompass satellite meteorology, oceanography, climatology, and cooperative research with academic institutions. The satellite-derived land, ice, ocean, and atmospheric environmental data products provide the basis for most, if not all, weather forecasting models of the NWS, another operating branch of NOAA. In addition to the working with current operational weather satellites, STAR plays a key consulting role with NASA in the design and development of new weather satellites and instruments.
5.2.1 Cross-project measures
As we began to interact with each of the eight selected projects, we realised that there was much more to the activities or tasks that these projects performed than what might be called original research. Although the work conducted by STAR is typically characterised as research, many of them in fact do not necessarily spend the majority of their time conducting research as it is normally defined. More typically, many of these projects focus on problems of quality control, particularly those activities identified as calibration and validation of satellite data. Since the work of STAR, and NOAA in general, relies on the quality of satellite data, this work is a critical component of the work of STAR. But it is also not the kind of research activity that traditional measures of performance would adequately capture.
370
J. Mote, G. Jordan and J. Hage
In general, seven of the eight we studied projects involved some amount of calibration and validation activities, which consists of four major activities: calibration and validation of satellite instruments, correction of existing algorithms, development of new algorithms, and the construction of time series. At one extreme are projects whose primary concern is with calibration/validation, such as the project centrally concerned with calibration (Cal/Val) and a project focused on the development of new datacompression algorithms. At the other extreme are projects whose primary concern is the improvement or development of new algorithms focused on specific weather phenomena, such as tropical storms and ocean winds. In general, a great deal of time is spent on continuous calibration and some on periodic validation and therefore much energy is devoted to the problem of maintaining the algorithms/time series or updating them in the light of new calibrations/validations. We began to define calibration/validation activities as 'quality control', and recognised a significant contrast between these activities and conducting sustained research that could result in the development of new methods of calibration and validation. Interestingly, these new methods have the potential to reduce the amount of time spent in quality control. Since the amount of time spent on quality control is such an important aspect of the work of many if not all of these projects, measuring the amount of time allocated to quality control was identified as a central performance measure common across most STAR projects. The measurement of quality control is significant because an existing performance measure at STAR is the number of new data products developed. However, the development of new products necessitate a considerable amount of research time and effort in developing new algorithms before the products become operational. In this manner, the recognition and measurement of quality control activities could have a significant impact on the amount of time allocated to more research-oriented activities. In addition, measures for each of the calibration/validation activities were derived and as listed in Table 3.
Table 3 Performance measures for calibration/validation Algorithms Increase in duration of algorithm Decrease in size of errors in data Increase in coverage data Time series Increase in duration of time series Reduction in inter-annual variation in data Increase in coverage of data
Calibration/Validation Reduction in frequency of calibration/validation Reduction in size of errors Increase in coverage of data
While the work of calibration and validation represents largely routine work and incremental innovation, researchers had developed a novel method for accurately predicting simultaneous nadir overpasses among different polar-orbiting meteorological satellites (Cao et al., 2004). As mentioned above, this represents a potential radical innovation, as it has the potential of significantly increasing the calibration consistency and traceability required for long-term climate studies by reducing the uncertainties in observations among weather satellites. The application of the performance measures developed (above) in the testing of this method can help managers determine whether this represents a radical innovation, and whether the costs associated with its implementation are justifiable.
Measuring radical innovation in real time
371
5.2.2 Project measures that link to science and mission goals
The exigencies of GPRA and PART have placed greater emphasis on evidence of impacts on mission goals, as well as on reducing costs and thus productivity. Some of the measures that were developed for STAR (Figure 3) address both forms of impacts and includes what we viewed as an important corollary to quality, that is, product development time. Of course, this measure is well-known in the industrial and new product development literature. STAR had recognised its importance by establishing a task force to look into ways of improving it, but had not actually defined a measure or measured it. Given STAR's location with a complex inter-organisational network, however, these measures of impact depend a great deal on communication and coordination. But these measures, along with shorter product development time, lies at the heart of STAR's research efforts. We would suggest that the use of such measures, while recognising the importance of inter-organisational partners and the need to innovate, allow STAR simultaneously to meet reporting requirements and better align research efforts mission goals.
Figure 3 Impact on weather prediction models
• • •
Improved prediction (accuracy, precision, coverage) Development of improved models (either more less variables) Reduction in product development time
To better illustrate the application of these measures, we will provide an example from a project that is focused on the development of data products on wind. In addition to maintaining current operational products, the wind project has been working on developing a new data product on winds in polar regions. The data is gathered from MODIS3 instruments aboard the Terra and Aqua satellites that circle the earth along a north-south flightplan. The Terra and Aqua satellites were launched in 1999 and 2002, respectively, as part of NASA's Earth Observing System, and both remain under NASA, not NESDIS, management. The advantage of developing products from these instruments and satellites is that they provide coverage in the polar regions of the globe, areas where wind data is lacking. Development of the algorithms to derive the wind observations was conducted in 2001 and 2002. Routine production of wind observations from the MODIS instruments, although still experimental, was established at NOAA/NESDIS in July 2003. The next step in the development of any satellite data product is the assimilation into weather prediction models and conducting impact studies, in terms of improved prediction, on forecasts. While STAR typically provides such data to other research centres around the world for impact studies, measuring the linkage to NOAA's mission goals is dependent on the data being assimilated into the prediction models of the National Center for Environmental Prediction (NCEP), housed within the National Weather Service. In essence, NCEP is the primary conduit for all meteorological data and provides the official NOAA forecasts of weather and environmental information. Hence, the development of a new product at STAR and, more importantly the determination of its radicalness and its linkage to NOAA's mission goals, depends on the assimilation and utilisation of data by NCEP.
372
J. Mote, G. Jordan and J. Hage
Another example of the application of measures for radical research is NOAA's work on exploring novel methods of data compression. New methods of data compression are critical for the next generation of Geostationary Operational Environmental Satellites (GOES), with the first of this series (GOES-R) to be launched in 2012. These satellites are expected to have sensors with resolution in the hyperspectral range, which would generate a raw unprocessed data stream of 130 Mbps and a processed data rate of 85 Mbps. Current GOES satellites have raw and processed data rates of 2.6 Mbps and 2.1 Mbps, respectively. Hence, the move to hyperspectral data will involve potentially significant increases in the power requirements and mass of satellites as well as the cost of broadcast rates and the archiving of data. In developing new methods of data compression, this project can draw on well-known existing state-of-the-art data compression methods, such as JPEG2000, in order to determine progress. However, the data represents a challenge to these methods because it is 3-dimensional (most compression methods are for 2D data) and any compression method must be lossless (or near lossless) due to the high sensitivity of error. The research project has been able to develop new compression methods that produces significant compression gains in the range of 50% over current state of the art methods, but improvements in compression rates were not tracked and utilised in assessing the performance of the research. The use of such measures, in conjunction with peer review to determine the radicalness of the research, would help to better determine the rate and scale of progress and allow managers the ability to better assess next steps and strategy. At the organisational level, the ability to develop measures and linkages to goals speaks to the need for a comprehensive performance management system. It would be necessary for such a system to address multiple impacts, as well as provide specific guidance in terms of both a technological (or research) roadmap and where to take corrective action, if needed. In our work with STAR, we utilised the Strategy Map and Balanced Scorecard approach of Kaplan and Norton. As they describe:
A strategy map for a Balanced Scorecard makes explicit the strategy's hypotheses. Each measure of a Balanced Scorecard becomes embedded in a chain of cause-and-effect logic that connects the desired outcomes from the strategy with drivers that will lead to the strategic outcomes. ? [The strategy map] provides executives with a framework for describing and managing strategy in a knowledge economy. (Kaplan and Norton, 2001)
In this manner, the balanced scorecard offered a way to provide a central focus for the activities and the measurement of progress for STAR research, that is, on the achievement of contributions to mission goals.
6
Concluding discussion
The aim in the studies we discussed above was primarily to explore the development of measures of research progress as a complement to existing measures. We are also suggesting how to measure the radicalness of that progress against a standard. We argue that the measures we developed could play a key role in the management of innovation. And as recent studies have shown, a key need for research that is focused on radical innovation or marked by high uncertainty is better data and documentation of the research process (Leifer et al., 2001; Doctor et al., 2001; Karlsson et al., 2004).
Measuring radical innovation in real time
373
As we have argued, the use of multiple measures of performance for radical research might provide a more accurate and more timely portrayal of technical progress than do the use of milestones or other performance measures, such as papers and patents. For instance, using the example of the figure of merit for the tunable filter, the measure of progress offers the ability to assess progress towards the completion of the filter. If the measure was simply a stage-gate determination of the completion of the filter, the inability to complete the filter would typically mean the discontinuation of the project. But the figure of merit provides significantly more information about the progress of the research along various important parameters which would allow managers make a more informed determination about the progress of the project prior to a specified stage and potentially suggest areas where further research might be more fruitful. In this manner, this approach would add timely measures of technical progress to the existing milestones and other performance measures, and it would seek to integrate these measures both into scientific knowledge, such as better understanding the causes of ozone depletion, and to the organisational/programmatic goal of reducing delays in forecasting various conditions, as in the case of NOAA's NWS. Of course, thesemeasures are complementary of existing measures, but the use of additional measures of the process of scientific and technological research allows for greater and more timely recognition of R&D accomplishments. For instance, with regard to publications, advances along one or more of these dimensions are typically not publishable depending upon the specific variable involved. Some variables are inherently more publishable than others, and it is perhaps the case that some variables in weather forecasting or model building are more likely to be favoured over others. Thus, these new measures of technical progress allow one to compare across research teams working in quite disparate areas so that the work of all is equally appreciated. To restate our basic argument, the use of real-time measures of progress and the assessment of the radicalness of that progress could provide critically needed information for the management of research. In particular, real-time measures would provide more timely information to determine the critical pathways of progress both within and across research projects, as well as larger research units. Further, the real-time measures that we discuss would more centrally locate the focus of performance management of research on the advance of knowledge and innovation, rather than bureaucratic processes. In the world of publicly funded or directed research, such measures could become an essential component for meeting GPRA and PART requirements while still directly valuing the research. As Feller (2002) points out, GPRA measures are expected to be objective and quantifiable, there are some allowances for agencies to develop alternative forms of performance measurements and reporting. In summary, the development of new metrics for R&D is an important step to successfully managing and generating innovation within research organisations. The challenge is developing measurements which allow managers to make informed and appropriate decisions about research and development, but do not pose unnecessary and distracting opportunity costs and obstruct the development of breakthroughs which, in fact, may alter the course of the research and knowledge.
374
J. Mote, G. Jordan and J. Hage
Acknowledgements
A previous version of this paper was presented at the February 2005 meeting of the American Association for the Advancement of Science (AAAS). The authors gratefully acknowledge the support of the US Department of Energy Office of Science and the Sandia National Laboratories Science, Technology and Engineering Foundations Strategic Management Unit and the National Oceanic and Atmospheric Administration (NOAA). This research has been performed under Sandia National Laboratories DOE contract DE-AC04-94AL85000. Sandia is operated by Sandia Corporation, a subsidiary of Lockheed Martin Corporation. Additional research was conducted under contract with NOAA. The opinions expressed are those of the authors, not the US Department of Energy or Sandia National Laboratories or NOAA.
References
Averch, H.A. (1994) 'Economic approaches to the evaluation of research', Evaluation Review, Vol. 18, pp.77-88. Blankenship, L.V. (1974) 'Management, politics, and science: a nonseparable system', Research Policy, Vol. 3, No. 3, pp.244-257. Cao, C., Weinreb, M. and Xu, H. (2004) 'Predicting simultaneous nadir overpasses among polar-orbiting meteorological satellites for the intersatellite calibration of radiometers', Journal of Atmospheric and Oceanic Technology, Vol. 21, pp.537-542. Colton, M., Powell, A., Jordan, G., Hage, J. and Mote, J. (2004) 'Minding the gaps: new insights into R&D management and operational transitions of NOAA satellite products', Proceedings of the International Society for Optical Engineering (SPIE), Vol. 5548, October, pp.1-10. Coombs, R., McMeekin, A. and Pybus, R. (1998) 'Toward the development of benchmarking tools for R&D project management', R&D Management, Vol. 28, No. 3, pp.175-186. Cooper, R.G. and Kleinschmidt, E.J. (1995) 'Benchmarking the firm's critical success factors in new product development', Journal of Product Innovation Management, Vol. 12, pp.374-391. Cozzens, S.E., Popper, S., Bonomo, J., Koizumi, K. and Flanagan, A. (1994) Methods for Evaluating Fundamental Science, DRU-875/2-CTI, RAND. Department of Commerce (2003) Strategic Plan, FY 2004-FY 2009. Deuten, J. and Rip, A. (2000) 'Narrative infrastructure in product creation processes', Organization, Vol. 7, No. 1, pp.69-94. Doctor, R.N., Newton, D.P. and Pearson, A. (2001) 'Managing uncertainty in research and development', Technovation, Vol. 21, pp.79-90. Feller, I. (2002) 'Performance measurement redux', American Journal of Evaluation, Vol. 23, No. 4, pp.435-452. Frenkel, A., Grupp, H. and Maital, S. (2000) 'Measuring dynamic technical change: a technometric approach', International Journal of Technology Management, Vol. 20, Nos. 3-4, pp.429-441. Garcia, R. and Calantone, R. (2002) 'A critical look at technological innovation typology and innovativeness terminology: a literature review', Journal of Product Innovation Management, Vol. 19, No. 2, pp.10-32. Geisler, E. (2000) The Metrics of Science and Technology, Westport, CT: Quorum Books. Green, S.G., Gavin, M.B. and Aiman-Smith, L. (1995) 'Assessing a multidimensional measure of radical technological innovation', IEEE Transactions on Engineering Management, Vol. 42, No. 3, pp.203-214. Grupp, H. (1994) 'The measurement of technical performance innovations by technometrics and its impact on establish technology indicators', Research Policy, Vol. 23, No. 2, pp.175-193.
Measuring radical innovation in real time
375
Grupp, H. and Hohmeyer, O. (1986) 'A technometric model for the assessment of technological standards and their application to selected technology-intensive products', Technological Forecasting and Social Change, Vol. 30, No. 2, pp.123-137. Jensen, K. and Warren, P. (2001) 'The use of options theory to value research in the service sector', R&D Management, Vol. 31, No. 2, pp.191-201. Jordan, G.B., Hage, J., Mote, J. and Hepler, B. (2005) 'Investigating differences among research projects and implications for managers', R&D Management, Vol. 35, No. 5, pp.501-511. Kaplan, R. and Norton, D. (2001) The Strategy-focused Organization: How Balanced Scorecard Companies Thrive in the New Business Environment, p.69. Karlsson, M., Trygg, L. and Elfstrom, B.O. (2004) 'Measuring R&D productivity: complementing the picture by focusing on research activities', Technovation, Vol. 24, pp.179-186. Lenz, R.C. (1985) 'A heuristic approach to technology measurement', Technological Forecasting and Social Change, Vol. 27, pp.249-264. Leifer, R., McDermott, C., O'Connor, G., Peters, L., Rice, M. and Veryzer, R. (2000) Radical Innovation: How Mature Companies can Outsmart Upstarts, Cambridge, MA: Harvard Business School Press. Leifer, R., O'Connor, G.C. and Rice, M. (2001) 'Implementing radical innovation in mature firms: the role of hubs', Academy of Management Executive, Vol. 15, No. 3, pp.102-113. McDermott, C. and O'Connor, G.C. (2002) 'Managing radical innovation: an overview of emergent strategy issues', Journal of Product Innovation Management, Vol. 19, No. 6, pp.424439. Narin, F., Carpenter, M. and Woolf, P. (1984) 'Technological performance assessment based on patents and patent citations', IEEE Transactions on Engineering Management, Vol. 31, No. 4, pp.172-183. Pavitt, K. (1991) 'What makes basic research economically useful?', Research Policy, Vol. 20, No. 2, pp.109-119. Pavitt, K., Robson, M. and Townsend, J. (1989) 'Technological accumulation, diversification and organisation in UK companies, 1945-1983', Management Science, Vol. 35, No. 1, pp.81-99. Raz, T., Shenhar, A.J. and Dvir, D. (2002) 'Risk management, project success, and technological uncertainty', R&D Management, Vol. 32, No. 2, pp.101-109. Rinia, E.J., van Leeuwen, T.N., van Vuren, H.G. and van Raan, A.F.J. (2001) 'Influence of interdisciplinarity on peer-review and bibliometric evaluations in physics research', Research Policy, Vol. 30, No. 3, pp.357-361. Roussel, P.A., Saad, K.M. and Erickson, T.J. (1991) 3rd Generation R&D, Boston, MA: Harvard Business School Press. Schainblatt, A.H. (1982) 'How companies measure the productivity of engineers and scientists', Research Management, Vol. 25, No. 3, pp.10-18. Schumann, P.A., Jr., Ransley, D.L. and Prestwood, C.L. (1995) 'Measuring R&D performance', Research-Technology Management, Vol. 38, No. 3, pp.45-55. Trucano, T., Easterling, R., Dowding, K., Paez, T., Urbina, A., Romero, V., Rutherford, B. and Hills, R. (2001) 'Description of the Sandia validation metrics project', Sandia Report, SAND2001-1339, Sandia National Laboratories. Zoltan, J.A., Anselin, L. and Varga, A. (2002) 'Patents and innovation counts as measures of regional production of new knowledge', Research Policy, Vol. 31, No. 7, pp.1069-1085.
376
J. Mote, G. Jordan and J. Hage
Bibliography
Baker, N., Green, S.G. and Bean, A.S. (1986) 'Why R&D projects succeed or fail', Research Management, November-December, Vol. 24, No. 6. Balachandra, R. and Friar, J.H. (1997) 'Factors for success in R&D projects and new product innovation: a contextual framework', IEEE Transactions on Engineering Management, Vol. 44, No. 3, pp.276-287. Brown, S. and Eisenhardt, K. (1995) 'Product development: past research, present findings, and future directions', Academy of Management Review, Vol. 20, No. 2, pp.343-378. Burns, T. and Stalker, G. (1961) The Management of Innovation, London: Tavistock Publications. Callon, M. (1980) 'The state and technical innovation: a case study of the electrical vehicle in France', Research Policy, Vol. 9, pp.358-376. Chandler, A. (1962) Strategy and Structures, Cambridge, MA: MIT Press. Chiesa, V. (2001) R&D Strategy and Organisation: Managing Technical Change in Dynamic Contexts, River Edge, NJ: Imperial College Press. Cooper, R. and Kleinschmidt, E. (1987) 'New products: what separates winners from losers', Journal of Product Innovation Management, Vol. 4, No. 3, pp.169-184. Damanpour, F. (1991) 'Organizational innovation: a meta-analysis of effects, determinants and moderators', Academy of Management Journal, Vol. 34, pp.555-590. Davis, J., Fusfeld, A., Scriven, E. and Tritle, G. (2001) 'Determining a project's probability of success', Research Technology Management, Vol. 44, No. 3, pp.51-57. Freeman, M. and Beale, P. (1992) 'Measuring project success', Project Management Journal, Vol. 23, No. 1, pp.8-17. Griffin, A. and Page, A. (1996) 'PDMA success measurement project: recommended measures for product development success and failure', Journal of Product Innovation Management, Vol. 13, No. 6, pp.478-496. Henderson, R.M. and Clark, K.B. (1990) 'Architectural innovation: there configuration of existing product technologies and the failure of established firms', Administrative Science Quarterly, Vol. 35, No. 1, pp.9-30. Knight, F. (1921) Risk, Uncertainty and Profit, Boston: Houghton Mifflin. Kumar, V., Persaud, A. and Kumar, U. (1996) 'To terminate or not an ongoing R&D project: a managerial dilemma', IEEE Transactions on Engineering Management, Vol. 43, No. 3, pp.273284. Lewis, M. (2001) 'Success, failure and organizational competence: a case study of the new product development process', Journal of Engineering and Technology Management, Vol. 18, No. 2, pp.185-206. Lipovetsky, S., Tishler, A., Dvir, D. and Shenhar, A. (1997) 'The relative importance of project success dimensions', R&D Management, Vol. 27, No. 2, pp.97-106. Paolini, A., Jr. and Glaser, M.A. (1997) 'Product selection methods to pick winners', Research Management, Vol. 20, pp.26-29. Pavitt, K. (1990) 'What we know about the strategic management of technology', California Management Review, Spring, pp.17-26. Pinto, J.K. and Slevin, D.P. (1988) 'Project success: definitions and measurement techniques', Project Management Journal, Vol. 19, No. 3, pp.67-73. Raz, T., Shenhar, A. and Dvir, D. (2002) 'Risk management, project success, and technological uncertainty', R&D Management, Malden, MA: Blackwell Publishers. Rothwell, R. (1985) 'Project SAPPHO: comparative study of success and failure in industrial innovation', Information Age, Vol. 7, No. 4, pp.215-220. Schumann, P.A., Jr. and Ransley, D.L. (1995) 'Measuring R&D performance', Research Technology Management, Vol. 38, No. 3, pp.45-55.
Measuring radical innovation in real time
377
Shenhar, A. (2001) 'One size does not fit all projects: exploring classical contingency domains', Management Science, Vol. 47, No. 3, pp.394-414. Shenhar, A. and Dvir, D. (1996) 'Towards a typological theory of project management', Research Policy, Vol. 25, pp.607-632. Wörner, S.D. and Grupp, H. (2003) 'The derivation of R&D return indicators within a real options framework', R&D Management, Vol. 33, No. 3, pp.313-325.
Notes
1 For instance, the goal of one of the projects we studied was to eliminate the need for reagents in detecting arsenic in water. The current state of art in arsenic detection requires two reagents, so the movement from two to zero limits the efficacy of this measurement. Government Performances Results Act of 1993 (GPRA) and Program Assessment Rating Tool (PART). Moderate resolution imaging spectroradiometer.
2 3
doc_402586477.docx
In commutative ring theory, a branch of mathematics, the radical of an ideal I is an ideal such that an element x is in the radical if some power of x is in I. A radical ideal (or semiprime ideal) is an ideal that is its own radical (this can be phrased as being a fixed point of an operation on ideals called 'radicalization'). The radical of a primary ideal is prime.
Research for Measuring Radical Innovation in Real Time
Abstract: As Leifer et al. have argued, the management of radical innovation projects necessitates different strategies and practices, including different ways of measuring performance. This paper discusses our efforts to explore indicators of progress for radical innovation in basic and applied research and technology development at both the project and programme levels. The paper provides examples of our work with scientific research projects in two research organisations to develop real-time indicators of the research progress. Finally, it is argued that the continued development of similar indicators can play an important role in managing research to achieve radical innovation. Keywords: innovation; basic research; R&D; performance; measurement. Reference to this paper should be made as follows: Mote, J., Jordan, G. and Hage, J. (2007) 'Measuring radical innovation in real time', Int. J. Technology, Policy and Management, Vol. 7, No. 4, pp.355-377. Biographical notes: Jonathon E. Mote is an Assistant Research Science at the University of Maryland in the Center for Innovation. Gretchen B. Jordan is a Principal Member of the Technical Staff at Sandia National Laboratories, New Mexico. Jerald T. Hage is Professor Emeritus and Director of the Center of Innovation at the University of Maryland.
1
Introduction
While radical innovation is widely recognised as an important goal, how does one actually manage and measure progress towards that goal? Everybody wants to increase radical innovation but nobody knows how to do it. Numerous recent studies have emphasised that the pursuit of radical innovation is something that must be managed differently from other research pursuits. Most prominently, Leifer et al. (2000) have argued that the management of radical innovation projects necessitates different strategies and practices, including different ways of measuring performance, than those directed at more incremental innovations. While the measurement of research performance has been an important issue at least since the emergence of 'Big Science' (Blankenship, 1974), it is clear that radical science needs to find more appropriate measures of scientific and technical progress. Given the heightened uncertainty of radical research, not only in terms of outcomes but also the scope of unknowns in the research process, the ability to manage this type of research effectively is hampered by the lack of adequate measures to assess ongoing progress and link progress to achieving higher-level goals. This kind of information provides both focus and inputs to decisions on the level of uncertainty and on research direction and continuation. In this regard, the ability to measure and assess progress in real-time could have potentially wide ranging consequences for improvement in the management of radical innovation. In this paper, we will be discussing just one aspect of performance assessment of research directed towards radical innovation, that is, measures of progress to assess progress towards stated goals. This narrower focus is akin to 'in-progress peer review' for example, and does not address project selection or assessment of societal outcomes. Further, we will not be discussing measures of research outcomes or processes. Rather, we will only be addressing measures of progress that eventually impact on outcomes or transitions to subsequent research. Our scope is further narrowed in that we focus solely on scientific and technological research at the basic, applied and early product development stages. Most importantly, the measures we discuss are in-progress (realtime), not prospective or retrospective. We argue that these real-time measures could be most helpful for managing radical innovation, particularly if they meet three criteria: continuous variables specific to an organisational performance goal; measured within a planning cycle without a time lag; and describe the extent of progress. While we argue that real-time measures would be useful for individual research projects, it is important to recognise that research projects do not exist in isolation. In the context of R&D organisation, particularly a large organisation with extensive portfolios of research projects, it is critical to develop measures that are commensurate across a range of projects. Hence, such measures should have the ability to aggregate to higher organisational levels and goals and allow for comparison across projects. In demonstrating the efficacy of real-time measures of radical innovation, we draw on the efforts of two studies that aimed to develop the types of measures and indicators of research performance discussed above, and, we would argue, these types of measures could play a key role in the measurement and management of radical innovation. The first study was supported by the US Department of Energy (DOE) and focused on a large multi-disciplinary national research laboratory comprised of a broad range of research units, centres and programmes. The second study was supported by the National Oceanic and Atmospheric Administration (NOAA) and sought not only to develop measures of
Measuring radical innovation in real time
357
research performance, but to incorporate these measures into a scheme that allowed for these measures to be applied up, down and across. In both projects, our approach to developing measures was largely inductive in nature, as we worked with scientists, engineers and technicians to define measures that made sense from the standpoint of the researchers. Overall, we achieved enough progress to assess the applicability of these measures to radical innovation and to indicate how we plan to improve upon what we have accomplished so far. While we suggest that the measures we developed are useful to all types of research, we argue that they could be especially useful in the measurement of radical innovation due to the higher levels of risk and uncertainty associated with this type of research. After a discussion of current measures of research performance and the need for real-time measures, we discuss the types of measures that are most appropriate for basic and applied research. Next, we provide a largely descriptive account of the two studies and the progress measures we developed. The paper concludes with a discussion of the results and implications for further research on the measurement of progress of radical innovation in basic and applied research.
2
The need for real-time measures of research progress
Why is measuring real-time progress important? We would argue that real-time measures are necessary for two primary reasons: accountability and programme improvement. As we discuss below, both of these items, which are necessary for the strategic management of research, are affected by the timeliness, or lack, of existing metrics. But it is important to state that we are not arguing for metrics that displace current metrics. Rather, we argue for complementary metrics that allow for appropriate measurement of the progress and advance of research. With regard to accountability and control, it is critical to demonstrate to investors and funders that research funds have been spent well. From the point of view of researchers, the continued flow of funds is more likely if this need for accountability is better met. But it is not sufficient to know that research is relevant, that the 'right research' is being conducted, or that past research was successful. As Feller (2002, p.444) states with regard to federally funded research, "perhaps the most telling limitation of performance measurement as applied to science policy is that whatever its value may be in tracking past performance, it is of limited value for prospective decisions". Some examples of existing measures that focus on outcomes would include: economic outcomes, like the degree of commercial success (Averch, 1994; Pavitt, 1991) or cost-benefit analysis (Jensen and Warren, 2001); bibliometric outcomes, such as publications and citations (Cozzens et al., 1994; Rinia et al., 2001); or technical outcomes, such as patents (Grupp, 1994; Narin et al., 1984; Zoltan et al., 2002) and awards and honours (Schainblatt, 1982). In general, these measures of research performance do not address the progress of the research and are lagging indicators. While lagging indicators can be useful in confirming patterns of past performance, the length of time between the actual research performance and traditional research metrics restricts their usefulness in measuring current research performance and addressing the need for control and accountability. Similarly, Geisler (2000) argues that the existing set of metrics suffers, in part because they miss the temporal dimension between scientific outputs and technology accomplishments.
358
J. Mote, G. Jordan and J. Hage
The issue of programme improvement speaks to the heart of R&D management and the need to improve research performance and take strategic action. In this regard, it is important to recognise the maxim that 'what gets measured, gets done'. Hence, it is essential to institute measures that stimulate performance towards strategic goals rather than hinder performance and result in goal displacement. For example, putting process indicators in place, such as quarterly reports, may be in conflict with the activities needed to produce the very value that comes from research. In this manner, the advance of science becomes secondary to meeting and completing arbitrary benchmarks that satisfy a bureaucratic need. In other words, the short-term criteria for control are often in conflict with the long-term criteria for evaluating value and strategy realisation. But in addition to stimulating progress and providing focus, it is essential that managers have relevant metrics that allow for making appropriate decisions regarding whether to continue or discontinue the research, maintain or change direction, or add resources or not in order to achieve desired goals. Existing methods of gathering information about research for project management, include benchmark or stage-gate methods (Coombs et al., 1998; Cooper and Kleinschmidt, 1995) and process management (Schumann et al., 1995). Arguably the most influential of these methods, the stage gate process, assesses progress at a series of 'gates' between idea generation and development and commercial launch. The widespread acceptance of the stage gate method is indicative of a need and desire to collect better information on current progress and provide guidance on project definition including scope and outputs. However, we would argue that such methods, particularly for the fuzzy front end that we are addressing, suffers from the same problem as other common measures. Progress is mostly defined in measures that are concrete, such as publications or the completion of an experiment or prototype, or in terms specific to a particular programme rather than general dimensions of performance that can be compared across similar projects or programmes. In summary, the existing types of progress measures, while useful for measuring various aspects of the research process, lack the ability to determine the progress or advance within the process of research itself. While the need for more timely measures ofprogress is a general one in research, it is especially critical for research aimed at radical innovation. It is to this topic we turn in the subsequent section.
3
The need for real-time measures of radical innovation in research
Before turning to a discussion of measures, it is important to ask a more basic question: What is the end goal, that is, what is radical innovation? Definitions differ, often times greatly. Typically, radical innovation is conceptualised as dramatic increases in functionality, performance or dramatic decreases in costs and time. But as McDermott and O'Connor (2002) pointed out with regard to decades of research on radical innovation, 'Researchers are far from a consensus regarding a formal definition of radical innovation'. We will not pursue a lengthy review of efforts to characterise radical innovation in this section, as this has been done more ably and exhaustively by others (see for example Garcia and Calantone, 2002; McDermott and O'Connor, 2002; Leifer et al., 2000).
Measuring radical innovation in real time
359
While definitions may differ, it is nonetheless clear that radical research, by which we mean basic and applied research that aims for radical breakthroughs, has several distinctive characteristics. First and foremost, radical innovation typically involves higher levels of uncertainty. A great amount of uncertainty arises from the fact that radical research involves a greater number of unknowns in the research, what Roussel et al. (1991) called technical novelty and Raz et al. (2002) called technological uncertainty. As Deuten and Rip (2000) discuss, the open-ended and non-routine nature of the research process itself makes innovation hard to predict. Because of this, the goals and outcomes of radical research can be uncertain, if not indefinable. Further, radical research must be flexible and opportunistic in order to capitalise on new bits of knowledge and insight, with the recognition that radical research produces intangible outcomes (learning) as well tangible outcomes (widgets). Finally, because radical research is not a straightforward endeavour, progress can be variable. Indeed, radical innovation often comes in the form of a step function, with progress coming in great leaps forward. Given the inchoate nature of radical research, it is understandable that the measurement of progress in radical research is also somewhat indeterminate (Green et al., 1995). Despite the challenges of measuring radical research, it is important to build on what is known and has been already done. Specifically, we can look to what has been done with technology, where radical is often characterised as a new feature or function. Typically, the radicalness of the technological innovation is characterised as some measurable increase in performance or decrease in cost compared to a fixed reference or baseline, such as the current state of art in the field. For example, Leifer et al. (2001, p.103) provide a good rule of thumb: "at least a five-fold improvement in known performance features, and a significant (30% or greater) reduction in cost". When such measures of degrees of radicalness are feasible, it is also generally possible to assess risk and the probability that this technical performance or cost is achieved is increased (Raz et al., 2002). As we discuss in greater detail below, these can also be applied to radical research with slight modifications. This most clearly applies to scientific advances that involve a tool or a technique, such as the speed and cost of genome sequencing, or increases in understanding, such as the properties of phenomena and materials. But in research, there is also a distinct need to apply measurement to 'learning', particularly in terms of hypotheses pursued (or eliminated) and theory building. Because we are talking about radical research, the fuzziest of the fuzzy front end, it is helpful that the measurement of radical research also relates to higher goals, within the research organisation, specifically, and the S&T community more generally. Two examples of organisational goals drawn from the public sector include increasing the lead time and accuracy of winter storm warnings (Department of Commerce, 2003, p.63) and an increase in the measurement of properties and interaction the top quark (Department of Energy, Office of Science). In this manner, direct linkages of radical research to such higher goals provides the same sort of direction and discipline that the market lends to new product development. But given the high degree of uncertainty of radical research, it is critical that performance metrics provide timely information on research progress in order to determine whether research is making sufficient progress towards stated goals, or whether new and unpredicted outcomes have occurred. In this manner, such metrics
360
J. Mote, G. Jordan and J. Hage
positively contribute to a key factor identified by Pavitt et al. (1989) for the successful management of innovation, that is, the flexibility and speed of the decision making. Similarly, Leifer et al. (2000) point out:
"The project manager's job in the radical innovation context is less one of listing tasks, delegating them, and controlling progress, and more one of monitoring where project stands on each uncertainty dimension and making decisions about which uncertainties to attack at any point in time."
Hence, measurement that can capture progress in real time that has these characteristics would provide very useful information to managers.
4
Assessing research and potential measures
4.1 Assessing the radicalness of the research
Before we turn to the different types of research to explore potential measures, it is useful to first address more specifically the challenge of determining whether an innovation is radical or not and the degree of that radicalness. On this topic, we focus on two primary metrics: the degree of change and the scope of research. We suggest that these are the standards by which to assess the degree of radicalness for whatever progress metric is defined. Thus they are essential to keep in mind when making prospective or retrospective determinations about radical innovations. As Garcia and Calantone (2002) suggest for new product development, it is more appropriate to assess innovation along a continuum, with varying degrees of innovativeness. Hence, the degree of change introduced by the innovation is an important element of radicalness. Unlike new product development, however, a baseline or state of the art for basic and applied research and early engineering and development can be difficult to determine quickly or easily or even conclusively. For instance, while it is relatively easy to determine baseline measures for piezo-resistance pressure sensors by assessing what is currently commercially available (Frenkel et al., 2000), it is altogether more difficult to determine a baseline for something like rarified gas dynamics, where research on the topic is being conducted on various aspects of the phenomena in laboratories around the world. Hence, it is even more essential to determine the extent of radicalness in these instances. Because radical innovation is often marked by steep advances, the degree of change can often be readily identified. But in those instances when it is less clear, particularly to non-technical managers, a key tool is the use of expert judgement, or peer review. In our experience, this is seldom done routinely, as when a major breakthrough is reported before it has been confirmed by other experts. For instance, the announcement that University of Utah researchers had discovered a method for cold fusion in 1989, or, more recently, the work on stem cells by South Korean researcher Hwang Woo-suk. Another important consideration is the scope of the research. Although we did not directly address the scope of research in our studies, it is an important aspect of our work going forward. At its most basic, the scope of research involves the number of variables or processes that are being researched at the same time. Some disciplines have a systemic quality, that is, a large number of variables have to be considered at the same time. Further, not all scientific problems can be approached with small research teams; some of them require a large scale focus. Again, consider the example of the Human Genome
Measuring radical innovation in real time
361
Project. While progress in mapping and sequencing the human genome could have proceeded with small research teams, it took a large scale programme to coordinate a range of efforts so that the time needed to complete the entire genome was appreciably lessened. Indeed, the systemic quality of some types of research is frequently overlooked as a critical dimension to the scientific problem. One example of what might be called a systemic problem is research on the weather, which encompasses both oceanic and atmospheric systems. NOAA was created in 1970 to unify and coordinate the government's research efforts on various aspects of the global environmental system, including the National Weather Service (NWS). Altogether, the scope of NOAA's mandate necessitates quite expensive and specialised equipment and teams to collect the relevant data, including satellites, ships, buoys, and planes. Indeed, one could argue that the choice to conduct large scale data collection represents one form of a broad scope of focus, such as global climate studies or the use of hyperspectral data in weather satellites. Two comments that might provide some additional explanation on this topic. First, the scope of the focus is not determined by how fundamental the research is in our theory. For example, Einstein's theory of special relativity is quite fundamental but consists of only three variables (radical because of its centrality to field and because of the broad scope of its impact). Second, as everyone is familiar, there are multiple levels within each discipline and therefore there can be multiple systems. In this respect, a broad scope of focus means that one is studying multiple levels or multiple systems.
4.2 Assessing different aspects of research progress
Some concrete measures of progress are suggested by the differences that one finds in the research process itself. In this vein, we would differentiate between technology research, focused more on products, and scientific research, focused more on discoveries. Within each of these there are differences that suggest different types of measures of progress. We should point out that we are primarily concerned with research at the project level, with the recognition that a critical challenge in utilising any measurements is aggregation across various levels of organisational structure.
4.2.1 Measures for technology research
In this category, research is typically focused on the earliest stages of development of a new product. Hence, the most important measure is the increase in functionality of the product relative to its predecessor or competition, which allows for straightforward measures of performance characteristics. In addition, such measures can be aggregated across products which share similar functional characteristics. For example, such measures might include:
• • • • •
miniaturisation or size reduction efficiency (in operating costs and training) speed of use or employment precision of use flexibility (frequently ignored), i.e., how easy is it to change from one use to the next.
362
J. Mote, G. Jordan and J. Hage
It is clear that measures that assess the extent of improvements in various performances lend themselves quite easily to the assessment of progress and innovation. But measures of progress towards the achievement of such improvements are complicated by how functionality is achieved. In some instances, the functionality simply reflects the replacement of one component with another, for example, the replacement of analog components with digital components in radar systems. But functionality might involve the synthesis of components, such as the combination of software and hardware components in the development of semi-autonomous modular robotic control technologies. Technology research can also be involved in improving the process of manufacturing products as well. Process measures would include the extent of the change in the machines used to produce the product. Similar to the measures suggested above, such improvements would be typically measured indirectly by the improvement in the following kinds of performance:
• • • •
increases in productivity increases in reliability increases in the flexibility of manufacturing (ability to move from one kind of product to the next on the same production line) reductions in the use of various resources (e.g., energy).
In general, the measures for technology research are closely akin to the types of measures that one typically finds in the new product development literature. But it is important to keep in mind that the measures we discussed are not directed towards assessing outcomes, but towards assessing progress towards outcomes.
4.2.2 Measures for scientific research
In scientific research, we would differentiate among three aspects of research, analytical, descriptive and control, and suggest measures for each. The first is analytical, that is, the explanation or characterisation of a particular phenomenon. As would be expected, this is an important focus of much scientific research. The characterisation of phenomena typically proceeds through repeated trials of experimentation and measurements, with the goal being the development of an accurate model of the phenomena and a more complete understanding of the phenomena's properties. Put another way, the goal of characterisation is to reduce the uncertainty of knowledge about a particular phenomena. Interestingly, we note that a great deal of analytical research now relies on the use of simulation as a tool in characterisation, which offers great potential for the use of validation metrics in measuring progress, such as precision of prediction and reduction of uncertainty. In addition, it is possible to include the complexity of the model as a measure, such as the number of unknowns or processes accounted for in the model. The key here is the percent improvement in the precision of prediction and the percent improvement in the processes included in the model. Another aspect of scientific research is largely descriptive, that is, the portrayals of new patterns. The emphasis is on discovery and the initial identification of patterns. For instance, NOAA researchers have recently designated a new type of hurricane, one that is symmetrical in formation and maintains its intensity much longer across
Measuring radical innovation in real time
363
ocean waters than current models predict. In this aspect of research, some suggested measures would be how many new patterns are discovered and how many concepts and hypotheses are developed to explain the pattern. In other words, the progress of research is measured in the movement towards explanation. The extent of radicalness in the description might be the degree of movement from a concept to a hypothesis to a theory to a paradigm. Intermediate between description and explanation of patterns is when it is possible to exert control to obtain some result. The extent of control in the properties of the phenomena reflects a kind of knowledge even if it is not clearly identified. We would argue that this constitutes an important element of much nanotechnology research where they are able to produce materials with fewer impurities and with certain improvements in performance characteristics such as transmission of electrical current. However, the use of the term control does not imply that there is a theory in place to explain the produced result. Apart from measuring progress of the research itself, another potential area of measurement is innovations in the research process. This is often an overlooked aspect of research and reflects the addition of a new instrument or technique for measurement, or improvements in existing ones, and the performance characteristics of that instrument or technique. The measure is the improvement in the measurement of the phenomena or function that is being measured, or the 'quality of the measurement'. For example, the ability to predict when the orbits of polar-orbiting weather satellites overlap could provide a significant improvement in the ability to calibrate all earth orbiting satellites as well as increasing the data quality for long-term climate studies (Cao et al., 2004). As we discuss in greater detail in the next section, the measures we developed for the projects in our studies closely follow the measures suggested above. As we hope to demonstrate, these measures are amenable to measuring progress of radical research as they allow for an assessment of the progress of research towards innovative outcomes. Once progress measures have been defined, however, it is not only objective quantitative measurement of research performance, but also objective, qualitative judgement of peers (Schumann et al., 1995), as to what constitutes a large or radical advance. In the next section, we present some of the measures we developed and discuss how they might be applied to radical innovation.
4.2.3 Measures for learning in research
Before we turn to the next section, however, it is important to address the notion of learning in research. During the process of developing research measures, it became clear that some of the projects we studied had not yet advanced to the stages of research where measurement of progress such as we described above was appropriate or applicable. In several of the projects this learning emerged in the face of new experimental results or difficulties with instruments. Often times, the learning resulted in some amount of reorienting or restructuring of the project. Clearly, however, learning is not simply a result of the 'fruitful error' in science, but is an integral component of all research. We would suggest that the learning process involves, but is not limited to, the identification of challenges or opportunities, learning something about these, and, in the process, closing in on some and perhaps identifying new challenges. In any event, this aspect of research is one that, while difficult to measure, is essential to the progress and advance of research.
364
J. Mote, G. Jordan and J. Hage
We discussed the desirability of attempting to measure learning and many of the researchers we interviewed, although not all, agreed that this might be potentially helpful. Although this type of progress is much harder to quantify, we feel it is important because it represents an important qualitative approach to the problem of technical progress. And this aspect is particularly important for research aimed at radical innovation given the high level of uncertainty and the potential for changes in direction or approach.
5
Our approach to developing measures of research progress
In this section, we report in much greater detail on the real-time progress measures that were developed in each research organisation and how they might be applied to radical research. Because of the unevenness in responses, measures of performance were not fully operationalised for all research projects. By fully operationalised, we mean that technical measures were agreed upon, data was collected on the measures, and progress on the measures was tracked retrospectively to the present. The two organisations we studied represented very different organisational settings. The first organisation was a large-scale, multi-disciplinary national laboratory. The organisation has a billion dollar annual budget and currently employs several thousand researchers. Research is conducted in a diverse range of fields, from physics and engineering to nanotechnology and biology. And the research work is organised in different ways, from small and large projects to long-term to short-term projects, and single discipline projects to multi-disciplinary projects. In addition, researchers typically work on several projects at the same time. The second organisation was a smaller-scale research unit housed within NOAA that focuses primarily on applied research on satellite data. Research is conducted in a narrower range of fields, primarily oceanography and meteorology, and the research work is organised around particular satellite systems, such as NOAA's family of geostationary weather satellites (GOES). Despite the differences in the organisations, data collection proceeded along similar lines at both sites. In our studies, we had two primary objectives for the measures of research performance. First, we were interested in real-time measures, that is, measures that would be useful to researchers and the management of the research work. The measures should be continuous and apply throughout the life of the project. We were motivated to address the limitations that we had identified with current metrics of research performance, primarily the temporal dimension of lagging indicators and to capture full value of the progress. Hence, we focused on developing measures within the process of research itself, with a particular focus on assessing of the health of the research environment and measures of 'outputs' or progress that occur prior to papers and provide more descriptive, rather than event-based, milestones. Following naturally from the first objective, our second objective was to focus on the scientific and technical aspects of the research. In this respect, we were interested in gaining insights from research personnel on their own interpretations of technical progress, and to conceptualise measures of technical progress that were meaningful to the researchers themselves. In short, we were interested in how researchers determined they were making scientific or technical progress or not. The development of metrics (units of measurement for the measures) for each of the projects was based on ongoing interviews with project principals and a thorough assessment of research materials. For each project, we sought to develop three measures
Measuring radical innovation in real time
365
of research performance that could be used to assess progress of the research throughout the life of the project. In many respects, our approach was similar to some approaches found in the technometrics and technological forecasting literature, such as Frenkel et al.'s (2000) analysis of industrial sensors, Grupp and Hohmeyer's (1986) work on technological standards, and Lenz's (1985) work on a heuristic approach to technology measurement. However, these approaches also differed from ours in a number of respects. First, this literature is primarily focused on the macro- or industrial level, rather than at the level of individual projects. Second, this literature tends to focus on research that is much closer to the market, such as new product development, rather than the fuzzy front end of basic and applied research. In many respects, our approach was similar to that of some of the US Department of Energy's Office of Science (DOE-SC) work on improving the performance measurement practices of Federal basic research organisations, particularly the DOE-SC's effort to develop metrics of research that satisfy the requirements of the PART. For example, Figure 1 illustrates the DOE-SC's approach to identifying a roadmap with specific metrics for research focused on advanced scientific computing. As the figure shows, the DOE-SC articulated a long-range goal at the level of scientific discovery and identified a key number of appropriate metrics to assess progress. However, our work differed in one important respect. Our work was focused on the level of the individual research project, not research portfolios or programmes. But as we were mindful of the importance of aggregating measures to higher levels of organisational structure, we sought to develop metrics at a level of generality that might allow for this possibility.
Figure 1 DOE-SC roadmap for scientific discovery needs for advanced
200
Deliver operating systems for scientific computers that incorporate fault tolerance (2007)
200
201
201
201
Complete programming model that enables scientists to use 100 000 processors (2009)
Deliver algorithm that scale to tends of thousands of processors key mathematical libraries (2007)
Deliver mathematics of complex systems that enables simulations of microbes (2013)
Deliver mathematics of complex systems that enables accurate linkage of multiple time and length scales (2011)
Demonstrate progress towards developing the mathematics, algorithms and software that enable effective, scientifically-critical models of complex systems (2015)
5.1 Sandia National Laboratories
We examined 22 research projects at this large, multidisplinary national laboratory. After an initial review of project documents we found that the projects could be sorted into two general categories, science and technology. The first set of eleven represents technological projects whose objective is to build an instrument, for example a sensor or mobile robot or catalytic converter. The second set of 11 represents science projects whose objective is to build the underlying knowledge about some specialty, e.g., the physics and chemistry of ceramics, plasma physics or the detection of proteins. In several cases, the science is designed to provide the basis for a new technology, for example the
366
J. Mote, G. Jordan and J. Hage
modelling membrane transport. Finally, in a few other projects, although the objective is a new technology, the project itself is more concerned with the science involved, e.g., the synthesis and characterisation of nano-scale energetic materials. The distinction between science and technology projects became important for three reasons. First, the measures that were developed were quite distinctive for each category. Most of the science projects were concerned with increased understanding of various phenomena, which can sometimes be characterised as the precision of experimental prediction, while many of the technology projects are involved with measures of reliability. Second, as it became apparent in our review of the results of an organisationwide survey, the science and the technology projects value aspects of their research environments quite differently (Jordan et al., 2005). Finally, the aggregation of measures was much simpler with the technology than with the science projects. In Table 1, we provide examples of measures of progress for technology research from our study. In addition to identifying specific measures for each project, we attempted to derive a set of measures that would allow for aggregation across projects. These aggregate measures were derived from a thorough review of project materials and identifying similarities and patterns. While the specific measures varied considerably, we were nonetheless able to aggregate the measures into several common categories, particularly along the dimension of science and technological projects.
Table 1 Examples of technology measures
Selected technical measures Aggregate Reliability Specific Overcoming dissipation processes Elimination of leakage, solvent resistance Reduction in high temperature corrosion Efficiency Improvement in harvesting isolate Ability to analyse multiple samples simultaneously Increase power output Precision Increase in dynamic frequency range to 10 GHz Increase in sensitivity in parts/million Increase in accuracy of micro injector Flexibility Band width selectability Performance scalability Number of toxins/agents detected Miniaturisation or reduction in size Decreases in mass size of particles (150 atoms) Decrease in receiver size/weight Decrease in size/weight of SARS module
Beyond the five categories in the table, the issues of ease of use and speed of use were also relevant in a handful of technological projects. Each of the 11 technical projects also involved a measure which can be termed functionality (and which could potentially contain multiple indices). Functionality represents progress relative to the basic goal of the project, such as the development of a hand-held device to detect pollutants without
Measuring radical innovation in real time
367
the use of chemical reagents. As we pointed out earlier, ideally, this progress and the degree of radicalness could be reported relative to the basic standards - the state of the art in the appropriate technological field. The most comprehensive technical measure developed was one where the project manager worked with a staff member to combine several of measures into a single figure of merit, including the initial measures we had agreed upon (see Figure 2). This is similar to the measure described by Lenz (1985) of a hierarchy of parameters whose values could be traded-off, but which culminate in one measure of performance. The approach taken by the project manager was to identify several critical variables for the development of a key element of the overall project, a tunable microwave filter. Each of these variables was then normalised and combined in such a way that progress on the filter could be tracked on a scale of 0 to 1, although the goal of the project was 0.3. The project manager then tracked the progress of the filter through a number of design iterations, as illustrated in Figure 2. In this manner, the measure becomes a useful single measure for showing the interactions among several variables, as well as allowing for useful comparisons across projects conducting research on filters using similar variables. On the opposite end of the spectrum were technical measures that were quantifiable, but were not fully illustrative of progress. Typically, this was due to a number of factors, such as limitations of the item chosen to be measured or the unit of measurement,1 or a small number of data points (both in terms of what was being measured and how long it had been measured). The limitations inherent in these technical measures meant we did not fully capture the progress of the projects within the scope of our study.
Figure 2
0.300 0.250 Actual measured FOM 0.200 FOM 0.150 0.100 0.050 0.000 10/1/2001 Ideal FOM for this filter
Filter figure of merit versus time
1/9/2002
4/19/2002 7/28/2002
11/5/2002 2/13/2003 5/24/2003 Date
9/1/2003 12/10/2003
In the 11 science projects, the specific measures varied, but there was remarkable similarity across a number of projects (see Table 2). In this respect, we identified two primary aggregate categories that are common in seven instances. These are the precision of the prediction and the complexity of the model, that is, the number of variables, forces, or processes involved. These categories partly arise from the increasing use of modelling and simulation in combination with experimentation, which allows for the use of validation metrics (Trucano et al., 2001). In general, the science projects typically focused on improving scientific models, and improvements in the predictive accuracy
368
J. Mote, G. Jordan and J. Hage
of the models represented advances in scientific understanding. However, the attempt to develop or utilise common categories of measures was more difficult in science projects that did not involve this approach. For instance, some science projects involved gaining a preliminary understanding of novel phenomena, such as a project that focused on the formation of plasma crystals. In general, this kind of research is largely descriptive as it moves towards the specification of models, and it is difficult to quantify performance on a real-time basis. But such progress can be indicated in terms of descriptive progress and/or learning.
Table 2 Categories Analytic Selected science measures Specific Increased precision of model of problematic horizontal permeability anisotropy Increased precision of model of transient strain on thermophoretic effects and soot formation/oxidation mechanisms Improved measurement of the effect of insufficient oxygen on the process of soot formation Development of stable atomistic simulation of lipid bilayer Development of model of friction at the mesoscopic scale Control Increase optimisation of novel MEMS friction structures Increase number of precursors for ceramic materials Increase ability to tailor properties of precursor-derived ceramic materials
Descriptive
In terms of measuring and managing research performance, the specific and aggregate measures developed at the national laboratory, as well as the reception of the measures by managers and researchers, indicated that this was a useful exploration. In this manner, we were pleased to demonstrate that specific and aggregate measures could be developed from the 'bottom up'. Finally, while above we have indicated how the specific measures can be generalised, and in turn these general measures can be aggregated, this does not solve all the many issues of a monitoring and evaluation system that functions at multiple levels. This was easier for the larger, broad scope projects because the project typically works to develop a set of progress and project goals to communicate and manage across levels of organisation. However, we recognise that in the context of the national laboratory, the practical application of the measures that we developed are limited. First, the 22 projects studied do not even begin to scratch the surface of overwhelming complexity of the laboratory's overall research portfolio. While aggregate measures were conceptualised for the projects we studied, a top down look would be needed to see how these measures could accurately and validly measure the heterogeneity that exists within relevant parts of the laboratory's work. Second, our study did not take into account the particular interests and concerns for measures at the upper levels of management, including national interests or GPRA and PART considerations.2 While this latter concern was not our charge in this study, it is of primary importance for developing relevant measures that can be utilised for performance measurement at various levels.
Measuring radical innovation in real time
369
5.2 NOAA's office of satellite applications and research
Building on our experience at the national laboratory, we subsequently embarked upon a similar exercise with NOAA's Office of Satellite Applications and Research (STAR). However, the aims of our effort at NOAA were much more ambitious (Colton et al., 2004). Specifically, our aim was not only to develop measures of research performance, but to develop a framework that would provide a monitoring and evaluation system for all of the research of about 150 researchers at STAR. We set out three primary objectives for our work: 1 2 3 respect the need for measures of technical progress at the research project level measure accomplishments at the project and organisational level establish linkages between project-level measures and the specific performance targets that have been established by NOAA as part of their planning process.
As with our work at the national laboratory, we set out to inductively derive measures of research performance as the first step in the development of a performance management system. The scope of the paper limits a complete discussion of our efforts, but below we provide examples of measures developed that attempt to meet the objectives above. STAR is housed within NOAA's National Environmental Satellite, Data and Information Service (NESDIS). NESDIS is an operating branch within NOAA that is charged with managing the operational environmental satellites and providing a series of weather-related data and information services. The primary function of STAR is two-fold: 1 2 researching, developing and using data products from new and existing weather satellites maintaining the quality and integrity of satellite-derived data.
STAR consists of three research and applications divisions that encompass satellite meteorology, oceanography, climatology, and cooperative research with academic institutions. The satellite-derived land, ice, ocean, and atmospheric environmental data products provide the basis for most, if not all, weather forecasting models of the NWS, another operating branch of NOAA. In addition to the working with current operational weather satellites, STAR plays a key consulting role with NASA in the design and development of new weather satellites and instruments.
5.2.1 Cross-project measures
As we began to interact with each of the eight selected projects, we realised that there was much more to the activities or tasks that these projects performed than what might be called original research. Although the work conducted by STAR is typically characterised as research, many of them in fact do not necessarily spend the majority of their time conducting research as it is normally defined. More typically, many of these projects focus on problems of quality control, particularly those activities identified as calibration and validation of satellite data. Since the work of STAR, and NOAA in general, relies on the quality of satellite data, this work is a critical component of the work of STAR. But it is also not the kind of research activity that traditional measures of performance would adequately capture.
370
J. Mote, G. Jordan and J. Hage
In general, seven of the eight we studied projects involved some amount of calibration and validation activities, which consists of four major activities: calibration and validation of satellite instruments, correction of existing algorithms, development of new algorithms, and the construction of time series. At one extreme are projects whose primary concern is with calibration/validation, such as the project centrally concerned with calibration (Cal/Val) and a project focused on the development of new datacompression algorithms. At the other extreme are projects whose primary concern is the improvement or development of new algorithms focused on specific weather phenomena, such as tropical storms and ocean winds. In general, a great deal of time is spent on continuous calibration and some on periodic validation and therefore much energy is devoted to the problem of maintaining the algorithms/time series or updating them in the light of new calibrations/validations. We began to define calibration/validation activities as 'quality control', and recognised a significant contrast between these activities and conducting sustained research that could result in the development of new methods of calibration and validation. Interestingly, these new methods have the potential to reduce the amount of time spent in quality control. Since the amount of time spent on quality control is such an important aspect of the work of many if not all of these projects, measuring the amount of time allocated to quality control was identified as a central performance measure common across most STAR projects. The measurement of quality control is significant because an existing performance measure at STAR is the number of new data products developed. However, the development of new products necessitate a considerable amount of research time and effort in developing new algorithms before the products become operational. In this manner, the recognition and measurement of quality control activities could have a significant impact on the amount of time allocated to more research-oriented activities. In addition, measures for each of the calibration/validation activities were derived and as listed in Table 3.
Table 3 Performance measures for calibration/validation Algorithms Increase in duration of algorithm Decrease in size of errors in data Increase in coverage data Time series Increase in duration of time series Reduction in inter-annual variation in data Increase in coverage of data
Calibration/Validation Reduction in frequency of calibration/validation Reduction in size of errors Increase in coverage of data
While the work of calibration and validation represents largely routine work and incremental innovation, researchers had developed a novel method for accurately predicting simultaneous nadir overpasses among different polar-orbiting meteorological satellites (Cao et al., 2004). As mentioned above, this represents a potential radical innovation, as it has the potential of significantly increasing the calibration consistency and traceability required for long-term climate studies by reducing the uncertainties in observations among weather satellites. The application of the performance measures developed (above) in the testing of this method can help managers determine whether this represents a radical innovation, and whether the costs associated with its implementation are justifiable.
Measuring radical innovation in real time
371
5.2.2 Project measures that link to science and mission goals
The exigencies of GPRA and PART have placed greater emphasis on evidence of impacts on mission goals, as well as on reducing costs and thus productivity. Some of the measures that were developed for STAR (Figure 3) address both forms of impacts and includes what we viewed as an important corollary to quality, that is, product development time. Of course, this measure is well-known in the industrial and new product development literature. STAR had recognised its importance by establishing a task force to look into ways of improving it, but had not actually defined a measure or measured it. Given STAR's location with a complex inter-organisational network, however, these measures of impact depend a great deal on communication and coordination. But these measures, along with shorter product development time, lies at the heart of STAR's research efforts. We would suggest that the use of such measures, while recognising the importance of inter-organisational partners and the need to innovate, allow STAR simultaneously to meet reporting requirements and better align research efforts mission goals.
Figure 3 Impact on weather prediction models
• • •
Improved prediction (accuracy, precision, coverage) Development of improved models (either more less variables) Reduction in product development time
To better illustrate the application of these measures, we will provide an example from a project that is focused on the development of data products on wind. In addition to maintaining current operational products, the wind project has been working on developing a new data product on winds in polar regions. The data is gathered from MODIS3 instruments aboard the Terra and Aqua satellites that circle the earth along a north-south flightplan. The Terra and Aqua satellites were launched in 1999 and 2002, respectively, as part of NASA's Earth Observing System, and both remain under NASA, not NESDIS, management. The advantage of developing products from these instruments and satellites is that they provide coverage in the polar regions of the globe, areas where wind data is lacking. Development of the algorithms to derive the wind observations was conducted in 2001 and 2002. Routine production of wind observations from the MODIS instruments, although still experimental, was established at NOAA/NESDIS in July 2003. The next step in the development of any satellite data product is the assimilation into weather prediction models and conducting impact studies, in terms of improved prediction, on forecasts. While STAR typically provides such data to other research centres around the world for impact studies, measuring the linkage to NOAA's mission goals is dependent on the data being assimilated into the prediction models of the National Center for Environmental Prediction (NCEP), housed within the National Weather Service. In essence, NCEP is the primary conduit for all meteorological data and provides the official NOAA forecasts of weather and environmental information. Hence, the development of a new product at STAR and, more importantly the determination of its radicalness and its linkage to NOAA's mission goals, depends on the assimilation and utilisation of data by NCEP.
372
J. Mote, G. Jordan and J. Hage
Another example of the application of measures for radical research is NOAA's work on exploring novel methods of data compression. New methods of data compression are critical for the next generation of Geostationary Operational Environmental Satellites (GOES), with the first of this series (GOES-R) to be launched in 2012. These satellites are expected to have sensors with resolution in the hyperspectral range, which would generate a raw unprocessed data stream of 130 Mbps and a processed data rate of 85 Mbps. Current GOES satellites have raw and processed data rates of 2.6 Mbps and 2.1 Mbps, respectively. Hence, the move to hyperspectral data will involve potentially significant increases in the power requirements and mass of satellites as well as the cost of broadcast rates and the archiving of data. In developing new methods of data compression, this project can draw on well-known existing state-of-the-art data compression methods, such as JPEG2000, in order to determine progress. However, the data represents a challenge to these methods because it is 3-dimensional (most compression methods are for 2D data) and any compression method must be lossless (or near lossless) due to the high sensitivity of error. The research project has been able to develop new compression methods that produces significant compression gains in the range of 50% over current state of the art methods, but improvements in compression rates were not tracked and utilised in assessing the performance of the research. The use of such measures, in conjunction with peer review to determine the radicalness of the research, would help to better determine the rate and scale of progress and allow managers the ability to better assess next steps and strategy. At the organisational level, the ability to develop measures and linkages to goals speaks to the need for a comprehensive performance management system. It would be necessary for such a system to address multiple impacts, as well as provide specific guidance in terms of both a technological (or research) roadmap and where to take corrective action, if needed. In our work with STAR, we utilised the Strategy Map and Balanced Scorecard approach of Kaplan and Norton. As they describe:
A strategy map for a Balanced Scorecard makes explicit the strategy's hypotheses. Each measure of a Balanced Scorecard becomes embedded in a chain of cause-and-effect logic that connects the desired outcomes from the strategy with drivers that will lead to the strategic outcomes. ? [The strategy map] provides executives with a framework for describing and managing strategy in a knowledge economy. (Kaplan and Norton, 2001)
In this manner, the balanced scorecard offered a way to provide a central focus for the activities and the measurement of progress for STAR research, that is, on the achievement of contributions to mission goals.
6
Concluding discussion
The aim in the studies we discussed above was primarily to explore the development of measures of research progress as a complement to existing measures. We are also suggesting how to measure the radicalness of that progress against a standard. We argue that the measures we developed could play a key role in the management of innovation. And as recent studies have shown, a key need for research that is focused on radical innovation or marked by high uncertainty is better data and documentation of the research process (Leifer et al., 2001; Doctor et al., 2001; Karlsson et al., 2004).
Measuring radical innovation in real time
373
As we have argued, the use of multiple measures of performance for radical research might provide a more accurate and more timely portrayal of technical progress than do the use of milestones or other performance measures, such as papers and patents. For instance, using the example of the figure of merit for the tunable filter, the measure of progress offers the ability to assess progress towards the completion of the filter. If the measure was simply a stage-gate determination of the completion of the filter, the inability to complete the filter would typically mean the discontinuation of the project. But the figure of merit provides significantly more information about the progress of the research along various important parameters which would allow managers make a more informed determination about the progress of the project prior to a specified stage and potentially suggest areas where further research might be more fruitful. In this manner, this approach would add timely measures of technical progress to the existing milestones and other performance measures, and it would seek to integrate these measures both into scientific knowledge, such as better understanding the causes of ozone depletion, and to the organisational/programmatic goal of reducing delays in forecasting various conditions, as in the case of NOAA's NWS. Of course, thesemeasures are complementary of existing measures, but the use of additional measures of the process of scientific and technological research allows for greater and more timely recognition of R&D accomplishments. For instance, with regard to publications, advances along one or more of these dimensions are typically not publishable depending upon the specific variable involved. Some variables are inherently more publishable than others, and it is perhaps the case that some variables in weather forecasting or model building are more likely to be favoured over others. Thus, these new measures of technical progress allow one to compare across research teams working in quite disparate areas so that the work of all is equally appreciated. To restate our basic argument, the use of real-time measures of progress and the assessment of the radicalness of that progress could provide critically needed information for the management of research. In particular, real-time measures would provide more timely information to determine the critical pathways of progress both within and across research projects, as well as larger research units. Further, the real-time measures that we discuss would more centrally locate the focus of performance management of research on the advance of knowledge and innovation, rather than bureaucratic processes. In the world of publicly funded or directed research, such measures could become an essential component for meeting GPRA and PART requirements while still directly valuing the research. As Feller (2002) points out, GPRA measures are expected to be objective and quantifiable, there are some allowances for agencies to develop alternative forms of performance measurements and reporting. In summary, the development of new metrics for R&D is an important step to successfully managing and generating innovation within research organisations. The challenge is developing measurements which allow managers to make informed and appropriate decisions about research and development, but do not pose unnecessary and distracting opportunity costs and obstruct the development of breakthroughs which, in fact, may alter the course of the research and knowledge.
374
J. Mote, G. Jordan and J. Hage
Acknowledgements
A previous version of this paper was presented at the February 2005 meeting of the American Association for the Advancement of Science (AAAS). The authors gratefully acknowledge the support of the US Department of Energy Office of Science and the Sandia National Laboratories Science, Technology and Engineering Foundations Strategic Management Unit and the National Oceanic and Atmospheric Administration (NOAA). This research has been performed under Sandia National Laboratories DOE contract DE-AC04-94AL85000. Sandia is operated by Sandia Corporation, a subsidiary of Lockheed Martin Corporation. Additional research was conducted under contract with NOAA. The opinions expressed are those of the authors, not the US Department of Energy or Sandia National Laboratories or NOAA.
References
Averch, H.A. (1994) 'Economic approaches to the evaluation of research', Evaluation Review, Vol. 18, pp.77-88. Blankenship, L.V. (1974) 'Management, politics, and science: a nonseparable system', Research Policy, Vol. 3, No. 3, pp.244-257. Cao, C., Weinreb, M. and Xu, H. (2004) 'Predicting simultaneous nadir overpasses among polar-orbiting meteorological satellites for the intersatellite calibration of radiometers', Journal of Atmospheric and Oceanic Technology, Vol. 21, pp.537-542. Colton, M., Powell, A., Jordan, G., Hage, J. and Mote, J. (2004) 'Minding the gaps: new insights into R&D management and operational transitions of NOAA satellite products', Proceedings of the International Society for Optical Engineering (SPIE), Vol. 5548, October, pp.1-10. Coombs, R., McMeekin, A. and Pybus, R. (1998) 'Toward the development of benchmarking tools for R&D project management', R&D Management, Vol. 28, No. 3, pp.175-186. Cooper, R.G. and Kleinschmidt, E.J. (1995) 'Benchmarking the firm's critical success factors in new product development', Journal of Product Innovation Management, Vol. 12, pp.374-391. Cozzens, S.E., Popper, S., Bonomo, J., Koizumi, K. and Flanagan, A. (1994) Methods for Evaluating Fundamental Science, DRU-875/2-CTI, RAND. Department of Commerce (2003) Strategic Plan, FY 2004-FY 2009. Deuten, J. and Rip, A. (2000) 'Narrative infrastructure in product creation processes', Organization, Vol. 7, No. 1, pp.69-94. Doctor, R.N., Newton, D.P. and Pearson, A. (2001) 'Managing uncertainty in research and development', Technovation, Vol. 21, pp.79-90. Feller, I. (2002) 'Performance measurement redux', American Journal of Evaluation, Vol. 23, No. 4, pp.435-452. Frenkel, A., Grupp, H. and Maital, S. (2000) 'Measuring dynamic technical change: a technometric approach', International Journal of Technology Management, Vol. 20, Nos. 3-4, pp.429-441. Garcia, R. and Calantone, R. (2002) 'A critical look at technological innovation typology and innovativeness terminology: a literature review', Journal of Product Innovation Management, Vol. 19, No. 2, pp.10-32. Geisler, E. (2000) The Metrics of Science and Technology, Westport, CT: Quorum Books. Green, S.G., Gavin, M.B. and Aiman-Smith, L. (1995) 'Assessing a multidimensional measure of radical technological innovation', IEEE Transactions on Engineering Management, Vol. 42, No. 3, pp.203-214. Grupp, H. (1994) 'The measurement of technical performance innovations by technometrics and its impact on establish technology indicators', Research Policy, Vol. 23, No. 2, pp.175-193.
Measuring radical innovation in real time
375
Grupp, H. and Hohmeyer, O. (1986) 'A technometric model for the assessment of technological standards and their application to selected technology-intensive products', Technological Forecasting and Social Change, Vol. 30, No. 2, pp.123-137. Jensen, K. and Warren, P. (2001) 'The use of options theory to value research in the service sector', R&D Management, Vol. 31, No. 2, pp.191-201. Jordan, G.B., Hage, J., Mote, J. and Hepler, B. (2005) 'Investigating differences among research projects and implications for managers', R&D Management, Vol. 35, No. 5, pp.501-511. Kaplan, R. and Norton, D. (2001) The Strategy-focused Organization: How Balanced Scorecard Companies Thrive in the New Business Environment, p.69. Karlsson, M., Trygg, L. and Elfstrom, B.O. (2004) 'Measuring R&D productivity: complementing the picture by focusing on research activities', Technovation, Vol. 24, pp.179-186. Lenz, R.C. (1985) 'A heuristic approach to technology measurement', Technological Forecasting and Social Change, Vol. 27, pp.249-264. Leifer, R., McDermott, C., O'Connor, G., Peters, L., Rice, M. and Veryzer, R. (2000) Radical Innovation: How Mature Companies can Outsmart Upstarts, Cambridge, MA: Harvard Business School Press. Leifer, R., O'Connor, G.C. and Rice, M. (2001) 'Implementing radical innovation in mature firms: the role of hubs', Academy of Management Executive, Vol. 15, No. 3, pp.102-113. McDermott, C. and O'Connor, G.C. (2002) 'Managing radical innovation: an overview of emergent strategy issues', Journal of Product Innovation Management, Vol. 19, No. 6, pp.424439. Narin, F., Carpenter, M. and Woolf, P. (1984) 'Technological performance assessment based on patents and patent citations', IEEE Transactions on Engineering Management, Vol. 31, No. 4, pp.172-183. Pavitt, K. (1991) 'What makes basic research economically useful?', Research Policy, Vol. 20, No. 2, pp.109-119. Pavitt, K., Robson, M. and Townsend, J. (1989) 'Technological accumulation, diversification and organisation in UK companies, 1945-1983', Management Science, Vol. 35, No. 1, pp.81-99. Raz, T., Shenhar, A.J. and Dvir, D. (2002) 'Risk management, project success, and technological uncertainty', R&D Management, Vol. 32, No. 2, pp.101-109. Rinia, E.J., van Leeuwen, T.N., van Vuren, H.G. and van Raan, A.F.J. (2001) 'Influence of interdisciplinarity on peer-review and bibliometric evaluations in physics research', Research Policy, Vol. 30, No. 3, pp.357-361. Roussel, P.A., Saad, K.M. and Erickson, T.J. (1991) 3rd Generation R&D, Boston, MA: Harvard Business School Press. Schainblatt, A.H. (1982) 'How companies measure the productivity of engineers and scientists', Research Management, Vol. 25, No. 3, pp.10-18. Schumann, P.A., Jr., Ransley, D.L. and Prestwood, C.L. (1995) 'Measuring R&D performance', Research-Technology Management, Vol. 38, No. 3, pp.45-55. Trucano, T., Easterling, R., Dowding, K., Paez, T., Urbina, A., Romero, V., Rutherford, B. and Hills, R. (2001) 'Description of the Sandia validation metrics project', Sandia Report, SAND2001-1339, Sandia National Laboratories. Zoltan, J.A., Anselin, L. and Varga, A. (2002) 'Patents and innovation counts as measures of regional production of new knowledge', Research Policy, Vol. 31, No. 7, pp.1069-1085.
376
J. Mote, G. Jordan and J. Hage
Bibliography
Baker, N., Green, S.G. and Bean, A.S. (1986) 'Why R&D projects succeed or fail', Research Management, November-December, Vol. 24, No. 6. Balachandra, R. and Friar, J.H. (1997) 'Factors for success in R&D projects and new product innovation: a contextual framework', IEEE Transactions on Engineering Management, Vol. 44, No. 3, pp.276-287. Brown, S. and Eisenhardt, K. (1995) 'Product development: past research, present findings, and future directions', Academy of Management Review, Vol. 20, No. 2, pp.343-378. Burns, T. and Stalker, G. (1961) The Management of Innovation, London: Tavistock Publications. Callon, M. (1980) 'The state and technical innovation: a case study of the electrical vehicle in France', Research Policy, Vol. 9, pp.358-376. Chandler, A. (1962) Strategy and Structures, Cambridge, MA: MIT Press. Chiesa, V. (2001) R&D Strategy and Organisation: Managing Technical Change in Dynamic Contexts, River Edge, NJ: Imperial College Press. Cooper, R. and Kleinschmidt, E. (1987) 'New products: what separates winners from losers', Journal of Product Innovation Management, Vol. 4, No. 3, pp.169-184. Damanpour, F. (1991) 'Organizational innovation: a meta-analysis of effects, determinants and moderators', Academy of Management Journal, Vol. 34, pp.555-590. Davis, J., Fusfeld, A., Scriven, E. and Tritle, G. (2001) 'Determining a project's probability of success', Research Technology Management, Vol. 44, No. 3, pp.51-57. Freeman, M. and Beale, P. (1992) 'Measuring project success', Project Management Journal, Vol. 23, No. 1, pp.8-17. Griffin, A. and Page, A. (1996) 'PDMA success measurement project: recommended measures for product development success and failure', Journal of Product Innovation Management, Vol. 13, No. 6, pp.478-496. Henderson, R.M. and Clark, K.B. (1990) 'Architectural innovation: there configuration of existing product technologies and the failure of established firms', Administrative Science Quarterly, Vol. 35, No. 1, pp.9-30. Knight, F. (1921) Risk, Uncertainty and Profit, Boston: Houghton Mifflin. Kumar, V., Persaud, A. and Kumar, U. (1996) 'To terminate or not an ongoing R&D project: a managerial dilemma', IEEE Transactions on Engineering Management, Vol. 43, No. 3, pp.273284. Lewis, M. (2001) 'Success, failure and organizational competence: a case study of the new product development process', Journal of Engineering and Technology Management, Vol. 18, No. 2, pp.185-206. Lipovetsky, S., Tishler, A., Dvir, D. and Shenhar, A. (1997) 'The relative importance of project success dimensions', R&D Management, Vol. 27, No. 2, pp.97-106. Paolini, A., Jr. and Glaser, M.A. (1997) 'Product selection methods to pick winners', Research Management, Vol. 20, pp.26-29. Pavitt, K. (1990) 'What we know about the strategic management of technology', California Management Review, Spring, pp.17-26. Pinto, J.K. and Slevin, D.P. (1988) 'Project success: definitions and measurement techniques', Project Management Journal, Vol. 19, No. 3, pp.67-73. Raz, T., Shenhar, A. and Dvir, D. (2002) 'Risk management, project success, and technological uncertainty', R&D Management, Malden, MA: Blackwell Publishers. Rothwell, R. (1985) 'Project SAPPHO: comparative study of success and failure in industrial innovation', Information Age, Vol. 7, No. 4, pp.215-220. Schumann, P.A., Jr. and Ransley, D.L. (1995) 'Measuring R&D performance', Research Technology Management, Vol. 38, No. 3, pp.45-55.
Measuring radical innovation in real time
377
Shenhar, A. (2001) 'One size does not fit all projects: exploring classical contingency domains', Management Science, Vol. 47, No. 3, pp.394-414. Shenhar, A. and Dvir, D. (1996) 'Towards a typological theory of project management', Research Policy, Vol. 25, pp.607-632. Wörner, S.D. and Grupp, H. (2003) 'The derivation of R&D return indicators within a real options framework', R&D Management, Vol. 33, No. 3, pp.313-325.
Notes
1 For instance, the goal of one of the projects we studied was to eliminate the need for reagents in detecting arsenic in water. The current state of art in arsenic detection requires two reagents, so the movement from two to zero limits the efficacy of this measurement. Government Performances Results Act of 1993 (GPRA) and Program Assessment Rating Tool (PART). Moderate resolution imaging spectroradiometer.
2 3
doc_402586477.docx