Description
This paper describes a novel approach to business intelligence and program management for large technology enterprises like the U.S. National Aeronautics and Space Administration (NASA).
1
The NASA Program Management Tool:
A New Vision in Business Intelligence
David G. Bell
[email protected]
650-604-0771
Peter Putz
[email protected]
650-604-2137
Universities Space Research Association
NASA Ames Research Center
Moffett Field, CA 94035
David A. Maluf
[email protected]
650-604-0611
Keith Swanson
[email protected]
650-604-6016
NASA Ames Research Center
Moffett Field, CA 94035
Yuri Gawdiak
NASA Headquarters
300 E Street SW
Washington DC 20546
202-358-1853
[email protected]
Abstract
1
—This paper describes a novel approach to
business intelligence and program management for large
technology enterprises like the U.S. National Aeronautics
and Space Administration (NASA). Two key distinctions of
the approach are that 1) standard business documents are the
user interface, and 2) a “schema-less” XML
2
database
enables flexible integration of technology information for
use by both humans and machines in a highly dynamic
environment. The implementation utilizes patent-pending
NASA software called the NASA Program Management
Tool (PMT) and its underlying “schema-less” XML
database called Netmark. Initial benefits of PMT include
elimination of discrepancies between business documents
that use the same information and “paperwork reduction”
for program and project management in the form of
reducing the effort required to understand standard reporting
requirements and to comply with those reporting
requirements. We project that the underlying approach to
business intelligence will enable significant benefits in the
timeliness, integrity and depth of business information
available to decision makers on all organizational levels.
TABLE OF CONTENTS
1. INTRODUCTION........................................................ 1
2. DESIGN REQUIREMENTS......................................... 2
3. PMT FEATURES ...................................................... 3
4. PMT ARCHITECTURE............................................. 4
5. CONCLUSIONS.......................................................... 5
REFERENCES................................................................ 6
BIOGRAPHY.................................................................. 7
1. INTRODUCTION
The Program Management Tool is a custom-built business
intelligence solution for NASA to successfully manage
large programs. It enables program and task managers to
communicate success critical information on the status and
1
0-7803-9546-8/06/$20.00© 2006 IEEE
2
Extensible Markup Language -http://www.w3.org/XML/
progress of all program levels in an efficient and always
current manner. PMT keeps track of project goals, risks,
milestones and deliverables, and assists the proper
allocation of financial, material, and human resources. It is
well integrated with other agency-wide information systems.
PMT is like the nervous system of a complex organization
like NASA —it rapidly transports vital information from its
operating parts to decision-making authorities and back,
allowing a program to move ahead in a coherent and
efficient manner and to master changes in the environment
fast and proactively.
PMT supports all essential program management activities
and corresponding documents like: creation and monitoring
of annual task plans; monthly reporting of technical,
schedule, management, and budget status; tracking budget
phasing plans; analyzing program risks and mitigation
strategies; reporting and evaluating project life cycle costs;
accessing convenient aggregated views; automatically
creating Earned Value Management assessments, Quad-
Charts and other reports.
PMT is more of an integrator than a standalone thus
connecting multiple distributed resources across the agency;
namely ERASMUS reporting system and to the NASA
Technology Inventory Database (NTI), and the Integrated
Financial Management System (IFMP) thereby significantly
reducing cost and time for entering the same data multiple
times into different systems.
The PMT data architecture is designed to support model-
based program management by abstracting NASA 7120.5
project planning guidelines into the PMT either directly via
data elements and management monitoring algorithms,
and/or via middleware links to portfolio, risk, configuration,
and other tools.
Management of large technology programs involves a
significant amount of information, which is often delivered
using standard templates that meet monthly, quarterly and
yearly reporting requirements. Managers who need to be
2
able to gather and understand large amounts of business
information in busy meetings have a vital interest in the
standardization of report formats. Therefore, information is
not only required based on templates, but also required in
the format of specific software packages such as
Microsoft® Excel, Microsoft® Word, Microsoft®
Powerpoint® and Adobe® Acrobat®. While the source of
this information is often found in multiple databases with
multiple schemas, and the destination of this information
often goes to multiple databases with multiple schemas, the
human readable format is most critical since management
decisions are typically based on conversations and
exchanges between humans not machines. Standard formats
are required for exchanges between technical experts and
decision-making authorities in order for the decision-
making authorities to make informed choices using compa-
rable information across the portfolio of selected projects.
Integrated business intelligence is an approach that not only
enables automated exchange of information between
machines, but also enables automated use of standard
business documents, spreadsheets and presentations that are
an integral part of the exchange of information between
humans for decision making. Business documents become
the user interface, and information entered once in business
documents gets automatically aggregated and repurposed
into standard outputs for use by both humans and machines.
This approach is enabled by patent-pending software
developed at the U.S. National Aeronautics & Space
Administration (NASA), including a novel document-based
program management application and its novel underlying
“schema-less” XML database.
Key distinctions of the approach are:
(1) Standardized business documents are a key user
interface (spreadsheets, presentations, documents),
eliminating the need for information administration by
automating the communication of information using
standard templates, and
(2) A 'schema-less' XML database enables integration and
query-based document composition, eliminating the
need for database administration by automating the
integration of information that have diverse schemas.
A key breakthrough of this approach is that the use of
business documents as the inputs and outputs of the
management system enables transparency up and down the
management chain. Project managers can see how their
project information gets summarized in the diversity of
presentations and documents used by program managers.
Program managers can also see the more detailed project
information beyond the project summaries they normally
receive. This approach eliminates discrepancies between
multiple documents that use the same project information,
since the multiple project documents are automatically
created from the XML database.
2. DESIGN REQUIREMENTS
For the management of large technology programs the
integration of heterogeneous and distributed technology
information is required, and is feasible with modern
information technologies. Heterogeneous information
involves diverse data structures in structured databases, and
unstructured information stored in spreadsheets, word
processing documents, presentations and other formats used
for human analysis and communication. In the end, it is
humans who make management decisions, so this is not just
an integration challenge, but also a challenge of composing
information from heterogeneous sources for use in human
communication and decision-making.
A key to addressing the data heterogeneity issue is the
adoption and enforcement of standards across information
systems. While XML is clearly emerging to be the lingua
franca for databases and internet systems, it is encouraging
to see the enforcement of standards even at the level of
Microsoft® Word and Microsoft® Powerpoint®
documents. For instance in the NASA enterprise the new
ESR&T
3
program has explicitly specified the requirements
and formats for monthly project reports. Each project is
required to submit a monthly progress report, in accordance
with its approved project plan or contract Statement of
Work, that provides information on the progress of the
project during the previous month, in the areas of technical
accomplishments, status against schedule, spending,
performance against Earned Value Management metrics,
planned activities for the following month, and other
information pertinent to the tracking and management of the
project. The formats have been specified for Word and
Powerpoint documents which will be the reporting formats
for this program.
The key capabilities that thus must be incorporated in an
effective program management information system for an
enterprise such as NASA are summarized as follows:
(1) Integrating information from heterogeneous
technology databases in a consistent manner, for
example project, program and enterprise-level
technology databases; past, current, and future
technology databases; technical, managerial and
financial databases.
(2) Composing analyses and reports from heterogeneous
technology databases in support of diverse technology
management processes, such as gap analyses,
technology assessments, and technology roadmaps;
and, historical problem/failure trend reports,
bug/requirements tracking, concept/feature tracking.
(3) Communicating technology information among diverse
technology development stakeholders and technology
database systems in order to assess data pedigree and
3
Exploration systems research and technology
3
support effective decision making for diverse
organizational processes, such as project and program
management, internal and external reviews, intellectual
property and investment management, gap analyses
and technology forecasting; and diverse database
systems.
3. PMT FEATURES
PMT has been developed to meet the above mentioned
requirements and the vision of a business intelligence
solution that enables humans to communicate efficiently
across organizational boundaries by seamlessly integrating
heterogeneous data sources. The two key distinctions of the
approach to integrated business intelligence incorporated in
PMT are: 1) standardized business documents as a key user
interface, and 2) a “schema-less” XML database enables
integration. These key features distinguish PMT from an
entire category of existing project and program management
tools that are commercially available.
Figure 1 – PMT Portal
The design philosophy of PMT has been to develop a
powerful program management information system while
providing a simple and intuitive user interaction through
standardized business documents in all stages of the
program management lifecycle. PMT supports all essential
program management activities and corresponding
documents such as creation and monitoring of annual task
plans; monthly reporting of technical, schedule,
management, and budget status; tracking budget phasing
plans; analyzing program risks and mitigation strategies;
reporting and evaluating project life cycle costs; accessing
convenient aggregated views; and automatically creating
Quad-Charts and other reports. It is a comprehensive, web-
enabled tool which provides an intuitive and enhanced web
interface for all user activities from setting up the basic
program configuration, to creating reports and charts.
Alternatively, data input can be easily done off-line by
downloading annual task plans, monthly reports and other
documents to a local desktop. They can be edited with
conventional spreadsheet software and uploaded to PMT by
the push of a button. Figure 2 provides an illustration of a
typical input document:
Figure 2 – Spreadsheet as PMT Input
At NASA input documents such as Task Plans and Monthly
reports are typically in formats such as Microsoft® Excel,
Microsoft® Word, Microsoft® Powerpoint® and Adobe®
Acrobat® since they can be easily distributed by email.
PMT users just take the spreadsheets, presentations, and
reports that they use in their day-to-day activities to input
data into the PMT system. This input process is extremely
simple, typically requiring users to upload information
through a web server, or simply drag and drop documents
into (Web DAV
4
) folders on the desktop. The input data is
automatically converted into XML and stored in a high-
throughput information management system (discussed in
detail in the Technical Details section).
PMT can automatically generate aggregated documents
from several input documents from multiple programs, very
quickly. For instance PMT has been used to automatically
generate NASA IBPDs (Integrated Budget Performance
Documents) spreadsheets using rollup approaches, thus
saving significant time and eliminating budget errors. The
details and benefits are discussed further in the conclusions
section. Other examples of the aggregated documents are
integrated budget/cost reports or summary Quad-Charts that
can be automatically created by PMT.
PMT is also interfaced and interoperable with existing
NASA information systems. For instance it is connected to
the ERASMUS
5
reporting system and to the NASA
Technology Inventory
6
, thereby significantly reducing cost
and time for entering the same data multiple times into
different systems.
4
Discussed shortly
5
http://www.hq.nasa.gov/erasmus/splash_nasa.htm : A NASA project
performance dashboard
6
http://inventory.gsfc.nasa.gov/ : Database of NASA ongoing technology
activities
4
Figure 3 – PMT Document Workflow
In essence, PMT acts like the nervous system of a complex
living organism—it rapidly transports vital information
from its operating parts to decision-making authorities and
back, allowing a program to move ahead in a coherent and
efficient manner and to master changes in the environment
fast and proactively.
A schematic illustration of the information workflow is
provided in Figure 3 above. While information storage and
other details are discussed in the following section, we must
stress that this approach manages information using open
and interoperable standards (i.e., XML) but without
enforcing day to day users to mark-up their data in any
particular format. We begin with enterprise information in
business documents which is converted automatically to
XML for storage, management, and integration, and which
is then composed and formatted for output to business
documents and applications as needed for end applications.
4. PMT ARCHITECTURE
A high level architectural description of PMT is illustrated
in Figure 4 on the next page. The architecture is based on
two primary levels: 1) client and 2) Netmark-XDB
7
service;
the figure also shows on a more detailed level internal
components and communication between components.
7
The term Netmark and XDB are used interchangeably in this paper.
At the heart of PMT is a high-throughput information
integration and management system, also developed at
NASA, called Netmark-XDB [1]. The key design
considerations held in the Netmark system design are
outlined below:
• Enabling meaningful government-to-government
information sharing using international standards, in
order to improve mission safety and success
• Enabling effective information management systems
that utilize indigenous user interfaces (e.g.,
spreadsheets as user interfaces), in order to improve
system usability
• Enabling rapid development of customized system
applications with an enabling platform, in order to
reduce time from system concept to system deployment
• Eliminating the need for database analysts using a
‘schema-less’ design, in order to reduce system
maintenance costs
Netmark supports the XML standard for metadata and
information interchange thus making it an open system that
is interoperable with a wide variety of other information
systems and tools.
Netmark is essentially a data management system for “semi-
structured” data i.e., data in the above enumerated kinds of
business documents (such as reports in Word or PDF,
5
presentations in formats such Powerpoint, Excel
spreadsheets etc.) which does not have a formal “schema”,
but where there is indeed some structure in the document
implicit from the format and layout. Data loaded into
Netmark (done by simple drag and drop) is converted into
XML and stored in the data store. The data store is an XML-
over-relational data management system. Data can be
retrieved very conveniently from Netmark using simple
Web-based interfaces that require simple keyword-based
input. Netmark also provides powerful data aggregation and
composition capabilities.
Figure 4 – PMT Software Architecture
Clients communicate with the XDB service exclusively over
the standards-compliant HTTP
8
and WebDAV
9
protocol
extensions. The protocol-specific headers, client
authentication, and feature negotiations are managed by the
HTTP server framework—currently Apache’s HTTPD
10
.
Information storage, retrieval, and recomposition modules
connect to the Apache server and provide their services
through handler API’s as defined in the Apache framework.
The XDB DAV module handles WebDAV requests for
document storage and retrieval—including decomposition
8
HyperText Transport Protocol –http://www.w3.org/Protocols/
9
Web Distributed Authoring and Versioning Protocol –http://www.webdav.org/specs/
10
Apache HTTPD Version 2 –http://httpd.apache.org/
via a pipeline system and storage of decomposed documents
in the XDB data store. Information is protected with a full-
featured access control mechanism to ensure data security.
The XDB Query module provides XDB datastore querying
capabilities including content and context querying,
information recomposition, server-side transformation
(using the Xalan
11
XSLT
12
processor), and information
access control enforcement.
5. CONCLUSIONS
PMT has been in use at NASA for about four years, initially
for all projects in a $40M technology research and
development program, subsequently for all projects in a
$500M+ research and development program, and has begun
to be used by some projects and is under consideration for
use by all projects in a multi-billion dollar technology
research and development program. PMT has been cited as
a key source of the successful passing of the $40M
program’s Non-Advocate Review (NAR) by the U.S. Office
of Management and Budget, and as a factor in the passing of
the $500M+ program’s NAR. The adoption and positive
feedback from program manager’s and non-advocate
reviewers provides qualitative measure of positive results
from using PMT.
Quantitative results can be measured in terms of the
integrity of the business documents used for management of
these technology programs. As one measure of the integrity
of information, program and project documents utilized
before and after the implementation of PMT on the $500M+
program were compared, with emphasis on discrepancies
that occurred between centrally administered registries and
local project documents around milestones and deliverables.
Milestones and deliverables represent one of the most
important aspects of the agreement between program and
project managers. Differences in program and project
manager’s understandings of the agreed upon milestones
can have significant impact on the project’s and program’s
success. Table 1 shows that prior to the use of PMT only
under 60% of all level three project milestones had complete
integrity between the project’s documents and the program’s
documents. Table 2 shows the breakdown in information
integrity for each of seven major projects, with levels of
information integrity varying from under 24% to 100%.
With the use of PMT the integrity of business documents
became 100% throughout, output documents were
automatically created from input documents without any
manual cutting & pasting of information.
11
Apache Xalan C++ XSLT Engine –http://xml.apache.org/xalan-
c/index.html
12
The Extensible Stylesheet Language –http://www.w3.org/Style/XSL/
6
Table 1 – Differences in milestone data for a major program prior to the use of PMT
Different
MS #
Different
Date
Different
MS Description
Any Difference
Count % Count % Count % Count %
match 267 85.3% 259 82.7% 232 74.1% 185 59.1%
mismatch 43 13.7% 51 16.3% 74 23.6% 121 38.7%
missing 7 1.0% 7 1.0% 7 2.2% 7 2.2%
Total 313 100.0% 313 100.0% 313 100.0% 313 100.0%
Table 2 – Differences in milestone data for specific projects in a major program prior to the use of PMT
Different MS # Different Date
Different MS
Description
Any Difference?
Count % Count % Count % Count %
match 57 76.0% 66 88.0% 54 72.0% 37 49.3%
mismatch 18 24.0% 9 12.0% 21 28.0% 38 50.7%
missing
P
r
o
j
e
c
t
1
Total 75 100.0% 75 100.0% 75 100.0% 75 100.0%
match 38 100.0% 38 100.0% 38 100.0% 38 100.0%
mismatch
missing
P
r
o
j
e
c
t
2
Total 38 100.0% 38 100.0% 38 100.0% 38 100.0%
match 28 90.3% 29 93.5% 27 87.1% 26 83.9%
mismatch 2 6.5% 1 3.2% 3 9.7% 4 12.9%
missing 1 3.2% 1 3.2% 1 3.2% 1 3.2%
P
r
o
j
e
c
t
3
Total 31 100.0% 31 100.0% 31 100.0% 31 100.0%
match 46 90.2% 45 88.2% 35 68.6% 29 56.9%
mismatch 5 9.8% 6 11.8% 16 31.4% 22 43.1%
missing
P
r
o
j
e
c
t
4
Total 51 100.0% 51 100.0% 51 100.0% 51 100.0%
match 21 100.0% 20 95.2% 18 85.7% 17 81.0%
mismatch 1 4.8% 3 14.3% 4 19.0%
missing
P
r
o
j
e
c
t
5
Total 21 100.0% 21 100.0% 21 100.0% 21 100.0%
match 15 42.9% 9 42.9% 9 42.9% 5 23.8%
mismatch 4 47.6% 10 47.6% 10 47.6% 14 66.7%
missing 2 9.5% 2 9.5% 2 9.5% 2 9.5%
P
r
o
j
e
c
t
6
Total 21 100.0% 21 100.0% 21 100.0% 21 100.0%
match 62 81.6% 52 68.4% 51 67.1% 33 43.4%
mismatch 10 13.2% 20 26.3% 21 27.6% 39 51.3%
missing 4 5.3% 4 5.3% 4 5.3% 4 5.3%
P
r
o
j
e
c
t
7
Total 76 100.0% 76 100.0% 76 100.0% 76 100.0%
7
REFERENCES
[1] David Maluf and Peter Tran, “NETMARK: A Schema-
less Extension for Relational Databases for Managing
Semi-Structured Data Dynamically,“ Proceedings of the
14
th
International Symposium on Methodologies for
Intelligent Systems, October 28-31, 2003, 231-241
BIOGRAPHY
David Bell is Director and Senior Scientist at the Research
Institute for Advanced Computer Science, located at the
NASA Ames Research Center. Prior to working at NASA,
David worked for ten years at the Xerox Palo Alto Research
Center, and previously held an appointment at MIT where
he led a research program in the Center for Innovation in
Product Development. David is co-inventor of multiple
patent and patent-pending information system technologies,
including XML query technologies related to NETMARK
and the NASA Program Management Tool, extensible blog
technology called Sparrow Web, and distributed knowledge
management software called Eureka. David received his
Ph.D. from Cornell University with a dissertation on the
dynamics of product development processes.
Yuri Gawdiak is a program integration manager for
NASA’s Aeronautics Research Mission Directorate and has
been working advanced information systems and strategic
planning for twenty years. He has worked in industry,
academia, and government covering space,
aeronautics, human factors, and engineering research. At
NASA he has worked at Johnson Space Flight Center on the
Space Station program including principal investigator for
the Wireless Network Experiment on STS-76/Mir-21. He
earned his BS in information systems with a concentration
in ergonomics at Carnegie Mellon University. For his work
in Personal Satellite Assistant as well as other programs
and project Mr. Gawdiak has been awarded both the NASA
Outstanding Leadership and Exceptional Achievement
medals.
David A. Maluf leads the NASA Advanced Exploration
Network laboratory (AEN), a laboratory consisting of 20+
staff with an average of 12 projects/year. He has over 70
technical publications in journals and conference
proceedings, over hundreds of presentations at
international conferences and is an inventor of numerous
patents. He has taught courses on system engineering and
databases, and has written two books. He received his PhD
from McGill University and conducted post-doctoral
research in information integration at Stanford University.
Peter Putz is a management scientist with the Research
Institute for Advanced Computer Science (RIACS) at the
NASA Ames Research Center. Previously he was a member
of research staff with the Xerox Palo Alto Research Center
(which is now PARC Inc.) where he was doing research on
learning and knowledge sharing strategies together with an
interdisciplinary group of social scientists in the
Knowledge, Interaction and Practice Area. Peter received
his Ph.D. for the Johannes Kepler University Linz, Austria.
There he was an assistant professor with the Department of
Business Information Systems and the Department of
Management for more then ten years.
Keith Swanson is a computer scientist in the Advanced
Exploration Networks laboratory of the Intelligent Systems
Division at NASA Ames Research Center. He has over 20
years of technology management and development
experience in the areas of system health management,
planning and scheduling, and knowledge-based systems.
Keith has a Master's degree in Computer Science from
Stanford University and a Master's Degree in Engineering
from UC Berkeley.
doc_703104334.pdf
This paper describes a novel approach to business intelligence and program management for large technology enterprises like the U.S. National Aeronautics and Space Administration (NASA).
1
The NASA Program Management Tool:
A New Vision in Business Intelligence
David G. Bell
[email protected]
650-604-0771
Peter Putz
[email protected]
650-604-2137
Universities Space Research Association
NASA Ames Research Center
Moffett Field, CA 94035
David A. Maluf
[email protected]
650-604-0611
Keith Swanson
[email protected]
650-604-6016
NASA Ames Research Center
Moffett Field, CA 94035
Yuri Gawdiak
NASA Headquarters
300 E Street SW
Washington DC 20546
202-358-1853
[email protected]
Abstract
1
—This paper describes a novel approach to
business intelligence and program management for large
technology enterprises like the U.S. National Aeronautics
and Space Administration (NASA). Two key distinctions of
the approach are that 1) standard business documents are the
user interface, and 2) a “schema-less” XML
2
database
enables flexible integration of technology information for
use by both humans and machines in a highly dynamic
environment. The implementation utilizes patent-pending
NASA software called the NASA Program Management
Tool (PMT) and its underlying “schema-less” XML
database called Netmark. Initial benefits of PMT include
elimination of discrepancies between business documents
that use the same information and “paperwork reduction”
for program and project management in the form of
reducing the effort required to understand standard reporting
requirements and to comply with those reporting
requirements. We project that the underlying approach to
business intelligence will enable significant benefits in the
timeliness, integrity and depth of business information
available to decision makers on all organizational levels.
TABLE OF CONTENTS
1. INTRODUCTION........................................................ 1
2. DESIGN REQUIREMENTS......................................... 2
3. PMT FEATURES ...................................................... 3
4. PMT ARCHITECTURE............................................. 4
5. CONCLUSIONS.......................................................... 5
REFERENCES................................................................ 6
BIOGRAPHY.................................................................. 7
1. INTRODUCTION
The Program Management Tool is a custom-built business
intelligence solution for NASA to successfully manage
large programs. It enables program and task managers to
communicate success critical information on the status and
1
0-7803-9546-8/06/$20.00© 2006 IEEE
2
Extensible Markup Language -http://www.w3.org/XML/
progress of all program levels in an efficient and always
current manner. PMT keeps track of project goals, risks,
milestones and deliverables, and assists the proper
allocation of financial, material, and human resources. It is
well integrated with other agency-wide information systems.
PMT is like the nervous system of a complex organization
like NASA —it rapidly transports vital information from its
operating parts to decision-making authorities and back,
allowing a program to move ahead in a coherent and
efficient manner and to master changes in the environment
fast and proactively.
PMT supports all essential program management activities
and corresponding documents like: creation and monitoring
of annual task plans; monthly reporting of technical,
schedule, management, and budget status; tracking budget
phasing plans; analyzing program risks and mitigation
strategies; reporting and evaluating project life cycle costs;
accessing convenient aggregated views; automatically
creating Earned Value Management assessments, Quad-
Charts and other reports.
PMT is more of an integrator than a standalone thus
connecting multiple distributed resources across the agency;
namely ERASMUS reporting system and to the NASA
Technology Inventory Database (NTI), and the Integrated
Financial Management System (IFMP) thereby significantly
reducing cost and time for entering the same data multiple
times into different systems.
The PMT data architecture is designed to support model-
based program management by abstracting NASA 7120.5
project planning guidelines into the PMT either directly via
data elements and management monitoring algorithms,
and/or via middleware links to portfolio, risk, configuration,
and other tools.
Management of large technology programs involves a
significant amount of information, which is often delivered
using standard templates that meet monthly, quarterly and
yearly reporting requirements. Managers who need to be
2
able to gather and understand large amounts of business
information in busy meetings have a vital interest in the
standardization of report formats. Therefore, information is
not only required based on templates, but also required in
the format of specific software packages such as
Microsoft® Excel, Microsoft® Word, Microsoft®
Powerpoint® and Adobe® Acrobat®. While the source of
this information is often found in multiple databases with
multiple schemas, and the destination of this information
often goes to multiple databases with multiple schemas, the
human readable format is most critical since management
decisions are typically based on conversations and
exchanges between humans not machines. Standard formats
are required for exchanges between technical experts and
decision-making authorities in order for the decision-
making authorities to make informed choices using compa-
rable information across the portfolio of selected projects.
Integrated business intelligence is an approach that not only
enables automated exchange of information between
machines, but also enables automated use of standard
business documents, spreadsheets and presentations that are
an integral part of the exchange of information between
humans for decision making. Business documents become
the user interface, and information entered once in business
documents gets automatically aggregated and repurposed
into standard outputs for use by both humans and machines.
This approach is enabled by patent-pending software
developed at the U.S. National Aeronautics & Space
Administration (NASA), including a novel document-based
program management application and its novel underlying
“schema-less” XML database.
Key distinctions of the approach are:
(1) Standardized business documents are a key user
interface (spreadsheets, presentations, documents),
eliminating the need for information administration by
automating the communication of information using
standard templates, and
(2) A 'schema-less' XML database enables integration and
query-based document composition, eliminating the
need for database administration by automating the
integration of information that have diverse schemas.
A key breakthrough of this approach is that the use of
business documents as the inputs and outputs of the
management system enables transparency up and down the
management chain. Project managers can see how their
project information gets summarized in the diversity of
presentations and documents used by program managers.
Program managers can also see the more detailed project
information beyond the project summaries they normally
receive. This approach eliminates discrepancies between
multiple documents that use the same project information,
since the multiple project documents are automatically
created from the XML database.
2. DESIGN REQUIREMENTS
For the management of large technology programs the
integration of heterogeneous and distributed technology
information is required, and is feasible with modern
information technologies. Heterogeneous information
involves diverse data structures in structured databases, and
unstructured information stored in spreadsheets, word
processing documents, presentations and other formats used
for human analysis and communication. In the end, it is
humans who make management decisions, so this is not just
an integration challenge, but also a challenge of composing
information from heterogeneous sources for use in human
communication and decision-making.
A key to addressing the data heterogeneity issue is the
adoption and enforcement of standards across information
systems. While XML is clearly emerging to be the lingua
franca for databases and internet systems, it is encouraging
to see the enforcement of standards even at the level of
Microsoft® Word and Microsoft® Powerpoint®
documents. For instance in the NASA enterprise the new
ESR&T
3
program has explicitly specified the requirements
and formats for monthly project reports. Each project is
required to submit a monthly progress report, in accordance
with its approved project plan or contract Statement of
Work, that provides information on the progress of the
project during the previous month, in the areas of technical
accomplishments, status against schedule, spending,
performance against Earned Value Management metrics,
planned activities for the following month, and other
information pertinent to the tracking and management of the
project. The formats have been specified for Word and
Powerpoint documents which will be the reporting formats
for this program.
The key capabilities that thus must be incorporated in an
effective program management information system for an
enterprise such as NASA are summarized as follows:
(1) Integrating information from heterogeneous
technology databases in a consistent manner, for
example project, program and enterprise-level
technology databases; past, current, and future
technology databases; technical, managerial and
financial databases.
(2) Composing analyses and reports from heterogeneous
technology databases in support of diverse technology
management processes, such as gap analyses,
technology assessments, and technology roadmaps;
and, historical problem/failure trend reports,
bug/requirements tracking, concept/feature tracking.
(3) Communicating technology information among diverse
technology development stakeholders and technology
database systems in order to assess data pedigree and
3
Exploration systems research and technology
3
support effective decision making for diverse
organizational processes, such as project and program
management, internal and external reviews, intellectual
property and investment management, gap analyses
and technology forecasting; and diverse database
systems.
3. PMT FEATURES
PMT has been developed to meet the above mentioned
requirements and the vision of a business intelligence
solution that enables humans to communicate efficiently
across organizational boundaries by seamlessly integrating
heterogeneous data sources. The two key distinctions of the
approach to integrated business intelligence incorporated in
PMT are: 1) standardized business documents as a key user
interface, and 2) a “schema-less” XML database enables
integration. These key features distinguish PMT from an
entire category of existing project and program management
tools that are commercially available.
Figure 1 – PMT Portal
The design philosophy of PMT has been to develop a
powerful program management information system while
providing a simple and intuitive user interaction through
standardized business documents in all stages of the
program management lifecycle. PMT supports all essential
program management activities and corresponding
documents such as creation and monitoring of annual task
plans; monthly reporting of technical, schedule,
management, and budget status; tracking budget phasing
plans; analyzing program risks and mitigation strategies;
reporting and evaluating project life cycle costs; accessing
convenient aggregated views; and automatically creating
Quad-Charts and other reports. It is a comprehensive, web-
enabled tool which provides an intuitive and enhanced web
interface for all user activities from setting up the basic
program configuration, to creating reports and charts.
Alternatively, data input can be easily done off-line by
downloading annual task plans, monthly reports and other
documents to a local desktop. They can be edited with
conventional spreadsheet software and uploaded to PMT by
the push of a button. Figure 2 provides an illustration of a
typical input document:
Figure 2 – Spreadsheet as PMT Input
At NASA input documents such as Task Plans and Monthly
reports are typically in formats such as Microsoft® Excel,
Microsoft® Word, Microsoft® Powerpoint® and Adobe®
Acrobat® since they can be easily distributed by email.
PMT users just take the spreadsheets, presentations, and
reports that they use in their day-to-day activities to input
data into the PMT system. This input process is extremely
simple, typically requiring users to upload information
through a web server, or simply drag and drop documents
into (Web DAV
4
) folders on the desktop. The input data is
automatically converted into XML and stored in a high-
throughput information management system (discussed in
detail in the Technical Details section).
PMT can automatically generate aggregated documents
from several input documents from multiple programs, very
quickly. For instance PMT has been used to automatically
generate NASA IBPDs (Integrated Budget Performance
Documents) spreadsheets using rollup approaches, thus
saving significant time and eliminating budget errors. The
details and benefits are discussed further in the conclusions
section. Other examples of the aggregated documents are
integrated budget/cost reports or summary Quad-Charts that
can be automatically created by PMT.
PMT is also interfaced and interoperable with existing
NASA information systems. For instance it is connected to
the ERASMUS
5
reporting system and to the NASA
Technology Inventory
6
, thereby significantly reducing cost
and time for entering the same data multiple times into
different systems.
4
Discussed shortly
5
http://www.hq.nasa.gov/erasmus/splash_nasa.htm : A NASA project
performance dashboard
6
http://inventory.gsfc.nasa.gov/ : Database of NASA ongoing technology
activities
4
Figure 3 – PMT Document Workflow
In essence, PMT acts like the nervous system of a complex
living organism—it rapidly transports vital information
from its operating parts to decision-making authorities and
back, allowing a program to move ahead in a coherent and
efficient manner and to master changes in the environment
fast and proactively.
A schematic illustration of the information workflow is
provided in Figure 3 above. While information storage and
other details are discussed in the following section, we must
stress that this approach manages information using open
and interoperable standards (i.e., XML) but without
enforcing day to day users to mark-up their data in any
particular format. We begin with enterprise information in
business documents which is converted automatically to
XML for storage, management, and integration, and which
is then composed and formatted for output to business
documents and applications as needed for end applications.
4. PMT ARCHITECTURE
A high level architectural description of PMT is illustrated
in Figure 4 on the next page. The architecture is based on
two primary levels: 1) client and 2) Netmark-XDB
7
service;
the figure also shows on a more detailed level internal
components and communication between components.
7
The term Netmark and XDB are used interchangeably in this paper.
At the heart of PMT is a high-throughput information
integration and management system, also developed at
NASA, called Netmark-XDB [1]. The key design
considerations held in the Netmark system design are
outlined below:
• Enabling meaningful government-to-government
information sharing using international standards, in
order to improve mission safety and success
• Enabling effective information management systems
that utilize indigenous user interfaces (e.g.,
spreadsheets as user interfaces), in order to improve
system usability
• Enabling rapid development of customized system
applications with an enabling platform, in order to
reduce time from system concept to system deployment
• Eliminating the need for database analysts using a
‘schema-less’ design, in order to reduce system
maintenance costs
Netmark supports the XML standard for metadata and
information interchange thus making it an open system that
is interoperable with a wide variety of other information
systems and tools.
Netmark is essentially a data management system for “semi-
structured” data i.e., data in the above enumerated kinds of
business documents (such as reports in Word or PDF,
5
presentations in formats such Powerpoint, Excel
spreadsheets etc.) which does not have a formal “schema”,
but where there is indeed some structure in the document
implicit from the format and layout. Data loaded into
Netmark (done by simple drag and drop) is converted into
XML and stored in the data store. The data store is an XML-
over-relational data management system. Data can be
retrieved very conveniently from Netmark using simple
Web-based interfaces that require simple keyword-based
input. Netmark also provides powerful data aggregation and
composition capabilities.
Figure 4 – PMT Software Architecture
Clients communicate with the XDB service exclusively over
the standards-compliant HTTP
8
and WebDAV
9
protocol
extensions. The protocol-specific headers, client
authentication, and feature negotiations are managed by the
HTTP server framework—currently Apache’s HTTPD
10
.
Information storage, retrieval, and recomposition modules
connect to the Apache server and provide their services
through handler API’s as defined in the Apache framework.
The XDB DAV module handles WebDAV requests for
document storage and retrieval—including decomposition
8
HyperText Transport Protocol –http://www.w3.org/Protocols/
9
Web Distributed Authoring and Versioning Protocol –http://www.webdav.org/specs/
10
Apache HTTPD Version 2 –http://httpd.apache.org/
via a pipeline system and storage of decomposed documents
in the XDB data store. Information is protected with a full-
featured access control mechanism to ensure data security.
The XDB Query module provides XDB datastore querying
capabilities including content and context querying,
information recomposition, server-side transformation
(using the Xalan
11
XSLT
12
processor), and information
access control enforcement.
5. CONCLUSIONS
PMT has been in use at NASA for about four years, initially
for all projects in a $40M technology research and
development program, subsequently for all projects in a
$500M+ research and development program, and has begun
to be used by some projects and is under consideration for
use by all projects in a multi-billion dollar technology
research and development program. PMT has been cited as
a key source of the successful passing of the $40M
program’s Non-Advocate Review (NAR) by the U.S. Office
of Management and Budget, and as a factor in the passing of
the $500M+ program’s NAR. The adoption and positive
feedback from program manager’s and non-advocate
reviewers provides qualitative measure of positive results
from using PMT.
Quantitative results can be measured in terms of the
integrity of the business documents used for management of
these technology programs. As one measure of the integrity
of information, program and project documents utilized
before and after the implementation of PMT on the $500M+
program were compared, with emphasis on discrepancies
that occurred between centrally administered registries and
local project documents around milestones and deliverables.
Milestones and deliverables represent one of the most
important aspects of the agreement between program and
project managers. Differences in program and project
manager’s understandings of the agreed upon milestones
can have significant impact on the project’s and program’s
success. Table 1 shows that prior to the use of PMT only
under 60% of all level three project milestones had complete
integrity between the project’s documents and the program’s
documents. Table 2 shows the breakdown in information
integrity for each of seven major projects, with levels of
information integrity varying from under 24% to 100%.
With the use of PMT the integrity of business documents
became 100% throughout, output documents were
automatically created from input documents without any
manual cutting & pasting of information.
11
Apache Xalan C++ XSLT Engine –http://xml.apache.org/xalan-
c/index.html
12
The Extensible Stylesheet Language –http://www.w3.org/Style/XSL/
6
Table 1 – Differences in milestone data for a major program prior to the use of PMT
Different
MS #
Different
Date
Different
MS Description
Any Difference
Count % Count % Count % Count %
match 267 85.3% 259 82.7% 232 74.1% 185 59.1%
mismatch 43 13.7% 51 16.3% 74 23.6% 121 38.7%
missing 7 1.0% 7 1.0% 7 2.2% 7 2.2%
Total 313 100.0% 313 100.0% 313 100.0% 313 100.0%
Table 2 – Differences in milestone data for specific projects in a major program prior to the use of PMT
Different MS # Different Date
Different MS
Description
Any Difference?
Count % Count % Count % Count %
match 57 76.0% 66 88.0% 54 72.0% 37 49.3%
mismatch 18 24.0% 9 12.0% 21 28.0% 38 50.7%
missing
P
r
o
j
e
c
t
1
Total 75 100.0% 75 100.0% 75 100.0% 75 100.0%
match 38 100.0% 38 100.0% 38 100.0% 38 100.0%
mismatch
missing
P
r
o
j
e
c
t
2
Total 38 100.0% 38 100.0% 38 100.0% 38 100.0%
match 28 90.3% 29 93.5% 27 87.1% 26 83.9%
mismatch 2 6.5% 1 3.2% 3 9.7% 4 12.9%
missing 1 3.2% 1 3.2% 1 3.2% 1 3.2%
P
r
o
j
e
c
t
3
Total 31 100.0% 31 100.0% 31 100.0% 31 100.0%
match 46 90.2% 45 88.2% 35 68.6% 29 56.9%
mismatch 5 9.8% 6 11.8% 16 31.4% 22 43.1%
missing
P
r
o
j
e
c
t
4
Total 51 100.0% 51 100.0% 51 100.0% 51 100.0%
match 21 100.0% 20 95.2% 18 85.7% 17 81.0%
mismatch 1 4.8% 3 14.3% 4 19.0%
missing
P
r
o
j
e
c
t
5
Total 21 100.0% 21 100.0% 21 100.0% 21 100.0%
match 15 42.9% 9 42.9% 9 42.9% 5 23.8%
mismatch 4 47.6% 10 47.6% 10 47.6% 14 66.7%
missing 2 9.5% 2 9.5% 2 9.5% 2 9.5%
P
r
o
j
e
c
t
6
Total 21 100.0% 21 100.0% 21 100.0% 21 100.0%
match 62 81.6% 52 68.4% 51 67.1% 33 43.4%
mismatch 10 13.2% 20 26.3% 21 27.6% 39 51.3%
missing 4 5.3% 4 5.3% 4 5.3% 4 5.3%
P
r
o
j
e
c
t
7
Total 76 100.0% 76 100.0% 76 100.0% 76 100.0%
7
REFERENCES
[1] David Maluf and Peter Tran, “NETMARK: A Schema-
less Extension for Relational Databases for Managing
Semi-Structured Data Dynamically,“ Proceedings of the
14
th
International Symposium on Methodologies for
Intelligent Systems, October 28-31, 2003, 231-241
BIOGRAPHY
David Bell is Director and Senior Scientist at the Research
Institute for Advanced Computer Science, located at the
NASA Ames Research Center. Prior to working at NASA,
David worked for ten years at the Xerox Palo Alto Research
Center, and previously held an appointment at MIT where
he led a research program in the Center for Innovation in
Product Development. David is co-inventor of multiple
patent and patent-pending information system technologies,
including XML query technologies related to NETMARK
and the NASA Program Management Tool, extensible blog
technology called Sparrow Web, and distributed knowledge
management software called Eureka. David received his
Ph.D. from Cornell University with a dissertation on the
dynamics of product development processes.
Yuri Gawdiak is a program integration manager for
NASA’s Aeronautics Research Mission Directorate and has
been working advanced information systems and strategic
planning for twenty years. He has worked in industry,
academia, and government covering space,
aeronautics, human factors, and engineering research. At
NASA he has worked at Johnson Space Flight Center on the
Space Station program including principal investigator for
the Wireless Network Experiment on STS-76/Mir-21. He
earned his BS in information systems with a concentration
in ergonomics at Carnegie Mellon University. For his work
in Personal Satellite Assistant as well as other programs
and project Mr. Gawdiak has been awarded both the NASA
Outstanding Leadership and Exceptional Achievement
medals.
David A. Maluf leads the NASA Advanced Exploration
Network laboratory (AEN), a laboratory consisting of 20+
staff with an average of 12 projects/year. He has over 70
technical publications in journals and conference
proceedings, over hundreds of presentations at
international conferences and is an inventor of numerous
patents. He has taught courses on system engineering and
databases, and has written two books. He received his PhD
from McGill University and conducted post-doctoral
research in information integration at Stanford University.
Peter Putz is a management scientist with the Research
Institute for Advanced Computer Science (RIACS) at the
NASA Ames Research Center. Previously he was a member
of research staff with the Xerox Palo Alto Research Center
(which is now PARC Inc.) where he was doing research on
learning and knowledge sharing strategies together with an
interdisciplinary group of social scientists in the
Knowledge, Interaction and Practice Area. Peter received
his Ph.D. for the Johannes Kepler University Linz, Austria.
There he was an assistant professor with the Department of
Business Information Systems and the Department of
Management for more then ten years.
Keith Swanson is a computer scientist in the Advanced
Exploration Networks laboratory of the Intelligent Systems
Division at NASA Ames Research Center. He has over 20
years of technology management and development
experience in the areas of system health management,
planning and scheduling, and knowledge-based systems.
Keith has a Master's degree in Computer Science from
Stanford University and a Master's Degree in Engineering
from UC Berkeley.
doc_703104334.pdf