Description
Strategic management analyzes the major initiatives taken by a company's top management on behalf of owners, involving resources and performance in internal and external environments.
Performance, Audit and Evaluation Strategy
Genome Canada
V.6.0
November 2007
PERFORMANCE, AUDIT AND EVALUATION STRATEGY
_______________________________________________________
A.
Introduction
The purpose of this document is to present Genome Canada’s Performance, Audit and Evaluation Strategy (PAES). The strategy has been developed as a high level framework which addresses key elements that Genome Canada has or will put in place to ensure accountability in the achievement of objectives from the perspective of performance, audit, evaluation and reporting. The principles and directions outlined in the PAES will serve as a guide to the organization for the continuation, initiation and implementation of activities in these areas starting in fiscal year 2007-08.
B.
Background
Genome Canada currently has in place a wide array of policies, systems and processes that were developed over time to address activities related to performance, audit, evaluation and risk. They were also developed to ensure compliance to the funding agreements signed with Industry Canada with respect to the use and accounting of funds received from the federal government. Genome Canada also signs individual funding agreements with each of the six Genome Centres where the undertakings agreed to with Industry Canada are essentially replicated. The most recent funding agreement between Genome Canada and Industry Canada includes a provision which calls for these activities to be consolidated into one overarching strategy. The strategy would receive approval by the Board of Genome Canada and by the Minister of Industry Canada. Genome Canada, in consultation with Industry Canada will develop a Performance, Audit and Evaluation Strategy, in accordance with recognized standards, that will be approved by its Board and provided for review and approval of the Minister within 180 days of the signing of this Agreement, and updated thereafter as required. (Article 8.2 of the Funding Agreement with Industry Canada, dated March 29, 2007) The PAES satisfies this requirement by presenting an integrated view to the implementation of initiatives in the required areas.
C.
Elements of the Strategy
2
The PAES is comprised of three key elements and two supporting elements each of which are integral to the strategy. The elements are discussed below in the context of activities and initiatives currently in place, and those planned for implementation over the next two years. The three key elements are: • Performance Monitoring and Measurement Framework • Audit Framework • Evaluation Framework The two supporting elements are: • Performance Reporting • Risk Management Each of the elements and their underlying components are displayed in the figure below.
PERFORMANCE REPORTING Corporate Plan Annual Report Special Publications Web Site
PERFORMANCE MONITORING
AUDIT
EVALUATION
Project Selection ** Project Monitoring Interim Review Performance Indicators Final Project Reports
Financial Audits Recipient Audits Compliance Audits Performance Audits
Results Based Management And Accountability Framework (RMAF) Logic Model Evaluations Every Five Years
RISK MANAGEMENT Risk Management Framework Operations and Management Governance Regime
** For purpose of this document, the term “Project” means both Research Projects and Science and Technology
Platforms.
1.
Performance Monitoring and Measurement Framework
3
Performance monitoring and measurement starts at the project selection phase, continues through the execution phase with effective project management and monitoring (including regular data collection by Genome Centres on established performance metrics, and a formal interim review by international experts), and ends with the assessment of final project reports.
Current Practice
Project Selection
a. An independent International Science Review Panel, supported by external reviews of the science by peers, and information taken from an extensive due diligence process of financial and management aspects, assess and rank each of the proposals based on evaluation criteria as described in Competition Guidelines, and make recommendations for the approval of the Board of Directors.
Project Management and Monitoring
a. All projects are “managed” through the Genome Centres by Program Managers employed by the Centres who are responsible for on-going monitoring of project performance, active liaison with project leaders, managers and other members of the research team, and assessment of quarterly project reports. b. Projects report on progress quarterly (on milestones and performance metrics) to Genome Centres allowing for continuous hands on monitoring and opportunities for interventions when appropriate. c. Based on financial reports received from projects, Centres submit a “Quarterly Draw Request” to Genome Canada for estimated cash requirements of projects in the subsequent quarter. The request is based on comparison of previous advances to funds expended and forecast of cash needs for next quarter. Genome Canada’s Finance Group also monitors co-funding that is recorded by Centres on a Genome Canada data base, and conducts regular site visits to Genome Centres to assess the efficacy of processes related to financial management of projects.
d
Performance Indicators
a Genome Centres collect performance information on a regular basis (quarterly) based on a list of indicators as shown in Appendix 1.
b. Currently, these data are collected independently by each Centre and there is no national repository which is easily accessible by Genome Canada. Integration of this information on a national basis into Genome Canada’s data base is a planned initiative in the coming year.
Interim Review
4
a. After approximately 18 months of funding for each project, a formal interim review is undertaken (again by an International Science Review Committee) to assess progress to date against approved milestones and outcomes. b. Recommendations to the Board address whether projects should proceed as proposed, if they must respond to specific issues prior to continuation, or whether a site visit should be undertaken to review significant issues first hand.
Final Reports
a. Upon completion of each project, Genome Canada requires a Final Report from the Project Leaders. b Final Reports address project outputs/outcomes relative to approved objectives in several categories including: research, technology transfer, training and recruitment, benefits to Canada, outcomes of public outreach, international linkages, outcomes of GELS activities, and a final financial accounting.
c. To date, Genome Canada has received Final Reports from Competition I and II projects, the only projects which have been completed so far. The reports have been used primarily to collect data and metrics such as numbers of publications, patents, people trained, etc., which are summarized in public reporting of performance information (i.e.in the Annual Report).This information will also serve as an important data source in the overall evaluation of Genome Canada.
Planned Initiatives
To enhance its performance monitoring, and to better evaluate the results of its research investments, Genome Canada will: . a. Implement a process whereby Final Reports for projects originating from specific competitions will undergo an evaluative external review. The objective of the review will be to assess and report on the overall impact and outcomes of each project individually, and of the competition as a whole, based on pre-determined criteria and methodologies. b. Validate our criteria against internationally accepted methodologies to improve the process for measurement of economic impact of investments in genomics and proteomics. c. Implement a process whereby Centres record performance metrics as listed in Appendix 1 in Genome Canada’s data base on a continuous basis.
2.
Audit Framework
5
Genome Canada’s audit activities are comprised of a number of components that together provide a comprehensive framework for the external examination of the organization’s policies, programs, systems and procedures.
Current Practice
Annual Financial Audits
Genome Canada’s financial statements are audited annually as required by the Canada Corporations Act, the Funding Agreement with Industry Canada, and normal business practices. An external public accounting firm performs the audit and provides an opinion as to whether Genome Canada’s financial statements present fairly, in all material respects, the financial position of the Corporation as at fiscal year end, and the results of operation for the year then ended, in accordance with generally accepted accounting principles.
Recipient Audits
Recipient audits are those conducted on the recipients of Genome Canada funding. The concept of recipient audits was introduced in fiscal year 2006-07 when the Board, upon recommendation of the Audit Committee, approved Genome Canada’s Recipient Audit Framework and Risk Assessment Tool. Fundamental to the Framework is the concept of a risk based approach to the selection of projects to be audited as well as prescribed audit criteria as specified by the Canadian Institute of Chartered Accountants. Genome Centres are responsible for implementing the framework and for initiating and managing the audits.
Compliance Audits
Compliance audits are those that assess compliance with the terms and conditions of an agreement. As per the Funding Agreement with Industry Canada, both the Minister and the Auditor General have the right to initiate a compliance audit of Genome Canada. Only one compliance audit to date has been carried and it was done by Industry Canada in 2006.
Performance Audits
Performance Audits were previously referred to as Value For Money Audits (VFM). As per the Funding Agreement with Industry Canada, both the Minister and the Auditor General have the right to initiate a performance audit of Genome Canada to ensure the economy, efficiency and effectiveness with which federal funds have been used.
Planned Initiatives
Two additional activities are planned for implementation over the next two years. a. With respect to Performance Audits, the current Funding Agreement (unlike previous agreements) requires that Genome Canada cause a Performance Audit to be conducted on itself by an independent third-party by March 1, 2009, to ensure the economy, efficiency and effectiveness of funds used. Genome Canada plans to have this audit undertaken by a firm of professional accountants. The methodology of the Office of the Auditor General of Canada will be
6
used to ensure it is conducted in accordance with generally accepted auditing principles in this area. Work will be conducted essentially in three phases: Planning Phase: In this phase, the audit team sets out to understand the organization, the applicable authorities, the way the organization is structured to achieve its mandate, and the challenges it faces in the environment in which it operates. They then develop the criteria against which the audited activities can be assessed Examination phase: In this phase, the audit team gathers evidence (eg,through interviews with stakeholders, review of business documents and records, etc.) to allow it to report on the strengths and weaknesses of the organization being audited. Reporting Phase: In this phase the auditors draft the report that will eventually be presented to the Audit Committee and Board. After validation of facts and findings with management, the final report to the Board will detail their audit findings (what is working well and what needs improvement) and provide recommendations for improvements to operations and performance. The Audit Committee will work with an independent professional firm to ensure effective oversight of the Performance Audit. Development of the audit criteria will be a critical task of the Audit Committee. b. With respect to Compliance Audits, Genome Canada has undergone one compliance audit from Industry Canada. However no such audits have yet to be done on Genome Centres. Within the the next two years, Genome Canada will undertake Compliance Audits of the Centres (compliance to the funding agreements) to provide further assurance of effective management and use of funds that are flowed to them.
3.
Evaluation Framework
Genome Canada’s program evaluation strategy is taken from its Results Based Management and Accountability Framework (RMAF) and Logic Model which was developed in 2003. The RMAF displays Genome Canada’s results expectations – from activities to outcomes – and forms the basis of its evaluation initiatives. Standards in respect to evaluations are taken primarily from Treasury Board policies and guidelines.
Current Practice
Genome Canada has had one formal evaluation of its activities conducted and reported on in 2006. This “Interim” Evaluation was essentially an overview of whether Genome Canada was on track towards meeting its nine objectives (generally, it was). The results of the evaluation are posted on Genome Canada’s web site. No other formal program evaluations have been conducted.
Planned Initiatives
Genome Canada’s current Funding Agreement with industry Canada specifies that Genome Canada shall carry out an independent third-party evaluation of its “grants to Eligible Projects”, including the activities and projects of Genome Canada every (5) five years. It further states that the evaluation will measure overall performance in achieving the objectives identified in the Funding Agreement. The evaluation will include a review of the following:
7
(a) (b) (c) (d) (e)
output and operations of Genome Centres; the research undertaken into GE3LS; efforts in relation to public communications and outreach; commercialization and corporate development; and overall results achieved.
According to this timetable the next evaluation is due by March 2009. Planning will therefore begin in fiscal year 2007-08. A Steering Committee with representation from Genome Canada’s Board and from Industry Canada, will work with an independent professional firm to undertake the evaluation. The firm will do the evaluation in accordance with an overall framework established by the Committee. It is expected that, in the context of the five areas listed above, the Committee will: re-examine and validate the current RMAF and Logic Model, including a review of currently established indicators and the method of reporting thereon. define the evaluation issues develop the appropriate questions to be answered by the evaluation confirm the data sources to be used by the evaluators
4.
Performance Reporting
Genome Canada’s primary public reporting documents are its Annual Report and Corporate Plan both published annually. The Funding Agreement with Industry Canada prescribes in great detail, the areas to be addressed in each of these documents, many of which relate to performance-type information.
Current Practice
The Funding Agreement specifies that Genome Canada’s Annual Report must have a statement of its objectives for the reporting year and a statement on the extent to which they have been met. It must also report on any evaluations that may have been undertaken during the year reported. In respect to the Corporate Plan, Genome Canada must refer to short and medium term outcomes, successes in respect to previous year’s plans, remaining challenges, anticipated results of planned activities and on-going performance management strategies. Genome Canada has attempted to address these and other performance related issues in the annual documents. In essence however, reporting has been more on process and outputs rather than outcomes and results, which must now become the reporting focus as projects end and evaluations are completed. For example, in 2006-07, Genome Canada commissioned the preparation of ten “success stories” which communicated in simple language, the achievements and outcomes of ten Genome Canada supported research projects.
Planned Initiatives
8
Genome Canada will continue to improve its performance reporting in its Annual Report, Corporate Plan, and other special publications. In addition, as part of a re-design of its web-site, a special area on Performance will be developed to highlight and provide prominence to performance issues. The reporting vehicles referred to above will take performance information from a variety of data sources including: Genome Canada and Genome Centre internal administrative and financial systems Formal evaluations Formal audits Special case studies, evaluations and reviews Review Reports from International Scientific Review Panels Final Project Reports and results of reviews thereon Project self-reported performance metrics (collected by Centres on each project), and Quality reviews published by independent third parties (eg. Review of Foundations by the Treasury Board of Canada, Genomics Review, the State of Science and Technology in Canada by the Council of Canadian Academies, etc.)
5.
Risk Management
Genome Canada’s Performance, Audit and Evaluation Strategy is supported by an Integrated Risk Management (IRM) Policy, a framework which outlines the organization’s approach to the management of risk. The policy includes methodology for risk identification, risk profiles, and related mitigating actions.
Current Practice
Risk management is integrated into all operational, managerial and governance activities of Genome Canada. It considers external strategic risks arising from the external operating environment as well as other internal operational risks. At the project selection level, the process is totally risk based, a process that ensures that only those projects judged to have the greatest probability of success from both a scientific and managerial point of view, are funded. At the operational level, officers identify risks and propose strategies for mitigating and reporting (eg, due diligence routines for review of draw requests and for interim reviews); At the managerial level, policies, systems, processes and procedures (e.g. administrative, financial, human resource management, etc.) are developed, implemented and monitored; At the Audit Committee level, the committee reviews and approves Genome Canada’s Risk Profile and mitigation strategies on at least an annual basis; At the governance level, the Board and its Committees are aware of its risk management responsibilities and exercise modern governance practices with respect to policy approval and oversight (eg, the Governance Committee of the Board monitors and oversees a comprehensive governance manual); and The internal environment is one that values honesty, integrity and ethical conduct.
-
-
-
-
Planned Initiatives
9
In the next year, Genome Canada will: Revalidate the Integrated Risk Management Policy to ensure that its principles are still valid in the context of the current internal and external environment. Through the Board’s Audit Committee, continue to detail and refine the organization’s risk profile and mitigation strategies.
-
Work with the Genome Centres to ensure that a risk management framework and supporting processes are in place for the management of risk at each Centre.
D.
Conclusion
The Performance, Audit and Evaluation Strategy described herein will be used by Genome Canada to consolidate and enhance its effectiveness in these important areas of accountability. Continuous improvement of existing policies and processes, and the implementation of new initiatives in the area of performance, audit, and evaluation, have as their objective, the provision of a more rigorous and effective approach to the measurement and reporting of outcomes and results.
List of Appendices
10
Appendix 1 Appendix 2 Appendix 3 Appendix 4 Appendix 5 -
Definitions Summary of Planned Activities Performance Indicators Related Documents References to Performance, Audit and Evaluation in the Funding Agreement with Industry Canada
Appendix 1
11
Definitions
Listed below are definitions of terms used in this document taken primarily from Treasury Board of Canada publications and those of other organizations including the Office of the Auditor General. Risk: The uncertainty of an event occurring that could have an impact on the achievement of objectives. Risk is measured in terms of likelihood and impact. Financial Audit: An audit of the financial statements of a company or other legal entity conducted by an independent external third party (typically by a firm of practicing public accountants) resulting in the publication of an independent opinion on whether or not those financial statements are relevant, accurate, complete, and fairly presented. The auditor's opinion is, by definition, the auditor's report on whether the financial statements are fairly presented and comply with generally accepted accounting principles. Performance Audit (Previously referred to as Value for Money Audit): A performance audit is a systematic, purposeful, organized and objective examination of the activities of an organizational entity, including the examination of economy, efficiency and cost-effectiveness; procedures to measure effectiveness; accountability relationships; protection of organizational assets, and; compliance with authorities. Compliance Audit: A compliance audit tests whether legal or statutory obligations have been complied with (rather than to test whether an entity is being managed efficiently, effectively and economically). Recipient Audit: An audit of a (Genome Canada) funded project conducted in accordance with Genome Canada’s Recipient Audit Framework. Performance: Actual achievements measured against defined goals, standards or criteria. Performance measurement: The ongoing monitoring of the results of a program, project, policy or initiative, and in particular, progress towards pre-established goals. Results: Relate to what was achieved. They are the collection of impacts and outcomes associated with a program, project, policy or initiative. Results-Based Management and Accountability Framework: An articulation of objectives, activities, outputs and key results to be achieved, by a program or initiative, along with any related linkages. It generally also includes a performance measurement strategy and performance information and indicators. Evaluation: A formalized approach to studying and assessing projects, policies, programs, organizations and initiatives determining if they 'work'. Program evaluations can involve both quantitative and qualitative methods and social research.
Appendix 2
12
Summary of Planned Activities
Performance Monitoring and Measurement Framework
Implement a framework and process for external reviews of Final Project Reports Validate review criteria against international methodologies Implement national system for data collection of performance information and metrics FY 07-08 FY 07-08 / 08-09 FY 08-09
Audit Framework
Undertake a Performance Audit (Value for Money Audit) of Genome Canada Implement a process for conducting compliance audits of funding agreements with Genome Centres FY 08-09
FY 08-09
Evaluation Framework
Undertake an evaluation of Genome Canada activities Review and update Genome Canada’s Results Based Management Accountability Framework (RMAF), including currently established performance indicators FY 08-09
FY 07-08 / 08-09
Performance Reporting
Continuous improvement of performance reporting in Annual Report and Corporate Plan Annually
Risk Management
Undertake a review of the current Integrated Risk Management Policy Continuous refinement of the Risk Profile and mitigation strategies Ensure and adequate Risk Management Framework is in place in each Genome Centre FY 07-08 FY 08-09
FY 08-09
Appendix 3
13
Performance Indicators
Performance Data Performance Metrics
Scientific achievements • • • • • • • • • • • • • • • • • • Scientific (peer reviewed) publications Citations Awards Number of presentations at major international conferences/congresses New database depositions Innovations brought to market Benefits to Canada New or improved products, services and processes Invention disclosures Patents filed Patents awarded Licenses granted (out) Revenues generated from products, services, license fees New companies created Name of collaborators Nature of collaborator (university, hospital, research organization, federal department, provincial department, international, industry) Country Mix of personnel (investigator, research assistant, technical personnel, postdoctoral fellow, graduate student, bioinformatician, programmer, project manager/coordinator, undergraduate/co-op student, administrative staff, consultant) and new vs. existing
Innovation and socio-economic metrics
Major collaborations
Jobs created/maintained and individuals trained
From the mix of personnel listed above, how many are: • new recruits from outside Canada • Researchers trained • Highly Qualified Personnel Mechanism to track career paths of trained support. Co-funding • Amount • Nature (cash, in-kind) • Source • Nature of source (province, federal, foreign, industry/private, NPO) • Status Mechanism to capture leveraged funding • Press releases issued • # of events or coverage – news stories in media (print. radio. TV) • Public forums / events Number and type of users: • Canadian research projects (GC funded versus others) • Canadian companies • Foreign research projects • Foreign companies Other data: • Gross recoveries • % capacity used • Units (e.g., sequence reads, samples processed, hours consulted)
Investment from others
Communications
Platform performance
Appendix 4
14
Related documents
Genome Canada has many reports and has developed and implemented numerous policies, frameworks and guidelines which relate directly to performance, audit and evaluation.
FRAMEWORKS AND GUIDELINES
Competition Guidelines and Evaluation Criteria for applicants – prepared for each competition Evaluation Criteria used by International Review Committees – prepared for each competition Interim Review Guidelines and Evaluation Criteria – prepared for each competition Performance Indicators (2003) Genome Canada’s Risk Management Framework - approved by Audit Committee (2004) Organizational Risk Profile - reviewed annually by the Audit Committee. Governance Manual – updated as required and monitored by the Board’s Governance Committee. Results-based Management and Accountability Framework (RMAF) and Logic Model (2003) Recipient Audit Framework and Risk Assessment Tool - used by Genome Centres to select individual research projects to be audited (2006)
REPORTS
Reports from International Review Panels – one set for each Competition Reports from International Scientific Review Committees for each Interim Review- one set for each Interim Review Final Project Reports submitted by Project Leaders in Competition 1 and 2 at end of funding period detailing project achievements (2005) Annual External Financial (Attest) Audits and Auditor’s Reports of the Financial Statements (Annual) Recipient Audit Reports, commissioned by Genome Centres in respect to each recipient audit (Annual) Compliance Audit, which assessed degree of compliance by Genome Canada to Funding Agreement with Industry Canada (2006) Interim Evaluation of Genome Canada which assessed whether Genome Canada was on track in meeting its objectives (2004) Corporate Plan addressing objectives and plans for up-coming year (annual) Annual Report published addressing performance and activities during preceding year (annual). Special Publications that address specific issues (as required, i.e. Success Stories)
Appendix 5
15
Funding Agreement Dated March 2007 References to Performance, Audit, and Evaluation
8.1 Ensure that results expected, performance expectations and indicators are measurable; Put in place evaluation, audit, accountability and reporting requirements, including the provision of information necessary to enable Genome Canada to assess the ongoing performance of the Genome Centre and to determine from time to time whether the funding to the Genome Centre should be continued; Evaluation Framework. In consultation with Industry Canada, Genome Canada shall carry out an independent third-party evaluation of its grants to Eligible Projects, including the activities and projects of Genome Canada, according to a framework and time frame approved by the Board, but to be completed no less frequently than once every (5) five years. The evaluation will measure its overall performance in achieving the Objectives identified in the Funding Agreement. The evaluation report shall be made public and a copy shall be sent to the Minister. The evaluation will include a review of the following: (f) output and operations of Genome Centres; (g) the research undertaken into GE3LS; (h) efforts in relation to public communications and outreach; (i) commercialization and corporate development; and (j) overall results achieved. Genome Canada, in consultation with Industry Canada will develop a Performance, Audit and Evaluation Strategy, in accordance with recognized standards, that will be approved by its Board and provided for review and approval of the Minister within 180 days of the signing of this Agreement, and updated thereafter as required. Audit for Compliance. Each of the Minister and the Auditor General may require that an audit of the books and records of Genome Canada be carried out, at their respective expense, by such person as the Minister or the Auditor General may appoint to ensure compliance with this Funding Agreement. Genome Canada will cooperate and provide access to the appropriate records to conduct such an audit. The Minister and the Auditor General, as the case may be, may share a copy of the resulting report with Genome Canada and agrees to discuss any concerns raised in the audit with Genome Canada. Evaluation Requested by the Minister. The Minister may, after consultation with Genome Canada, choose to conduct his own evaluation of the Funding Agreement as an instrument of policy of the Government of Canada, at his own cost and by an evaluator of his own choosing. Genome Canada will cooperate and provide access to the appropriate records to conduct such an evaluation. The Minister may share a copy of the resulting report with Genome Canada and agrees to discuss any concerns raised in the evaluation with Genome Canada. Performance (value-for-money) Audit. Each of the Minister and the Auditor General, after consultation with Genome Canada, elect that a performance (value-for-money) audit of Genome Canada be carried out, at their respective expense, by such person as the Minister or the Auditor General may appoint, to ensure the economy, efficiency and effectiveness with which the Amount has been used. Genome Canada will cooperate and provide access to the appropriate records to conduct such an audit. The Minister and the Auditor General, as the case may be, may share a copy of the resulting report with Genome Canada and agrees to discuss any concerns raised in the audit with Genome Canada.
Performance Audit: Genome Canada shall cause a performance (value-for-money) audit to be done, at its own expense, by an independent third-party by March 1, 2009, to ensure the economy, efficiency and effectiveness with which funds have been used. The report shall be made public and a copy shall be sent to the Minister. The Minister may cause the copy of the report to be laid before each House of Parliament on any of the first fifteen (15) days which the House is sitting after the Minister receives it.
8.2
8.4
8.5
8.6.1
8.6.2
16
doc_591099383.pdf
Strategic management analyzes the major initiatives taken by a company's top management on behalf of owners, involving resources and performance in internal and external environments.
Performance, Audit and Evaluation Strategy
Genome Canada
V.6.0
November 2007
PERFORMANCE, AUDIT AND EVALUATION STRATEGY
_______________________________________________________
A.
Introduction
The purpose of this document is to present Genome Canada’s Performance, Audit and Evaluation Strategy (PAES). The strategy has been developed as a high level framework which addresses key elements that Genome Canada has or will put in place to ensure accountability in the achievement of objectives from the perspective of performance, audit, evaluation and reporting. The principles and directions outlined in the PAES will serve as a guide to the organization for the continuation, initiation and implementation of activities in these areas starting in fiscal year 2007-08.
B.
Background
Genome Canada currently has in place a wide array of policies, systems and processes that were developed over time to address activities related to performance, audit, evaluation and risk. They were also developed to ensure compliance to the funding agreements signed with Industry Canada with respect to the use and accounting of funds received from the federal government. Genome Canada also signs individual funding agreements with each of the six Genome Centres where the undertakings agreed to with Industry Canada are essentially replicated. The most recent funding agreement between Genome Canada and Industry Canada includes a provision which calls for these activities to be consolidated into one overarching strategy. The strategy would receive approval by the Board of Genome Canada and by the Minister of Industry Canada. Genome Canada, in consultation with Industry Canada will develop a Performance, Audit and Evaluation Strategy, in accordance with recognized standards, that will be approved by its Board and provided for review and approval of the Minister within 180 days of the signing of this Agreement, and updated thereafter as required. (Article 8.2 of the Funding Agreement with Industry Canada, dated March 29, 2007) The PAES satisfies this requirement by presenting an integrated view to the implementation of initiatives in the required areas.
C.
Elements of the Strategy
2
The PAES is comprised of three key elements and two supporting elements each of which are integral to the strategy. The elements are discussed below in the context of activities and initiatives currently in place, and those planned for implementation over the next two years. The three key elements are: • Performance Monitoring and Measurement Framework • Audit Framework • Evaluation Framework The two supporting elements are: • Performance Reporting • Risk Management Each of the elements and their underlying components are displayed in the figure below.
PERFORMANCE REPORTING Corporate Plan Annual Report Special Publications Web Site
PERFORMANCE MONITORING
AUDIT
EVALUATION
Project Selection ** Project Monitoring Interim Review Performance Indicators Final Project Reports
Financial Audits Recipient Audits Compliance Audits Performance Audits
Results Based Management And Accountability Framework (RMAF) Logic Model Evaluations Every Five Years
RISK MANAGEMENT Risk Management Framework Operations and Management Governance Regime
** For purpose of this document, the term “Project” means both Research Projects and Science and Technology
Platforms.
1.
Performance Monitoring and Measurement Framework
3
Performance monitoring and measurement starts at the project selection phase, continues through the execution phase with effective project management and monitoring (including regular data collection by Genome Centres on established performance metrics, and a formal interim review by international experts), and ends with the assessment of final project reports.
Current Practice
Project Selection
a. An independent International Science Review Panel, supported by external reviews of the science by peers, and information taken from an extensive due diligence process of financial and management aspects, assess and rank each of the proposals based on evaluation criteria as described in Competition Guidelines, and make recommendations for the approval of the Board of Directors.
Project Management and Monitoring
a. All projects are “managed” through the Genome Centres by Program Managers employed by the Centres who are responsible for on-going monitoring of project performance, active liaison with project leaders, managers and other members of the research team, and assessment of quarterly project reports. b. Projects report on progress quarterly (on milestones and performance metrics) to Genome Centres allowing for continuous hands on monitoring and opportunities for interventions when appropriate. c. Based on financial reports received from projects, Centres submit a “Quarterly Draw Request” to Genome Canada for estimated cash requirements of projects in the subsequent quarter. The request is based on comparison of previous advances to funds expended and forecast of cash needs for next quarter. Genome Canada’s Finance Group also monitors co-funding that is recorded by Centres on a Genome Canada data base, and conducts regular site visits to Genome Centres to assess the efficacy of processes related to financial management of projects.
d
Performance Indicators
a Genome Centres collect performance information on a regular basis (quarterly) based on a list of indicators as shown in Appendix 1.
b. Currently, these data are collected independently by each Centre and there is no national repository which is easily accessible by Genome Canada. Integration of this information on a national basis into Genome Canada’s data base is a planned initiative in the coming year.
Interim Review
4
a. After approximately 18 months of funding for each project, a formal interim review is undertaken (again by an International Science Review Committee) to assess progress to date against approved milestones and outcomes. b. Recommendations to the Board address whether projects should proceed as proposed, if they must respond to specific issues prior to continuation, or whether a site visit should be undertaken to review significant issues first hand.
Final Reports
a. Upon completion of each project, Genome Canada requires a Final Report from the Project Leaders. b Final Reports address project outputs/outcomes relative to approved objectives in several categories including: research, technology transfer, training and recruitment, benefits to Canada, outcomes of public outreach, international linkages, outcomes of GELS activities, and a final financial accounting.
c. To date, Genome Canada has received Final Reports from Competition I and II projects, the only projects which have been completed so far. The reports have been used primarily to collect data and metrics such as numbers of publications, patents, people trained, etc., which are summarized in public reporting of performance information (i.e.in the Annual Report).This information will also serve as an important data source in the overall evaluation of Genome Canada.
Planned Initiatives
To enhance its performance monitoring, and to better evaluate the results of its research investments, Genome Canada will: . a. Implement a process whereby Final Reports for projects originating from specific competitions will undergo an evaluative external review. The objective of the review will be to assess and report on the overall impact and outcomes of each project individually, and of the competition as a whole, based on pre-determined criteria and methodologies. b. Validate our criteria against internationally accepted methodologies to improve the process for measurement of economic impact of investments in genomics and proteomics. c. Implement a process whereby Centres record performance metrics as listed in Appendix 1 in Genome Canada’s data base on a continuous basis.
2.
Audit Framework
5
Genome Canada’s audit activities are comprised of a number of components that together provide a comprehensive framework for the external examination of the organization’s policies, programs, systems and procedures.
Current Practice
Annual Financial Audits
Genome Canada’s financial statements are audited annually as required by the Canada Corporations Act, the Funding Agreement with Industry Canada, and normal business practices. An external public accounting firm performs the audit and provides an opinion as to whether Genome Canada’s financial statements present fairly, in all material respects, the financial position of the Corporation as at fiscal year end, and the results of operation for the year then ended, in accordance with generally accepted accounting principles.
Recipient Audits
Recipient audits are those conducted on the recipients of Genome Canada funding. The concept of recipient audits was introduced in fiscal year 2006-07 when the Board, upon recommendation of the Audit Committee, approved Genome Canada’s Recipient Audit Framework and Risk Assessment Tool. Fundamental to the Framework is the concept of a risk based approach to the selection of projects to be audited as well as prescribed audit criteria as specified by the Canadian Institute of Chartered Accountants. Genome Centres are responsible for implementing the framework and for initiating and managing the audits.
Compliance Audits
Compliance audits are those that assess compliance with the terms and conditions of an agreement. As per the Funding Agreement with Industry Canada, both the Minister and the Auditor General have the right to initiate a compliance audit of Genome Canada. Only one compliance audit to date has been carried and it was done by Industry Canada in 2006.
Performance Audits
Performance Audits were previously referred to as Value For Money Audits (VFM). As per the Funding Agreement with Industry Canada, both the Minister and the Auditor General have the right to initiate a performance audit of Genome Canada to ensure the economy, efficiency and effectiveness with which federal funds have been used.
Planned Initiatives
Two additional activities are planned for implementation over the next two years. a. With respect to Performance Audits, the current Funding Agreement (unlike previous agreements) requires that Genome Canada cause a Performance Audit to be conducted on itself by an independent third-party by March 1, 2009, to ensure the economy, efficiency and effectiveness of funds used. Genome Canada plans to have this audit undertaken by a firm of professional accountants. The methodology of the Office of the Auditor General of Canada will be
6
used to ensure it is conducted in accordance with generally accepted auditing principles in this area. Work will be conducted essentially in three phases: Planning Phase: In this phase, the audit team sets out to understand the organization, the applicable authorities, the way the organization is structured to achieve its mandate, and the challenges it faces in the environment in which it operates. They then develop the criteria against which the audited activities can be assessed Examination phase: In this phase, the audit team gathers evidence (eg,through interviews with stakeholders, review of business documents and records, etc.) to allow it to report on the strengths and weaknesses of the organization being audited. Reporting Phase: In this phase the auditors draft the report that will eventually be presented to the Audit Committee and Board. After validation of facts and findings with management, the final report to the Board will detail their audit findings (what is working well and what needs improvement) and provide recommendations for improvements to operations and performance. The Audit Committee will work with an independent professional firm to ensure effective oversight of the Performance Audit. Development of the audit criteria will be a critical task of the Audit Committee. b. With respect to Compliance Audits, Genome Canada has undergone one compliance audit from Industry Canada. However no such audits have yet to be done on Genome Centres. Within the the next two years, Genome Canada will undertake Compliance Audits of the Centres (compliance to the funding agreements) to provide further assurance of effective management and use of funds that are flowed to them.
3.
Evaluation Framework
Genome Canada’s program evaluation strategy is taken from its Results Based Management and Accountability Framework (RMAF) and Logic Model which was developed in 2003. The RMAF displays Genome Canada’s results expectations – from activities to outcomes – and forms the basis of its evaluation initiatives. Standards in respect to evaluations are taken primarily from Treasury Board policies and guidelines.
Current Practice
Genome Canada has had one formal evaluation of its activities conducted and reported on in 2006. This “Interim” Evaluation was essentially an overview of whether Genome Canada was on track towards meeting its nine objectives (generally, it was). The results of the evaluation are posted on Genome Canada’s web site. No other formal program evaluations have been conducted.
Planned Initiatives
Genome Canada’s current Funding Agreement with industry Canada specifies that Genome Canada shall carry out an independent third-party evaluation of its “grants to Eligible Projects”, including the activities and projects of Genome Canada every (5) five years. It further states that the evaluation will measure overall performance in achieving the objectives identified in the Funding Agreement. The evaluation will include a review of the following:
7
(a) (b) (c) (d) (e)
output and operations of Genome Centres; the research undertaken into GE3LS; efforts in relation to public communications and outreach; commercialization and corporate development; and overall results achieved.
According to this timetable the next evaluation is due by March 2009. Planning will therefore begin in fiscal year 2007-08. A Steering Committee with representation from Genome Canada’s Board and from Industry Canada, will work with an independent professional firm to undertake the evaluation. The firm will do the evaluation in accordance with an overall framework established by the Committee. It is expected that, in the context of the five areas listed above, the Committee will: re-examine and validate the current RMAF and Logic Model, including a review of currently established indicators and the method of reporting thereon. define the evaluation issues develop the appropriate questions to be answered by the evaluation confirm the data sources to be used by the evaluators
4.
Performance Reporting
Genome Canada’s primary public reporting documents are its Annual Report and Corporate Plan both published annually. The Funding Agreement with Industry Canada prescribes in great detail, the areas to be addressed in each of these documents, many of which relate to performance-type information.
Current Practice
The Funding Agreement specifies that Genome Canada’s Annual Report must have a statement of its objectives for the reporting year and a statement on the extent to which they have been met. It must also report on any evaluations that may have been undertaken during the year reported. In respect to the Corporate Plan, Genome Canada must refer to short and medium term outcomes, successes in respect to previous year’s plans, remaining challenges, anticipated results of planned activities and on-going performance management strategies. Genome Canada has attempted to address these and other performance related issues in the annual documents. In essence however, reporting has been more on process and outputs rather than outcomes and results, which must now become the reporting focus as projects end and evaluations are completed. For example, in 2006-07, Genome Canada commissioned the preparation of ten “success stories” which communicated in simple language, the achievements and outcomes of ten Genome Canada supported research projects.
Planned Initiatives
8
Genome Canada will continue to improve its performance reporting in its Annual Report, Corporate Plan, and other special publications. In addition, as part of a re-design of its web-site, a special area on Performance will be developed to highlight and provide prominence to performance issues. The reporting vehicles referred to above will take performance information from a variety of data sources including: Genome Canada and Genome Centre internal administrative and financial systems Formal evaluations Formal audits Special case studies, evaluations and reviews Review Reports from International Scientific Review Panels Final Project Reports and results of reviews thereon Project self-reported performance metrics (collected by Centres on each project), and Quality reviews published by independent third parties (eg. Review of Foundations by the Treasury Board of Canada, Genomics Review, the State of Science and Technology in Canada by the Council of Canadian Academies, etc.)
5.
Risk Management
Genome Canada’s Performance, Audit and Evaluation Strategy is supported by an Integrated Risk Management (IRM) Policy, a framework which outlines the organization’s approach to the management of risk. The policy includes methodology for risk identification, risk profiles, and related mitigating actions.
Current Practice
Risk management is integrated into all operational, managerial and governance activities of Genome Canada. It considers external strategic risks arising from the external operating environment as well as other internal operational risks. At the project selection level, the process is totally risk based, a process that ensures that only those projects judged to have the greatest probability of success from both a scientific and managerial point of view, are funded. At the operational level, officers identify risks and propose strategies for mitigating and reporting (eg, due diligence routines for review of draw requests and for interim reviews); At the managerial level, policies, systems, processes and procedures (e.g. administrative, financial, human resource management, etc.) are developed, implemented and monitored; At the Audit Committee level, the committee reviews and approves Genome Canada’s Risk Profile and mitigation strategies on at least an annual basis; At the governance level, the Board and its Committees are aware of its risk management responsibilities and exercise modern governance practices with respect to policy approval and oversight (eg, the Governance Committee of the Board monitors and oversees a comprehensive governance manual); and The internal environment is one that values honesty, integrity and ethical conduct.
-
-
-
-
Planned Initiatives
9
In the next year, Genome Canada will: Revalidate the Integrated Risk Management Policy to ensure that its principles are still valid in the context of the current internal and external environment. Through the Board’s Audit Committee, continue to detail and refine the organization’s risk profile and mitigation strategies.
-
Work with the Genome Centres to ensure that a risk management framework and supporting processes are in place for the management of risk at each Centre.
D.
Conclusion
The Performance, Audit and Evaluation Strategy described herein will be used by Genome Canada to consolidate and enhance its effectiveness in these important areas of accountability. Continuous improvement of existing policies and processes, and the implementation of new initiatives in the area of performance, audit, and evaluation, have as their objective, the provision of a more rigorous and effective approach to the measurement and reporting of outcomes and results.
List of Appendices
10
Appendix 1 Appendix 2 Appendix 3 Appendix 4 Appendix 5 -
Definitions Summary of Planned Activities Performance Indicators Related Documents References to Performance, Audit and Evaluation in the Funding Agreement with Industry Canada
Appendix 1
11
Definitions
Listed below are definitions of terms used in this document taken primarily from Treasury Board of Canada publications and those of other organizations including the Office of the Auditor General. Risk: The uncertainty of an event occurring that could have an impact on the achievement of objectives. Risk is measured in terms of likelihood and impact. Financial Audit: An audit of the financial statements of a company or other legal entity conducted by an independent external third party (typically by a firm of practicing public accountants) resulting in the publication of an independent opinion on whether or not those financial statements are relevant, accurate, complete, and fairly presented. The auditor's opinion is, by definition, the auditor's report on whether the financial statements are fairly presented and comply with generally accepted accounting principles. Performance Audit (Previously referred to as Value for Money Audit): A performance audit is a systematic, purposeful, organized and objective examination of the activities of an organizational entity, including the examination of economy, efficiency and cost-effectiveness; procedures to measure effectiveness; accountability relationships; protection of organizational assets, and; compliance with authorities. Compliance Audit: A compliance audit tests whether legal or statutory obligations have been complied with (rather than to test whether an entity is being managed efficiently, effectively and economically). Recipient Audit: An audit of a (Genome Canada) funded project conducted in accordance with Genome Canada’s Recipient Audit Framework. Performance: Actual achievements measured against defined goals, standards or criteria. Performance measurement: The ongoing monitoring of the results of a program, project, policy or initiative, and in particular, progress towards pre-established goals. Results: Relate to what was achieved. They are the collection of impacts and outcomes associated with a program, project, policy or initiative. Results-Based Management and Accountability Framework: An articulation of objectives, activities, outputs and key results to be achieved, by a program or initiative, along with any related linkages. It generally also includes a performance measurement strategy and performance information and indicators. Evaluation: A formalized approach to studying and assessing projects, policies, programs, organizations and initiatives determining if they 'work'. Program evaluations can involve both quantitative and qualitative methods and social research.
Appendix 2
12
Summary of Planned Activities
Performance Monitoring and Measurement Framework
Implement a framework and process for external reviews of Final Project Reports Validate review criteria against international methodologies Implement national system for data collection of performance information and metrics FY 07-08 FY 07-08 / 08-09 FY 08-09
Audit Framework
Undertake a Performance Audit (Value for Money Audit) of Genome Canada Implement a process for conducting compliance audits of funding agreements with Genome Centres FY 08-09
FY 08-09
Evaluation Framework
Undertake an evaluation of Genome Canada activities Review and update Genome Canada’s Results Based Management Accountability Framework (RMAF), including currently established performance indicators FY 08-09
FY 07-08 / 08-09
Performance Reporting
Continuous improvement of performance reporting in Annual Report and Corporate Plan Annually
Risk Management
Undertake a review of the current Integrated Risk Management Policy Continuous refinement of the Risk Profile and mitigation strategies Ensure and adequate Risk Management Framework is in place in each Genome Centre FY 07-08 FY 08-09
FY 08-09
Appendix 3
13
Performance Indicators
Performance Data Performance Metrics
Scientific achievements • • • • • • • • • • • • • • • • • • Scientific (peer reviewed) publications Citations Awards Number of presentations at major international conferences/congresses New database depositions Innovations brought to market Benefits to Canada New or improved products, services and processes Invention disclosures Patents filed Patents awarded Licenses granted (out) Revenues generated from products, services, license fees New companies created Name of collaborators Nature of collaborator (university, hospital, research organization, federal department, provincial department, international, industry) Country Mix of personnel (investigator, research assistant, technical personnel, postdoctoral fellow, graduate student, bioinformatician, programmer, project manager/coordinator, undergraduate/co-op student, administrative staff, consultant) and new vs. existing
Innovation and socio-economic metrics
Major collaborations
Jobs created/maintained and individuals trained
From the mix of personnel listed above, how many are: • new recruits from outside Canada • Researchers trained • Highly Qualified Personnel Mechanism to track career paths of trained support. Co-funding • Amount • Nature (cash, in-kind) • Source • Nature of source (province, federal, foreign, industry/private, NPO) • Status Mechanism to capture leveraged funding • Press releases issued • # of events or coverage – news stories in media (print. radio. TV) • Public forums / events Number and type of users: • Canadian research projects (GC funded versus others) • Canadian companies • Foreign research projects • Foreign companies Other data: • Gross recoveries • % capacity used • Units (e.g., sequence reads, samples processed, hours consulted)
Investment from others
Communications
Platform performance
Appendix 4
14
Related documents
Genome Canada has many reports and has developed and implemented numerous policies, frameworks and guidelines which relate directly to performance, audit and evaluation.
FRAMEWORKS AND GUIDELINES
Competition Guidelines and Evaluation Criteria for applicants – prepared for each competition Evaluation Criteria used by International Review Committees – prepared for each competition Interim Review Guidelines and Evaluation Criteria – prepared for each competition Performance Indicators (2003) Genome Canada’s Risk Management Framework - approved by Audit Committee (2004) Organizational Risk Profile - reviewed annually by the Audit Committee. Governance Manual – updated as required and monitored by the Board’s Governance Committee. Results-based Management and Accountability Framework (RMAF) and Logic Model (2003) Recipient Audit Framework and Risk Assessment Tool - used by Genome Centres to select individual research projects to be audited (2006)
REPORTS
Reports from International Review Panels – one set for each Competition Reports from International Scientific Review Committees for each Interim Review- one set for each Interim Review Final Project Reports submitted by Project Leaders in Competition 1 and 2 at end of funding period detailing project achievements (2005) Annual External Financial (Attest) Audits and Auditor’s Reports of the Financial Statements (Annual) Recipient Audit Reports, commissioned by Genome Centres in respect to each recipient audit (Annual) Compliance Audit, which assessed degree of compliance by Genome Canada to Funding Agreement with Industry Canada (2006) Interim Evaluation of Genome Canada which assessed whether Genome Canada was on track in meeting its objectives (2004) Corporate Plan addressing objectives and plans for up-coming year (annual) Annual Report published addressing performance and activities during preceding year (annual). Special Publications that address specific issues (as required, i.e. Success Stories)
Appendix 5
15
Funding Agreement Dated March 2007 References to Performance, Audit, and Evaluation
8.1 Ensure that results expected, performance expectations and indicators are measurable; Put in place evaluation, audit, accountability and reporting requirements, including the provision of information necessary to enable Genome Canada to assess the ongoing performance of the Genome Centre and to determine from time to time whether the funding to the Genome Centre should be continued; Evaluation Framework. In consultation with Industry Canada, Genome Canada shall carry out an independent third-party evaluation of its grants to Eligible Projects, including the activities and projects of Genome Canada, according to a framework and time frame approved by the Board, but to be completed no less frequently than once every (5) five years. The evaluation will measure its overall performance in achieving the Objectives identified in the Funding Agreement. The evaluation report shall be made public and a copy shall be sent to the Minister. The evaluation will include a review of the following: (f) output and operations of Genome Centres; (g) the research undertaken into GE3LS; (h) efforts in relation to public communications and outreach; (i) commercialization and corporate development; and (j) overall results achieved. Genome Canada, in consultation with Industry Canada will develop a Performance, Audit and Evaluation Strategy, in accordance with recognized standards, that will be approved by its Board and provided for review and approval of the Minister within 180 days of the signing of this Agreement, and updated thereafter as required. Audit for Compliance. Each of the Minister and the Auditor General may require that an audit of the books and records of Genome Canada be carried out, at their respective expense, by such person as the Minister or the Auditor General may appoint to ensure compliance with this Funding Agreement. Genome Canada will cooperate and provide access to the appropriate records to conduct such an audit. The Minister and the Auditor General, as the case may be, may share a copy of the resulting report with Genome Canada and agrees to discuss any concerns raised in the audit with Genome Canada. Evaluation Requested by the Minister. The Minister may, after consultation with Genome Canada, choose to conduct his own evaluation of the Funding Agreement as an instrument of policy of the Government of Canada, at his own cost and by an evaluator of his own choosing. Genome Canada will cooperate and provide access to the appropriate records to conduct such an evaluation. The Minister may share a copy of the resulting report with Genome Canada and agrees to discuss any concerns raised in the evaluation with Genome Canada. Performance (value-for-money) Audit. Each of the Minister and the Auditor General, after consultation with Genome Canada, elect that a performance (value-for-money) audit of Genome Canada be carried out, at their respective expense, by such person as the Minister or the Auditor General may appoint, to ensure the economy, efficiency and effectiveness with which the Amount has been used. Genome Canada will cooperate and provide access to the appropriate records to conduct such an audit. The Minister and the Auditor General, as the case may be, may share a copy of the resulting report with Genome Canada and agrees to discuss any concerns raised in the audit with Genome Canada.
Performance Audit: Genome Canada shall cause a performance (value-for-money) audit to be done, at its own expense, by an independent third-party by March 1, 2009, to ensure the economy, efficiency and effectiveness with which funds have been used. The report shall be made public and a copy shall be sent to the Minister. The Minister may cause the copy of the report to be laid before each House of Parliament on any of the first fifteen (15) days which the House is sitting after the Minister receives it.
8.2
8.4
8.5
8.6.1
8.6.2
16
doc_591099383.pdf