Description
Intellectual property (IP) is a legal concept which refers to creations of the mind for which exclusive rights are recognized.
E
CDIP/10/6 ORIGINAL: ENGLISH DATE: SEPTEMBER 28, 2012
Committee on Development and Intellectual Property (CDIP)
Tenth Session Geneva, November 12 to 16, 2012
EVALUATION REPORT OF THE PROJECT ON DEVELOPING TOOLS FOR ACCESS TO PATENT INFORMATION
prepared by the Secretariat
1. The Annex to this document contains an independent evaluation report of the project on Developing Tools for Access to Patent Information undertaken by WIPO’s Internal Audit and Oversight Division. 2. The CDIP is invited to take note of the information contained in the Annex to this document.
[Annex follows]
CDIP/10/6 ANNEX
TABLE OF CONTENT EXECUTIVE SUMMARY.............................................................................................................. 3 MAIN CONCLUSIONS ............................................................................................................. 3 RECOMMENDATIONS ............................................................................................................ 5 I. BACKGROUND .................................................................................................................... 7 a. PROJECT BACKGROUND AND DESCRIPTION ............................................................ 7 B. purpose, methodology and limitations of this evaluation .................................................. 7 (i) Key purpose ................................................................................................................. 8 (ii) Main limitations to this evaluation ................................................................................. 8 II. FINDINGS AND ASSESSMENT ...................................................................................... 9 A. PROJECT PREPARATION AND MANAGEMENT........................................................... 9 (i) Project identification ..................................................................................................... 9 (ii) Use of project planning tools ...................................................................................... 10 (iii) Project Management............................................................................................... 11 B. RELEVANCE.................................................................................................................. 11 (i) Policy relevance ......................................................................................................... 11 (ii) Relevance to target groups ........................................................................................ 12 C. EFFECTIVENESS .......................................................................................................... 12 (I) Output 1: PLRs accessible on PATENTSCOPE® Website ....................................... 12 (II) Output 2: E-tutorial .................................................................................................... 14 (iii) Output 3: Regional conferences ............................................................................ 15 (iv) No unexpected negative effects found ................................................................... 16 B. Efficiency ........................................................................................................................ 16 e Likelihood of Sustainability of Results ............................................................................ 17 iii. CONCLUSIONS ............................................................................................................. 19 IV. Recommendations.......................................................................................................... 20
CDIP/10/6 Annex, page 2 LIST OF ACRONYMS AATF ARIPO CDIP CERN DA DAC DNDi FAO GIWEH IAOD IGO IP IRENA ISF ITU LDCs MDGs NGO OECD PLR R&D SMART TISC ToR UN UNEP WIPO WMO
The African Agricultural Technology Foundation African Regional Intellectual Property Organization Committee on Development and Intellectual Property The European Organization for Nuclear Research Development Agenda Development Assistance Committee (of the OECD) The Drugs for Neglected Diseases Initiative Food and Agriculture Organization The Global Institute for Water, Environment and Health WIPO’s Internal Audit and Oversight Division Inter-governmental organization Intellectual property International Renewable Energy Agency The International Seed Federation The International Telecommunication Union Least Developed Countries Millennium Development Goals Non-governmental organization Organization for Economic Co-operation and Development Patent Landscape Report Research and Development Specific, measurable, achievable, relevant and time-bound (indicators) Technology Information and Support Centre Terms of Reference United Nations United Nations Environment Programme World Intellectual Property Organization World Meteorological Organization
CDIP/10/6 Annex, page 3
EXECUTIVE SUMMARY This report covers the independent final evaluation of the Project “Developing Tools for Access to Patent Information” (CDIP 4/6) subsequently referred to as “the Project”. Evaluation work was conducted by the Internal Audit and Oversight Division of the World Intellectual Property Organization (WIPO) according to WIPO’s evaluation guidelines with the assistance of an external consultant. The Project with a total duration of 30 months and a budget of 1,576,000 Swiss francs started in January 2010. It aimed at enhancing access of developing countries to patent information, by publishing patent landscape reports (PLRs), developing an e-tutorial and organising regional conferences. MAIN CONCLUSIONS The findings and assessment of the evaluation resulted in the following conclusions: Conclusion 1: The Project was generally well prepared and managed, but there is room for further enhancing existing tools for planning, monitoring and evaluating projects. The Project made proper use of existing planning and monitoring tools, but there is room to improve the tools in consideration of internationally recognized best practices in managing development assistance. While the project document does not include a logical framework, it contains most of its information, such as clearly defined objectives at different levels linked to objectively verifiable performance indicators. Yet, assumptions on external factors that need to be in place to achieve each of the results and objectives at the impact level are not defined. The involvement of other WIPO sectors and links to WIPO Programmes are well explained, without however specifying concrete actions and responsibilities. Good coordination with other WIPO departments and external partners was more the result of personal initiatives than of a systematically planned approach. Progress reports provide evidence that management actively mitigated most risks identified at the planning stage and additional ones that became apparent during implementation. The most important risk not identified, which subsequently materialized, was a possible lack of inputs from Member States on the topics of PLRs. Unlike for programmes, WIPO does not require result-based financial budgeting and reporting for projects. This Project established for managerial purposes a financial report that links expenditures to budget lines and outputs, which has been identified as a good practice. The Project made significant efforts to use well designed surveys for a systematic monitoring of quality and use of outputs. This was however not possible for users who downloaded PLRs from the dedicated PLR website. The current statistics of WIPO’s Internet Services Sector provide only information related to the number of unique page views, the geographical distribution of the page visitors, along with the number of downloads of each of the PLRs. They do not provide detailed and unambiguous information about who accessed a particular resource and how and for what purpose it was subsequently used. Systematically retrieving contact details about users of on-line services would allow WIPO to profile its clients and to conduct online surveys as a basis for specifically tailoring its services to different target groups. Conclusion 2: The project design was overambitious, especially for achieving the objectives set for the PLRs. The planned time of 30 months to achieve the objectives, in particular producing 12 PLRs, was not commensurate and seems to have been determined by the need to align the project duration to biennial budgeting cycles rather than the time required to deliver the results. The number of PLRs the Project was actually able to publish (see conclusion 3) is evidence of this.
CDIP/10/6 Annex, page 4 Conclusion 3: While the Project generally provided the right type of support in the right way, not all of its expected outputs (PLRs, e-tutorials, regional conferences) were delivered. PLRs: While only seven instead of 12 PLRs as foreseen in the project document were published as of September 15, 2012, users considered them of high quality and generally useful for their work. Two more PLRs have been commissioned and are under preparation. As an additional output, various innovative and useful tools to analyse information of the PLRs and to directly access patent information used in the reports were made available on the Internet. In addition, the Project refined their concept and developed standard procedures and documentation for commissioning work from outside suppliers. Key value added of the Project was the capacity building element through disclosure of the methodology and cooperation with different partners in drafting Terms of Reference (ToR). Evidence provided through interviews suggests that some of the reports were used mainly as an input to the work of other organizations, including in policy making. Due to the lack of direct inputs from Member States, the Project exclusively cooperated with different international, inter-governmental and non-governmental organizations. Topics selected covered areas of high relevance to internationally agreed priorities, including the Millennium Development Goals. Expanding the direct benefits to developing countries would call for a second phase, which would also allow further testing, refining and disseminating of the methodologies. Long-term sustainability will require pursuing the Project’s efforts to prepare for providing PLRs as a regular service of WIPO’s Global Infrastructure Sector. The e-tutorial is of high quality and potential relevance to users, especially to those with a technical background, but no prior knowledge in the field of patents. It complements existing tools, which are either not comprehensive or are oriented towards the use of specific databases. The fact that it became only available after the official ending date of the Project had three main reasons: late start of preparation work, procurement issues (changes in procurement procedures led to delays in awarding the contract), and the complex task of developing an e-tutorial in coordination with various WIPO departments, external content providers and editors, and the application developer. Participants rated the quality of regional conferences as highly relevant to their work. The conferences mainly promoted the Technology and Information Support Centres (TISCs), also supported by project DA_8_01. Furthermore, basic capacity building, mainly to staff of Intellectual Property (IP) Offices, was provided. Conferences were only marginally linked to the e-tutorial and the PLRs. Assessing long-term benefits at the outcome level and their sustainability depends on whether and how the TISCs will use the information provided. Looking at this was neither required nor possible within the scope of this evaluation. While awareness raising and networking is important at the initial stage of technical support, WIPO has rightly recognized the need to also provide training and information exchange through modern online tools (e.g. webinars) to complement more resource-intensive means of skills development such as on-site training.
CDIP/10/6 Annex, page 5 RECOMMENDATIONS Recommendation 1: To Project Managers, the Development Agenda Coordination Division and Resource Planning, Programme Management and Performance Division on project planning (from Conclusions 1 and 2). (a) Wherever possible, project duration should be determined based on a reasonable estimate of the time required to achieve each expected result rather than based on the duration of funding cycles. (b) In addition to defining expected outputs and outcomes and linking them to objectively verifiable indicators (OVIs), as done, longer-term changes the Project is expected to contribute to (impact level) as well as related OVIs should be formalized. (c) Where monitoring and/or self-assessment of results by projects requires significant resources (e.g. comprehensive studies), they need to be included into the budget. (d) As a basis for internal result-based financial monitoring, project documents should include a result-based budget that allocates expenditures according to budget lines (e.g., travel) to each of the expected outcomes and to project management cost (overhead). (e) Besides identifying risks and the way of mitigating them, risks should be rated according to the degree of their potential negative impact on achieving results and the likelihood they materialize. (f) Project documents should include assumptions (external conditions that need to be in place to achieve objectives). (g) Coordination within WIPO and other organizations should be clearly specified (explaining which specific joint actions will be taken and who is responsible). Recommendation 2: To Project Managers and Development Agenda Coordination Division, the Resource Planning, Programme Management and Performance Division on project monitoring (from Conclusion 1). (a) In order to improve regular reporting as a tool to take well-informed management decisions, self-evaluation reports should not only assess results against objectives by using objectively-verifiable indicators, but also regularly self-assess on-going relevance, efficiency and likelihood of sustainability. (b) For internal management purposes, financial reporting should link expenditures to budget lines and allocate them to different outcomes and project overhead cost. This would enhance transparency of financial reporting, provide managers with a sound basis for budgeting future projects, allow a bench marking of projects within the Development Agenda (DA) and provide the necessary information to assess efficiency of projects in detail. Recommendation 3: To Committee on Development and Intellectual Property (CDIP) on a follow-up project covering PLRs (from Conclusion 3). The evaluation recommends to Member States to favourably consider a proposal for a possible follow-up phase, focusing exclusively on completing the PLRs under preparation and on the further development of the concept, with a view to establish a regular service provided by the
CDIP/10/6 Annex, page 6 Global Infrastructure Sector that assists and advises on patent analytics, and coordinates the development of PLRs in a systematic way. A possible follow-up (if approved by Member States) may in particular: (a) Promote the concept among a broad range of possible users through IP offices, TISCs, universities, business support service providers and other institutions, with a particular focus on those in developing countries and Least Developed Countries (LDCs); (b) Support capacity building through TISCs (where appropriate);
(c) Refine the methodology through publicizing additional reports in other technical fields and using different formats; and (d) Enlarge the range of partners under particular consideration of cooperation with institutions in developing countries. Recommendation 4: To WIPO Senior Managers on establishing a system that allows WIPO to systematically track users of online service (from Conclusion 1). WIPO may consider establishing additional means to track users of its online services (e.g. through requiring users to register). This would provide information on who uses existing services as a basis to provide tailored information to specific target groups and to actively collect feed-back from them for the purpose of continuous improvement of its services. Recommendation 5: To WIPO Senior Managers to strongly promote e-learning tools in order to complement on-site training, where possible (from Conclusion 3). With a view on increasing efficiency of different training activities, WIPO may consider to reinforce existing efforts to complement on-site training involving high travel cost through interactive online courses, such as for example providing them through webinars, taking into account the infrastructure capacities of potential beneficiaries of such services. Recommendation 6: To the WIPO Global Infrastructure Sector on formalizing coordination with other Sectors. The WIPO Global Infrastructure Sector may further formalize with other Sectors their roles and responsibilities in the implementation of a future follow-up covering PLRs through agreements. Defining specific responsibilities to be assumed by each Programme and requiring a formal sign-off by the Programmes involved would help to ensure that coordination is less dependent on informal co-operation.
CDIP/10/6 Annex, page 7 I. BACKGROUND
1. This report covers the independent final evaluation of the Project “Developing Tools for Access to Patent Information” (DA_19_30_31_01) executed by WIPO, subsequently referred to as “the Project”. A. PROJECT BACKGROUND AND DESCRIPTION
2. The Project with a total budget 1,576,000 Swiss francs (of which 936,000 Swiss francs are non-personnel costs) was adopted in November 2009 at the fourth session of the CDIP in Geneva. It was formulated in response to recommendations 19, 30, and 31 of the DA 1 relating to facilitating access and use of patent-related information, knowledge and technology for developing countries, including LDCs. Implementation started in January 2010 and was planned to end in June 2012 (30 months). Main expected outputs included: (a) (b) The drafting of PLRs, The production of an e-tutorial for training on using/exploiting patent information and
(c) Training and information exchange on exploiting patent information through courses and workshops. 3. All the above outputs were to be delivered by the Global Infrastructure Sector (Global Information Services, now Services for Access to Knowledge and Information Division) in collaboration with other sectors of WIPO. 4. Main project purpose is to provide developing countries with demand-driven services to facilitate the use of patent information on specific technologies with the aim to spur indigenous innovation and research and development (R&D) of developing countries. Access to patent information by the public is an important element for achieving a balanced patent system. On the one hand, patent protection encourages investment into R&D, as the owner of a patent is granted exclusive rights to exploit the patented invention in the countries in which the patent is valid. On the other hand, information disclosed about an invention in a patent or patent application is generally made available to the public and to everyone who wants to further exploit a new technology. B. PURPOSE, METHODOLOGY AND LIMITATIONS OF THIS EVALUATION
5. Evaluation work was conducted by the IAOD of WIPO according to WIPO’s evaluation guidelines with the assistance of an external consultant 2 . 6. The work undertaken was guided by the ToRs dated June 2012 and the Inception Report dated August 3, 2012 approved by IAOD. 7. The evaluation covers the period from January 1, 2010 to September 15, 2012.
1
The DA was formally established in October 2007, to mainstream social and economic development into all activities of WIPO, and to keep development at the core of the IP System. In order to enhance the development dimension of the Organization’s activities, a set of 45 recommendations was adopted. The list of DA Recommendations can be found under the following link: http://www.wipo.int/ip-development/en/agenda/recommendations.html 2 Daniel P. Keller, Director, Swiss Consulting Co. Ltd, Hanoi - Vietnam
CDIP/10/6 Annex, page 8 8. Evaluation is part of WIPO’s regular oversight functions that are guided by the WIPO Internal Oversight Charter 3 . WIPO’s Evaluation Policy 4 , which is aligned to the Organization for Economic Co-operation and Development – Development Assistance Committee (OECD-DAC) evaluation criteria and quality standards 5 , provided the framework for this evaluation. (i) Key purpose
9. The key purpose of this evaluation was to assess whether the Project provided the right type of support in the right way based on four main evaluation criteria: (a) Relevance: The extent to which project objectives were consistent with beneficiaries’ requirements, member countries’ needs, global priorities and policies. (b) Efficiency: How economically inputs (e.g. funds, expertise, and time) were converted into results (“value for money”). (c) Effectiveness: The extent to which objectives were achieved, or are expected to be achieved, taking into account their relative importance. (d) Sustainability: assesses the likelihood of continuation of project benefits after the assistance has been completed. 10. Beyond this, the evaluator was also asked to look at WIPO’s contribution to fulfil the general objectives of the Development Agenda (DA) recommendations 19, 30, and 31. 11. The emphasis of this particular evaluation was on organizational learning. The evaluation approach was interactive and participatory. The process itself was designed to contribute to continuous improvement of WIPO’s services. The report further aims at providing input to the decision making of the CDIP and ensure accountability of WIPO towards its Member States. 12. Discussions with stakeholders were in general open and constructive. All persons interviewed openly shared information and exchanged views. The evaluators were able to work freely and without interference. All WIPO staff members supported the evaluation process actively and provided access to all relevant information. Overall, factual information obtained has been comprehensive, consistent and clear. Different evaluation tools were combined to ensure an evidence-based qualitative and quantitative assessment. Particular emphasis was given to cross-validation of data and an assessment of plausibility of the results obtained. The methodological mix included desk studies, literature review, individual interviews, interviews of focal groups and direct observation. 13. The key evaluation results presented below received endorsement by the project managers. (ii) Main limitations to this evaluation
14. Two of the key outputs (E-tutorial and most PLRs) have only recently been delivered. An assessment of the outcome level (the use of the Project’s outputs) was therefore only marginally
3
See WIPO Financial Regulations and Rules, Chapter 7, Rule 7.1, applicable from January 1, 2008, last revised on October 5, 2011. 4 WIPO, Revised Evaluation Policy, May 2010, in particular Annex 1 on evaluation criteria, which makes reference to the DAC Criteria of evaluating development assistance. 5 DAC Guidelines and Reference Series, Quality Standards for Development Evaluation, OECD-DAC, OECD 2010.
CDIP/10/6 Annex, page 9 possible. Looking at impact would be premature. The inclusion of indirect beneficiaries was only possible to the degree the Project had obtained feed-back from them. Fact finding focused on actors directly involved into the Project (different sections of WIPO, key direct beneficiaries and organizations WIPO has cooperated with). Furthermore, the conferences organized under the Project (Output 1) were mainly aimed at promoting and strengthening the TISCs. Assessing their effects on the TISCs would need to be done in conjunction with the relating project (DA_08_01), which was not within the scope of this evaluation. All of this necessarily limited the dimension and depth of the assessment. II. FINDINGS AND ASSESSMENT
This section presents the findings of the evaluation and provides an assessment of the Project against the evaluation criteria. A. (i) PROJECT PREPARATION AND MANAGEMENT Project identification
15. The Project was designed 6 in response to recommendations 19, 30, and 31 7 approved by Member States within the framework of the DA. 16. Specific aspects of those recommendations were addressed by the Project, while others were integrated into other DA projects or covered within the framework of existing WIPO services. 17. The Project was closely linked to project DA_08_01 (“Specialized Databases’ Access and Support”) by strengthening the TISCs. Except for the development of the e-learning tutorial developed under the Project, training programmes on patent information were generally delivered through project DA_08_01. Conferences organized under the Project (Output 3) mainly aimed at promoting the TISCs and were only marginally related to the two other outputs of the project (PLRs, e-tutorial). Given the close linkages to project DA_08_01, the conferences could equally have been funded under this project. 18. Aspects of Recommendation 19 are also implemented by different projects on IP, Technology Transfer and IP, Information Communication and Technology (ICT) and the Digital Divide, and Specialized Databases’ Access and Support (see DA Cluster B). 19. Apart from the dedicated PLR and TISCs website under the aforementioned DA projects, the PLR, the improvement of the PATENTSCOPE portal and the digitization and dissemination of patent information resources that contribute to addressing Recommendation 31 are also covered within WIPO’s regular technical capacity building activities. 20. The project document rightly identifies specific linkages to different strategic goals of WIPO Programmes 1 (Patents), 14 (Global IP Information Services) and 18 (IP and Global Challenges), without however translating them into specific joint activities 8 .
6 7
See Project Document (CDIP 4/6) Recommendation 19 (Cluster B): relates to facilitate access of developing countries to knowledge and technology to foster creativity and innovation and to strengthen relating existing activities within WIPO; recommendation 30 (Cluster C): promotes WIPO’s cooperation with other intergovernmental organizations to provide advice on how to make use of IP-related information on technology; recommendation 31 (Cluster C): aims at contributing to technology transfer to developing countries and LDCs through facilitating access to publicly available patent information.
CDIP/10/6 Annex, page 10 21. Defining specific responsibilities to be assumed by each Programme and requiring a formal sign-off by the Programmes involved would help to ensure that coordination is less dependent on informal cooperation. (ii) Use of project planning tools
22. A logical framework, the standard instrument for project preparation used by almost all development actors, was not an element of the standard DA template on the basis of which the project document was developed. Nevertheless, most information that is typically part of a logical framework is available in the project document. 23. Clear objectives derived from DA recommendations are broken down in specific outputs to be delivered and outcomes to be achieved. Expected outputs and outcomes are linked to specific, measurable, achievable, relevant and time-bound (SMART) performance indicators. Retrieving the information needed to assess outcomes according to some of the indicators would however require extensive surveys among a wide array of companies and policy makers, for which no resources were allocated in the project budget. 24. Clear objectives at the impact level 9 were not defined in the project document. This would not only be important to explain the wider benefits to beneficiaries, but also provide a basis for an impact assessment at a later stage. Since impact is what ultimately matters, it would be important to explain to which longer-term benefits projects aim to contribute. 25. A timeline of implementation is presented in a Gantt chart. Sequencing of activities is in general logical and appropriate. A delivery strategy in the project document clearly outlines the different steps of implementation. The planned project duration of 30 months for delivering all outputs was not commensurate with the limited personnel resources that were allocated. In retrospect it proved to be unrealistic for two WIPO staff(one of them not fully allocated to the Project) working on PLRs to identify their topics, draft the ToRs, commission and publish two high quality PLRs within only three months, as planned. Project duration seems to have been determined to fit the project into the regular biennial budget cycles rather than being based on a realistic estimate of time needed to deliver the outputs required. 26. Some but not all major risks and the way of mitigating them were identified. For instance the risk of not receiving proposals on PLRs from Member States, which subsequently materialized, was overlooked. Categorizing risks according to the likelihood they materialize and the degree of negative impact they may result in would help project managers to focus more closely on monitoring those risks that need to be controlled in order to achieve objectives. Moreover, project preparation did not look at assumptions 10 . 27. Financial planning based on the standard DA template was rudimentary and not in line with good project management practices. Presenting each budget item according to budget lines and outcomes would allow WIPO to compare budgets against expenditures in detail, which is needed for assessing efficiency of implementation. Furthermore, WIPO would be able to benchmark efficiency of projects internally and with similar interventions of other development
[Footnote continued from previous page]
See WIPO Program and Budget for 2010/2011. Program 1: Patents, Program 14: Global IP Information Services and Program 18: IP and Global Challenges 9 Impact: Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended (OECD, Glossary of Key Terms in Evaluation and Results Based Management, 2010). 10 Hypotheses about factors or risks which could affect the progress or success of a development intervention (OECD, Glossary of Key Terms in Evaluation and Results Based Management, 2010)
8
CDIP/10/6 Annex, page 11 actors. Last but not least, it would provide project managers with detailed information as a basis for accurate budgeting for similar future projects. (iii) Project Management
28. Project management was flexible in addressing arising challenges and tailoring the implementation strategy to changed circumstances: Progress reports indicate that management continued to actively identify and analyse risks arising during implementation. Regular monitoring and mitigation of risks is evidenced by the Project’s response to the lack of proposals from Member States for PLRs. The mitigation strategy chosen consisted in identifying topics in cooperation with different international organizations and non-governmental organizations NGOs). 11 Management also actively identified and exploited new cooperation opportunities and was responsive to requests received. 29. Management also looked for new opportunities to expand the reach of the Project. Moreover, the cooperation with other UN organizations in the area of patent landscaping (see table 1 below), which was also foreseen in the project document, is evidence of WIPO’s efforts to create synergies and complementarities within the UN system and with inter-governmental organizations (IGOs) and NGOs. B. RELEVANCE
Relevance assesses the extent to which project objectives were consistent with beneficiaries’ requirements, member countries’ needs, global priorities and WIPO’s policies. (i) Policy relevance
30. The fact that the Project was approved by the CDIP provides strong evidence for the Project’s relevance to Member States at the macro policy level. 31. The Project responds well to a number of international priorities, which have been defined by UN Member States, most of which are also members of WIPO. The topics selected for the PLRs are all of high relevance to international priorities and to the developing world in particular. Several of the areas covered by the PLRs potentially contribute to different Millennium Development Goals (MDGs). To be mentioned are in particular those relating to child mortality (MDG 4), maternal health (MDG 5), combating malaria, HIV/AIDS and other forms of diseases in developing countries and LDCs (MDG 6) and providing access to affordable essential drugs (MDG 8, Target 4). Other PLRs relate to MDG 7 (environmental sustainability), in particular to target 7C (access to clean drinking water). Two PLRs (one on solar cooking and another one on solar cooling) are potentially relevant to international efforts to reduce greenhouse gases under different initiatives. 32. Further mentioned should be the Project’s contribution to the UN’s efforts to better coordinate its services through cooperation, as explained in section 2.1 above. 33. The Project is also well aligned with WIPO’s policies, reflected by the Strategic Goals and Programmes 12 and capitalizes on WIPO’s core competencies.
11 12
See CDIP 6/2, Annex XIII, page 3: risk of demand and expertise for PLRs. In particular Strategic Goal I, Program 1; Strategic Goal IV, Programme 14 and Strategic Goal VII, Programme 18
CDIP/10/6 Annex, page 12 (ii) Relevance to target groups
34. PLRs: Relevance was confirmed through interviews with a sample of representatives from different target groups of direct users (one international organization, one NGO, one researcher and one patent practitioner). The evaluation also received indirect confirmation of positive feedback on the two PLRs on desalination and use of renewable energy, and on membrane filtration and UV water treatment, presented in an international conference (World Water Day). 35. E-tutorial: The need for training on using and exploiting patent information is evidenced by interviews, surveys of conference participants and initial e-tutorial users similar training material known to the evaluator either have only one section on patent information or explain only one specific internet search system. The e-tutorial is potentially highly relevant, but as it was only available at the end of the Project, it was not possible to validate this with a broader range of beneficiaries who have used the tool in practice. 36. Conferences: According to the surveys conducted at the end of the conferences 13 , participants confirmed the relevance of the topics covered for their work. C. EFFECTIVENESS
37. This section compares planned against expected results using the OVIs defined in the project document. As a basis, results reported by the Project were validated and assessed against key performance indicators defined in the project document. As mentioned under limitations in section 1 (B) above, outcomes was only marginally evaluable, since some PLRs and the e-tutorial have only recently or not yet been delivered. Short term effects of the training were measured through participants’ surveys. Outcome data, which is for instance required to assess the longer-term effects of the trainings or PLRs, was not yet available at the time of the evaluation. (I) Output 1: PLRs accessible on PATENTSCOPE® Website
38. The Project planned to make a total of 12 PLRs accessible through a dedicated Website with a “quality corresponding to the ToR requirements”. By September 15, 2012, a total of seven reports were published on the PLR website. The PLR on Ritonavir covers five instead of four innovation tracks (through a subsequent amendment covering Ritonavir pro-drugs). The PLR on vaccines combines the resources of two reports (one general part, one on vaccines for specific diseases). Two more PLRs had been commissioned and were under preparation. Topics and partner organizations for three more PLRs had been identified, but contracts to draft them not yet awarded. In retrospect, the target to make12 PLRs accessible via the Internet appears to have been overambitious and not achievable with the personnel and financial resources available. 39. In addition, 50 PLRs established by other organizations were also made available on the WIPO website 14 . 40. Due to the lack of specific proposals from Member States and their national institutions following circulars sent to the permanent missions in Geneva, the WIPO Secretariat initiated
See summary of participant evaluations in the reports established for each of the conferences. A compilation of links to publicly available PLRs in different areas prepared by various entities is available on (http://www.wipo.int/patentscope/en/programs/patent_landscapes/published_reports.html). PLRs established by the Project can be found on (http://www.wipo.int/patentscope/en/programs/patent_landscapes/ongoing_work.html)
14 13
CDIP/10/6 Annex, page 13 cooperation with other UN Agencies, IGOs and NGOs with expertise in the areas of technology, as described in the original project document (Public Health, Food and Agriculture, Environment and Climate Change, Disabilities). Those partners included one NGO from a developing country (African Agriculture Technology Foundation (AATF)). 41. The evaluation was unable to shed light on the reasons for the lack of inputs from developing countries, though it may be conjectured that low awareness on PLRs and their possible use was among the reasons. This indicates the need to promote PLRs in order to foster broader inputs to their planning and implementation. A possible forum to further promote PLRs would be the TISCs with their networks at the country level. 42. The cooperation with various IGOs and NGOs, including the World Health Organization (WHO), the Food and Agriculture Organization (FAO), Medicines Patent Pool, the International Renewable Energy Agency (IRENA), the Drugs for Neglected Diseases Initiative (DNDI), the AATF, the Global Institute for Water Environment and Health (GIWEH), the International Seed Federation (ISF), the International Telecommunication Union (ITU), the European Organization for Nuclear Research (CERN), the United Nations Environment Programme (UNEP), the World Meteorological Organization (WMO), Village Earth, and others has led to the identification of several specific topics for PLRs 15 as presented in Table 1 below. Table 1: Status of progress in publishing PLRs (as per 15 September 2012) Title of Report 1. Solar cooking 2. Solar cooling 3. Ritonavir 4. Atazanavir 5. Desalination and the use of alternative energies 6. Membrane Filtration and UV water treatment 7. Vaccines for selected diseases 8. Selected Neglected Diseases 9. Plant Salinity tolerance 10. E-waste recycling 11. Animal Genetic Resources 12. Accelerator technologies Cooperation Partner UNITAID/MPP UNITAID/MPP IRENA/GIWEH GIWEH WHO DNDI FAO, AATF & ISF UNEP FAO CERN Status Published Published Published Published Published Published Published Under finalization Under preparation Procurement stage ToR drafted ToR drafted
Source: Information provided by the Project on request 43. Topics were mostly selected based on those addressed by a specific project or programme of the collaborating partner institution. The fields covered relate to areas of potential interest to developing countries and LDCs (e.g. procurement of HIV medicines, access to clean water, renewable energy etc.). Selected partners interviewed confirmed their active involvement in the preparation of the ToRs for the reports and in the evaluation of the draft reports. While two of the partners interviewed had extensive experience in using patent information, for others the consultations on drafting the ToRs also served as a vehicle to familiarize themselves with broader aspects of patent information and protection for those partners with no prior knowledge in the field of patents. 44. PLRs are designed to identify what is patented, where and by whom. Each report further depicts the technology for which the patenting activity was investigated. Important for capacity building purposes is the disclosure of the patent search methodology, which enables the reader
See summary of discussions held in document “contacts and discussions for establishment of cooperation partners and preparation of landscaping reports (prepared by the Project)
15
CDIP/10/6 Annex, page 14 to learn about general search strategies (combination of classification and keyword search) or specific search strategies (e.g. various names used for pharmaceutical compounds). Commercially produced reports would typically not disclose the methodology in detail. 45. Reports facilitate technology transfer through identifying relevant patents. However, they do not include information about the actual legal status of those patents since such status naturally changes and requires verification. Each report includes meanwhile valuable patent family information, i.e. information regarding other jurisdictions where patent applications for the same invention have been filed. A company that wants to exploit a certain technology would have to verify its freedom to operate for each individual country based on this family information. Since patent data from many countries is not readily accessible through online patent registers, this task of determining the patent status of each family member remains time-consuming and expensive. WIPO in collaboration with other IPOs assists IGOs, NGOs or institutions in developing countries in obtaining up-to-date legal status data through WIPO's Patent Information Service (WPIS) for developing countries, and has provided such data, e.g. on several Ritonavir and Atazanavir related patents included in the respective reports. 46. The four published PLRs focus on providing facts and evidence rather than drawing strong conclusions or recommendations for policy makers, which might conflict with WIPO’s mandate as a specialised UN Agency. The evaluators received confirmation of the usefulness of the PLRs as input to policy documents at the international level drafted by technical specialists in the field. While PLRs could of course also be directly commissioned by partner organizations, WIPO enjoys the reputation of high technical competence and neutrality, which increases the credibility of the PLRs as source of reference. In some fields (e.g. pharmaceuticals), it might be interesting to summarize key findings derived from several reports in a format tailored to the needs of policy makers in developing countries. 47. The Project further delivered two additional (unplanned) outputs: the patent database for each of the reports is published on the website. These spread sheets contain all relevant patent families, on which the patent analysis was based, including hyperlinks to the complete patent documents. In addition, an interactive visualization of the patent analysis was made available. Users are able to explore various automatic statistical analysis options used in patent analysis. The tools on the website are highly useful. Moreover, a standardized procurement procedure for PLRs was developed in coordination with the Procurement and Contracts Section of WIPO. 16 48. All stakeholders interviewed unanimously confirmed the high quality of all PLRs that have so far been published. In-depth interviews with partner organizations confirmed that reports were used as a factual input to policy documents (e.g. the WHO’s Initiative for Vaccine Research) or to identify possible licensing arrangements for patents (e.g. Atazanavir and Ritonavir, both used for the treatment of HIV/AIDS). The quality and usefulness of the reports for users is also validated through survey results 17 . Respondents stated they had used PLRs mainly to learn more about patent information (21%), learn more about methodologies in patent search and analysis (26%), getting information about trends (21%), get information about specific technologies (16%) or get information about companies and people (10%). (II) Output 2: E-tutorial
49. By the end of August 2012 (three months later than planned), the e-tutorial was available online, while the production of DVDs is projected to be completed by the end of September
See as an example document “Call for Expression of Interest for the Request of Proposal No PTD/10/007” See survey Report dated August 20, 2012 established by the Project. Due to the small sample of 35 users, results are statistically not relevant. But the survey provides some indication on results at the outcome level.
17 16
CDIP/10/6 Annex, page 15 2012. According to own observation of the evaluators, the e-tutorial is of high quality. It seems to be particular useful for technical specialists with no prior experience in the field of IP. The modular design allows users to select those topics they are specifically interested in. Initial feed-back obtained through a test survey conducted by the Project confirmed this. 50. At this stage, an assessment of outcomes (use of the e-tutorial) is not yet possible. The e-tutorial focuses on practical knowledge and complements well the more theoretical courses of the WIPO. According to the interviews with the Project, the e-tutorial was developed in consultation with the WIPO Academy. 51. Delays in delivering this output had three main reasons: late start of preparation work, procurement issues of temporary nature (changes in procurement procedures led to delays in awarding the contract) and the complex task of developing an e-tutorial in coordination with various WIPO departments, external content providers and editors, and the application developer. (iii) Output 3: Regional conferences
52. Regional conferences promoting the TISC project and offering initial training on technology and innovation support and a platform to exchange institutional experiences in the field of innovation and best practices among countries were held in Singapore in May 2010 (40 participants from the Asia-Pacific region), in Addis Ababa in November 2010 (39 participants from the African region), in Moscow in December 2010 (32 participants from certain European and Asian countries), as well as in Buenos Aires in March 2011 (150 participants from Latin America and the Caribbean). A further regional conference, attended by 44 participants from the African Regional Intellectual Property Organization (ARIPO), was carried out in Harare in June 2011. 53. Participants were selected through invitation sent to heads of IP offices or their alternates in the respective regions, with guidance to nominate “a patent information specialist or person in charge of patent documentation and databases” as additional participants. According to the Project’s managers, selection of participants was based on the objective of raising awareness and developing skills among the management and technical staff of IP offices potentially involved in the establishment of TISCs and preparation or dissemination of PLRs. 54. There is no sign of a formal selection process for attendees by WIPO (which would be challenging), so it is not possible to draw any conclusions on whether the participants invited would be those who are able to practically apply what they have learned. An analysis of conference agendas and the material showed that the main benefit of the conferences was to promote the TISCs and to provide capacity building to their staff. The specific content of the conferences seems to have been tailored to the different needs in the regions covered, as evidenced by the different formats. Participants rated the conferences overall above the mark “good (4)”. 18 A survey conducted at the end of the conference indicated that the conferences had been “very useful” (4) for the daily work of participants and had generally be “useful” (3) or “very useful” (4) in raising their awareness and building their understanding of patent information in general, providing training, organizing and managing services, monitoring and evaluating services. The survey suggested that the conferences had been less useful for building an understanding of how to obtain resources. Detailed information on how the information provided in the conferences had been used (outcome level) was not available.
18
Survey results for conferences in Singapore, Addis Ababa, Moscow, Buenos Aires and Harare.
CDIP/10/6 Annex, page 16 (iv) No unexpected negative effects found
55. PLRs are a service that is provided by public institutions such as patent offices as well as private companies. The Project’s objectives, implementation strategy and outputs suggest that it produced a public good and did not compete directly with services offered by private companies. The topics covered by the PLRs produced under the Project related to areas of public interest. Moreover, privately commissioned reports would not be publicly accessible, and if, the methodology is typically not disclosed. Should WIPO decide to offer PLRs as a regular service on the free market, it would be important to differentiate pricing in a way that market rates are charged where WIPO services compete with private service providers. B. EFFICIENCY
56. Based on the official financial reports per end of December 31, 2011, 95.5% or 894,000 Swiss francs of the 936,000 Swiss francs allocated to the Project (excluding personnel costs) had been disbursed. Table 2 shows an analysis of non-personnel expenditures according to different budget lines. Table 2: Non-personnel expenditures according to outputs and budget lines (in 1,000 CHF) PLRs 8 8 293 e-Tutorial Conferences 3 336 3 125 211 138 108 94 All outputs 347 136 211 539 94 % of total 39% 15% 24% 60% 11%
Travel and fellowships Staff Missions Third-party Travel Contractual services Conferences Publishing Experts' Honoraria and Other Equipment and Supplies Furniture & Equipment Supplies and Materials Total % of total expenditures
293 8 8 309 35%
138 0
14 0
445 8 8 894 100%
50%
141 16%
444 50%
1% 100%
Source: Information provided by the Project (unofficial figures) and analysis by evaluators 57. This analysis shows that around half of the expenditures relate to promoting the TISCs, also financially supported under Project DA_08_01, one third on the PLRs, and around one eighth on the e-tutorial. Accordingly, it is not surprising that, as foreseen in the project document approved by Member States, almost 40% of the expenditures were spent for travel and missions (96% of travel costs are relating to the conferences). A large slice of the expert fees (66%) relate to the PLRs and the e-tutorial (31%), whose development was mostly outsourced. 58. PLRs: Costs per PLR were significantly higher than expected, evidenced by the smaller number of reports produced than planned, given the higher complexity of the subject matter and expected depth of analysis involved in their preparation, according to the Project managers. Consequently, the completion of all 12 PLRs (see table 1) would not be feasible without
CDIP/10/6 Annex, page 17 additional funding. The average cost 19 for commissioning one PLR as reported by the Project was around 31,780 Swiss francs. 59. E-tutorial: Relating the potentially high number users to the costs, the e-tutorial is a highly efficient way of capacity building. 60. Conferences: Assuming that the number of 306 participants reported by the Project (see 2.2 above) is correct, the cost per conference participant would be 1’010 Swiss francs. While awareness raising and networking are important for the building of the TISC network, the evaluators strongly endorse WIPO’s efforts to complement resource-intensive on-site training through internet-based tools for seminars (e.g. webinars), wherever this is possible without jeopardizing the quality of capacity building efforts. 61. Due to the lack of a detailed result-based budget in the project document 20 , a direct comparison between budget and expenditures per category and output was not possible. E LIKELIHOOD OF SUSTAINABILITY OF RESULTS
62. Sustainability assesses the likelihood of continuation of project benefits after the assistance has been completed. 63. E-Tutorial: The sustainability of the e-learning tool is rather obvious. It will continue to be available on-line and WIPO does have the technical and financial capacities to maintain it. 64. International conferences: The evaluation was unable to assess the sustainability of the outcomes achieved by the conferences, which mainly aimed at strengthening the TISCs. This question depends on the sustainability of the TISCs, which exceeds the scope of this evaluation (see also comments on limitations in section 1 (B) above) 21 . 65. PLRs: The sustainability of PLRs and relating capacity building as a service depends on its integration in the portfolio of regular WPO services. The intention of integrating PLRs into the DA was to test demand, institutionalise, improve the concept and ultimately take a decision on whether to mainstream PLR services under WIPO’s regular budget. While prior to publishing PLRs under a DA project, several thematic departments had already undertaken similar work, a decision was taken to entrust PLRs to the Global Infrastructure Division. The evaluation found emerging evidence of two advantages of centralizing PLRs within the Global Infrastructure Division as done under the Project: Firstly, drafting PLRs and advising partners on defining ToRs requires a high specialization in the field of patent search methodology. Secondly, centralizing the service reduces the risk that the selection of patent reports to be published is driven by WIPO’s own internal needs and interests rather by those of possible clients. On the other hand, the Global Infrastructure Division lacks in-depth knowledge in certain thematic fields. Therefore, coordination with thematic departments at an early stage of drafting the ToRs for the reports for their inputs and strengthening cooperation with external partners with expertise in the relevant fields remains crucial. 66. Before taking a decision whether and in which way WIPO provides PLR services, further testing of the concept and the demand is needed. While demand of international organizations is obviously given and likely to continue, it will be important to explore the demand of institutions in developing countries and LDCs. This might also allow covering subjects that are of particular
Total cost of the subcontracts for producing the nine reports that have either been published or are under preparation was 286,021 Swiss francs, contract values ranging from 16,212 Swiss francs to 63,400 Swiss francs. 20 See budget CDIP/4/6 Annex, page 9, which presents details of non-personnel cost only according to staff missions, third-party travel and experts’ honoraria (contractual services). 21 Project DA 8_01 was evaluated, but the report (CDIP 9/5 April 2012) does refer to input received from the Project.
19
CDIP/10/6 Annex, page 18 national and/or regional relevance. More awareness rising and capacity building is needed to create demand among a broader range of users and refining the concept of PLRs. This will require a follow-up phase, after which a final decision should be taken on whether or not provide PLRs as a regular service. A proposal is currently in the initial stage of preparation.
CDIP/10/6 Annex, page 19 III. CONCLUSIONS
67. Based on the findings and assessment above, the evaluation draws the following conclusions: Conclusion 1: The Project was generally well prepared and managed, but there is room for further enhancing existing tools for planning, monitoring and evaluating projects. The Project made proper use of existing planning and monitoring tools, but there is room to improve the tools in consideration of internationally recognized best practices in managing development assistance. While the project document does not include a logical framework, it contains most of its information, such as clearly defined objectives at different levels linked to objectively verifiable performance indicators. Yet, assumptions on external factors that need to be in place to achieve each of the results and objectives at the impact level are not defined. 68. The involvement of other WIPO sectors and links to WIPO Programmes are well explained, without however specifying concrete actions and responsibilities. Good coordination with other WIPO departments and external partners was more the result of personal initiatives than of a systematically planned approach. 69. Progress reports provide evidence that management actively mitigated most risks identified at the planning stage and additional ones that became apparent during implementation. The most important risk not identified, which subsequently materialized, was a possible lack of inputs from Member States on the topics of PLRs. 70. Unlike for programmes, WIPO does not require result-based financial budgeting and reporting for projects. This Project established for managerial purposes a financial report that links expenditures to budget lines and outputs, which has been identified as a good practice. 71. The Project made significant efforts to use well designed surveys for a systematic monitoring of quality and use of outputs. This was however not possible for users who downloaded PLRs from the dedicated PLR website. The current statistics of WIPO’s Internet Services Sector provide only information related to the number of unique page views, the geographical distribution of the page visitors, along with the number of downloads of each of the PLRs. They do not provide detailed and unambiguous information about who accessed a particular resource and how and for what purpose it was subsequently used. Systematically retrieving contact details about users of on-line services would allow WIPO to profile its clients and to conduct online surveys as a basis for specifically tailoring its services to different target groups. Conclusion 2: The project design was overambitious, especially for achieving the objectives set for the PLRs. 72. The planned time of 30 months to achieve the objectives, in particular producing 12 PLRs, was not commensurate and seems to have been determined by the need to align the project duration to biennial budgeting cycles rather than the time required to deliver the results. The number of PLRs the Project was actually able to publish (see conclusion 3) is evidence of this. Conclusion 3: While the Project generally provided the right type of support in the right way, not all of its expected outputs (PLRs, e-tutorials, regional conferences) were delivered. 73. PLRs: While only seven instead of 12 PLRs as foreseen in the project document were published as of 15 September 2012, users considered them of high quality and generally useful for their work. Two more PLRs have been commissioned and are under preparation. As an additional output, various innovative and useful tools to analyse information of the PLRs and to directly access patent information used in the reports were made available on the Internet. In
CDIP/10/6 Annex, page 20 addition, the Project refined their concept and developed standard procedures and documentation for commissioning work from outside suppliers. Key value added of the Project was the capacity building element through disclosure of the methodology and cooperation with different partners in drafting ToR. Evidence provided through interviews suggests that some of the reports were used mainly as an input to the work of other organizations, including in policy making. Due to the lack of direct inputs from Member States, the Project exclusively cooperated with different international, inter-governmental and non-governmental organizations. Topics selected covered areas of high relevance to internationally agreed priorities, including the Millennium Development Goals. 74. Expanding the direct benefits to developing countries would call for a second phase, which would also allow further testing, refining and disseminating of the methodologies. Longterm sustainability will require pursuing the Project’s efforts to prepare for providing PLRs as a regular service of WIPO’s Global Infrastructure Sector. 75. The e-tutorial is of high quality and potential relevance to users, especially to those with a technical background, but no prior knowledge in the field of patents. It complements existing tools, which are either not comprehensive or are oriented towards the use of specific databases. 76. The fact that it became only available after the official ending date of the Project had three main reasons: late start of preparation work, procurement issues (changes in procurement procedures led to delays in awarding the contract), and the complex task of developing an etutorial in coordination with various WIPO departments, external content providers and editors, and the application developer. 77. Participants rated the quality of regional conferences as highly relevant to their work. The conferences mainly promoted the TISCs, also supported by project DA_8_01. Furthermore, basic capacity building, mainly to staff of IP Offices, was provided. Conferences were only marginally linked to the e-tutorial and the PLRs. Assessing long-term benefits at the outcome level and their sustainability depends on whether and how the TISCs will use the information provided. Looking at this was neither required nor possible within the scope of this evaluation. While awareness raising and networking is important at the initial stage of technical support, WIPO has rightly recognized the need to also provide training and information exchange through modern online tools (e.g. webinars) to complement more resource-intensive means of skills development such as on-site training. IV. RECOMMENDATIONS
Recommendation 1: To Project Managers, the Development Agenda Coordination Division and Resource Planning, Programme Management and Performance Division on project planning (from Conclusions 1 and 2): 78. Wherever possible, project duration should be determined based on a reasonable estimate of the time required to achieve each expected result rather than based on the duration of funding cycles. 79. In addition to defining expected outputs and outcomes and linking them to objectively verifiable indicators (OVIs), as done, longer-term changes the Project is expected to contribute to (impact level) as well as related OVIs should be formalized. 80. Where monitoring and/or self-assessment of results by projects requires significant resources (e.g. comprehensive studies), they need to be included into the budget.
CDIP/10/6 Annex, page 21 81. As a basis for internal result-based financial monitoring, project documents should include a result-based budget that allocates expenditures according to budget lines (e.g. travel) to each of the expected outcomes and to project management cost (overhead). 82. Besides identifying risks and the way of mitigating them, risks should be rated according to the degree of their potential negative impact on achieving results and the likelihood they materialize. 83. Project documents should include assumptions (external conditions that need to be in place to achieve objectives). 84. Coordination within WIPO and other organizations should be clearly specified (explaining which specific joint actions will be taken and who is responsible). Recommendation 2: To Project Managers and Development Agenda Coordination Division, the Resource Planning, Programme Management and Performance Division on project monitoring (from Conclusion 1) 85. In order to improve regular reporting as a tool to take well-informed management decisions, self-evaluation reports should not only assess results against objectives by using objectively-verifiable indicators, but also regularly self-assess on-going relevance, efficiency and likelihood of sustainability. 86. For internal management purposes, financial reporting should link expenditures to budget lines and allocate them to different outcomes and project overhead cost. This would enhance transparency of financial reporting, provide managers with a sound basis for budgeting future projects, allow a bench marking of projects within the DA and provide the necessary information to assess efficiency of projects in detail. Recommendation 3: To CDIP on a follow-up project covering PLRs (from Conclusion 3) 87. The evaluation recommends to Member States to favourably consider a proposal for a possible follow-up phase, focusing exclusively on completing the PLRs under preparation and on the further development of the concept, with a view to establish a regular service provided by the Global Infrastructure Sector that assists and advises on patent analytics, and coordinates the development of PLRs in a systematic way. 88. A possible follow-up (if approved by Member States) may in particular:
89. Promote the concept among a broad range of possible users through IP offices, TISCs, universities, business support service providers and other institutions, with a particular focus on those in developing countries and LDCs; 90. Support capacity building through TISCs (where appropriate);
91. Refine the methodology through publicizing additional reports in other technical fields and using different formats; 92. Enlarge the range of partners under particular consideration of cooperation with institutions in developing countries.
CDIP/10/6 Annex, page 22
Recommendation 4: To WIPO Senior Managers on establishing a system that allows WIPO to systematically track users of online service (from Conclusion 1) 93. WIPO may consider establishing additional means to track users of its online services (e.g. through requiring users to register). This would provide information on who uses existing services as a basis to provide tailored information to specific target groups and to actively collect feed-back from them for the purpose of continuous improvement of its services. Recommendation 5: To WIPO Senior Managers to strongly promote e-learning tools in order to complement on-site training, where possible (from Conclusion 3) 94. With a view on increasing efficiency of different training activities, WIPO may consider to reinforce existing effortsto complement on-site training involving high travel cost through interactive online courses, such as for example providing them through webinars, taking into account the infrastructure capacities of potential beneficiaries of such services. Recommendation 6: To the WIPO Global Infrastructure Sector on formalizing coordination with other Sectors 95. The WIPO Global Infrastructure Sector may further formalize with other Sectors their roles and responsibilities in the implementation of a future follow-up covering PLRs through agreements. Defining specific responsibilities to be assumed by each Programme and requiring a formal sign-off by the Programmes involved would help to ensure that coordination is less dependent on informal co-operation.
CDIP/10/6 Annex, page 23
LIST OF ANNEXES (AVAILABLE IN THE FULL VERSION OF THE REPORT) Project document Annex 1 Inception report Annex 2 List of persons met Annex 3 List of documents Annex 4
[End of Annex and of document]
doc_248618044.pdf
Intellectual property (IP) is a legal concept which refers to creations of the mind for which exclusive rights are recognized.
E
CDIP/10/6 ORIGINAL: ENGLISH DATE: SEPTEMBER 28, 2012
Committee on Development and Intellectual Property (CDIP)
Tenth Session Geneva, November 12 to 16, 2012
EVALUATION REPORT OF THE PROJECT ON DEVELOPING TOOLS FOR ACCESS TO PATENT INFORMATION
prepared by the Secretariat
1. The Annex to this document contains an independent evaluation report of the project on Developing Tools for Access to Patent Information undertaken by WIPO’s Internal Audit and Oversight Division. 2. The CDIP is invited to take note of the information contained in the Annex to this document.
[Annex follows]
CDIP/10/6 ANNEX
TABLE OF CONTENT EXECUTIVE SUMMARY.............................................................................................................. 3 MAIN CONCLUSIONS ............................................................................................................. 3 RECOMMENDATIONS ............................................................................................................ 5 I. BACKGROUND .................................................................................................................... 7 a. PROJECT BACKGROUND AND DESCRIPTION ............................................................ 7 B. purpose, methodology and limitations of this evaluation .................................................. 7 (i) Key purpose ................................................................................................................. 8 (ii) Main limitations to this evaluation ................................................................................. 8 II. FINDINGS AND ASSESSMENT ...................................................................................... 9 A. PROJECT PREPARATION AND MANAGEMENT........................................................... 9 (i) Project identification ..................................................................................................... 9 (ii) Use of project planning tools ...................................................................................... 10 (iii) Project Management............................................................................................... 11 B. RELEVANCE.................................................................................................................. 11 (i) Policy relevance ......................................................................................................... 11 (ii) Relevance to target groups ........................................................................................ 12 C. EFFECTIVENESS .......................................................................................................... 12 (I) Output 1: PLRs accessible on PATENTSCOPE® Website ....................................... 12 (II) Output 2: E-tutorial .................................................................................................... 14 (iii) Output 3: Regional conferences ............................................................................ 15 (iv) No unexpected negative effects found ................................................................... 16 B. Efficiency ........................................................................................................................ 16 e Likelihood of Sustainability of Results ............................................................................ 17 iii. CONCLUSIONS ............................................................................................................. 19 IV. Recommendations.......................................................................................................... 20
CDIP/10/6 Annex, page 2 LIST OF ACRONYMS AATF ARIPO CDIP CERN DA DAC DNDi FAO GIWEH IAOD IGO IP IRENA ISF ITU LDCs MDGs NGO OECD PLR R&D SMART TISC ToR UN UNEP WIPO WMO
The African Agricultural Technology Foundation African Regional Intellectual Property Organization Committee on Development and Intellectual Property The European Organization for Nuclear Research Development Agenda Development Assistance Committee (of the OECD) The Drugs for Neglected Diseases Initiative Food and Agriculture Organization The Global Institute for Water, Environment and Health WIPO’s Internal Audit and Oversight Division Inter-governmental organization Intellectual property International Renewable Energy Agency The International Seed Federation The International Telecommunication Union Least Developed Countries Millennium Development Goals Non-governmental organization Organization for Economic Co-operation and Development Patent Landscape Report Research and Development Specific, measurable, achievable, relevant and time-bound (indicators) Technology Information and Support Centre Terms of Reference United Nations United Nations Environment Programme World Intellectual Property Organization World Meteorological Organization
CDIP/10/6 Annex, page 3
EXECUTIVE SUMMARY This report covers the independent final evaluation of the Project “Developing Tools for Access to Patent Information” (CDIP 4/6) subsequently referred to as “the Project”. Evaluation work was conducted by the Internal Audit and Oversight Division of the World Intellectual Property Organization (WIPO) according to WIPO’s evaluation guidelines with the assistance of an external consultant. The Project with a total duration of 30 months and a budget of 1,576,000 Swiss francs started in January 2010. It aimed at enhancing access of developing countries to patent information, by publishing patent landscape reports (PLRs), developing an e-tutorial and organising regional conferences. MAIN CONCLUSIONS The findings and assessment of the evaluation resulted in the following conclusions: Conclusion 1: The Project was generally well prepared and managed, but there is room for further enhancing existing tools for planning, monitoring and evaluating projects. The Project made proper use of existing planning and monitoring tools, but there is room to improve the tools in consideration of internationally recognized best practices in managing development assistance. While the project document does not include a logical framework, it contains most of its information, such as clearly defined objectives at different levels linked to objectively verifiable performance indicators. Yet, assumptions on external factors that need to be in place to achieve each of the results and objectives at the impact level are not defined. The involvement of other WIPO sectors and links to WIPO Programmes are well explained, without however specifying concrete actions and responsibilities. Good coordination with other WIPO departments and external partners was more the result of personal initiatives than of a systematically planned approach. Progress reports provide evidence that management actively mitigated most risks identified at the planning stage and additional ones that became apparent during implementation. The most important risk not identified, which subsequently materialized, was a possible lack of inputs from Member States on the topics of PLRs. Unlike for programmes, WIPO does not require result-based financial budgeting and reporting for projects. This Project established for managerial purposes a financial report that links expenditures to budget lines and outputs, which has been identified as a good practice. The Project made significant efforts to use well designed surveys for a systematic monitoring of quality and use of outputs. This was however not possible for users who downloaded PLRs from the dedicated PLR website. The current statistics of WIPO’s Internet Services Sector provide only information related to the number of unique page views, the geographical distribution of the page visitors, along with the number of downloads of each of the PLRs. They do not provide detailed and unambiguous information about who accessed a particular resource and how and for what purpose it was subsequently used. Systematically retrieving contact details about users of on-line services would allow WIPO to profile its clients and to conduct online surveys as a basis for specifically tailoring its services to different target groups. Conclusion 2: The project design was overambitious, especially for achieving the objectives set for the PLRs. The planned time of 30 months to achieve the objectives, in particular producing 12 PLRs, was not commensurate and seems to have been determined by the need to align the project duration to biennial budgeting cycles rather than the time required to deliver the results. The number of PLRs the Project was actually able to publish (see conclusion 3) is evidence of this.
CDIP/10/6 Annex, page 4 Conclusion 3: While the Project generally provided the right type of support in the right way, not all of its expected outputs (PLRs, e-tutorials, regional conferences) were delivered. PLRs: While only seven instead of 12 PLRs as foreseen in the project document were published as of September 15, 2012, users considered them of high quality and generally useful for their work. Two more PLRs have been commissioned and are under preparation. As an additional output, various innovative and useful tools to analyse information of the PLRs and to directly access patent information used in the reports were made available on the Internet. In addition, the Project refined their concept and developed standard procedures and documentation for commissioning work from outside suppliers. Key value added of the Project was the capacity building element through disclosure of the methodology and cooperation with different partners in drafting Terms of Reference (ToR). Evidence provided through interviews suggests that some of the reports were used mainly as an input to the work of other organizations, including in policy making. Due to the lack of direct inputs from Member States, the Project exclusively cooperated with different international, inter-governmental and non-governmental organizations. Topics selected covered areas of high relevance to internationally agreed priorities, including the Millennium Development Goals. Expanding the direct benefits to developing countries would call for a second phase, which would also allow further testing, refining and disseminating of the methodologies. Long-term sustainability will require pursuing the Project’s efforts to prepare for providing PLRs as a regular service of WIPO’s Global Infrastructure Sector. The e-tutorial is of high quality and potential relevance to users, especially to those with a technical background, but no prior knowledge in the field of patents. It complements existing tools, which are either not comprehensive or are oriented towards the use of specific databases. The fact that it became only available after the official ending date of the Project had three main reasons: late start of preparation work, procurement issues (changes in procurement procedures led to delays in awarding the contract), and the complex task of developing an e-tutorial in coordination with various WIPO departments, external content providers and editors, and the application developer. Participants rated the quality of regional conferences as highly relevant to their work. The conferences mainly promoted the Technology and Information Support Centres (TISCs), also supported by project DA_8_01. Furthermore, basic capacity building, mainly to staff of Intellectual Property (IP) Offices, was provided. Conferences were only marginally linked to the e-tutorial and the PLRs. Assessing long-term benefits at the outcome level and their sustainability depends on whether and how the TISCs will use the information provided. Looking at this was neither required nor possible within the scope of this evaluation. While awareness raising and networking is important at the initial stage of technical support, WIPO has rightly recognized the need to also provide training and information exchange through modern online tools (e.g. webinars) to complement more resource-intensive means of skills development such as on-site training.
CDIP/10/6 Annex, page 5 RECOMMENDATIONS Recommendation 1: To Project Managers, the Development Agenda Coordination Division and Resource Planning, Programme Management and Performance Division on project planning (from Conclusions 1 and 2). (a) Wherever possible, project duration should be determined based on a reasonable estimate of the time required to achieve each expected result rather than based on the duration of funding cycles. (b) In addition to defining expected outputs and outcomes and linking them to objectively verifiable indicators (OVIs), as done, longer-term changes the Project is expected to contribute to (impact level) as well as related OVIs should be formalized. (c) Where monitoring and/or self-assessment of results by projects requires significant resources (e.g. comprehensive studies), they need to be included into the budget. (d) As a basis for internal result-based financial monitoring, project documents should include a result-based budget that allocates expenditures according to budget lines (e.g., travel) to each of the expected outcomes and to project management cost (overhead). (e) Besides identifying risks and the way of mitigating them, risks should be rated according to the degree of their potential negative impact on achieving results and the likelihood they materialize. (f) Project documents should include assumptions (external conditions that need to be in place to achieve objectives). (g) Coordination within WIPO and other organizations should be clearly specified (explaining which specific joint actions will be taken and who is responsible). Recommendation 2: To Project Managers and Development Agenda Coordination Division, the Resource Planning, Programme Management and Performance Division on project monitoring (from Conclusion 1). (a) In order to improve regular reporting as a tool to take well-informed management decisions, self-evaluation reports should not only assess results against objectives by using objectively-verifiable indicators, but also regularly self-assess on-going relevance, efficiency and likelihood of sustainability. (b) For internal management purposes, financial reporting should link expenditures to budget lines and allocate them to different outcomes and project overhead cost. This would enhance transparency of financial reporting, provide managers with a sound basis for budgeting future projects, allow a bench marking of projects within the Development Agenda (DA) and provide the necessary information to assess efficiency of projects in detail. Recommendation 3: To Committee on Development and Intellectual Property (CDIP) on a follow-up project covering PLRs (from Conclusion 3). The evaluation recommends to Member States to favourably consider a proposal for a possible follow-up phase, focusing exclusively on completing the PLRs under preparation and on the further development of the concept, with a view to establish a regular service provided by the
CDIP/10/6 Annex, page 6 Global Infrastructure Sector that assists and advises on patent analytics, and coordinates the development of PLRs in a systematic way. A possible follow-up (if approved by Member States) may in particular: (a) Promote the concept among a broad range of possible users through IP offices, TISCs, universities, business support service providers and other institutions, with a particular focus on those in developing countries and Least Developed Countries (LDCs); (b) Support capacity building through TISCs (where appropriate);
(c) Refine the methodology through publicizing additional reports in other technical fields and using different formats; and (d) Enlarge the range of partners under particular consideration of cooperation with institutions in developing countries. Recommendation 4: To WIPO Senior Managers on establishing a system that allows WIPO to systematically track users of online service (from Conclusion 1). WIPO may consider establishing additional means to track users of its online services (e.g. through requiring users to register). This would provide information on who uses existing services as a basis to provide tailored information to specific target groups and to actively collect feed-back from them for the purpose of continuous improvement of its services. Recommendation 5: To WIPO Senior Managers to strongly promote e-learning tools in order to complement on-site training, where possible (from Conclusion 3). With a view on increasing efficiency of different training activities, WIPO may consider to reinforce existing efforts to complement on-site training involving high travel cost through interactive online courses, such as for example providing them through webinars, taking into account the infrastructure capacities of potential beneficiaries of such services. Recommendation 6: To the WIPO Global Infrastructure Sector on formalizing coordination with other Sectors. The WIPO Global Infrastructure Sector may further formalize with other Sectors their roles and responsibilities in the implementation of a future follow-up covering PLRs through agreements. Defining specific responsibilities to be assumed by each Programme and requiring a formal sign-off by the Programmes involved would help to ensure that coordination is less dependent on informal co-operation.
CDIP/10/6 Annex, page 7 I. BACKGROUND
1. This report covers the independent final evaluation of the Project “Developing Tools for Access to Patent Information” (DA_19_30_31_01) executed by WIPO, subsequently referred to as “the Project”. A. PROJECT BACKGROUND AND DESCRIPTION
2. The Project with a total budget 1,576,000 Swiss francs (of which 936,000 Swiss francs are non-personnel costs) was adopted in November 2009 at the fourth session of the CDIP in Geneva. It was formulated in response to recommendations 19, 30, and 31 of the DA 1 relating to facilitating access and use of patent-related information, knowledge and technology for developing countries, including LDCs. Implementation started in January 2010 and was planned to end in June 2012 (30 months). Main expected outputs included: (a) (b) The drafting of PLRs, The production of an e-tutorial for training on using/exploiting patent information and
(c) Training and information exchange on exploiting patent information through courses and workshops. 3. All the above outputs were to be delivered by the Global Infrastructure Sector (Global Information Services, now Services for Access to Knowledge and Information Division) in collaboration with other sectors of WIPO. 4. Main project purpose is to provide developing countries with demand-driven services to facilitate the use of patent information on specific technologies with the aim to spur indigenous innovation and research and development (R&D) of developing countries. Access to patent information by the public is an important element for achieving a balanced patent system. On the one hand, patent protection encourages investment into R&D, as the owner of a patent is granted exclusive rights to exploit the patented invention in the countries in which the patent is valid. On the other hand, information disclosed about an invention in a patent or patent application is generally made available to the public and to everyone who wants to further exploit a new technology. B. PURPOSE, METHODOLOGY AND LIMITATIONS OF THIS EVALUATION
5. Evaluation work was conducted by the IAOD of WIPO according to WIPO’s evaluation guidelines with the assistance of an external consultant 2 . 6. The work undertaken was guided by the ToRs dated June 2012 and the Inception Report dated August 3, 2012 approved by IAOD. 7. The evaluation covers the period from January 1, 2010 to September 15, 2012.
1
The DA was formally established in October 2007, to mainstream social and economic development into all activities of WIPO, and to keep development at the core of the IP System. In order to enhance the development dimension of the Organization’s activities, a set of 45 recommendations was adopted. The list of DA Recommendations can be found under the following link: http://www.wipo.int/ip-development/en/agenda/recommendations.html 2 Daniel P. Keller, Director, Swiss Consulting Co. Ltd, Hanoi - Vietnam
CDIP/10/6 Annex, page 8 8. Evaluation is part of WIPO’s regular oversight functions that are guided by the WIPO Internal Oversight Charter 3 . WIPO’s Evaluation Policy 4 , which is aligned to the Organization for Economic Co-operation and Development – Development Assistance Committee (OECD-DAC) evaluation criteria and quality standards 5 , provided the framework for this evaluation. (i) Key purpose
9. The key purpose of this evaluation was to assess whether the Project provided the right type of support in the right way based on four main evaluation criteria: (a) Relevance: The extent to which project objectives were consistent with beneficiaries’ requirements, member countries’ needs, global priorities and policies. (b) Efficiency: How economically inputs (e.g. funds, expertise, and time) were converted into results (“value for money”). (c) Effectiveness: The extent to which objectives were achieved, or are expected to be achieved, taking into account their relative importance. (d) Sustainability: assesses the likelihood of continuation of project benefits after the assistance has been completed. 10. Beyond this, the evaluator was also asked to look at WIPO’s contribution to fulfil the general objectives of the Development Agenda (DA) recommendations 19, 30, and 31. 11. The emphasis of this particular evaluation was on organizational learning. The evaluation approach was interactive and participatory. The process itself was designed to contribute to continuous improvement of WIPO’s services. The report further aims at providing input to the decision making of the CDIP and ensure accountability of WIPO towards its Member States. 12. Discussions with stakeholders were in general open and constructive. All persons interviewed openly shared information and exchanged views. The evaluators were able to work freely and without interference. All WIPO staff members supported the evaluation process actively and provided access to all relevant information. Overall, factual information obtained has been comprehensive, consistent and clear. Different evaluation tools were combined to ensure an evidence-based qualitative and quantitative assessment. Particular emphasis was given to cross-validation of data and an assessment of plausibility of the results obtained. The methodological mix included desk studies, literature review, individual interviews, interviews of focal groups and direct observation. 13. The key evaluation results presented below received endorsement by the project managers. (ii) Main limitations to this evaluation
14. Two of the key outputs (E-tutorial and most PLRs) have only recently been delivered. An assessment of the outcome level (the use of the Project’s outputs) was therefore only marginally
3
See WIPO Financial Regulations and Rules, Chapter 7, Rule 7.1, applicable from January 1, 2008, last revised on October 5, 2011. 4 WIPO, Revised Evaluation Policy, May 2010, in particular Annex 1 on evaluation criteria, which makes reference to the DAC Criteria of evaluating development assistance. 5 DAC Guidelines and Reference Series, Quality Standards for Development Evaluation, OECD-DAC, OECD 2010.
CDIP/10/6 Annex, page 9 possible. Looking at impact would be premature. The inclusion of indirect beneficiaries was only possible to the degree the Project had obtained feed-back from them. Fact finding focused on actors directly involved into the Project (different sections of WIPO, key direct beneficiaries and organizations WIPO has cooperated with). Furthermore, the conferences organized under the Project (Output 1) were mainly aimed at promoting and strengthening the TISCs. Assessing their effects on the TISCs would need to be done in conjunction with the relating project (DA_08_01), which was not within the scope of this evaluation. All of this necessarily limited the dimension and depth of the assessment. II. FINDINGS AND ASSESSMENT
This section presents the findings of the evaluation and provides an assessment of the Project against the evaluation criteria. A. (i) PROJECT PREPARATION AND MANAGEMENT Project identification
15. The Project was designed 6 in response to recommendations 19, 30, and 31 7 approved by Member States within the framework of the DA. 16. Specific aspects of those recommendations were addressed by the Project, while others were integrated into other DA projects or covered within the framework of existing WIPO services. 17. The Project was closely linked to project DA_08_01 (“Specialized Databases’ Access and Support”) by strengthening the TISCs. Except for the development of the e-learning tutorial developed under the Project, training programmes on patent information were generally delivered through project DA_08_01. Conferences organized under the Project (Output 3) mainly aimed at promoting the TISCs and were only marginally related to the two other outputs of the project (PLRs, e-tutorial). Given the close linkages to project DA_08_01, the conferences could equally have been funded under this project. 18. Aspects of Recommendation 19 are also implemented by different projects on IP, Technology Transfer and IP, Information Communication and Technology (ICT) and the Digital Divide, and Specialized Databases’ Access and Support (see DA Cluster B). 19. Apart from the dedicated PLR and TISCs website under the aforementioned DA projects, the PLR, the improvement of the PATENTSCOPE portal and the digitization and dissemination of patent information resources that contribute to addressing Recommendation 31 are also covered within WIPO’s regular technical capacity building activities. 20. The project document rightly identifies specific linkages to different strategic goals of WIPO Programmes 1 (Patents), 14 (Global IP Information Services) and 18 (IP and Global Challenges), without however translating them into specific joint activities 8 .
6 7
See Project Document (CDIP 4/6) Recommendation 19 (Cluster B): relates to facilitate access of developing countries to knowledge and technology to foster creativity and innovation and to strengthen relating existing activities within WIPO; recommendation 30 (Cluster C): promotes WIPO’s cooperation with other intergovernmental organizations to provide advice on how to make use of IP-related information on technology; recommendation 31 (Cluster C): aims at contributing to technology transfer to developing countries and LDCs through facilitating access to publicly available patent information.
CDIP/10/6 Annex, page 10 21. Defining specific responsibilities to be assumed by each Programme and requiring a formal sign-off by the Programmes involved would help to ensure that coordination is less dependent on informal cooperation. (ii) Use of project planning tools
22. A logical framework, the standard instrument for project preparation used by almost all development actors, was not an element of the standard DA template on the basis of which the project document was developed. Nevertheless, most information that is typically part of a logical framework is available in the project document. 23. Clear objectives derived from DA recommendations are broken down in specific outputs to be delivered and outcomes to be achieved. Expected outputs and outcomes are linked to specific, measurable, achievable, relevant and time-bound (SMART) performance indicators. Retrieving the information needed to assess outcomes according to some of the indicators would however require extensive surveys among a wide array of companies and policy makers, for which no resources were allocated in the project budget. 24. Clear objectives at the impact level 9 were not defined in the project document. This would not only be important to explain the wider benefits to beneficiaries, but also provide a basis for an impact assessment at a later stage. Since impact is what ultimately matters, it would be important to explain to which longer-term benefits projects aim to contribute. 25. A timeline of implementation is presented in a Gantt chart. Sequencing of activities is in general logical and appropriate. A delivery strategy in the project document clearly outlines the different steps of implementation. The planned project duration of 30 months for delivering all outputs was not commensurate with the limited personnel resources that were allocated. In retrospect it proved to be unrealistic for two WIPO staff(one of them not fully allocated to the Project) working on PLRs to identify their topics, draft the ToRs, commission and publish two high quality PLRs within only three months, as planned. Project duration seems to have been determined to fit the project into the regular biennial budget cycles rather than being based on a realistic estimate of time needed to deliver the outputs required. 26. Some but not all major risks and the way of mitigating them were identified. For instance the risk of not receiving proposals on PLRs from Member States, which subsequently materialized, was overlooked. Categorizing risks according to the likelihood they materialize and the degree of negative impact they may result in would help project managers to focus more closely on monitoring those risks that need to be controlled in order to achieve objectives. Moreover, project preparation did not look at assumptions 10 . 27. Financial planning based on the standard DA template was rudimentary and not in line with good project management practices. Presenting each budget item according to budget lines and outcomes would allow WIPO to compare budgets against expenditures in detail, which is needed for assessing efficiency of implementation. Furthermore, WIPO would be able to benchmark efficiency of projects internally and with similar interventions of other development
[Footnote continued from previous page]
See WIPO Program and Budget for 2010/2011. Program 1: Patents, Program 14: Global IP Information Services and Program 18: IP and Global Challenges 9 Impact: Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended (OECD, Glossary of Key Terms in Evaluation and Results Based Management, 2010). 10 Hypotheses about factors or risks which could affect the progress or success of a development intervention (OECD, Glossary of Key Terms in Evaluation and Results Based Management, 2010)
8
CDIP/10/6 Annex, page 11 actors. Last but not least, it would provide project managers with detailed information as a basis for accurate budgeting for similar future projects. (iii) Project Management
28. Project management was flexible in addressing arising challenges and tailoring the implementation strategy to changed circumstances: Progress reports indicate that management continued to actively identify and analyse risks arising during implementation. Regular monitoring and mitigation of risks is evidenced by the Project’s response to the lack of proposals from Member States for PLRs. The mitigation strategy chosen consisted in identifying topics in cooperation with different international organizations and non-governmental organizations NGOs). 11 Management also actively identified and exploited new cooperation opportunities and was responsive to requests received. 29. Management also looked for new opportunities to expand the reach of the Project. Moreover, the cooperation with other UN organizations in the area of patent landscaping (see table 1 below), which was also foreseen in the project document, is evidence of WIPO’s efforts to create synergies and complementarities within the UN system and with inter-governmental organizations (IGOs) and NGOs. B. RELEVANCE
Relevance assesses the extent to which project objectives were consistent with beneficiaries’ requirements, member countries’ needs, global priorities and WIPO’s policies. (i) Policy relevance
30. The fact that the Project was approved by the CDIP provides strong evidence for the Project’s relevance to Member States at the macro policy level. 31. The Project responds well to a number of international priorities, which have been defined by UN Member States, most of which are also members of WIPO. The topics selected for the PLRs are all of high relevance to international priorities and to the developing world in particular. Several of the areas covered by the PLRs potentially contribute to different Millennium Development Goals (MDGs). To be mentioned are in particular those relating to child mortality (MDG 4), maternal health (MDG 5), combating malaria, HIV/AIDS and other forms of diseases in developing countries and LDCs (MDG 6) and providing access to affordable essential drugs (MDG 8, Target 4). Other PLRs relate to MDG 7 (environmental sustainability), in particular to target 7C (access to clean drinking water). Two PLRs (one on solar cooking and another one on solar cooling) are potentially relevant to international efforts to reduce greenhouse gases under different initiatives. 32. Further mentioned should be the Project’s contribution to the UN’s efforts to better coordinate its services through cooperation, as explained in section 2.1 above. 33. The Project is also well aligned with WIPO’s policies, reflected by the Strategic Goals and Programmes 12 and capitalizes on WIPO’s core competencies.
11 12
See CDIP 6/2, Annex XIII, page 3: risk of demand and expertise for PLRs. In particular Strategic Goal I, Program 1; Strategic Goal IV, Programme 14 and Strategic Goal VII, Programme 18
CDIP/10/6 Annex, page 12 (ii) Relevance to target groups
34. PLRs: Relevance was confirmed through interviews with a sample of representatives from different target groups of direct users (one international organization, one NGO, one researcher and one patent practitioner). The evaluation also received indirect confirmation of positive feedback on the two PLRs on desalination and use of renewable energy, and on membrane filtration and UV water treatment, presented in an international conference (World Water Day). 35. E-tutorial: The need for training on using and exploiting patent information is evidenced by interviews, surveys of conference participants and initial e-tutorial users similar training material known to the evaluator either have only one section on patent information or explain only one specific internet search system. The e-tutorial is potentially highly relevant, but as it was only available at the end of the Project, it was not possible to validate this with a broader range of beneficiaries who have used the tool in practice. 36. Conferences: According to the surveys conducted at the end of the conferences 13 , participants confirmed the relevance of the topics covered for their work. C. EFFECTIVENESS
37. This section compares planned against expected results using the OVIs defined in the project document. As a basis, results reported by the Project were validated and assessed against key performance indicators defined in the project document. As mentioned under limitations in section 1 (B) above, outcomes was only marginally evaluable, since some PLRs and the e-tutorial have only recently or not yet been delivered. Short term effects of the training were measured through participants’ surveys. Outcome data, which is for instance required to assess the longer-term effects of the trainings or PLRs, was not yet available at the time of the evaluation. (I) Output 1: PLRs accessible on PATENTSCOPE® Website
38. The Project planned to make a total of 12 PLRs accessible through a dedicated Website with a “quality corresponding to the ToR requirements”. By September 15, 2012, a total of seven reports were published on the PLR website. The PLR on Ritonavir covers five instead of four innovation tracks (through a subsequent amendment covering Ritonavir pro-drugs). The PLR on vaccines combines the resources of two reports (one general part, one on vaccines for specific diseases). Two more PLRs had been commissioned and were under preparation. Topics and partner organizations for three more PLRs had been identified, but contracts to draft them not yet awarded. In retrospect, the target to make12 PLRs accessible via the Internet appears to have been overambitious and not achievable with the personnel and financial resources available. 39. In addition, 50 PLRs established by other organizations were also made available on the WIPO website 14 . 40. Due to the lack of specific proposals from Member States and their national institutions following circulars sent to the permanent missions in Geneva, the WIPO Secretariat initiated
See summary of participant evaluations in the reports established for each of the conferences. A compilation of links to publicly available PLRs in different areas prepared by various entities is available on (http://www.wipo.int/patentscope/en/programs/patent_landscapes/published_reports.html). PLRs established by the Project can be found on (http://www.wipo.int/patentscope/en/programs/patent_landscapes/ongoing_work.html)
14 13
CDIP/10/6 Annex, page 13 cooperation with other UN Agencies, IGOs and NGOs with expertise in the areas of technology, as described in the original project document (Public Health, Food and Agriculture, Environment and Climate Change, Disabilities). Those partners included one NGO from a developing country (African Agriculture Technology Foundation (AATF)). 41. The evaluation was unable to shed light on the reasons for the lack of inputs from developing countries, though it may be conjectured that low awareness on PLRs and their possible use was among the reasons. This indicates the need to promote PLRs in order to foster broader inputs to their planning and implementation. A possible forum to further promote PLRs would be the TISCs with their networks at the country level. 42. The cooperation with various IGOs and NGOs, including the World Health Organization (WHO), the Food and Agriculture Organization (FAO), Medicines Patent Pool, the International Renewable Energy Agency (IRENA), the Drugs for Neglected Diseases Initiative (DNDI), the AATF, the Global Institute for Water Environment and Health (GIWEH), the International Seed Federation (ISF), the International Telecommunication Union (ITU), the European Organization for Nuclear Research (CERN), the United Nations Environment Programme (UNEP), the World Meteorological Organization (WMO), Village Earth, and others has led to the identification of several specific topics for PLRs 15 as presented in Table 1 below. Table 1: Status of progress in publishing PLRs (as per 15 September 2012) Title of Report 1. Solar cooking 2. Solar cooling 3. Ritonavir 4. Atazanavir 5. Desalination and the use of alternative energies 6. Membrane Filtration and UV water treatment 7. Vaccines for selected diseases 8. Selected Neglected Diseases 9. Plant Salinity tolerance 10. E-waste recycling 11. Animal Genetic Resources 12. Accelerator technologies Cooperation Partner UNITAID/MPP UNITAID/MPP IRENA/GIWEH GIWEH WHO DNDI FAO, AATF & ISF UNEP FAO CERN Status Published Published Published Published Published Published Published Under finalization Under preparation Procurement stage ToR drafted ToR drafted
Source: Information provided by the Project on request 43. Topics were mostly selected based on those addressed by a specific project or programme of the collaborating partner institution. The fields covered relate to areas of potential interest to developing countries and LDCs (e.g. procurement of HIV medicines, access to clean water, renewable energy etc.). Selected partners interviewed confirmed their active involvement in the preparation of the ToRs for the reports and in the evaluation of the draft reports. While two of the partners interviewed had extensive experience in using patent information, for others the consultations on drafting the ToRs also served as a vehicle to familiarize themselves with broader aspects of patent information and protection for those partners with no prior knowledge in the field of patents. 44. PLRs are designed to identify what is patented, where and by whom. Each report further depicts the technology for which the patenting activity was investigated. Important for capacity building purposes is the disclosure of the patent search methodology, which enables the reader
See summary of discussions held in document “contacts and discussions for establishment of cooperation partners and preparation of landscaping reports (prepared by the Project)
15
CDIP/10/6 Annex, page 14 to learn about general search strategies (combination of classification and keyword search) or specific search strategies (e.g. various names used for pharmaceutical compounds). Commercially produced reports would typically not disclose the methodology in detail. 45. Reports facilitate technology transfer through identifying relevant patents. However, they do not include information about the actual legal status of those patents since such status naturally changes and requires verification. Each report includes meanwhile valuable patent family information, i.e. information regarding other jurisdictions where patent applications for the same invention have been filed. A company that wants to exploit a certain technology would have to verify its freedom to operate for each individual country based on this family information. Since patent data from many countries is not readily accessible through online patent registers, this task of determining the patent status of each family member remains time-consuming and expensive. WIPO in collaboration with other IPOs assists IGOs, NGOs or institutions in developing countries in obtaining up-to-date legal status data through WIPO's Patent Information Service (WPIS) for developing countries, and has provided such data, e.g. on several Ritonavir and Atazanavir related patents included in the respective reports. 46. The four published PLRs focus on providing facts and evidence rather than drawing strong conclusions or recommendations for policy makers, which might conflict with WIPO’s mandate as a specialised UN Agency. The evaluators received confirmation of the usefulness of the PLRs as input to policy documents at the international level drafted by technical specialists in the field. While PLRs could of course also be directly commissioned by partner organizations, WIPO enjoys the reputation of high technical competence and neutrality, which increases the credibility of the PLRs as source of reference. In some fields (e.g. pharmaceuticals), it might be interesting to summarize key findings derived from several reports in a format tailored to the needs of policy makers in developing countries. 47. The Project further delivered two additional (unplanned) outputs: the patent database for each of the reports is published on the website. These spread sheets contain all relevant patent families, on which the patent analysis was based, including hyperlinks to the complete patent documents. In addition, an interactive visualization of the patent analysis was made available. Users are able to explore various automatic statistical analysis options used in patent analysis. The tools on the website are highly useful. Moreover, a standardized procurement procedure for PLRs was developed in coordination with the Procurement and Contracts Section of WIPO. 16 48. All stakeholders interviewed unanimously confirmed the high quality of all PLRs that have so far been published. In-depth interviews with partner organizations confirmed that reports were used as a factual input to policy documents (e.g. the WHO’s Initiative for Vaccine Research) or to identify possible licensing arrangements for patents (e.g. Atazanavir and Ritonavir, both used for the treatment of HIV/AIDS). The quality and usefulness of the reports for users is also validated through survey results 17 . Respondents stated they had used PLRs mainly to learn more about patent information (21%), learn more about methodologies in patent search and analysis (26%), getting information about trends (21%), get information about specific technologies (16%) or get information about companies and people (10%). (II) Output 2: E-tutorial
49. By the end of August 2012 (three months later than planned), the e-tutorial was available online, while the production of DVDs is projected to be completed by the end of September
See as an example document “Call for Expression of Interest for the Request of Proposal No PTD/10/007” See survey Report dated August 20, 2012 established by the Project. Due to the small sample of 35 users, results are statistically not relevant. But the survey provides some indication on results at the outcome level.
17 16
CDIP/10/6 Annex, page 15 2012. According to own observation of the evaluators, the e-tutorial is of high quality. It seems to be particular useful for technical specialists with no prior experience in the field of IP. The modular design allows users to select those topics they are specifically interested in. Initial feed-back obtained through a test survey conducted by the Project confirmed this. 50. At this stage, an assessment of outcomes (use of the e-tutorial) is not yet possible. The e-tutorial focuses on practical knowledge and complements well the more theoretical courses of the WIPO. According to the interviews with the Project, the e-tutorial was developed in consultation with the WIPO Academy. 51. Delays in delivering this output had three main reasons: late start of preparation work, procurement issues of temporary nature (changes in procurement procedures led to delays in awarding the contract) and the complex task of developing an e-tutorial in coordination with various WIPO departments, external content providers and editors, and the application developer. (iii) Output 3: Regional conferences
52. Regional conferences promoting the TISC project and offering initial training on technology and innovation support and a platform to exchange institutional experiences in the field of innovation and best practices among countries were held in Singapore in May 2010 (40 participants from the Asia-Pacific region), in Addis Ababa in November 2010 (39 participants from the African region), in Moscow in December 2010 (32 participants from certain European and Asian countries), as well as in Buenos Aires in March 2011 (150 participants from Latin America and the Caribbean). A further regional conference, attended by 44 participants from the African Regional Intellectual Property Organization (ARIPO), was carried out in Harare in June 2011. 53. Participants were selected through invitation sent to heads of IP offices or their alternates in the respective regions, with guidance to nominate “a patent information specialist or person in charge of patent documentation and databases” as additional participants. According to the Project’s managers, selection of participants was based on the objective of raising awareness and developing skills among the management and technical staff of IP offices potentially involved in the establishment of TISCs and preparation or dissemination of PLRs. 54. There is no sign of a formal selection process for attendees by WIPO (which would be challenging), so it is not possible to draw any conclusions on whether the participants invited would be those who are able to practically apply what they have learned. An analysis of conference agendas and the material showed that the main benefit of the conferences was to promote the TISCs and to provide capacity building to their staff. The specific content of the conferences seems to have been tailored to the different needs in the regions covered, as evidenced by the different formats. Participants rated the conferences overall above the mark “good (4)”. 18 A survey conducted at the end of the conference indicated that the conferences had been “very useful” (4) for the daily work of participants and had generally be “useful” (3) or “very useful” (4) in raising their awareness and building their understanding of patent information in general, providing training, organizing and managing services, monitoring and evaluating services. The survey suggested that the conferences had been less useful for building an understanding of how to obtain resources. Detailed information on how the information provided in the conferences had been used (outcome level) was not available.
18
Survey results for conferences in Singapore, Addis Ababa, Moscow, Buenos Aires and Harare.
CDIP/10/6 Annex, page 16 (iv) No unexpected negative effects found
55. PLRs are a service that is provided by public institutions such as patent offices as well as private companies. The Project’s objectives, implementation strategy and outputs suggest that it produced a public good and did not compete directly with services offered by private companies. The topics covered by the PLRs produced under the Project related to areas of public interest. Moreover, privately commissioned reports would not be publicly accessible, and if, the methodology is typically not disclosed. Should WIPO decide to offer PLRs as a regular service on the free market, it would be important to differentiate pricing in a way that market rates are charged where WIPO services compete with private service providers. B. EFFICIENCY
56. Based on the official financial reports per end of December 31, 2011, 95.5% or 894,000 Swiss francs of the 936,000 Swiss francs allocated to the Project (excluding personnel costs) had been disbursed. Table 2 shows an analysis of non-personnel expenditures according to different budget lines. Table 2: Non-personnel expenditures according to outputs and budget lines (in 1,000 CHF) PLRs 8 8 293 e-Tutorial Conferences 3 336 3 125 211 138 108 94 All outputs 347 136 211 539 94 % of total 39% 15% 24% 60% 11%
Travel and fellowships Staff Missions Third-party Travel Contractual services Conferences Publishing Experts' Honoraria and Other Equipment and Supplies Furniture & Equipment Supplies and Materials Total % of total expenditures
293 8 8 309 35%
138 0
14 0
445 8 8 894 100%
50%
141 16%
444 50%
1% 100%
Source: Information provided by the Project (unofficial figures) and analysis by evaluators 57. This analysis shows that around half of the expenditures relate to promoting the TISCs, also financially supported under Project DA_08_01, one third on the PLRs, and around one eighth on the e-tutorial. Accordingly, it is not surprising that, as foreseen in the project document approved by Member States, almost 40% of the expenditures were spent for travel and missions (96% of travel costs are relating to the conferences). A large slice of the expert fees (66%) relate to the PLRs and the e-tutorial (31%), whose development was mostly outsourced. 58. PLRs: Costs per PLR were significantly higher than expected, evidenced by the smaller number of reports produced than planned, given the higher complexity of the subject matter and expected depth of analysis involved in their preparation, according to the Project managers. Consequently, the completion of all 12 PLRs (see table 1) would not be feasible without
CDIP/10/6 Annex, page 17 additional funding. The average cost 19 for commissioning one PLR as reported by the Project was around 31,780 Swiss francs. 59. E-tutorial: Relating the potentially high number users to the costs, the e-tutorial is a highly efficient way of capacity building. 60. Conferences: Assuming that the number of 306 participants reported by the Project (see 2.2 above) is correct, the cost per conference participant would be 1’010 Swiss francs. While awareness raising and networking are important for the building of the TISC network, the evaluators strongly endorse WIPO’s efforts to complement resource-intensive on-site training through internet-based tools for seminars (e.g. webinars), wherever this is possible without jeopardizing the quality of capacity building efforts. 61. Due to the lack of a detailed result-based budget in the project document 20 , a direct comparison between budget and expenditures per category and output was not possible. E LIKELIHOOD OF SUSTAINABILITY OF RESULTS
62. Sustainability assesses the likelihood of continuation of project benefits after the assistance has been completed. 63. E-Tutorial: The sustainability of the e-learning tool is rather obvious. It will continue to be available on-line and WIPO does have the technical and financial capacities to maintain it. 64. International conferences: The evaluation was unable to assess the sustainability of the outcomes achieved by the conferences, which mainly aimed at strengthening the TISCs. This question depends on the sustainability of the TISCs, which exceeds the scope of this evaluation (see also comments on limitations in section 1 (B) above) 21 . 65. PLRs: The sustainability of PLRs and relating capacity building as a service depends on its integration in the portfolio of regular WPO services. The intention of integrating PLRs into the DA was to test demand, institutionalise, improve the concept and ultimately take a decision on whether to mainstream PLR services under WIPO’s regular budget. While prior to publishing PLRs under a DA project, several thematic departments had already undertaken similar work, a decision was taken to entrust PLRs to the Global Infrastructure Division. The evaluation found emerging evidence of two advantages of centralizing PLRs within the Global Infrastructure Division as done under the Project: Firstly, drafting PLRs and advising partners on defining ToRs requires a high specialization in the field of patent search methodology. Secondly, centralizing the service reduces the risk that the selection of patent reports to be published is driven by WIPO’s own internal needs and interests rather by those of possible clients. On the other hand, the Global Infrastructure Division lacks in-depth knowledge in certain thematic fields. Therefore, coordination with thematic departments at an early stage of drafting the ToRs for the reports for their inputs and strengthening cooperation with external partners with expertise in the relevant fields remains crucial. 66. Before taking a decision whether and in which way WIPO provides PLR services, further testing of the concept and the demand is needed. While demand of international organizations is obviously given and likely to continue, it will be important to explore the demand of institutions in developing countries and LDCs. This might also allow covering subjects that are of particular
Total cost of the subcontracts for producing the nine reports that have either been published or are under preparation was 286,021 Swiss francs, contract values ranging from 16,212 Swiss francs to 63,400 Swiss francs. 20 See budget CDIP/4/6 Annex, page 9, which presents details of non-personnel cost only according to staff missions, third-party travel and experts’ honoraria (contractual services). 21 Project DA 8_01 was evaluated, but the report (CDIP 9/5 April 2012) does refer to input received from the Project.
19
CDIP/10/6 Annex, page 18 national and/or regional relevance. More awareness rising and capacity building is needed to create demand among a broader range of users and refining the concept of PLRs. This will require a follow-up phase, after which a final decision should be taken on whether or not provide PLRs as a regular service. A proposal is currently in the initial stage of preparation.
CDIP/10/6 Annex, page 19 III. CONCLUSIONS
67. Based on the findings and assessment above, the evaluation draws the following conclusions: Conclusion 1: The Project was generally well prepared and managed, but there is room for further enhancing existing tools for planning, monitoring and evaluating projects. The Project made proper use of existing planning and monitoring tools, but there is room to improve the tools in consideration of internationally recognized best practices in managing development assistance. While the project document does not include a logical framework, it contains most of its information, such as clearly defined objectives at different levels linked to objectively verifiable performance indicators. Yet, assumptions on external factors that need to be in place to achieve each of the results and objectives at the impact level are not defined. 68. The involvement of other WIPO sectors and links to WIPO Programmes are well explained, without however specifying concrete actions and responsibilities. Good coordination with other WIPO departments and external partners was more the result of personal initiatives than of a systematically planned approach. 69. Progress reports provide evidence that management actively mitigated most risks identified at the planning stage and additional ones that became apparent during implementation. The most important risk not identified, which subsequently materialized, was a possible lack of inputs from Member States on the topics of PLRs. 70. Unlike for programmes, WIPO does not require result-based financial budgeting and reporting for projects. This Project established for managerial purposes a financial report that links expenditures to budget lines and outputs, which has been identified as a good practice. 71. The Project made significant efforts to use well designed surveys for a systematic monitoring of quality and use of outputs. This was however not possible for users who downloaded PLRs from the dedicated PLR website. The current statistics of WIPO’s Internet Services Sector provide only information related to the number of unique page views, the geographical distribution of the page visitors, along with the number of downloads of each of the PLRs. They do not provide detailed and unambiguous information about who accessed a particular resource and how and for what purpose it was subsequently used. Systematically retrieving contact details about users of on-line services would allow WIPO to profile its clients and to conduct online surveys as a basis for specifically tailoring its services to different target groups. Conclusion 2: The project design was overambitious, especially for achieving the objectives set for the PLRs. 72. The planned time of 30 months to achieve the objectives, in particular producing 12 PLRs, was not commensurate and seems to have been determined by the need to align the project duration to biennial budgeting cycles rather than the time required to deliver the results. The number of PLRs the Project was actually able to publish (see conclusion 3) is evidence of this. Conclusion 3: While the Project generally provided the right type of support in the right way, not all of its expected outputs (PLRs, e-tutorials, regional conferences) were delivered. 73. PLRs: While only seven instead of 12 PLRs as foreseen in the project document were published as of 15 September 2012, users considered them of high quality and generally useful for their work. Two more PLRs have been commissioned and are under preparation. As an additional output, various innovative and useful tools to analyse information of the PLRs and to directly access patent information used in the reports were made available on the Internet. In
CDIP/10/6 Annex, page 20 addition, the Project refined their concept and developed standard procedures and documentation for commissioning work from outside suppliers. Key value added of the Project was the capacity building element through disclosure of the methodology and cooperation with different partners in drafting ToR. Evidence provided through interviews suggests that some of the reports were used mainly as an input to the work of other organizations, including in policy making. Due to the lack of direct inputs from Member States, the Project exclusively cooperated with different international, inter-governmental and non-governmental organizations. Topics selected covered areas of high relevance to internationally agreed priorities, including the Millennium Development Goals. 74. Expanding the direct benefits to developing countries would call for a second phase, which would also allow further testing, refining and disseminating of the methodologies. Longterm sustainability will require pursuing the Project’s efforts to prepare for providing PLRs as a regular service of WIPO’s Global Infrastructure Sector. 75. The e-tutorial is of high quality and potential relevance to users, especially to those with a technical background, but no prior knowledge in the field of patents. It complements existing tools, which are either not comprehensive or are oriented towards the use of specific databases. 76. The fact that it became only available after the official ending date of the Project had three main reasons: late start of preparation work, procurement issues (changes in procurement procedures led to delays in awarding the contract), and the complex task of developing an etutorial in coordination with various WIPO departments, external content providers and editors, and the application developer. 77. Participants rated the quality of regional conferences as highly relevant to their work. The conferences mainly promoted the TISCs, also supported by project DA_8_01. Furthermore, basic capacity building, mainly to staff of IP Offices, was provided. Conferences were only marginally linked to the e-tutorial and the PLRs. Assessing long-term benefits at the outcome level and their sustainability depends on whether and how the TISCs will use the information provided. Looking at this was neither required nor possible within the scope of this evaluation. While awareness raising and networking is important at the initial stage of technical support, WIPO has rightly recognized the need to also provide training and information exchange through modern online tools (e.g. webinars) to complement more resource-intensive means of skills development such as on-site training. IV. RECOMMENDATIONS
Recommendation 1: To Project Managers, the Development Agenda Coordination Division and Resource Planning, Programme Management and Performance Division on project planning (from Conclusions 1 and 2): 78. Wherever possible, project duration should be determined based on a reasonable estimate of the time required to achieve each expected result rather than based on the duration of funding cycles. 79. In addition to defining expected outputs and outcomes and linking them to objectively verifiable indicators (OVIs), as done, longer-term changes the Project is expected to contribute to (impact level) as well as related OVIs should be formalized. 80. Where monitoring and/or self-assessment of results by projects requires significant resources (e.g. comprehensive studies), they need to be included into the budget.
CDIP/10/6 Annex, page 21 81. As a basis for internal result-based financial monitoring, project documents should include a result-based budget that allocates expenditures according to budget lines (e.g. travel) to each of the expected outcomes and to project management cost (overhead). 82. Besides identifying risks and the way of mitigating them, risks should be rated according to the degree of their potential negative impact on achieving results and the likelihood they materialize. 83. Project documents should include assumptions (external conditions that need to be in place to achieve objectives). 84. Coordination within WIPO and other organizations should be clearly specified (explaining which specific joint actions will be taken and who is responsible). Recommendation 2: To Project Managers and Development Agenda Coordination Division, the Resource Planning, Programme Management and Performance Division on project monitoring (from Conclusion 1) 85. In order to improve regular reporting as a tool to take well-informed management decisions, self-evaluation reports should not only assess results against objectives by using objectively-verifiable indicators, but also regularly self-assess on-going relevance, efficiency and likelihood of sustainability. 86. For internal management purposes, financial reporting should link expenditures to budget lines and allocate them to different outcomes and project overhead cost. This would enhance transparency of financial reporting, provide managers with a sound basis for budgeting future projects, allow a bench marking of projects within the DA and provide the necessary information to assess efficiency of projects in detail. Recommendation 3: To CDIP on a follow-up project covering PLRs (from Conclusion 3) 87. The evaluation recommends to Member States to favourably consider a proposal for a possible follow-up phase, focusing exclusively on completing the PLRs under preparation and on the further development of the concept, with a view to establish a regular service provided by the Global Infrastructure Sector that assists and advises on patent analytics, and coordinates the development of PLRs in a systematic way. 88. A possible follow-up (if approved by Member States) may in particular:
89. Promote the concept among a broad range of possible users through IP offices, TISCs, universities, business support service providers and other institutions, with a particular focus on those in developing countries and LDCs; 90. Support capacity building through TISCs (where appropriate);
91. Refine the methodology through publicizing additional reports in other technical fields and using different formats; 92. Enlarge the range of partners under particular consideration of cooperation with institutions in developing countries.
CDIP/10/6 Annex, page 22
Recommendation 4: To WIPO Senior Managers on establishing a system that allows WIPO to systematically track users of online service (from Conclusion 1) 93. WIPO may consider establishing additional means to track users of its online services (e.g. through requiring users to register). This would provide information on who uses existing services as a basis to provide tailored information to specific target groups and to actively collect feed-back from them for the purpose of continuous improvement of its services. Recommendation 5: To WIPO Senior Managers to strongly promote e-learning tools in order to complement on-site training, where possible (from Conclusion 3) 94. With a view on increasing efficiency of different training activities, WIPO may consider to reinforce existing effortsto complement on-site training involving high travel cost through interactive online courses, such as for example providing them through webinars, taking into account the infrastructure capacities of potential beneficiaries of such services. Recommendation 6: To the WIPO Global Infrastructure Sector on formalizing coordination with other Sectors 95. The WIPO Global Infrastructure Sector may further formalize with other Sectors their roles and responsibilities in the implementation of a future follow-up covering PLRs through agreements. Defining specific responsibilities to be assumed by each Programme and requiring a formal sign-off by the Programmes involved would help to ensure that coordination is less dependent on informal co-operation.
CDIP/10/6 Annex, page 23
LIST OF ANNEXES (AVAILABLE IN THE FULL VERSION OF THE REPORT) Project document Annex 1 Inception report Annex 2 List of persons met Annex 3 List of documents Annex 4
[End of Annex and of document]
doc_248618044.pdf