EVALUATION GUIDELINES

Transcription

1 EVALUATION GUIDELINES

2 Introduction This document outlines the reviewing process and defines the responsibilities of the participants in the process. It details a number of important issues, such as: FCT s mission, objectives of the present Call and application components, evaluation criteria, scoring system, evaluation process, feedback, confidentiality and conflict of interest. FCT S MISSION Fundação para a Ciência e a Tecnologia, (FCT), the Portuguese Foundation for Science and Technology, is the Portuguese public Research Council that funds all scientific areas of science and technology. FCT promotes excellence, innovation and international competitiveness across all areas of scientific research. FCT supports, funds and assesses people (including grants fellowships, studentships and contracts), ideas (research grants) and internationally competitive research centres. FCT aims to promote research talent through sustainable advanced training and scientific careers of excellence; foster international competitiveness and visibility of scientific research and innovation carried out in Portugal; encourage knowledge transfer between R&D centres and businesses; facilitate access of the scientific community to state-of-the-art infrastructures and support the development of internationally leading research centres. FCT s main functions are: to promote, evaluate, fund and accompany research units, programmes, projects and qualification of human resources; to promote and support infrastructures for scientific research and technological development; to promote the diffusion of scientific and technological culture and knowledge; to stimulate the updating, interconnection, reinforcement and availability of science and technology information sources. FCT funds all areas of knowledge, including exact, natural and health sciences, engineering, social sciences and humanities. RESEARCH INFRASTRUCTURES OF STRATEGIC RELEVANCE

3 CALL FOR RESEARCH INFRASTRUCTURES AND THEIR INTEGRATION IN THE NATIONAL ROADMAP OF RESEARCH INFRASTRUCTURES OF STRATEGIC RELEVANCE In 2013, FCT launches a call aiming to: a) evaluate national Research Infrastructures, in order to create a national Roadmap for infrastructures of strategic interest; b) develop a strategic plan for investment in Research Infrastructures up to 2019, promoting synergies and overcoming redundancies; c) prioritize funding, identify areas and, when possible, potential beneficiaries; d) develop a national database of Research Infrastructures. The selected Research Infrastructures will be included in the first National Roadmap for Research Infrastructures to be published in Portugal. This call will impact on the national research and innovation system and on its sustainability and implementation and must be articulated with the regional action plans for research and innovation across the country, if already defined. Definition of Research Infrastructure (RI) A Research Infrastructure (RI) is here defined as an organizational system used by the scientific community to conduct top-level research and innovation in their respective fields. It may include large scientific equipment or sets of scientific instruments, collections and other knowledge based resources, data files and scientific data, computational and programming systems, communication networks that promote digital open access as well as other infrastructures of a unique nature, essential to achieve excellence in research. Research infrastructures can be single-sited or distributed as organized resource networks. To be included in the Roadmap, a Research Infrastructure must have: professional management that guarantees implementation of an action plan and the accomplishment of the specific aims there defined, with an efficient and transparent internal management of resources; capacity to relate with, and provide services to, the scientific, educational, business and industrial communities; clear, well defined and widely advertised policy of access conditions to national and international researchers external to the infrastructure, integrated in their aims and action plan. Eligibility The following institutions are eligible to apply: a) Public or Private National Higher Education Institutions; b) State Laboratories or International Laboratories based on national territory; RESEARCH INFRASTRUCTURES OF STRATEGIC RELEVANCE

4 c) Private non-profit institutions whose main objective is to carry out S&T activities; d) Other public and private non-profit institutions which carry out or participate in scientific research activities. Each application must have a Principal Investigator PI and an implementation team, which will ensure the adequate implementation of the work plan and other activities that are explicit in the application. One PI cannot submit more than one application. In the case of applications that include more than one institution, a copy of the consortium contract or equivalent collaboration agreement is compulsory. The equipment resources available in a given institution can only be included in one application. Scope Applications are to be submitted within seven thematic areas: a) Social sciences and humanities; b) Physical sciences and engineering; c) Environmental sciences; d) Biological and medical sciences; e) Materials and analytical facilities; f) Energy; g) E.infrastructures. APPLICATION COMPONENTS Applications are submitted online via a dedicated FCT Web application. The application form is organized in 2 parts, both compulsory, broken down into the following components: A - Characterization of the Research Infrastructure (RI) 1. Identification i. Name ii. Acronym iii. Website iv. Type (Distributed, Single sited, Virtual) v. Institutional partners (nodes) vi. Research units vii. Main thematic area viii. Complementary scientific area(s). RESEARCH INFRASTRUCTURES OF STRATEGIC RELEVANCE

5 2. Description i. General description ii. Keywords iii. Status (operational/not operational) iv. Equipment/resources v. Research services vi. Integration in International Research Infrastructures vii. Average users per year (National, EU, non-eu) viii. Access conditions 3. Management structure i. Identification of the management team ii. Brief description of the management structure 4. SWOT Analysis (strengths, weaknesses, opportunities and threats) B - Research Infrastructure Proposal Strategic action, diagnosis of future investment needs and potential funding sources 5. Scientific aims 6. Action plan 7. Identification of the team responsible for implementing the proposed action plan 8. Expected results and impact i. Training indicators ii. National/Regional socio-economic impact iii. Other relevant impact indicators 9. Multi-annual financing plan (budget and funding sources) RESEARCH INFRASTRUCTURES OF STRATEGIC RELEVANCE

6 Evaluation Criteria The evaluation and selection process considers both scientific merit and strategic relevance of the research infrastructure. The scientific panel focuses on the scientific quality and merit of the infrastructure; and the strategic panel on the strategic relevance of the infrastructure. The latter will mainly focus on the applications articulation with regional and national political priorities, in order to ensure the financial sustainability and maximum impact of the investment. The strategic panel will only assess the applications after the scientific merit panel has finished its work and will consider its result as a key basis for any further recommendations. The criteria and respective sub-criteria for each component of the evaluation are detailed below. SCIENTIFIC ASSESSMENT A - Scientific and technological excellence of the RI The significance of the RI for the specific research fields, including: o Relevance of the scientific objectives of the RI to facilitate and promote top-level science in Portugal; o Capacity of providing potential for world class research and scientific breakthrough; o Expected benefits for the Portuguese scientific and technological system for conducting cutting edge research at an international level, namely to increase the participation in international collaborative research projects, such as, those of the Horizon 2020; Adequate identification of the RI's strengths, weaknesses, opportunities and threats (SWOT analysis); Degree of internationalization, including the integration in international RI initiatives, namely those of the European Strategy Forum for Research Infrastructures (ESFRI) Roadmap; Degree of interdisciplinarity, including the effect of the RI on strengthening interdisciplinary research in Portugal; Quality of the proposed training of researchers. B Governance capacity and implementation feasibility Degree of adequacy of the management structure and governance of the RI to the proposed scientific aims; Adequate management and action plan implementation including: o Leadership; o Distribution of responsibilities; o Experience and capacity; o Identification of (and adequate strategy to address) RI's strengths, weaknesses, opportunities and threats (SWOT analysis); Competence and complementarities of the nodes and added value of the national RI at the regional, national and international levels, including contribution to increase access to knowledge resources and scientific capacity in the field of operation of the RI; RESEARCH INFRASTRUCTURES OF STRATEGIC RELEVANCE

7 Adequate equipment and relevance of improvements to the existing and/or acquisition of new equipment, considering the scientific aims of the RI; Quality of the access policy and data management plan: o Transparent policy for access to the infrastructure, including international access activities, conditions for provision of access, addressing remote access needs in relation to availability of e-infrastructures and data management issues; o Access policy for industry (addressing IP rights - if applicable - fees and confidentiality issues); Operational readiness: maturity of the RI and appropriate relations between partners of the infrastructure and, if relevant, of the integration in an international research infrastructure. C Budget and sustainability Technical feasibility, human resource costs and cost-effectiveness of the proposed infrastructure, based on adequacy of: o Requested funding and envisaged sources of funds; o Multi-annual budget plan with funding sources information; o Long-term sustainability plan of the investment. Adequate identification of (and answer to) RI's strengths, weaknesses, opportunities and threats (SWOT analysis); The scientific criteria presented will be weighted as follows: A) Scientific and technological excellence of the RI - 50%; B) Governance capacity and implementation feasibility - 25%; C) Budget and sustainability - 25%. STRATEGIC ASSESSMENT Strategic assessment will focus on the articulation of the RI proposal with national sectorial policies and regional strategies and priorities, taking into account the following criteria: D RI s contribution to the regional and/or national development strategy Degree of adequacy of the RI proposal to the national policies; Integration of the proposal in sectors / technologies considered key to one or more regions (NUT II), in articulation with the smart specialization objectives defined by each region for the structural funds programming period. E - RI's contribution to the strengthening of national and international competitiveness Potential of the research infrastructure to become a national and international scientific and technological reference hub as a service provider; RESEARCH INFRASTRUCTURES OF STRATEGIC RELEVANCE

8 Potential of the research infrastructure to increase the industrial knowledge base and innovation capacity; Strategic anchor effect of the infrastructure for the emergence of new research and technology initiatives. F - Potential for social and economic development and for the implementation of public policies on science and technology RI s contribution to the growth and consolidation of national and regional scientific competences; Degree of engagement and impact on regional and national stakeholders activities. RESEARCH INFRASTRUCTURES OF STRATEGIC RELEVANCE

9 Scoring System The current FCT scoring system uses a 9-point scale: Impact Score Additional Guidance on Strengths/Weaknesses 9 Exceptionally strong with essentially no weaknesses High 8 Extremely strong with negligible weaknesses 7 Very strong with only some minor weaknesses 6 Strong but with numerous minor weaknesses Medium 5 Strong but with at least one moderate weakness 4 Some strengths but also some moderate weaknesses 3 Some strengths but with at least one major weakness Low 2 A few strengths and a few major weaknesses 1 Very few strengths and numerous major weaknesses Minor weakness: An easily addressable weakness that does not substantially lessen impact. Moderate weakness: A weakness that lessens impact. Major weakness: A weakness that severely limits impact. A score of 9 indicates an exceptionally strong application with essentially no weaknesses. A score of 1 indicates an application with serious and substantive weaknesses with very few strengths. Impact, regards the research infrastructure likelihood to have a sustained, influence or strong impact: High impact = 7 to 9; Medium impact = 4 to 6; Low impact = 1 to 3. RESEARCH INFRASTRUCTURES OF STRATEGIC RELEVANCE

10 Evaluation Process The evaluation process of eligible applications comprises two stages: scientific and strategic. For each stage a distinct panel will carry out differentiated readings towards a final evaluation and selection. The scientific merit and quality of the research infrastructure will be evaluated by an international panel of experts. The panel will be subdivided in seven sub-panels, corresponding to the thematic areas defined in the Call. One member of each sub-panel will be designated as the thematic coordinator. A national strategic relevance panel will focus on key criteria of national and regional policy coherence and potential socio-economic impact at national and regional levels. The reading of the strategic relevance panel will only take place after the scientific panel assessment and will consider its result as the basis for any further recommendations SCIENTIFIC ASSESSMENT All applications will be subjected to scientific evaluation and will be distributed by 7 sub-panels, which are responsible for the preliminary remote reviewing. This distribution is in accordance with the 7 thematic areas corresponding to those defined in the Call, complemented with the scientific areas indicated in the application. Each application will be remotely assessed by 3 panel members. Remote panel members will produce individual evaluation reports for each application (about 10 to each panel member, accordingly to their field of expertise), and a consensus report drafted by one of the 3 readers and validated by all readers before being forwarded to the final Evaluation Panel Meeting. Individual remote report includes: The scores of each evaluation criterion and sub criterion; A global average weighted score, based on the scores of each criterion; A succinct but substantial explanatory global comment addressing the extent to which the proposal actually meets the criteria and that explains the evaluator s overall judgment on the proposal. Consensus report includes: The consensus scores for each evaluation criterion; A global average weighted score, based on the scores of each criterion; A succinct but substantial explanatory consensus global comment, based on the 3 individual reviews submitted beforehand; An overall score, that is not a direct result of the individual scores; Recommendation for integration of the RI in the roadmap; RESEARCH INFRASTRUCTURES OF STRATEGIC RELEVANCE

11 Specific directions and suggestions for the final evaluation panel meeting not to be transmitted to the applicants. These suggestions can, for example, take the form of recommendations about possible fusion of RIs in larger partnerships or elimination of parts of the RI, or recommendations to include selected RI in national or international networks of RI; Confidential comments to FCT, if necessary. Final Scientific Evaluation Panel Meeting The final Scientific Evaluation Panel Meeting will take place in Lisbon and should gather the 7 thematic coordinators. For the final meeting one of the thematic coordinators will be designated the panel chair. The panel chair will have the added duties of coordinating and moderating the meeting, of elaborating the panel report, and of conveying the results of the discussions to the Board of Directors of FCT. The thematic coordinators will validate all consensus reports from their thematic field, prior to the final meeting and will be fully acquainted with those applications to be presented and discussed during the final meeting. The main aims of the Panel Meeting are to: Ensure that each application receives a fair judgment and is discussed appropriately; Produce a consolidated ranking list of the applications; Define the quality threshold above which the proposals will be included in the Roadmap; Elaborate a short final evaluation report of each application, to be made available to the applicants; Recommend changes in the proposed RI, if needed. Final evaluation report includes: The final scores for each evaluation criterion; An overall score that can be independent from the average scores for each criterion; A succinct but substantive explanatory final comment including eventual recommendations for changes in the proposed RI; Recommendation for integration of the RI in the roadmap; Confidential comments to FCT, if necessary. STRATEGIC ASSESSMENT After the scientific assessment, applications will be submitted to a strategic relevance evaluation. These applications will be assessed by at least one national level ministerial representative and one regional level representative. National level representatives will be defined in accordance with the main thematic domain of the research infrastructure. Regional level representatives will be defined according the territorial implementation of the RI. RESEARCH INFRASTRUCTURES OF STRATEGIC RELEVANCE

12 The readings of the strategic relevance panel members will take the form of qualitative reports, with the main aim of analyzing coherence with national and regional strategies and policies. These reports may include eventual recommendations for a better alignment of the RI with national and regional policies and priorities. FCT, as chair of the strategic relevance panel, will compile the final evaluation report to be transmitted to the applicants, including the decision of inclusion in the Roadmap. RESEARCH INFRASTRUCTURES OF STRATEGIC RELEVANCE

13 Feedback to Applicants FEEDBACK REPORT TO APPLICANTS Applicants will receive the final evaluation report, including scientific merit and strategic relevance panel comments and possible recommendations, positioning the research infrastructure in one of following three categories of priority: Category 1 - those that have demonstrated high scientific potential and are considered to have high strategic regional and/or national relevance; Category 2 those that have demonstrated high scientific potential and quality but still need further work on the implementation and sustainability capacity and have not been considered closely articulated with the highest priority regional and national policies; Category 3 those that have demonstrated medium or low scientific quality or impact and/or have been considered to have or demonstrated none or weak relation with regional and national policies and priorities. Members of the evaluation committee are encouraged to observe the following additional guidelines: Avoid comments that give a description or a summary of the proposal. Avoid the use of the first person or equivalent: "I think " or "This reviewer finds. Always use dispassionate and analytical language: avoid dismissive statements about the applicant, the proposed science, or the scientific field concerned. Avoid asking questions, as the applicant will not be able to answer them. Evaluate the proposed work and not the work you consider should have been proposed. In the case of a very large number of proposals, some standardization of the comments may be implemented. RESEARCH INFRASTRUCTURES OF STRATEGIC RELEVANCE

14 Confidentiality and Conflict of Interest CONFIDENTIALITY The confidentiality of written proposals must be protected. All experts involved in the evaluation are asked not to copy, quote or otherwise use material from them. They are requested to sign a statement of confidentiality relative to the contents of the project proposals and to the results of the evaluation. The text to be accepted, which appears the first time each member of the evaluation committee uses his/her username and password to access the evaluation area, is the following: STATEMENT OF CONFIDENTIALITY Thank you for accepting to participate in the scientific evaluation of RI's proposals submitted to the Portuguese Foundation of Science and Technology (Fundação para a Ciência e a Tecnologia) FCT. The reader of this message pledges, on his/her honor, not to quote or use in any way, the contents of the applications, nor to make available, other than to FCT or to the Evaluation Panel, the results of the evaluation. CONFLICT OF INTEREST (COI) Any CoI must be declared prior to the evaluation process. No reviewer shall make an individual review of a proposal if in CoI with it. Circumstances that could be interpreted as a disqualifying conflict of interest are laid down in the following criteria: 1. First-degree relationship, marriage, life partnership, domestic partnership with the applicant(s); 2. Personal interest in the application's success or financial interest by persons listed under no.1; 3. Current or planned close scientific cooperation; 4. Dependent employment relationship or supervisory relationship (e.g. teacher-student relationship up to and including the postdoctoral phase) extending five years beyond the conclusion of the relationship with the applicant(s); 5. The affiliation or pending transfer to this or to a participating institution; 6. Researchers who are active in a council or similar supervisory board of the applying institution are excluded from participating in the review and decision-making process for applications originating from this institution; RESEARCH INFRASTRUCTURES OF STRATEGIC RELEVANCE

15 A potential conflict of interest may exist, even in cases not covered by the clear disqualifying conflicts indicated above, in the following circumstances: 7. Relationships that do not fall under no. 1, other personal ties or conflicts with the applicant(s); 8. Financial interests of persons listed under no. 7; 9. Participation in university bodies other than those listed under no. 6, e.g. in scientific advisory committees in the greater research environment; 10. Research cooperation within the last three years, e.g. joint publications; 11. Preparation of an application or implementation of a project with a closely related research topic (competition); 12. Participating in an on-going scientific or inter-personal conflict with the applicant(s). In the latter case, FCT will make a decision whether the situation in question constitutes an actual CoI or whether no CoI exists. RESEARCH INFRASTRUCTURES OF STRATEGIC RELEVANCE

16