Evaluation of Genome Canada

June 2020

This publication is available online at http://www.ic.gc.ca/eic/site/ae-ve.nsf/eng/h_03914.html

To obtain a copy of this publication or an alternate format (Braille, large print, etc.), please fill out the Publication Request Form at www.ic.gc.ca/Publication-Request or contact:

Web Services Centre
Innovation, Science and Economic Development
Canada
C.D. Howe Building
235 Queen Street
Ottawa, ON K1A 0H5
Canada

Telephone (toll-free in Canada): 1-800-328-6189
Telephone (Ottawa): 613-954-5031
TTY (for hearing-impaired): 1-866-694-8389
Business hours: 8:30 a.m. to 5:00 p.m. (Eastern Time)
Email: info@ic.gc.ca

Permission to Reproduce

Except as otherwise specifically noted, the information in this publication may be reproduced, in part or in whole and by any means, without charge or further permission from Innovation, Science and Economic Development Canada, provided that due diligence is exercised in ensuring the accuracy of the information reproduced; that Innovation, Science and Economic Development Canada is identified as the source institution; and that the reproduction is not represented as an official version of the information reproduced, nor as having been made in affiliation with, or with the endorsement of, Innovation, Science and Economic Development Canada.

For permission to reproduce the information in this publication for commercial purposes, please fill out the Application for Crown Copyright Clearance at www.ic.gc.ca/copyright-request or contact the Web Services Centre mentioned above.

© Her Majesty the Queen in Right of Canada, as represented by the Minister of Innovation, Science and Economic Development Canada, 2019.

Cat. No. Iu-4-318/2020-E-PDF
ISBN 978-0-660-35759-1

Aussi offert en français sous le titre Genome Canada Rapport D'Evaluation.

List of acronyms

ARC
Average of Relative Citation
CFI
Canada Foundation for Innovation
CIHR
Canadian Institutes of Health Research
GC
Genome Canada
GE3LS
Genomics, ethical, environmental, economic, legal and social implications
HQP
Highly Qualified Personnel
ISED
Innovation, Science, and Economic Development Canada
NSERC
Natural Sciences and Engineering Research Council of Canada
OECD
Organization for Economic Cooperation and Development
SIAC
Science and Industry Advisory Committee
SSHRC
Social Sciences and Humanities Research Council of Canada

List of figures

  1. Figure 1: Percentage of Genome Canada Funding and Co-Funding Directed to Strategic Sectors from 2000 to 2019
  2. Figure 2: Percentage of Genome Canada Funding and Co-Funding Directed to Programs from 2000 to 2019
  3. Figure 3: Total Annual Value of Genome Canada Funding (in millions of $)
  4. Figure 4: Genome Canada's Logic Model
  5. Figure 5: Genome Canada Evaluation Issues and Methods of Data Collection
  6. Figure 6: Trend in the Yearly Average Number of Papers per Researcher across Sub-Groups of the 500 Most Published Genomics Scientists in Canada, 1997–2018
  7. Figure 7: Percentage of Operation Costs out of Total Budgets for GC and the Genome Centres
  8. Figure 8: Operation Costs as a Share of Total Budget for the Genome Centres (2009-10 to 2018-19)

List of Tables

  1. Table 1: GC Funding Model in Comparison to Other National and International Research Funding Organizations with Similar Mandates

Executive summary

This report presents the results of an evaluation of Genome Canada (GC), a not-for-profit organization established in 2000 to develop and implement a national strategy in genomics research for the benefit of all Canadians.

Program overview

Since 2000, the Government of Canada, through the Department of Innovation Science and Economic Development Canada (ISED), has committed $1.5 billion in funding to GC, of which an average of $6.6 million per year was dedicated to GC's operational costs and $5 million per year to cover operational costs of the network of six independently incorporated regional Genome Centres.

GC's mandate involves:

  1. increasing breadth and depth of knowledge in genomics by supporting large-scale genomics research and knowledge related to ethical, environmental, economic, legal and social issues (GE3LS) of genomics;
  2. supporting the translation of research discoveries into application in the public and private sectors;
  3. providing leading-edge genomics technology platforms;
  4. enabling the creation of a solid base of trained genomics researchers; and
  5. enhancing Canada's international profile in genomics research.

GC's mandate covers seven strategic sectors (health, agriculture, environment, forestry, fisheries, energy, and mining) and is fulfilled in partnership with the Genome Centres.

Evaluation purpose and methodology

The GC evaluation meets the requirement of subsection 42.1(1) of the Financial Administration Act for reviewing every five years the relevance and performance of each ongoing program. The GC evaluation covered the period from April 1, 2014 to December 31, 2018 and was conducted by ISED's Audit and Evaluation Branch in accordance with the Treasury Board Policy on Results.

The evaluation assessed the relevance, performance and efficiency of GC, including the extent to which ISED's support to GC addresses a demonstrable need; examining the extent to which GC was able to meet its expected results as stated in the 2017-18 Contribution Agreement with ISED; and assessing the efficiency of the GC model of delivering federal support to genomics researchers.

Findings

Relevance

The field of genomics has advanced significantly over the past two decades. However, further research is still required to enhance the understanding of genomics and contribute to the emergence of an internationally competitive Canadian bio-economy.

The evaluation found that government support to GC was important to ensure the coverage of Canada's strategic sectors and alignment with the national science vision, which could not have been accomplished through dispersed funding support from other sources. However, the time-limited nature of the ISED-GC funding agreement can impact GC's ability to secure long-term co-funding commitments and planning.

Performance

GC contributes to the increased breadth of knowledge in the field of genomics by supporting research in a number of multidisciplinary topics and sectors that address the needs of different regions across Canada and foster international collaborations. GC-funded researchers achieved a higher publication rate compared to unfunded researchers (two papers more per researcher from 2014 to 2018).

GC has also contributed to the depth of genomics knowledge as research supported by GC enabled the development of new knowledge, which was subsequently used in practical applications in GC strategic sectors.

Further, GC's programs allowed the participation of highly qualified personnel in 84% of the 2014-18 GC-funded projects, where they benefited from the research experience – providing opportunities to move forward in their careers.

Moreover, GC-funded researchers benefited from the availability and quality of services provided to them by the GC-supported technology platforms. The platform experts provide advice and give access to researchers from various disciplines to leading-edge technologies needed for their projects at a competitive service cost. However, the continued technological competitiveness of the platforms may be further improved by focusing GC funding on technology development and better aligning operating funding for the five platforms, which also receive Canada Foundation for Innovation (CFI) funding.

GC-funded research results led to practical applications in GC strategic sectors. Although the number of Canadian genomics patents outpaces those of the top ten most productive countries in the field, the number of Canadian intellectual property rights (IP) retained is among the lowest, consistent with other research fields in Canada, but beyond GC's mandate.

Overall, GC has contributed to Canada being among the top ten performing countries in genomics research. However, Canada is outperformed by other nations in sequencing research that covers large populations, indicating that more support would be required to develop and reach international standards. GC's 2019 Strategic Vision report outlines strategies to help address this challenge.

Efficiency

The GC mandate focuses solely on genomics, unlike other key government organizations (e.g., the federal granting councils), which cover other fields as well. GC is also more capable of funding large-scale genomic projects. Further, it is the main national organization funding GE3LS research and extending its support to strategic sectors beyond health. GC also has a stronger regional presence through the Centres, with whom they work in collaboration. Following consultation with stakeholders, GC sets a national strategy and works collaboratively with the Centres and their regional strategies to set an overall agenda for genomics in Canada. While there is a great deal of collaboration among the Centres during the implementation of the funded projects, additional coordination at the application phase would help align their project submissions and minimize duplication of effort.

Funded researchers consider the GC funding application process to be clear and supported by an international peer review process that is transparent and equitable. However, the GC project reporting requirement was considered onerous for lower cost projects and could be better aligned with the amount of funding provided. Moreover, GC faces data consistency challenges while reporting on funded research results and socio-economic impacts.

GC and the Centres have established measures to help ensure the efficiency of their operations, but maintaining networking and outreach activities, necessary for establishing partnerships, can affect the operations of smaller Centres to a greater extent.

GC's average annual operating cost as a share of the total budget from 2009-10 to 2018-19 was comparable to larger national organizations with a similar mandate. Moreover, among the Centres, the average annual operating cost share was lower for larger Centres, which may be benefiting from economies of scale and their ability to attract more funding from sources other than GC.

Recommendations

Based on the evaluation findings, the following recommendations may help support GC's efforts toward continuous improvements of operations and the achievement of its strategic objectives.

Recommendation 1: Technology Platforms

ISED's Science and Research Sector should ensure Genome Canada coordinates with the Canada Foundation for Innovation to better align the operating funding they provide to the technology platforms, in order to minimize duplication of effort and focus Genome Canada's funding on the technology development and competitiveness of the platforms.

Recommendation 2: Efficiency of Service Delivery

To improve the efficiency of service delivery, ISED's Science and Research Sector will ensure Genome Canada:

  • Considers making project reporting requirements commensurate with the project funding level to limit the reporting burden on researchers; and
  • Develops systematic data gathering techniques and defines concepts related to knowledge translation to better measure the socioeconomic impacts of funded projects, in collaboration with the Centres.

1.0 Introduction

This report presents the results of the 2019 evaluation of Genome Canada (GC), a not-for-profit organization established in 2000 to develop and implement a national strategy in genomics research for the benefit of all Canadians.

The purpose of the evaluation is to assess the relevance, performance and efficiency of GC. The report is organized into four sections:

  • Section 1 provides the context, background, target population and stakeholders, and logic model of GC;
  • Section 2 presents the evaluation objectives, methods, and limitations;
  • Section 3 provides the evaluation findings; and
  • Section 4 summarizes the conclusions and provides recommendations.

1.1 Context

GC Mandate:

Genome Canada acts as a catalyst for developing and applying genomics and genomic-based technologies to create economic and social benefits for Canadians by:

  • Connecting ideas and people across public and private sectors to find new uses for genomics;
  • Investing in large-scale science and technology to fuel innovation; and
  • Translating discoveries into solutions across key sectors of national importance, including health, agriculture and agri-food, forestry, fisheries and aquaculture, the environment, energy and mining.

Genomics is the science that studies the genetic code and the function of genes within the DNA of all living things. Understanding the function of genes is central to the understanding of many biological processes. Given that many of the world's challenges a re controlled directly or indirectly by biological processes, genomics research can help find solutions to these challenges and spur innovation.

Since 2000, the Government of Canada, through the Department of Innovation Science and Economic Development Canada (ISED), has been directly funding GC through a number of Contribution Agreements in order to enhance the understanding of genomics and its uses. From 2000-01 to the latest Contribution Agreement in 2017-18, ISED committed $1.5 billion in funding to GCFootnote 1, of which $11.6 million per year, on average, was dedicated to operational costs for GC and the six CentresFootnote 2.

1.2 Background

GC's mandate, as per the Contribution Agreement with ISED, is to harness the transformative power of large-scale genomics research for the benefit of Canadians. This mandate is fulfilled in cooperation with a network of six independently incorporatedFootnote 3 regional Genome Centreslocated in:

  1. British Columbia;
  2. Alberta;
  3. Saskatchewan and Manitoba (represent the Prairie Provinces);
  4. Ontario;
  5. Quebec; and
  6. Nova Scotia (represents the Atlantic Provinces).

The rationale for decentralizing operations via the Centres relies on the notion that they are best positioned to identify local research needs and priorities, partnerships, and co-funding opportunities.

GC is responsible for developing strategies and partnerships at both the national and international levels. It launches national competitions and manages a peer review process for the selection of research projects. GC then transfers funds to the regional Genome Centres to be distributed to the funded projects. The Centres play a critical role in a number of areasFootnote 4:

  • Secure regionally-based partners to co-invest in genomics projects that cross provincial and national borders;
  • Advance regional priorities;
  • Ensure effective management and monitoring of GC-funded projects;
  • Facilitate access to leading-edge technology for researchers; and
  • Deliver public outreach at a regional level.

1.2.1 GC Governance

GC's Board of Directors is comprised of up to 16 individuals, drawn from the academic, private and public sector communities. The presidents of five major federal research funding agencies – the Canadian Institutes of Health Research (CIHR), the Canada Foundation for Innovation (CFI), the National Research Council Canada, the Natural Sciences and Engineering Research Council of Canada (NSERC), and the Social Sciences and Humanities Research Council of Canada (SSHRC) – are ex-officio advisors to the Board and may be invited to attend and participate in Board meetings whenever deemed advisable by the Board of Directors.

The GC Board is supported by the Science and Industry Advisory Committee (SIAC), which is a permanent committee of GC's Board of Directors that provides advice and recommendations to the Board on emerging areas of scientific and strategic importance to Canada.

1.2.2 GC Strategic Sectors

GC supports research in genomics and its application across seven strategic sectors including:

  • Health;
  • Agriculture;
  • Environment;
  • Forestry;
  • Fisheries;
  • Energy; and
  • Mining.

Since GC's inception, human health is the sector containing the majority of GC's funded projects (62%), primarily due to human genome research pioneering the study of genomics and the continued focus in this sector.

Figure 1: Percentage of Genome Canada Funding and Co-Funding Directed to Strategic Sectors from 2000 to 2019

Source: Genome Canada Financial Database

Note: The percentage directed to the strategic sectors is based on total Genome Canada funding and co-funding (i.e., including funding directed to the technology platforms).

Text version
Funded sectors % of funding directed to strategic sectors
Agriculture 12.66%
Energy 1.58%
Environment 6.14%
Fisheries 2.84%
Forestry 4.37%
Health 61.73%
Mining 0.13%

GC funds large-scale research projects, which include research into the ethical, environmental, economic, legal and social (GE3LS) implications of genomics. This can either be the major focus of the project or an integrated component that is shaped by, and helps shape, the overall project by investigating key factors that may facilitate or hinder the uptake of the genomic-based application(s) being developed by the project. GE3LS research may be conducted by researchers from universities, government, industry, not-for-profit or other organizations, who generally are trained in disciplines other than the life sciences, including social or behavioral scientists or humanities scholars.

1.2.3 GC Programs

GC supports a range of activities from fundamental research to applied research, proof-of-concept, validation, and product/tool development.

This support is delivered through the following programs:

  • Large-scale applied research projects (77%), which includes:
    • National competitions to support genomics research projects (including GE3LS) between $5 to $10 million over a term of up to four years, and with at least 50% of the project co-funded from other sources.
    • Strategic initiatives, which range from large international consortia, such as the Structural Genomics Consortium or International Barcode of Life, to emerging issues and opportunities that require immediate attention and timely resolution such as the Zika virus or Lyme Disease.

      Figure 2: Percentage of Genome Canada Funding and Co-Funding Directed to Programs from 2000 to 2019

      Source: Genome Canada Financial Database

      Text version
      Type of program Percentage of GC funding and Co-funding
      Large-scale Projects 77%
      Translation 13%
      Technology Platforms 10%
  • Technology platforms (10%) – supports the operations of ten technology platforms, which provide researchers across Canada and internationally with access to tools and expert advice needed to analyze genomes, proteomes and metabolomes in various ways, such as laboratory services for DNA mapping and sequencing and statistical analysis. Moreover, GC issued funding opportunities to support the development of new and improved technologies of the platforms (see Annex A for the evolution of GC investment in the technology platforms over time).
  • Translation (13%) – supports partnerships between academic researchers and knowledge users to help translate genomics-based discoveries into applications and/or marketable products through the Genomic Application Partnership Program (GAPP). GC funds one-third of the project costs, with another third required from the end-user.

1.2.4 GC Co-Funding Model

Securing co-funding through partnerships is central to the GC business model. Bringing together diverse partners to co-invest in Canadian genomics research aligns efforts and benefits society. From 2000-01 to 2018-19, GC, in collaboration with the Centres, leveraged a total of $2.1 billion from sources other than ISED, which had invested $1.5 billion in GC over the same period of time.1 Co-funders included other federal organizations, provincial governments, academic institutions, industries, non-profit organizations, and international organizations.

Figure 3: Total Annual Value of Genome Canada Funding (in millions of $)

Source: Genome Canada Annual Report 2017-18

Text version
- 2013-14 2014-15 2015-16 2016-17 2017-18
Co-funding received  87.2 95.1 103.6 124.6 153.6
GC project expenditures 68.1 59.9 45.8 55.4 63.2
GC operating expenses  4.7 5.0 5.6 5.0 3.2
Total  160 160 155 185 220

GC's goal to increase engagement of end-users of applied genomics has gone beyond its historical requirement to leverage a dollar for every dollar invested in some programs (1:1), leveraging over a dollar and-a-half or more for every dollar invested (1.58:1), depending on the program. This is now a requirement in the 2017-18 Contribution Agreement.

1.3 Program target population and stakeholders

GC's primary target population is the research community located in Canadian universities and their affiliated research hospitals and institutes. Further, the GC target population includes the end users of research results, which entails users from the public, private or non-profit sectors, who could also possibly be participating members of the funded research.

GC has a number of other stakeholders, such as the six Genome Centres, which participate in the delivery of the programs, and co-funding partners such as provincial governments and other federal funders, which includes the granting councils (i.e., SSHRC, NSERC and CIHR) and CFI, in addition to industry, international organizations, and not-for-profit organizations.

1.4 Logic model

The logic model in Figure 4 visually presents the activities undertaken and the expected outputs and outcomes resulting from the delivery of GC's mandate.

Figure 4: Genome Canada's Logic Model

Text version

Genome Canada's logic model is presented in Figure 4. The logic model visually presents the activities undertaken and the expected outputs and outcomes resulting from the delivery of Genome Canada's mandate.

The logic model has three distinct sections:

Section 1 comprises the left-third of the diagram. The mission at the bottom is Strategy and Partnerships. There is an upward arrow from the mission leading to three separate outputs/activities:

  • Research/investment strategies in various sectors of the Canadian bioeconomy.
  • Relationships and partnerships with the regional, national and international genomics community in areas of strategic importance to Canada.
  • Requirements for supported projects to leverage co-funding from various sources, including industry.

Each of the above outputs/activities has an upward arrow leading to three separate immediate outcomes:

  • Funded genomics research projects are relevant to sectors of the Canadian economy.
  • New opportunities arise for research collaboration in strategic areas of importance to Canada, within Canada and globally.
  • Increased level of engagement and investment by other partners, including industry.

Each of the above immediate outcomes has an upward arrow leading to the intermediate outcome, which is that Canada's international profile and visibility in genomics research is enhanced.

Section 2 comprises the middle-third of the diagram. The mission at the bottom is Large-Scale Genomics Research and Technology. There is an upward arrow from the mission leading to three separate outputs/activities:

  • Support for large-scale genomics research projects in Canadian research institutions, including GE3LS.
  • Support for operations, technology development and networking of the Genomics Innovation Network (GIN).
  • Support for the development of technologies that enable genomics research.

Each of the above outputs/activities has an upward arrow leading to two separate immediate outcomes:

  • Enhanced knowledge and HQP capacity in Canada in genomics research including GE3LS.
  • Canadian genomics research is enabled through the provision of leading-edge genomics, proteomics, metabolomics, and bioinformatics technologies.

Each of the above immediate outcomes has an upward arrow leading to two intermediate outcomes:

  • Increased breadth and depth of knowledge in genomics, including GE3LS.
  • Canada has a solid base of researchers trained to undertake future genomics research.

Section 3 comprises the right-third of the diagram. The mission at the bottom is Translation into Application. There is an upward arrow from the mission leading to four separate outputs/activities:

  • Support for partnerships between academia and users to advance a product, tool or process closer to market or address a significant unmet need.
  • Support for researchers to better understand how to create and capture entrepreneurial value from their research.
  • Support for research that investigates societal implications of genomics research and its application.
  • Support for mechanisms to strengthen communications and engagement with stakeholders.

Each of the above outputs/activities has an upward arrow leading to four separate immediate outcomes:

  • Increased partnerships between academia and users.
  • Increased translation of genomics technology, research prototypes, early stage products, tools and processes.
  • Increased policy, practice and regulatory engagement informed by GE3LS evidence.
  • Increased stakeholder awareness of genomics contribution to the social and economic health of Canada.

Each of the above immediate outcomes has an upward arrow leading to the intermediate outcome, which is translation of research discoveries into application in the public and private sectors leading to socio-economic benefits.

Each of the three sections of Figure 4 has an upward arrow on the right-hand side leading to the top, where it states "Canada has a world-leading research capacity".

2.0 Methodology

This section provides information on the evaluation scope and objective, issues and questions that were addressed, data collection methods, and limitations.

2.1 Evaluation context

GC was last evaluated in 2014 and assessed the organization's relevance, retrospective performance from 2009-10 to 2013-14, and the GC design and delivery model. The 2014 evaluation provided five main recommendations:

  1. GC, working with the Genome Centres, should seek out and/or create joint initiatives with a broader range of public and private organizations aiming to achieve similar objectives (e.g., R&D funding programs, partnership programs, business innovation, etc.).
  2. GC should address current information gaps on the effectiveness and weaknesses of integrated GE³LS to confirm its value in facilitating translation of genomics research and to develop criteria and guidelines to help adjust practices for the integration of GE³LS.
  3. GC should further improve working relationships with Genome Centres and collaboratively develop focused and customized funding programs that address the needs of specific sectors, including both large and small-scale projects, as appropriate.
  4. GC should encourage the five Science and Technology Innovation Centres (now ten technology platforms) to build on their unique strengths (e.g., providing analytical expertise, developing training programs and providing leading-edge technologies at an affordable cost) and to develop clearer policies and guidelines regarding data sharing and intellectual property, with a view to promote more open access to data.
  5. GC should continue to improve its performance measurement and reporting structures, as well as seek to better integrate its different databases.

2.2 Evaluation scope and objectives

The GC evaluation meets the requirement of subsection 42.1(1) of the Financial Administration Act that each department is required to conduct a review every five years of the relevance and performance of each ongoing program for which it is responsible. The evaluation covered the period from April 1, 2014 to December 31, 2018.

The current evaluation was conducted by ISED's Audit and Evaluation Branch. It examined the continued relevance, performance and efficiency of GC in accordance with the Treasury Board Secretariat Policy on Results. This entailed assessing the extent to which ISED's support to GC addresses a demonstrable need, examining the extent to which GC was able to meet its expected results as stated in the 2017-18 Contribution Agreement with ISED, and assessing the efficiency of the GC model of delivering federal support to genomics researchers.

2.3 Evaluation issues and questions

The evaluation considered the following questions to assess GC's relevance, performance and efficiency.

Relevance

  1. To what extent does ISED's contribution to GC address a demonstrable need?

Performance

  1. To what extent has GC contributed to increasing the breadth and depth of knowledge in genomics, including knowledge related to ethical, environmental, economic, legal and social issues (GE3LS) of genomics?
  2. To what extent has GC contributed to the development of a solid base of researchers that are trained to undertake future genomics research?
  3. To what extent has Canadian genomics research been enabled through the provision of leading-edge genomic technologies?
  4. To what extent has GC contributed to translating genomics research discoveries into applications in the public and private sectors leading to socio-economic benefits for Canadians?
  5. To what extent has GC contributed to enhancing Canada`s international profile and visibility in genomics research? 

Efficiency

  1. To what extent has GC been an efficient model of delivering federal support to genomics research?

2.4 Data collection methods

Multiple lines of evidence were used to address the evaluation questions. As Figure 5 below highlights, data sources included:

  • Document review;
  • Administrative and financial data analysis;
  • Interviews with GC stakeholders;
  • Case studies of funded research projects;
  • Survey of funded researchers and end users, as well as a survey of highly qualified personnel (HQP) working on funded projects;
  • Bibliometric analysis of the publications of GC-funded researchers and of Canadian genomics researchers;
  • Literature review comparing research funding models in Canada and internationally.

Figure 5: Genome Canada Evaluation Issues and Methods of Data Collection

Evaluation Issue Method
Document Review Administrative and Financial Data Analysis Interviews Case studies Survey Bibliometric Analysis Literature Review
Relevance

yes

yes

yes

no

no

no

yes

Performance

yes

no

yes

yes

yes

yes

no

Efficiency

yes

yes

yes

no

no

yes

Document Review

A review of GC's key documents was conducted in order to facilitate an understanding of GC's operations, assess its design and delivery model, and progress made toward achieving its objectives. Documents analyzed included:

  • Foundational documents (e.g., Treasury Board Submissions, Contribution Agreements);
  • Annual and impact reports; and
  • GC strategic and operational plans.

Administrative and Financial Data Analysis

An analysis of GC's administrative and financial data was undertaken to provide information regarding the extent to which the delivery model is effective and efficient and assess the extent to which GC is achieving its objectives. The data analyzed included the funding application database, as well as GC and the Centres' annual expenditures and co-funding database.

Interviews

The evaluation also included 29 interviews with key GC stakeholders. Findings from the interviews supported analysis on relevance, GC's design and delivery model, and performance. The interviews also helped identify areas for improvement. Stakeholders interviewed included:

  • ISED program management (n=1);
  • Genomics international experts (n=3);
  • Members of GC's Science and Industry Advisory Committee (SIAC) (n=4);
  • GC Board members (n=4);
  • GC and Centres management (n=9); and
  • GC partners and national experts (n=8).

Case Studies

Eight case studies were selected from funded projects: (four follow-ups from the previous evaluation and four new cases). The cases documented success stories and possible impacts of GC funding on different sectors, regions and research teams.

Surveys

Two surveys were conducted to document the opinions and perceptions related to the impact of GC funding on funded researchers, their projects and teams. The groups surveyed included:

  • Lead researchers, co-leads, co-applicants and end users of genomics results who were funded from 2014 to 2018 (n= 411, with a response rate of 31%); and
  • HQP, which includes technicians and Doctoral and Post-Doctoral and Masters students who were hired to work on research funded from 2014 to 2018 (n=78, with a response rate of 29%).

Bibliometric Analysis

The bibliometric analysis examined the scientific productivity and quality of publications of GC-funded researchers in comparison to those who were not funded by GC in order to assess the contribution of GC funding to the creation and translation of knowledge in the field of genomics. It also analyzed the national-level publications in the field and compared them to the publications of the top ten countries in genomics research with the goal of assessing the international status of Canada regarding genomics research.

Literature Review

The literature review documented GC's delivery model of funding genomics research in Canada and compared it to other funding models of genomics research in Canada and around the world in order to assess the efficiency of the funding model.

2.5 Limitations and mitigation strategies

The challenges encountered during the GC evaluation and the mitigation techniques applied to address them are outlined below.

The Assessment of Impact

Research projects often require a longer timeframe to show a tangible impact on the sectors and/or communities they study, thereby presenting a limitation to fully assess the possible socio-economic impacts of GC-funded research. As a mitigation technique, whenever possible, anecdotal evidence of impacts was collected through the interviews, surveys, some case studies, and available GC impact reports.

Attribution of Outcomes

With the presence of several players and donors of funds in the field of genomics, it can be challenging to attribute the outputs and outcomes achieved to the work of GC. To mitigate this issue, the lines of evidence were designed and articulated in a way that respondents can answer, to the extent possible, questions about the role played by GC in the outcomes achieved in the field. Furthermore, the bibliometric analysis contributed to demonstrating impacts directly attributable to GC, by comparing the productivity and impact of the publications of GC-funded and non-funded researchers.

Data Issues

There was some misalignment between the definitions and concepts describing the operational data collected by the Centres and those collected at the GC level. This made it challenging to conduct further analysis to describe and compare the operations of the Centres and GC over time, as well making it difficult to conduct socio-economic impact analysis. As a result, findings from the other lines of evidence were used to compensate for the missing information from the data.

3.0 Findings

3.1 Relevance

This section analyzes findings from the interviews, surveys, document review and literature review to assess the following:

3.1.1 The extent to which ISED's contribution to GC addresses a demonstrable need. This includes assessing:

  • 3.1.1.1 The need for genomics research.
  • 3.1.1.2 ISED support to GC.
3.1.1.1 The need for genomics research

Key Finding: The scientific and economic benefits of the field of genomics are acknowledged nationally and internationally. Although the field has advanced significantly over the past two decades, further research is required to enhance the understanding of genomics and contribute to the emergence of an internationally competitive Canadian bio-economy.

Significance of genomics

Genomics is a transformative technology that plays a key role in addressing the most pressing challenges facing society in the 21st century. The Organization for Economic Cooperation and Development (OECD) recognizes genomics as one of the most important technologies that will fuel the development of a global bio-economy, placing genomics at the heart of the world's economic development for years to come. In its landmark report "The Bio-economy to 2030", the OECD projects that "biotechnology could contribute to 2.7% (or about US$1.1 trillion) of the GDP of OECD countries in 2030."

Reinforcing the OECD projection, the Centre for the Study of Living Standards (CSLS) has estimated that biotechnology could represent up to 4.0% (or about $144 billion) of Canadian GDP in 2030, driven by factors such as increased demand for food, energy, and healthcareFootnote 5.

Need to support genomic research

Genomics is becoming widely recognized as a critical foundation for numerous applications that will contribute to the emergence of an internationally competitive Canadian bio-economy. The document review and opinions from the interviews showed that although genomics has produced dramatic advances in the understanding of living organisms and the biotechnological capabilities in the past two decades, the field is still young and further research needs to be supported in order to:

  • Enhance the understanding of the biological systems beyond the health sector;
  • Address the management and analysis of big data collected via research; and
  • Provide solutions to the many serious challenges facing Canada and the world today, such as climate change, global population growth, increasing food and energy demand, and chronic and acute health issues.
3.1.1.2 ISED support to GC

Key Finding: Government support to GC was important to ensure the coverage of Canada's strategic sectors and alignment with the national science vision, which could not have been accomplished through dispersed funding support from other sources. However, the time-limited nature of the ISED-GC funding agreement can impact GC's ability to secure long-term co-funding commitments and planning.

There was consensus among interviewees that ISED's support to GC is important. GC's SIAC members and the international experts interviewed noted that many genomics discoveries require significant funding that often only governments are able to provide.

Although separate national agencies and research initiatives could still provide valuable support to genomics research if ISED's support to GC is withdrawn, these fragmented efforts do not guarantee the coverage of the seven strategic sectors. Further, other organizations do not ensure alignment with the national vision and directionFootnote 6 of supporting science and innovation that foster evidence-based decision making, encouraging the next generation of scientists, and making Canadian science more collaborative, which are enabled by ISED's support to GC.

This opinion was further supported by the survey findings, as 71% of the GC-funded researchers and end users surveyed reported that, in the absence of GC, it is unlikely that the GC-funded research projects would have been supported at the same level and scope via other funding sources.

Findings from the case studies also showed that GC funding allowed the projects studied to proceed on a much larger scale and move faster than they would have in the absence of that funding. According to several case study interviews, grants from the Granting Councils (i.e., CIHR, SSHRC and NSERC) would have been an alternative principal funding source for their projects, but they would provide much lower annual fundingFootnote 7 compared to that provided by GC, which is deemed insufficient for running a large-scale genomics research project.

However, GC Board members indicated that there is uncertainty inherent in the time-limited federal funding agreement with GC. As an independent, not for-profit organization, GC does not receive A-Base funding (i.e., a source of funding accessed through parliamentary votes), but rather depends on federal funding in the form of a time-limited grant or contribution.

The 2-3 year funding agreements are believed to have negatively impacted the ability of GC and the Centres to maintain medium-to long-term co-funding partnerships, as co-funding partners usually require a multi-year planning horizon for the kind of large-scale and long-term investments that genomics research and innovation entailsFootnote 8.

It is worth noting that the 2017-18 Contribution Agreement offered GC funding that is allocated over three years and disbursed over a period of seven years. Beyond the 2017-18 Contribution Agreement, Budget 2019 proposed to establish the Strategic Science Fund, which will be the new mechanism for funding third-party organizations (like GC) starting in 2022-23. Going forward, the selection of recipient organizations and corresponding level of support will be determined through the Fund's competitive allocation process.

3.2 Performance

This section analyzes findings from the interviews, surveys, case studies, document review, and bibliometric analysis to assess the following:

  • 3.2.1 The extent to which GC contributed to increasing the breadth and depth of knowledge in genomics, including knowledge related to the GE3LS of genomics;
  • 3.2.2 The extent to which GC contributed to the development of a solid base of researchers that are trained to undertake future genomics research;
  • 3.2.3 The extent to which Canadian genomics research has been enabled through the provision of leading-edge genomic technologies;
  • 3.2.4 The extent to which GC has contributed to translating genomics research discoveries into applications in the public and private sectors leading to socio-economic benefits for Canada; and
  • 3.2.5 The extent to which GC has contributed to enhancing Canada's international profile and visibility of genomics research.

3.2.1 The extent to which GC contributed to increasing the breadth and depth of knowledge in genomics, including knowledge related to the GE3LS of genomics

3.2.1.1 GC's Contribution to the Increased Breadth of Knowledge in Genomics

Key Finding: By supporting research in a number of multidisciplinary topics and sectors that address the needs of different regions across Canada and foster international collaborations, GC contributes to the increased breadth of knowledge in the field of genomics. Further, GC-funded researchers achieved a higher publication productivity compared to unfunded researchers (two papers more per researcher from 2014 to 2018).

From 2014 to 2019, GC has funded about 201 projects. According to most of the funded researchers and end users surveyed (80%), GC contributed to increasing the quantity of produced genomics research in Canada. The other lines of evidence also showed how GC managed to increase the breadth of knowledge in genomics, as noted below.

Supporting various research topics and sectors

Results from the interviews showed that GC's work enabled a number of research studies in various research areas, such as wheat genomics, microbiome, rare diseases, infectious diseases, agri-food and fisheries. However, some interviewees highlighted that further research is still needed in emerging genomics sectors such as environment, mining, and energy.

Supporting multidisciplinary research

Through its support to GE3LS research, GC has successfully brought researchers from the natural sciences and social sciences and humanities to work together on multidisciplinary research. Moreover, 81% of the researchers surveyed reported that GC had a significant impact on enabling research production in GE3LS.

Supporting wider research outreach and international collaborations

The majority (62%) of survey respondents indicated that GC contributed to increasing coordination among Canadian researchers across regions via its national-provincial model of delivery. As GC and Genome Centre management highlighted, funded research is supported via on-going interaction with scientists to help move projects forward, oversight committees that help direct and advise research, GE3LS review of research work, and via fostering participation in public events and conferences.

Further, results from the interviews showed that GC funding has transferred certain centres in Ontario and BC into significant hubs for supporting research in human health. In Alberta and Saskatchewan, the contribution to animal health research was evident, and the East Coast supported more research in aquaculture and fisheries.

GC's partnerships with other countries, such as the UK, enabled the significant expansion of the database on infectious disease, microbiome, and human data through allowing data sharing and standardization.

Higher productivity of GC-funded researchers

Bibliometric analysis comparing the volume of publications between GC-funded (lead researchers, co-leads and co-applicants) and non-GC-funded genomics researchers showed a positive impact that is attributable to GC funding (an average of two more publications per researcher annually from 2014 to 2018). Figure 6 shows that the growth in publication productivity of funded researchers has seen sharp acceleration following receipt of GC funding, at a pace greater than the trend observed for researchers never funded by GC. The volume generally plateaued or decreased after some time, however, as a lag in publication is expected during the research conduct phase. Nonetheless, it remained higher than the volume of publications in the initial years.

Figure 6: Trend in the Yearly Average Number of Papers per Researcher across sub-Groups of the 500 Most Published Genomics Scientists in Canada, 1997–2018
Text version
Group 1997-2001 1998-2002 1999-2003 2000-2004 2001-2005 2002-2006 2003-2007 2004-2008 2005-2009 2006-2010 2007-2011 2008-2012 2009-2013 2010-2014 2011-2015 2012-2016 2013-2017 2014-2018
Never funded by GC 2.6 2.6 2.8 3.0 3.2 3.4 3.7 4.0 4.2 4.4 4.7 4.9 5.2 5.4 5.6 5.6 5.5 5.3
GC-funded 2.9 3.1 3.3 3.5 3.9 4.2 4.7 5.0 5.3 5.6 6.0 6.3 6.7 7.2 7.6 7.6 7.6 7.3

Results from an econometric analysisFootnote 9 examining the relationship between GC funding and the publication volume of 619 GC-funded researchers from 2009 to 2013Footnote 10 showed that the effect of funding on the number of papers produced is positive and statistically significant. The analysis also showed that the studied GC-funded researchers published two to six papers more per $1 million received from GC, compared to their productivity during periods when they were not being funded by GC.

3.2.1.2 GC's Contribution to the Increased Depth of Knowledge in Genomics

Key Finding: Research supported by GC enabled the development of new genomics knowledge in the funded strategic sectors.

Most of the survey respondents (79%) reported that GC contributed to increasing the quality of genomics research in Canada. Moreover, 69% of the researchers reported that the improvement in GE3LS research could also be attributed to the GC contribution to the field.

Some of the GC-funded projects in the various sectors provide illustrative examples that GC funding has contributed to the creation of new knowledge, as noted belowFootnote 11.

Health

Through GC funding for Finding of Rare Disease Genes and CARE4RARE, researchers were able to determine that whole-exome sequencing is a highly successful strategy to diagnose patients with rare pediatric-onset epilepsies when no clear diagnosis was possible based on regular standards of care.

Agriculture

Though the GC-funded project "Genomics for a Competitive Greenhouse Vegetable Industry" (Completed June 2017), the TTM2 gene variants have increased pathogen resistance levels by 25-30%, and tomato and pepper hybrids with TTM2 variants will be released commercially in 2022.

Mining

The "Mine Wastewater Solutions: Next Generation Biological Treatment through Functional Genomics" project has led to the important discovery of how best to measure sulphur compounds in wastewaters. The team is currently working on a new tool, which would enable mining companies to effectively account for all of the sulphur in wastewater and track the amount of sulphur balance that has the risk of being released into the environment.

Forestry

SMarTForest, a large-scale applied research project from 2011-15, developed the first genome sequence of white spruce using novel bioinformatics tools. It was one of the first published conifer genome assemblies. The project also developed genomic selection methods for use as tree breeding tools that provide an accurate prediction of breeding values for tree breeding programs. These tools have the potential to cut the breeding cycle for mature traits by a third of the time required (28 years to 9 years) using traditional methods.

Fisheries and Aquaculture

The GC-funded project "Sustaining freshwater recreational fisheries in a changing environment" sequenced the genome of rainbow trout populations in BC and constructed a tool (WildTroutChip) that can identify genetic variation across the species range. The GE³LS component of the project is contributing to a better understanding of the highly complex regulatory environment for rainbow trout in BC. It successfully gathered the oral histories from 50 Indigenous elders on the traditional relationship of First Nations with salmon and rainbow trout.

3.2.2 GC's Contribution to the Development of a Solid Base of Trained Researchers

Key Finding: GC's programs allowed the participation of highly qualified personnel in 84% of the 2014-18 GC-funded projects, where they benefited from the research experience, which gave them opportunities to advance in their careers.

GC's suite of programs do not include direct funding to train emerging researchers in the field of genomics. However, these programs have enabled lead researchers to hire a number of highly qualified personnel (HQP) to work in their research teams and gain experience. Survey results showed that 42% of the projects had fewer than five HQP, 23% hired five to nine HQP, and 19% hired 10 to 19 HQP per project. Fewer international HQP, however, were reported by lead researchers to have worked on GC-funded projects during the same period, with 78% of the projects involving less than five international HQP.

Half of the HQP surveyed reported that they would not have had the opportunity to be involved in genomics research without the support provided by GC to their research teams. According to the case studies and interviews (with SIAC members and national partners), the opportunities offered to HQP via GC-funded research include: working on large-scale and multidisciplinary research projects, having access to leading-edge research infrastructure, attending conferences and workshops in their fields, and participating in research networks that involve academic partners, government and international stakeholders. Almost all (91%) surveyed researchers noted that the HQP gained research experience such as new knowledge and skills and 76% saw that the HQP benefited from increased networking opportunities. Further, 78% of the researchers and 92% of the HQP surveyed indicated the HQP's experience contributed directly to their degree, thesis or publication.

Findings from the case studies and the surveys also showed that the training received while working on GC-funded projects has enabled the career development of the HQPinvolved. For example, about 60% of the surveyed researchers reported that former HQP got research positions within their research teams, while 56% said HQP were offered a subsequent academic opportunity. Further, the interviews cited two former post-doctoral fellows on GC-funded projects that have gone on to work as project leads on other GC projects.

Moreover, between 2014 and 2018, 66% of those who were involved in GC-funded projects as end users hired HQP who have participated in GC-funded projects, mainly because those HQP had the experience relevant to the organization's needs (95%). Other influences for hiring HQP include that the HQP facilitated the transfer of research results from the project to their organizations (57%). Project leads of the various case studies also reported they had trouble retaining HQP, as their skills are in high demand and they are actively recruited by other employers.

3.2.3 GC's Contribution to Genomic Research Through the Technology Platforms

Key Finding: GC-funded researchers benefit from the availability and quality of services provided by the GC-supported technology platforms. The platform experts provide advice and give access to researchers from various disciplines to leading-edge technologies needed for their projects at a competitive service cost. However, the continued technological competitiveness of the platforms may be improved by focusing GC funding on technology development and better aligning operating funding for the platforms, which also receive CFI funding.

Among the researchers using a technology platform, 76% said that the platform enabled the conduct of their research to a high extent. A further 67% of the lead researchers and co-applications/collaborators surveyed said that GC and/or the Genome Centres made a significant contribution to the improvement of the quality of genomics infrastructure, while 63% believe they contributed significantly to the quantity and accessibility of the infrastructure available.

Findings from the case studies also showed that the McGill University and Genome Quebec Innovation Centre platform (used most frequently by researchers, per the survey results) provides work that is nationally and internationally competitive. Moreover, the GC Board members interviewed highlighted that the sequencing centres in Montreal, Toronto and Vancouver are all among the top 10 largest academic genomics centres in the world.

The interviews and case study findings showed that the platforms contribute to genomics research in the following ways:

  • Provide access to cutting-edge technologies needed for large-scale research that would not have otherwise been available in many of the researchers' institutions;
  • Offer researchers services at a cost that is competitive to other labs;
  • Provide services to researchers from disciplines other than health;
  • Allow the storage of generated research data in Canada, which consequently enables more control over data security, access and accuracy; and
  • Assist researchers in the development of research proposals through their innovation centres, by providing advice on appropriate technologies and study design that improve the quality of the research.

Despite the success of the technology platforms in supporting research, the results from the interviews and document review identified the following areas for improvement:

  • Holding separate technology development competitions regularly every 2-3 years to help ensure the platform technologies are up-to-date and supportive to the funded research.
  • Continuing to invest in the platforms, as noted by SIAC members, GC Board Members and GC management. However, the role GC plays should be more focused on technology development rather than funding operations as is currently the case, considering that half the platforms also receive CFI operating fundingFootnote 12. In that respect, better coordination and collaboration between GC and CFI would help avoid duplication of effort and better focus the resources of both organizations.

3.2.4 GC's Contribution to Translating Genomics Research Discoveries into Applications

Key Finding: GC-funded research results led to practical applications in GC strategic sectors. Although the number of Canadian genomics patents outpaces those of the top ten most productive countries in the field, the number of Canadian intellectual property rights (IP) retained is among the lowest, consistent with other research fields in Canada, and beyond GC's mandate.

Survey respondents reported that the GC-funded research they participated in led to practical applications, including the exploration, development or application of new or improved public policy programs (61%); new or improved health care protocols, diagnostics or therapeutics (57%); or direct (56%) or indirect (58%) technology transfer for new or improved products, processes or services. A further 52% pointed out practical applications in environmental benefits. Moreover, the researchers rated the contribution made by GC to the practical applications of funded research results as high.

Bibliometric analysis also showed that Canadian genomics researchers are particularly active in patenting their research outcomes. Over time, the number of Canadian genomics patents came second to only the United States.

Further, bibliometric analysis showed that GC funding has contributed to the writing of influential genomics publications that impacted other research, as measured by the Average of Relative Citation (ARC) of publicationFootnote 13. Results showed that $1 million of GC funding increased the ARC of researchers by slightly more than 20% compared to the periods where they did not receive GC funding.

Examples of how some GC-funded projects were able to translate their results into practice include the following:

  • Researchers of the "Development and commercialization of next generation enzyme supplements for swine and poultry" and "Lysozyme feed additives to improve gut health and productivity of food animals" projects are currently in commercialization talks with a U.S. feed company to license, manufacture and market two enzymes for use in animal feed. Those enzymes demonstrated a decrease in the amount of food compared to the amount of weight gain in animals. This could translate into substantial cost savings for farmers, as adding enzymes to improve digestibility would allow them to use more locally available feed.
  • Due to the "Metagenomic Approach to Evaluate the Impact of Cheese-making Technologies and Ripening Conditions on the Microbial Ecosystem of Premium Washed Rind Cheeses" project, the production of Oka cheese plant has tripled and sales have increased by 83%.
  • As a result of the "SALMON and CHIPS: Commercial Application of Genomics to Maximize Genetic Improvement of Farmed Atlantic Salmon on the East Coast of Canada" project, The Kelly Cove Salmon company can now breed selectively for improved saltwater growth and resistance to diseases and parasites. This is expected to improve the quality of Atlantic salmon production, increase sales, and create a sustainable aquaculture strain with minimal impact on wild salmon.
  • The instrument-free diagnostic device developed due to the "Development of Low-Cost Testing Chip and Device for Hepatitis C Testing" project provided genotype-specific ID of Hep C in minutes using a device attached to a smartphone. The intellectual property developed, as a result, was acquired by General Atomics to support initiatives related to microelectronics.

Despite the achievements accomplished so far in knowledge translation, the bibliometric analysis showed that Canada placed last on the IP indicator compared to the top twenty nations with the most applications in the sector of genomics. This means Canadian inventors did not retain property of (or transfer to Canadian partners) their IP rights. Additional evidence indicates that Canada's record on the development of IP is modest, with Canadians increasingly likely to transfer or sell their IP to foreign entities rather than continuing to develop it in CanadaFootnote 14. This was noted as being due to the rise of dominant U.S. technology firms that are both fierce competitors and keen buyers of Canadian IP – increasing the incentive for Canadian firms to sell and not scale up. This suggests that IP retention in the field of genomics is beyond GC's mandate. It is worth noting that, according to interviewees and the bibliometric analysis, the IP retention issue is not unique to the knowledge translation of genomics research, as Canadian researchers in other fields have identified similar challenges.

3.2.5 GC's Contribution to Enhancing Canada's International Profile and Visibility of Genomics Research

Key Finding: GC has contributed to Canada being among the top ten performing countries in genomics research. However, Canada is outperformed by other nations in sequencing research that covers large populations, indicating that more support would be required to develop and reach international standards. GC's 2019 Strategic Vision report outlines strategies to help address this challenge.

Bibliometric analysis showed that when it comes to the number and influence of genomics publications, Canada was well positioned internationally. Of the years studied, between 2012 and 2016, Canada consistently appears to be among the top 10 national performers and has maintained performances ranking around seventh or eighth across the indicators of quantity and impact of genomics publications.

From 1997 to 2016, Canada was among the top countries that combine both very specialized and impactful research publications in GE³LS research, outperforming the United Kingdom and the United States.

Even though the ranking of Canada's international profile in genomics cannot be fully attributed to the work of GC, there is some evidence to show that GC's work still contributed to this ranking:

  • The researchers and end-users surveyed reported that GC contributed to increasing coordination among Canadian researchers (62%) and Canadian and international researchers (52%). Further, 61% reported that GC enabled Canada to become a world leader in genomics research, a finding that was also supported in the interviews.
  • The case studies showed that the GC-funded projects contributed to international research partnerships and/or world-class scientific discoveries. They have also contributed to the appointment of Canadian researchers to key positions on international bodies. For example, the Care4Rare project extends Canada's profile as a world leader in the discovery and diagnosis of rare diseases. The project lead serves as the chair of the Diagnostics Committee of the International Rare Diseases Research Consortium and recently presented her work at the United Nations Rare Disease Day Policy event.

Moreover, there was a consensus among the interviewees that GC work has contributed toward Canada's current position in genomics research. Some further noted that GC supported large-scale initiatives such as the Structural Genomics Consortium and the International Barcode of Life have put Canada 'on the international genomics map'.

While Canada is well positioned internationally in the field of genomics in general, the interviews showed that there are a number of fast moving fields within genomics where Canada is outperformed by other countries. These include sequencing projects that cover large a population where large dataset storage and analysis are necessary, indicating that more support would be required to develop and reach international standards.

Findings from the interviews also showed that addressing regional research needs have been one of the GC enterprise successes. However, for Canada to compete at an international level in the field of genomics, it was noted by interview respondents that GC, in collaboration with the Centres and other stakeholders, need also to focus on a national strategic direction, which will consolidate the regional efforts and minimize duplication. A 2019 publication of a strategic vision for GC, emanating from stakeholder consultations including each of the six Centres, could help address this challengeFootnote 15. The vision includes Canada becoming a world leader in the application of genomics-based biosciences for human health, the environment and across the bio-economy. As well, the vision includes new strategies including big data initiatives to handle large and complex data sets; a commitment to promoting equity, diversity and inclusion; and an enhanced commitment to genomics in society.

3.3 Efficiency

This section analyzes findings from the interviews, data review, surveys, case studies, and literature review to assess the following:

3.3.1 The extent to which GC has adopted an efficient model of delivering federal support for genomics research that includes examining:

  • 3.3.1.1 The GC funding model in comparison to other national and international models.
  • 3.3.1.2 Aspects of the GC design and delivery model such as the application submission, review process, grant reporting requirement, and measuring funded research impact.
  • 3.3.1.3 The coordination between GC and the Genome Centres and among the Centres.
  • 3.3.1.4 How GC and the Centres manage operational efficiency.
  • 3.3.1.5 GC and Centres' share of operation costs out of total budget.

Overall, GC was noted as having an efficient model of delivering federal support for genomics research. However, a number of opportunities to improve efficiency were identified, including areas related to project reporting requirements and data consistency.

3.3.1.1 GC Funding Model Compared to Other National and International Models

Key Finding: National and international genomics research is primarily funded by government organizations. Unlike other key Canadian players, the GC mandate focuses solely on genomics and is more capable of funding large-scale projects. Further, it is the main national organization funding GE3LS research and extending its support to strategic sectors beyond health. GC also has a stronger regional presence through the Centres, which are coordinated via GC's central office.

The literature review compared the GC funding model to that of other national and international research funding organizations with a similar mandate. Most of the models examined were for government organizations that directly deliver research funding to the researchers. The majority fund external researchers and a few fund and conduct research, such as Centre national de la recherche scientifique (CNRS), the National Human Genome Research Institute (NHGRI) and the Wellcome Sanger Institute (WSI) – see Table 1.

Unlike GC, most of the national models examined, such as that of the Granting Councils, fund genomics and other fields of research.

Table 1: GC Funding Model in Comparison to Other National and International Research Funding Organizations with Similar Mandates
- Country Organization Name Type of Organization Funds Extramural Research Organization Conducts Research The Funded/Conducted Research is dedicated to Genomics
International France

Centre national de la recherche scientifique (CNRS)

Government organization

Yes

Yes

No

USA

National Human Genome Research Institute (NHGRI)

Government organization

Yes

Yes

Yes, specifically health related research

Germany

National Genome Research Network (NGFN)

Government organization

Yes

No

Yes, specifically health related research

UK

Wellcome Sanger Institute (WSI)

Not-for-profit organization funded by Wellcome Trust.

Funds only few

Yes

Yes

National Canada

Genome CanadaFootnote 1

Not-for-profit organization funded by the Government of Canada.

Yes

No

Yes

Canada

CIHR

Government organization

Yes

No

No

Canada

SSHRC and NSERC

Government organizations

Yes

No

No

Canada

CFI

Not-for-profit organization funded by the Government of Canada

Funds research-related infrastructure

No

No

Canada

Genomics Research and Development Initiative (GRDI)

Government organizations

No

Funds genomics research conducted in eight federal science departmentsFootnote 2 and agencies

Yes

According to the interviews, the GC research-funding model, which includes GC and the regional Centres, is important for the reasons outlined below.

Uniqueness of the Institution's Mandate

Genomics focus: None of the other national organizations' mandates and programs fully coincide with that of GC and that, according to the interviewees, makes GC's mandate unique. Genomics is the main focus of GC's mandate whereas the other Canadian institutions' mandates cover other fields as well. This helps channel GC's efforts and resources toward the field and enables the development of more focused studies of genomics.

Funding multidisciplinary research: GC is the main national institution that currently funds research in GE3LS. This is besides funding research in a number of strategic sectors that go beyond the health sector. It is worth noting that SSHRC, NSERC and GRDI fund genomics research outside the health sciences, but in much smaller proportions than GC.

Large-scale research: GC is fully equipped to fund large-scale projects that often require access to complex infrastructure and involve a large number of collaborators and personnel, whereas other organizations fund mainly small-scale projects.

National and Regional Platform for Genomics Research

Importance of the Regional Centres: GC has a comparative advantage over the other national organizations due to its strong regional presence via the Genome Centres, which enables proximity and wider outreach to researchers across Canada that helps identify regional research and practical application needs. It also fosters collaboration with provincial governments, which enables leveraging of additional funds.

Importance of GC: GC ensures the work of the regional offices aligns with the GC objectives, provides a national platform for international collaboration, and a point of contact with federal government organizations.

3.3.1.2 GC Funding Application, Review Process and Reporting Requirement

Key Finding: The GC funding application process is clear and application information is accessible to researchers, supported by an international peer review process that is transparent and equitable. The GC project reporting requirement was, however, considered onerous for lower cost projects and could be better aligned with the amount of funding provided. Moreover, GC faces data consistency challenges for reporting on the impact of funded research.

GC Application Submission Process

Most of the lead researchers (73%) surveyed agreed that the information needed for GC's funding application was easy to find and 66% indicated that the instructions needed to fill in the application were clear and easy to understand.

GC Application Review Process

Of the lead researchers surveyed, 67% said the GC application review and approval process was transparent and 62% noted the process as equitable, unbiased and inclusive.

The document review also showed that, because GC funds research projects of considerable monetary value, its evaluation process of applications to its programs is extensive, rigorous and competitive in order to ensure that only the highest-calibre projects with the greatest likelihood of success are funded. To avoid a conflict of interest, an international panel of experts are invited to Canada to conduct a peer review of the proposals by interviewing the applicants face-to-face, which enables reviewers to interact directly with applicants and clarify some aspects of their proposals that lead to a better-informed decisionFootnote 16.

GC Project Reporting Requirement

The case studies and interviews flagged that the project reporting requirementFootnote 17 is burdensome despite GC's efforts to reduce the frequency of progress reports that funded researchers have to submit. Researchers who were involved in larger projects and had considerable experience with GC-funded projects tended to view these processes as appropriate given the large amounts of funding involved. However, researchers working on projects with smaller amounts of GC funding (typically less than $250,000), or who were involved in their first GC project, felt that the reporting burden was excessive.

Reporting on Funded Research Results and Socio-Economic Impact

In response to the 2014-15 evaluation, GC developed an annual impact report, which provides information about the achievements of some of the funded research as they become available. As the interviews flagged, despite GC's efforts to report on funded research impact, there are still challenges in monitoring performance in a systematic way because of the variation of the data collected across the Centres.

3.3.1.3 Coordination between GC and Genome Centres and among the Centres

Key Finding: GC and the Centres work collaboratively to operationalize national and regional strategies for genomics in Canada. While there is a great deal of collaboration among the Centres during the implementation of the funded projects, additional coordination at the application phase would help align their project submissions and minimize duplication of effort.

Coordination between GC and the Centres

SIAC members, GC Board members, and GC and Centre management highlighted that the relationship between GC and the Centres is characterized more by complementarity rather than duplication. In collaboration with their stakeholders, GC and the Centres set national and regional strategies and work collaboratively to operationalize this plan. Moreover, GC works more at the federal level while the Centres work closely with their respective provinces. There is also collaboration at both levels, especially as the Centres work toward aligning their regional strategies with the strategic direction of GC.

Survey results showed that 47% of the lead researchers indicated that the roles and responsibilities of GC and the Centres are clear and well defined. A further 55% of the lead researchers surveyed valued the role played by the Centres, more specifically in helping them secure co-funding. Further, several case study interviewees involved with Genome BC projects said they were impressed with the Centre's support for communications, marketing and outreach to end-users. According to one researcher, "we would not have had as far reaching impact without Genome BC." It is worth noting that the researchers deal mainly with the Centres throughout the duration of their funding.

Coordination among the Genome Centres

As indicated by GC Board members, SIAC members, and GC and Centre management, the Centres collaborate well during the implementation of the funded projects (e.g., 70% of Genome Prairie's projects involve another Centre). Moreover, regular discussions among the CEOs of the Centres allow the sharing of best practices, how they can better work together, and how to build capacity. As well, the Centres work together to help implement GC's overall strategy. However, because the Centres are regionally focused, rather than aligning their submission efforts, sometimes they submit similar projects, which compete over the limited funding opportunities. This suggests that the national office could play a greater role in the coordination of project submissions.

3.3.1.4 Managing Operational Efficiency

Key Finding: GC and the Centres have established measures to help ensure the efficiency of their operations. However, maintaining networking and outreach activities, necessary for establishing partnerships, can impact the operations of smaller Centres.

How GC manages Operational Efficiency

Findings from the interviews showed that GC uses a number of measures to manage the efficiency of its operations and those of the Centres:

  • GC's Board regularly evaluates its effectiveness in fulfilling its roles and responsibilities using assessment questionnaires and comprehensive governance reviews.
  • The Centres raise funds from the provincial governments and other sources to complement the funding received from GC in support of their operations (an average ratio of 1:2 from 2009-10 to 2018-19).
  • GC and the Centres coordinate their planning and budgets. They periodically review their processes to ensure they are efficient and cost-effective.
  • The peer review process for the competitions is centralized at GC to minimize costs and avoid duplication with the regions.
Challenges to GC and Centre Operations

Creating partnerships, economic opportunities and strategic consultations requires conducting several communication, outreach and networking activities, which sometimes calls for dedicating more GC and Centres' funding to the communication budget than what is currently allocated, as reported in the interviews.

Some Centres receive provincial support for operations while others do not, putting pressure on the operational budget of these Centres. Further, provincial governments generally show limited interest in funding Centres that allocate most of their resources outside the provincesFootnote 18 own borders, and those that do fund these Centres will put conditions on how funding can be spent. This is mainly the case for the Prairie and Atlantic Centres, which work with multiple provincial governments and require frequent travel that significantly increases travel and meeting costs.

3.3.1.5 Share of Operation Cost out of Total Budget

Key Finding: GC's average annual operating cost as a share of the total budget from 2009-10 to 2018-19 was comparable to larger national organizations with a similar mandate. Moreover, among the Centres, the average annual operating cost share was found to be lower for larger Centres, which may be benefiting from economies of scale and their ability to attract more funding from sources other than GC.

GC's 2017 Performance Audit report assessed the operational efficiency of GC by comparing its performance to that of similar national organizationsFootnote 19 and showed that GC and the Centres' operating cost share of total cost was 13.2% in 2015-16, which they concluded was comparable to national organizations of the same size (total annual funding between $40 million and $80 million) and higher than larger organizations (9.8%, total annual funding of $80 million or greater). Using a similar methodology, GC's operating costs as a share of the total budget for the 2009-10 to 2018-19 period was calculated to be about 10% (see Figure 7), making GC more comparable to the efficiency level of larger organizations.

For the Centres, even though the collective operation costs share of the total budget between 2009-10 and 2018-19 ranged from 13.8% in 2009-10 to 20.7% in 2015-16 (as shown in Figure 7), averaging about 17%Footnote 20, some of the Centres had higher average annual shares (e.g., Genome Atlantic at 43.1% - see Figure 8) compared to others (e.g., Genome Quebec at 11.5%). This shows that larger Centres may be benefiting more from the economies of scale regarding operating costs compared to smaller size Centres, as larger Centres were more able to attract funding from sources other than GC to fund their programs. For example, in 2018-19, Genome Atlantic secured $13.3 million in funding from provincial governments. By comparison, Genome Quebec managed to raise $237.2 million from the Province of Quebec. Further, although the six Centres collectively secured $347.2 million in 2018-19 from foreign sources (industry, government, institutions and foundations), Genome Atlantic received only $2.1 million. Note, however, that the data does not disaggregate between operational funding versus research funding.

Figure 7: Percentage of Operation Costs out of Total Budgets for GC and the Genome Centres
Text version
- 2009-10 2010-11 2011-12 2012-13 2013-14 2014-15 2015-16 2016-17 2017-18 2018-19
Genome Centres Operations to total budget  13.76% 16.66% 17.88% 16.74% 14.56% 16.17% 20.70% 18.60% 16.69% 18.45%
GC Operations to total budget  7% 10% 10% 11% 9% 9% 14% 11% 9% 9%
Figure 8: Operation Costs as a Share of Total Budget for the Genome Centres (2009-10 to 2018-19)
Text version
- 2009-10 2010-11 2011-12 2012-13 2013-14 2014-15 2015-16 2016-17 2017-18 2018-19
Genome Atlantic 17.51% 38.1% 44.8% 43.6% 40.9% 41.3% 57.6% 46.5% 46.7% 54.5%
Genome Quebec 13.39% 14.9% 18.9% 12.7% 10.0% 8.6% 9.1% 8.5% 8.0% 10.8%
Ontario Genome 9.11% 13.4% 17.3% 20.5% 14.2% 15.8% 21.4% 20.7% 17.7% 16.0%
Genome Prairie 21.56% 16.1% 15.9% 10.7% 12.4% 19.6% 24.7% 19.1% 16.5% 17.9%
Genome Alberta 22.99% 17.1% 20.1% 14.5% 15.1% 23.3% 33.1% 25.9% 19.9% 15.9%
Genome BC 13.61% 18.7% 15.8% 18.5% 17.0% 21.6% 30.8% 23.8% 21.4% 25.5%

4.0 Conclusions and recommendations

4.1 Conclusions

Relevance

The field of genomics has advanced significantly over the past two decades. However, further research is still required to enhance the understanding of genomics and contribute to the emergence of an internationally competitive Canadian bio-economy.

The evaluation found that government support to GC was important to ensure the coverage of Canada's strategic sectors and alignment with the national science vision, which could not have been accomplished through dispersed funding support from other sources. However, the time-limited nature of the ISED-GC funding agreement can impact GC's ability to secure long-term co-funding commitments and planning.

Performance

GC contributes to the increased breadth of knowledge in the field of genomics by supporting research in a number of multidisciplinary topics and sectors that address the needs of different regions across Canada and foster international collaborations. GC-funded researchers achieved a higher publication productivity compared to unfunded researchers (two papers more per researcher from 2014 to 2018).

Research supported by GC enabled the development of new genomics knowledge, which was subsequently used in practical applications in GC strategic sectors.

GC's programs allowed the participation of HQP in 84% of the 2014-18 GC-funded projects, where they benefited from the research experience, providing opportunities to move forward in their careers.

GC-funded researchers also benefited from the availability and quality of services provided by the GC-supported technology platforms. The platform experts provide advice and give access to researchers from various disciplines to leading-edge technologies needed for their projects at a competitive service cost. However, the continued technological competitiveness of the platforms may be further improved by focusing GC funding on technology development and better aligning operating funding for the five platforms, which also receive CFI funding.

GC-funded research results led to practical applications in GC strategic sectors. Although the number of Canadian genomics patents outpaces those of the top ten most productive countries in the field, the number of Canadian intellectual property rights (IP) retained is among the lowest, consistent with other research fields in Canada, but beyond GC's mandate.

Overall, GC has contributed to Canada being among the top ten performing countries in genomics research. However, Canada is outperformed by other nations in sequencing research that covers large populations, indicating that more support would be required to develop and reach international standards. GC's 2019 Strategic Vision report outlines strategies to help address this challenge.

Efficiency

The GC mandate focuses solely on genomics, unlike the mandates of other funding organizations in Canada which cover other fields as well. GC is also more capable of funding large-scale genomic projects. Further, it is the main national organization funding GE3LS research and extending its support to strategic sectors beyond health. GC also has a stronger regional presence through the Centres, with whom they work in collaboration. Following consultation with stakeholders, GC sets a national strategy and works collaboratively with the Centres and their regional strategies to set an overall agenda for genomics in Canada. While there is a great deal of collaboration among the Centres during the implementation of the funded projects, additional coordination at the application phase would help align their project submissions and minimize duplication of effort.

Funded researchers consider the GC funding application process to be clear and supported by an international peer review process that is transparent and equitable. The GC project reporting requirement was, however, considered onerous for lower cost projects and could be better aligned with the amount of funding provided. Moreover, GC faces data consistency challenges while reporting on funded research results and socio-economic impacts.

GC and the Centres have established measures to help ensure the efficiency of their operations. However, maintaining networking and outreach activities, necessary for establishing partnerships, can impact more the operations of smaller Centres.

GC's average annual operating cost as a share of the total budget from 2009-10 to 2018-19 was comparable to larger national organizations with a similar mandate. Moreover, among the Centres, the average annual operating cost share was lower for larger Centres, which may be benefiting from economies of scale and their ability to attract more funding from sources other than GC.

4.2 Recommendations

Based on the evaluation findings, the following recommendations may help support GC's efforts toward continuous improvements of operations and the achievement of its strategic objectives.

Recommendation 1: Technology Platforms

ISED's Science and Research Sector should ensure Genome Canada coordinates with the Canada Foundation for Innovation to better align the operating funding they provide to the technology platforms, in order to minimize duplication of effort and focus Genome Canada's funding on the technology development and competitiveness of the platforms.

Recommendation 2: Efficiency of Service Delivery

To improve the efficiency of service delivery, ISED's Science and Research Sector will ensure Genome Canada:

  • Considers making project reporting requirements commensurate with the project funding level to limit the reporting burden on researchers; and
  • Develops systematic data gathering techniques and defines concepts related to knowledge translation to better measure the socioeconomic impacts of funded projects, in collaboration with the Centres.

Sources

Canada's Science vision can be found at: https://www.ic.gc.ca/eic/site/131.nsf/eng/h_00000.html

Genome Canada Annual Impact Report Outcome Examples 2014-15

Genome Canada (2015). Performance, Evaluation, Risk and Audit Framework (PERAF): 2012–2017

Genome Canada, 2018 Corporate Plan, 2018

Genome Canada, Annual Report, 2017-2018: https://www.genomecanada.ca/annualreport/2017-2018/#5

Genome Canada Annual Impact Report (2016-2017).

Genome Canada Annual Impact Report (2017-2018).

Genome Canada (2017). Canada a leader in global coalition to accelerate genomic data sharing that will benefit Canadian patients, retrieved from: https://www.genomecanada.ca/en/news/canada-leader-global-coalition-accelerate-genomic-data-sharing-will-benefit-canadian-patients

Genome Canada (2019). Genomic Applications Partnership Program, retrieved from: https://www.genomecanada.ca/en/programs/translation/funding-opportunities/genomic-applications-partnership-program

Genome Canada (2019). Leading-Edge Technologies - Past Competitions, retrieved from: https://www.genomecanada.ca/en/programs/leading-edge-technologies/past-competitions

Genome Canada (2019). Membership to the Genomics Innovation Network and Core Operations Support Funds, retrieved from: https://www.genomecanada.ca/en/programs/leading-edge-technologies/past-competitions/membership-genomics-innovation-network-and-core

ISED, Evaluation of Industry Canada's Contribution to Genome Canada (2015)

Raymond Chabot Grant Thornton Consulting (2017). Genome Canada Performance Audit.

Science-Metrix Inc., Genome Canada Five-Year Evaluation, Transformative research, March 2014

Annex A: GC Technology Platform Funding Evolution

Text version

The figure in Annex A illustrates the funding evolution for the Genome Canada (GC) technology platform.

The legend on the left-hand side of the diagram provides a current list of the 10 GC-funded platforms in British Columbia, Alberta, Ontario and Quebec:

  • British Columbia
    • The Pan-Canadian Proteomics Centre
    • BC Cancer Agency Genome Sciences
    • Centre Genomics Technology Platform
  • Alberta
    • The Metabolomics Innovation Centre
  • Ontario
    • The Centre for Applied Genomics
    • The Centre for Phenogenomics
    • Network Biology Collaborative Centre
    • Canadian Data Integration Centre
  • Québec
    • McGill University and Génome Québec Innovation Centre
    • Centre for Advanced Proteomic and Chemogenomic Analyses
    • Canadian Centre for Computational Genomics

To the right of the legend is a timeline (starting in March 2001 and ending in February 2019) of the 10 key events which took place:

  • March 2001: Put in place research infrastructure to support science and technology platforms essential for the large-scale projects (10 platforms, cost of $42.7 million).
  • April 2004: One-time capital investment in platforms to be decided by the Centres (cost of $4.5 million).
  • July 2005: A new funding model that classifies the platform support to service, operations and technology development costs (open competition for operations and technology development, cost of $37.5 million for 6 platforms).
  • August 2009: Interim funding (2009 to 2011) to fund platforms' operations (open competition, cost of $22.7 million for 6 platforms).
  • April 2011: Open competition (April 2011 to March 2013) for platforms' operation support (5 platforms, cost of $24.3 million).
  • April 2013: Renewal of existing platforms (April 2013 to March 2015, cost of $29 million for 5 platforms).
  • April 2015: Open competition (April 2015 to March 2017, cost of $16 million for 10 platforms).
  • October 2015: Technology development funds (October 2015 to September 2017, cost of $8 million for 10 platforms).
  • April 2017: Open competition (April 2017 to March 2022, $45 million committed for years 1 to 3 for 10 platforms).
  • February 2019: Budget 2019 allocated $21 million for the 10 current platforms (years 4 and 5).