Evaluation of Industry Canada's Grant to the Institute for Quantum Computing

December 2013

Recommended for approval to the Deputy Minister by the Departmental Evaluation Committee on November 29, 2013

Approved by the Deputy Minister on December 12, 2013

Table of contents

• Executive Summary
• 1.0 Introduction
  • 1.1 Program Context
  • 1.2 Description of IQC
  • 1.3 Program Design and Governance
  • 1.4 Stakeholders
  • 1.5 Resources
  • 1.6 Expected Results of the Program and Logic Model
• 2.0 Methodology
  • 2.1 Evaluation Approach
  • 2.2 Objective and Scope
  • 2.3 Evaluation Issues and Questions
  • 2.4 Data Collection Methods
  • 2.5 Data Limitation
• 3.0 Findings
  • 3.1 Relevance
  • 3.2 Performance
• 4.0 Conclusions
  • 4.1 Relevance
  • 4.2 Performance

Acronyms, Abbreviations and Definitions used in this Report

List of Tables

List of Figures

Executive Summary

Program Overview

The Institute for Quantum Computing (IQC) is a multidisciplinary research organization within the University of Waterloo and is supported by a partnership of the federal and provincial governments and the philanthropy of Mike and Ophelia Lazaridis. Building on the University of Waterloo's internationally recognized strengths in mathematics and computer science, IQC was officially established as a research institute at the University of Waterloo in 2002.

In order to expand its leadership role and build on the reputation it had developed, IQC sought to create a research environment and facility conducive to growth. In 2008, the Institute launched a $300 million project to build, purchase and recruit the resources needed to support IQC's growth. The project included:

• the construction of an $80 million new facility to house IQC activities;
• the acquisition of $20 million worth of quantum science equipment;
• up to $100 million to establish and staff a world-leading program in quantum information science, including training programs and knowledge transfer mechanisms; and,
• the establishment of a $100 million endowment to secure the future operating costs of IQC.

As part of this project the Government of Canada's Budget 2009 allocated a $50 million grant, over five years, to IQC to support the construction and establishment of a new world class research facility that would contribute to achieving the goals of the federal Science and Technology (S&T) Strategy. The project was also supported by private donors (Mike and Ophelia Lazaridis), the Government of Ontario, and the University of Waterloo.

Evaluation Purpose and Methodology

In accordance with the Treasury Board Policy on Evaluation and the Directive on the Evaluation Function, the purpose of this evaluation was to assess the core issues of relevance and performance of Industry Canada's grant to IQC. The evaluation covers the period of 2009-10 to 2013-14 and findings are based on the analysis of multiple lines of evidence. The methodology included a document review, literature review, interviews, environmental scan, scientific data analysis, and case studies.

Findings

Relevance

There is a continued need to increase Canada's research and innovation capacity as a means to provide social and economic benefits to Canadian society. IQC responds to this need through a multi-disciplinary approach that spans the innovation spectrum and focuses on quantum information and quantum computing sciences, a technology area with widespread potential benefits. IQC is uniquely positioned, within Canada and internationally, to contribute to the development of quantum sciences and its related technologies.

Support for IQC is consistent with federal government priorities related to science and technology as set out in the 2007 S&T Strategy and subsequent Federal Budgets. The objectives and activities of IQC are also in line with Industry Canada's strategic outcomes.

Support for IQC is consistent with federal roles and responsibilities to encourage the development of science and technology and aligns with Industry Canada's mandate. In addition, the delivery of support to IQC through Industry Canada appears to be logical and complimentary with other funding initiatives.

Performance

Overall, IQC is achieving the majority of its immediate outcomes: IQC researchers have access to some of the best quantum information sciences facilities and equipment in the world; and IQC is attracting and developing top-ranked researchers and students. With respect to increased awareness and knowledge, IQC has attracted interest among a variety of audiences.

IQC's research has increased in intensity and excellence as demonstrated by improved collaboration networks, increased publication and citation rates, as well as its reputation in the scientific community. IQC is also on track to being recognized as a leader in quantum information.

IQC's current delivery structure has demonstrated efficiency and economy by leveraging additional resources, minimizing transaction costs and using existing University of Waterloo management processes.

Recommendations

Overall, the evaluation did not find any major issues with the grant to IQC and as a result makes no recommendations.

1.0 Introduction

This report presents the results of an evaluation of Industry Canada's Grant to the Institute for Quantum Computing (IQC). The purpose of the evaluation was to assess the relevance and performance of Industry Canada's grant to IQC. The report is organized into four sections:

• Section 1 provides the program context and profile of IQC;
• Section 2 presents the evaluation methodology along with a discussion of data limitations;
• Section 3 presents the findings pertaining to the evaluation issues of performance and relevance; and
• Section 4 summarizes the study's conclusions.

1.1 Program Context

IQC is a multidisciplinary research organization within the University of Waterloo and is supported by a partnership of the federal and provincial governments and the philanthropy of Mike and Ophelia Lazaridis. Building on the University of Waterloo's internationally recognized strengths in mathematics and computer science, IQC was officially established as a research institute at the University of Waterloo in 2002. IQC was created to foster pioneering research in quantum information science, as part of a larger vision that "harnessing quantum mechanics will lead to transformational technologies that will benefit society and become a new engine of economic development in the 21st century"^{Footnote 1}. The organization is guided by three strategic objectives:

• to establish Waterloo as a world-class centre for research in quantum technologies and their applications;
• to become a magnet for highly qualified personnel (HQP) in the field of quantum information science; and,
• to establish IQC as the authoritative source of insight, analysis and commentary on quantum information science.

In the years following the establishment of IQC, world-class researchers were recruited and scientific output and collaborations intensified. In order to expand its leadership role and build on the reputation it had developed, IQC sought to create a research environment and facility conducive to growth. In 2008, the Institute launched a $300 million project to build, purchase and recruit the resources needed to support IQC's growth. The project included:

• the construction of an $80 million new facility to house IQC activities;
• the acquisition of $20 million worth of quantum science equipment;
• up to $100 million to establish and staff a world-leading program in quantum information science, including training programs and knowledge transfer mechanisms; and,
• the establishment of a $100 million endowment to secure the future operating costs of IQC.

As part of this project, Budget 2009 committed to providing $50 million, over five years, to IQC to support the construction and establishment of a new world class research facility that would contribute to achieving the goals of the federal Science and Technology (S&T) Strategy. The project was also supported by private donors (Mike and Ophelia Lazaridis), the Government of Ontario, and the University of Waterloo.

1.2 Description of IQC

IQC brings together researchers to conduct collaborative research, provides training and mentorship opportunities to graduate students, and delivers various scientific outreach activities.

The research approach at IQC is fundamentally interdisciplinary, aiming to bring together a critical mass of researchers in computer science, mathematics, physical science and engineering. IQC's research breadth spans from the foundations of quantum information science to the development of quantum technologies. Theorists and experimentalists pursue a number of different but complementary research avenues to better understand and utilize quantum information. The focus is on three main applications: computing, communication, and sensing.

Quantum mechanics allow these applications to be more efficient and more precise than what can be done through classical devices. For instance, quantum computing is a future design for computers based on quantum mechanics. It uses the "qubit," or quantum bit, which can hold an infinite number of values as opposed to the binary 1s and 0s in a classic computer. It is believed that such a device can handle multiple operations simultaneously and can factor large numbers 10,000 times faster than today's computers. While the science is still at the research stage, the potential applications of this technology are innumerable.

IQC, in collaboration with the University of Waterloo's Faculties of Science, Mathematics and Engineering, offers graduate students opportunities to learn about and engage in research in quantum information through a wide range of advanced research projects and advanced courses on the foundations, applications and implementation of quantum information processing. Students complete the requirements of their home faculty alongside the specific requirements of the quantum information program. As well, a mentorship program for post-doctoral fellows has been developed.

IQC also pursues a broad outreach agenda aimed at sharing and explaining its research. Outreach efforts are tailored to different target audiences, from prospective students and faculty to the general public and partners from government, academia and industry. Activities include graduate fairs, public lectures, conferences, cultural collaborations, high school visits, publications, and social media.

1.3 Program Design and Governance

Industry Canada's Science Partnerships Directorate (part of the Program Coordination Branch within the Science and Innovation Sector) is responsible for the delivery of the federal grant to IQC, as well as the ongoing management and oversight of the funding. The funding agreement with IQC is one of a number of agreements within Industry Canada that use third-party organizations as a delivery mechanism for the government's S&T Strategy.

IQC is governed through an Executive Committee, a Board of Directors, an Executive Director and a Scientific Advisory Committee. Roles and responsibilities are as follows:

• Executive Committee: is made up of senior administrators from the University of Waterloo who provide guidance to IQC's Executive Director and senior management team. The Executive Committee is responsible for monitoring IQC's overall direction and performance.
• Board of Directors: is made up of internationally recognized leaders from academia, business and government. The Board meets twice a year, and provides strategic advice on all aspects of management including finances, planning, commercialization and outreach.
• Executive Director: oversees the IQC Directors, faculty and committees. Faculty members at IQC hold appointments in departments at the University of Waterloo and as such, are governed by the University's policies on appointment, promotion and tenure. All faculty participate in annual evaluations (performance assessments) conducted by their home departments. The Executive Director of IQC gives input to the heads of departments about the contributions of each member.
• Scientific Advisory Committee: is an independent body that meets on an annual basis (more often, if necessary). The Scientific Advisory Committee consists of seven internationally-recognized scientists in the field of quantum information.^{Footnote 2} It provides advice on research direction, focus, and long-term strategy of IQC and reports to the Executive Committee.

In addition, the institute tracks information on research, outreach and other contributions to IQC for its own membership renewal process. Members are elected to IQC for a period of five years. IQC holds monthly faculty meetings to discuss issues arising related to faculty and postdoctoral fellow hiring, visiting scientists, the graduate program, upcoming colloquia and seminars, scholarships and other matters as they arise.

Figure 1 depicts IQC's current organizational structure:

Figure 1: IQC's Current Governance Structure

1.4 Stakeholders

In addition to Industry Canada, a number of stakeholders have been and continue to be involved in the development of IQC. The most prominent of these stakeholders are as follows:

• Funding partners: Including Mike and Ophelia Lazaridis, the Government of Ontario, University of Waterloo, The Canada Foundation for Innovation (CFI), and The Natural Sciences and Engineering Research Council (NSERC).
• Industry partners: IQC works with industry partners such as COM DEV (a leading supplier to the aerospace industry), Blackberry, Excelitas Technologies (optoelectronics and electronic systems), and Neptec (3D machines for space, industrial and military applications).
• The Perimeter Institute for Theoretical Physics: With its focus on computer science and experimental physics, IQC is meant to complement the Perimeter Institute's programs in theoretical physics.
• Business Incubators: The science and technology ecosystem in the Waterloo Region includes other centres that are engaged in entrepreneurship and commercialization such as: The Waterloo Accelerator Centre, Communitech, and Accelerator for Commercialization Excellence.

1.5 Resources

Funding of $50 million over five years was provided, as a grant, by the federal government to:

• support the construction costs of a new building ($25 million);
• purchase small equipment ($5 million); and
• recruit and retain highly qualified personnel, conduct knowledge transfer/communication activities, support administrative/technicians staff, and cover material/supplies costs ($20 million).

The funding was allocated as follows:

IQC allocated $11.1 million of the expenditures in the People and Operations category to Research, HQP Development and Innovation. $3.1 million and $3.2 million were allocated respectively to Communication and Outreach and Management, Administration and Finance and $1.5 million to Information Systems and Technology. Figure 2 summarizes this breakdown as percentages of the total expenditures on People and Operations.

Figure 2: Breakdown of IQC Expenditures on People and Operations 2009-10 to 2013-14

1.6 Expected Results of the Program and Logic Model

The expected results of the $50 million federal investment in the IQC are:

• to increase knowledge in the various fields and sub-fields of quantum computing;
• to create new opportunities for students to learn and to apply new knowledge to the benefit of Canada;
• to brand Canada as the destination of choice for conducting research in quantum technologies, attracting the best in the world to Canada; and
• to position Canada to take full and privileged advantage of the economic and social benefits of research in this field.

The logic model for the program, represented in Figure 3, was developed in 2009 as part of IQC's Performance Measurement Strategy. The immediate and intermediate outcomes that align with the expected results of the $50 million federal investment were used to guide the assessment of the core evaluation issue "achievement of expected outcomes".

Figure 3: Logic Model of the Institute of Quantum Computing

2.0 Methodology

This section provides information on the evaluation approach, objective and scope, the specific evaluation issues and questions that were addressed, the data collection methods, and data limitations for the evaluation.

2.1 Evaluation Approach

Like many evaluations of government programs, this evaluation was based on expected outcomes of the program as stated in the program's foundational documents and logic model. The evaluation used a variety of research methods, including a document review, a literature review, interviews, case studies, an environmental scan, a scientific peer review, and an analysis of social media.

2.2 Objective and Scope

An evaluation of IQC is required under section 42.1 of the Financial Administration Act. In accordance with the Treasury Board Policy on Evaluation and Directive on the Evaluation Function, the purpose of this evaluation was to assess the core evaluation issues of relevance and performance.

The evaluation study covered the period from April 2009 to September 2013.

2.3 Evaluation Issues and Questions

Based on the program Performance Measurement Strategy, and subsequent consultations with the program, the evaluation addressed the following questions:

Relevance

1. Is there a continued need to increase Canada's research and innovation capacity? Does IQC respond to this need?
2. To what extent do the objectives and activities of IQC align with federal government priorities and Industry Canada's strategic outcomes?
3. Does support to IQC align with federal roles and responsibilities?

Performance

4. To what extent has IQC achieved its immediate outcomes?
   • To what extent have IQC research facilities and equipment improved?
   • To what extent has IQC attracted and developed highly qualified personnel (top-ranked researchers and students)?
   • To what extent have awareness and knowledge of quantum information and IQC increased?
5. To what extent is IQC on track to achieving its intermediate outcomes?
   • Has the research and training conducted at IQC increased in intensity and/or excellence?
   • Is IQC on track to being recognized as a leader in quantum information?
6. To what extent does the program demonstrate efficiency and economy?

2.4 Data Collection Methods

Multiple lines of evidence were used to address all evaluation questions. The data collection methods included a document review, a literature review, interviews, an environmental scan, a scientific peer review, data analysis, and case studies.

Document Review

The document review was conducted to gain an understanding of the program and to gain insight into both the relevance and the performance of the program. Key documents included Federal Budgets and Speeches from the Throne, IQC's Treasury Board Submission, other policy documents, relevant federal and provincial legislation, Departmental Reports on Plans and Priorities, Departmental Performance Reports, program business plans, annual reports, performance reports and the recipient-led mid-term evaluation of IQC. Internal policies and processes within IQC were also assessed to provide evidence of the extent to which resource optimization mechanisms to ensure efficiency and economy are in place.

Literature Review

The literature review primarily addressed the core evaluation issues of continued need and federal roles and responsibilities. Specifically, the literature review examined the continued need to increase research and innovation capacity and the role of federal funding in supporting R&D in Canada and other jurisdictions outside of Canada.

Interviews

The objective of the interviews was to gather in-depth information for evaluation purposes, including views, explanations and factual information that address the evaluation questions. The interviews were designed to obtain qualitative feedback from a range of respondents. The interviews were conducted in-person, in conjunction with two site visits to IQC, or by telephone if an in-person interview was not possible.

Interviews were conducted with a total of 38 participants and include the following types of respondents:

• IQC management (4)
• IQC Board member / University of Waterloo management (4)
• Industry Canada (4)
• Advanced level students (8 conducted as 2 group interviews)  
• Interviews were also conducted to support other lines of evidence, specifically the environmental scan (5), scientific peer review (6), and case studies (7)

Environmental Scan

An environmental scan was conducted to assess the role of IQC within the innovation and commercialization ecosystem of the Kitchener-Waterloo area or "Quantum Valley" vision. Quantum Valley is a term used to characterize the vision of the future of the Kitchener-Waterloo region. The vision includes creating a cluster of research around quantum information sciences, while fostering an entrepreneurial culture and supporting the development of a related technology sector focused on the commercialization of quantum technologies. Literature examining the theory and success of regional clusters in fostering innovation was reviewed. Interviews with 5 key stakeholders, who form part of the Quantum Valley vision, were conducted as follows:

• Business Incubators (2): Several organizations designed to support the creation, growth and success of entrepreneurial companies through an array of business support resources and services have been established in the Kitchener-Waterloo area. These include Communitech, and the Accelerator for Commercialization Excellence.
• Related Research Institutes (1): Perimeter Institute, as part of the cluster of complementary research.
• University of Waterloo (2): Representatives from the VeloCity program^{Footnote 3} and the Technology Transfer Office (within the University's Office of Research) were asked about processes in place to foster an entrepreneurial culture.

Scientific Peer Review

Scientific peer review is widely recognized as a key mechanism for assessing research quality and excellence. The evaluation team leveraged the existing experience and findings of the IQC Scientific Advisory Committee through a review of its annual reports and subsequent focused telephone interviews with select members of the Committee. Interview guides were designed to build upon the findings of the Scientific Advisory Committee panel and structured to require a low level of effort from panel members (e.g. opinions based on current knowledge was solicited rather than requiring panel members to review additional material). Additional interviews with independent scientific experts were also conducted. These included experts from the Massachusetts Institute of Technology (MIT), the California Institute of Technology (Caltech) and Hebrew University in Jerusalem. This line of evidence enabled the objective assessment of the overall quality of research outputs and of IQC more generally (i.e., as a world-leading facility, as a destination of choice for highly qualified personnel).

Data Analysis

Two types of data were analysed, including:

• Administrative data: Information on the operations of the Institute were analysed to assess the efficiency and economy of program delivery.
• Web analytics: Social media tracking and web metrics, as published in the 2013 annual report, were analyzed to measure awareness and knowledge of quantum information and IQC. Examples include trends in the numbers of Youtube hits for IQC videos and Twitter followers.

Case studies

In conjunction with the 2012 annual report, IQC prepared four case studies that demonstrate several scientific achievements that have led to transferable knowledge in the field of quantum information. The evaluation built upon three of these case studies by conducting seven interviews with Principal Investigators, Co-investigators, and industry partners who are the potential users of IQC research. This line of evidence addressed performance issues, specifically the achievement of immediate outcomes and whether IQC is on track to achieving intermediate outcomes.

2.5 Data Limitation

Timing was a limitation for this evaluation. To fulfill the requirements under Section 42.1 of the Financial Administration Act, this evaluation was scheduled to be completed by March 2014. As a result, the evaluation was conducted only one year following completion of the construction of the new facilities. This limited the evaluation's ability to assess the full impact of the new facilities on enabling new forms of advanced research. To mitigate this, the evaluation design included a mix of lines of evidence to assess the results to date as well as anticipated results.

3.0 Findings

3.1 Relevance

3.1.1 Is there a continued need to increase Canada's research and innovation capacity? Does IQC respond to this need?

The need to increase Canada's research and innovation capacity

In May 2007, the Prime Minister released the Government of Canada's national S&T Strategy, Mobilizing Science and Technology to Canada's Advantage, setting out a multi-year framework to improve Canada's long-term competitiveness and quality of life. This strategy was introduced to address Canada's economic and societal challenges, most notably Canada's widening productivity gap relative to the United States.^{Footnote 4} Since 2007, the labour productivity gap between Canada and the United States has continued to widen.^{Footnote 5}

It is generally accepted that the material standard of living of a society depends on productivity, which drives increases in average per capita incomes and business competitiveness, and that productivity is primarily the result of innovation.^{Footnote 6} According to the 2012 Science, Technology and Innovation Council (STIC) Report, innovation requires the introduction of knowledge or technology into the marketplace, where value is created, or into an organization, where efficiencies are generated.^{Footnote 7} At the heart of the innovation process are the people who "generate the ideas and knowledge…and then apply this knowledge and the resulting technologies, products and services in the workplace and as consumers."^{Footnote 8}

Science and technology, including research and development, involves the creation of new knowledge or technology.^{Footnote 9} Canada's science, technology and innovation ecosystem involves numerous players, including governments, businesses, universities and colleges, non-governmental organizations, communities and individuals. The links among these players facilitate the exchange and creative deployment of the knowledge, capital, talent and other resources required for innovation. Specifically, universities are seen to be essential in the innovation system, both producing and attracting the human capital needed for innovation.^{Footnote 10}

IQC's response to these needs

IQC responds to the needs outlined above through a multi-disciplinary approach that spans the innovation spectrum and focuses on quantum information and quantum computing sciences, a technology area with widespread potential benefits.

As part of the University of Waterloo, IQC is positioned within a unique science and technology ecosystem in Canada. It is located close to the Perimeter Institute, which is considered a world-leader in theoretical physics and supports IQC by conducting basic research in quantum physics. IQC's research includes six academic faculties and is focused on the full continuum of research related to quantum computers and quantum information. This includes 'basic research' (experimental or theoretical work, undertaken without any particular application or use in view); 'applied research' (original investigations directed towards a specific application); and 'experimental development' (systematic work directed at improving or producing new materials, products or devices) – all essential components of the innovation system.^{Footnote 11}

Literature suggests that on the commercialization end of the innovation spectrum, the Waterloo region has a well-developed and high-performing technology cluster. The University is highly integrated into this cluster and is actively supporting its development by building a large pool of highly-skilled labour, patenting new technologies, spin-off companies and other entrepreneurial activities. In this regard IQC has developed a number of relationships with the University of Waterloo Programs and not-for-profit organizations to help facilitate interaction with the existing network of local and global firms. The following organizations are part of the science, technology and innovation ecosystem in the Waterloo Region and are engaged with IQC to help facilitate entrepreneurship and commercialization of research:

• The University of Waterloo Commercialization Office provides services in the form of intellectual property assessments, investments and management (e.g. patents, copyright); commercialization-pathway assessments and strategies; and prototype-development and assisting in writing applications to government programs;
• The University of Waterloo VeloCity program supports students through workshops, discussions and networking events; mentoring, advice and free work spaces to build start-ups; and a venture fund that allows students to compete for a $25,000 start-up grant;
• Communitech supports tech companies at all stages of their growth and development—from start-ups to rapidly-growing mid-size companies, and large global players. This includes facilitating partnerships between industry and academics, such as IQC's participation under the DATA.BASE program that seeks to invest in data capturing capabilities and improve data mining techniques;
• The Accelerator Centre cultivates technology entrepreneurship by promoting commercialization of research and technology rising out of academic institutions.
• The Quantum Valley Investments is a $100 million private fund that invests in breakthroughs in quantum information science that have the potential to lead to new commercializable technologies and applications. Currently, two IQC faculty members sit on the Investment Fund's scientific advisory committee.

According to the Jenkins Report (2011), an ecosystem such as this is essential for effective collaboration between business and higher education and eventual commercialization of research.^{Footnote 12} It also helps ensure that Waterloo (and Canada for that matter) is positioned to benefit from commercialization opportunities related to quantum technologies.^{Footnote 13}

As indicated by interviewees, quantum technologies (including computers and other devices) are a logical extension of the current technology trajectory^{Footnote 14} and will have enormous advantages over classical devices that will likely have widespread social and economic impacts. For example, according to David Cory, Chair of the Quantum Valley Investments Scientific Advisory Committee and IQC faculty:

Quantum mechanics is the ultimate law of nature and when we build quantum devices we can realize efficiencies that greatly exceed any existing classical devices. Today we have robust quantum technologies that can be deployed as sensors, actuators, communication channels and small processors. The opportunity is to creatively connect such devices to important applications in a broad range of areas including medical, health, energy, security, environment, nanoscience, nondestructive testing, etc…(http://quantumvalleyinvestments.com/management/)

In addition, the National Research Council Strategic Technical Information Analysis (STIA) Assessment of Quantum Capability (2010) found that there is a high capacity for quantum applications to be disruptive.^{Footnote 15} While several companies have already entered the market with products in the area of quantum key distribution, encryption, and communication networks, certain technical issues are still being observed and most projects are still at the demonstration stage of development. This report also found that the quantity and quality of IQC research was highly regarded – in 2010 IQC was ranked fifth on NSERC's top 20 international list for quantum capabilities and was the only Canadian organization on the list.^{Footnote 16}

Interviews with independent quantum scientists also indicated that IQC has one of the largest quantum computing programs in the world and is setting the international research agenda in a number of quantum information areas. It has consistently attracted top-ranked researchers, is actively partnering with internationally recognized researchers and has partnered with a number of private sector organizations, all of which are viewed to be essential to improving the strength of Canada's science and technology.^{Footnote 17}, ^{Footnote 18}, ^{Footnote 19}

3.1.2 To what extent do the objectives and activities of IQC align with federal government priorities and Industry Canada's strategic outcomes?

The S&T Strategy outlines the Government's intention to foster three distinct Canadian S&T advantages: a Knowledge Advantage, a People Advantage, and an Entrepreneurial Advantage. Industry Canada's 2012-13 Report on Plans and Priorities emphasized the continued need to focus on the S&T framework's three criteria. IQC, in its activities and mandate, are aligned with all three:

• Knowledge Advantage: is based on the premise that Canadians must be positioned at the leading edge of the important developments that generate health, environmental, societal, and economic benefits. The S&T Strategy identifies information and communications technologies as one of four priority knowledge areas. IQC is directly aligned with this advantage by conducting ground-breaking/leading-edge research with a particular focus on interdisciplinary and/or international collaboration related to quantum information and quantum computing;
• People Advantage: is based on the premise that Canada must be a magnet for the highly skilled people we need to thrive in the modern global economy with the best-educated, most-skilled, and most flexible workforce in the world. IQC is aligned with this advantage by providing training and learning opportunities to highly qualified personnel;
• Entrepreneurial Advantage: is based on the premise that Canada must do more to translate knowledge into commercial applications. IQC research projects partner with private and industrial sectors to facilitate knowledge transfer. IQC is actively working with the existing Waterloo high-tech cluster infrastructure to help realize the Quantum Valley vision. The vision includes creating a cluster of research around quantum physics, while fostering an entrepreneurial culture and supporting the development or a related technology sector focused on the commercialization of quantum technologies.

Subsequent Budgets have reiterated the Government's commitment to the S&T Strategy Advantages. Some examples of these references are outlined in Table 2:

In addition, support for IQC aligns with Industry Canada's priorities under the Science, Technology and Innovation Capacity Program Activity of Industry Canada's Program Alignment Architecture (PAA). This Program Activity contributes to the Strategic Outcome: Advancements in Science and Technology, Knowledge, and Innovation Strengthen the Canadian Economy. Industry Canada's grant to IQC also aligns with and supports the department's priority to foster a knowledge-based economy by further developing the federal S&T Strategy and continuing to build Canada's advantages in knowledge, people and entrepreneurship.^{Footnote 20}

3.1.3 Does the support to IQC align with federal roles and responsibilities?

Alignment between support for IQC, the mandate of Industry Canada and the federal government's roles and responsibilities

Industry Canada's mandate for supporting Canadian S&T activities and policy goals stems from the Department of Industry Act, 1995. Subsection 4(1) defines the powers, duties and function of the Minister, which include matters related to industry, technology, and science in Canada. In exercising these powers, the Minister is responsible for initiating, recommending, coordinating, directing, promoting and implementing national policies, programs, projects and practices with respect to the objectives set out in section 5 of the Act.

The grant to IQC is provided under the authority of paragraph 14(1) of the Act^{Footnote 21} and is consistent with the objectives established under section 5(d) "[to] encourage the fullest and most efficient and effective development and use of science and technology" and 5(e) "[to] foster and promote science and technology in Canada."

The legitimacy of the federal government's role in supporting S&T is supported by a review of policy documents and peer-reviewed literature. S&T policy-related documents outlining the rationale for such support were observed in multiple jurisdictions, including the US,^{Footnote 22} Australia,^{Footnote 23} Japan,^{Footnote 24} the OECD,^{Footnote 25} and Europe^{Footnote 26}. These policy statements and recommendations all identify a key role for direct and indirect public sector support for research and innovation in order to ensure the country/region achieves a strong and competitive economy. Importantly, this support is needed for basic and applied research conducted in the higher education sector, including activities that contribute to downstream business innovation.

In the Canadian context, the importance of government support for basic and applied research was highlighted in the Jenkins Report (2011), "The federal and provincial governments play an important role in fostering an economic climate that encourages business innovation–for example, by supporting basic and applied research and related training of highly qualified, skilled people […] the higher education and government sectors are key players in Canada's innovation system and complement the role of business."^{Footnote 27} The contributions of the higher education sector to the innovation system have also been highlighted in numerous policy documents and academic literature.^{Footnote 28}, ^{Footnote 29} In fact, the higher education sector is currently the second largest performer of R&D in Canada (about 38% based on expenditures) and relies on the federal government to provide for about 25% of these activities. Universities are widely seen to play a strong and strategic role in the Canadian innovation system,^{Footnote 30} including the development and maintenance of a highly skilled workforce^{Footnote 31},^{Footnote 32}.

In addition, Industry Canada's grant to IQC appears to be consistent with support provided to quantum information sciences by governments in other countries. All interviewees, including researchers from other countries, indicated that other governments provide funding to quantum research. Although it is difficult to determine the exact amount of funding, a steep increase in international publication rates suggests that governments are supporting researchers in this field. The National Research Council Strategic Technical Information Analysis (STIA) Assessment of Quantum Capability (2010)^{Footnote 33} reports that there was a 300% increase internationally in publications between 2000 and 2009, with the top 20 organizations located in the following countries (shown alphabetically):

• Australia
• Canada
• China
• Germany
• Japan
• Singapore
• UK
• USA

Degree of overlap between Industry Canada's contribution to IQC and other government programs

IQC receives funding from a variety of sources, including federal funds provided by NSERC, Canadian Institute for Advanced Research (CIFAR) and CFI. All three of these funding initiatives are within the Industry Canada portfolio but administered independently from the department. Generally speaking, they each support research, equipment and networking through competitive processes. Although there is potential for overlap between these funding sources, interviewees pointed out that IQC is already making use of these other funding initiatives and that without Industry Canada's grant there would not have been enough funding to cover all the targeted aspects of its proposed $300 million project.

Moreover, Industry Canada's grant and the other federal funding sources (such as NSERC, CIFAR and CFI) are guided by the S&T Strategy and therefore have supportive rationales and goals. In terms of implementation approaches, the flexible design of the Industry Canada's grant^{Footnote 34} allowed IQC to manage the funding in a way that eliminated conflicts with other funding sources. In this regard, interviewees reported a high degree of strategic thinking and coordination around how the different funding sources fit together.

3.2 Performance

3.2.1 To what extent has IQC achieved the following expected immediate outcomes?

• Improved research facilities and equipment
• IQC attracts and develops HQP (top-ranked researchers and students)
• Increased awareness and knowledge of quantum information and IQC

Improved research facilities and equipment

In September 2012, IQC expanded into its new headquarters, the Mike and Ophelia Lazaridis Quantum-Nano Centre (QNC), a 285,000 square foot (sq. ft.) facility. Within QNC, the institute occupies a total of 51,832 sq. ft. of lab space and shares the 6,700 sq. ft cleanroom/fabrication facility with the University of Waterloo Institute for Nanotechnology. This new facility has over twice the square footage compared to its previous facilities and is designed to control for vibration, temperature fluctuation and electromagnetic radiation. It also has spaces to accommodate conferences, public lectures, and common areas for scientists to meet and collaborate.

The building was completed in 2012-13 and some researchers are still in the process of moving to the new location, which limits the ability of the evaluation to assess the impact the facility is having on IQC. Nonetheless, the evaluation was able to assess early impacts and gather opinions of its potential from researchers internal and external to IQC, as well as from students attending the institute.

Interviewees, including scientific peers independent of IQC, indicated that the new facilities are "state-of-the-art", "the best in the world" and "a huge step up over the past five years". It was also noted that although different institutions might have the same or similar equipment, interviewees did not know of any single institution with the breadth of equipment available to IQC researchers. Having the breadth of equipment on site is important for conducting experiments because it allows researchers to control the entire process (including fabrication of highly sensitive devices). This helps reduce inconsistencies and improves the quality of research. For example, one researcher explained that he recently developed a new tool that allows his team to work with magnetic fields with superconductors without contamination. With this tool the researcher and his team can work more precisely than their competitors.

Industry partners indicated that the equipment available to IQC researchers is unique and allows the researchers to work with industry in areas that would not otherwise be possible. For example, according to interviewees it would not be cost-effective for industry to build a lab similar to IQC's optics lab, and therefore it is seen to be essential for IQC's partnerships on the Quantum Communications Network via Satellite project.

In addition, interviewees indicated that because of the facilities and the equipment, IQC has been able to attract top-ranked researchers and students who would have gone elsewhere. For example, students indicated that IQC is on par with bigger labs, but where IQC stands out is the collaborative approach that fosters interaction and provides opportunities for students to work with the labs and equipment.

IQC attracts and develops HQP (top-ranked researchers and students)

The ability to attract and develop HQP (top-ranked researchers and students) is a key measure in any academic institution's success. As show in Figure 4, over the past five years, IQC has seen an overall growth in the number of graduate students, Postdoctoral Fellows and researchers.

Figure 4: IQC Membership

Source: IQC Annual Report to Industry Canada, 2013

With regard to graduate students, IQC launched a collaborative graduate studies program in 2009-10 that includes courses in quantum information processing and theory, quantum algorithms, open science systems, nanoelectronics, quantum electronics and photonics. The program attracted 104 applications in its first year, and approximately 120 applications every year since (those applying directly to the quantum information graduate program). IQC admits approximately 20% of those applying with 20-25 new graduate students accepted annually. Currently there are 103 students actively participating in the program. According to interviewees this is one of the largest (in terms of number of students and breadth of courses offered) graduate studies programs in quantum information sciences in the world and it is consistently attracting students from top-ranked international universities.

IQC has also instituted a growing Postdoctoral Fellowship program that currently includes 46 members, which is more than double the number in 2009. Positions in this program are highly competitive, as seen with the rising number of applicants in recent years. In the past five years there have been 87-119 applicants annually with IQC accepting 5-18 new Postdoctoral Fellows each year. Interviews with Postdoctoral Fellows indicated that IQC's reputation and quality of research were key factors in their decision to choose IQC over other institutions.

With respect to researchers, IQC has maintained approximately 20 faculty members, 3 Research Assistant Professors and 14 associate members/long term visitors. The goal for the institute is to recruit a total of 33 faculty members with a focus on attracting the right researchers to compliment existing research areas and improve the overall caliber of the institute. Interviewees, including members of the Scientific Advisory Committee, concurred with the preference for prioritizing the quality of new hires over quantity.

Since 2009, IQC was able to recruit Professor David Cory, one of the leading quantum experimentalists and Canada Excellence Research Chair in quantum information, and more recently Dr. Amir Yacoby of Harvard University as a visiting faculty member. Although these are major accomplishments, interviewees acknowledged that recruiting researchers is extremely competitive and scientific peers felt that IQC should continue developing capacity in experimental research in order to meet future needs.

Increased awareness and knowledge of quantum information and IQC

IQC has established a communications and scientific outreach team that helps to share its research with a variety of audiences including other research institutions, government, industry and the general public. IQC organizes annual meetings, workshops, conferences, and uses social media (such as Facebook, Twitter and Youtube) to highlight the research undertaken at the institute. Interviewees indicated that the communications team is very important and that IQC plays a leading role in the quantum information sciences field with respect to raising awareness and disseminating knowledge to the scientific community, students, as well as to the general public.

Since 2009 IQC has held 27 public outreach events that attracted approximately 5,500 participants. Events include specialized conferences, workshops, summer school for young students, open houses, public lectures and panel discussions. Within this same timeframe, 675 people toured IQC's facilities, including 580 academics (including visitors from 108 research institutions worldwide), 61 industry members and 34 people from government. It is challenging for the evaluation to assess the impact in terms of improving general awareness and knowledge of quantum information and IQC because no information, beyond the numbers of events and visitors, is collected.

According to data tracked by the analytic services associated with social media outlets, IQC social media presence has grown in recent years. Table 3 below shows the cumulative increase in social media attention at IQC since 2010.

IQC has garnered significant attention on social media sites. Although the type of online conversations and shared content can vary widely, the public is increasingly using social media as a way to learn about science by following scientific opinions, reading updates from conferences and meetings, learning about upcoming events and watching scientific clips. As such, social media offer a powerful way for scientists to boost their professional profile and act as a public voice for science.

3.2.2 To what extent has IQC achieved the following expected intermediate outcomes?

• Increased research intensity and excellence
• IQC is recognized as a leader in quantum information

Increased research intensity and excellence

To determine research intensity, the evaluators reviewed the trends in the numbers of research collaborations and publications. To assess excellence in research the evaluators relied on the number of citations of IQC research, rankings of journals in which IQC researchers published, as well as opinions gathered through interviews with scientific peers, business incubators and industry partners.

Collaboration in research

Collaboration is widely regarded as a practice that can improve the quality, impact and reach of research. Academic collaboration with fellow university scientists drives knowledge creation, whereas collaboration with industry partners drives knowledge application.^{Footnote 35} At IQC, researchers from a variety of disciplines come together to form the Institute. IQC spans six departments including: Applied Mathematics; Physics & Astronomy; Chemistry; Combinatorics & Optimization; Computer Science; and Electrical & Computer Engineering. Areas of specialization explored by IQC's theoretical and experimental scientists include:

• Spin-Based Quantum Information Processing
• Quantum Algorithms Nanoelectronics-Based
• Quantum Information Processing
• Quantum Complexity
• Optical Quantum Information Processing
• Quantum Error Correction
• Fault Tolerance
• Quantum Cryptography

IQC research is further enhanced by collaborations between researchers in a variety of fields from different institutions and with industry partners worldwide. The number of collaborations, including research projects and joint publications, with researchers from other institutions has grown from 141 researchers from 61 institutes in 2009 to 221 researchers from 185 institutions in 2012-13.^{Footnote 36} The growth in the number of collaborations, with institutions worldwide, allows IQC researchers opportunities to advance knowledge creation by building social capital, enhancing creativity, integrating specialized skills, and pooling resources.^{Footnote 37}

In terms of collaborations with industry partners, the case studies revealed a number of industry partnerships in applied and experimental projects. For example, the Quantum Communications Network via Satellite project is collaborating with COM DEV, a global designer and manufacturer of space hardware, to create global quantum communications networks by sending equipment into orbit on a satellite. Although still in the testing phases, interviewees indicated that they are working towards developing a quantum communications network. Another example involves an IQC researcher who works with industry partner ID Quantique^{Footnote 38} to test the security of quantum cryptography systems. The company is able to maintain a level of security and performance of their systems by collaborating with the IQC researcher who tests the systems' security. In both of these examples, relationships with industry happened at the researcher level such that partners work directly with IQC researchers and relationships were developed through networking opportunities.

Industry partners indicated that IQC researchers provide the expertise and theoretical knowledge as well as access to world-class facilities and equipment to address some of the practical problems identified by industry. Other interviewees indicated that partnerships with industry are "the next frontier" and that quantum technologies are getting closer to the stage where the private sector is getting more interested.

IQC researchers: publications and citations

Publications are one of several indicators of research output used by organizations that create or support the creation of new knowledge. In addition, citation count is used to gauge the importance of a publication by counting the number of times it has been cited by other scholars. Although a formal bibliometric analysis was beyond the scope of this evaluation, the evaluators nonetheless reviewed IQC's publication and citation rates, as reported in the annual reports to Industry Canada. These reports were approved by the University of Waterloo Board of Governors. Since IQC's inception in 2002, there has been an overall increase in the number of publications and citations, as shown in Figure 5^{Footnote 39} below.

Figure 5: Annual Number of IQC Publications and Annual Number Citations of all Papers Published by IQC Researchers

Source: IQC 2013 Annual Report

Beyond counting publications and citations, journal ranking is widely used in academic circles to assess an academic journal's impact and quality. IQC researchers regularly publish in world-leading journals. Between 2009 and 2012, 113 (or 20%) of IQC publications were published in the following journals (reported as prominent by IQC):

• Nature
• Nature Photonics
• Nature Physics
• Nature Communications
• Physical Review Letters
• Science
• Symposium on Theory of Computing proceedings
• Foundations of Computer Science proceedings
• Journal of Mathematical Physics

In addition, interviewees, independent from IQC, indicated that IQC's research "leaves a huge foot print, both in terms of sheer volume but also in the quality". Furthermore, independent interviewees indicated that IQC is "driving the agenda" in a number of quantum science areas. This is corroborated by the National Research Council Strategic Technical Information Analysis (STIA) Assessment of Quantum Capability (2010) that concluded IQC was ranked fifth internationally for quantum capabilities.^{Footnote 40}

IQC is recognized as a leader is quantum information

The previous findings indicate the following: IQC's researchers have access to some of the best quantum science facilities and equipment in the world; IQC is attracting and developing top-ranked researchers and students; IQC's research has increased in intensity and excellence; and that scientific peers expressed that IQC is one of the most respected institutions in this field of research. Based on these findings, the evaluation concludes that at this stage of development IQC is on track to being recognized as a leader in quantum information. As research moves along the innovation spectrum, it will become increasingly important to measure how IQC is having an impact on industry.

3.2.3 To what extent does the program demonstrate efficiency and economy?

The efficiency and economy of the delivery of Industry Canada's grant to IQC was considered over the following areas: the general efficiency and economy of using a third-party delivery model and evidence that mechanisms are in place to facilitate an efficient and economic use of funds.

Efficiency and Economy of Third-Party Delivery Model

The ability to leverage additional funding is a principle financial benefit of conducting research through an independent institute. As previously discussed, the $50 million Industry Canada grant was provided to support a larger $300 million project, which represents a 1:5 leverage ratio. Furthermore, as shown in Figure 6, IQC reported that its researchers had attracted an additional $32.1 million in grants, donations, gifts, and awards between 2009-10 and 2012-13 to support research undertaken at the institute. This demonstrates IQC's ability to gain wider support and remain relevant to a broad number of stakeholders.

Figure 6: Breakdown of Grants, Donations, Gifts and Awards Received by IQC, 2009-10 to 2012-13

Source: IQC Annual Reports to Industry Canada, 2011-12 and 2012-13

In addition to the ability to leverage, a third-party delivery model has a number of advantages that would not be possible if IQC were a government lab. These include an increased ability for IQC to attract highly qualified representatives, including those from industry, to participate in the institute's governance structure; an increased ability to take risks and partner with industry; and, as an institute within the University of Waterloo, IQC is better able to offer advanced courses and train students.

According to the literature, the principle drawback of this delivery model is the additional transaction costs incurred by arm's length organizations and by Industry Canada. In the case of IQC, there do not appear to be significant transaction costs resulting from the delivery structure. Industry Canada allocated roughly one FTE to manage the delivery of the grant. This included drafting the foundation documents, developing and overseeing the implementation of the funding agreement, and liaising with IQC. Although these activities are considered transaction costs, interviewees noted that there are a number of efficiencies (such as developing insights on science policy and expertise in managing funding agreements) that result from having one program manage all of the department's grants and contributions to arm's length science organizations.

At the University of Waterloo, the Office of Research is responsible for administration of the funding. IQC is responsible for developing annual budgets to access the funding, liaising with Industry Canada and providing the Minister with an annual report on the activities and results of the previous year as well as anticipated activities and results for the following year. It is important to note that the annual budgets are not limited to the Industry Canada grant; they are developed for IQC as a whole and therefore are not considered to be additional transaction costs (i.e., budgets would be developed, with or without Industry Canada funding). Also, IQC purposefully designed the annual report to Industry Canada to support outreach activities, a key priority for the institute, in order to maximize the benefits of these requirements.

Analysis of Governance Structure

In 2006, NSERC commissioned a review of IQC that was led by a committee of independent experts. This review concluded that the governance structure of the institute is appropriate to ensure proper guidance to the IQC management team, specifically noting the commitment from the University of Waterloo Executive, the impressive list of Board of Directors and the eminent international scientists on the Scientific Advisory Committee.^{Footnote 41}

The Board of Directors continues to be made up of internationally recognized leaders from academia, business and government and is seen to be effective at providing overall strategic direction to the institute, including outreach and ensuring that IQC maintains a commercialization orientation. The Scientific Advisory Committee includes internationally recognized researchers with expertise related to quantum information sciences and is responsible for advising on research direction, recruitment and faculty performance assessments.

With respect to administration, IQC uses the University administrative functions when appropriate, to reduce duplication. IQC is overseen by an Executive Committee of Senior University of Waterloo officials that is responsible for reviewing academic direction, annual budgets and major capital expenditures. This committee is chaired by the Vice President of Research and includes the Deans of the faculties of Mathematics, Sciences and Engineering. In the past, the Dean of Sciences was responsible for financial management of the grant. However, given the collaborative nature of IQC, it has recently been decided to consolidate this function under the Executive Committee to avoid duplication between faculties.

This structure ensures that IQC is subject to same level of scrutiny and internal controls as a faculty department, including major capital investments such as purchasing equipment. For example, as a result of the governance structure, IQC is responsible for ensuring that management of the Industry Canada grant adheres to the Standard from the Tri Council Memorandum^{Footnote 42}, is subject to the same budget review process, and is considered in the scope of the University's annual audit.

4.0 Conclusions

4.1 Relevance

• There is a continued need to increase Canada's research and innovation capacity as a means to provide social and economic benefits to Canadian society. IQC responds to this need through a multi-disciplinary approach that spans the innovation spectrum and focuses on quantum information and quantum computing sciences, a technology area with widespread potential benefits. IQC is uniquely positioned, within Canada and internationally, to contribute to the development of quantum sciences and its related technologies.
• Support for IQC is consistent with federal government priorities related to science and technology as set out in the 2007 S&T Strategy and subsequent Federal Budgets. The objectives and activities of IQC are also in line with Industry Canada's Strategic Outcome: Advancements in Science and Technology, Knowledge, and Innovation Strengthen the Canadian Economy.
• Support for IQC is consistent with federal roles and responsibilities to encourage the development of science and technology and aligns with Industry Canada's mandate. In addition, the delivery of support to IQC through Industry Canada appears to be logical and complimentary with other funding initiatives.

4.2 Performance

• Overall, IQC is achieving the majority of its immediate outcomes: IQC researchers have access to some of the best quantum information sciences facilities and equipment in the world; and IQC is attracting and developing top-ranked researchers and students. With respect to increased awareness and knowledge, IQC has attracted interest among a variety of audiences.
• IQC's research has increased in intensity and excellence as demonstrated by improved collaboration networks, increased publication and citation rates, as well as its reputation in the scientific community. IQC is also on track to being recognized as a leader in quantum information.
• IQC's current delivery structure has demonstrated efficiency and economy by leveraging additional resources, minimizing transaction costs and using existing University of Waterloo management processes.

Overall, the evaluation did not find any major issues with the grant to IQC and as a result makes no recommendations.