Evaluation of Aerospace Programming

Audit and evaluation branch

January 2019

This publication is available online at http://www.ic.gc.ca/eic/site/ae-ve.nsf/eng/h_03893.html.

To obtain a copy of this publication or an alternate format (Braille, large print, etc.) please fill out the Publication Request Form at www.ic.gc.ca/Publication-Request or contact:

Web Services Centre
Innovation, Science and Economic Development Canada
C.D. Howe Building
235 Queen Street
Ottawa, ON K1A 0H5
Canada

Telephone (toll-free in Canada):
1-800-328-6189
Telephone (Ottawa):
613-954-5031
TTY (for hearing-impaired):
1-866-694-8389
Business hours:
8:30 a.m. to 5:00 p.m. (Eastern Time)
Email:
info@ic.gc.ca

Permission to Reproduce

Except as otherwise specifically noted, the information in this publication may be reproduced, in part or in whole and by any means, without charge or further permission from Innovation, Science and Economic Development Canada, provided that due diligence is exercised in ensuring the accuracy of the information reproduced; that Innovation, Science and Economic Development Canada is identified as the source institution; and that the reproduction is not represented as an official version of the information reproduced, nor as having been made in affiliation with, or with the endorsement of, Innovation, Science and Economic Development Canada.

For permission to reproduce the information in this publication for commercial purposes, please fill out the Application for Crown Copyright Clearance at www.ic.gc.ca/copyright-request or contact the Web Services Centre mentioned above.

© Her Majesty the Queen in Right of Canada, as represented by the Minister of Innovation, Science and Economic Development Canada, 2018.

Cat. No. Iu4-265/2019E-PDF
ISBN 978-0-660-31500-3

Aussi offert en français sous le titre Évaluation de la programmation aérospatiale.

List of abbreviations and acronyms

A&D
Aerospace, Defence, Space and Security
CARIC
Consortium for Aerospace Research and Innovation in Canada
CRIAQ
Consortium for Research and Innovation in Aerospace in Quebec
GDP
Gross Domestic Product
ISED
Innovation, Science, and Economic Development Canada
MDA
MacDonald Dettwiler and Associates Ltd. (MDA Systems)
MRO
Maintenance, Repair and Overhaul
R&D
Research and Development
SADI
Strategic Aerospace and Defence Initiative
SIF
Strategic Innovation Fund
SME
Small and Medium-Sized Enterprise
STAR
Space Technology and Advanced Research
TDP
Technology Demonstration Program
TRL
Technology Readiness Level

List of figures


List of Tables


Executive summary

This report presents the results of an evaluation of two separate Innovation, Science and Economic Development Canada (ISED) programs – Bombardier's C-Series program and the Technology Demonstration Program (TDP).  The C-Series program was established in 2008 and later extended to end in March 2020, while the TDP, established in 2013, was consolidated into the Strategic Innovation Fund which was announced in Budget 2017.

Program overview

The C-Series program and TDP both supported large-scale research and development (R&D) projects in the aerospace, defence, space and security industries to strengthen their global competitiveness by accelerating innovation and leveraging private sector investment.  The C-Series program was launched in 2008 and has provided $470 million in repayable contributions over time, while the TDP was established in 2013 with a commitment of $187.5 million in non-repayable contributions.

The C-Series program helped fund R&D activities related to the C-Series family of aircraft and new generic aerospace technologies.  The TDP has supported three large-scale collaborative technology demonstration projects as well as a national network (the Consortium for Aerospace Research and Innovation in Canada or CARIC) to facilitate communication and collaboration among industry, academic and research institutions.

Evaluation purpose and methodology

The objective of this evaluation was to address the issues of relevance, performance and design and delivery in accordance with the Policy on Results and the Financial Administration Act.  The evaluation covered the period from April 1, 2013 to March 31, 2018.  For the C-Series program, the evaluation was calibrated to build on the findings of the 2013 C-Series evaluation to assess the progress toward the immediate and intermediate outcomes, while the TDP has never been evaluated before.  The evaluation employed multiple data collection methods: a document review, literature review, project document and data review, interviews and case studies.

Findings

Through programs such as the C-Series and TDP, the federal government promotes R&D and encourages private sector investment.

The C-Series program and TDP have achieved their expected outcomes.  Specifically, the programs have helped strengthen the technological and financial capacity of the industry, while also enhancing collaboration on R&D projects between industry, academic and research institutions.  In addition, the programs have assisted in creating and maintaining R&D employment, augmenting the talent pool, and enabling the emergence of aerospace expertise.  Further, the Consortium for Aerospace Research and Innovation in Canada (CARIC) has helped increase networking opportunities and membership for industry, academic and research institutions across Canada.  Finally, the C-Series program has contributed to R&D that played a part in the eventual commercialization of the C-Series aircraft, as well as generated some benefits for C-Series suppliers.

In terms of design and delivery for the TDP, some phases of the TDP application and approval processes took longer than anticipated, although this was mainly due to circumstances outside the control of the program.  The long and unpredictable waiting period resulted in recipients losing project partners, especially SMEs and universities.  However, in light of lessons learned from the first two rounds of TDP funding, the program modified the application process for the third round to allow for better communication and transparency.

Lessons learned

Given that the TDP was consolidated into the Strategic Innovation Fund and that the C-Series program was a specific-purpose program, the following lessons learned were developed to inform the design and delivery of future programs which aim to provide support to key sectors of the Canadian economy.

Lesson Learned 1: Government support for aerospace programming

Federal government support for the industrial sector continues to be important, including the aerospace, defence, space and security industries which are strategically significant for the country.  Government support facilitates Canada's competitiveness, as it helps accelerate R&D and innovation.  For the aerospace sector, it helps level the playing field with aerospace companies in other countries who receive more support than Canadian companies.

Lesson Learned 2: Role of government as a catalyst for collaboration

The government has a role as a catalyst for fostering collaboration and networking among companies and academia across Canada.  Collaboration helps develop the supply chain and accelerates innovation.  The design of programs to require participation from small and medium-sized enterprises and academic and research institutions is helpful for bringing industry and academia together to work on R&D projects.

Lesson Learned 3: Timely and predictable application process

To better respond to the needs of industry, consideration should be given to a timelier and more predictable application process.


1.0 Introduction

1.1 Report overview

This report presents the results of an evaluation of two separate Innovation, Science and Economic Development Canada (ISED) programs – Bombardier's C-Series program and the Technology Demonstration Program (TDP).  Together with the Strategic Aerospace and Defence Initiative (SADI), these programs comprised the aerospace programming of Innovation, Science and Economic Development Canada (ISED).

The C-Series program was established in 2008 and later extended to end in March 2020.  The TDP and SADI were consolidated into the Strategic Innovation Fund (SIF). The SIF was created in 2017 and consolidated existing departmental programming targeted at the aerospace and automotive sectors, and expanded support to all key industrial sectors.Footnote 1 Since SADI was recently evaluated in 2016-17, it is not included as part of this evaluation.

The purpose of this evaluation is to assess the relevance, performance, and design and delivery of the C-Series program and TDP. The report is organized into four sections:

  • Section 1 provides the context and profile of the C-Series program and TDP;
  • Section 2 presents the evaluation methodology and the challenges for the evaluation;
  • Section 3 presents the findings; and
  • Section 4 summarizes the conclusions and provides lessons learned.

At a glance:

  • The Bombardier C-Series program was established in 2008 to provide repayable contributions to encourage R&D in the development of technologies for the C-Series and other aircraft platforms.
    • $350 million announced in 2008
    • $120 million announced in 2017
  • The TDP was established in 2013 to support collaborative R&D projects that leverage private sector investment to accelerate innovation in the aerospace, defence, space and security industries.  As of March 31, 2018, $187.5 million in non-repayable contributions has been committed.

1.2 Context

Canada's aerospace, defence, space and security (A&D) industries are major contributors to Canada's economy. The aerospace and space systems industries alone contributed $24.5 billion to Canada's gross domestic product (GDP), with the non-aerospace defence industry contributing another $4.2 billion to the economy.Footnote 2

The strategic significance of these industries for the Canadian economy is amplified when considering the inter-industry linkages. For example, 14% of Canadian aerospace industry sales were related to defence and space systems in 2017.Footnote 3 Defence and space systems technologies are also frequently adopted by the civil aerospace sector.

Canada's support for innovation in these industries is long-standing, beginning in 1959 with the Defence Industry Productivity Program.  It was replaced by Technology Partnerships Canada in 1996, followed by SADI in 2007.

1.3 Objectives and description

By supporting R&D efforts in the A&D industries, the C-Series program and TDP helped to maintain and grow technological capabilities and further leverage private sector investment among Canadian A&D industries.

Bombardier C-Series Program

In 2008, the Government of Canada committed $350 million in repayable contributions through the C-Series program to help fund Bombardier's R&D activities associated with building the C-Series family of aircraftFootnote 4 and to develop new generic aerospace technologies. In February 2017, a further commitment of $120 million in repayable contributions for the C-Series was announced.

The objectives of the C-Series program are:

  • To encourage R&D that will result in the development of technologies for Bombardier's C-Series commercial aircraft; and
  • To encourage R&D that will result in the development of generic technologies including advanced materials, technologies and manufacturing processes which are applicable to a variety of aircraft platforms and other commercial applications.

In July 2018, Airbus acquired a 50.01% majority stake in the C-Series program, with Bombardier retaining a 34% stake and Investissement Québec holding the remaining 16%.

Technology Demonstration Program

The 2012 Emerson report entitled Volume 1: Beyond the Horizon: Canada's Interests and Future in Aerospace identified a need for aerospace programming to promote and accelerate collaborative R&D and networking among industry, universities and research centres, as well as the need for support at the technology demonstration stage (i.e. Technology Readiness Levels 4 to 6).  See Appendix A for more detail on Technology Readiness Levels (TRLs).
In response, the government created the TDP in 2013 with the following objectives:

  • To ensure a concentration of technology development in priority areas with significant potential for broad based, long-term and material economic benefits for Canada; and
  • To serve as the basis for the next-generation of manufacturing, technical capabilities and services in Canada.

The TDP funded two types of activities: large-scale technology demonstration projects; and, a national network to promote communication and collaboration among stakeholders in the aerospace industry and provide financial support to launch R&D projects in partnership with these stakeholders.

TDP – Technology Demonstration Projects

The TDP projects were large-scale collaborative technology demonstration projects that leveraged private sector investment to accelerate innovation. The projects provided non-repayable contributions to a maximum of $54 million per project.Footnote 5 As of March 31, 2018, a total of $157.5 million was committed for the following three projects:

  • MDA Systems-led Space Technology and Advanced Research (STAR) project ($54 million) – the project was initiated in 2016 with an aim to develop new technologies for a next generation of radar, optical and communication satellites;
  • Bombardier-led Horizon project ($54 million) – the project started in 2015 with the objective to develop new technologies related to advanced systems for a highly efficient next generation aircraft and the creation of a shared multi-system rig as an infrastructure hub for use during and after the project; and
  • Bell Helicopter Textron Canada Ltd. consortium project ($49.5 million) – this project was launched in 2018 to support the development of next-generation helicopter technologies.
TDP – Consortium for Aerospace Research and Innovation in Canada

In 2014, the TDP provided funding of $30 million for the Consortium for Aerospace Research and Innovation in Canada (CARIC), a Canada-wide not-for-profit network with the purpose to facilitate communication and collaboration among industry, academia, research institutions and associations and provide financial support to launch R&D projects in partnership with these stakeholders. These collaborative R&D projects were at low-to-mid-TRLs (i.e. TRLs 1 to 6).

Both the C-Series program and TDP have been managed and delivered by ISED through Innovation Canada.

1.4 Stakeholders

In addition to recipient firms, there were a number of key stakeholders in the C-Series program and TDP including:

Universities and Research Institutions

  • As part of the development of the C-Series aircraft and associated technologies, Bombardier collaborated with various post-secondary education and research institutions.
  • The TDP projects were large scale, complex, and required specialized equipment, facilities, and researchers (e.g. academics, graduate students, etc.).  As a result, they were almost always conducted through collaborative efforts which included universities and research institutions.

Supply Chain Companies

  • The development of A&D products, services and processes included a number of Canadian suppliers and other firms who were engaged in technical collaboration projects with the lead firm.

Networks and Associations

  • CARIC: fosters collaboration between the industry, academic and research institutions.
  • Aerospace Industries Association of Canada: understands, builds consensus and provides leadership on policy issues of interest to the industry; and, promotes Canadian aerospace companies in foreign markets.
  • Aéro Montréal: a strategic think tank which seeks to increase the cohesion and competitiveness of the Quebec aerospace cluster.
  • Consortium for Research and Innovation in Aerospace in Quebec (CRIAQ): promotes industry-led research collaboration projects involving universities and research centres in Quebec's aerospace cluster.
  • Canadian Association of Defence and Security Industries: the national industry voice for Canadian defence and security companies that produce Canadian-made goods, services and technologies.

Funding Partners

  • The federal government was one of several financial participants, each with their own objectives. Other than the federal government and recipients, funding partners included provincial/territorial levels of government, governments of other countries and the private sector (e.g. suppliers).

1.5 Logic model

The Logic Model in Figure 1 describes the design and delivery and intended results for aerospace programming (i.e. the Bombardier C-Series program and TDP).

Figure 1: Aerospace Programming Logic Model

Long description below.
Text version

Figure 1 depicts the logic model describing the design and delivery and intended results for aerospace programming (i.e. the Bombardier C-Series program and the Technology Demonstration Program (TDP)).  In this logic model, there are three sets of activities (one for the TDP, one for the C-Series program, and one for the Consortium for Aerospace Research and Innovation in Canada (CARIC) – a sub-component of the TDP) leading to their respective outputs.  These three sets of outputs lead to three common immediate outcomes, three common intermediate outcomes, and one common final outcome.  The activities, outputs and outcomes are all described as contributing to the achievement of technological, environmental and socio-economic benefits to Canada.

The three sets of activities are:

  1. TDP: Communicate program availability, eligibility, processes, expected outcomes and benefits; receive applications, assess eligibility of applicants and their R&D projects; conduct due diligence on recipients and their projects; formulate decision documents and seek appropriate approval; prepare Contribution Agreements; and monitor Agreements, R&D activities and program outcomes.
  2. C-Series: Internal program management activities: implementation of management processes, monitoring and reporting; post-work phase: monitoring of project benefits and repayments.
  3. CARIC: Development and management of the Contribution Agreement with CARIC.

The three sets of outputs are:

  1. TDP: Execution of Contribution Agreements including processing of claims and disbursements of non-repayable contributions, and amendments (if warranted) of Contribution Agreements.
  2. C-Series: Funding for R&D activities: processing of claims.
  3. CARIC: Disbursement of funds to CARIC.

The three immediate outcomes are:

  1. Increased technology development and demonstration of R&D activities.
  2. Increased networking opportunities and collaboration on R&D projects for Canadian industry, academic and research institutions.
  3. Creation and maintenance of highly qualified personnel and employment.

The three intermediate outcomes are:

  1. Strengthened technological and economic capacities.
  2. Commercialization of new and improved products, processes and service technologies.
  3. A skilled and experienced R&D aerospace workforce.

The final outcome is 'increased competitiveness of Canadian aerospace sector'.

2.0 Methodology

This section provides information on the evaluation context, approach, objectives and scope, issues and questions, data collection methods, and limitations.

2.1 Evaluation context

An evaluation of the C-Series program was conducted in 2013 which covered the period from September 2008 to March 2013.Footnote 6 The evaluation found that the program addressed a demonstrable need for aerospace funding due to the sector's financial risks and the importance of the economic benefits that the sector offered to Canadians.  As well, the evaluation noted that the program:

  • Contributed to the creation and maintenance of direct and indirect R&D jobs;
  • Led to increased R&D investment and R&D activities undertaken at Bombardier;
  • Contributed to the development of improved products, manufacturing processes and services related to the C-Series aircraft and for future aircraft platforms;
  • Enhanced collaborative activities for Bombardier; and
  • Demonstrated operational efficiency.

While the 2013 evaluation made no recommendations, the lessons learned highlighted the importance that future grant and contribution programs with intended results for collaboration have clearly defined expected collaborative activities.  As well, it noted that there should be processes implemented to monitor and document results on a regular basis and ensure that collaborative results are regularly reported.

This is the first evaluation of the TDP.

2.2 Evaluation approach

This evaluation was managed and conducted by ISED's Audit and Evaluation Branch and was calibrated to build on the findings of the previous C-Series evaluation, maximize the use of secondary research, and make targeted use of primary research to focus on the progress toward the immediate and intermediate outcomes identified in the logic model in Section 1.5.

2.3 Objective and scope

An evaluation of the C-Series program and TDP was required to be completed in 2018-19 to address the issues of relevance, performance and design and delivery in accordance with the Treasury Board Policy on Results and the Financial Administration Act. The evaluation covered the period from April 1, 2013 to March 31, 2018.

The evaluation examined the C-Series program and three of the four TDP-funded projects (including CARIC). The TDP Bell Helicopter Textron Canada Ltd. consortium project was not included as it was only launched in 2018.  Therefore, it is too early to assess its results.

2.4 Evaluation issues and questions

The evaluation addressed the following questions:

Relevance

  1. Is there a need for federal support for aerospace programming and, if so, to what extent did the C- Series program and TDP address a demonstrable need?

Performance

  1. To what extent have the C-Series program and TDP contributed to strengthened technological and financial capacities (including technology demonstration for the TDP)?
  2. To what extent has the TDP contributed to increased networking opportunities for Canadian industry, academic and research institutions?
  3. To what extent have the C-Series program and TDP contributed to increased collaboration on R&D projects between Canadian industry, academic and research institutions?
  4. To what extent have the C-Series program and TDP contributed to a skilled and experienced R&D aerospace workforce?
  5. To what extent have the C-Series program and TDP contributed to the commercialization of new and improved products, processes and service technologies?

Design and Delivery

  1. To what extent have the application and approval processes of the TDP been efficient and effective?

2.5 Data collection methods

Multiple lines of evidence were used to address the evaluation questions. The data collection methods included a document review, literature review, interviews, project document and data review, and case studies.

Document Review

A document review was conducted to gain a thorough understanding of the programs and to provide insights into relevance and performance.  The review included:

  • Program foundational documents (e.g. Terms and Conditions, Contribution Agreements);
  • Government priority-setting documents (e.g. Budgets, Speeches from the Throne); and
  • Other key program documents (e.g. industry reports, annual reports, previous audit and evaluation reports related to aerospace programming).

Literature Review

A literature review was conducted to complement the other lines of evidence on the relevance and performance of the C-Series program and TDP.  The review included:

  • Information on how other countries support their aerospace industry;
  • Stakeholders' viewpoints on the federal government's aerospace programming; and
  • Collaboration impacts of other programs with similar objectives to the TDP.

Interviews

The objective of the interviews was to gather in-depth information related to the relevance, performance, and design and delivery questions.  The interviews were semi-structured in nature to help collect qualitative information from a range of key stakeholders. Interviews were conducted either in-person or by telephone, as required.  Where appropriate, group interviews were conducted.

In total, the evaluation included 35 interviews from the following groups:

  • ISED management and staff (6);
  • Recipient firms – Bombardier Inc. and MDA Systems Ltd. (8);
  • CARIC representatives (4);
  • Tier 1 and small- and medium-sized (SME)Footnote 7 suppliers (9);
  • Universities and research institutes (5); and
  • Industry experts, networks and associations (3).

Project Document and Data Review

Project documents and data related to the C-Series program and TDP were reviewed to assess their results, including but not limited to the following:

  • Due diligence reports – provide information on what benefits could be expected from undertaking a project;
  • Progress reports – provide information on the progress towards completion of the stated goals for a project and milestones;
  • Annual benefits reports – allow for the tracking of project performance; and
  • Other documents (e.g. project proposals, site-visit reports, etc.).

Performance data templates filled out by recipients as well as ISED data on the TDP application and approval processes were also reviewed and analyzed.

Case Studies

Case studies were conducted for the C-Series program and TDP (including CARIC). These project case studies included in-depth document reviews, interviews and, in some cases, site visits in order to gather specific project information to address relevance, performance, and design and delivery issues.

2.6 Limitations

Program Maturity of TDP

Large-scale technology demonstration projects such as the ones funded by the TDP take many years to conclude and may take even longer before the full impacts of the projects can be realized. The two TDP projects started in 2015 and 2016 and are expected to be completed in the early 2020s. Therefore, not enough time has elapsed for these projects to have achieved some of their intended outcomes.

Performance Data

Certain commercially-sensitive information such as R&D investment as well as the identity and nature of collaborations were confidential. For this reason, the evaluation relied on self-reported information provided by companies to assess the achievement of these outcomes. Where possible, the evaluation addressed this limitation by supplementing the self-reported information with information gathered from interviews and case studies.

Attribution

Though leveraging of incremental private sector investment in R&D was a program requirement under the TDP, the presence of multiple funding partners (e.g. recipient firms, suppliers, other levels of government, foreign government, etc.) makes it difficult to isolate and measure the impact of the federal government's contribution.

3.0 Findings

3.1 Relevance

3.1.1 Is there a need for federal support for aerospace programming and, if so, to what extent did the C- Series program and TDP address a demonstrable need?

Key Finding: There is a need for the federal government to promote R&D and encourage private sector investment.

Federal support for aerospace programming aids the aerospace, defence, space and security industries – key economic drivers and major contributors of R&D and innovation for Canada's manufacturing industry.

Canada's aerospace industry, which includes space systems manufacturing, contributed $24.5 billion to Canada's GDP and close to 188,500 jobs in 2017.Footnote 8 Aerospace jobs are generally highly skilled and well paid (e.g. aerospace employees earn 29% more than the average manufacturing employee).Footnote 9 As well, over 70% of aerospace manufacturing revenues were derived from exports in 2017.Footnote 10

Furthermore, the Canadian non-aerospace defence industry and its value chain accounted for 38,300 jobs and close to $4.2 billion in GDP in 2016, based on an estimate from the State of Canada's Defence Industry report published by the Canadian Association of Defence and Security Industries and ISED. 

As the number one R&D investor across all manufacturing industries in 2017, aerospace manufacturing accounted for $1.7 billion, or close to 25%, of all R&D in the Canadian manufacturing industry.Footnote 11 R&D intensityFootnote 12 was over seven times that of the manufacturing average.Footnote 13 In parallel, R&D intensity was eight times higher in the space industry and four-and-a-half times higher in the defence industry than the Canadian manufacturing average in 2016. During the same year, R&D spending was $254 million for the space industry and close to $400 million for the defence industry.Footnote 14

Canada is a leading aerospace nation in the world. In 2017, it ranked first in the world's production of civil flight simulators and third in civil aircraft and engine production.Footnote 15 To maintain the global significance of a key Canadian industry, the evidence collected suggested the need for the federal government to promote R&D and encourage private sector investment through aerospace programming.

To stay globally competitive, the Canadian aerospace industry must remain at the forefront of innovation. Aerospace R&D projects are often expensive, decade-long investments, with up-front expenditures required for years prior to generating revenues. Because of this, and due to the risks and amount of funding involved, the private sector can be hesitant to invest in these R&D endeavours. Hence, the government has a valid role to play in encouraging and facilitating aerospace R&D.

Evidence also pointed to the need for government support for R&D to level the playing field for Canada with its international competitors. The aerospace industries in the U.S., U.K., France, Germany, and Brazil have been receiving government support since the early-to-mid 20th century. As well, competition has intensified among emerging aerospace nations such as China and Russia, who exert considerable amounts of public support to establish themselves on the global stage.Footnote 16 By international standards, Canada's support to promote R&D in the aerospace sector is not large.Footnote 17 For example, U.S. government funding of R&D for the aerospace and defence industry was $84.7 billion in 2016.Footnote 18

In addition, some interviewees reiterated the need identified in the 2012 Emerson reportFootnote 19 to have a national network such as CARIC to promote industrial-academic collaborative R&D and networking opportunities across Canada. The need for CARIC is also notable because aerospace activities have progressively become more prominent not only in Quebec and Ontario but also across Canada, particularly for aerospace maintenance, repair and overhaul (MRO) activities. Traditionally, aerospace manufacturing activities have been clustered in Montreal and Toronto. However, since 2004 direct GDP growth in MRO activities has outpaced the growth of manufacturing activities, with the majority of MRO activities in 2017 taking place in Western Canada (43%) and Atlantic Canada (13%). As a result, manufacturing took up 62% ($15.2 billion) of aerospace industry GDP in 2017 and MRO activities accounted for 38% ($9.3 billion).Footnote 20

Without government support, interviewees suggested that the aerospace industry might be smaller in size and R&D and innovation would be slower.  Further, supply chain development and industrial-academic knowledge transfer might be hampered since SMEs and universities, with minimal capital resources for R&D, would be less likely to participate on collaborative projects.

3.2 Performance

3.2.1 To what extent have the C-Series program and TDP contributed to strengthened technological and financial capacities (including technology demonstration for the TDP)?

Key Findings: The C-Series program and Technology Demonstration Program have contributed to strengthened technological capacity, particularly the C-Series program, which supported R&D which led to the development of a brand new C-Series aircraft incorporating many Canadian technical firsts.  The Technology Demonstration Program helped to accelerate technologies to higher Technology Readiness Levels.  Through additional private sector investment and the sharing of resources, the two programs have helped to strengthen financial capacity.

Strengthened Technological Capacity
C-Series Program

The C-Series program contributed to strengthened technological capacities.  The program supported R&D, which played a role in the development of a brand new commercial aircraft that is better performing (e.g. longer range, lighter, and safer), more environmentally friendly (e.g. quieter, lower carbon dioxide and nitrogen oxide emissions, and lower fuel burn), with more modern interiors (e.g. more storage, larger windows, and wider aisles), and less costly to operate (e.g. lower cost per seat).  To achieve these goals, Bombardier incorporated many technical firsts in Canada into the C-Series aircraft including the fly-by-wire system,Footnote 21 the extensive use of composite and lightweight metal alloys, advanced manufacturing processes, and Pratt & Whitney's Geared Turbofan engine.Footnote 22 Evidence indicates that Bombardier has successfully achieved these technological advances.

The C-Series program also helped to advance technological capacity in other areas.  In terms of knowledge diffusion, interviewees agreed that knowledge was shared between Bombardier and its suppliers.  The transfer and adoption of technologies to other applications has also been noted.  A review of documents and interview findings revealed that some of the technologies developed for the C-Series program have been adopted by other aerospace firms and the transportation industry (e.g. composite materials developed have been used in the automotive sector).

TDP

For the TDP projects, the program has played a role in strengthening technological capacity, most notably by accelerating the advancement of Technology Readiness Levels (TRLs) with the goal of advancing technologies to reach TRL 6. For the STAR project, all eleven project partners reported that, as a direct result of the TDP, technologies have advanced to later TRLs compared to when the project first started (for the Horizon project, two-thirds of project partners reported increased TRLs). Interviewees noted that without the TDP, progress would have been slower. Two of the SMEs involved in the STAR project reported technology development has moved beyond the technology demonstration stage (TRLs 4 to 6) to the pre-commercialization stage (TRLs 7 to 9). Moreover, all project partners of the two TDP projects stated that the projects have helped them to acquire new skills, competencies and capabilities related to technology innovation.

Knowledge diffusion for the TDP was particularly evident between industry and university partners and for the Horizon project, where regular meetings stimulate discussions among project partners to understand the needs of the recipient and increasing know-how.  As for the C-Series program, other firms have adopted some of the technologies developed under the TDP.

Lastly, though there is little indication that the funded projects resulted in the creation of intellectual property such as patents,Footnote 23 interviewees agreed that know-how was developed and shared among project partners.

Strengthened Financial Capacity

To assess the contribution of the C-Series program and TDP in strengthening the financial capacity of project partners, the evaluation examined the leveraging of project funding for the TDP projects from other sources, private sector investments received for the building of the C-Series family of aircraft, and the extent to which partners have access to each other's resources under both the TDP and C-Series program.

C-Series Program

For the building of the C-Series family of aircraft, the federal government invested $350 million in 2008-09 and $120 million in 2017.  Suppliers contributed $1.2 billion and Bombardier contributed $1.15 billion.

In terms of the sharing of resources, suppliers interviewed indicated that limited, if any, sharing of resources happened among project partners.  However, it should be noted that resource sharing was not expected to be a key component of the C-Series program.

TDP

The project data review showed that TDP funding contributed to the leveraging of funding from other sources. For the TDP Horizon project, every dollar invested by the federal government translated into an additional investment of $2.40 from other sources. For the STAR project, the leveraging ratio was 1.35, while the ratio was 2.13 for CARIC.

Interviewees noted that TDP funding provided an impetus for recipients to increase their own R&D investment as well. For the STAR project, one of the project partners interviewed said that TDP funding has led to significant internal R&D investment. Tier-1 Horizon suppliers interviewed indicated that the project has contributed to an increase in their companies' R&D budgets.

Sharing of resources (i.e. human resources, facilities and equipment) was a common practice for the TDP and CARIC projects.  For the Horizon project, some interviewees indicated that regular sharing of human resources and equipment has occurred among Bombardier, its suppliers and university partners.  Further, a project document review suggested that the multi-system rig will remain available post-project to support subsequent collaborative technology development efforts.  CARIC representatives noted that because they funded joint projects there was a sharing of resources – in particular, companies made use of labs and computer equipment at universities.

3.2.2 To what extent has the TDP contributed to increased networking opportunities for Canadian industry, academic and research institutions?

Key Findings: CARIC has helped increase networking opportunities and membership for industry, academic and research institutions across Canada.  Though CARIC's presence has continued to be dominant in Quebec, the share of events and membership have been increasing in other regions.

Through CARIC, the TDP has increased networking opportunities for Canadian industry, academic and research institutions across Canada. As of March 31, 2018, CARIC has helped organize 262 events – consisting of 142 workshops, 59 conferences, 16 forums, 21 international events, 20 regional events, and four other types of events in support of collaborative research and technology development activities. Although the majority of events continue to be conducted by Quebec and Ontario, these events have been increasingly conducted by other regions. For example, the share of events conducted by the Pacific, Central, and Atlantic regionsFootnote 24 have grown significantly, rising from 3.5% in 2014-15 to 34.0% in 2017-18. Collectively, these events have attracted well over 17,000 participants.

Interviews and a review of project documents suggested that CARIC's networking activities have been working well to bring industry and academia together to discuss collaborative research and technology development projects. It was noted that at CARIC events, industry and universities present their projects and use the opportunity to seek partnerships. As a result, participants are able to see what others are working on, which sometimes leads to a merging of projects which can then benefit from the synergy of expertise and coordination of efforts. CARIC events go beyond the presentation of academic research or discussion of collaborative projects.  They also serve as forums to discuss the needs and future of the aerospace sector.

As well, CARIC has continued to grow its membership across Canada.  CARIC's membership has more than doubled since inception, increasing from 73 members in 2014-15 to 152 members in 2017-18.  Most of CARIC's current members are either SMEs (59%) or academic and research institutions (30%).  The rest are associate membersFootnote 25 (e.g. Aéro Montreal, Canada Space Agency) and one is a project memberFootnote 26 (Air Canada).

Further, the changing composition of CARIC's membership, particularly by region, highlights CARIC's importance in helping to expand networking opportunities in Canada. For example, only 15% of CARIC's initial 73 members came from outside of Quebec.  By 2017-18, 37% of all CARIC members were from outside Quebec, with 18% from Ontario, 8% from the Pacific region, 7% from Central region and 4% from the Atlantic region (Figure 2).

Figure 2: Distribution of CARIC's Membership, by Region

Long description below.
Text version

Figure 2 compares the distribution of CARIC's membership (by region) for fiscal year 2014-15 and fiscal year 2017-18 using two separate pie charts. 

The first pie chart in Figure 2 shows that there were 73 CARIC members in 2014-15, of which:

  • 85.0% were from Quebec;
  • 11.0% were from Ontario;
  • 2.7% were from the Pacific region;
  • 1.4% were from the Central region;
  • 0.0% were from the Atlantic region; and
  • 0.0% were from the U.S.

The second pie chart in Figure 2 shows that there were 152 CARIC members in 2017-18, of which:

  • 62.6% were from Quebec;
  • 17.8% were from Ontario;
  • 7.9% were from the Pacific region;
  • 7.2% were from the Central region;
  • 4.0% were from the Atlantic region; and
  • 0.7% were from the U.S.

Although CARIC has helped to promote networking and collaborative research projects, there is some indication to suggest that more outreach activities are needed, as some interviewees indicated they were unaware of CARIC's existence or they perceived CARIC as a provincial and not a national network.  Further, interviewees pointed out that the CARIC requirement that each project have at least two industry and two academic/research partners can sometimes be difficult, since projects may not always have a second suitable industry or academic/research partner.

3.2.3 To what extent have the C-Series program and TDP contributed to increased collaboration on R&D projects between Canadian industry, academic and research institutions?

Key Findings: Collaboration on research and development projects between industry, academic and research institutions has been enhanced under the C-Series program and Technology Demonstration Program (TDP), but the extent has varied partly due to program objectives.  Further, CARIC has been successful in increasing collaborative research projects between industry, academic and research institutions.  Other collaborative benefits such as supply chain development and innovation acceleration have also been realized.

C-Series Program

There were no formal collaboration requirements for the C-Series program. However, as of October 31, 2018, Bombardier reported having 55 collaborative relationships with different institutions throughout various stages of the program, including 39 with industry (22 domestic, 17 international), 11 with the academia (seven domestic, four international), and five with research institutes (all domestic).

Suppliers interviewed for the C-Series program indicated that they have worked well with Bombardier.  They also noted that in general, they have worked with Bombardier as partners and not just as suppliers and that over time closer ties have been formed.  However, these interviewees suggested that compared to the more recent Bombardier-led TDP Horizon project, collaborative efforts have not been as significant.

TDP – STAR and Horizon Projects

The two TDP projects have exceeded program requirements in terms of the number of collaborators required. The TDP required at least one Canadian SME and one Canadian academic or research institute as project partners.  As of March 31, 2018, the Horizon project had 15 project partners (four large/Tier 1 companies, five SMEs, five academic institutions and one research institute), while the STAR project had 10 project partners (three large/Tier 1 companies, six SMEs, and one university).

Further, these projects have fulfilled the requirement to have a significant portion (target of 50%) of TDP funding distributed to project partners. As of March 31, 2018, TDP funding of $54 million for the Horizon project was shared equally between Bombardier and its partners, with SMEs and academic and research partners receiving 17% of the total and large/Tier 1 companies receiving 33%. Of the 108 Horizon project activities, 70 activities were led by project partners with the rest being led by Bombardier. For the STAR project, project partners led four of the six major work activities, with close to $40 million in TDP funding (74%) allocated to project partners (46% allocated to large/Tier 1 companies and 28% allocated to SMEs and academic and research partners).

The extent of collaboration has varied between the two projects.  Close collaboration among the recipient, large suppliers, SME suppliers, and academic and research partners have been demonstrated for the Horizon project. There is a clear governance structure to foster close working relationships among all project partners, from senior management down to the working level, with each partner having an equal say on project issues. Interviewees noted that presentations at meetings can be open and transparent because partners' work is complementary to each other. Interview findings also suggested extensive collaboration has existed in terms of resource and knowledge sharing, with partners working at each other's sites and helping to jointly develop students.

In contrast, interview findings suggested that the extent of collaboration has not been as noticeable for the STAR project. Project partners knew little about what other partners were working on and activities were performed and submitted to the recipient in the form of deliverables. Interviewees explained that some partners were competitors, thereby making it difficult to have fulsome collaboration.

TDP – CARIC Projects

For CARIC, there has been success in forging collaborative research projects between industry, academic and research institutions.  Each CARIC funded project must have at least four Canadian partners – two from industry and two from academic/research institution.  The number of project partners increased from 43 for 13 projects in 2014-15 to 121 for 46 projects in 2017-18,Footnote 27 with the majority of them being from Quebec and Ontario.  Interviewees noted that it would be helpful if there was increased flexibility in terms of which firms and institutions could be considered as project partners. For example, the Composites Innovation Centre Manitoba Inc.Footnote 28 is not considered to be an eligible partner, which is restrictive since there are few academic/research institutions available in the Manitoba/Saskatchewan region with an applied research focus on projects at the mid-TRL level.Footnote 29

Other Collaborative Benefits

Interview and case study findings suggested additional collaborative benefits as a result of the C-Series program and TDP. First, there is evidence to indicate that the programs have supported supply chain development. Some SMEs interviewed noted that the expertise gained, increased visibility, better understanding of recipients' needs, and trust developed during the projects have led them to have ongoing collaborative relationships with other partners and, in some cases, have brought about new business opportunities. University partners interviewed also emphasized the value of collaboration in developing applied aerospace expertise for professors and students.  Secondly, interviewees noted that the opportunity to work with partners who possess specialized expertise has brought new ideas and helped accelerated innovation.

3.2.4 To what extent have the C-Series program and TDP contributed to a skilled and experienced R&D aerospace workforce?

Key Findings: The C-Series program and Technology Demonstration Program have contributed to creating and maintaining research and development employment, augmenting the talent pool, and enabling the emergence of aerospace expertise.

C-Series Program

The C-Series program has contributed to creating and maintaining R&D aerospace employment. Data provided by Bombardier indicates that the R&D of the C-Series aircraft has led to a sustained increase in employment. From 2008 to 2017, the number of knowledge-based jobs created and maintained at Bombardier increased from 476 to 1,995, with a peak of 2,278 in 2013.  As well, C-Series suppliers, notably the SMEs interviewed, said that the program has led to the maintenance and creation of jobs in their companies, as was also reported in the 2013 C-Series evaluation.

Bombardier has estimated that the number of students working on the C-Series program rose from 12 in 2015 to 74 in 2018.

TDP

The TDP has created significant employment opportunities through both the STAR and Horizon projects. As of March 31, 2018, close to 940 R&D employees have worked on the two projects. MDA Systems Ltd. reported that 495 R&D employees have worked on the STAR project, with 230 of them being MDA employees, 100 from Tier 1 suppliers, 149 from SMEs, and 16 from an academic or research partner. Similarly, 438 R&D employees have worked on the Horizon project, with 290 from Bombardier, 92 from Tier 1 suppliers, 51 from SMEs, and five from an academic or research partner (Figure 3).

Figure 3: TDP's Contribution to R&D Jobs Supported

Long description below.
Text version

Figure 3 compares TDP's contribution to R&D employment (by collaboration partner) for Project STAR and Project Horizon using two separate pie charts. 

The first pie chart in Figure 3 shows that, for Project STAR, there were 495 jobs supported as of March 31, 2018, of which:

  • 46.5% were supported by the lead recipient (MDA Systems Ltd.);
  • 30.1% were supported by small- and medium-sized enterprise (SMEs);
  • 20.2% were supported by Tier 1 suppliers; and
  • 3.2% were supported by an academic or research partner.

The second pie chart in Figure 3 shows that, for Project Horizon, there were 438 jobs supported as of March 31, 2018, of which:

  • 66.2% were supported by the lead recipient (Bombardier);
  • 11.6% were supported by small- and medium-sized enterprise (SMEs);
  • 21.0% were supported by Tier 1 suppliers; and
  • 1.1% were supported by an academic or research partner.

In addition, the TDP has contributed to increasing the talent pool by providing training opportunities for students.  As of March 31, 2018, a total of 234 students have worked on the two TDP projects (186 on Horizon and 48 on STAR), with about one-third of them at the graduate or post-doctorate level (Figure 4).

Figure 4: TDP's Contribution to Training Opportunities for Students

Long description below.
Text version

Figure 4 compares TDP's contribution to training opportunities for students (by education level) for Project STAR and Project Horizon using two separate pie charts. 

The first pie chart in Figure 4 shows that, for Project STAR, there were 48 training opportunities provided to students as of March 18, 2018, of which:

  • 60.4% were provided to undergraduate students;
  • 31.3% were provided to PhD students;
  • 6.3% were provided to post-doc students; and
  • 2.1% were provided to Master's students.

The second pie chart in Figure 4 shows that, for Project Horizon, there were 186 training opportunities provided to students as of March 18, 2018, of which:

  • 68.8% were provided to undergraduate students;
  • 9.1% were provided to PhD students;
  • 3.2% were provided to post-doc students; and
  • 18.8% were provided to Master's students.

CARIC has also played a role in facilitating students' involvement in its funded projects. The number of students who were involved with or trained on one of CARIC's research and technology development projects more than tripled, from 43 students in 2015-16 to 136 students in 2017-18 with more than 90% of them at the graduate or post-doctorate level.

Interviewees noted the benefits of hiring interns and co-op students at the graduate and post-doctorate levels.  While students benefit from acquiring industrial skills, companies benefit from the new ideas and research knowledge (e.g. digital technologies) that students bring.  This cycle continuously augments the knowledge base of the talent pool and the aerospace workforce, resulting in a more experienced and skilled R&D aerospace workforce.

3.2.5 To what extent have the C-Series program and TDP contributed to the commercialization of new and improved products, processes and service technologies?

Key Findings: The C-Series program has contributed to R&D that played a part in the eventual commercialization of the C-Series aircraft.  For the Technology Demonstration Program, commercialization is not a key component of the program and it is still too early to draw conclusions.

C-Series Program

The C-Series aircraft has successfully commercialized with the first seven aircraft delivered in 2016.  As well, interviewees commented that the C-Series could be considered a commercial success, based on continued market interest. By May 2018, there were 402 firm orders.Footnote 30 Orders have been received from numerous customers including Delta Air Lines (75 firm orders and an option to buy an additional 50 aircraft), Air Canada (45 and 30), Air Baltic (30 and 15), and JetBlue Airways Corp. (60 and 60).Footnote 31 Interviewees commented that feedback from customers has been positive with the company reporting the same. Airbus expects that the C-Series aircraft can secure 3,000 unit sales over the next 20 years.Footnote 32

Some C-Series suppliers indicated that the knowledge acquired from their involvement with the C-Series program allowed them to develop their own competitive niche (e.g. proprietary product). This, in turn, has helped them to further advance their technologies to be used in other aircraft for new customers.

TDP

For the TDP, commercialization is not a key component of the program since it was designed to fund projects at the earlier TRL stages (i.e. up to TRL 6).  Furthermore, because the two TDP projects were only recently launched, commercialization has yet to occur.  Having said that, interviewees said that the TDP projects have contributed to improving and/or creating new technologies and some suppliers interviewed mentioned that the TDP projects have contributed to bringing their products to market-ready status and, in some cases, to commercialization.

3.3 Design and delivery

3.3.1 To what extent have the application and approval processes of the TDP been efficient and effective?

Key Findings: Some phases of the Technology Demonstration Program application and approval processes took longer than anticipated, although this was mainly due to circumstances outside the control of the program.

The TDP Program GuideFootnote 33 projected that the application and approval processes would take between 12-17 months.  However, data provided by the program revealed that the processes took longer during certain phases.  Interviewees also said that for Round 1, additional delays of at least six months were encountered because of the requirement for policy and program approvals.

Table 1: Duration of TDP Application and Approval Processes (in Months)
- Projected Timeline Round 1 Round 2 Round 3
Statement of Interest Phase 4 months 3.6 4.1 4.2
Project Proposal Phase 3 months 3.0 2.8 3.2
Due Diligence Phase 2 to 3 months 2.0 2.8 8.7
Approval Phase 2 to 6 months 15.8 0.6 0.5
Contribution Agreement Phase 1 month 13.8 5.3 12.8
Total Duration 12 to 17 38.2 15.6 29.4
Text version

Table 1 compares the duration of the TDP application and approval processes (in months) for the five different phases (statement of interest phase, project proposal phase, due diligence phase, approval phase, Contribution Agreement phase) for the first three rounds of the call for proposals.

The actual duration of each phase (by round) and the total duration for each of the three rounds, including the projected timeline for each phase, is provided in Table 1.

  • Statement of Interest Phase:
    • Projected timeline of 4 months.
    • Actual durations of 3.6, 4.1 and 4.2 months for rounds 1, 2 and 3 respectively.
  • Project Proposal Phase:
    • Projected timeline of 3 months.
    • Actual durations of 3.0, 2.8 and 3.2 months for rounds 1, 2 and 3 respectively.
  • Due Diligence Phase:
    • Projected timeline of 2 to 3 months.
    • Actual durations of 2.0, 2.8 and 8.7 months for rounds 1, 2 and 3 respectively.
  • Approval Phase:
    • Projected timeline of 2 to 6 months.
    • Actual durations of 15.8, 0.6 and 0.5 months for rounds 1, 2 and 3 respectively.
  • Contribution Agreement Phase:
    • Projected timeline of 1 month.
    • Actual durations of 13.8, 5.3 and 12.8 months for rounds 1, 2 and 3 respectively.
  • Total Duration:
    • Projected timeline of 12 to 17 months.
    • Actual durations of 38.2, 15.6 and 29.4 months for rounds 1, 2 and 3 respectively.

Interviewees said that a long and unpredictable waiting period from the time of application (project proposal phase) to time of awarded contract (contribution agreement phase) resulted in recipients losing project partners, especially SMEs and universities. For example, it was difficult for SMEs to commit capital without having firm timelines of when TDP funding would be awarded.

Although not related to timelines, following lessons learned from the first two rounds of TDP funding, the program modified the application process for the third round, requiring that all project partners sign a contribution agreement with ISED to allow for better communication and transparency related to roles and responsibilities on R&D activities. Previously, only the recipient signed an agreement with ISED.

4.0 Conclusions and lessons learned

4.1 Conclusions

Relevance

  • There is a need for the federal government to promote R&D and encourage private sector investment.

Performance

  • The C-Series program and TDP have contributed to strengthened technological capacity, particularly the C-Series program, which supported R&D that led to the eventual development of a brand new C-Series aircraft incorporating many Canadian technical firsts.  The TDP helped to accelerate technologies to higher Technology Readiness Levels.
  • Through additional private sector investment and the sharing of resources, the two programs have helped to strengthen financial capacity.
  • CARIC has helped increase networking opportunities and membership for industry, academic and research institutions across Canada.  Though CARIC's presence has continued to be dominant in Quebec, the share of events and membership have been increasing in other regions.
  • Collaboration on research and development projects between industry, academic and research institutions has been enhanced under the C-Series program and TDP, but the extent has varied partly due to program objectives.  Further, CARIC has been successful in increasing collaborative research projects between industry, academic and research institutions.  Other collaborative benefits such as supply chain development and innovation acceleration have also been realized.
  • The C-Series program and TDP have contributed to creating and maintaining R&D employment, augmenting the talent pool, and enabling the emergence of aerospace expertise.
  • The C-Series program has contributed to R&D that played a part in the eventual commercialization of the C-Series aircraft.  For the TDP, commercialization is not a key component of the program and it is still too early to draw conclusions.

Design and Delivery

  • Some phases of the TDP application and approval processes took longer than anticipated.

4.2 Lessons learned

Given that the TDP was consolidated into the Strategic Innovation Fund and that the C-Series program was a specific-purpose program, the following lessons learned were developed to inform the design and delivery of future programs which aim to provide support to key sectors of the Canadian economy.

Lesson Learned 1: Government support for aerospace programming

Federal government support for the industrial sector continues to be important, including the aerospace, defence, space and security industries which are strategically significant for the country.  Government support facilitates Canada's competitiveness, as it helps accelerate the speed of R&D and innovation.  For the aerospace sector, it helps level the playing field with aerospace companies in other countries who receive more support than Canadian companies.

Lesson Learned 2: Role of government as a catalyst for collaboration

The government has a role as a catalyst for fostering collaboration and networking among companies and academia across Canada.  Collaboration helps develop the supply chain and accelerates innovation.  The design of programs to require participation from small and medium-sized enterprises and academic and research institutions is helpful for bringing industry and academia together to work on R&D projects.

Lesson Learned 3: Timely and predictable application process

To better respond to the needs of industry, consideration should be given to a timelier and more predictable application process.