Evaluation of the Program for Strategic Industrial Projects (PSIP)—Final Report

May 2009

Tabled and approved at Departmental Evaluation Committee on June 26, 2009

Table of Contents

Executive Summary

1. Introduction

2. Background Information - PSIP

2.1 Canada's Automotive Industry

2.2 Focus on three funded PSIP projects

3. Goals of the Evaluation

4. Evaluation Methodology

4.1 Limitations of the PSIP methodology

5. Findings

5.1 Relevance

5.2 Program Design and Implementation

5.3 Performance Measurement

5.4 Success/Progress

5.5 Cost-Effectiveness of Program Delivery

6. Conclusions

Executive Summary

The Program for Strategic Industrial Projects (PSIP) was created in early October 2005 to strengthen the Canadian auto industry through support for the implementation of flexible manufacturing capacity at Canadian assembly plants. PSIP was created because the then existing federal programs, including Technology Partnership Canada (TPC)¹, were limited in their ability to support major vehicle assembly projects. However, like TPC, PSIP aimed to support the development, adaptation and adoption of new technologies with benefits to the Canadian economy, as well to promote the long-term stability of the auto assembly sector and, by extension, the Canadian auto parts sector.

This evaluation focuses on the three projects funded through the PSIP: those with Ford Motor Company of Canada Ltd. (in Oakville, Ontario), General Motors of Canada Limited (a project mainly in Oshawa, Ontario), and Toyota Motor Manufacturing Canada Inc. (a project located in Woodstock, Ontario). The federal government's investment in these three projects totals $355 million. The evaluation was intended to examine five main themes: relevance; program design and implementation; performance measurement; success and progress; and cost-effectiveness of program delivery. Field work for the evaluation was conducted between June 2008 and February 2009 during a period of great turmoil in the automotive industry and a downturn in the overall economy.

Lines of evidence included interviews with Industry Canada officials, Government of Ontario officials, company representatives, and municipal officials and other community representatives (such as leaders of chambers of commerce), and union representatives, which provided the evaluators with useful and in-depth insights into the successes, issues and lessons learned of and from PSIP. As well, an extensive document review was undertaken, and a visit was made to one of the companies.

Relevance

The PSIP did not duplicate or overlap with other federal or provincial programs. Most importantly, PSIP was complementary to its main counterpart, the Ontario Automotive Investment Strategy (OAIS) Program. As well, the PSIP was aligned with Federal Government and Industry Canada objectives in relation to the economy, innovation, technology, and sustainability.

Design and Implementation

The negotiation and monitoring phases of PSIP were managed by different individuals and sectors of the Department, the Automotive and Transportation Industries Branch (ATIB) of Industry Sector, and the Industrial Technologies Office (ITO). Interviews and program documents suggest that some information was lost in the "hand-off" at each stage, or the quality of information declined. There was also some lack of clarity in responsibilities between the ATIB and the ITO and sometimes inconsistent participation of officials. More recently efforts have been made to maintain a more integrated approach to discussions with the industry, which has resolved some of the issues noted above. While the negotiations were lengthy and generated some frustration on the part of company representatives and Industry Canada (IC), the three projects were successfully negotiated. Overall, the process for delivering PSIP funds was largely effective (i.e. robust projects, timely approval of projects, etc.).

Performance Measurement

The evaluation found that no performance measurement system was in place in the usual sense for the program as a whole. While data was regularly collected, this information was not "rolled up" into an overall program assessment. However, a high level of reporting was obtained from two of the companies and this was deemed to be positive. For the third project, reporting had just begun at the time of the evaluation and it was thus not possible to review and assess the level of reporting. Companies' difficulties with the reporting requirements, for the most part have been worked out. That said, the creation of standard reporting forms would facilitate the collection of performance information and ensure consistency across projects.

Success and Progress

PSIP is seen by both Industry Canada and Ontario government staff to have reached its intended beneficiaries, with positive results, notwithstanding the general economic uncertainty currently facing the automotive industry in North America. Company investments are regarded as highly successful, with a key conclusion being that the two older existing assembly plants have been modernized to a very high degree and an additional world-class assembly plant has been built. As well, numerous spin-offs were noted in the areas of parts suppliers, technology and training. Positive economic and community impacts were attested to by all community, business and union representatives interviewed. Sustainability of these results, however, remains an open question, given the current state of the industry and the economy generally.

Cost-Effectiveness of Program Delivery

Qualitative data gathered from interviews suggested that program resources were satisfactory for delivery of the program, and that program delivery by Industry Canada was cost effective e.g. comparable to other Industry Canada programs; however, only limited quantitative data could be obtained. As a result, cost effectiveness of delivery could not be addressed definitively.

As a result of the fieldwork conducted for this evaluation, and subsequent discussions with the Treasury Board Secretariat's Centre of Excellence for Evaluation, a decision was made to forego the conduct of another evaluation of this program. This decision was based on the following considerations:

• The PSIP is a legacy program and will not fund any additional projects before it sunsets on March 31, 2011. A successor program for the automotive industry, the Automotive Innovation Fund, is already in place.
• This evaluation includes a discussion of program results, and fulfills the requirements of the Evaluation Policy to assess program performance. Given the current challenges being faced by the automotive industry and the large scale changes that are likely to occur, it would be difficult, in any future evaluation, to attribute subsequent outcomes to the program.
• The program will continue to be monitored through to the repayment phase of the projects. Any significant issues arising at the repayment phase could be covered through the audit process.

[1] Technology Partnerships Canada (TPC) closed in December 2006. The Industrial Technologies Office (ITO) was created in April 2007 to manage Canadian government investments in R&D. It also manages legacy projects previously contracted through TPC. Many corporate interviewees continued to refer to TPC as the agency to which they sent their reports.

Introduction

This evaluation was managed by the Audit and Evaluation Branch (AEB) of Industry Canada (IC). The evaluation was conducted by SPR Associates Inc.  This firm was chosen in part because of its direct access to academic expertise in the automotive industry¹. In addition, one AEB resource was imbedded in the consulting team throughout the conduct of the evaluation. The evaluation was done in consultation with a Steering Committee established for the purpose of the evaluation.

The remainder of this report is organized into four sections: Section 2 provides background information on the program; Section 3 presents goals of the evaluation; Section 4 presents the methodology followed in conducting the evaluation work; Section 5 provides findings, organized in relation to the main themes of program relevance, design and implementation, performance measurement, success/progress, and, to the extent possible, cost-effectiveness; and Section 6 provides a summary of the main conclusions.

[1] SPR Associates team included: Dr. Ted Harvey (SPR), Dr. Maureen Molot (Carleton University), Dr. John Holmes (Queen's University), Dr. Johannes Van Biesebroeck and Dr. David Wolfe (University of Toronto).

Background Information –PSIP^{Footnote 1}

The Program for Strategic Industrial Projects (PSIP) was established in October 2005 and is scheduled to sunset on March 31, 2011.

PSIP was mandated to provide a framework within which strategic auto investment projects could be administered by IC:

Three auto companies were recipients of this funding (Ford Motor Company of Canada, Ltd, General Motors of Canada, Limited, and Toyota Motor Manufacturing Canada, Inc), in support of significant investments in large, complex projects that would contribute to long-term federal government priorities beyond the lifespan of the program. One other firm was also negotiating an agreement at the time of the evaluation. Projects administered under the PSIP were individually funded, in whole or in part, from the fiscal framework. PSIP was modeled on the Technology Partnerships Canada (TPC) program processes established for project management and monitoring. A total of $355 million has been allocated by the federal government to these three projects.

PSIP Eligibility: Activities eligible under the PSIP Terms and Conditions are those involving three main components: (1) industrial research focused on gaining new industry insights that will prove to be useful in future industry development more generally, and improving current industry standards and practices; (2) pre-competitive development, which explains why PSIP projects are extensively involved in production activities which will result in competitive new technologies on the open market; and (3) technology adaptation/adoption which requires that PSIP projects not only develop new technologies, practices, etc. but also implement these technologies, thus contributing to innovation in the sector in general. Under these rules of eligibility, PSIP was a special purpose program developed for the automotive sector.

Project Characteristics: Projects selected for PSIP support were to (1) contribute to the strategic objectives of the government through the provision of technological and net economic benefits to Canada; (2) include a significant element of technology development demonstration, adaptation or adoption; (3) commit to larger strategic endeavours or investments by the firm; and (4) seek federal government investment.^{Footnote 2}

Accountability: Once PSIP projects were negotiated by IC officials in the Automotive and Transportation Industries Branch (ATIB) and approved by Treasury Board, they were administered originally by TPC and then by the ITO.^{Footnote 3} Annual progress reports are submitted to ITO staff, with site visits scheduled at least once per year. A Result-Based Management and Accountability Framework (RMAF) and Risk-Based Audit Framework (RBAF)^{Footnote 4} exist for PSIP to aid in ensuring accountability.

Results: PSIP projects were intended to achieve specific results. Depending on their nature, projects were to support the development of new technologies with benefits to the Canadian economy. They were to promote long-term growth and capabilities in industries and corporations stimulating further corporate sustainability (i.e., job growth). Projects were also intended to stimulate the growth of related industry firms, for example parts suppliers. Further, PSIP projects were to be of value to Canadians in general (e.g. fiscal growth, technological advancement).

Funding

Table 1 presents the payments made to the three companies from 2005-06 to 2007-08.^{Footnote 5}

Source : Public Accounts of Canada

Canada's Automotive Industry

The automotive industry has long been one of Canada's most important manufacturing industries, accounting for close to 12% of the country's manufactured GDP in 2007.^{Footnote 6} The industry accounted for some 24% of Canada's manufacturing trade; close to 85% of all vehicles assembled in Canada and approximately two-third of Canadian-made parts are exported. Ontario which until October 2008 was the leading auto producing jurisdiction in North America^{Footnote 7} is home to all of Canada's vehicle assemblers and the majority of the country's parts producers. The auto industry generates a large number of spin-off jobs and is a big consumer of steel, rubber and processed aluminium as well as other commodities.

The Canadian auto industry is part of a globalized and highly competitive international industry in which countries vie for investment in new assembly plant capacity and upgrades to existing capacity. Governments recognize that assembly investment is frequently the catalyst for the establishment of new components production. At the beginning of the new millennium, key auto industry issues for Canada included not only the augmentation of assembly capacity but also investment in innovation, engineering capacity, and environmentally friendly research and development. It is in this context that initiatives to enhance industry performance must be considered. As this assessment of PSIP was being done, Canada, the United States and all other auto producing economies fell into recession. All assemblers are experiencing falling sales, as a result of which they have decreased production. The situations of GM and Chrysler are such that the Canadian and U.S. governments offered each of these companies financial assistance.

2.2 Focus on three funded PSIP projects

Information presented here on the PSIP projects is from public sources (e.g., corporate websites, IC press releases, and newspaper articles) at the time these projects were announced.

Beacon Project (General Motor of Canada Limited (GMCL))

On March 2, 2005, GMCL announced a $2.5 billion reinvestment in its Canadian operations representing the largest and most comprehensive automotive investment in Canadian history. Following agreements with the Ontario and Federal Governments, GMCL noted that the "Beacon Project" investments would strengthen automotive engineering, R&D and manufacturing capabilities in Canada. The Government of Canada invested $200 million while the Ontario Province committed $235 million to the project. The Beacon Project was designed to expand Canadian production capacity, improve production technology, develop "best in class" environmental technology, and introduce flexible manufacturing capability. The Beacon Project would also invest in the GMCL workforce, adding over 500 jobs in total at GM's Canadian Engineering Centre and CAMI Automotive. The project includes: a New Vehicle Programs; Enhanced Vehicle Engineering Activities; Flex Manufacturing and Environmental Leadership; Employment and Skills Training; and a Canadian Automotive Innovation Network.^{Footnote 8}.

Project Centennial (Ford Motor Company of Canada Ltd. (FMCC))

Ford's Project Centennial, announced in October 29, 2004, was designed to introduce a state-of-the-art, flexible manufacturing system at the company's Oakville plant, making it Ford's first flexible assembly plant in Canada capable of producing more new vehicles, more quickly and more efficiently. Ford was to invest a minimum of $1billion in the Oakville Assembly Complex (OAC) for two new vehicle programs, and a series of engineering/research and development commitments. In addition, it was to provide training in new production technologies for employees, create opportunities for suppliers, and invest in research and development, particularly in the area of environmental technologies. It was expected that Project Centennial would maintain some 3,900 direct jobs^{Footnote 9} as well as a significant number of other jobs in the area that depend on the Ford assembly plant.

The Government of Canada contribution of $100 million supports areas of the project related to innovative process technologies, environmental research and advanced engineering. The government also committed to work with Ford on its Canadian supplier development initiatives. The Government of Ontario also committed $100 million to the project, focusing mostly on workforce training.

Project Jewel (Toyota Motor Manufacturing Canada Inc. (TMMC))

The Government of Canada invested $55 million in Toyota's Jewel Project as part of a $1.1 billion project to construct a state-of-the-art, flexible assembly plant that uses the latest Toyota manufacturing and environmental systems. In addition to increased Canadian capacity, the long-term benefits of the new Woodstock assembly plant were expected to be significant. The additional production was also expected to create significant new opportunities for local parts sourcing, investment and employment by Toyota's suppliers in Canada, as well as substantial spin-off benefits in the local community. It was anticipated that several suppliers would build new plants or expand existing plants in the Woodstock area to supply the new assembly plant.

Ground was broken at the new Woodstock facility on October 11, 2005 and on December 4, 2008 Toyota officially celebrated the Grand Opening of the plant. This new facility, situated on almost 1,000 acres, employs 1,200 people and produces Toyota's compact SUV, the RAV4. It will produce 75,000 vehicles annually with one shift and has the capacity to produce up to 150,000 vehicles annually. The facility is the first greenfield assembly site in Canada since the mid-1980s^{Footnote 10}.

Footnotes

1. ^{1 back to footnote reference 1} From Terms of Reference for the Mid-Term Evaluation of the Program for Strategic Industrial Projects, Industry Canada, 2008.
2. ^{2 back to footnote reference 2} Adapted from PSIP Terms and Conditions.
3. ^{3 back to footnote reference 3} Technology Partnerships Canada (TPC) closed in December 2006. The Industrial Technologies Office (ITO) was created in April 2007 to manage Canadian government investments in R&D. It also manages legacy projects previously contracted through TPC. Many corporate interviewees continued to refer to TPC as the agency to which they sent their reports.
4. ^{4 back to footnote reference 4} November 16, 2005.
5. ^{5 back to footnote reference 5} Information for the year 2008-2009 had not yet been released at the time of the evaluation.
6. ^{6 back to footnote reference 6} From the Canadian Automotive Partnership Council (CAPC) website.
7. ^{7 back to footnote reference 7} Mexico surpassed Ontario as North America's leading automotive producing area in October 2008 when it produced 17,000 more vehicles than did Ontario.
8. ^{8 back to footnote reference 8} Michael Grimaldi, president of GMCL said when the Beacon Project was announced, "This "2.5 billion investment demonstrates GM's strong commitment to Canada's automobile industry. GM's investment is looking to the present and the future as we focus on great new cars and trucks, flexible manufacturing, the environment, and R&D. The partnerships comprehended in the Beacon Project are critical to successfully compete in an industry going through a challenging global transformation with production over-capacity."
9. ^{9 back to footnote reference 9} Backgrounder, Government of Canada Support of Ford's Project Centennial, October 29, 2004, Industry Canada website
10. ^{10 back to footnote reference 10} A new manufacturing plant built on a previously undeveloped site (in contrast to the upgrading or refurbishment of an existing plant).

3. Goals of the Evaluation

This evaluation is being conducted to provide program management with insight on program delivery and early success. This evaluation fulfills the requirement outlined in the program's RMAF. The issues or themes addressed by the evaluation include: program relevance, design and implementation, performance measurement, early success and, to the extent possible, cost-effectiveness. The following questions were included in the evaluation:

Relevance:

1. Does the PSIP overlap, duplicate or complement other programs designed to address the needs of the automotive industry?
2. Is PSIP aligned with government of Canada and Industry Canada objectives?

   Program design and implementation:

3. Are the roles and responsibilities of the Industry Sector and ITO staff involved in the negotiation, and monitoring processes efficient?
4. Are the processes in place effective (i.e. approval process, negotiation process, monitoring process, etc.)?

   Performance measurement:

5. How effective is the ongoing performance measurement system of the PSIP? What information will be useful for future reporting? (This issue should also address the timing of a final evaluation of the program.)

   Success/Progress:

6. Is the PSIP reaching its intended beneficiaries as planned?
7. Recognizing that results may take years to occur given the nature of the program, what evidence is there that results have or are likely to occur?

   Cost-effectiveness of program delivery:

8. Does/Did the program have sufficient resources to ensure adequate implementation?
9. Is the program being delivered in a cost-effectiveness manner?

As noted above, the evaluation included a question related to the timing of a subsequent evaluation, in response to comments contained in the program RMAF. As a result of the fieldwork conducted for this evaluation, and subsequent discussions with the Treasury Board Secretariat's Centre of Excellence for Evaluation, a decision was made to forego the conduct of another evaluation of this program. This decision was based on the following considerations:

• The PSIP is a legacy program and will not fund any additional projects before it sunsets on March 31, 2011. A successor program for the automotive industry, the Automotive Innovation Fund, is already in place.
• This evaluation includes a discussion of program results, and fulfills the requirements of the Evaluation Policy to assess program performance. Given the current challenges being faced by the automotive industry and the large scale changes that are likely to occur, it would be difficult, in any future evaluation, to attribute subsequent outcomes to the program.
• The program will continue to be monitored through to the repayment phase of the projects. Any significant issues arising at the repayment phase could be covered through the audit process.

4. Evaluation Methodology

Planning for this evaluation began in April 2008 and the data collection and analysis took place between June 2008 and February 2009.

Initial steps and planning

Initial communications were a key step in the evaluation, since these prepared interviewees for the study, and outlined the type of information needed. Initial letters were sent to IC staff in August and initial interviews were conducted with ITO and ATIB officers in September, which aided evaluators in gaining a broader understanding of the projects (including project completion, implementation, issues, etc.) and identified key individuals within PSIP and the companies to interview. Initial contact also took place with Ford, GM and Toyota representatives to schedule a one-day site visit for each. Company representatives were also asked to identify municipal, business and other important individuals in their respective and surrounding communities who would be able to discuss the economic and other impacts that PSIP has had on their communities.

Three main lines of evidence were used in this evaluation: interviews, site visits, and a document review.

Interview with key stakeholders

As indicated in Table 2, a total of 38 .

Table 2: Number of interviews

Site Visits

Interviews with companies were to include site visits for observation. The evaluators were given a comprehensive tour of Ford facilities and Ford's new flex body shop as well as the Fumes to Fuel large scale fuel cell project. The evaluators were not provided with access to the GM facility but received detailed information including photographs and other information on the flex body operation and the paint shop. A site visit to the Toyota plant could not be arranged within the timelines of this evaluation; however, three separate sets of interviews were conducted with Toyota officials via teleconference.

Document Review

A comprehensive document review was conducted as part of the evaluation. This process included a review of the foundation documents of the PSIP program, correspondence, contributions agreements and company records, projects files for the three funded projects as well as for the firm which, at the time of the evaluation, was negotiating an agreement, and documents outlining the roles and responsibilities of various parties, anticipated short-, medium- and long-term impacts of the projects and the results achieved to date.

4.1 Limitations of the PSIP methodology

The PSIP evaluation faced a number of obstacles and limitations.

Temporal factors: Most important of these limitations is the fact that PSIP investments have long time horizons, in terms of their financial agreements (agreements have repayment terms of 10+ years) as well as impacts. The nature of the projects therefore limits the assessment to shorter-term impacts.

As well, as the evaluation unfolded, a substantial down-turn in the North American automotive industry occurred, making assessments of impacts more difficult (although positive impacts were noted in the short term). The supplier base for GM and Ford is in a state of economic turmoil which made it difficult to document with any degree of confidence the short term economic impact on suppliers of the Beacon and Centennial projects. It is easier to measure the impact of a new assembly plant, which not only adds new direct employment but also attracts supplier firms to the area.

Lack of data for assessing cost-effectiveness of program delivery: One goal of the evaluation was to evaluate the cost effectiveness of the delivery of the PSIP (e.g. effective use of staff resources and cost impacts on recipients (the three companies)). Unfortunately, the evaluators determined that suitable systems for the ongoing collection for data on these matters did not exist. Thus the only useful information available on the cost effectiveness of program delivery was qualitative (i.e., comments provided during interviews with ITO officials)¹.

Gaps in corporate memory: Staff turnover, moves to other departments and retirements of those involved in the startup of PSIP created gaps in the information available to the evaluators. For example, none of the ATIB officials interviewed were involved in the PSIP negotiations.

Challenges of company interviews: Officials from the companies were generous with their time in meeting with the evaluators. However, some company representatives were more forthcoming in responding to evaluators' questions than others. The evaluators appreciate that the receipt of government funding might have led some corporate officials to be more circumspect in their responses to questions posed by the evaluators. As well, the evaluators reviewed the files, but were not able to conduct interviews with officials from the firm that was involved in the negotiations phase at the time of the evaluation.

Challenges in evaluation planning: The evaluation was delayed at the planning stage in order to address issues related to the commercially sensitive nature of project documents and related confidentiality issues. A further delay occurred during the data collection phase because of a federal election in the fall of 2008.

Impact of limitations: The above limitations posed some difficulties; however, overall, the evaluators concluded that data and information collected were satisfactory for assessing the program.

[1] In early 2009, AEB collected extensive information on time spent by ATIB and ITO officials on monitoring the three PSIP projects [as well as data for one project that was being negotiated] but no comparative information was collected. As well, key cost elements were not available, including costs for portfolio management functions such as director and team led reviewing, legal opinion and advice, quality assurance group for policy and documents, financial analysis input, or time for reviewing claims. Without information which compared the full monitoring of PSIP with other TPC projects the evaluators concluded that the information could not be utilized for an assessment.

5. Findings

This section of the report outlines the findings of the evaluation, organized by the four major themes: relevance, program design and implementation, performance measurement, success and progress, and cost-effectiveness of program delivery.

5.1 Relevance

This section relied on discussions with government officials and company representatives as well as the document review to help assess the following two questions:

1. Does the PSIP overlap, duplicate or complement other programs designed to address the needs of the automotive industry?
2. Is PSIP aligned with Government of Canada and Industry Canada objectives?

Conclusions: PSIP was devised to allow the federal government to support major investments by the auto assemblers. At about the same time the Government of Ontario developed the Ontario Automotive Investment Strategy (OAIS)^{Footnote 1} and provided funding to the same projects^{Footnote 2}. The two programs were complementary and demonstrated the capacity of the two levels of governments to work together to support one of Canada's most important manufacturing industries. The PSIP projects advanced and supported government initiatives by investing strategically in industrial research, pre-competitive development and technology adaptation and adoption projects. These projects are aligned with Government of Canada and IC objectives as regards to technology, the environment, sustainability, and industry growth.

Key findings:

All IC interviewees were aware of other programs at the federal and provincial levels that target the automotive industry among other sectors, for example, the Hydrogen Early Adopters Program. However, all of these IC staff stated that there was no duplication between PSIP and other federal/provincial government programs – rather the programs were complementary. Discussions with OAIS officials and company representatives reaffirmed that the OAIS and PSIP programs complemented each other, and provided mutual incentives for investment by all parties. Where there was potential for overlap, Government of Canada "stacking rules" prevented this from occurring. External evidence, mainly from company representatives, also indicated that there was very little, if any, overlap between PSIP and other programs.

The document review confirmed that the federal and provincial programs did not duplicate each other. Specifically, the PSIP terms and conditions indicated that the program supports federal government initiatives by investing strategically in industrial research, pre-competitive development and technology adaptation and adoption projects proposed by firms in the automotive sector in order to encourage private investments. This differs from the Government of Ontario's program, which supports projects which primarily advance skills and training – a task not identified as a PSIP objective^{Footnote 3}.

From the perspective of company representatives, there was a clear division of elements of the projects funded under each program (no capital elements were double-funded), and Ontario's emphasis was more on training. Company comments included:

A wide range of evidence supported the assessment that PSIP is aligned with the Government of Canada's and Industry Canada's objectives. All interviewees were unanimous about this, and indicated that PSIP was aligned with the Government of Canada objectives, e.g., promoting a knowledge-based economy; improving the competitiveness and productivity of the industry; and promoting both environmental and economic sustainability. Comments included:

IC through the Industry Canada Act is mandated to "increase the international competitiveness of Canadian industry, good and services and assist in the adjustment to changing domestic and international conditions". PSIP projects are also linked to the Departmental strategy to foster the knowledge-based economy through enhanced research and innovation^{Footnote 4}.

The document review also affirmed that PSIP was aligned with federal objectives. For example, technological goals were reflected by Fumes-to-Fuel technology, flexible manufacturing facilities, and bringing "new to Canada technology". Environmental goals were reflected in the reduction of CO₂ and VOC emissions in the manufacturing process (e.g., new paint shop and fuel cell technology and active fuel management technology), a reduction in natural gas consumption, recycling facilities at plants, and production of reduced greenhouse gas emitting vehicles. Economic benefits were evidenced in: maintenance of jobs; new job creation; increase in tax revenue; and creation of indirect jobs (e.g., construction, parts manufacturing, etc.).^{Footnote 5}

5.2 Program Design and Implementation

Interviews and the document review were used in developing the findings in this section.

Conclusions: The program design was effective in delivering funds for the PSIP projects. Although the negotiations were lengthy and sometimes generated frustration on the part of company and IC negotiators, the three projects were successfully negotiated and all three were completed in accordance with agreed-upon timelines. Experience with the program suggests that it is important to keep all discussions/negotiations within the same branch if at all possible in order to facilitate the negotiation process. This approach has been adopted for the Automotive Innovation Fund (AIF) program.

Key Findings:

Roles and responsibilities

Interviews and documents indicated that officials from ATIB participated in the negotiation process and ITO staff participated in the monitoring process^{Footnote 6}. ATIB officials began the consultations with the companies and, as the projects continued, ITO officials joined the discussions. A review of redacted emails also suggested that at the outset of discussions with the companies, most of the conversations took place with Industry Sector (ATIB) officials.

Nearly all IC interviewees found it difficult to comment on the efficiency of the roles and responsibilities of the Industry Sector and ITO. More than half of IC interviewees felt that the roles and responsibilities of the Industry Sector and ITO staff were clearly delineated; others found it difficult to comment. None was able to comment on the clarity of the negotiation process, because none were involved in that process (which speaks to the high rate of staff turnover the evaluators observed for these units of IC (retirements and transfers of key officials)). Over 50% of interviewees found the monitoring process to be somewhat clear or very clear. Satisfaction with the working relationship between Industry Sector and ITO was good, with over 50% indicating that they were "somewhat satisfied" and, at most, "very satisfied". Some comments from IC staff illustrating the above include:

Communication between Industry Sector officials and those from ITO also were reported to have improved over the time of PSIP's implementation. Although company representatives were not asked about this matter, some did comment on their working relationship with Industry Sector and ITO officials in the course of discussing the evolution of their projects. There was general agreement that the working relationship improved over time. A few company representatives also commented that it was easier to work with the province than with the federal government^{Footnote 7} and that there were periods during the course of their project when they had limited if any contact with Industry Canada personnel.  One commented:

Company representatives also commented on the development of their projects and the prolonged negotiation process that had created extra work. For example:

The document review provided some information on the delineation of roles between Industry Sector and ITO in some instances (i.e., development of the investment decision document or a memorandum for the Minister of Industry) but a specific document that clearly delineated the roles of each group did not exist. Email correspondence also revealed some communication difficulties between the two branches at the outset of the process.

In the end, the question regarding the effectiveness of processes could not be fully answered with the evidence at hand. A high level of turnover in IC staff and the lack of access to some staff were obstacles to the evaluators examining some issues.

However, the fact that different people and sections were involved in negotiations and phases through to monitoring suggested that information was lost in the "hand-off" at each stage. Whatever the lack of clarity of responsibilities between ATIB and ITO and the sometimes inconsistent participation of officials from both units in negotiations, current policy is to maintain a more integrated approach to all discussions with the auto industry.

Processes

Because PSIP was developed specifically to fund large scale automotive investments, the length of time to negotiate the projects was not deemed to be an unusual issue, although the companies found some aspects of the process frustrating. The interviews did not allow the evaluators to determine the efficiency of the negotiation process because the IC staff interviewed had not been involved in key parts of the negotiations, and documents were ambiguous about roles. Without data comparing the PSIP process to other ITO projects, it is difficult to assess the effectiveness of the processes, save for the obvious success in the negotiations. IC staff opinions regarding the effectiveness of the monitoring and reporting requirements varied. As well, none of the IC interviewees was able to assess the effectiveness of company interaction with government officials. Finally, none of the interviewees was able to provide an opinion regarding the initial steps in the coordination between federal and provincial funding programs. IC personnel were, however, very positive about the effectiveness of the current monitoring process.

OAIS officers felt that there were some aspects of management of PSIP that both impeded and facilitated the management of the projects. OAIS officers emphasized that communications in the form of all-party annual meetings greatly aided in the facilitation of the project management for both PSIP and OAIS. OAIS interviewees felt that staffing and differences of timetables between IC and OAIS has at times impeded efficient management. OAIS interviewees felt that maximum communications throughout the projects was a great benefit.

Company representatives expressed generally positive assessments of PSIP delivery. At two of the companies there was a shared sense that there had been initial challenges in sorting out reporting protocols, especially when the project was passed to ITO following negotiation with Industry Sector. One company spokesperson noted that in his view there was an initial mismatch between the paperwork required by ITO and the scale of the project. However, these challenges had been effectively addressed and there was now a good system in place for reporting to both PSIP and OAIS, and it was felt that in general, the initial negotiating process had gone smoothly.

IC documents reviewed also indicated both difficulties and successes in the processes. As noted above a broad analysis of inter-branch email correspondence indicates that there were difficulties during the negotiation phase of the projects. Companies, ITO and Industry Sector were reportedly left out of conversations at different points in the process. On the other hand, project documents demonstrate that the companies were appreciative of the time which ITO employees had provided throughout the negotiation process and one company interviewee commented that "when the contract was being developed the same people were involved throughout the process which really helped. The negotiation process went well."

5.3 Performance Measurement

This particular issue included one question:

Conclusions: The evaluation found that no performance measurement system was in place in the usual sense for the program as a whole. While data was regularly collected, this information was not being "rolled up" into an overall program assessment. However, a high level of reporting was obtained from two of the companies and this was deemed to be positive. For the third project, reporting had just begun at the time of the evaluation and it was thus not possible to review and assess the level of reporting. Companies' difficulties with the reporting requirements, for the most part have been worked out. That said, the creation of standard reporting forms would facilitate the collection of performance information and ensure consistency across projects.

While efforts were made to determine the appropriate timing of a final evaluation of the program, as a result of the fieldwork conducted for this evaluation, and subsequent discussions with the Treasury Board Secretariat's Centre of Excellence for Evaluation, a decision was made to forego the conduct of another evaluation of this program. This decision was based on the following considerations:

• The PSIP is a legacy program and will not fund any additional projects before it sunsets on March 31, 2011. A successor program for the automotive industry, the Automotive Innovation Fund, is already in place.
• This evaluation includes a discussion of program results, and fulfills the requirements of the Evaluation Policy to assess program performance. Given the current challenges being faced by the automotive industry and the large scale changes that are likely to occur, it would be difficult, in any future evaluation, to attribute subsequent outcomes to the program.
• The program will continue to be monitored through to the repayment phase of the projects. Any significant issues arising at the repayment phase could be covered through the audit process.

Key Findings:

There is no formal performance measurement system in place for PSIP as a whole, based on evidence gathered from the document review and interviews. This is the case in spite of the fact that a detailed RMAF was developed (a countervailing positive factor is that companies were requested to adhere closely to reporting of key data required for performance measurements).

Reporting documents such as progress reports, claims reports and update reports shed good light on the quality of performance at the project level. Four kinds of reporting were required as part of each project funding decision. Companies were required to submit claims reports, an annual review, project progress reports, and an annual information update. Quarterly reports were available for two of the projects. In these the companies identified the progress made during the reporting quarter on a range of components of the overall project and on the number of jobs created and/or maintained. These Quarterly reports were submitted as part of the claims process. Although these reports contain more or less the same information, the format differs between the two companies which have provided regular reports.

Interviews with company representatives revealed general satisfaction with the reporting process as it now stands, although there were some "start-up" challenges. There had been initial challenges in meeting ITO reporting requirements and coordinating reporting to the two levels of government for the two complementary programs (PSIP and OAIS). However, companies appreciated the flexibility shown by PSIP and the willingness of ITO staff to work with them to iron out difficulties. One company, however, felt that there was a tendency for ITO to request information beyond what was stipulated in the contractual agreement.

5.4 Success/Progress^{Footnote 8}

The following two questions were addressed based on interviews, the document review and the site visit.

Conclusions: PSIP is having intended economic and technological impacts benefiting communities, industry and workers. All lines of evidences pointed to initial achievement of goals, including the establishment of more up-to-date assembly plants, and in the case of Toyota, an increase in employment with the opening of their new plant. Sustainability of these results remains an open question, given the current state of the automobile industry.

Key Findings:

Intended Beneficiaries

Overall findings from all sources indicate that important benefits have flowed to the intended beneficiaries. IC interviews indicated that the PSIP projects, although still in their early phases, have reached many of the intended beneficiaries and other benefits are unfolding/will occur in the future. IC interviewees suggested that PSIP:

Generally the interviewees felt that PSIP has been, or will be, effective in benefiting workers and unions. Supporting comments from IC staff include:

OAIS interviewees also felt that PSIP is reaching its intended beneficiaries. All OAIS interviewees suggested that PSIP's contributions have had positive impacts for the companies, the industry and the specific Canadian communities affected and their neighbors. Comment illustrating the above views of OAIS staff includes:

These findings were corroborated through the document review. IC documents indicated that PSIP projects are in the process of reaching intended beneficiaries, particularly in terms of new jobs created. These job creation efforts were also seen as linked to high level skill objectives, an OAIS objective.

Results Achieved

All OAIS interviewees felt that PSIP has strengthened recipient firms and that there had been diffusion and adoption of technology beyond the recipient firms. All OAIS interviewees felt that there was a positive impact on economic growth for Canada and that sustainable development was aided. The following comment illustrates the above views of OAIS staff:

This was reaffirmed by the company representatives. They provided extensive evidence on positive benefits accruing from the PSIP investments, particularly in terms of flexible production for the two renovated assembly plants which have strengthened their plants significantly and provided a degree of 'protection' during the current industry restructuring. The site visit revealed observable improvements as a result of the development of flexible manufacturing facilities that are expected to provide economic and technological benefits to the company. The company representatives all stated that their projects would not have happened without support from PSIP and that the investments have resulted in plants that are competitive and productive within their respective company facility networks.

One company stated that while it had met its job targets under the OAIS/PSIP agreements, it would be difficult to make similar forward commitments if entering into a new agreement today given the level of economic turmoil. Comments supplied by company representatives which support the above statements include:

Interviews with community representatives were generally positive about the present and future program successes. An interviewee indicated that the Woodstock project investment has created many direct and indirect jobs which have greatly aided the local economy, noting that many new parts manufacturers have moved into the area as a result of the PSIP investment. Another one noted "[The].project was a great initiative and a good investment for the government to make. From our perspective it has done a lot for our community and put into play some significant changes that will be long lasting". However, when the company representatives were asked "what benefits do you expect within the next 5 years and beyond?" all commented on the economic uncertainties currently confronting the automobile industry, both in North America and globally, and how this makes forecasting difficult.

Several community representatives commented on the fact that without the PSIP/OAIS investment the auto industry in southern Ontario would be in worse shape during the current economic downturn.

Another community representative indicated that the Beacon Project has resulted in the creation of state of the art research facilities at the University of Ontario Institute of Technology (UOIT) and created 20 full-time technical positions in addition to the additional academic staff hired. He noted that the new trend for companies with the aid of government assistance to establish automotive research design and development centres in Canada has given the Canadian automotive industry a competitive edge which is a "tremendous step in the right direction". The Automotive Centre of Excellence (ACE) at UOIT in partnership with GM and other companies will help build a new kind of automotive economy in Ontario that is founded on the existing industry but geared to the production of high value-added vehicles.

The Union officials at Ford Oakville and GM Oshawa and an official in the Canadian Auto Workers (CAW) National Office all commented very favourable on the impact of the investments.

The document review revealed that new paint technologies have been developed and implemented to benefit the automotive industry, the Flex Cell body shop at Oakville is complete, and the Advanced Manufacturing Technology C-Flex in Oshawa is on schedule for completion in early 2009.^{Footnote 9} PSIP repayments will not begin to flow for three to five years [assuming there are no requests for delays as a result of the financial difficulties currently being experienced by vehicle assemblers].^{Footnote 10} A sampling of IC documents reveals that the projects are all on track or completed, and that a number of the employment goals were met in spite of the current economic downturn. The closing of the GMCL Oshawa Truck Plant would have triggered the early repayment provision of the Beacon agreement. However, in September 2008, the Minister agreed to waive this provision in return for a $290 million new investment by the company in the GM Pathway Project.^{Footnote 11}

Assessing the impact of a Greenfield investment is easier than measuring/assessing the impact of the introduction of flexible manufacturing and new technologies. A new assembly plant not only adds new direct employment but also attracts supplier firms with which the assembler has a relationship. The establishment of a number of new suppliers and the expansion of some existing ones in the Woodstock area and the jobs they created can be documented, as can other spin-off effects (e.g. increase in the municipal tax base).

In the case of the revitalization of existing plants by converting them to flexible production it is unlikely that there will be a significant number of new jobs created either in the assembly plant (where the very purpose of technological upgrading is to increase efficiency and labour productivity which will likely reduce the total number of workers needed) or among the supplier base. Rather the impact of such investments is probably best measured by the degree to which productivity and capacity utilization rates increase ensuring the longer term competitiveness of the assembly plant to attract new product assignments and retain employment.

In the most recent (2008) edition of the influential Harbour Report, Ford OAC ranked as the second most improved vehicle assembly plant in North America in 2007 in terms of labour productivity (measured as hours per vehicle); recording a 38.4% year-over-year improvement. Plant capacity utilization in 2007 was 93% which represented a very significant improvement over 2006 and launch quality for the new vehicles was on a par with Toyota and Honda.

Overall, the initial goals of the projects are all on track or completed, and a number of the employment goals were met, although the across-the-board decreases in vehicle production will temper the job numbers in the short to medium term because of the current economic downturn.

5.5 Cost-Effectiveness of Program Delivery

Key questions were:

Conclusions: Interview information suggested that resources were satisfactory for the delivery and monitoring of the PSIP funding. However, the gaps in co-participation of the various IC units in all stages of the projects (from negotiations to monitoring), may have reflected a shortfall in resources. As well frequent staff turnover resulted in inefficiencies because new officials had to get up to speed on existing files. However, without comparative data on time spent on monitoring other ITO projects it is impossible to draw any firm conclusions with respect to the adequacy of resources available for monitoring PSIP funding. Overall, the question on cost effectiveness could not be addressed definitively.

Key Findings:

In general, information available indicated that resources were satisfactory for program monitoring and the delivery of PSIP funding, although as noted below human resources issues have had an impact on project monitoring. There was no on-going collection of data on monitoring in the sense of hours normally required to monitor projects of this sort. Therefore the assessment can only be made based on information obtained from interviews.

Some IC interviewees could not comment on the adequacy of IC resources for delivering PSIP. ITO interviewees noted that each officer has a number of files for which he/she is responsible, but none commented on whether their PSIP files took more or less of their time than other ITO project files. Staff turnover as well as fewer employees at ITO when the evaluation was being done than in the previous year has had an impact on project monitoring.

Company representatives interviewed were generally satisfied with IC resources and availability and commented favourably on the relationships they enjoyed at the staff level with ITO. Comments offered during the company interviews included:

The cost-effectiveness of delivering the program has not yet been addressed in detail since no data has been supplied that would allow the assessors to consider this question definitively. Some indications on this topic came from interviews. IC interviewees felt that PSIP administration costs were comparable to other federal programs.

No information was obtained from the document review which assesses the cost-effectiveness of PSIP. However, ITO provided some estimated data on FTEs^{Footnote 12}, travel costs, and salaries.

Footnotes

1. ^{1 back to footnote reference 1} Launched in 2004, the government of Ontario unveiled $500 million to strengthen the industry's competitiveness and train workers. Particularly, the investment was to support private sector investment with: Skills training; innovation and research; improved infrastructure; and energy efficiencies and environmental technologies. (From the website of the Ministry of Economic Development and Trade).
2. ^{2 back to footnote reference 2} Note that OAIS also provided funding to projects beyond these three co-funded with PSIP. Other OAIS funded projects include ones by Ford, Daimler Chrysler, Navistar and a number of automotive component manufacturers.
3. ^{3 back to footnote reference 3} www.investinontario.com
4. ^{4 back to footnote reference 4} The PSIP contributes to the achievement of the Department's strategic outcome " science and technology, knowledge and innovation are effective drivers of a strong Canadian economy".
5. ^{5 back to footnote reference 5} Documents also indicated that PSIP contributed to strategic objectives of IC (as articulated when PSIP was created): new jobs, direct and indirect, and industry growth; an innovative economy through the adoption of new technology, e.g., new paint and advanced manufacturing technology; and, ensuring a competitive industry and sustainable communities (in environmental benefits and new clean technology).
6. ^{6 back to footnote reference 6} Please note that ITO and ATIB are two separate branches within Industry Canada reporting to two different Assistant Deputy Ministers (ADMs).
7. ^{7 back to footnote reference 7} Some company officials suggested Ontario officials were easier to work with than their federal counterparts because the Ontario officials had more in depth knowledge of the auto industry. This may be the result of greater movement/mobility of IC personnel than those in the government of Ontario.
8. ^{8 back to footnote reference 8} This section of the report seeks to identify results achieved through PSIP funding. The size and scope of the projects, however, go beyond the funding received through PSIP. This fact should be taken into consideration when reading this section of the report.
9. ^{9 back to footnote reference 9} From PSIP files.
10. ^{10 back to footnote reference 10} From PSIP files.
11. ^{11 back to footnote reference 11} Exchange of letters between Industry Canada and General Motors of Canada Limited, Industry Canada press release, September 6, 2008.
12. ^{12 back to footnote reference 12} The full-time equivalent (FTE) is the ratio of the assigned hours of work to the scheduled hours of work.

6. Conclusions

The results of the evaluation suggest that PSIP is relevant, has a good program design, was implemented successfully, and has achieved its short-term goals as reflected in reported successes. The interviews with company officials, the document review, and the site visit reinforced these findings as did interviews with community and union officials.

More specifically, results indicate:

• The PSIP and Ontario programs were seen as complementary; each designed to support discrete aspects of the overall projects.
• PSIP was well aligned with Government of Canada and IC objectives at the time the program was developed; namely the promotion of an innovative and knowledge-based economy; improving the competitiveness and productivity of the industry; and fostering development that is both economically and environmentally sustainable.
• The overall process for delivering the PSIP funds appears to have been effective. However, no definitive conclusions could be reached on whether the overall PSIP program, including monitoring, was cost-effective.
• There is no formal performance measurement system for PSIP as a whole (with a "roll-up" of measures for PSIP as a whole), although reporting by project has been reasonably thorough. At least two of the companies encountered some initial challenges in meshing their project as negotiated with IC with the reporting protocols used by ITO. However, the reporting process is generally now working well and company representatives expressed the hope that a similar system would be used in any subsequent Federal Government program.
• Company investments are regarded as successful. Although the Ford Centennial Project and, in particular, the GM Beacon Project involve a number of elements supported by PSIP, perhaps the most significant is the installation of advanced flexible manufacturing technology. The productivity and the competitiveness of the established assembly complexes at Oshawa and Oakville have been significantly increased with the investments in flexible body build facilities. In the case of the Toyota Jewel Project, the PSIP contribution enabled added capacity in a planned new Greenfield assembly plant. The Toyota plant incorporates leading edge flexible production technology and is one of the most advanced assembly plants in North America. In retrospect, and given the current turmoil in the North American automotive industry, the timing of the PSIP investments may have contributed to protecting Canada's place in the North American automotive industry. The companies stated that without the PSIP and OAIS contributions the projects would not have gone forward.
• PSIP is seen to have reached its intended beneficiaries, with positive results. It has enhanced the competitiveness of vehicle assembly in Canada by maintaining and augmenting assembly capacity, which may help sustain a market for Canadian produced automotive parts and components. Although the projects have not currently produced as many additional jobs as originally projected, it is possible that without these investments Canadian job losses could have been even more substantial during the current downturn in the North American automotive industry.