Evaluation of the Marquee Tourism Events Program

Final Report

June 30, 2011

Tabled and approved at the Departmental Evaluation Committee on June 30, 2011

Table of Contents

• Executive Summary
• 1.0 Introduction
  • 1.1 Program Context and Overview
  • 1.2 Program Resources
  • 1.3 Program Design and Delivery
  • 1.4 Expected Results
  • 1.5 MTEP Logic Model
• 2.0 Evaluation methodology
  • 2.1 Evaluation Approach
  • 2.2 Evaluation Objectives and Scope
  • 2.3 Evaluation Questions
  • 2.4 Data Collection Methods
  • 2.5 Data limitations
• 3.0 Evaluation findings
  • 3.1 Relevance
  • 3.2 Performance
• 4.0 Conclusions and lessons learned

Appendices (Separate document)

Appendices are available via an Access to Information

Appendix A: MTEP Evaluation Matrix
Appendix B: List of Individuals Interviewed
Appendix C: Interview Guides
Appendix D: List of Approved Summer Marquee Tourism Events and Tier One Events

Executive Summary

Program Overview

Recognizing the importance of tourism to the visitor economy, through Budget 2009, Canada's Economic Action Plan (EAP) allocated $100 million over 2 years (2009–2011) to provide short-term financial assistance to existing marquee tourism events. The objective of the Marquee Tourism Events Program (MTEP) was to assist existing marquee tourism events deliver enhanced world-class programs and experiences, thereby increasing or sustaining the number of tourists from within and outside Canada.

Support for marquee tourism events was expected to contribute to the short-term and long-term growth and viability of Canada's visitor economy by increasing or sustaining the level of tourism and tourism-related spending, and providing economic stimulus to ensure that tourism events could remain competitive within and outside Canada.

Evaluation Purpose and Methodology

The purpose of the evaluation was to assess the relevance and performance of the program. As MTEP was an EAP initiative, emphasis was placed on assessing the program's success in delivering financial assistance in a timely and targeted manner.

The evaluation findings are based on the analysis of multiple lines of evidence. The methodology included a review of documents, an analysis of financial and performance data, along with interviews with program officials, project recipients, and key tourism stakeholders.

Evaluation Findings

Relevance

The global economic crisis created challenges for key sectors of the Canadian economy, including the many service sectors that make up the tourism industry. By providing targeted financial assistance to marquee tourism events, the program responded to the need for an immediate economic stimulus to the tourism sector. MTEP's objective aligns with federal government priorities and is consistent with the mandate and strategic objectives of Industry Canada (IC).

Performance

MTEP was designed and delivered under very tight timelines. The mandate to develop MTEP was given to IC in February 2009, the program was launched in April 2009, and the application deadline for first year project proposals was May 8, 2009. Despite the significant challenge of designing and delivering a new contribution program under such tight timelines, the department succeeded in implementing MTEP within the required timeframe. Planned levels of contribution funding to support events were approved each fiscal year. However, at a project level, many year one recipients did not receive notification and confirmation of funding with sufficient time to plan and implement all of their activities as intended. As a result, the potential impact of the program was likely limited in the first year.

An analysis of the overall efficiency of the program delivery was conducted, with an emphasis on identifying changes in efficiency in year two of the program. Earlier application deadlines in year two of the program allowed for more time to process project proposals and approve funding. MTEP also streamlined several processes that improved the timeliness of program delivery and the overall efficiency of the program in year two.

The assessment of the program's cost-efficiency was limited to an analysis of planned versus actual operating costs. MTEP was allocated $4 million, representing 4% of the total program costs. Actual operating costs were higher than planned, largely due to increased salary expenditures needed to respond to the high demand for the program, to deliver MTEP funding within tight timelines, and to maintain program capacity in year two. In addition, as MTEP was a two year EAP initiative, no funding was allocated to close the program in 2011–2012. However, despite higher than planned expenditures, MTEP's actual operating costs remained at just under 6% of total program costs - a relatively low ratio for a Grants and Contributions program.

MTEP achieved its immediate program outcome. MTEP funding stimulated improvements to the quality of funded marquee tourism events through increased spending on incremental activities, such as new or enhanced marketing and promotion, product development, programming, capital projects, and equipment. MTEP project activities aligned with the intended outcomes of the program and created positive benefits for recipients.

Due to a lack of comparable baseline data, an assessment of the extent to which the program achieved its intermediate program outcomes was not possible. Specifically, an analysis of the change in the number of domestic and international tourists attending supported events, the change in the level of tourism related spending, along with the change in event revenues, could not be comprehensively conducted.

Lessons Learned

Given that the program was temporary and sunsetted in March 2011, the evaluation does not include recommendations. Instead, "lessons learned" were identified with the overall objective of informing organizational learning and program excellence.

Lesson Learned 1: Applicants need time to comprehensively develop project proposals and require timely notification of funding to ensure that projects can be implemented successfully.

Lesson Learned 2: As part of the program design, take the time necessary to develop a suitable data collection system to support ongoing program performance reporting and evaluation needs.

Lesson Learned 3: Temporary programs may result in staffing problems and gaps, which could limit the efficiency of program delivery. Quick approvals for assignments, the creation of opportunities to build skills and gain valuable career experience, along with departmental support to transition staff as a program closes can help address these challenges.

1.0 Introduction

This report presents the results of an evaluation of the Marquee Tourism Events Program (MTEP).

The purpose of the evaluation was to assess the relevance and performance (effectiveness, efficiency and economy) of MTEP. The report is organized into four sections:

• Section 1 provides the program context and profile of MTEP;
• Section 2 presents the evaluation methodology along with a discussion of data limitations;
• Section 3 presents the findings pertaining to the evaluation issues of relevance and performance; and
• Section 4 summarizes the study's conclusions and provides lessons learned.

1.1 Program Context and Overview

Due to the international economic downturn in 2008, along with decreased private sector sponsorship and competition from other tourist destinations, the tourism industry in Canada faced significant challenges. In January 2009, the Government of Canada introduced a stimulus package in response to the potential effects of the global recession on the Canadian economy.

Canada's Economic Action Plan (EAP) was developed to provide economic stimulus to encourage growth and restore confidence in the economy. The Action Plan was based on three guiding principles—that the stimulus should be timely, targeted, and temporary. Recognizing the contribution of tourism to the visitor economy, through Budget 2009, Canada's EAP allocated $100 million over two years (2009–2011) to provide financial assistance to existing marquee tourism events.

MTEP was formally launched on April 6, 2009, by the Minister of State (Small Business and Tourism). MTEP was a contribution program, which provided short-term (one or two year) financial assistance to existing marquee tourism events. The objective of MTEP was to assist existing marquee tourism events to deliver enhanced world-class programs and experiences, thereby increasing or sustaining the number of tourists from within and outside Canada. The program defined a marquee event as an annually recurring world-class event that is well established and has a long-standing tradition of programming and management excellence.

Support for marquee events was expected to contribute to the short-term and long-term growth and viability of Canada's visitor economy by increasing or sustaining the level of tourism and tourism-related spending, and providing economic stimulus to ensure that tourism events could remain competitive within and outside Canada.

The program was managed and delivered through the Small Business and Tourism Branch (SBTB) of Industry Canada (IC) and sunsetted at the end of the 2010–2011 fiscal year.

1.2 Program Resources

In terms of funding, the program was initially allocated $100 million over two years. The following table provides details of the initial budget allocation for each year of the program.

In April 2010, a transfer of $8 million was approved from IC Vote 10 Grants & Contributions (Gs&Cs) to the Canadian Tourism Commission (CTC), to invest in new activities in domestic and priority international markets. This reduced the total MTEP budget to $92 million. Available contribution funds were reduced to $88 million and total operating expenses remained at $4 million. Table two provides details of the final budget allocations for each year of the program.

1.3 Program Design and Delivery

The program was designed with three funding streams:

• Summer Marquee Tourism Events – To provide a rapid and targeted response, a one-time funding envelope was allocated to ten events taking place between April 1, 2009 and September 30, 2009 (maximum contribution of $3 million in 2009–2010);
• Tier One – To help larger marquee events that had an existing international marketing strategy that incorporated overnight tour packages (maximum contribution of $3 million per fiscal year); and
• Tier Two – To support smaller marquee events that did not meet Tier 1 criteria, but had a domestic marketing strategy and an existing or proposed international marketing strategy that would incorporate overnight tour packages (maximum contribution of $1 million per fiscal year).

With the support of information gathered from industry sources and the experiences of other government organizations, eligibility criteria were developed to identify those events that would meet the definition of a marquee event and would offer the best prospect of drawing increased numbers of tourists. Some of the criteria included the number of years in existence, overall attendance at the event, consecutive days of programming, size of cash operating budget, and existence of an international marketing strategy or the intention to develop one with program support.

Summer Marquee Tourism Events

To allow for a quick roll out of the program, and to ensure that summer tourism events received timely support, a Summer 2009 Advisory Committee^{Footnote 1} was established to identify a limited number of world class events that contributed to each region of Canada's visitor economy. The Committee met twice, once in March 2009 to identify tourism events that would be invited to apply; and, a second time in April 2009 to evaluate and confirm the eligibility of the events for recommendation to the Minister of State. The Minister of State provided final funding approval for 10 summer marquee events in the Spring of 2009.

Tier One and Tier Two Events

In April 2009, there was one call for proposals for both Tier 1 and Tier 2 events occurring in the fiscal year 2009–10. The deadline for applications was May 8, 2009. In year 2, the call for proposals went out on November 30, 2009, with a deadline of January 8, 2010.

MTEP program officials were responsible for ensuring that all applications submitted were complete and met program eligibility requirements, as well as determining whether the requested funding was within program funding limits based on submitted financial information.

All eligible Tier 1 events were reviewed by a Tier One Advisory Committee^{Footnote 2} to evaluate and confirm eligibility. Upon review by the Advisory Committee, eligible event proposals were submitted to the Minister of State or Minister of Industry for approval.

Tier 2 proposals exceeding $500,000 were submitted to the Department's Programs and Services Board (PSB) for review. In year one, a merit-based score card was completed for each eligible Tier 2 proposal in order to determine projects with the greatest potential impact. Events were sorted based on region and ranked according to the score received. The scoring for these Tier 2 proposals was completed by an independent third party, temporarily employed for this purpose only. In year two of MTEP, a list of eligible Tier 2 proposals was submitted to the Minister for his review. Program funding for the second year of the MTEP was focused on a broader regional distribution of support.

The Minister receiving the eligible Tier 1 and Tier 2 event proposals provided the approval on all funding decisions.^{Footnote 3}

Program Reporting

Standard Contribution Agreements were developed and all Contribution Agreements included a Statement of Work (SOW) setting out the activities that the recipient agreed to perform and toward which the department was making its contribution.

Recipients were required to provide monthly progress reports. Site visits were conducted by MTEP officials. Monitoring also included third party monitoring reviews and recipient audits conducted by external audit firms. Approximately 20% of MTEP projects were identified for review or audit using a risk based approach.

Final reports were required upon completion of all projects, including a third party economic impact study and audited financial statements.

1.4 Expected Results

It was anticipated that in the short-term the program would enhance funded marquee tourism events through increased spending on incremental activities such as new or enhanced marketing and promotion, product development, programming, capital projects, and equipment. This in turn was expected to translate into sustaining or increasing the revenue generated by the event, sustaining or increasing the number of tourists attending these events, as well as sustaining or increasing the tourism related spending in the local regions for the duration of the supported marquee events. This tourism-related spending was expected to benefit hotels, restaurants and other local businesses.

The following logic model for MTEP was developed as part of the program's performance measurement strategy (May 2009). The logic model outlines the program's inputs, activities, and outputs, as well as the intended short-term, intermediate, and longer-term outcomes.

1.5 MTEP Logic Model

Figure: MTEP Logic Model

2.0 Evaluation Methodology

This section describes the overall evaluation approach and scope, the specific questions that were addressed, the data collection methods used, and the limitations of the methodology.

2.1 Evaluation Approach

The evaluation was conducted using a hybrid approach. The overall evaluation study was managed and conducted by IC's Audit and Evaluation Branch (AEB), with two components completed by other groups. Interviews with project recipients and key tourism stakeholders were conducted by a third party contract with the consulting firm Performance Management Network. The analysis of tourism and economic data collected through economic impact studies was independently conducted by the Research and Analysis group of SBTB.

2.2 Evaluation Objectives and Scope

Consistent with the Treasury Board (TB) Policy on Evaluation, the evaluation addressed core issues of relevance and performance. As MTEP was designed to provide temporary fiscal stimulus to meet the challenges of a specific economic crisis, and sunsetted in March 2011, the evaluation was calibrated to address the core issues as follows:

Relevance – An assessment of the relevance of MTEP was the secondary focus of the evaluation. The need for the program was assessed within the specific context of the EAP and the need for short-term fiscal stimulus for the tourism sector in response to the economic crisis.

Performance – An assessment of the performance of MTEP was the main focus of the evaluation. This included an assessment of progress toward the achievement of expected outcomes (effectiveness), and the demonstration of efficiency and economy. As MTEP was an EAP initiative, emphasis was placed on assessing the program's success in delivering financial assistance in a timely and targeted manner.

Given the timing of the evaluation, the assessment of effectiveness focused on the achievement of immediate outcomes (increased spending to enhance marquee events). The achievement of intermediate outcomes (sustained or increased tourism attendance, event revenues, and tourism related spending) was assessed to the extent possible and was limited to the results of the 27 projects funded in the first year under the Summer Marquee Tourism Events and Tier One funding streams.

Given that the program was temporary and sunsetted in March 2011, the evaluation does not include recommendations. Instead, "lessons learned" are identified with the overall objective of informing organizational learning and program excellence.

2.3 Evaluation Questions

The evaluation addressed the following questions:

Relevance

• Did the program meet a need of the tourism sector by providing targeted financial assistance to marquee events?
• Did the program align with government priorities and federal roles and responsibilities?

Performance

• Was MTEP delivered in a cost-efficient and timely manner?
• How did MTEP projects stimulate improvements to the quality of marquee tourism events? Did these improvements contribute to the achievement of short-term and intermediate program outcomes?
• Did the program achieve its intermediate program outcomes?
• Were there unintended impacts, positive or negative, that can be attributed to the program?

The Evaluation Matrix (Appendix A) includes the indicators, data collection methods and data sources for each evaluation question.

2.4 Data Collection Methods

Multiple lines of evidence, along with the triangulation of data, were used where possible to address all evaluation questions of performance. As the assessment of relevance was the secondary focus of the evaluation, data collection and analysis were limited to a review of key documents to address these questions.

Evaluation methods included a review of documents, analysis of financial and performance data, along with in-depth interviews with program staff, project recipients, and key tourism industry stakeholders.

Document Review

A number of documents were reviewed, including: program documents, such as the MTEP TB Submission, Program Delivery Procedures, and MTEP application forms and guides; program audits and reviews, such as the Review of Design and Delivery (2010) and the Management Control Framework Assessment (2010); along with evaluations of similar programs and contextual documents on trends in the tourism sector.

Data Analysis

Financial and performance data was reviewed at the program and project level to inform the assessment of all evaluation questions on the program's performance. Secondary data on tourism statistics was reviewed for contextual and comparative purposes.

Relevant primary data reviewed at the program and project level included: financial data, such as planned and actual operating expenditures; Contribution Management Information System (CMIS) data on allocated contribution funding and approved contribution funding; and financial data on planned and actual project expenditures by cost category. Additional project data was reviewed for the 27 projects funded under the Summer Marquee Tourism Events and 2009 Tier One funding streams. This included an analysis of: application data; audited financial statements; site visit and final project reports; and economic impact studies.

Relevant secondary data sources reviewed included Statistics Canada National Tourism Indicators 2007–2009 and provincial and municipal tourism statistics.

Interviews

A total of 30 interviews were conducted:

• 7 in-person interviews were conducted with IC program staff and management;
• 20 telephone interviews were conducted with 22 individuals representing 20 of the 27 Summer Marquee Tourism Events and 2009 Tier One recipients; and
• 3 telephone interviews were conducted with tourism industry officials in Toronto and Montreal.

Interviews with project recipients were distributed as follows:

Interview guides were provided to each interviewee prior to the interview. A list of individuals interviewed is included in Appendix B. A copy of each interview guide is included in Appendix C.

2.5 Data limitations

The following are the three main data limitations for this evaluation:

Timing and scope of evaluation

As the evaluation was conducted in the Fall of 2010, mid-way through the second year of the program, performance and economic impact data was only available for projects completed in year one of the program. The assessment of intermediate outcomes (data analysis and interviews) was limited to projects funded under the Summer Marquee Tourism Events and 2009 Tier One funding streams. These 27 projects represent 81% of program funds approved in 2009. A list of approved Summer Marquee Tourism Events and 2009 Tier One projects is included in Appendix D. The evaluation study did not collect data on unfunded marquee tourism events.

Availability of cost-efficiency data

The Policy on Evaluation requires that all evaluations provide an assessment of the degree to which programs have used resources efficiently to produce outputs which support the achievement of intended results. Efforts were made to compare MTEP's percentage of operating costs to total program costs with similar contribution programs for benchmarking purposes. Comparable data was either unavailable or the program context and characteristics were too different to allow for a meaningful comparison. While there was no available data within MTEP to calculate costs for different program activities (such as time tracking in the approval and administration of contribution agreements), the evaluation was able to analyze planned versus actual program operating costs, as well as identify areas in the program delivery process where efficiencies were gained in the second year.

Availability of baseline data

Due to a lack of comparable baseline data, it was not possible to comprehensively assess the change in the number of domestic and international tourists attending supported events, the change in the level of tourism related spending, along with the change in event revenues. For many events, this was the first time they had been required to collect consistent data on the number of international and domestic tourists attending their events, as well as the associated tourism-related spending that was generated. While all events provided data on tourist attendance from 2007 and 2008, in many cases these figures were collected using different methodologies and different definitions from those required for the economic impact studies. The 2009 data for tourism related spending was also calculated by third parties using a wide range of methodologies. For all MTEP funded projects, eligible expenses and MTEP payments were recorded in detail by program officials. However, for event revenues, third party audited financial statements were prepared using different reporting presentations.

3.0 Evaluation Findings

3.1 Relevance

3.1.1 Did the program meet a need of the tourism sector by providing targeted financial assistance to marquee events?

Canada's tourism industry makes a significant contribution to the overall economy and is an important source of employment. Between 2008 and 2010, Canada's tourism industry represented approximately 2% of Canada's Gross Domestic Product (GDP) and generated over 600,000 jobs each year in a range of tourism-related industries.

The global economic crisis created challenges for key sectors of the Canadian economy, including the many service sectors that make up the tourism industry. In 2008 and 2009 the Canadian tourism industry faced a number of challenges, including:

• global economic uncertainty;
• the H1N1 influenza pandemic;
• high fuel prices;
• high Canadian dollar; and
• new traveller passport requirements.

Festivals and events are important contributors to the tourism economy in Canada as they attract visitors to communities and regions across the country. As a result of the global economic crisis, a number of Canadian festivals and events lost private sector support. As a short-term measure, the government took on the role to deliver targeted financial assistance to marquee festivals and events.

While there are other federally funded programs that assist festivals and events, most promote elements of culture, heritage, the arts, and sports excellence. As MTEP's primary focus was on tourism and the visitor economy, it complemented other federal programs.

By providing targeted financial assistance, the program enabled existing marquee events to conduct new and enhanced activities in support of attracting increased tourists from within and outside Canada to these events; thus providing an immediate stimulus to the tourism sector. The program also helped events to fill short term funding gaps, where private sector support had been withdrawn due to the economic crisis.

3.1.2 Did the program align with government priorities and federal roles and responsibilities?

The need for economic stimulus to the tourism sector was identified as a government priority in Budget 2009, Canada's Economic Action Plan. To stimulate the growth of tourism and help position the industry for success in the growing and increasingly competitive global environment, Canada's EAP provided important financial support for federal programming, including MTEP, through Budget 2009.

Tourism is an increasingly important industry that supports small businesses and provides employment to Canadians across the country. Tourism is part of IC's mandate under the Industry Canada Act, S.4.1(o). Industry Canada leads federal tourism policy development by engaging with other departments on issues that impact tourism and in working cooperatively with provincial and territorial governments to help expand tourism.

The objective of the program was to assist existing marquee tourism events deliver enhanced world-class programs and experiences, thereby increasing or sustaining Canada's prominence as a destination of choice for tourists from within and outside Canada. This objective is consistent with IC's mandate to help make Canadian industry more productive and competitive in the global economy, thus improving the economic and social well-being of Canadians. It is linked to IC's strategic objective, "competitive businesses are drivers of sustainable wealth creation". MTEP contributed to IC's Program Activity Architecture sub-objective 3.2 "Global Reach and Agility in Targeted Canadian Industries" and more specifically, 3.2.6 "Global Reach and Agility in Service and Consumer Products Industries".

3.2 Performance

3.2.1 Was the MTEP program delivered in a cost-efficient and timely manner?

The assessment of the cost-efficiency and timeliness of the MTEP program included an analysis of:

• the timeliness of program implementation;
• the overall efficiency of program delivery—focusing on the use of program resources to achieve timely program delivery; and
• the planned versus actual program operating costs.

3.2.1.1 Timeliness of program implementation

Program implementation

Designing and delivering the MTEP program was a new undertaking for IC and presented several challenges. To respect the intent of the EAP, the design and implementation of the program had to be done under tight timelines. IC was advised that it had been given the mandate to develop and administer MTEP in early February 2009. A team needed to be quickly assembled to put together a program that would be ready for launch within two months (April 2009), in order to support festivals and events planned for early summer. The application deadline for first year project proposals was May 8, 2009. IC had not previously funded festivals or events and therefore had limited experience in this area. In addition, the capacity of IC to deliver a new program in such a short timeframe was limited, as a new tourism related program delivery unit had to be established, and reassigning staff from other areas or other departments to this unit was challenging. At the same time that MTEP was being developed, IC also had to implement other Budget 2009 programs under similar constraints.

Providing funding to marquee tourism events in a very short timeframe was the most significant challenge for the program in year one. Despite significant internal and external constraints, the program succeeded in implementing MTEP within the required timeframe. Planned levels of contribution funding to support events were approved each fiscal year. In total, funding was approved for 107 projects, for a maximum contribution of close to $88 million, as planned in the program budget allocations. Sixty events were approved in year one; 10 Summer Marquee Tourism Events, 17 Tier One events, and 33 Tier Two Events. Forty-seven events were approved in year two; 16 Tier One events, and 31 Tier Two events.

Table four shows the distribution of approved funding amounts by event tier and program year.

Figures one and two show the distribution of approved funding by event tier and region.

Figure 1: MTEP Funding Distribution by Tier and Region (2009–2010)

Figure 2: MTEP Funding Distribution by Tier and Region (2010–2011)

Timing of project applications and approval processes

At a project level, the timing of applications and approval processes was a significant challenge for prospective applicants. In year one, events had under five weeks from the initial call for proposals to the application deadline. It was difficult for event organizers to define a project in this short period of time.

On the part of the program, there were several constraints that prevented timely notification of funding and negotiation of Contribution Agreements with recipients. While tight timelines in year one (May 8, 2009 application deadline) made a lengthy advance notice impossible for many summer events, areas within the program process that also contributed to delays were identified. In year one, notification of funding approval, along with the subsequent public announcements, required multiple levels of approvals and coordination. This resulted in delays between the date that projects received approval and the date that the recipients received notification of funding. Depending on whether recipients required modifications to their proposed activities, along with their familiarity and experience with contribution programs, it then took days or even weeks to finalize their Contribution Agreements. In year one of the program, 57% of Contribution Agreements (34 out of 60) were signed after the event start date.

Impacts of late timing for notification and confirmation of funding

When asked about the main challenges in implementing MTEP activities, almost all Summer Marquee Tourism Events and 2009 Tier One recipients indicated that the late timing of their notification and confirmation of funding was a major challenge. This negatively impacted the recipients' ability to plan and implement activities as intended. Specifically, recipients noted the following impacts in interviews and final reports:

• Increased risk—Many recipients noted that, because the activities funded under MTEP were new or significantly enhanced activities, they could not wait for notification of funding before they started some of the detailed planning for these activities. Furthermore, in many cases, events had to implement plans following project approval, but did not have the cash flow as their Contribution Agreements were not yet signed.
• Inefficient spending—Late confirmations of funding inhibited the ability of some events to secure more competitive prices for advertising and media purchases.
• Inability to carry out all planned activities—In many cases, performers were no longer available or affordable and deadlines for securing ads in publications or media purchases were missed. In some cases, not all approved funds could be used. Many recipients noted that last minute changes to planned activities, along with the inability to complete all activities, limited the potential impact of projects on tourist reach.
• Complicated logistics—Given tight timelines, the logistical aspects of developing and implementing a totally new product were more difficult than expected.

Tight timelines prevented recipients from implementing some activities as planned. This resulted in modifications to recipients' Statements of Work (SOWs). As well, in some cases, the events were not able to spend all of the approved funds. Analysis of project expenditures for 2009–2010 show that while a total of $48.7 million^{Footnote 5} was approved in the initial Contribution Agreements, the actual project expenditures were $43 million^{Footnote 6}, approximately $5.7 million less than originally planned.

3.2.1.2 Overall efficiency of program delivery

The assessment of program efficiency included an analysis of the program controls, operational processes and work methods used to achieve timeliness throughout all stages of program delivery. The focus was on identifying any changes in efficiency in year two of the program.

As MTEP was designed in an extremely short timeframe, the program began while processes were still being put in place. Many staff members were new to contribution programs and had to be trained at each step in the project cycle. An evaluation of a similar contribution program^{Footnote 7} noted that it took two years for their program to reach a "steady state", both in terms of project volume and operational procedures.

In year two of the program, an application deadline of January 8^th, 2010, allowed for more time to process project proposals and approve funding. Further, overall program efficiency increased as the recipients became more familiar with the program and Departmental staff became more experienced with respect to MTEP processes and procedures. The program was also able to streamline certain delivery processes and make changes in program delivery tools that resulted in increased timeliness and cost-efficiency.

Program processes that were streamlined in year two include:

Funding announcements—The timing of funding announcements with respect to event start dates improved significantly. All announcements for project funding were made on the same day (May 7, 2010). This resulted in all but four projects receiving notification of funding approval prior to the event start dates. However, a time lapse remained between the decision to fund a project and the notification date.

Contribution Agreements—Improvements were also made to the timing of signed Contribution Agreements. Earlier funding announcements made it possible for the program to begin negotiations sooner with events. This enabled the process of finalizing Contribution Agreements with recipients to proceed more quickly, and decreased the need for subsequent amendments, saving time on both the part of event officials and program staff. However, while these improvements increased the timeliness of program delivery, there were still 25% of events (12 of 47) that did not have a signed Contribution Agreement by the event start date in 2010–11. In many cases this was due to early event start dates, as well as the need for recipients to complete their audited financial statements^{Footnote 8} from the previous year.

Claims processing—The speed of the claims process was identified as a challenge in year one by both recipients and program staff. This was mostly due to the labour intensive nature of processing a large volume of claims (recipients submitted over 300 claims involving approximately 22,000 expense transactions). The major improvement in claims submissions for year two was that forms detailing expenses were submitted electronically versus in paper. This allowed claims officers to upload the claims directly into the MTEP database and process them more efficiently.

Program tools—The program improved the application form, guide, and templates in year two. For instance, greater clarity around "in-kind" contributions and "incrementality" was provided in program guidelines. A standard format for interim progress reports as well as an electronic detailed cost form was developed, simplifying reporting procedures for recipients and data management for program staff.

3.2.1.3 Planned versus actual operating costs

Efforts were made to compare MTEP's ratio of operating costs to total program costs with similar contribution programs for benchmarking purposes, but comparable data was either unavailable or the program context and characteristics were too different to allow for a meaningful comparison. As a result, the assessment of the program's cost-efficiency was restricted to an analysis of planned versus actual operating costs.

MTEP was allocated operating costs of $4 million (4% of total program costs); $2.4 million in the first year and $1.6 million in the second. It was anticipated that MTEP would use 15 full time equivalent employees (FTEs) to deliver the program in the first year and reduce that number to 10 in the second year.

The actual operating costs of MTEP were higher than planned. Total operating expenditures for the program are estimated to be $5.4 million. In year one, the program ran a deficit of close to $200,000, spending 8% more than planned on operating expenses. In year two, the program ran a deficit of $652,000, spending 41% more than planned. In addition, while no resources were allocated for the closing of the program in 2011–12, operating costs are estimated to be $550,000 and will be absorbed by SBTB.

Increased spending in year one was seen as necessary in order to respond to the volume of applications received and to deliver MTEP funding within tight timelines; 165 applications were received as of the May 2009 application deadline, many of which were for summer events. There was pressure to quickly assess applications to determine whether they met all eligibility requirements and to finalize Contribution Agreements once the projects were approved. Salary expenditures were higher than expected given the requirement for extensive overtime. In addition, given gaps in staffing, the program relied on consultants to ensure timelines were respected.

In the second year of the program, spending was higher largely due to the number of staff employed. While it was estimated that the program would require a higher level of operating costs in year one for the start up of the program and less in year two, both years of the program were equally challenging and required a steady level of staffing. The actual number of FTEs used to deliver the program was 15.5 in the first year and 15.7 in the second^{Footnote 9}. This was due to two factors. Demand for the program in year two remained high, and staff members were processing a large volume of claims from the year one projects at the same time as they were negotiating Contribution Agreements for year two.

However, despite higher than planned operating expenditures, MTEP's actual operating costs remained at just under 6% of total program costs^{Footnote 10}. This was a relatively low percentage, as in general, Grants and Contributions programs have operating costs of between 8-15% of total program costs.

3.2 Performance (continued)

3.2.2 How did MTEP supported projects stimulate improvements to the quality of marquee tourism events? Did these improvements contribute to the achievement of short-term and intermediate program outcomes?

Given the timing of the evaluation, the analysis of how MTEP supported projects enhanced marquee tourism events focused on activities completed in year one of the program.

In 2009–2010, actual MTEP support for eligible activities totalled $43 million^{Footnote 11}. Eligible expenses under MTEP were restricted to incremental costs related to new activities or enhancements to existing activities that would not have been incurred in the absence of MTEP funding.

Recipients used MTEP funds to support increased spending on the following types of new or enhanced activities:

• marketing and promotion—such as enhancing websites, expanding international media coverage and attracting international journalists to events;
• product development—such as adding new activities or services to events;
• programming—such as the addition of higher calibre artists to events;
• capital and fixed assets—such as the addition of new venues^{Footnote 12};
• equipment—such as interactive display structures; as well as
• audit and reporting requirements.

Figure three below presents a summary of actual 2009–2010 project expenditures by activity category^{Footnote 13}.

Figure 3: Three: Summary of actual 2009–2010 MTEP project expenditures by activity category

While activities were completed in all of categories of eligible funding, the majority of funds were used to support new or enhanced programming (39%) and new or enhanced marketing and promotional activities (35%).

A detailed analysis of activities completed by Summer Marquee Tourism Events and 2009 Tier One recipients was conducted through interviews and a review of final reports. Illustrative examples of how funding was used to support activities across different categories can be found in the following two examples:

2009 Vancouver International Jazz Festival—The Vancouver International Jazz Festival (VIJF) is a ten day multi-venue music festival presenting over 1800 artists and 400 concerts in over 40 venues across Vancouver. MTEP funding enabled the festival to enhance the overall quality of the event, through the addition of a high profile concert, the "Pink Martini and Orchestra". The reach of the festival was expanded across Canada and internationally, through a comprehensive marketing and promotion strategy. Overnight tour packages were created, and in the process a relationship with tour operators was built. Enhancements to online search engines allowed greater targeting of marketing efforts to potential audiences and priority markets. Social media and mobile marketing applications were developed, including the creation of the VanJazzFest iPhone application.

2009 Montreal High Lights Festival – The Montreal High Lights Festival is a multidisciplinary event that brings together a diverse array of artistic performers and the digital arts. Programming includes two flagship events: the High Lights Event, a free outdoor production combining multimedia presentations, circus arts and music; and the All-Nighter, which presents some 200 activities, most of them free, during a night of exhibitions, shows and workshops. Funding from MTEP allowed the event to add new and enhanced programming, including a series of dining and culinary activities, music and dance performances, pyrotechnics, acrobatics and other new attractions. Additions included the installation of a new multimedia event, L'Ambiosphère; a series of musical performances from tropical countries, Caliente; and the development of new family event, a giant labyrinth. The festival also expanded the scope and reach of its marketing and promotional efforts. Media coverage was expanded through inviting foreign journalists to cover the event, and promotional efforts were increased through print and online advertising, an improved website, and participation in tourism trade shows

While it was difficult for most recipients to provide evidence on the extent to which MTEP funding directly contributed to intended program outcomes, the Summer Marquee Tourism Events and 2009 Tier One recipients indicated during interviews that the activities they completed with MTEP funding aligned with the intended outcomes of the program. Examples provided included:

• increasing event attendance and attracting more domestic and international tourists by expanding the reach of events with new marketing and promotional initiatives, along with enhanced programming and product development;
• contributing to the events' ability to attract additional revenues through the establishment of new partnerships, cash and in-kind contributions; and
• alleviating poor economic conditions through increased spending and attracting more visitors to local regions.

As well, the majority of project recipients interviewed noted that MTEP funding was critical to the success of their events in 2009 and created other positive benefits. While MTEP support directly enhanced recipients' ability to add new activities or elements to their events, attract "big names" and expand marketing and outreach activities, recipients noted a wide range of other benefits, including:

• helping to increase the prestige or profile of the event;
• developing new relationships, strengthening alliances, and attracting new sponsors;
• helping to make their events more competitive internationally;
• allowing them to take risks in testing new mediums and piloting new activities; and
• establishing benchmarks through the economic impact surveys.

Based on interviews, most project recipients indicated that MTEP-supported activities continued to have a positive impact on attendance in 2010. This was attributed to increased advertising and media coverage, enhanced websites and marketing tools, new tourism partnerships, increased access to international journalists, as well as an increase in the profile and overall awareness of the supported events.

3.2.3 Did the program achieve its intermediate program outcomes?

MTEP's intermediate program outcomes included:

• sustained or increased revenues generated by supported events;
• sustained or increased number of tourists attending the supported events; and
• sustained or increased tourism related spending in the local regions for the duration of the supported events.

The analysis of the change in event revenues, tourist attendance, and tourism related spending relied on data provided by recipients through applications, economic impact studies, final reports and audited financial statements.

As discussed in Section 2.5 (Data Limitations), due to a lack of comparable baseline data, it was not possible to comprehensively assess the extent to which the program achieved its intermediate outcomes. For all MTEP funded projects, eligible expenses and MTEP payments were recorded in detail by program officials. However, for event revenues, third party audited financial statements were prepared using different reporting presentations. For tourism related data, the 2008 baseline data provided by many recipients was not comparable to data provided from 2009. This was largely due to different methodologies and definitions used to collect the data.

3.2.4 Were there unintended impacts, positive or negative, that can be attributed to the program?

The Summer Marquee Tourism Events and 2009 Tier One recipients identified a number of positive benefits that can be attributed to the program. These benefits have largely been reported on in previous sections and include:

• an increased profile of the marquee event in the marketplace;
• the reduction of risk for new or enhanced activities;
• the development of new relationships, strengthened alliances, and the attraction of new sponsors;
• the opportunity to test new mediums and pilot new activities; and
• the ability to use the economic impact study as a benchmark for data collection in future years.

4.0 Conclusions and Lessons Learned

Conclusions

Regarding the relevance and performance of the program, the following conclusions can be reached.

Relevance:

• By providing targeted financial assistance to marquee tourism events, the program responded to the need for an immediate economic stimulus to the tourism sector.
• The objective of the program aligns with federal government priorities and is consistent with the mandate and strategic objectives of Industry Canada.

Performance:

• Despite the significant challenge of designing and delivering a new contribution program under tight timelines, the Department succeeded in implementing MTEP within the required timeframe.
• At a project level, many year one recipients did not receive notification and confirmation of funding with sufficient time to plan and implement all of their activities as intended. As a result, the potential impact of the program was likely limited in the first year.
• Earlier application deadlines in year two of the program allowed for more time to process project proposals and approve funding. MTEP streamlined several processes to improve the timeliness of program delivery and overall efficiency of the program in the second year.
• Actual operating costs were higher than planned, largely due to increased salary expenditures needed to respond to the high demand for the program, to deliver MTEP funding within tight timelines, and to maintain program capacity in year two. In addition, as MTEP was a two year EAP initiative, no funding was allocated to close the program in 2011–2012.
• Despite higher than planned expenditures, MTEP's actual operating costs remained at just under 6% of total program costs—a relatively low ratio for a Grants and Contributions program.
• MTEP funding stimulated improvements to the quality of funded marquee tourism events through increased spending on incremental activities.
• MTEP project activities aligned with the intended outcomes of the program and created positive benefits for recipients.

Lessons Learned

As MTEP sunsetted at the end of March 2011, and there are no plans for Industry Canada to deliver similar programming in the tourism sector in the near future, the lessons learned were identified with the intent of informing the design and delivery of programs with similar contexts—new contribution programs, designed and delivered with short timelines.

The lessons learned include:

Lesson Learned 1: Applicants need time to comprehensively develop project proposals and require timely notification of funding to ensure that projects can be implemented successfully.

Lesson Learned 2: As part of the program design, take the time necessary to develop a suitable data collection system to support ongoing program performance reporting and evaluation needs.

Lesson Learned 3: Temporary programs may result in staffing problems and gaps, which could limit the efficiency of program delivery. Quick approvals for assignments, the creation of opportunities to build skills and gain valuable career experience, along with departmental support to transition staff as a program closes can help address these challenges.