Final Evaluation of the Language Industry Initiative

Final report

May 2008

Tabled and Approved at DEC on June 9, 2008

This publication is available upon request in accessible formats. Contact:

Multimedia Services Section
Communications and Marketing Branch
Industry Canada
Room 264D, West Tower 235 Queen Street
Ottawa ON K1A 0H5

Tel.: 613-948-1554
Fax: 613-947-7155
Email: ic.cmb-creative.ic@canada.ca

Permission to Reproduce
Except as otherwise specifically noted, the information in this publication may be reproduced, in part or in whole and by any means, without charge or further permission from Industry Canada, provided that due diligence is exercised in ensuring the accuracy of the information reproduced; that Industry Canada is identified as the source institution; and that the reproduction is not represented as an official version of the information reproduced, nor as having been made in affiliation with, or with the endorsement of, Industry Canada.

For permission to reproduce the information in this publication for commercial redistribution, please email: copyright.droitdauteur@pwgsc.gc.ca

Cat. No.Iu4-138/2008E-PDF

ISBN 978-1-100-10136-1
60486

Aussi offert en français sous le titre Évaluation finale de l'Initiative de l'industrie de la langue.

Translated version. In instances of inconsistency, the French shall prevail.

Table of contents

• Executive Summary
• 1.0 Background
  • 1.1 Profile of the Canadian language industry
  • 1.2 Initiative profile
  • 1.3 Logic model
• 2.0 Methodology/approach
  • 2.1 Final evaluation objectives
  • 2.2 Evaluation questions
  • 2.3 Methodological approach
  • 2.4 Methodology limitations
• 3.0 Results
  • 3.1 Success
  • 3.2 Cost-effectiveness
• 4.0 Conclusions
• The Language Industry Initiative or
  PDF version (6.3 KB, 1 page)

Annexes (Separate document)

(Note: Annexes are available via an Access to Information)

Appendix A Mid-term Evaluation Summary
Appendix B Evaluation Committee Members
Appendix C List of Interviewees
Appendix D List of Documents Consulted
Appendix E Evaluation Instruments
Appendix F Case Study Summaries
Appendix G Description of Comparator Programs
Appendix H Detailed Survey Findings

Note: Appendices are available via an Access to Information Request.

Executive Summary

The language industry is made up of three sectors: translation, language instruction and language technologies. According to The Conference Board of Canada, the language industry in Canada contributed $2.7 billion to gross domestic product (GDP) in 2004. The 2003 Action Plan for Official Languages (APOL) predicted that the demand for language services would increase noticeably over the coming years and the language industry would be unable to meet this increased demand. The purpose of the Language Industry Initiative (LII) was therefore to improve the capacity of the language industry to deliver its products and services in Canada and abroad. The LII had two key objectives:

• to help the language industry establish strategic directions, strengthen links among partners, improve its capability to respond to the increasing demand for linguistic services and products, develop the language industry and give it greater visibility; and
• to establish a sustainable succession plan following the implementation of the APOL.

The key activities of the LII were divided into two components: 1) building a Canadian language industry network, in the form of support for the Language Industry Association (AILIA); and 2) marketing and positioning, which included a contribution program for language industry companies and research and branding activities carried out by the Department.

This final evaluation of the LII covers the period from 2003 to 2007. The purpose of the evaluation was to assess the results and effectiveness of the Initiative. The evaluation included four lines of evidence: in-depth interviews (n=16), a documentation review, case studies (n=4) and a survey of AILIA members as well as companies that had received contributions under the Language Industry Program (LIP) (n=109).

Outcomes

AILIA success in positioning itself as a national language industry association

Overall, the AILIA has not yet fully succeeded in branding itself as a national association. According to the results of the evaluation, the association has yet to attract a critical mass of members, and the satisfaction rates with AILIA services are below what is expected for this type of association. The geographic distribution of members also indicates that language companies in Western Canada, especially language schools, are underrepresented in the AILIA. The AILIA has not succeeded in achieving its recruitment objectives. The association also depends largely on public funds for its survival, and its long-term survival is not assured under its current structure. Nevertheless, the Association has made major progress in the past year.

AILIA and LII contribution in improving industry capacity

By and large, the LII has had a positive impact on the ability of the industry to meet demand. The LIP had a positive impact on language schools, which were very successful in attracting an international clientele. Translation firms and language technology companies indicated that they were also satisfied with the LIP and that they used the program to attract new clients. However, the AILIA has had a marginal impact on professional and corporate development in general. In terms of human resource capacity, a minority of companies seem to have difficulty recruiting workers.

Success of the LII in raising awareness of business opportunities

Results indicate that the industry—especially translation firms—values the market studies and reports developed by the LII. Trade fairs and missions are also appreciated, mainly by the language schools, which participate in these events in order to attract many foreign students.

LII contribution in increasing the awareness of industry products and services among potential clients in other industrial sectors in Canada and abroad

Barring a few exceptions, most of the respondents to the in-depth interviews were unsure whether the Initiative helped to increase awareness of industry products and services among potential clients in other industrial sectors in Canada and abroad. No single industry representative could confirm that the LII helped them establish contact with clients outside their traditional markets.

Contribution to creating synergies between language industry organizations

The results indicate that, generally, the LIP contributed to building links between beneficiaries and other partners, especially language training and technology companies. These links were established during trade missions, participation at trade fairs, etc. The AILIA also established links with other institutions; however, a small majority of its members believes that the association helped them establish new contacts with people in their industry.

Cost-effectiveness analysis

The evaluation team examined program cost effectiveness and efficiency. A systematic comparison of the costs incurred in managing the LII contribution agreements with those of two other programs indicates that LII management costs are very high, i.e. greater than 30% of the value of contributions. This can be explained in part by the small scope of the program and the value of individual contributions. However, forecast revenues generated by projects funded by the LIP, estimated at over $6 million, tend to show that the short-term financial impact of the LIP exceeds program costs, at least those related to contribution agreements.

The Language Industry Initiative has sunset on March 31, 2008. As a result, the evaluation report does not contain any recommendations, and a management action plan from the program was not required. The management response indicated that the program concurs with the findings and conclusions of the evaluation.

1.0 Background

This document presents the results of a final evaluation of the Language Industry Initiative (LII). The purpose of the evaluation is to assess the results and effectiveness of the LII. The evaluation meets current policy requirements pertaining to program evaluation. This first chapter presents the background to the study. Subsequent chapters describe the methodological approach, results and conclusions of the study.

1.1 Profile of the Canadian language industry

The language industry is a recent concept stemming from a desire to bring together professional sectors that traditionally operated independently. According to The Conference Board of Canada,^{Footnote 1} the contribution of the Canadian language industry to GDP is minimal, i.e. $2.7 billion or 0.2% of total GDP in 2004. Experts estimate that the language industry will grow by over 15% per year over the next five to ten years.^{Footnote 2} The industry employs 51,700 people (directly and indirectly) across Canada. With over 2,000 companies and 30,000 specialists, this industry has different activity sectors grouped around three axes: translation, language training and language technologies.

1.1.1 Translation

As a sector, translation includes, among others, firms and professionals in translation, oral and sign language interpretation, and terminology. According to the National Occupational Classification (NOC), these professionals work in various types of organizations, as described below.

Canada has about 800 firms whose principal activity is translation and interpretation. They mainly serve firms and public or government establishments. These firms employ an average of 6.2 workers, and 76.9% of translation companies have fewer than five employees.

Although Census 2001^{Footnote 3} data indicate that over 13,000 people are employed in the translation sector, only a little more than half of translators, terminologists and interpreters work in companies in this sector, while 47% work for firms in other sectors. Approximately 70% of people in the sector work full-time and 38% are self-employed.

Almost 80% of these professionals are concentrated in Quebec and Ontario. The titles of certified translator, certified interpreter and certified terminologist are legally recognized in four Canadian provinces.

1.1.2 Language training

The language training sector includes language instruction, online language training and language technology training. According to the North American Industry Classification System (2002), the classification 611630 – Language schools comprises establishments primarily engaged in foreign language instruction. The courses provided by these establishments range from conversational courses for personal development to intensive training courses intended to further one's education or career.

Private language schools provide their services primarily to individuals. Overall, 64% of Canadian language schools target foreign students. According to a 2004 survey by the Canadian Association of Private Language Schools (CAPLS), students of foreign language schools mainly come from Asia (58%) and Latin America (24%). Asked about their future plans, 86% of students wanted to continue their education or profession in their country of origin, whereas 4% were considering registering at a Canadian university or college. The 2006 Survey of Language Industry Companies also indicates that 52% of private language schools provide programs and courses lasting six months or less for foreign students.

The 2005 Industry Canada report on career choices indicates that, of the 526 schools surveyed, 57% are private. From 1999 to 2002, the number of language schools increased by 15% per year. Ontario and Quebec each account for 30% of language schools, while British Columbia has 23%.

The national average number of employees is 21.5, whereas 37% of companies in this sector have 20 or more employees. Contrary to the translation sector, the proportion of self-employed workers in language training is low, but the percentage of part-time work is higher. According to the 2006 Industry Canada survey, 38.5% of employees work full-time and 56.9% part-time.

According to Statistics Canada, private language schools had approximately $240 million in revenues in 2004, whereas public language schools earned over $165 million. There were approximately 11,200 direct jobs in this sector in 2004. The sector is represented by a number of associations, the largest of which are:

• The Canada Language Council (CLC) is a national association representing both public and private institutions that provide intensive French and English training.
• CAPLS is currently the largest association of French and English private language schools in Canada. Established in 1997 with 20 members, the association today represents more than 60 language schools in over 100 educational establishments across the country.

In the spring of 2008, the Canadian Association of Private Language Schools and the Canada Language Council will merge to create Languages Canada. Its mission will be to promote quality, accredited English and French language training in Canada, both nationally and internationally.

1.1.3 Language technologies

This last sector includes a variety of technology companies involved in automated translation, translation tools, multilingual word processing, speech processing (voice recognition, biometric voice recognition, text-to-speech conversion), management of textual information (Customer Relationship Management (CRM), knowledge management, content management) and training tools. Given that this sector is closely integrated with other technology sectors, including computer science, it is more difficult to obtain statistics. Furthermore, according to the Economic Assessment of the Canadian Language Industry, workers in this sector have little or no identification with the language industry, but rather with computer services.

According to a report published in 2005,^{Footnote 4} this sector has approximately 180 companies and 40 research centres; however, no study to date provides a professional profile of workers in this sector. In 2004, Statistics Canada carried out a survey and attempted to obtain data to create a profile. The results were inconclusive, although Statistics Canada did reach some conclusions. The number of companies in the language technologies sector was very low, and their activities in this field represented only a small proportion of their overall activities. Nevertheless, it seems that the demand for language technologies is growing. Another survey, this one published in 2006,^{Footnote 5} indicates very low research and development (R&D) in language technologies in translation companies (5.3%) and language schools (1%).

1.2 Initiative profile

The Language Industry Initiative (LII) was launched in 2003 for a five-year period, until March 31, 2008. It was based on two main initiatives, Canada's Innovation Strategy and the Action Plan for Official Languages (APOL).^{Footnote 6} The language industry component of APOL primarily deals with the main industry challenges, as identified at a symposium sponsored by Industry Canada in 2002. At the time, demand for language services was expected to increase noticeably and the language industry was considered to not have the capacity to meet this heightened demand. Four major industry challenges were identified:

• industry fragmentation: Most Canadian firms in the language industry sectors are micro-enterprises that are relatively isolated from each other. They have few resources to invest in R&D, marketing, skills development, recruitment and technology.
• lack of visibility: The Canadian language industry is not very well known, either in Canada or abroad. Due to fragmentation and a lack of visibility, it is difficult to highlight the success of activities or products and to influence the adoption of government policies.
• shortage of human resources: In Canada, the demand for specialized language industry workers (especially translators) exceeds supply. Each year, the industry notes that there are too few graduates to meet recruitment needs.
• lack of R&D and language technology skills: Because of industry, its sectors have neither the critical mass nor the strategic planning and leadership to benefit from R&D investments. The private sector lacks the necessary funds and the government has not addressed the needs of this sector. According to the APOL, this challenge is more the responsibility of the National Research Council of Canada (NRC).

The purpose of the LII is to increase the capability of the language industry to provide its products and services in Canada and abroad. The LII covers the three industry sectors: translation, language training and language technologies.

In total, $20 million was allocated to the Initiative for the period 2003–2004 to 2007–2008. Of this amount, $10 million was allocated to the National Research Council of Canada (NRC) to improve research and development and competencies in the language technologies sector. The remaining $10 million was allocated to Industry Canada to focus on the three other key challenges identified above. The Industry Canada component of the LII included two main objectives:

• to help the language industry establish strategic directions, strengthen links with partners, improve its ability to respond to the increased demand for linguistic services and products, develop the language industry and give it greater visibility; and
• to put in place a sustainable succession plan for human resources following the implementation of the Action Plan for Official Languages.

The key LII activities were divided into two components: 1) the establishment of a Canadian language industry network and 2) marketing and positioning. The following table presents the financial framework for the Initiative.

Two contribution agreements were also signed with the Language Technologies Research Centre (LTRC), for a total commitment of $450,000, which came from unused LIP funds and a transfer of funds from the NRC. These agreements helped to market and promote the industry.

The table below presents the actual expenditures related to contributions and the LII operating and maintenance (O&M) budget.

1.2.1 Canadian Network of Language Industries

The Canadian network of language industries was designed to build one or more viable and sustainable language industry associations, facilitate networking, represent the interests of all sectors of the language industry and support efforts to recruit new talent. The $5 million allocated to this component included a non-refundable contribution of $3 million over five years to support industry associations and $2 million for salaries and operating expenditures of the Initiative.

The Language Industry Association (AILIA) was established in 2003 to represent the three industry sectors (i.e. translation, language training and language technologies). Its mission was to bring together key players, become the voice of the language industry, and make Canada a world leader in the field. To achieve its mission, the AILIA has adopted the following objectives:

• Promote the interests of the language industry.
• Act as a common base for action for all aspects of the industry.
• Raise the visibility of the sector.
• Implement initiatives to overcome issues.
• Develop common human resources strategies for sector firms.
• Promote innovation and R&D.
• Ensure networking between industry firms and other public and private sector partners.
• Become a key industry voice when dealing with government.
• Encourage the emergence of alliances and appropriate projects to further the industry.

The association is governed by a Board of Directors and an Executive Committee. Together they are responsible for and direct the actions of Canada's language industry network, through the close co-operation of various types of stakeholders in the sector, be they decision makers, business owners, heads of professional and business associations, or representatives from the public sector and education.^{Footnote 7}

The AILIA has translation, human resources, communication and technology committees. The key AILIA activities are intended to help the industry tackle these four main challenges. Among its priorities are the following activities:

• participation at conferences, exhibits and trade fairs in Canada, as well as trade fairs and missions abroad;
• annual recruitment campaigns, annual general meetings, production of promotional material;
• succession planning, which includes developing a strategy, producing material on occupations in the industry, participating in job fairs and creating an "Employment Opportunities" section on the AILIA Web site;
• participation in international conferences and the establishment of a "Year of Languages" to promote branding;
• development of the technological roadmap, which will contribute to R&D through an analysis of language industry trends and technological priorities; and
• development of a national standard^{Footnote 8} for translation services that will provide industry clients with access to recognized quality services and increase the market share of firms certified at the national and international levels.

Under the contributions program, the Association has received a total of $2,523,848 over the past 5 years to fund its activities, with an additional $76,888 anticipated to be provided in 2007–2008. Its membership breaks down as follows:

1.2.2 Marketing and positioning

The purpose of the second component, marketing and positioning, was to improve the image of the language industry as well as its products and services in Canada and abroad, increase its visibility and support efforts to recruit new talent. There are two components.

1.2.2.1 Marketing and branding

Activities completed under this component include surveys, market studies, the development of information products, and trade missions and fairs to showcase industry capability. In 2007, seven market studies were carried out, and trade missions were held in China, Brazil and Spain. According to program documentation, over 75 events and presentations were held. Although it is difficult to obtain exact attendance figures, an estimate based on documentation suggests that over 8,500 people attended these events.

As much as $600,000 ($3 million over five years) was available for these activities in the form of O&M funds.

1.2.2.2 Language Industry Program

The Language Industry Program (LIP) was created to work in collaboration with private-sector firms to build a strong, competitive industry. The contribution program therefore focuses on assistance for firms to help them undertake marketing and branding activities in Canada and abroad. Under the program, funding was limited to 50% of eligible costs, up to $10,000 annually per company. In total, $400,000 per year ($2 million over five years) was available for the three key industry sectors (i.e. translation services, language training and language technologies).

From 2003 to 2008, the LIP funded 219 projects across Canada in the three industry sectors. Table 5 presents a breakdown of projects by sector and region. According to the table, 76% of beneficiary firms are in the language training sector and 54% are from Ontario and Quebec.

The LIP received 327 contribution applications, as indicated in Table 6 below. The overall financing rate is 67% (219 approved applications), while the average contribution is $8,808.

The number of proposals received and the number approved differs due to withdrawals by applicants or rejections by the program review committee.

Contributions supported 123 firms; some of them (44%) received more than one contribution:

• firms that received one contribution: 69
• firms that received two contributions: 22
• firms that received three contributions: 22
• firms that received four contributions: 10

As presented in Table 7, activities related to the LIP primarily include participation in trade fairs and missions, the development of marketing plans and communication strategies, and the creation of promotional tools.

1.3 Logic model

In August 2003, a Results-based Management and Accountability Framework (RMAF) was developed for the Initiative.

Logic model

[Description of Logic model]

Footnotes

1. ¹ Economic Assessment of the Canadian Language Industry, Conference Board of Canada, March 2007. (back to footnote reference 1)
2. ² The Language Industry – A Source of Interesting Career Choices, Industry Canada, January 2005. (back to footnote reference 2)
3. ³ Census 2006 data were not available. (back to footnote reference 3)
4. ⁴ The Language Industry – A Source of Interesting Career Choices, Industry Canada, January 2005. (back to footnote reference 4)
5. ⁵ Survey of Language Industry Companies in Canada – Translation, Interpretation and Language Training, Industry Canada, 2006. (back to footnote reference 5)
6. ⁶ The Next Act: New Momentum for Canada's Linguistic Duality – Action Plan for Official Languages, March 2003. (back to footnote reference 6)
7. ⁷ AILIA website http://www.ailia.ca/ (back to footnote reference 7)
8. ⁸ Industry Canada and the AILIA have signed a memorandum of understanding with the Canadian General Standards Board (CGSB) to develop this standard. (back to footnote reference 8)

2.0 Methodology/approach

2.1 Final evaluation objectives

The objective of this project was to conduct a final evaluation of the LII for the period from 2003 to 2007. The purpose of the evaluation was to assess the outcomes and effectiveness of the LII. The evaluation meets the requirements of the Treasury Board Secretariat based on the current policies for program evaluation and transfer payments.

The evaluation was carried out by Goss Gilroy Inc.. for the Audit and Evaluation Branch (AEB) of Industry Canada. It was managed by an AEB evaluation manager. A member of the AEB evaluation team helped collect and analyze data during the project. An Evaluation Steering Committee made up of Industry Canada representatives and key stakeholders provided input and guidance during the course of the evaluation.

2.2 Evaluation questions

The evaluation seeks to answer the questions below. As the LII was scheduled to sunset on March 31, 2008, the issue of relevance, covered in the 2006 mid-term evaluation, was not included in this evaluation (see Appendix A for a summary of the mid-term evaluation results), and this report is limited to presenting conclusions based on the findings of the evaluation.

Evaluation questions

Success

1. Has the Language Industry Association (AILIA) succeeded in positioning itself as a national language industry association among both government and industry stakeholders?
2. Have the AILIA and the LII helped to improve industry capability to meet the demand for its products and services?
3. Has the LII contributed to greater awareness among industry stakeholders of national and international commercial opportunities for language industry products and services? If so, how?
4. Has the LII contributed to greater awareness of industry products and services among potential customers in other industrial sectors in Canada and abroad?
5. Has the LII helped to create synergies between organizations in the language industry sector? If so, how?
6. Did the LII have any unintended impacts?

Cost-effectiveness

7. How cost-effectiveness were the major components of the LII (contribution to the AILIA, steering committee and related secretariat services, language industry contributions program, marketing and positioning activities)?

Lessons

8. What lessons can be learned from the program development and implementation?

2.3 Methodological approach

The evaluation relied on both qualitative and quantitative methods. The sources of information listed below were used in the evaluation.

2.3.1 In-depth interviews

In-depth interviews, lasting between 30 and 90 minutes, were conducted for evaluation purposes. The Evaluation Steering Committee provided the list of interviewees (see Appendix C). This non-random selection was deemed satisfactory because the evaluation would also rely upon a survey of AILIA members and contribution beneficiaries. Sixteen people were interviewed:

• seven program representatives/managers (including two group interviews);
• three advisory committee members;
• three representatives of associations and organizations;
• three funding recipients; and
• two external stakeholders (provincial organizations, sponsors, etc.).

The interviews were conducted in person, except with people outside of the National Capital Region who were contacted by telephone. All interviews were conducted in the respondent's official language of choice.

2.3.2 Documentation and secondary data review

Many documents were reviewed for the evaluation, including various program budgets, the Results-based Management and Accountability Framework, various research studies and reports, the LII Formative Evaluation, meeting minutes, the AILIA business plans, etc. (see Appendix D for the complete list). These documents were used to establish the context for the Initiative and answer the evaluation questions. In addition, the documentation review helped in understanding the language industry sectors in Canada and the challenges facing this industry.

2.3.3 Case studies

Case studies were carried out to obtain more in-depth examples of firms that have benefited from the program. The goal was to understand how and to what extent the program helped firms to capture markets and expand their activities and workforce. The cases were proposed by the managers of the Initiative. This approach was deemed acceptable in light of the other methods, such as the survey aimed at providing a more representative view of LII outcomes.

Four case studies were carried out on organizations that received a contribution through the program. Three of the studies concerned organizations in the key sectors, i.e. translation services, language training and language technologies. A case study was also conducted on the AILIA, which helped in better understanding the development of the association and its services, successes, challenges and future.

Discussions with program representatives helped determine the profile of these cases. Documentation was gathered for the case studies through in-depth interviews and documentation reviews. Quantitative data were collected where possible. From three to four business managers and executives were interviewed in depth for each study (in addition to the other in-depth interviews) using an interview guide that included specific questions for the AILIA. The interviewees were chosen following the recommendations of the project leaders.

Each case is summarized in Appendix F of this report.

2.3.4 Survey of firms and members

Given that the evaluation mainly focuses on the outcomes of the Initiative, the evaluation team collected quantitative information on its scope and impact. To do this, an electronic survey was sent to firms that had received funding through the Language Industry Program and to AILIA members. The list of AILIA members and recipients of a program contribution formed the basis of the survey. To maximize the response rate, some organizations were surveyed over the telephone.

Organizations that submitted an application as part of the Language Industry Program but did not receive funding were not surveyed. These organizations took part in the 2005–2006 mid-term evaluation survey. Because the questions in the final evaluation focus on the success of the program, the evaluation team decided that it would not be useful to survey these organizations a second time.

All of the respondents were asked about the impacts of the Initiative's various measures, i.e. through the AILIA, program contributions received, and program marketing projects. Among other things, the evaluators measured the extent to which firms increased their work force (including self-employed workers) and their market diversification, indicators used to analyze business growth since the start of the Initiative. The survey questions were tailored to the component of the initiative (the AILIA, the contribution program, departmental activities).

Note that for technical reasons, the survey was not designed to accurately measure the economic impacts of the contributions. The mid-term evaluation tried to measure these impacts accurately but it obtained a very low response rate in this regard. In accordance with the contribution agreements, organizations that generated revenues as the result of the contributions had to repay the Department amounts equal to these revenues.^{Footnote 9} Consequently, it is thought that many respondents of the previous survey refused to answer questions related to revenues resulting from their contribution and to the survey in general.

For this evaluation, we tried to maximize the response rate by not asking specific questions about impacts, including on sales. We decided to measure the general trends in business revenues and jobs created as impact indicators. From these answers, we deduced whether the contributions had a positive or negative impact on the firms, while hoping that the other sources of information, including the case studies, would enable us to establish a link between the contributions and any commercial impacts. We also obtained other information, such as the number of students enrolled in language schools as a result of activities funded through contribution agreements. These answers helped us identify impact estimates based on other information sources.

Sampling and response rate

Given that it was an online survey, the electronic questionnaire was sent to all potential respondents, i.e. AILIA members and organizations that received contributions. The evaluation team received three lists of AILIA members (covering three years) and a list of organizations that benefited from the LIP. The table below summarizes the parent populations and the samples obtained:

The total sample was distributed as follows:

• Translation services: 36% of respondents (n=39);
• Language training: 54% of respondents (n=59);
• Language technologies: 19% of respondents (n=21);
• Other: 14% (n=15).

Other respondents included government institutions, consultants and firms that provide generic language services.

Respondents who benefitted from the LIP had the following profile:

• Translation services: 18% of respondents (n=11);
• Language training: 72% of respondents (n=42);
• Language technologies: 20% of respondents (n=12);
• Other: 8% (n=5).

Respondents who were AILIA members had the following profile:

• Translation services: 46% of respondents (n=35);
• Language training: 42% of respondents (n=32);
• Language technologies: 18% of respondents (n=14);
• Other: 18% (n=14).

Many respondents indicated they were in more than one sector, which is why the percentages add up to over 100.

2.3.5 Analysis of cost-effectiveness

Financial data and the survey results were used to analyze cost-effectiveness and efficiency. The analysis compared program costs, the volume of contribution agreements and the program results with those of similar programs. The cost-effectiveness analysis was based on two ratios, i.e. the ratio of administration costs for salaries to the value of contributions (only this ratio was applied to all three of the compared programs); and the ratio of the value of contributions to revenues generated by the contributions. The financial and results information were obtained from program managers and the Department of Canadian Heritage Grants and Contributions Centre of Expertise.

For the purposes of the analysis, the LII was compared with the Trade Routes and the Book Publishing Industry Development programs, both Canadian Heritage initiatives. These two programs were chosen because of their similarities with the LII in that both offer contribution agreements to organizations to take part in trade fairs or events. They also provide support for promotional activities. The Trade Routes program also includes a research component.

The comparison has its limitations, however, given that the programs fund very different industries, as explained below. The programs also have very different budgets for contribution agreements (financial data are provided later in the chapter on results). The LII had a much smaller budget than that of the other two programs.

The major parameters of these programs are presented in Appendix G.

2.4 Methodology limitations

As with any other evaluation study, this one has its methodological limitations.

The in-depth interviews have their limitations in terms of their qualitative nature and limited scope (i.e. 16 respondents, of which only 10 are outside the program).

Owing to the number of people who were reached, the survey complements the other sources of information. Although not everyone in the parent population responded, resulting in a margin of error, the team did manage to achieve a satisfactory response rate by combining survey methods (Internet and telephone). The initial sample also included some obsolete or incorrect data. To overcome this, the team obtained new contact information for a number of organizations through additional research on the Internet and in the telephone book. However, this delayed the project by a few weeks.

The survey also includes rather inaccurate metrics of the financial impact of contributions, for the reasons explained in section 2.3.4.

The cost-effectiveness analysis also has its limitations. The team was unable to obtain product or performance indicators of research activities, or at least comparable indicators for the three programs, to measure the cost-effectiveness ratio or even the cost-efficiency ratio. Although we know how much was invested in research, in terms of both government human resources and research contracts, we cannot compare efficiency on the same basis because some of the research conducted by the various programs differs in nature. The LII produced (or created) over 20 research projects, but this work is diverse and we are uncertain of its impact on the industry. In theory, we could have compared the distribution of documents as an impact indicator of the research projects, but research projects carried out by the LII are mainly distributed electronically, and download statistics for research documents from the Strategis Web site were not available. However, the evaluation team included survey questions on the usefulness of LII market studies and reports, which provided some indication of the impact of these documents on the industry.

The cost-effectiveness analysis was therefore largely based on two ratios, i.e. the administration costs for salaries to the value of contributions (only this ratio was applied to all three of the compared programs); and the value of contributions to revenues generated by the contribution. However, only the LII was analyzed according to the second ratio, assuming the objective is to obtain a ratio of less than one (where generated revenues exceed program costs). We considered these limitations during the results analysis.

Footnotes

1. ⁹ "In the case of the LIP, it seems difficult to obtain information from beneficiaries on the impact of the financial contribution, undoubtedly because it is a contribution that is partly repayable when there are financial benefits" (back to footnote reference 9)

3.0 Results

The following sections describe the evaluation results, in basically the same order as the evaluation questions.

3.1 Success

3.1.1 Has the Language Industry Association (AILIA) succeeded in positioning itself as a national language industry association?

General findings

Although the results show that the AILIA made considerable progress last year, it did not entirely succeed in positioning itself as a national association. The distribution of its members indicates that organizations from Western Canada, particularly language schools, are underrepresented. As well, the AILIA still has work to do to establish a critical mass of members, at least if it wants to remain influential. The overall satisfaction of its members with the AILIA is below expectations, particularly in the language training sector. The AILIA depends heavily on public funds to survive, and its long-term survival is uncertain under its current structure.

The detailed results are described in the following paragraphs by information source (or by method).

In-depth interview results

Most respondents were of the opinion that the AILIA has some recognition, more so in Eastern Canada (particularly in the National Capital Region) than in the West, especially with respect to language schools, where the level of representation in the AILIA is lower than their actual economic weight. According to some respondents, the AILIA has succeeded in providing the language industry with some visibility.

A number of respondents said that the AILIA made considerable progress in the last year, particularly on two major projects. First, its working group on national standards for translation services made significant headway (producing a set of preliminary standards). Second, the AILIA played a leadership role in the standing offer for Public Works and Government Services Canada (PWGSC). The AILIA held several meetings with PWGSC, which was thinking of establishing a standing offer for translation and interpretation services.

Most respondents felt, however, that the AILIA had not entirely succeeded in positioning itself as a national association. According to one industry representative, the AILIA failed to meet expectations, or at least not for this organization; the organization members participated in the annual AILIA conference last year and were disappointed. They had the impression that the AILIA was not well organized and that it lacked leadership. According to a number of respondents, the relationship between the three sectors is unclear and its grouping is somewhat artificial. One respondent pointed out that the AILIA is mainly associated with the language training and translation sectors because, unlike these sectors, the language technologies sector provides products rather than services. Some respondents questioned the relevance of the AILIA for these sectors, given that several professional associations already exist, including the Canada Language Council (CLC), the Canadian Association of Private Language Schools (CAPLS) which, as previously mentioned will merge in the spring of 2008 to create Languages Canada.^{Footnote 10}

Documentation review results

As with the interviews, the documentation review primarily indicates that the AILIA has more members from Eastern Canada. The geographic distribution of its list of members in 2007–2008 confirms that AILIA members are concentrated in Eastern Canada, as shown in Table 9. Table 10 represents the actual distribution of the language industry using data from another source.

A comparison of the two tables reveals the following:

• In terms of training schools, Ontario schools have significantly more AILIA weighting (57%) than within the industry (25%). Conversely, British Columbia has half as many members as it should have, if the goal of AILIA is to represent the entire industry.
• The table reveals similar results, albeit to a lesser extent, for translation firms; Ontario represents 42% of AILIA members, whereas its actual weight within the industry is 30%. British Columbia has half as many AILIA members as its actual representation (i.e. 5% of its members, whereas 9% of Canadian translation firms are located in British Columbia).

According to the 2005 Industry Canada report on career choices, Ontario and Quebec each account for 30% of language schools, while British Columbia has 23%. If these percentages are accurate, the difference between the membership and economic distributions would be less (but significant, nonetheless).

Case study results

The case study results reveal that the AILIA has created a major hub for industry representatives. Through its activities, members have created partnerships, participate in business opportunities, share information and network, thus allowing them to realize they are not isolated.

The AILIA performance reports indicate a number of AILIA business activities, including the following in 2006–2007:

• annual general meeting (with 46 members participating);
• the second Language Industry Showcase in November 2006 with 26 exhibitors; approximately 100 people visited the exhibitors and attended the conference luncheon;
• participation in annual conferences and conventions of Canadian and foreign organizations;
• workshops on export strategies;
• publication of 10 issues of Info-AILIA; and
• publication and distribution of industry brochures to more than 1,000 recipients, including the 500 largest companies in Canada.

AILIA directors admit that it has not always been easy to convince AILIA members of the benefits of bringing the three language industry sectors together despite their different needs. According to the AILIA, each of these sectors is too small to have the same impact as with AILIA. It is therefore necessary to identify what the three sectors have in common.

According to its directors, the AILIA is an effective advocate for its members before the federal government. The AILIA has played an important role in discussions between the language industry and PWGSC on a future standing offer for the translation sector. The AILIA accompanied a PWGSC delegation as part of a pan-Canadian consultation. According to current AILIA directors, this enhanced the visibility of the association.

One of the questions raised by this evaluation was AILIA's longevity, given that the LII has been its main source of funding since it was created. According to data from AILIA annual reports (see table below), government contributions were its main source of revenue between 2003 and 2007.

Due to difficulties in securing government funding, the AILIA had to suspend its activities for a four month period near the end of the third year of operations. This suspension had a negative impact on its ability to maintain and launch projects at a time when it was gearing up its operations.

According to the latest financial data available (2006–2007), over 73% of AILIA revenue comes from Industry Canada, while only 8% is generated through membership fees, sponsorships and activities. Industry Canada funding for the AILIA varied between 73% (2006–2007) and 87% (2003–2004) of AILIA funding. According to a manager of the Initiative, the basic AILIA operating costs total approximately $200,000.

The AILIA successfully obtained funding from other federal sources, but its membership numbers have stagnated in the last two years, which limits revenue from membership fees.

The AILIA instituted a number of measures to recruit members, including two membership drives and the hiring of a recruitment officer; however, this was insufficient. At an AILIA meeting in 2005–2006, the following reasons were given for the failure of AILIA to achieve membership recruiting objectives:

• Membership provided an inadequate concrete return on investment;
• Membership provided an insufficient number of services generating financial spinoffs; and
• The variety of services was limited compared with services offered by other associations or organizations.^{Footnote 11}

However, the AILIA is working on some promising projects, notably the standards projects, which would lead to the certification of firms.^{Footnote 12} This could result in a new source of funding for the association. For AILIA directors, funding remains the main challenge of the organization.

Survey results

The survey conducted for this evaluation sheds some light on the above-mentioned issues and, to some extent, confirms the AILIA analysis of its membership opinions. Some respondents, who were not members of the AILIA, were asked in the survey to explain their non-member status. The results are presented in the figure below. Among non-member respondents, almost half (45%) said that they were members of another association. Close to one-third (32%) of these respondents did not know the AILIA.

Figure 1: Reasons Why Respondents Were Not Members of AILIA

[Description of Figure 1]

According to respondents who gave "others" as a reason, AILIA services were not adapted to their company's needs, were not easily accessible, or seemed irrelevant.

Other survey results provide an idea of AILIA strengths and weaknesses, hinting at other reasons for membership stagnation during the last two years. According to the figure below, although two-thirds of respondents are satisfied with reports and publications, only half were satisfied with the conferences, trade fairs or meetings organized by the association, or with other member services. Fewer than half of respondents were satisfied with AILIA promotional and advocacy activities and its efforts to develop sector standards and good practices. Members of the language training sector were slightly less likely to be satisfied with the AILIA in general (see Appendix H for the survey results by sector).

Figure 2: Level of Satisfaction with AILIA Services

[Description of Figure 2]

By way of comparison, the statistics from another organization, the Canadian Evaluation Society (CES), indicate that these rates are not optimal. In 2003, the CES surveyed its members and the results revealed that over 80% of members found the CES annual conferences useful.^{Footnote 13} At the CES 2007 annual conference, 84% of participants found the conference good or very good overall. Another example is the Canadian Standards Association (CSA), whose satisfaction rates are also relatively high. According to its annual member surveys, its satisfaction rates with the CSA have been approximately 90% since 2001.^{Footnote 14} (The evaluation team conducted research to obtain comparative data for other organizations. It only succeeded in obtaining results for the two above-mentioned organizations.)

3.1.2 Have the AILIA and the LII helped to improve industry capability to meet the demand for its products and services?

General findings

Overall, the LII has had a positive impact on industry capability to meet demand, notably through contribution agreements, which have enabled several firms to attract clients abroad (especially schools) and also in Canada (especially translation firms and language technology firms). However, AILIA services have had a rather limited impact on professional development or even business development, in general.

Only a few firms said that they were unable to recruit staff in the last few years, mainly because they had difficulty finding qualified candidates.

In-depth interview results

As we have mentioned, human resources is one of the four challenges in the language industry. When the Initiative was developed, it was reported that the language industry was suffering from a shortage of qualified staff. The LII and the AILIA were to help increase industry capability to better meet demand. According to the interview results, the LII has indeed helped some firms recruit human resources. The in-depth interviews also revealed that LIP contributions had generally been useful to companies, particularly those interested in hiring human resources from abroad.

Views on the impact of the AILIA on the industry were more varied. One respondent, a representative of a language training centre, told us that AILIA activities had benefited his company. Conversely, another respondent from the language industry and a member of the AILIA had expected the association to provide information on best practices, networking, participation in events (conferences), etc., but they were disappointed.

Case study results

The case studies illustrate how the LIP could help the language industry to develop and meet demand. For example, the translation firm in the case studies successfully used the contribution program to recruit translators in France. The program helped the firm travel to France to participate in Expolangues, a major annual trade fair that brings together approximately 25,000 representatives of the various sectors of the language industry and members of the general public. During the conference, representatives of the firm met with several young graduates, freelance translators and translation firms. The firm was thus able to attract approximately a dozen new translators to Canada, most of who have been trained in computer-assisted translation. According to the firm's management, the new employees have brought new ideas, and some of them will help the firm improve its service offering in different language combinations. It is interesting to note that, according to the representatives of this firm, some of their success was attributable to the pre-travel preparations, on which it worked hard, testing and pre-selecting 50 candidates who were interviewed during Expolangues. The company also established contacts with universities in the months leading up to the event and produced pamphlets highlighting the advantages of a career in translation in Canada.

Another example comes from the technology sector. The firm in question, one of only two of its kind in Canada, designs and manufactures latest-generation language laboratories. Founded in 1984, it has offices in Latin America, Europe, Asia and the Middle East and has approximately 20 employees at its facility in Laval, Quebec. The firm has grown substantially over the last few years, going from 15 to 22 employees in 2007, and has sales of $3.5 million. The LIP contribution enabled this firm to meet with potential distributors and clients in the United States. It also participated in a mission to Chile, which resulted in several digital laboratory equipment projects, including projects with five universities (valued at $125,000) and two government departments. It is estimated that these trips produced considerable benefits—$375,000 to date and a further $200,000 in sales in the next two years.

Documentation review results

As we mentioned, the LII was developed because of a shortage of human resources. Some studies shed light on this issue. According to the Survey of Language Industry Companies in Canada (2006), only 20% of firms in the translation sector indicated they had difficulty recruiting, while the remainder had none (42.4%) or were not recruiting at the time of the survey (37.6%). The greatest difficulty recruiting was found in Quebec, where close to 24% of translation firms were experiencing difficulty. The main difficulty expressed during the survey was finding qualified workers.

In the training sector, 27.8% of firms indicated they were experiencing difficulty recruiting. Quebec firms were more likely to experience difficulty (37.4%), while Ontario was the least affected (16.9%).

Survey results

The survey indicates the extent to which the AILIA and the LIP have helped develop the industry. The survey measured to what extent AILIA members and LIP beneficiaries were able to meet their personnel needs. According to the results, 83% of respondents had been trying to hire staff since 2004 and 77% of these respondents had found all of the staff they sought. Therefore, 14% of total survey respondents did not find the staff they sought. The results indicate that those who did hire recruited eight full-time and five part-time workers. This suggests that if there is a shortage, it has had a long-term effect on only a few firms overall (14%), which reflects the literature.

The survey indicates why companies did not find the staff they sought. Of the respondents who were unsuccessful in their search for personnel (n=19 or 15% of the sample), 84% cited difficulty in finding qualified workers as a reason (particularly language technology companies); 42% indicated difficulty in meeting salary expectations (especially schools); and 21% indicated a lack of time for recruiting.

The AILIA did not contribute much to the development of human resources according to the survey. The results reveal that only 10% of respondents agreed that the AILIA had helped their organization to hire personnel, while 21% agreed that the association had supported the professional development of their company personnel. Only 30% of respondents said that the AILIA had contributed to the overall development of their organization (see the following figure). Translation firms were the least likely to say that the AILIA had contributed to their development. However, of the three sectors, translation firms were also more likely to say that the AILIA had helped them find staff.

Figure 3: The AILIA's Impact on its Members (1)

[Description of Figure 3]

The survey also looked at the effectiveness of AILIA and LIP contributions in helping the industry take advantage of markets. According to the survey results, the AILIA had no substantial impact on the industry in this regard. As the following figure illustrates, few members (12%) agreed that the AILIA contributed to the development of products. A minority of respondents also said that the AILIA helped their organization to find new domestic or foreign clients (18% and 20%, respectively). However, representatives of training companies were more likely to say that the AILIA helped them find clients abroad (26%).

Figure 4: The AILIA's Impact on its Members (2)

[Description of Figure 4]

The following figure suggests that the LIP has had a more significant impact. According to the results, a considerable majority of respondents who received contributions said that these had helped their organizations attract new students (78%), increase the demand for their products (68%) or find new business opportunities or clients in national (75%) or international (73%) markets. A majority of respondents (62%) also said that the contributions helped their organizations develop marketing or communication strategies; a minority of respondents said that the contribution helped them develop new services or products.

Figure 5: Impact on LIP Contributions

[Description of Figure 5]

According to the following figures, most schools (83%) said they had attracted students. Training companies were more likely to say that the contribution allowed them to increase demand, develop new products, develop a marketing or communication strategy, or find new outlets or new international clients. However, technology and translation firms were more likely to say that the contributions allowed them to find new business opportunities or clients in national markets.

For technical reasons, the survey was not designed to measure the exact economic impacts of LIP contributions (as explained in section 2.3). However, the results indicate that most of the companies that were members of the AILIA or that received contributions saw some growth in the last five years: 72% of survey respondents confirmed that their sales figures had increased since 2004; 13% said that there had been no change in their sales, and 15% said that their sales had decreased. Over three-quarters (81%) of the respondents believed that demand for the services and products they offered would increase over the next three years, mainly because of growing demand (n=43).

By way of comparison with the language industry as a whole, in the 2006 Statistics Canada survey, 41% of respondents anticipated an increase, 45% expected stability and 14% forecast a decrease.

Lastly, the current survey measured the number of students attracted as a result of activities undertaken with funds received through contribution agreements. According to the results, contributions allowed organizations to attract an average of 46 students, a figure that is of primary importance in the cost-effectiveness calculation described later in this report.

3.1.3 Has the LII contributed to greater awareness among industry stakeholders of national and international commercial opportunities for language industry products and services? If so, how?

General findings

Overall, the results indicate that most respondents appreciated the market studies and reports produced through the Initiative and found them to be useful for their organization (especially translation firms). The same is true for trade fairs and missions (especially on the part of schools). The LII has produced many market studies that can be found on the Industry Canada Web site. The AILIA also publishes industry information on its Web site. Moreover, the personnel of the Department and a number of LIP contribution recipients have participated in trade missions in a number of countries. The in-depth interviews revealed that these information sources and activities can help schools attract foreign students.

In-depth interviews

The LII supported the language industry in terms of providing it with information on business opportunities on three fronts: through Department research and publications, through trade missions undertaken by Department staff and through the LIP. The LII also supports the industry through the AILIA.

With regard to AILIA publications, one company representative said that employees at his organization used study results, including those of the Technology Roadmap. They also used Info-AILIA, the association's newsletter.

Employees from this organization also greatly appreciated information from the Department. Apparently, because of information from the Department and the AILIA, as well as the contribution received from the LIP, the organization's European sales more than tripled, while Canadian sales increased by 20%. Before receiving the contributions, the organization conducted no promotional activities. However, the organization's representative would have preferred higher contributions, of approximately $20,000. The representative of another company said that the LIP Web site was very user-friendly and that he had used documents produced by the LII to obtain statistics.

As is indicated further on (documentation review), a number of LIP beneficiaries accompanied the Department team on trade missions.

Case study results

The three case studies carried out for the evaluation confirm that LIP contributions had positive repercussions for their organization. In the case of the language school, for example, school trade missions abroad helped to attract foreign students. The school went on three trade missions to Asia, namely, to Japan, China and Korea. School representatives participated in (and sometimes organized) sales seminars and met with local representatives. They also met with potential clients and established a partnership with a representative in China. According to school representatives, these efforts helped increase business activities, in which the number of clients increased by between 500 and 1,000 per year in the last five years. Today, the company attracts over 6,000 students to its schools per year. However, the company acknowledged that some of these activities would have been initiated without LIP support.

Documentation review results

According to the documentation review, the market information produced by the LII is available to the industry via the Internet. The information is disseminated through the Strategis Web site. It contains the reports of all the studies produced by the LII through its Marketing and Branding component. The documentation review indicates that the LII has produced seven market studies: China, Hong Kong/Macao, Korea, Brazil, Czech Republic, Germany and Taiwan. These studies generally describe market conditions. For example, the German study gives an overview of the language training sector in Germany by focusing primarily on its education system, the key factors contributing to market growth, its characteristics and its opportunities. It also deals with the competitive environment and provides conclusions drawn from a survey and interviews.

The LII also supports the language industry portal on the Internet. The portal provides an introduction to the Canadian language industry, the AILIA and the LTRC. It also contains information bulletins (news, events, and conferences). The portal receives over 11,000 visits per year (2006–2007).

The AILIA Web site also contains information about the industry, particularly by means of the Technology Roadmap. The AILIA Web site was launched in January 2005, and its pages are viewed an average of 4,500 times a month by 1,800 visitors. Apart from the Web site, the documentation review indicates that the AILIA took part in 18 annual conferences and seven trade fairs and missions.

The recipients of LIP contributions took part in trade fairs in the following countries:

Survey results

According to the survey results, LII products and services are appreciated, particularly the market information products. According to the results, 66% of respondents found the market studies produced for the Marketing and Branding component useful, and 67% found the reports and publications useful. A similar percentage (65%) found the information on professions in the industry useful. Overall, 69% found the LIP useful to them.

Figure 6: Perceived Usefulness of the LII

[Description of Figure 6]

By sector, it was found that training firms were more likely to appreciate the program (78% find it useful, versus 56% in translation and 58% in language technologies). However, more representatives from the translation sector found information on the professions useful.

According to the survey results, the training sector found trade fairs or missions more useful, with 53% rating them as useful. However, the translation sector appreciates reports and publications more, along with market studies. Finally, respondents were asked about the usefulness of the language industry Web site (portal), and there was a slightly higher proportion of language technology sector representatives who found it useful, as did respondents from the "Others" category.

3.1.4 Has the LII contributed to greater awareness of industry products and services among potential customers in other industrial sectors in Canada and abroad?

In-depth interview results

With some exceptions, the majority of in-depth interview respondents were unable to say whether the Initiative contributed to raising awareness of industry products and services among potential clients from other industry sectors in Canada and abroad. No industry representative was able to confirm that the LII had contributed to their making contacts with clients outside of their traditional markets.

However, one program manager said that a number of companies, particularly those in the manufacturing (export) industry, have changed their opinion with respect to language services. It would appear that some companies in other sectors no longer consider language costs as an "expense", but rather as an investment. These companies have understood how language industry products and services can help them position themselves in international markets and communicate with clients.

Survey results

According to the survey results, half of LIP beneficiaries agreed with the statement that the department's contribution had helped them make contact with people outside of their industry. This is especially true for language technology and language training companies (see Appendix H).

Figure 7: Impact on LIP Contributions

[Description of Figure 7]

3.1.5 Has the LII helped to create synergies between organizations in the language industry sector? If so, how?

General findings

The results generally indicate that the LII has contributed to creating links between the AILIA and other institutions. The LIP has contributed to building links between beneficiaries and other partners, particularly among language training and language technology firms. However, a small majority of AILIA members said that the association has helped to create new contracts with people in their industry.

In-depth interview results

One of the challenges facing the sector is fragmentation. The dispersion of the industry compromises its ability to take advantage of major markets, such as those at the international level. The evaluation focussed on the impact of the LII on synergy among sector organizations.

A number of respondents of in-depth interviews gave examples of synergies created among industry organizations through the LII, such as the links it established through the AILIA with committees (including the translation committee working on the standards project) and exchanges among members. The AILIA has created links by taking a leadership role with respect to the PWGSC standing offer. Employees of the Initiative indicated that the LII had also created new synergies between the LTRC, the AILIA and the Université du Québec en Outaouais (UQO) that did not previously exist and that support industry development. The AILIA subcommittees enable discussions among committees and members. Currently, the AILIA has a Translation Committee, Human Resources Committee, Communications Committee, Technology Roadmap Committee, and four subcommittees (language processing, translation, content management and language training).

In addition, networks were created through the AILIA, according to respondents interviewed. Other examples and results of synergies:

• The Translation Bureau worked with the AILIA to develop promotional materials.
• Translation school associations and Canadian Francophone associations worked together to publish data such as the number of graduates (they also identified 13 translation programs in Canada).
• Partnerships were established between companies for the development of industry standards (e.g. for language schools, discussions on translation – international codes) (see attached case study).

For its part, the LIP enabled small firms to explore potential business development opportunities in countries that would normally be inaccessible.

One of the program managers said that, as a result of the requirement for translation companies to be incorporated to gain access to contributions, a number of self-employed translators decided to collaborate with others to become incorporated. An association representative observed that the LIP also contributed to the emergence of partnerships between training firms and technology-based firms.

Survey results

According to the survey results (see figure below), the AILIA has had a definite impact in this area. As indicated, approximately half of the respondents felt that the AILIA had enabled them to make new contacts with people in the industry. One-third of respondents said that the AILIA helped their organization build partnerships.

Figure 8: AILIA's Impact on its Members

[Description of Figure 8]

LIP contributions seem to have had greater impacts on contacts and partnerships with other firms. According to the results (see figure below), 71% agreed that the contributions had enabled them to make contacts with people in their industry, and just over half said that the contributions helped them build partnerships with other firms or organizations. This is consistent with statements made during the interviews.

Figure 9: Impact of LIP contributions

[Description of Figure 9]

According to the results, training and technology-based companies were more likely to build partnerships with other firms or institutions.

3.1.6 Did the LII have unintended impacts?

In-depth interview results

Few respondents reported unexpected or unanticipated effects. The lack of cohesion between the three sectors was mentioned the most often. The sectors do not identify with each other, and one of the respondents noted that the disparity between the three sectors had almost certainly been underestimated, and that they appeared to have little in common. An observer nonetheless remarked that language technology firms serve the other two sectors.

Some respondents also said that the LIP seemed to respond to the needs of the language training sector better than to those of the translation and language technologies sectors (which was confirmed by the survey).

Two external stakeholders also stated that the Initiative was launched as part of the Action Plan for Official Languages, but it does not comply with the criteria of the plan (an observation made during the mid-term evaluation – see Appendix A). The LII was created in the wake of the APOL but was modified so that the language industry in general could benefit from it, and not just in French and English. For example, Anglophone companies are less interested in the domestic Francophone market, but more so in the international market.

As well, LIP managers noted that the work done to develop and establish a national standard was not planned for at the outset. The people questioned as part of the case study felt that this standard would have a considerable impact on the industry, particularly on its ability to export services to the international market.

Finally, survey respondents were asked overall whether the Initiative had succeeded in helping the language industry set strategic directions, strengthen links between partners, improve its capability to respond to the increasing demand for language products and services, and develop and increase its accessibility. According to 22% of respondents, this objective was achieved, while 70% felt that it was only partly achieved.

Figure 10: Agree that the LII Achieved its Objective

[Description of Figure 10]

As indicated in the following figure, technology and training companies were more likely to think that the LII had met its objective.

Figure 11: Agree that the LII Achieved its Objective

[Description of Figure 11]

3.2 Cost-effectiveness

3.2.1 How cost-effectiveness were the major components of the LII (contribution to the AILIA, steering committee and related secretariat services, language industry contribution program, and marketing and positioning activities)?

General observations

A systematic comparison of management costs for the contribution agreements between the LII and the two other programs shows that the LII management costs were very high. However, estimates of revenue generated by the LIP-funded projects shows that the LIP short-term financial impacts exceed LIP costs.

The evaluation focussed on two aspects of cost-effectiveness, i.e. the cost-efficiency ratio and the cost-effectiveness ratio. The first ratio essentially deals with the proper use of resources in administering the program. In other words, it shows to what extent administrative costs are reasonable in relation to managing the program. The second ratio focuses on program impacts, in view of the investments made.

The efficiency ratios were compared with those of two other fairly similar programs, i.e. the Book Publishing Industry Development Program and Trade Routes, both Canadian Heritage programs (see Appendix G for program descriptions).

3.2.1.1 Cost-efficiency ratio

To assess the cost-efficiency ratio, we compared the administrative costs associated with managing the LIP contribution agreements, the contribution agreement with the AILIA, and the components of two contribution agreement programs of Canadian Heritage, i.e. Trade Routes and the Book Publishing Industry Development Program. The administrative costs were calculated based on the annual salaries of personnel directly involved in the management of contribution agreements, i.e. project selection, review of requests for expense reimbursement, and other monitoring work. A ratio was established between these administrative costs and the volume of contribution agreements.

The team chose to compare these ratios between the three programs for a two-year period, i.e. 2005–2006 and 2006–2007. With respect to the LII, the period of 2004–2005 was a start-up phase, to some extent. In terms of 2007–2008, it was not completed during the field work. The years 2005–2006 and 2006–2007 were therefore selected, which was also appropriate for the other program comparisons.

3.2.1.2 Analyses results of the Cost-efficiency ratio

The table below summarizes the results of the analysis.

Cost-efficiency analysis

According to the data, LIP administrative costs are clearly higher than those of other programs, with a ratio representing 32.9% of contribution agreement costs. This can be explained in part by the large number of contribution agreements to be managed in relation to the total budget. In fact, the average value of LIP contributions is clearly lower than that of other programs, meaning that it has a higher management cost ratio. This can also be explained by the fact that the program offered up to four rounds of funding per year. With respect to the AILIA contribution management costs, the ratio of 14.1% seems quite comparable to that of the other programs, even if the nature of the contribution is somewhat different. Moreover, it is below the generally accepted standard for program administration costs, i.e. 15% (or less).

However, as shown in the following section, the estimated generated revenue exceeds direct program costs.

3.2.1.1 Cost-effectiveness ratio: approach

The analysis of the cost-effectiveness ratio is essentially an examination of program costs versus its effects or impacts. It was not possible to obtain impact data for the other programs compared. However, the evaluation team was able to estimate the value of the revenue generated by the LIP contribution agreements by means of the survey. Program revenues and costs were compared, assuming that the objective was for revenues to exceed costs.

Calculation of revenue generated

The revenue generated can be estimated using two sources of information, i.e. case studies and the survey.

1) Revenue generated according to case studies

According to two case studies, the contributions generated considerable revenue. One of the case studies revealed that the contribution agreement allowed the company to recruit 10 translators from abroad. By estimating that each translator generated at least $25,000 in revenue for the company, the total impact is $250,000 for a single year from a $10,000 contribution, or a ratio of 25 to 1. The language technology company experienced a benefit of approximately $500,000 for participating in a fair and trade mission. The ratio in this case was also 25 to 1, given that there were two contributions.

Therefore, these cases alone generated $750,000 in revenue from an investment of $30,000.

2) Revenue generated by language schools based on the survey

The survey conducted for this evaluation did not allow us to precisely measure revenues generated, for the reasons explained earlier in the report. However, the survey measured the number of students attracted by the schools as a result of activities undertaken through the contribution agreements. According to the survey, 37 organizations attracted an average of 46 students. The revenue generated can therefore be estimated, as described below.

We assumed that one-third of the students were foreign, which is undoubtedly a very conservative estimate.^{Footnote 15}Based on this assumption, we can estimate the benefit using other sources of information. According to a survey of 1,734 foreign students conducted by the Canadian Association of Private Language Schools (CAPLS),^{Footnote 16} we can base our estimates on the following data:

• The average cost of language courses is $250 per week.
• In the case of foreign students, there will be impacts associated with living expenses, which averaged $398 per week according to the CAPLS survey.
• The survey also showed that half of the students travelled to Canada on a Canadian airline. Their travel costs could therefore be estimated at $500 per student per trip (or $1,000 per student, divided by two). Therefore, each foreign student spent an average of $898 for transportation, housing and other living expenses.

In all cases, if we consider that the LIP attracted 1,702 students (37 respondents attracting an average of 46 students) for courses lasting nine weeks (which is the average course length, according to the CAPLS survey), we can calculate the following:

In short, for the Canadian economy as a whole, we can estimate that the LIP made it possible to generate revenue in excess of $6,000,000. This amount easily exceeds the contribution agreement costs ($400,000 per year) and salary costs ($184,000 per year), for a total of $1.7 million over three years.

The LIP cost-effectiveness ratio is therefore reasonable, particularly when we consider that our estimates are essentially based on immediate and partial benefits in one industry sector (we surveyed only about half of the population of organizations that received contribution agreements) and do not take into account the indirect impacts (which could be calculated using an input-output model). We estimate that future and non-estimated revenue will exceed investments even further.

What remains to be seen is what portion of these impacts would have occurred without the LII. This is a difficult question to answer. However, even if the estimate of impacts was overestimated by 100% (or double the actual value), the amounts would nonetheless remain substantially higher than expenses, according to our estimates.

Footnotes

1. ¹⁰ The evaluation team conducted some additional research on the structure of some of these organizations. The CAPLS has three membership categories. Full members are firms that have operated for three or more years and that provide ESL or FSL instruction as their primary activity. Mentored members satisfy the same criteria as full members but firms need to be in operation for a minimum of one year, to a maximum of three. Associate members are firms that support ESL and FSL teaching, but that do not teach as their main activity. (back to footnote reference 10)
2. ¹¹ Minutes of the April 24, 2006, Executive Committee meeting. (back to footnote reference 11)
3. ¹² Industry standards, which have to do with services and monitoring methods, differ from professional standards in that the latter are governed by professional associations. (back to footnote reference 12)
4. ¹³ Canadian Evaluation Society Membership Survey, December 2003, Summary Report. (back to footnote reference 13)
5. ¹⁴ Perspectives (CSA newsletter), Spring 2004. (back to footnote reference 14)
6. ¹⁵ This is undoubtedly a low estimation. According to the Survey of Language Industry Companies in Canada – Translation, Interpretation and Language Training (August 2006), Canadian language schools mainly target foreign students (64% of schools). (back to footnote reference 15)
7. ¹⁶ Canadian Association of Private Language Schools — Preliminary Report, Student Profile Survey – February 2007. (back to footnote reference 16)

4.0 Conclusions

The purpose of this report was essentially to assess the magnitude of Initiative impacts and examine its cost-effectiveness. We examined the impacts from five standpoints, i.e. program success in creating a language industry association at the national level; increasing industry capacity; raising awareness of business opportunities; raising the awareness among potential clients from other sectors about industry products and services; and creating synergies among language industry organizations. The major conclusions from each of these standpoints follows.

AILIA success in positioning itself as a national language industry association. Overall, the results indicate that AILIA membership is not entirely representative of the industry, particularly in terms of training schools. The results also show that a majority of members are dissatisfied with AILIA service delivery and that the AILIA has yet to attract a critical mass of members. The AILIA depends heavily on public funds to survive, and its long-term survival is not guaranteed, at least within the current parameters of the association. The AILIA has nonetheless made significant headway in the past year.

AILIA and LII contribution in improving industry capability. According to the results, LIP contribution agreements have had a positive impact. Language schools were successful in attracting an international clientele, and translation firms and language technology firms say they benefitted from the program in attracting new clients. However, the results of the AILIA are more mixed in that few members agreed that the AILIA had an impact on professional development or even on the development of their firm, as a whole.

Success of the LII in raising awareness of business opportunities. The results indicate that the market studies and reports produced by the LII are appreciated, particularly by translation firms. Trade fairs and missions are also appreciated, particularly by language schools, which used them to attract several foreign students.

LII contribution in increasing the awareness of industry products and services among potential clients in other industrial sectors in Canada and abroad. The results do not allow us to answer this question with any certainty.

Success in increasing awareness of industry products and services among potential clients in other industrial sectors in Canada and abroad. According to the results, half of LIP beneficiaries said that the contribution provided through the LIP allowed them to make contacts with people outside of their industry.

Contribution to creating synergies between language industry organizations. LIP contributions helped to build links between beneficiaries and other partners, particularly with respect to training and language technology firms. The AILIA built links with other institutions. However, a small majority of AILIA members felt that the AILIA helped them to create new contacts with people from their industry.

Cost-effectiveness analysis. Cost-effectiveness was examined in two ways, i.e. the cost-efficiency ratio and the cost-effectiveness ratio. A systematic comparison of management costs for contribution agreements between the LII and two other programs indicates that LII management costs were very high, exceeding the generally accepted standard of 15%. According to analyses of the revenue generated by the contributions, the short-term financial impacts from the LIP exceed program costs, or at least those associated with the contribution program.

The Language Industry Initiative

The Language Industry Initiative has sunset on March 31, 2008. As a result, the evaluation report does not contain any recommendations, and a management action plan from the program was not required. The management response indicated that the program concurs with the findings and conclusions of the evaluation.

Logic Model

The objectives phase has the following written: "To help language industries (Lis) establish a strategic direction, strengthen links among partners, improve their capacity to respond to the increasing demand for linguistic services and products; develop access language markets in Canada and abroad; establish a sustainable succession plan following the implantation of the Action Plan for Official languages (APOL)" It branches down to two different program elements.

The first is "Canadian Network of Language industries Coordination and governance Enhancing potential." The goes down to the activities box that has the heading "Lii implementation group's steering committee." Inside the section are the following points:

• Ensures Language Industry Initiative Implementation
• Supports the creation of representative associations
• Monitors Outcomes
• Organizes and coordinates steering committee activities
• Assesses the language industry profile
• Manages and supervises Language industry transfer payments to the Association
• Supports the development strategy for La Reliève
• Identifies research and analysis priorities.

The next section is output and it contains the following:

• Reports on activities and outcomes
• Updates the Li profile
• Meetings, workshops
• Creation of at least one association
• Contributions to the AILIA for networking activities and HR development
• Annual report to the Privy Council office (PCO)

The next section is labelled Immediate Outcomes

• HR development action plan
• Clear strategic direction
• Industry consensus on and adoption of a strategic direction
• Opportunities knowledge.

The next section is labelled Provisional outcomes

• Synergies between Li sectors
• Curriculum development of teaching establishments
• Elimination of the difference between market demand and available HR.

The last section is labelled Final outcomes and says within it "Sustainable infrastructure for Lis".

The second program element is "Marketing and brand promotion Enhancing Potential." The connects to a box in the activities section with the heading Lii implementation group. In the box are the following bullet points:

• Supports the recruitment of new talent (HR development_
• Supports and coordinates fairs and trade missions
• Supports the development of information products to promote Lis in various markets in Canada and abroad
• Makes public servants and Li companies aware of government programs intended for Lis
• Manages and supervises language industry transfer payments to firms
• Gathers data on national and international markets.

This box then branches down to two more boxes under the output category. The first has the following points in it:

• Marketing strategy – Canada (clients/HR development)
• Li national trade fairs
• Li promotional material
• Printed and online list of government Li programs
• Contributions to Li firms for image brand marketing and promotional activities for participation in trade fairs (national)

It leads next to the immediate outcomes category with a box containing:

• Better Li knowledge among potential students and clients
• Increased participation in trade fairs
• Understanding of market opportunities
• Increased use of government programs by Lis.

Next it goes to Provisional outcomes

• Higher sales
• Industry viability in terms of HR

The second output category of the split had the following written.

• Marketing strategy-international market
• Canada booth at trade fairs
• Information products inviting participation at international events
• Contributions to Li firms for image brand marketing and promotional activities and for participation in international trade fairs and missions.

It leads to an immediate outcomes category box with the following

• Increased participation in trade fair missions and fairs (international)
• Better knowledge of Canadian Lis among Canadian representatives abroad
• Understanding of market opportunities

This leads to a provisional outcomes box with the following:

• More contracts and greater international market share for Canadian Li firms
• Increased presence of Canadian Lis at international promotional activities

The two Provisional outcomes boxes together lead to one under the final outcomes category with "Secure and improved access to national and international markets" inside

Back to Logic Model

Figure 1

The image is of a bar graph with the heading "Reasons why respondents were not members of AILIA" and 5 reasons answered as well as the number of respondents for that answer. 45 percent of respondents answered that they were "Already a member of another organization." 32 percent answered that they "Didn't know the organization." 16 percent answered that "Membership fees too high." 13 percent respondents answered that "It does not add value to our organization." Finally, 26 percent responded "Other (specify)" The of the graph is labelled that "number of respondents (n=31)"

Back to figure 1

Figure 2

The image depicts a graph showing the percentages of responses who answered for varying levels of satisfaction in various categories. The heading of the chart is "Levels of satisfaction with AILIA services". The first category is "AILIAs contribution to the development of standards and best practices for your sector (n=61)"; 27% of respondents said Dissatisfied, 29% said Neutral, and 44% said Satisfied. The next category is "Overall promotional and advocacy activities by AILIA (n=67)"; 22% of respondents said Dissatisfied, 31% said Neutral, and 47% said Satisfied. The next category is "Overall services to AILIA members (n=70)"; 23% of respondents said Dissatisfied, 27% said Neutral, and 50% said Satisfied. The next category is "Conferences, fairs, and/or meetings organized by AILIA(n=61)"; 25% of respondents said Dissatisfied, 25% said Neutral, and 50% said Satisfied. The final category is "AILIA reports and publications (n=65)"; 11% of respondents said Dissatisfied, 23% said Neutral, and 66% said Satisfied.

Back to figure 2

Figure 3

The image is a graph titled "The AILIA's impact on its members." The graph shows responses depicting the various levels of agreement with three statements. The first statement is "Contributed to the development of my organization in general (n=68)" where 40% of respondents answered Disagree, 30% answered Neutral, and 30% answered Agree. The next statement is "Allowed my organization to hire new staff (n=60)" where 70% of respondents answered Disagree, 20% answered Neutral, and 10% answered Agree. The last statement is "Helped the professional training of organization's staff (n=68)" where 45% of respondents answered Disagree, 34% answered Neutral, and 21% answered Agree.

Back to figure 3

Figure 4

The image is a graph titled "The AILIA's impact on its members." The graph shows responses depicting the various levels of agreement with three statements. The first statement is "Allowed my organization to develop new services or products (n=65)" where 67% of respondents answered Disagree, 21% answered Neutral, and 12% answered Agree. The next statement is "Allowed my organization to find new outlets or clients on the international markets (n=60)" where 52% of respondents answered Disagree, 30% answered Neutral, and 18% answered Agree. The last statement is "Allowed my organization to find new outlets or clients on the national markets (n=60)" where 53% of respondents answered Disagree, 27% answered Neutral, and 20% answered Agree.

Back to figure 4

Figure 5

The image is of a graph with the heading "Impact of LIP contributions". The graph shows the percentage of respondents who either agree, disagree or are neutral to a series of statements. The first statement is "Attract new students (n=49) and 8% Disagree, 14% are neutral, and 78% Agree. The next statement is "Observe an increase in the demand for our products and/or services (n=53) and 11% Disagree, 21% are neutral, and 68% Agree. The next statement is "Develop new services or products (n=34) and 35% Disagree, 21% are neutral, and 44% Agree. The next statement is "Develop a business or communications strategy in order to benefit from market opportunities (n=42) and 17% Disagree, 21% are neutral, and 62% Agree. The next statement is "Find new outlets or clients on international markets (n=51) and 8% Disagree, 20% are neutral, and 72% Agree. The final statement is "Find new outlets or clients on national markets (n=49) and 13% Disagree, 13% are neutral, and 75% Agree.

Back to figure 5

Figure 6

The image is of a graph with the heading "Perceived usefulness of the LII". The graph shows the percentage of respondents and how useful they find various LII products and services. The first service is "Language Industry Program (LIP) (n=87)" and 18% say Not Useful, 13% Fairly Useful, and 69% Useful. The next service is "Informations on language industry occupations (n=62)" and 14% say Not Useful, 21% Fairly Useful, and 65% Useful. The next service is "Fairs or Missions (n=57)" and 29% say Not Useful, 19% Fairly Useful, and 52% Useful. The next service is "Reports and Publications (n=63)" and 11% say Not Useful, 22% Fairly Useful, and 67% Useful. The next service is "Market Studies (n=60)" and 18% say Not Useful, 16% Fairly Useful, and 66% Useful. The final service is "Language industry web site (n=75)" and 20% say Not Useful, 25% Fairly Useful, and 55% Useful.

Back to figure 6

Figure 7

The image shows a chart depicting the response to a statement and the degree to which respondents agree with that statement. The title of the graph is "Impact of LIP contributions." The statement "Enabled me to make contact with people outside my industry (n=42)". 33% of the respondents said they disagree, 17% were neutral, and 50% agreed.

Back to figure 7

Figure 8

The image shows a chart depicting the response to two statements and the degree to which respondents agree with the statements. The title of the graph is "AILIA's impact on its members." The first statement is "Enabled me to make new contacts among people in my industry (n=70)." 28% of the respondents said they disagree, 18% were neutral, and 54% agreed. The second statement is "Enabled my organization to build partnerships with other firms or institutions (n=68)."  49% of the respondents said they disagree, 22% were neutral, and 29% agreed.

Back to figure 8

Figure 9

The image shows a chart depicting the response to two statements and the degree to which respondents agree with the statements. The title of the graph is "Impact of LIP contributions." The first statement is "Enabled me to make new contacts among people in my industry (n=48)." 13% of the respondents said they disagree, 17% were neutral, and 71% agreed. The second statement is "Enabled my organization to build partnerships with other firms or institutions (n=43)."  23% of the respondents said they disagree, 21% were neutral, and 56% agreed.

Back to figure 9

Figure 10

The image shows a graph titled "Agree that the LII Achieved its Objective." The statement "The initiative achieved its objective" and the percentage of respondents are given. Of 88 respondents, 8% said not at all, 70% said partially and 22% said completely.

Back to figure 10

Figure 11

This image shows a graph with the title "Agree that the LII achieved its objective." The graph shows respondents based on the type of company and the degree to which they agree. The first type is "Translation (n=28)"; 14% said not at all, 72% said partially, and 14% said completely. The next type is "Training (n=53)"; 11% said not at all, 64% said partially, and 25% said completely. The next type is "Language technology (n=18)"; 22% said not at all, 50% said partially, and 28% said completely. The final type is "Other (n=11)"; 91% said partially, and 9% said completely.

Back to figure 11