Evaluation of the Office of Consumer Affairs' Contributions Program for Non-Profit Consumer and Voluntary Organizations

Final Report December 2014

Presented to the Departmental Evaluation Committee on December 1, 2014

Approved by the Deputy Minister on January 8, 2015

Table of contents

• Executive Summary
• 1.0 Introduction
  • 1.1 Contributions Program Context
  • 1.2 Contributions Program Objective
  • 1.3 Governance of the Contributions Program
  • 1.4 Target Population and Stakeholders of the Contributions Program
  • 1.5 Resources
  • 1.6 Logic Model
• 2.0 Methodology
  • 2.1 Evaluation Approach
  • 2.2 Objective and Scope
  • 2.3 Evaluation Issues and Questions
  • 2.4 Evaluation Methodology
• 3.0 Findings
  • 3.1 Relevance
  • 3.2 Performance
• 4.0 Conclusions and Recommendations
  • 4.1 Relevance
  • 4.2 Performance
  • 4.3 Recommendations
• Appendix A: Examples of Research Projects Referenced in Policy, Legislative or Regulatory Process or Appearing in Media
• Management Response and Action Plan

List of Acronyms used in this Report

List of Tables

List of Figures

Annexes (Separate Document)

Executive Summary

Program Overview

In 1997, the Office of Consumer Affairs of Industry Canada established the Contributions Program for Non‑Profit Consumer and Voluntary Organizations (the Contributions Program) in its current form. The overall objective of the Contributions Program is to strengthen the consumer's role in the marketplace through the promotion of sound research and analysis of consumer issues, and encourage the financial self‑sufficiency of consumer and voluntary organizations. This Program supports consumer and consumer‑focussed research by non‑governmental organizations in Canada. Its intent is to support organizations in producing quality, independent and timely research on consumer issues affecting the marketplace and developing policy advice that is both credible and useful to decision makers. The Contributions Program can allocate up to $1.69M in contributions annually.

Evaluation Purpose and Methodology

The objectives of this evaluation are to assess the core issues of relevance and performance in accordance with the Treasury Board Policy on Evaluation and Directive on the Evaluation Function. The methodology was calibrated to take into account the materiality of the Contributions Program. Nonetheless, the evaluation findings and conclusions are based on the analysis of multiple lines of evidence and included a document/file review, literature review, interviews and financial analysis. The evaluation covered the period from April 2010 to June 2014.

Findings

Relevance

There is a continued need for the Contributions Program as it helps to increase capacity for consumer organizations and facilitates their participation in the marketplace and the legislative and regulatory process. Consumer organizations play a vital role in the marketplace and help identify existing and emerging consumer issues for decision makers.

The Contributions Program aligns with the roles and responsibilities and priorities of the federal government, including the Consumers First Agenda. In line with the Industry Canada strategic outcome, the Contributions Program also contributes to an efficient and competitive Canadian marketplace. The Contributions Program is the only federal funding source for consumer policy research and development.

Performance

The Contributions Program has strengthened research on consumer issues affecting the marketplace. The Program has increased the credibility of the consumer organizations and helped to fill a research gap in the Canadian marketplace. The Program ensures rigor in the research through administrative requirements, such as the involvement of a methodologist and the use of an external firm when conducting public opinion research, and has enabled organizations to develop internal capacity.

The Contributions Program has fostered collaboration among the various consumer groups. For example, projects by the Canadian Consumer Initiative that received funding from the Contributions Program resulted in a culture of collaboration among participants. Consumer organizations are increasingly choosing to work together in order to have a greater impact.

Since the last evaluation in 2010, the Office of Consumer Affairs (OCA) has increased its efforts to track the impact of research reports. Program‑funded research results have been cited in policy, regulatory and legislative decision‑making processes at both the federal and provincial levels. Research results have also appeared in the media on numerous occasions, suggesting the Contributions Program has had the expected impact of promoting advice on consumer issues that is useful to decision‑makers.

Organizational development funding has assisted consumer organizations in improving their internal capacity. Organizations have increasingly become more active in developing projects and activities, but attaining self‑sufficiency is still a long‑term objective for many of these consumer organizations. While the impacts of development projects are typically internal to an organization, more effort is required in tracking the results of these projects.

The Contributions Program has been administered efficiently and economically. A number of process improvements have been implemented to improve efficiency. Opportunities exist to reduce the reporting requirements and suggestions were made by Program recipients and other government representatives related to the timing of the intake process.

Recommendations

The findings of the evaluation led to the following recommendations:

1. That the Contributions Program consider implementing a systematic follow‑up of development projects to track their long‑term impacts on recipient organizations, including progress towards financial self‑sufficiency, without adding significant additional reporting requirements for recipients; and,
2. That the Contribution Program review their risk‑based monitoring approach to ensure that reporting requirements adequately reflect variables such as the amount of funding, the history with the client and the sensitivity and complexity of the project.

1.0 Introduction

This report presents the results of an evaluation of the Office of Consumer Affairs' (OCA) Contributions Program for Non‑Profit and Voluntary Organizations (the Contributions Program). The purpose of the evaluation was to assess the Program's relevance and performance. The report is organized into four sections:

• Section 1 provides the profile of the Contributions Program;
• Section 2 presents the evaluation methodology;
• Section 3 presents the findings pertaining to the evaluation issues of relevance and performance; and
• Section 4 summarizes the study's conclusions.

1.1 Contributions Program Context

The Government of Canada has been providing financial assistance to consumer organizations since the 1970s, following the creation of the Department of Consumer and Corporate Affairs Canada. The primary objective of the funding program at the time was to strengthen the consumer movement in Canada by assisting non‑profit community organizations in providing direct services to consumers (e.g., consumer information and education, redress and other assistance).

With the establishment of provincial and territorial ministries responsible for consumer protection in the late 1970s, the Department withdrew from consumer matters falling within provincial jurisdiction. Consequently, the Department no longer awarded funding for projects relating to consumer information and educational activities.

At that time, studies^{Footnote 1} demonstrated that while consumer representatives were included in the regulatory decision‑making process, their input was qualitatively inferior to other contributors, such as industry representatives. Some attributed this to limited scientific training and others to timidity in the face of the greater professional knowledge of the scientific and industry representatives. With their limited financial resources it was difficult for consumer organizations to obtain adequately trained representatives. In this context, the OCA in Industry Canada (IC) established the current Contributions Program in 1997.

Today, consumer affairs are a shared responsibility between the federal and provincial/territorial governments. The federal government is explicitly responsible for consumer protection with respect to the safety of products, including food and drugs; the transportation system; telecommunications; nationally offered financial services and payment systems; and national economic framework policies such as intellectual property, bankruptcy, and weights and measures. The provinces and territories are responsible for most matters relating to consumer contract law, particularly the basic buyer‑seller relationship in a retail setting—which supports the transactions consumers make in the sectors regulated by the federal government.

1.2 Contributions Program Objective

The overall objective of the Contributions Program is to strengthen the consumer's role in the marketplace through the promotion of sound research and analysis on consumer issues, and by encouraging the financial self‑sufficiency of consumer and voluntary organizations. This Program supports consumer and consumer‑focussed research by non‑governmental organizations in Canada. Its intent is to support organizations in producing quality, independent and timely research on consumer issues affecting the marketplace and developing policy advice that is both credible and useful to decision makers. It is the only mechanism of its type in Canada for funding broad‑based consumer research independent of industry influence.

There are two types of funding that can be awarded under the Program: research project contributions, which aim to help organizations conduct consumer‑based research and improve their research capacity; and development project contributions, which are aimed at institution and capacity building for consumer and voluntary organizations. There are two categories of development project contributions: organizational development project contributions aimed at helping organizations become financially self‑sufficient (i.e. multi‑year business plans and feasibility studies); and consumer movement development project contributions aimed at increasing collaboration and cooperation among consumer organizations. Figure 1, shows the number and type of approved projects for fiscal years 2010‑11 to 2013‑14 and for projects approved as of June 30, 2014 for fiscal year 2014‑15^{Footnote 2}.

Figure 1: Number of Approved Projects by Year and Type

A total of 133 projects were funded during the evaluation period, including 86 research projects and 47 development projects. Five of the projects funded spanned two fiscal years (three development projects and two research projects).

1.3 Governance of the Contributions Program

The Contributions Program is managed and administered by the OCA in accordance with the Treasury Board's Policy on Transfer Payments. Formal accountability rests with the Director General of the OCA who reports to the Assistant Deputy Minister, Strategic Policy Sector. The Director of Consumer Services and Outreach is responsible for the overall daily operations and management of the Program.

The maximum contribution per project is $100,000. The maximum amount that can be awarded to a single organization per year is $500,000 for all projects undertaken in a fiscal year. Projects are funded through a formal Contribution Agreement. The Contributions Program has a fixed $1.69M budget available per fiscal year.

The Program launches its annual call for proposals every October with a late November closing date. Funding decisions are communicated to all applicants in April and funded projects must be completed by March 31^st of the following year^{Footnote 3}.

All proposals submitted in each call for proposals are evaluated by OCA policy analysts and by other relevant departments, and agencies as appropriate. The evaluations of all proposals as well as specific comments from the evaluators are reviewed by OCA Program Management Committee composed of the Director General, the Director of Consumer Policy and the Director of Consumer Services and Outreach.

1.4 Target Population and Stakeholders of the Contributions Program

The Contributions Program targets registered Canadian non‑profit consumer and voluntary organizations working in the consumer interest and who perform activities such as:

• providing objective information, and education which enables consumers to protect their interests in the marketplace;
• responding to consumer enquiries and complaints;
• undertaking independent research and analysis of current consumer protection issues and concerns; and/or,
• promoting and representing the consumer interest to industry and government.

The Contributions Program ultimately benefits Canadian consumers by providing an avenue for the consideration of the consumer perspective in the policy‑making process which in turn results in better representation of consumer interests.

Key stakeholders for the Contributions Program include:

• Other federal government departments, provincial and territorial governments

  In addition to internal assessments at OCA for research proposals, OCA will usually consult with other sectors within IC and, when appropriate, other federal departments and provincial/territorial governments, as part of the assessment process.

• The Consumer Measures Committee (CMC)

  OCA also co‑chairs the CMC, a joint federal/provincial/territorial committee which provides a forum for national cooperation to improve the marketplace for Canadian consumers, through the harmonization of laws, regulations and practices and through actions to raise public awareness. The CMC is informed of the outcomes of the projects that receive funding under the Contributions Program.

• Financial Services Advisory Committee (FSAC)

  OCA, Finance Canada and the Financial Consumer Agency of Canada have formed FSAC based on a Memorandum of Understanding (MOU). FSAC assesses proposals dealing with financial issues and are also involved in the review of the final deliverables.

1.5 Resources

The Contributions Program can provide up to $1.69M in contributions annually. This remained constant during the evaluation period (2010‑11 to 2014‑15). In addition, the ongoing cost of operating and maintenance for the Contributions Program are included in OCA's overall annual budget. The following table provides details of the allocation of Program resources over the evaluation period.

1.6 Logic Model

The logic model presented on the following page depicts the program theory for the Contributions Program. That is, it shows how the Contributions Program's activities are expected to lead to certain outputs and various levels of outcomes, and ultimately, to one of IC's strategic outcomes.

This logic model was updated and streamlined in 2014 to focus on the Contribution Program's key activities and outcomes for this evaluation. The revision was conducted during the planning phase of the evaluation in consultation with OCA staff. This logic model provides the basis for developing the evaluation strategy, including the evaluation matrix.

Figure 2: Logic Model of the Contributions Program for Non‑Profit Consumer and Voluntary Organizations

2.0 Methodology

This section provides information on the evaluation approach, objective and scope, the specific evaluation issues and questions, the data collection methods, and data limitations for the evaluation.

2.1 Evaluation Approach

This evaluation was goal‑based, that is, it was based on the expected outcomes of the Contributions Program as stated in its foundational documents and logic model. The evaluators measured the outcome variables using a variety of research methods. The evaluation study was conducted in‑house by IC's Audit and Evaluation Branch (AEB).

2.2 Objective and Scope

The objectives of this evaluation are to assess the core issues of relevance and performance in accordance with the Treasury Board (TB) Policy on Evaluation and Directive on the Evaluation Function. With respect to the Contributions Program's performance, the evaluation addressed the immediate, intermediate and long‑term outcomes (to the extent possible) and covers the period from April 1, 2010 to June 30, 2014.

2.3 Evaluation Issues and Questions

The evaluation addressed the following questions on relevance and performance:

Relevance

1. Is there a continued need for the Contributions Program?
2. To what extent do the objectives of the Contributions Program align with the priorities of the federal government and the strategic outcomes of IC?
3. To what extent does the Contributions Program align with the roles and responsibilities of the federal government?

Performance

4. To what extent has the Contributions Program strengthened research on consumer issues affecting the marketplace?
5. What are the impacts of the Contributions Program on the level of collaborative work among organizations aimed at representing consumers in the marketplace?
6. To what extent has the work of consumer organizations contributed to policy advice on consumer issues?
7. To what extent has the Contributions Program improved the organizational capacity of consumer organizations to move closer to attaining self‑sufficiency?
8. Has the Contributions Program been administered efficiently and economically?

2.4 Evaluation Methodology

The methodology for this evaluation has been calibrated to take into consideration the materiality of the Contributions Program, which represents 0.15% of planned departmental spending for 2014‑15, as well as the fact that it was evaluated in 2009‑10 and has undergone few changes since that evaluation.

Nevertheless, multiple lines of evidence were used to address all the evaluation questions. The data collection methods included a document/file review, literature review, interviews, and financial analysis.

Document/File Review

A document review was conducted to gain a thorough understanding of the Contributions Program. Departmental and Program documents and recipient files were reviewed to address issues of relevance and to gain insight into the performance of the Contributions Program (i.e. extent to which projects have contributed to the main objectives and outcomes of the Program).

Key documents and files reviewed included:

• Government priority setting documents (e.g. Budgets, Speeches from the Throne);
• Program foundational documents (e.g. Treasury Board Submissions, Terms and Conditions, Performance Measurement Strategy);
• Departmental documents (e.g., Report on Plans and Priorities, Departmental Performance Report, Performance Measurement Framework);
• Program documentation (i.e. compendium of program results including policy discussions and media coverage);
• Recipient project files (project proposals/reports, contribution agreements);
• Applicants and recipient guides for the studied period; and
• Previous evaluations (2005 and 2010).

Literature Review

This review focused on the core evaluation issues of continued need, and alignment with federal roles and responsibilities. Specifically, it looked at the evaluation question regarding continued need to support non‑profit consumer and voluntary organizations. In addition, the literature review examined best practices and alternative approaches from other jurisdictions. A list of the articles and reports reviewed is contained in Annex A.

Interviews

The objective of the interviews was to gather in‑depth information for evaluation purposes, including views, explanations and factual information that address the evaluation questions. Interviews allowed evaluators to gain insight into the performance of the Contributions Program from the perspective of individuals with first‑hand knowledge of various aspects of the Program. The interviews were semi‑structured in nature and designed to obtain qualitative feedback from a range of respondents. Interviews were conducted in the language of the participant's choice and interview guides in both official languages were provided beforehand. The interviews were conducted in‑person where possible and, where not possible, by telephone. A generic interview guide is provided in Annex B.

In total, 22 individuals were interviewed from the following groups:

• OCA, Industry Canada (5);
• Executive directors / senior members of 11 recipient consumer organization (12);
• Other federal government representatives (3); and
• Provincial government representatives (2).

Financial Analysis

A high level financial analysis of the Contributions Program was conducted to inform the findings on continued need and performance. Quantitative data gathered from the Contributions Program (for the period April 1, 2010 to June 30, 2014) ranged from basic program usage data to trends among recipients with respect to the utilization of the Program. These included total funds allocated by type of project (Research and Development); dollar value of each proposal/project; and funds awarded each fiscal year for the two types of contributions' streams.

Limitations

Randomization in the selection of interviews was not possible because of the small number of individuals in IC, other government departments, and among consumer organizations that have a solid knowledge of the Contributions Program and its impact. Eighteen different consumer organizations received contributions from the Program during the evaluation period. Given that it is a small number, the evaluation team interviewed a sufficient number of individuals from various stakeholder groups to ensure all views are represented.

It is difficult to fully assess some of the Contributions Program outcomes outlined in the logic model, because they relate to the provision of policy advice. Policy advice is difficult to assess and it is not always possible to attribute decisions to particular advice that was provided in the form of a Program result. It was only possible to track outcomes provided by Program recipients that contributed and/or supported policy discussions.

Interviews helped to mitigate this limitation, particularly as the key focus was to gain insight into the performance of the Contributions Program. The views and opinions of interviewees provided the necessary context and understanding in assessing the achievement of outcomes as did the compendium of the Program's results. Further, this limitation was judged to be acceptable in relation to the level of calibration set out for this evaluation.

3.0 Findings

3.1 Relevance

3.1.1 Is there a continued need for the Contributions Program?

Consumer research has been identified by the Organization for Economic Co‑operation and Development (OECD) as a key element to evidence‑based consumer policy development.^{Footnote 4} Consumer research is increasingly important given the breadth and scope of changes that have taken place in the market over the last decades. New tools, technologies and the growing consumer consumption of services have given rise to new issues, both for consumers and policy makers (e.g., e‑commerce, mobile services, electronic payments).^{Footnote 5} This has reinforced the need for timely research in order to identify, monitor and quantify issues and facilitate the development of evidence‑based policies with consideration of the consumer's perspective. All interviewees confirmed that there is a continued need for the Contributions Program. Recipients affirmed that their research projects would not have proceeded without contributions received from the Program.

Consumer organizations play a vital role in the marketplace. They support consumer education, and they are well positioned to help identify existing and emerging consumer issues.^{Footnote 6} This reinforces the need to have consumer organizations contribute to the policy‑making process to ensure that the consumers' perspective is considered. This is particularly true in light of the different levels of resources available to consumer groups compared to industry groups and the private sector. The existence of the Contributions Program allows consumer groups to conduct their own research and better participate in the marketplace and legislative/regulatory processes.

The Contributions Program is able to offer its recipients the opportunity to explore existing and emerging consumer issues in an in‑depth manner which helps contribute to effective policy interventions by generating independent research results. These types of analyses assist consumer groups to build their capacity on a given topic or issue, and can improve their credibility in their interactions with governments. An interviewee from a provincial government noted that the research funded through the Program is helpful in developing policy approaches as the projects normally provide an overview of how a given issue has been addressed in other jurisdictions. An interviewee from a recipient organization noted that this depth of analysis on consumer issues would not be possible without funding provided by the Program.

The Contributions Program is oversubscribed which helps to substantiate the continued need. Figure 3 illustrates the types and amounts of funding requested and awarded by fiscal year. During the evaluation period, requests for research funding amounted to 3.5 times the dollars awarded, while requests for development funding amounted to 2.8 times the dollars awarded. Since demand exceeds available Program resources, eligible projects are being declined through the competitive process.

Figure 3: Amount of Funding Requested and Awarded by Fiscal Year

3.1.2 To what extent do the objectives of the Contributions Program align with the priorities of the federal government and the strategic outcomes of Industry Canada?

In Canada, consumer spending accounts for over half of the Gross Domestic Product. Protecting consumer interests, such as confidence in the safety of products, is therefore crucial to ensure economic growth and fairness in the marketplace. During the period of the evaluation the federal government has been focusing increasingly on consumer issues. This is evidenced by the number of consumer‑related measures introduced in federal budgets during the evaluation period, as well as commitments made by the government via the Speech from the Throne 2013. Since 2010 the government has introduced measures to protect consumers on issues related to taxes and tariffs, marketplace competition in various sectors including the payment sector, financial services, and telecommunications. This has culminated with the introduction of the Government's Consumers First Agenda in Economic Action Plan 2014. The research projects funded through the Contributions Program are well aligned with the recent consumer measures implemented, as well as the actions outlined in the Consumers First Agenda.

To ensure that the research projects are relevant, OCA utilizes federal and provincial government representatives to assist with assessing project proposals (i.e. Financial Consumer Agency of Canada, Canadian Radio‑television and Telecommunications Commission (CRTC), Heritage Canada, provincial and territorial governments). A number of these organizations who evaluate proposals were interviewed and stated that when assessing proposals they also consider the relation to their priorities and organizational research plan in terms of timeliness and relevance.

The Contributions Program also contributes, when possible to the Federal Sustainable Development Strategy through contributions provided for research projects that address consumer issues such as sustainable consumption.

A strong consumer movement has also been identified as an essential component to well‑functioning markets.^{Footnote 7} Through its support to consumer groups via both development and research contributions, the Contributions Program contributes to an efficient and competitive marketplace which is a Strategic Outcome under IC's Program Alignment Architecture.

3.1.3 To what extent does the Contributions Program align with the roles and responsibilities of the federal government?

Section 4.1 (d) of the Department of Industry Act (1995) extends powers, duties and functions to the Minister of Industry in the area of consumer affairs, while section 5(i) requires the Minister to promote the interests and protection of Canadian consumers. The Contribution Program helps the Minister fulfill this role by supporting consumer organizations and consumer‑related research to help foster evidence‑based approaches to policy making.

The OECD, in their Consumer Policy Toolkit, indicate that monitoring markets to identify areas where consumers are being disadvantaged is one of the principal roles of competition, and consumer authorities can help identify where government intervention may be required to protect consumer interests.^{Footnote 8} Although data on consumer complaints have traditionally been the primary measure used by governments to gauge where policy interventions may be required, complaint data may be biased as a number of socio‑economic factors impact the likelihood of consumers complaining.^{Footnote 9} In‑depth analysis stemming from consumer complaints may also reveal issues of which consumers are unaware.^{Footnote 10} As a result, there is a role for the government to play in funding research by consumer groups. Two interviewees noted that, in Canada, the role for the government is even greater in light of the absence of a national consumer testing organization, which in other countries serves as a major source of funding for consumer policy research.

The Contributions Program is the only federal funding source for consumer policy research and development. The Office of the Privacy Commissioner of Canada has a complementary program which solely funds privacy related research; however, that program is not limited to consumer groups, and the majority of the funding in recent years has been secured by academic institutions. Other departments and provincial governments may also fund specific projects to address a given need; however, this funding is ad hoc and not part of a formal ongoing program.

Internationally, a number of funding models are in place to support consumer groups and consumer policy research ranging from the provision of core funding to allowance of limited funding for specific projects / outcomes. However, comparisons are difficult as a number of factors have an impact on the institutional arrangements (i.e. constitutional responsibility for consumer interest protection varies). For instance, the United States government does not provide direct funding to consumer groups^{Footnote 11} but many American consumer organizations conduct consumer policy research using alternative sources of funding or mechanisms. For example, Consumers Union, the publisher of the Consumer Reports, spent $15.5M USD in 2013 on consumer advocacy and education while having generated $234M USD in subscriptions, newsstand and other sales.^{Footnote 12} Conversely in Norway, the main consumer group, the Consumer Council of Norway, receives 95% of its funding from the government (€11.3M in 2008).^{Footnote 13} Similarly in France, a block of funding is available to support the consumer movement (€7.4M) and is distributed among the various consumer groups.^{Footnote 14}

3.2 Performance

3.2.1 To what extent has the Contributions Program strengthened research on consumer issues affecting the marketplace?

Interviewees confirmed that the Contributions Program has been instrumental in strengthening research on consumer issues. As one recipient noted: "It's either we do it or we don't. If the Program wasn't there we couldn't do research on consumer issues."

During the evaluation period a total of 86 research projects from 15 consumer organizations were funded through the Contributions Program which amounted to $5,428,179. The average amount of funding requested per project (Research and Development) has increased during the evaluation period (Figure 4) and Program staff indicated that the higher value proposals reflect a shift towards better and more thorough projects. The Contributions Program has also helped improve the quality of research produced by requiring the involvement of a methodologist and the use of an external firm when conducting public opinion research.

Figure 4: Average Dollar Value Per Proposal

Four of the interviewees indicated that the research projects have helped to improve their credibility and their ability to consult on various issues. They are better equipped to advocate on behalf of consumers as a result of their research projects. Interviewees from other government departments also echoed this by highlighting the extent to which the research projects have helped the consumer organizations effectively contribute to various consultations. This is further evidenced by the extent to which the reports have been able to help influence both policy and industry discussions (outlined in section 3.2.3).

An interviewee from another government organization noted that the Program has been able to fill research gaps on the Canadian marketplace for the industry that they regulate, noting that they are often asked how Canada compares internationally on given issues, but that they have limited resources to conduct their own research. The research funded through the Contributions Program has helped to fill this gap.

Four of the recipient consumer organizations indicated that they have been able to develop their internal human resources as a result of the Contributions Program funding, and been able to hire dedicated research staff. In some instances, the organizations have been able to develop expertise on a given topic, while in others the research capacity itself has developed. For example, the Automobile Protection Association (APA) has developed mystery shopping expertise.

With increased credibility, demand for input and involvement from recipient consumer organizations on consumer issues (from government, industry and/or media) has increased, especially following the release of a research report. Although this is beneficial for the organizations, it can pose some challenges because once the research funding has ended, the organizations must absorb the costs of follow‑up activities. Half of the recipient consumer organizations suggested funding should be provided to support follow‑up activities or to support their core mandate.

Two interviewees also noted that in the past the Contributions Program has organized skill development activities for recipient consumer organizations which were beneficial (e.g., training sessions, guest speakers). It was suggested that additional activities in this area could help recipient organizations further tailor their research projects to increase their utility in the policy development process.

3.2.2 What are the impacts of the Contributions Program on the level of collaborative work among organizations aimed at representing consumers in the marketplace?

Under the consumer movement development stream, the Contributions Program has supported the development of the Canadian Consumer Initiative (CCI), which has served to increase collaboration among consumer organizations. The CCI brought together four key consumer groups (Consumers Council of Canada (CCC), Option consommateurs (OC), the Public Interest Advocacy Centre (PIAC) and Union des consommateurs (UC)) to facilitate information sharing and the development of common positions on various consumer issues. Fourteen CCI projects received a total of $353,700 during the evaluation period. In addition to facilitating the development of common policy positions, the CCI projects enabled the participating groups to hold regular discussions, participate in meetings with government officials, engage with Members of Parliament, contribute to government consultations, monitor parliamentary and policy development and publish joint media releases.

The CCI has recently been discontinued as the initiative was reported to have served its purpose. Through the CCI, the consumer groups gained familiarity with each other and have now developed a "reflex of collaboration". This is evidenced by the fact that three of the groups (OC, PIAC and UC) are now working together on a common topic (consumer financial service issues and payment system processing) despite the absence of continued funding for the CCI.

Interviews with consumer organizations also revealed collaboration over and above what had been supported through CCI. For instance, both Les Éditions Protégez‑Vous (ÉPV) and the Automobile Consumer Coalition (ACC) have reported collaborating with the APA for content on automobile related projects.

A new network has also recently been announced which will help to increase collaboration to support consumer research. The OCA as well as the four groups formerly part of CCI have recently partnered with a number of academic institutions, as well as government organizations and representatives from the private sector to form the Canadian Partnership for Public Policy‑Oriented Consumer Interest Research. The project is being co‑led by researchers from six universities, and has secured $200,000 in Social Sciences and Humanities Research Council funding over three years. The group will be conducting and sharing their research which aims to improve consumer policy, and ultimately the well‑being of consumers.

3.2.3 To what extent has the work of consumer organizations contributed to policy advice on consumer issues?

Since the last evaluation in 2010, OCA has increased its media scanning efforts to continuously track the impact of its results (more specifically research reports produced through the Contributions Program). As well, in the application form, consumer organizations need to demonstrate that the project results will be available in time to provide input into the decision‑making process; provide a detailed communications plan; and describe how they will inform relevant stakeholders. Recipients are also required to provide a Communications Report at the end of the project which includes information such as the distribution of final project results.

The majority of Program recipients, through their websites, are providing copies of research reports and also media releases to highlight the findings of their studies. In addition, their reports are also sent to key stakeholders (government and industry) and their membership and/or other decision makers. Citations of, and references to, Program‑funded research are used in a variety of manner, including witness testimonies in parliamentary committees, stakeholder briefings for ministers of the Crown and members of Parliament, federal and provincial regulatory and utilities board hearings, input to government and to private‑sector consultations, task forces, and articles and reports in media.

OCA prepared a document to serve as a compendium of projects that received funding from the Contributions Program. Part of the document contains evidence and actual data on research and analysis by Program recipients which contributed to policy discussions, public forums (i.e. conferences, symposiums, etc.) and other initiatives. The document shows that there were 30 policy‑type discussion references to funded projects from seven different consumer organizations from 2010‑11 to 2013‑14. The following table provides a summary of the types of references:

In terms of media coverage of projects funded by the Contributions Program, over 90 news article references, 11 radio shows and 12 segments on television (e.g. news, W5, L'Épicerie) were identified through interviews, the file review and the OCA compendium document. Appendix A provides examples of research projects referenced in policy, legislative or regulatory processes or appearing in media.

Consumer organizations have become recognized as a credible reference group. A number of the consumer organizations have developed good relations with government departments and agencies and/or parliamentarians and, as a result, they seek input from them during consultations on consumer issues. As a result of these relationships, consumer organizations are also invited to attend events, such as the Speech from the Throne, and are often contacted by media to comment on consumer issues following government announcements. For example, PIAC has made issues associated with the regulatory process a priority (i.e. CRTC and the Ontario Energy Board (OEB)).

As mentioned earlier, the provision of policy advice is difficult to assess and it is challenging to attribute decisions to particular advice. It is only possible to track which type of policy advice was provided by recipients (through its research reports) to support consumers in the marketplace. However, the evaluation found that funded research projects are useful to decision‑makers, but realizing its impact is often a long‑term process.

3.2.4 To what extent has the Contributions Program improved the organizational capacity of consumer organizations to move closer to attaining self‑sufficiency?

The majority of consumer organizations are small. The average recipient employs around five people and has annual revenues in the hundreds of thousands of dollars. Organizational development projects funded through the Contributions Program aim at assisting organizations to become self‑sufficient (i.e. developing business and strategic plans to increase membership, undertaking a feasibility study to see how revenue diversification can be achieved or implementing a pilot‑project to offer new services to consumers).

The need to support the capacity of consumer organizations has led to an increased percentage of Program funding towards developmental projects. Of the 43 development projects funded from 2010‑11 to 2013‑14, 30 have been from the organizational stream and benefited 12 consumer organizations. The number of development projects as a percentage of the total number of projects approved has increased proportionately from 28% to 46% during this four year period.

Figure 5: Percentage of Research and Development Projects Approved by Year

The impacts of development projects are typically internal to an organization, and thus can be more difficult to track. Although the Program does require regular reporting during the Contribution Agreement (e.g. reporting on activities and expenditures), they do not systematically track the results of development projects after the end of this agreement making it difficult to assess the extent to which the Contributions Program is helping to improve the capacity of consumer organizations.

Nonetheless, below are three examples of where contributions to support organizational development projects have had an impact on organizations revenue sources.

• APA received funding in 2012‑13 from the development stream of the Program to modernize the transactional portion of their website. As a result, they have automated an initiative where consumers can request a dealer cost report on a given vehicle for $25. Automation has allowed APA to significantly increase the revenues from this service.
• The Consumers' Association of Canada (Manitoba) Inc. (CAC‑MB) received funding in 2013‑14 for a feasibility study "Fee‑for‑Service as means to greater self‑sufficiency". The purpose of the project was to look at what consumers and other non‑profit organizations and agencies would be prepared to pay for in terms of consumer information and what other organizations are doing to increase their revenues. CAC‑MB was able to use the exposure from this project to secure fee‑generating services from several clients. For example, CAC‑MB undertook the provincial management of a program offered by a national non‑profit organization; signed a letter of retainer with another client to conduct mystery shopping; and, agreed to a contract to conduct focus groups and facilitate stakeholder meetings.
• ÉPV receives 85% to 90% of their funding from consumers who are purchasing access to their information, either through their magazine or digital content (over 100,000 subscribers). ÉPV has received development funding for one project in each of the past five years. Four of these projects related to digital content and have helped ÉPV remain relevant as there is a growing demand for digital content, while paper magazine subscriptions are generally declining. A recent project allowed them to develop two digital versions of their auto guide, thus increasing revenues as they are now selling the guides through the Apple store, which is a new distribution channel for revenue.

An example of an organization that has been able to reduce its financial reliance on the Contributions Program by diversifying its sources of funding is PIAC. They have become increasingly involved with regulatory proceedings and have been granted intervenor status by both the CRTC and the OEB. This enables them to participate in regulatory proceedings to represent the interests of consumers. They are compensated for their participation through cost awards. PIAC received a $42,800 development contribution from the Program in 2012‑13 to assist them to build their internal capacity to participate in regulatory proceedings related to broadcasting. As a result of their increased participation in public proceedings, they have diversified their revenue sources and decreased their reliance on the Contributions Program as the funding they receive from cost awards has increased, while the funding they received from the Contributions Program has steadily decreased over the evaluation period from $409,000 in 2011‑12 to $260,500 in 2013‑14.

Consumer organizations that receive development contributions are examining ways to diversify and increase their revenue sources, and believe progress is being made towards financial self‑sufficiency; however, progress is slow and varies from one organization to another. Many still require contributions provided through the Program to survive; while a few could survive without the funding, but they would no longer be able to conduct national research projects on consumer issues.

3.2.5 Has the Contributions Program been administered efficiently and economically?

Interviewees agreed that the Contributions Program has been administered efficiently. A team of three staff administer the Contributions Program, with O&M costs being absorbed by the broader OCA budget. In general, the administrative costs are low, and the Program makes use of human resources elsewhere within IC, as well as from other federal departments and provincial ministries in the assessment process at no cost.

During the evaluation period a number of process improvements were introduced which have resulted in increased efficiency. For instance, in the current year, the Contributions Program automated certain aspects of the assessment process to simplify the routing of completed assessments by all evaluators and automating the compilation of assessment results. This is estimated to have reduced work by the Program staff associated with compiling individual assessments by at least 37.5 hours. Also, in the 2013‑14 fiscal year the Program adopted an online application system based on a functional system being used by other programs in IC. As the intake system had already been developed by colleague programs, the costs to adapt the system to the Program's unique needs were minimal. Four interviewees reported that the changes to the application system had resulted in improvements, however three interviewees indicated that there was some duplication within the application form questions^{Footnote 15}. OCA is also currently working on improving the automation of reporting forms (financial and progress) in order to reduce errors (i.e. missing information, calculation errors, etc.) which often resulted in delays in processing claims which can have an impact on the recipient's cashflow.

The main concern noted by recipient organizations interviewed was regarding the administration of the Contributions Program related to the reporting requirements. Half of the recipients interviewed indicated that the amount of reporting was excessive in light of the dollar value of the contributions received. Reporting was particularly challenging for recipients with multiple projects. The quarterly reporting required creates work both for OCA staff and for the consumer organizations that could be reduced. In order to respond to this concern, the Contributions Program should explore the adoption of a risk‑based monitoring approach that would tailor the reporting requirements based on variables such as the amount of funding, the history with the client and the sensitivity and complexity of the project. This would align with the IC Risk Management Framework and is consistent with the Policy on Transfer Payments and the Government Action Plan in response to the independent Blue Ribbon Panel on Grant and Contribution Programs.

Half of the recipient organizations interviewed noted challenges related to the timing of the Contributions Program's project cycle. The fixed timing of the application process and research schedules which conform to fiscal year funding realities, limit the ability to be responsive to an emerging issue. Two recipients noted that because of delays in approvals in 2014‑15 they had less time to complete their research projects. Two recipients also noted that having a single project intake per year limits their ability to capitalize on developing issues. This was also noted by two interviewees from other government agencies, who suggested that the timing poses challenges for aligning with the government policy cycle^{Footnote 16}.

4.0 Conclusions and Recommendations

Regarding the relevance and performance of the Contributions Program, the following conclusions can be reached.

4.1 Relevance

• There is a continued need for the Contributions Program as it facilitates evidence‑based approaches to policy making and helps to increase capacity for Canadian consumer organizations.
• The Contributions Program aligns well with the priorities of the federal government, including the Consumers First Agenda. Consistent with one of IC's strategic outcome, the Program also contributes to an efficient and competitive Canadian marketplace.
• The Contributions Program aligns with the roles and responsibilities of the federal government under the Department of Industry Act. The Program is the only federal funding source of its kind and that provides contributions for consumer policy research and development projects.

4.2 Performance

• The Contributions Program has strengthened research on consumer issues and increased the credibility of consumer organizations, which allowed them to build and retain in‑house research capacity that is vital in the longer‑term.
• The Contributions Program has promoted collaboration among the various Canadian consumer organizations. Consumer organizations are now increasingly choosing to work together in order to have a greater impact.
• Contributions Program‑funded research results have been cited in policy, regulatory and legislative decision making processes at both the federal and provincial levels and have also appeared in the media on many occasions.
• Organizational capacity of consumer organizations has improved but attaining self‑sufficiency is still a long‑term objective for many of these consumer organizations. While the impacts of development projects are typically internal to an organization, more effort is required in tracking the results of these projects.
• While the Contributions Program has been administered efficiently and economically, opportunities exist to reduce the reporting requirements and suggestions were made related to the timing of the intake process.

4.3 Recommendations

The conclusions of the evaluation led to the following recommendations:

1. That the Contributions Program consider implementing a systematic follow‑up of development projects to track their long‑term impacts on recipient organizations, including progress towards financial self‑sufficiency, without adding significant additional reporting requirements for recipients; and,
2. That the Contributions Program review their risk‑based monitoring approach to ensure that reporting requirements adequately reflect variables such as the amount of funding, the history with the client and the sensitivity and complexity of the project.

Appendix A: Examples of Research Projects Referenced in Policy, Legislative or Regulatory Process or Appearing in Media

Below are summaries of a number of research projects funded through the Contributions Program. They provide examples of the project findings being referenced in policy, legislative or regulatory processes or appearing in media.

Public Interest Advocacy Centre (PIAC) ‑ Paying a Premium: Consumers and Mobile Premium Services ($59,000)

This 2010‑11 report called for better consumer protection for mobile premium services, also known as premium text messaging services. PIAC participated in policy discussions with the telecommunications industry on this issue. On Friday, September 14, 2012, following a five‑month investigation, the Competition Bureau began legal proceedings against Bell Canada, Rogers Communications, Inc., TELUS Corporation and the Canadian Wireless Telecommunications Association, requiring them to stop misleading advertising that promotes costly "premium texting services", and to compensate consumers. The Bureau is seeking full customer refunds and administrative monetary penalties — $10 million each from Bell, Rogers and Telus, and $1 million from the Canadian Wireless Telecommunications Association.

Homeowner Protection Centre (HPC) – Domestic Hot Water Tanks and Other Equipment: A Consumer Perspective ($76,200)

In 2012‑13, HPC's report looked at the issue of door to door sales of water heaters, the second most frequent complaint received by the Ontario Ministry of Consumer Services in 2012. The Ontario government received more than 3,200 written complaints and verbal inquiries about door‑to‑door water heater rentals that year. The report suggests that without the necessary information to assess new choices and without safeguards and penalties to address market misconduct, homeowners who rent their water heaters are left to pay higher prices and endure poorer service. News outlets (i.e. Toronto Star, Canada.com) and CTV Toronto reported on the findings and recommendations of the study.

The Ontario Minister of Consumer Services announced on April 11, 2013 that there would be strengthened protections against aggressive door to door sales of water heaters. Excerpts from the Second Reading – Bill 55, Strong Protection for Ontario Consumers Act, 2013, made reference to HPC and the work they are doing in this area. The HPC report was referred and shared with Committee members of the Government of Ontario – Standing Committee on the Legislative Assembly on Bill 55. A number of changes to the regulations that affect the industry were recommended. Bill 55 was passed by the Ontario Legislature on December 12, 2013. HPC's report helped in framing the issue and formed part of the advice from the Ministry to Government regarding regulatory reform.

Automobile Protection Association (APA) – Research Auto Repair Investigation ($75,151)

The APA has produced a number of studies over the years funded through the Contributions Program that have looked at the performance of auto repair facilities (the last being in 2010‑11). Several APA studies of auto repairs have revealed that overselling of parts by the large chain stores occurred in one‑third to one‑half of the stores visited. A report on their investigation aired on W5, a CTV current affairs show and was available on CTV's website. APA met with regulators from Quebec to discuss its findings, following the W5 telecast and the media coverage it received. APA's report provided recommendations for government and industry and strategies that consumers can use to improve the service they receive at repair facilities.

Canadian Consumer Initiative 2012‑13 – Genuine or fake sales? ($25,000)

In Quebec, there is a strong relationship between l'Office de la protection du consommateur (OPC) and consumer organizations with many examples of them working closely together. Since 2006, the OPC has been modernizing the laws and regulations that it administers and the interviewee from the OPC confirmed that the research projects funded through OCA have been useful in contributing to this exercise. One specific example which was included in the fourth phase of the modernization process focuses on misleading advertising practices, an issue which was examined by Option consommateurs (OC) under the Contributions Program in 2011. The OPC proposed modifications to the Loi sur la protection du consommateur to further clarify these misleading practices using a verbatim recommendation from the 2011 OC research report that was also the base document of OC's CCI development project in 2012‑13. The public consultation document seeking input on the fourth phase of the modernization process also referenced the OC report when detailing the issue. The fourth phase has yet to be concluded. The OPC also initiated legal proceedings against the Hudson's Bay Company as a result of the OC report. The company ultimately pleaded guilty in March 2014 and was fined $32,510 for the infraction.

Management Response and Action Plan