Final Evaluation of the Contributions Program for Nonprofit Consumer and Voluntary Organizations

Final Report

January 2010

Approved by the Deputy Minister at the Departmental Evaluation Committee meeting March 26, 2010

Table of Contents

• Executive Summary
  • Background
  • Purpose of the Evaluation
  • Methodology
  • Conclusions
• 1.0 Introduction
  • 1.1 History of the Program
  • 1.2 Program Objectives
  • 1.3 Program Theory and the Strategic Position of the OCA Contributions Program
  • 1.4 Program Structure
  • 1.5 Target Population
  • 1.6 Program Resources
  • 1.7 OCA Program Governance
• 2.0 Scope/Objective of the Study
  • 2.1 Evaluation Objectives
  • 2.2 Scope of the Evaluation
  • 2.3 Management/Governance of the Evaluation
• 3.0 Methodology/Approach
  • 3.1 Overview
  • 3.2 Data Collection Methods
• 4.0 Findings and Analysis
  • 4.1 Issue 1: Relevance
  • 4.2 Issue 2: Achievement of Expected Outcomes
  • 4.3 Issue 3: Demonstration of Efficiency and Economy
  • 4.4 Issue 4: IC Issues: Performance Measurement/RMAF Update/Program Design and Delivery
• Management Response and Action Plan

Appendices

(Note: Annexes are available via an Access to Information Request)

• Appendix 1: List of Project Steering Committee Members
• Appendix 2: OCA Contributions Program Logic Model
• Appendix 3: Final Evaluation Question Matrix
• Appendix 4: Case Study Document Coding Sheet and Interview Questionnaire
• Appendix 5: Document Review Coding Template
• Appendix 6: List of Documents Reviewed—Bibliography
• Appendix 7: List of Key Informant Interviewees
• Appendix 8: Key Informant Interview Guide and Questionnaires
• Appendix 9: Stakeholder Survey Form
• Appendix 10: Program Expenditures for 2005–06 to 2007–08
• Appendix 11: Formative Evaluation Recommendations in Relation to Management Response with Action Taken Since 2005
• Appendix 12: OCA Case Study Final Report
• Appendix 13: OCA Program Impact: 2005–09

List of Acronyms

Executive Summary

Background

In 1997 the Office of Consumer Affairs (OCA) in Industry Canada (IC) established the current Contribution Program for Non-profit Consumer and Voluntary Organizations.

The aim of the Program is to strengthen the consumer's role in the marketplace through the promotion of high quality, independent and timely research on consumer issues and through the encouragement of financial self sufficiency of consumer and voluntary organizations. Non-profit and voluntary organizations are funded through Contributions Agreements. The average annual program funding is $1.69 million for the 80 or so organizations involved in representing consumers on marketplace issues.

Over the past four years¹ a total of $6.45 million was provided for two types of projects: Research Projects (taking an average 85% of available funds) aimed at helping organizations to conduct consumer-based research and to improve their research capacity; and Development Projects (using an average of 15% of the funds). Two categories of Development Projects were supported: Organizational Development aimed at helping organizations become financially self sufficient (i.e. multi-year business plans and feasibility studies); and Consumer Movement Development aimed at increasing collaboration and cooperation among consumer organizations.

Purpose of the Evaluation

This final evaluation provides results for the fiscal years 2005–2006 to 2009–2010. A mid-term evaluation of the Program was completed in 2005. The Contribution Program's current funding cycle expires on March 31, 2011. The evaluation addressed four issues:

1. Relevance: continued need for the Program and the role of the federal government in this area;
2. Performance: achievement of expected outcomes, particularly those expected in the medium-term;
3. Performance: efficiency and economy in program delivery; and,
4. Management: progress made in the implementation of the 2005 evaluation recommendations and the impact of design or delivery adjustments.

Methodology

Five data collection methods were employed: document and literature review; OCA financial database review and analysis; program recipient case studies; key informant interviews; and stakeholder survey. The evaluators integrated the data from these five "lines of evidence" in a triangulation of findings to help draw conclusions and offer recommendations. This increased the reliability of the study by providing an opportunity for the cross-checking and validating of data from one source. Data collected were accurate and of high quality because of the consistency of results obtained using the different methods.

The evaluation was carried out by Capra International Inc. An Evaluation Manager was assigned by the Audit and Evaluation Branch (AEB) of IC as the Project Authority for the evaluation. A member of the AEB evaluation team was embedded in the evaluation work. The evaluation was done in consultation with an Evaluation Steering Committee composed of key stakeholders that met three times during the evaluation process.

Conclusions

Need, Relevance and Role of the Federal Government

The Organization for Economic Cooperation and Development (OECD) indicated that in most member countries consumer organizations are frequently not able to raise the level of resources equivalent to those available to business and industry to effectively research and present positions on marketplace issues. The OECD encourages national governments to provide resources for consumer organizations to research and present positions on marketplace issues.

The objectives of the OCA Contributions Program directly support the role of the Federal Government as a facilitator in the marketplace to ensure the participation of consumers, through consumer organizations, in government policy, regulatory and legislative development. Evaluation findings clearly affirm the legitimacy and importance of the OCA Contributions Program in supporting collaboration among consumer organizations and consumer-focused research particularly in areas where government was considering policy, regulatory or legislative action. No other federal government department or agency has a mandate to act, or generally undertakes activities unilaterally, in this specific area although two federal departments (Health and Justice Canada) have commissioned occasional studies.

Recommendation aimed at improving program relevance:

1. OCA, to enhance its role as a "gateway" for promoting and sharing consumer research, should actively facilitate the coordination of consumer organization input and consumer marketplace research among federal government departments.

Achievement of Expected Outcomes

The Program Logic Model identifies the following medium-term expected outcomes:

• The consumer organizations provide policy advice on consumer issues affecting the marketplace that is credible and used by decision makers.
• Consumer organizations have an increased capacity to represent consumer views on issues affecting consumers in the marketplace.

Regarding the first outcome, OCA-funded research results have been cited in the policy, regulatory and legislative decision making processes both at the federal and provincial levels.

Research results also appeared in the media on many occasions suggesting the Program has had the expected impact. However, improvements are required in two areas:

• The dissemination of project results should be expanded.
• The collective knowledge acquired through the conduct of consumer research should be retained within the staff of beneficiary organizations.

Regarding the second outcome, an increased capacity for organizations to represent consumers on marketplace issues was partially realized through the establishment of the Canadian Consumer Initiative (CCI), a coalition of consumer organizations that effectively coordinated many of the consumer representational activities and improved collaboration. In pooling their resources, organizations will be able to achieve a critical mass. However, the capacity of consumer organizations to represent consumers was constrained by resource limitations stemming from a continued dependence on government funding and a failure to move any closer to financial self sufficiency, something that was expected in the OCA Program Logic Model.

The Research Component seemed to have the most immediate and direct impact on public policy, regulatory decisions and legislative initiatives. The other two components (i.e., Organizational Development and Consumer Movement Development) were needed as "enablers"—something that helped consumer organizations to gain self-sufficiency and collaborate effectively on marketplace issues.

Recommendations aimed at improving program achievement are:

2. Efforts should continue to be made to facilitate and improve the dissemination of Research Contribution Program project results.

3. To more effectively build research capacity (staff with the right mix of research skills) and the organizational capabilities of non-profit and voluntary consumer organizations, OCA should consider funding research and organizational development projects that extend more than one year for the same program beneficiary.

4. OCA should consider directing more of the current funding to the establishment of organizations similar to the CCI but constituted by different sets of consumer organizations.

Achievement of Efficiency and Economy

As a percentage of the total program budget, the overhead/administrative costs of the program, including salary and O&M (operations and maintenance) averaged 8% over three years. This amount was within the requirements specified in the original program documentation. Small changes in program administration might be made to improve efficiency but they will not substantially alter the overall ratio of overhead to contribution expenditures.

¹ 2005–06 to 2008–09 were the last full years of funding. (Return to Reference 1)

1.0 Introduction

The Financial Administration Act requires that all Transfer Payment programs be evaluated over a five-year cycle with a focus on relevance and program performance. The current form of the Contributions Program for Non-profit Consumer and Voluntary Organizations was last evaluated in 2005. The Contribution Program's current funding cycle expires on March 31, 2011. This final evaluation was completed for the period of 2005–2006 to 2009–2010.

1.1 History of the Program

In 1985 the United Nations formally recognized the importance of ensuring that the consumer voice is heard in the making and execution of government policy by adopting its Guidelines for Consumer Protection.² Canada is a signatory to the Guidelines. Further, Canadian government policy requires that Canadians be provided an opportunity to participate in developing or modifying policies, regulations and regulatory programs.³ In this context, consumer organizations have an important role to play as a voice of the consumer in the public policy process. Consequently, the federal government has provided funding over the years to consumer organizations to help them undertake the research needed to support their effective representation on issues of concern to consumers.

The Federal Government has been providing financial assistance to consumer organizations for over forty years. In the 1970's, following the creation of the department of Consumer and Corporate Affairs Canada (CCAC), the primary objective of the funding program was to strengthen the consumer movement in Canada by assisting non-profit community organizations to provide direct services to consumers (e.g., consumer information and education, redress and other assistance). Government support consisted of grants aimed at sustaining voluntary organizations working for Canadian consumers.

In 1997 the Office of Consumer Affairs (OCA) in Industry Canada (IC) established the current Contributions Program for Non-profit Consumer and Voluntary Organizations. The aim of the current program is to strengthen the consumer's role in the marketplace through the promotion of high quality, independent and timely research on consumer issues and by encouraging the financial self sufficiency of consumer and voluntary organizations.⁴

1.2 Program Objectives

In the Industry Canada Management Resources and Results Structure (MRRS) the strategic outcome to which the Contributions Program for Non-profit Consumer and Voluntary Organizations contributes is: "ensuring that consumer interests are protected and promoted throughout Canada". The MRRS also indicates that the program's chief performance indicator is the number of major challenges that affect the consumer interest in Canada being addressed and/or documented.

In the RMAF (2005) the program mandate is stated as:

1.3 Program Theory and the Strategic Position of the OCA Contributions Program

Canadian Federal Government policy⁵ and Organization for Economic Cooperation and Development (OECD) documents⁶ suggest that consumers need to have the information and skills to engage effectively in the marketplace. The OECD suggests that when consumers make informed choices they contribute to effective competition and well-functioning markets. The OECD proposes that it is in the best interest of governments, consumers and businesses that consumers are empowered with an awareness of their rights, knowledge of how to function and cope in the marketplace, and with the ability to be proactive on issues that are of concern to them.

The OECD has noted that "… consumer associations appear to be one of the most important stakeholders for educating consumers", and "Consumer associations frequently serve as consultative or advisory bodies to governments."⁷ However, in our economies producers and sellers and often governments have more information, and a greater capacity to use that information, than consumers and consumer organizations.⁸

The OECD observed from a recent survey of members⁹ that consumer organizations in most of the countries are frequently not able to raise the level of resources similar to those available to business and industry associations to effectively research and present positions to governments on marketplace issues of concern to consumers. In the survey, many member countries suggested that it is the role of national governments to provide financial assistance to assist consumer organizations in undertaking those kinds of activities.

The OCA Contributions Program is a response in support of the Canadian Government policy that Canadians have the opportunity to participate in developing or modifying policies, regulations and regulatory programs, and that consumer organizations have the capacity to serve as a voice of the consumer in the public policy process.¹⁰ The Program is positioned in Industry Canada to ensure that "consumer groups and other non-government organizations (NGOs) can provide effective input into policy development and play their role in creating demanding consumers, largely through the strategic use of the Contributions Program for Non-Profit Consumer and Voluntary Organizations…"¹¹.

Federal and provincial government departments use contracts to purchase goods and services that are consumed by the government itself. Contributions and grants are used when some other element within society is the intended beneficiary, but not the government per se. The OCA Contributions Program is intended to lever in-kind effort on the part of consumer groups and to help the organizations build their expertise in marketplace issues. Industry Canada is not the main consumer of this work. Thus, a contributions program, rather than government contracting, is the appropriate financial instrument to use.

1.4 Program Structure

Two types of Project Contributions are awarded under the Program¹²: Research Project Contributions, and Development Project Contributions.

1. Research Project Contributions

Research Project Contributions support quality research and analysis directed at consumer protection issues which are national in scope. They improve the capacity of consumer organizations to carry out such activities. Projects aimed at supporting educational activities, conferences, workshops, exhibitions and promotional materials, magazines, guides, folders, web sites, training programs and the like are not eligible.

2. Development Project Contributions

Development Project Contributions are aimed at institution and capacity building for consumer and voluntary organizations, and fall into two categories.

• Organizational Development Project Contributions are awarded to encourage organizations to reach financial self-sufficiency through the development of multi-year business plans, by carrying out feasibility studies for new products and services (including online services), or by supporting pilot projects with the potential to generate revenues and increase an organization's visibility.
• Consumer Movement Development Project Contributions strengthen recipient organizations' ability to work collaboratively on joint plans and initiatives regarding major consumer issues. These projects must be jointly sponsored and governed by two or more eligible consumer associations, with one participant designated as the lead organization.

Eligible organizations are those which:

• Are incorporated as non-profit corporations in Canada;
• Demonstrate competence, credibility, and accountability in carrying out projects;
• Have a governance structure which assures accountability to a membership representing the consumer interest¹³;
• Are capable of communicating the results of their work to key stakeholders or the general public; and,
• Are guided by objectives which are consistent with the program objective of strengthening "the consumer's role in the marketplace through the promotion of timely and sound research and analysis and the financial self-sufficiency of consumer and voluntary organizations."

1.5 Target Population

Canadian non-profit consumer and voluntary organizations working in the consumer interest are eligible under the program¹⁴.

Working in the consumer interest includes activities such as:

• Providing objective information and education which enable consumers to protect their interests in the marketplace;
• Responding to consumers enquiries and complaints;
• Undertaking independent research and analysis of current consumer protection issues and concerns; and,
• Promoting and representing the consumer interest before industry and government.

1.6 Program Resources

From 2005–06 to 2007–08 the annual Contributions Program budget averaged $1.64 million of which an average 85% was expended annually in Research Project Contributions and 15% in Development Contributions. During that period the cost of operating the Program (salaries, operations and maintenance) averaged 8% of the total OCA expenditures (excluding one-time expenses). ¹⁵ In total 81 organizations submitted 277 proposals of which 145 (or 52%) were funded.¹⁶. In the last full fiscal year (2008–09) the Contributions Program provided $1.3 million in Research Project contributions and $170,000 in Development Project contributions. These contributions went to 10 different consumer organizations to carry out 32 research projects and five development projects.¹⁷ Tables 1.6.1 and 1.6.2 give an overview of program activity and expenditures for the three fiscal years for which complete data were available, broken out by fiscal year, component and province.

1.7 OCA Program Governance

Formal accountability for program management rests with the Director General of the Office of Consumer Affairs. Financial management is conducted in accordance with Ministerial delegation of authorities in the department.

An Office of Consumer Affairs Program Management Committee is advisory to the OCA Director General. It is chaired by the Director General of OCA, and composed of two OCA Directors (Director of Consumer Policy and Director of Consumer Services and Outreach), and the OCA senior officer responsible for administration of the Contributions Program.

The Committee advises the Director General on all matters relating to the Program, including approval and management of the RMAF and Risk Based Audit Framework (RBAF); program management, priorities, and evaluation; organization and project assessments; funding recommendations; and disbursement or reallocation of reserves¹⁸.

² Robert S. Friedman, "Representation in Regulatory Decision Making: Scientific, Industrial, and Consumer Inputs to the F.D.A.", Public Administration Review, May/June 1978, pp: 205–214. (Return to Reference 2).

³ The Contributions Program for Non-Profit Consumer and Voluntary Organizations, Results Based Management and Accountability Framework (RMAF) Revised November 4, 2005. (Return to Reference 3).

⁴ The Contributions Program for Non-Profit Consumer and Voluntary Organizations, Results Based Management and Accountability Framework (RMAF) Revised November 4, 2005. (Return to Reference 4).

⁵ The Contributions Program for Non-Profit Consumer and Voluntary Organizations, Results Based Management and Accountability Framework (RMAF) Revised November 4, 2005. (Return to Reference 5).

⁶ OECD, Promoting Consumer Education: Trends, Policies and Good Practices, 2007. (Return to Reference 6).

⁷ OECD, Promoting Consumer Education: Trends, Policies and Good Practices, 2007. (Return to Reference 7).

⁸ Hadfield, R. Howse and M. Trebilcock, Rethinking Consumer Protection Policy, University of Toronto, 1996. (Return to Reference 8).

⁹ OECD, Promoting Consumer Education: Trends, Policies and Good Practices, 2007. (Return to Reference 9).

¹⁰ The Contributions Program for Non-Profit Consumer and Voluntary Organizations, Results Based Management and Accountability Framework (RMAF) Revised November 4, 2005, p. 3. (Return to Reference 10).

¹¹ OCA Work Plan for 2007–2009 at www.ic.gc.ca/eic/site/oca-bc.nsf/eng/ca02286.html. (Return to Reference 11).

¹² Contributions Program for Non-profit Consumer and Voluntary Organizations, Terms and Conditions, dated November 18, 2005. (Return to Reference 12).

¹³ Office of Consumer Affairs, Contributions Program for Non-profit Consumer and Voluntary Organizations, Eligibility Criteria. Available online at: http://www.ic.gc.ca/eic/site/oca-bc.nsf/eng/h_ca00175.html. (Return to Reference 13).

¹⁴ Contributions Program for Non-profit Consumer and Voluntary Organizations, Terms and Conditions, dated November 18, 2005. (Return to Reference 14).

¹⁵ Financial data on Contributions Program Expenditures and Operating Costs provided by OCA May 2009. (Return to Reference 15).

¹⁶ Organizations Matrix Excel File for period 2005–06 to 2007–08, created by OCA February 18, 2009. (Return to Reference 16).

¹⁷ Funded Projects 1997–2009, Excel Spreadsheet of Projects, created by OCA December 11, 2008. (Return to Reference 17).

¹⁸ The Contributions Program for Non-Profit Consumer and Voluntary Organizations, Results Based Management and Accountability Framework (RMAF) Revised November 4, 2005. (Return to Reference 18).

2.0 Scope/Objective of the Study

2.1 Evaluation Objectives

To meet the requirements of the most recent Treasury Board (TB) policy on the evaluation of Transfer Payment Programs, the following issues were addressed in this evaluation¹⁹:

1. Relevance: continued need for the Program and role of the federal government in this area;
2. Performance: achievement of expected outcomes, particularly those expected in the medium-term;
3. Performance: efficiency and economy in program delivery; and,
4. Management, design and delivery: progress made in the implementation of the 2005 evaluation recommendations and the impact of design or delivery adjustments.

Industry Canada approved a Results-Based Management and Accountability Framework (RMAF) for the Contributions Program for Non-profit Consumer and Voluntary Organizations in 2000. The RMAF (last revised in 2005) provides a results-based logic model (reproduced in Appendix 2) that identifies the chain of results emanating from discrete program activities, from outputs to eventual outcomes and key results.

The performance measures cited in the RMAF formed the basis for the matrix of evaluation questions and the performance indicators that guided data collection and analysis (Appendix 3).

2.2 Scope of the Evaluation

The evaluation period extends over fiscal years 2005–06 to 2009–10. The evaluation target population included: OCA staff; members or affiliates of the 80 or more consumer groups that received OCA funding, submitted proposals, or expressed interest in the program's activities; representatives of other federal or provincial government departments involved in areas where consumer protection or consumer awareness was of interest (e.g. Justice, Health); and, legislators, public board or committee members involved in areas potentially influenced by consumer research and advocacy.

2.3 Management/Governance of the Evaluation

The evaluation was carried out by Capra International Inc. An Evaluation Manager was assigned by the Audit and Evaluation Branch (AEB) of IC as the Project Authority for the evaluation. A member of the AEB evaluation team was embedded in the evaluation work. The evaluation was done in consultation with an Evaluation Steering Committee composed of key stakeholders that met three times during the evaluation process. The list of Evaluation Steering Committee members is provided at Appendix 1.

¹⁹ Treasury Board of Canada, Directive on the Evaluation Function, April 2009. (Return to Reference 19).

3.0 Methodology/Approach

3.1 Overview

To address the questions outlined in Appendix 3 both qualitative and quantitative data were collected by several different means and from several sources. These multiple lines of evidence were integrated in a triangulation of findings to support conclusions and recommendations. In triangulating findings the evidence obtained for each evaluation question was analyzed to estimate its reliability and the influence of biases. Findings from a particular method that deviated from those obtained using the other methods were investigated and reasons for the deviation explained.

3.2 Data Collection Methods

Five data collection methods were used:

1. Program Recipient Case Studies and a Subsequent Update;
2. Document and Literature Review;
3. OCA Financial Database Review and Analysis;
4. Key Informant Interviews; and,
5. Stakeholder Survey.

3.2.1 Program Recipient Case Studies

Program recipient case studies were undertaken in the January to March 2009 period. Six cases were selected purposefully to ensure equal representation in Ontario and Quebec (where 95% of program funds were expended), to include one-time and repeat recipients, and to represent each of the three types of program activities. Project success was not a criterion in selection²⁰.

The sample cases were:

• In Quebec: Action pour la solidarité, l'équité, l'environnement et le développement Équiterre; Union des consommateurs UC; and Les Éditions Protégez-vous.
• In Ontario: Infant Feeding Action Coalition INFACT; Public Interest Advocacy Centre PIAC; and Consumers Council of Canada CCC.

For the six cases the analysts reviewed documents provided by OCA, conducted a site visit and interviewed individuals in the organizations. A separate report on the studies was submitted to IC prior to the commencement of the current full evaluation (contained at Appendix 13). An update of the case studies was completed during the full evaluation by having one member of the evaluation team review the case studies final report as well as the individual case studies themselves.

The document review template and the interview questionnaire for the case studies are shown at Appendix 4.

3.2.2 Document and Literature Review

The document review was critical in establishing the official expectations for the Program which could then be compared to the actual expectations of staff and beneficiaries that unfolded during the period of the evaluation (determined from interview and survey data). The document review was aimed at uncovering evidence of the impact of the various Contributions Program projects on public discussions and/or decisions of government bodies related to consumer issues.

An examination of project files for the four years of 2005–06 to 2008–09 was done. In that period a total of 277 project proposals were received and 145 projects were approved²¹. The evaluators were able to obtain 110 project files for which they did a cursory review to establish the type of content and level of completeness. Subsequently, a total of five approved project files per fiscal year and five rejected project files were randomly selected for each of the four years. These 40 files were examined in detail.

The sample of approved projects was investigated most closely to locate evidence, if any, of the impact of the project outcomes on public discussion of consumer issues (e.g. research report used in media coverage, submission of brief or an appearance at a committee hearing, etc.).

A media analysis was done to assess the extent to which program projects had an impact on public discussions, policies and regulatory decisions related to consumer protection issues. The OCA program staff provided the evaluation team with numerous articles and press releases published by government bodies and other organizations as well as a comprehensive report on impacts (contained at Appendix 14). These were reviewed and an estimate made of the potential impact on public dialogue.

In addition, a brief scan of recent literature related to consumer issues and consumer research was done with a particular focus on the Organization for Economic Cooperation and Development (OECD) countries aimed at identifying programs similar to the OCA Contributions Program. Several different types of programs for facilitating consumer protection and information research were found.

In order to facilitate the collection of comparable data in the document review, a coding structure (also called a data collection template) was developed and applied. The template is provided at Appendix 5. A list of the documents and literature reviewed is given at Appendix 6.

3.2.3 OCA Financial Database Review and Analysis

To assess cost-effectiveness and economy the Contributions Program expenditures for fiscal years 2005–06 to 2007–08 (the last three years for which a complete record was available) were analyzed. The Contributions Program expenditures were available for each of those fiscal years by individual project beneficiary and by project type (research, organizational development, consumer movement). For the research projects specifically, the expenditures were presented by the consumer area of interest (e.g. Food & Drug). Data were also available showing the number of program applicants by program type by fiscal year, and the number of applications accepted and rejected each year. The Program expenditure data were used to determine if there were any trends over time.

As well, summaries of OCA annual operational expenditures (operations, maintenance and salaries) over the same fiscal years were analyzed enabling the calculation of the ratio of administrative costs to Contributions Program expenditures. This was done to assess OCA performance against the expectations contained in the Terms and Conditions for the Program (section 15):

3.2.4 Key Informant Interviews

OCA staff identified 32 individuals that had a sufficient knowledge of the program and its context to serve as key informants (called the experimentally accessible population). Twenty-two individuals were to be interviewed. In the end, 24 key informants (or 75% of the accessible population) completed the interviews with representation as shown in Table 3.2.1. A list of the interviewees is provided at Appendix 7.

Three tailored key informant questionnaires were used, one for each of the three groups included in Table 3.2.1. All versions (in English and French) were approved by IC prior to commencing the interviews. They are contained at Appendix 8.

3.2.5 Stakeholder Survey

The target population for the stakeholder survey included a total of 124 individuals as follows (names and contact information provided by OCA staff):

• Members or affiliates of the 80 or more groups²² that had received OCA funding, submitted proposals, or expressed interest in the program's activities;
• Representatives of other federal or provincial government departments that were involved since 2005 in public policy and/or government regulation of areas where consumer protection or consumer awareness was of interest (e.g. Justice, Energy, Health); and,
• Legislators, policy officers and other decision-makers on a variety of public boards or committees involved in areas potentially influenced by consumer research and advocacy.

A web-administered survey questionnaire was developed. It was approved by IC prior to use and is displayed at Appendix 9. The questionnaire contained different sets of questions that were relevant to the different groups of respondents cited above.

Using the list of 124 names with contact information provided by OCA staff, the evaluation team sent a notification letter signed by IC to 117 individuals for whom complete email addresses were available. After a three-week period potential respondents that had not completed the survey were sent a follow-up email. Two additional email reminders were sent and the closing date for the survey extended to allow maximum time for potential respondents to complete a survey. The survey response statistics are presented in Table 3.2.2.

Survey respondents represented a broad spectrum of stakeholders as shown in Table 3.2.3.

3.2.6 Limitations of the Methodology

Randomization in the selection of individuals for the key informant interviews was not possible because of the small number of individuals in IC, other government departments, and among consumer organizations that had a comprehensive knowledge of the Program and its impacts. The evaluators interviewed a total of 24 people, and this represented 75% of the 32 individuals the OCA program staff identified as having the level of knowledge required to address the program and impact issues. This provides confidence that the views of the potential respondents were adequately represented by those interviewed.

Six case studies were completed prior to commencement of the full evaluation. However, the six studies represent 11% of the estimated total number of consumer groups that received funding over the previous five years. This provides a level of comfort that the responses to questions and data collected through documents and on site reviews for the six groups reflect what one might have expected to find had the number of studies been greater.

The final number of stakeholder online survey respondents was 37 out of a potential pool of 117. This was a 32% response level. While it would have been preferable if the number of respondents had been greater, the response rate is sufficient for the purposes of this evaluation. The limited proportion of potential respondents that actually completed the survey dictates caution in generalizing any of the findings from the survey alone.

Because the findings from multiple lines of evidence were triangulated to draw conclusions for the key questions, the limitations of the different data collection methods cited above did not have a detrimental impact on the accuracy and reliability of the conclusions. Through the crosschecking and validating of data from one source with that from the other sources the data were shown to be accurate and of high quality. The study is considered to be reliable and of good quality.

²⁰ Case Studies Non-Profit Consumer and Voluntary Organizations, Final Report, dated March 31, 2009. (Return to Reference 20).

²¹ Organizations Matrix created by OCA February 18, 2009—Excel spreadsheet of all projects funded and applications made. (Return to Reference 21).

²² Organizations Matrix created by OCA February 18, 2009—Excel spreadsheet of all projects funded and applications made. (Return to Reference 22).

4.0 Findings and Analysis

The Evaluation Question Matrix at Appendix 3 contains 28 questions. Many of these were combined in the analysis process, resulting in a total of 9 sets of questions that are addressed in this section of the report. The question numbering in this section does not correspond directly with the numbering in Appendix 3 because of the consolidation of questions.

4.1 Issue 1: Relevance

Conclusion

Findings

In the OECD publication "Promoting Consumer Education: Trends, Policies and Good Practices" the authors note that it is in the best interest of governments, consumers and businesses that consumers are aware of their rights, know how to function and cope in the marketplace, and have the ability to be proactive on issues that are of concern to them. The OECD observed that most of its countries have an institutionalized framework to foster consumer education and effective representation. The OECD pointed out²³ that national governments need to provide national leadership and financial support for consumer education activities including consumer research and the development of consumer positions on marketplace issues. And non-governmental stakeholders were observed to be key partners in this area.

From a survey of member countries, the OECD noted²⁴ that consumer organizations are frequently not able to raise the level of resources similar to those available to business and industry associations to effectively research and present positions to governments and industry on marketplace issues of concern to consumers. In the survey, member countries suggested it is the role of government to financially assist consumer organizations in undertaking those kinds of activities. Similarly, the United Nations (UN) recognized the importance of ensuring there is a consumer voice in the making and execution of government policy in its 1985 "Guidelines for Consumer Protection", to which Canada is a signatory.

The OCA Work Plan for 2007–2009 supports the UN call for a consumer voice and the OECD's emphasis on the important role of central governments in this area. It states²⁵: "The Office of Consumer Affairs will continue its efforts to ensure that consumer groups and NGOs can provide effective input into policy development and play their role in creating demanding consumers, largely through the strategic use of the Contributions Program for Non-Profit Consumer and Voluntary Organizations…"

Consumer organization representatives unanimously affirmed in the case studies and key informant interviews their belief that it is a valid role for the federal government to support the development and expansion of organizations that represent consumers, and to fund consumer-based, independent research. They said that without OCA Contributions Program funding the consumer organizations would not have been able to develop research findings and positions free from industry influence, and the volume and quality of research would have been less than was achieved.

In fact, OCA project administrative files and financial records show that demand for program support has remained strong over the past decade. The annual budget has remained at approximately $1.69 million for the entire period of 2001–02 to 2006–07. For the five-year period leading up to the 2005 program evaluation the number of applications averaged about 98 with about 33–42% being approved for an annual total of 40 funded projects. From 2005 to 2009 roughly the same number of projects received funding, totaling annually between 38 and 40. The program continues to be over subscribed.

In the six case studies no significant overlap of OCA funding with program assistance offered by other governments or agencies was noted. In two cases additional funding was identified as being available from other departments, notably Justice Canada and Health Canada. Several of the organizations studied received financial support from provincial government ministries or agencies such as the Government of Ontario (energy efficiency agency), Office of the Privacy Commissioner of Ontario, the Ontario Trillium Foundation, Ontario Hydro and the Office of the Privacy Commissioner of Canada. But in the majority of these cases the funding was for the production of items like brochures that related to the research projects. These other government departments and agencies generally supported research targeted at assisting them to develop "community sensitization" plans in specific areas (e.g. eco-action by Environment Canada; the role of the individual in public health by Health Canada; or privacy rights for the deaf, elderly and youth by the Privacy Commissioner).

It appears that other federal departments funded relatively little consumer research. The organizations studied suggested that having the OCA office manage all federal government consumer projects, including those related to ancillary product development, would increase efficiency through a "one-stop shop" approach and at the same time maintain the "arms length principle" from the line departments that might have a vested interest in certain positions on consumer issues. The suggestion was made that OCA could act as "gateway" to funding from other federal departments that require consumer-based input to marketplace policy or legislative issues. This approach is consistent with the recent federal initiative on "harmonization among federal departments".

The final case studies report also notes that there was a consistency between what the organizations received for funding research projects and government "priority areas".

Key informant interviewees all agreed on the continued need for the program. Similarly, 17 (or 90%) of the stakeholders responding to the online survey said that there is "definitely a need" or a "great need" for the Program. And 18 out of 25 (or 72%) of the responses to the question about the ability of non-profit consumer and voluntary organizations to effectively represent consumers on marketplace issues indicated they could do so only to a "small extent" without OCA Contributions Program assistance. One survey respondent included the following statement about the need for the Program:

Key informants and some case study respondents identified several areas that they saw as program "gaps", none of which are major in the view of the evaluators.

• Consumer organizations cannot apply for a communication project (i.e. they cannot be funded to print and distribute reports and engage in other activities such as workshops to promote their completed work after research project wind-up).
• Each contribution agreement is limited to $100,000 per project and this has not changed in a number of years.
• The OCA program did not formally fund face to face meetings among consumers groups and it did not do much to help them in networking to learn from each others' experiences. The exception to this was the funding of CCI. That kind of activity was critical to the development of the capacity of consumer organizations to effectively develop and present positions on marketplace issues to government and industry.

Recommendation

²³ OECD, Promoting Consumer Education: Trends, Policies and Good Practices, 2007. (Return to Reference 23).

²⁴ OECD, Promoting Consumer Education: Trends, Policies and Good Practices, 2007. (Return to Reference 24).

²⁵ ARCHIVED—Work Plan 2007-2009. (Return to Reference 25).

4.2 Issue 2: Achievement of Expected Outcomes

Conclusion

Findings

The Research Component might be expected to have the most immediate impact on public policy, regulatory decisions and legislative initiatives because its purpose is "to improve the capacity of consumer organizations to represent the interests of consumers in the marketplace decision-making process through sound research and analysis of consumer protection issues…". On the other hand, the purposes of the Organizational Development and the Consumer Movement Development components are to help consumer organizations attain sustainability without compromising their independence, and to contribute to effective collaboration among the organizations. These purposes are equally as important as the Research Component aim, but serve as an "enabler" for equipping consumer organizations to develop and present positions on consumer issues.

The perceived need to support capacity building for non-profit and voluntary organizations was seen in a recent shift of focus in OCA funding towards assisting consumer organizations to increase their self-sufficiency. Even though the total number of Research and Organizational Development projects approved by OCA declined by 23% from 2005–06 to 2007–08, the number of Organizational Development projects increased proportionately from 12% to 27% over the same time frame. This shift is displayed in Figure 4.2.1.

Figure 4.2.1: Proportion of Projects Approved by the Two Types of Projects

Description Link

Key informants generally observed that Canadian consumer organizations are small and do not have a solid membership base. They are largely dependent on the government for funds to conduct research. Without OCA help to undertake work aimed at building memberships and securing other forms of financing for their operations, key informants suggested the organizations will remain dependent on the OCA Contributions Program.

A common remark from interviewees was that some organizations that have received repeated funding over the years have failed to develop a business model that would help lead them to self sufficiency. Several respondents suggested that in the Canadian marketplace non-profit consumer and voluntary organizations need to find a means of developing long-term funding that will allow for sustainable activity in order to build their capacity for effectively representing consumers regarding marketplace issues.

Conclusion

Findings

One method for making OCA-funded research project results known was that of listing the projects in the Consumer Policy Research Database on the OCA website. The website is open to the public.

In the impact and media analysis the evaluators acquired a diverse selection of examples where research information derived from OCA-funded projects was cited and/or referenced by consumer organizations at various venues. The total number of examples found is shown in Table 4.2.1. A detailed listing of examples by organization and venue is provided at Appendix 14.

To facilitate the tracking of the impact of their projects, the beneficiaries could provide evidence, if any; of the project results quoted in media reports or cited by decision makers.

Case Study beneficiary organizations said they tried to ensure that project findings did not simply "sit on a shelf". For example, in its second OCA-funded study called "Ecological ornamental horticulture: Getting to know consumers to guide their decision-making" Équiterre directly informed the large chain stores that sell decorative horticultural products, a number of consumer and environmental associations, and the relevant federal and provincial government departments of their research report and findings. Similarly, the Consumers Council of Canada (CCC) made the results of six research studies public by using press releases. The press release on Nanotechnology was found on the internet on several different news and magazine sites (e.g. CTV interview on energy efficiency and nanotechnology). The report on "Nanotechnology and Its Impact on Consumers" was picked up by the National Post, and the primary CCC author delivered a number of speeches on the report, both nationally and internationally.

The Public Interest Advocacy Centre (PIAC) categorized the impact of the results of its 20 projects on the extent of use by stakeholders (i.e. extent of possible changes made in regulatory or policy development) using a scale from 1 (low) to 10 (very high) and the average rating was 6.1. This reflects the PIAC's belief that the results of projects have been used by stakeholders.

The Infant Feeding Action Coalition (INFACT) stated its belief that the results of its research on "Impact of Health and Nutrition Claims on Infant Feeding Practices" had been broadly used by Health Canada. INFACT provided its results to the World Health Organization (WHO) and was involved with that body in policy development on international standards related to the infant formulae. The results were also used in the EU and the feedback from that organization on the study results were positive due to the systematic approach of the research.

The Union des consommateurs (UC) suggested that the results of consumer research were useful when the time came to advocate for action on particular issues. Solid findings from systematic research supported with OCA funding made their representations to government credible. For example, the UC research results were the foundation of the positions the organization and its collaborators presented to government departments regarding eco-labelling of food products and eco-energy labelling on vehicles.

Like the case study respondents, the key informants suggested that research results were generally made broadly available. A total of 76% (18 of 24 interviewees) indicated that the Program had "greatly increased" or "somewhat increased" the availability of quality research. Some respondents stated that funding had produced research that allowed consumer groups to influence policy development and legislation specifically in regard to amendments to the Competition Act and the National Building Code (energy efficiency). Other respondents expressed the view that some project reports were not read by the audience that should have been interested and had only a limited impact on policy or legislative decision-making. These respondents suggested that the reason for poor dissemination of findings was the cost of mass communication—that without core funding from government to operate their organizations they were less able to publish magazines containing the research findings and to build awareness of their websites among the public.

Regarding program impacts, 72% (or 17 of the 24 key informants) felt that the Program had "greatly increased" or "somewhat increased" the use by decision makers of quality research on issues of interest and/or concern to consumers. Respondents remarked that decision makers would not have made the choices and/or recommendations they did in the absence of the information presented to them by consumer organizations that was developed from the OCA-funded research.

Online survey respondents provided results similar to the interviewees. Twelve out of 16 (or 75%) of respondents indicated that the Program had "definitely an impact" or a "great impact" on the eventual availability of consumer-based marketplace research. And 12 of 16 respondents (or 75%) indicated that the Program had "definitely an impact" or a "great impact" on increases in the capacity of non-profit consumer and voluntary organizations to distribute their consumer research. Furthermore, 15 out of 19 respondents (or 79%) indicated that decision-makers generally make "some good use" or "very good use" of information from consumer organizations.

Overall, consumer organization representatives suggested that much more could be done to see that the valuable results from OCA-funded research would find their way into the decision making process. This is a question, they noted, of the OCA Program providing the additional funding required for organizations to actively promote the use of research results. For example, eight survey respondents suggested that the funding of research projects had ended before the organizations could make the results broadly known. One respondent wrote:

Recommendation

Conclusion

Findings

Through its two research studies (one on consumer choices of eco-friendly food products and one related to eco-friendly ornamental horticultural products) Équiterre said it achieved a level of recognition that allowed it to work with other organizations, the related industries and retail chains to do further research. For example, a large chain called upon the organization to advise it on the products to be promoted or avoided given the level of sensitization of the consumers to labelling.

INFACT indicated that its OCA-funded research on infant food nutritional claims made by industry producers contributed to Canada's ability to monitor the infant food industry and determine the need for changes to national, international and WHO standards. This kind of research activity attracted the interest of other federal organizations (Health Canada; CFIA) but

INFACT has not received additional financial support from the federal government. The program suggested that doing such work for these government departments might prevent the organizations from serving as consumer advocates. INFACT said it was the information source for human rights and legal issues involving breastfeeding.

Interviewees noted that some consumer organizations have become well-recognized by specific government bodies. For example, the PIAC gets funding to appear fairly regularly before the Ontario Energy Board and CRTC. However, other organizations stated a belief that a number of important regulatory boards are not aware of their capacity and role in providing advice and information on issues of interest/concern to consumers.

Regarding the level of demand for consumer input by marketplace stakeholders, 53% (or 13 of 24) of key informants indicated that demand had "greatly increased" or "somewhat increased" after their organizations had completed OCA-funded research projects. An important outcome, they said, of the OCA-funded work was that their organizations became recognized by industry and government bodies as "reference groups". But a number of the key informants also suggested that the level of demand for consumer input by marketplace stakeholders has not yet reached its full potential. This view was supported by the finding from the online survey that 5 of 19 (or 31%) of respondents said that the Program had "definitely an impact" or a "great impact" on increases over time in the demand by industry and commercial organizations for research. Half of the respondents (8 or 50%) were not sure if there was any impact while 3 (or 19%) indicated the impacts were small or there were none at all. It seems that many industry lobby groups are still generally dismissive of work done by consumer organizations.

As well, up to 63% (or 15 of 24) of key informants indicated that as a result of undertaking OCA-funded research the number of requests from government departments, agencies, and other public and private sector organizations for this kind of research had "greatly increased" or "somewhat increased". Similarly, 9 of 16 (or 56%) of online survey respondents indicated that the Program had "definitely an impact" or a "great impact on increases over time in the number of requests from government, regulatory bodies or other non-government organizations for consumer organizations to undertake research. And 16 of 19 (or 84%) indicated that government and/or regulatory bodies expressed an interest in information about consumer concerns that was provided by consumer organizations "somewhat often" or "very often".

Interviewees noted that the completion of OCA-funded research projects increased their visibility and credibility. A major step forward in this regard, they said, was the formation through program funding of the Canadian Consumer Initiative (CCI), a coalition of five consumer organizations. The CCI hired a consultant (Rideau Institute) to lead and conduct representations at the various Parliamentary Committees.

Conclusion

Findings

The case studies findings support the view that consumer organizations improved their research capacity as a result of OCA funding. For example, the PIAC estimated that the impact across its 18 research projects on building the organization's capacity for consumer research averaged 7.6 on a scale of 1 (lowest) to 10 (highest). PIAC also asserted that its 18 research projects "significantly increased" the in-house core knowledge. They noted that PIAC did most of the research with in-house personnel and outsourced less than 10% of the work. INFACT provided a similar example. Its research project helped the organization "increase its knowledge base" regarding the use of questionnaires and methods to network with academics. It indicated that such knowledge is vital in assisting in-house consultants in other advisory capacities. It also increased the knowledge within the organization on how to execute larger projects as well as positively impacted its ability on how to handle ongoing projects. A third example was given by UC. Through its research projects the UC said it was able to considerably expand its expertise on the issues researched and to begin to engage other consumer association in work on the issues. In several cases UC did research and developed policy positions that did not at the time fit within the government priorities. But because of the level of expertise the organization and its collaborators had developed, the government was influenced to take the issue into consideration.

The case study findings also support the position that developmental funding improves the capacity of consumer organizations to represent consumers on marketplace issues. The two Development component projects undertaken by Les Éditions Protégez-Vous enabled the organization to develop an internet site that became one of the pillars of its activities. The operation, maintenance and upgrading of the organization's website was crucial to the continued development of the organization's consumer information dissemination role to the achievement of organizational self-sufficiency through increased revenues. Through a second contribution agreement Éditions Protégez-Vous produced a business plan that included organizational development. This project allowed Les Éditions Protégez-Vous to analyze the risks of establishing a membership scheme and to set up a membership process. Similarly, the CCC reported that its organizational development projects increased the organization's core knowledge base and enabled them to hire a project manager. Further, the CCC's exchanges with the media expanded and the CCC staff became better networked with others involved in consumer advocacy as well people in policy-making roles.

Case study findings showed that the interconnectivity with others improved once the Canadian Consumer Initiative (CCI) got underway—an outcome of a contribution of $100,000 from the OCA Program. The CCI funding greatly improved the co-operation between the consumer advocacy groups, adding to their capacity to address key consumer issues. The success of the CCI in adding to the capacity of consumer organizations was also seen in the key informant interview results. For example, 50% (or 12 of 24) of the key informants indicated that overall the OCA Development component projects "greatly increased" or "somewhat increased" their organizations' capacity to effectively represent consumers regarding marketplace issues, and that was largely a result of the creation of the Canadian Consumer Initiative (CCI). The CCI approach was viewed as the way ahead for facilitating collaboration and thereby improving the effectiveness of the organizations.

In general, the findings from the key informant interviews did not indicate as strongly as the case studies that organizations improved their capacity to undertake research and represent consumers as a result of OCA funding. For example, up to 53% (or 13 of 24) of key informants indicated that program funding had "greatly increased" or "somewhat increased" the collective knowledge within their organizations. And 50% (or 12 of 24) indicated that program funding had "greatly increased" or "somewhat increased" the capacity of non-profit and voluntary organizations to undertake quality research. Without the OCA funding they said they would not have gained the necessary knowledge and expertise required to represent consumers in the specific areas that their organizations were generally involved with. On the other hand, some of the interview respondents said that the consultants engaged by the organizations to undertake the research projects, and not organization staff, gained in terms of their "collective knowledge" as a result of the OCA-funded work. The only way the organizations themselves would add to their own collective knowledge was if they were funded on a multi-year basis to carry out a research program, rather than a single project. Then they would hire the expertise on a permanent basis rather than engaging different consultants for each study.

Some 17% (or 4 of 24) of the key informants said their capacity to undertake important research on consumer issues had declined over the past couple of years because fewer of their projects got funded by OCA each successive year. Key informants said they are highly dependent on OCA funding for their consumer research because few other sources of funding exist. They have experienced a funding squeeze that is threatening their ability to do essential research and, as a result, their ability to attract and retain senior research staff and consultants has been diminished. Getting good researchers takes more than one successful research project completed over one or two years.

Several interviewees (17% or 4 of 24) suggested that capacity building was not really possible with OCA funding because funding is only provided on a year by year basis. They said that multi-year funding would afford organizations a better chance of hiring permanent staff, and developing strategies in common with other organizations for research and presentation of positions to government. Another suggested way to facilitate capacity building, is to provide core funding to organizations. The underlying logic of this approach would be for the consumer organizations to acquire members and build a base for obtaining funds from other sources, something that has not been done very well up to the present time.

Online survey respondents were somewhat more positive than key informants about improvements in capacity resulting from OCA-funded projects. Of 16 respondents 13 (or 81%) indicated that the Program had "definitely an impact" or a "great impact" on increases over time in the collective knowledge/accumulated knowledge of non-profit consumer and voluntary organizations. And 12 of 14 respondents (or 86%) indicated that the Program had "definitely an impact" or a "great impact" on the organization's capacity to effectively represent consumers. Four of 8 respondents (or 50%) indicated that the Program had "perhaps a small impact" or had "no impact at all" on increases in financial self-sufficiency. On the other hand, 6 of 8 respondents (or 75%) said that the Program had "definitely an impact" or "a great impact" on increases over time in the level of collaborative work among consumer organizations.

Overall, the evaluation findings suggest that consumer organizations lack the level and consistency of financial support required to maintain their positions as a voice for consumers on a number of key issues. Although self-sufficiency in terms of their financial situation is certainly desired, they have found it difficult to attract and retain the level of paying membership, and to derive adequate revenues from their products and services, to maintain the organizations without government funding. Their funding only covers costs related to a specific project which leads to the situation where consumer groups attempt to submit to OCA as many projects as possible in an effort to get at least one approved to help offset some of their fixed costs.

The fact that financial self-sufficiency is not attainable by many consumer organizations in the near term was also seen in the interview responses and online survey results. For example, 21% (or 5 of 24) of key informants responded positively when asked to rate the progress made by consumer organizations involved in marketplace issues toward becoming financially self-sufficient (i.e. not dependent on OCA funding). It seems that a few of the consumer organizations have succeeded in expanding their membership base and in raising additional funds while the rest have not. Improved visibility and credibility of the organizations has helped them have a greater impact on the resolution of specific issues but that has not translated into substantially more funding from other government departments or agencies or from the public in general.

Two comments from the interviews that may explain the minimal progress made by consumer organizations in reaching self-sufficiency are:

Regarding obstacles to effective representation the most frequently cited one was "lack of necessary funds" given by 7% of online stakeholder respondents. The next most frequently listed obstacle was "insufficient consumer membership to be credible" given by 26% of online survey respondents.

Recommendations

4.3 Issue 3: Demonstration of Efficiency and Economy

Conclusion

Findings

Internationally, it was found that most OECD countries have recognized the need for government to support the development of consumer research and protection of consumers by consumer organizations. For example, Australia has had a centralized approach for many years to the provision of support to NGOs for the conduct of consumer-based research.²⁶ The Government of Great Britain announced that it will publish a Consumer Rights Bill that would go further than an EU Directive on consumer rights.²⁷ Government research had found a low level of awareness of consumer rights. Consequently, consumer groups were asked by Consumer Focus, the Office of Fair Trading and Consumer Direct to develop a campaign to improve awareness of consumer rights, and to urge consumers to use Consumer Direct as their main source of online and telephone advice and support on consumer issues.

The OCA funding for the creation of CCI is similar to the proposal made in Australia to move toward centralization of consumer research and advocacy through the funding of coordinative consumer organizations. The case studies showed that the CCI funding greatly improved the cooperation between the consumer advocacy groups. Through the CCI organizations like UC were able to exchange strategic information and to concentrate their actions on particular issues at the federal level. Formation of the CCI enabled consumer groups to redirect information requests to the organization with the most experience in a particular area. All of the case study respondents supported the idea of the CCI, and a majority of key informants (75% or 18 of 24 respondents) suggested that more collaborative initiatives like that involved with the formation of the CCI are needed.

Conclusion

Findings

A complete record of the expenditures for the Contributions Program and OCA overhead was available only for the three fiscal years of 2005–06 to 2007–08.

The program financial data indicate clearly that demand for funding consistently outstripped the supply of the available (and relatively constant) OCA funds. In Figures 4.3.1 and 4.3.2 the trend over the three years in the rate of project submissions to approvals is shown in terms of funding and number of projects. The value of the funding requests increased by $1,376,898 or 38% between 2005 and 2008, while the actual funding of projects increased by $124,368 or 7.4% for the same time frame. The increase in dollar value of annual funding requested over the three years can be attributed to the 0% increase in average value of project funds sought (see Figure 4.3.3). This is likely a result of an increase in project complexity.

The average number of projects submitted over the three years was 92 per year and can be viewed as constant. The average number of projects approved over the three years was 37 per year. The number approved as a percentage of the number submitted declined by 23%. The decline in the number approved as a percentage of the number submitted was likely due to the increased dollar value in the average amount of funding requested per project that was not compensated by the 7.4% increase in contributions funding made available over the three years.

Figure 4.3.1: Trend in Project Approval Rate ($) over Years 2005–06 to 2007–08

Description of Figure 4.3.1

Figure 4.3.2: Trend in Project Approval Rate (Number of Projects) over Years 2005–06 to 2007–08

Description of Figure 4.3.2

Figure 4.3.3: Average Dollar Value of Submitted and Approved Projects

Description of Figure 4.3.3

The analysis of administrative costs included an assessment of the operations and maintenance (O&M) budget line objects (e.g. translation, telephone, training, publishing and printing, etc.) by fiscal year to determine if there were any trends over time and to possibly identify areas, if any, where improvements in efficiency might be found. The ratio of overhead costs to program costs was also to be calculated to determine if the commitment made in the original program documentation, as contained in section 15 of the Terms and Conditions, was met. The commitment statement was:

In Figure 4.3.4 the total program costs (one-time costs, overhead and contributions funding) for each of the three years from 2005–06 to 2007–08 are shown. Overhead costs as a percentage of total program costs (not including one-time expenses) ranged from 8% in 2005–06, to 10% in 2006–07, and 5% in 2007–08. Over the three years the average percentage was 8% not including one-time costs and 10% including one-time costs and that is within the commitment made in the original program documentation.

Figure 4.3.4: Break Down of Total Program Costs in Dollars and Percent for the Three Years of 2005–06 to 2007–08

Description of Figure 4.3.4

A detailed break-down of expenditures by Program Component and categories of expenditures is given at Appendix 10.

Three of the six organizations included in the case studies suggested improvements in program delivery aimed at improving cost-effectiveness. The suggestions follow.

• The amount of funding provided by OCA for each project is generally considerably less than that required for effective execution. For example, organizations said that their ongoing analysis suggests they generally get $290 and $350 per hour for research work done under contract with other government agencies, with non-profit organizations or for the private sector. The OCA funding only allows for a rate of $50–$60 per hour. This impacts the results of the OCA projects.
• Co-funding with a branch of government implicated in a particular research issue would broaden the funding base for consumer groups. If this is done through the OCA and not directly with the relevant department, concerns about independence of findings would be lessened.
• Perhaps research programs rather than projects can be funded. A calendar or fiscal year is not often the best time frame for completion of research. Greater flexibility is required to allow organizations to do longer-term research. This would help their strategic research planning and reduce the administrative burden of some recipient organizations as they have to file multiple reports on smaller research projects on small size contributions.

With respect to inefficiencies in the ratio of overhead to program costs, some OCA staff suggested in the interviews that:

• The level of contributions funding could increase substantially (e.g. from $1.6 million up to $10 million) without adding appreciably to the overhead (salary) costs;
• The quarterly reporting required creates needless work both for OCA staff and for the consumer organizations and could be reduced to annual reporting;
• The contract documents should specify the list of deliverables and schedule of payments. Then the requirement of having consumer organizations, once the project is approved, justify each expense would not be needed, reducing paperwork; and,
• Requiring methodologist reports for the projects is of questionable value. OCA staff should be able to review project methodologies and suggest changes.

A repeated argument was that the consumer organizations need sustained funding provided as core funding on a multi-year basis. It was said this would help organizations build a sustainable capacity for research and advocacy similar to that found among the consumer organizations in other countries (e.g. US and Britain).

Stakeholders also expressed doubt about the adequacy of OCA funding to meet program objectives. While the responses to the question about adequacy of funding (percentage responses summarized for adequacy to "some/great extent" or "small extent/ not at all") differed for each of the three sub-components, overall less than half (47%) indicated that OCA funding was sufficient "to some or a great extent". A comparison of the responses to the question of adequacy of funding levels for the Program overall, and for each of the three components, is shown in the following graph (Figure 4.3.5).

Figure 4.3.5: Sufficiency of OCA Funding for Program Success Overall and by Component

Description Link

²⁶ Commonwealth of Australia, (2000), Ministerial Council on Consumer Affairs Website (www.consumer.gov.au). (Return to Reference 26).

²⁷ Secretary of State for Business, Innovation and Skills of Great Britain (2009), A Better Deal for Consumers: Delivering Real Help Now and Change for the Future, Government White Paper, July 2009. (Return to Reference 27).

4.4 Issue 4: IC Issues: Performance Measurement/RMAF Update/Program Design and Delivery

Conclusion

Findings

The findings derived from the document review with respect to this question are presented in a table at Appendix 11. The table shows that 10 of the 13 recommendations of the 2005 evaluation were implemented and demonstrable action taken. Two were not implemented because of concerns about privacy issues and no action was required on one recommendation.

The 2005 evaluation report was reviewed by the management team and it was decided that some of the recommendations were not feasible. Those that could be implemented were completed in the first year and a half after the evaluation was submitted.

OCA staff said that the current performance measurement system is, in their view, adequate but needs to be formalized. Improved outcome measurement would be possible if program beneficiaries were required, as a part of their contributions agreements, to provide evidence, if any, of the impact of their projects at the end of one year following project completion.

Conclusion

Findings

The organizations included in the case studies offered the following ideas about the project solicitation and selection process.

• The priorities of consumer associations and not those of the government might be used to determine the projects that are funded. This would put the focus on the protection of consumers' rights rather than on government's desire to respond in particular areas.
• The time frame for the processing of funding requests should allow for the receipt by recipient organizations of project approvals before the beginning of the new fiscal year.
• Improvements could be made to the program guides and related documentation such that these would include a clear indication of the factors to be used in assessing funding requests. This would give the process greater credibility.

Consumer organization key informants suggested that:

• The application guidelines for each year be made available much earlier;
• The application process be year round and should not take place just from October to December of each year;
• Organizations should have the ability to apply online; and,
• Rather than use a lengthy list of criteria for proponents to respond to, the process might occur in two stages: 
  (1) Proponent provide a description of the research issue, its importance, capacity to execute the research, and dollar value; and 
  (2) Those that make it through the first stage would submit their methodologies with help from the OCA.

Management Response and Action Plan

Figure Descriptions

Figure 4.2.1: Proportion of Projects Approved by the Two Types of Projects

The image represents an histogram which the title is: Proportion of Projects Approved by the Two Types of Projects

Return to Figure 4.2.1

Figure 4.3.1: Trend in Project Approval Rate ($) over Years 2005–06 to 2007–08

The image represents an histogram which the title is: Trend in Project Approval Rate ($) over Years 2005–06 to 2007–08

Return to Figure 4.3.1

Figure 4.3.2: Trend in Project Approval Rate (Number of Projects) over Years 2005–06 to 2007–08

The image represents an histogram which the title is: Trend in Project Approval Rate (Number of Projects) over Years 2005–06 to 2007–08

Number of Submitted and Recommended Projects

Return to Figure 4.3.2

Figure 4.3.3: Average Dollar Value of Submitted and Approved Projects

The image represents an histogram which the title is: Average Dollar Value of Submitted and Approved Projects

Average Dollar Value of Submitted and Approved Projects

Return to Figure 4.3.3

Figure 4.3.4: Break Down of Total Program Costs in Dollars and Percent for the Three Years of 2005–06 to 2007–08

The image represents an histogram which the title is: Break Down of Total Program Costs in Dollars and Percent for the Three Years of 2005–06 to 2007–08

Return to Figure 4.3.4

Figure 4.3.5: Sufficiency of OCA Funding for Program Success Overall and by Component

The image represents an histogram which the title is: Sufficiency of OCA Funding for Program Success Overall and by Component

Return to Figure 4.3.5