Evaluation of the Communications Research Centre

Final Report

January 2012

Recommended for approval to the Deputy Minister by the Departmental Evaluation Committee on

Approved by the Deputy Minister on


Appendices (Separate document)
Appendices are available via an Access to Information

Appendices (Separate document)
Appendices are available via an Access to Information
Appendix Title
Appendix 1 Description of Research Programs
Appendix 2 Crosswalk Between the Research Programs and Core Competency Groupings
Appendix 3 Document Review
Appendix 4 Literature Review
Appendix 5 Interview Guides
Appendix 6 Client Survey Questionnaire
Appendix 7 Client Survey—Results
Appendix 8 Managers' Survey Questionnaire
Appendix 9 Review of Other Labs—Questionnaire and List of Participating Organizations
Appendix 10 Excellence Indicators

List of Tables

List of Tables
Table Title
Table 1 2011–2012 Source of Funds
Table 2 CRC Employees by Activity
Table 3 Shirleys Bay Buildings
Table 4 Recent Awards for Scientific Excellence
Table 5 CRC Research Outputs
Table 6 Summary of 2007–2010 Outcomes
Table 7 Why clients choose to work with CRC
Table 8 Campus Operation Costs
Table 9 Profile of Organizations Participating in Review of Other Labs
Table 10 Comparison of Corporate Services Provided Internally

List of Figures

List of Figures
Figure Title
Figure 1 Overview of CRC Budget Allocation, 2011-12
Figure 2 Percentage Split between Research Expenditures and Corporate Expenditures between Comparable Organizations

Acronyms, Abbreviations and Definitions used in this Report

Acronyms, Abbreviations and Definitions used in this Report
Acronym Meaning
AEB Audit and Evaluation Branch
CEB Certification and Engineering Bureau
CRC Communications Research Centre
CSA Canadian Space Agency
DND Department of National Defence
DPR Departmental Performance Report
DRB Canadian Defence Research Board
DRDC Defence Research and Development Canada
DRTE Research Telecommunications Establishment
FTE Full-time equivalent
GDP Gross domestic product
IC Industry Canada
ICT Information and Communication Technologies
IEEE Institute of Electrical and Electronics Engineers
IT Information Management and Information Technology
IP Intellectual Property
ITU International Telecommunication Union
LAC Library and Archives Canada
MOU Memorandum of Understanding
NCC National Capital Commission
NRC National Research Council
O&M Operating and Maintenance
OGD Other Government Departments
R&D Research and Development
SITT Spectrum, Information Technology and Telecommunications
TTO Technology Transfer Office

Executive Summary

Program Overview

The Communications Research Centre (CRC) is the federal government's primary laboratory for research and development in advanced telecommunications and is a centre of excellence in Information and Communications Technologies (ICTs). The organization oversees twenty individual research programs and a full suite of in-house corporate services ranging from a technology transfer office to graphic design services. Additionally, the CRC acts as landlord to the Shirleys Bay Campus in Ottawa, where the organization is co-located with a number of other government departments as well as some private industry.

Created in 1969 and part of Industry Canada since 1994, the organization is currently housed in the department's Spectrum, Information Technology and Telecommunications (SITT) Sector. Its annual funding is approximately $52.7 million, of which approximately $18.0 million is funded by other government departments and the private sector.

Evaluation Purpose and Methodology

This evaluation is intended to inform management decision-making with respect to future priorities and resource allocation. This evaluation was conducted in accordance with the Treasury Board Policy on Evaluation and the Directive on the Evaluation Function.

The evaluation findings and conclusions are based on the analysis of multiple lines of evidence. The methodology included a review of documents and literature, an administrative data review, key interviews, a client survey, expert panels and a review of other federal labs.

Findings

Relevance

The CRC is responding to the roles and responsibilities of the federal government as set out by legislation via both its function in ICT research as well as its capacities to provide unbiased technical advice in the policy and regulatory forums.

The CRC's activities align with the federal government's priorities as stated in the June 2011 Speech from the Throne, as well as the Science and Technology Strategy, Mobilizing Science and Technology to Canada's Advantage. The CRC's activities also align with Industry Canada's strategic outcomes; however, the CRC research lines reflect a very broad interpretation of them. This interpretation should be narrowed, and research should be reviewed for relevance more frequently in order to ensure work carried out by the organization is appropriately aligned.

The CRC sits at the technological intersection of spectrum management, innovation and regulation in Canada. Evidence indicates that there is a continued need for the organization due to the increasing importance of the ICT industry as well as reliance on the CRC by its client base which includes Industry Canada's SITT, other government departments, academia, industry and other players in the international forum. The CRC is the Government of Canada's source of sound and unbiased advice to inform policies and regulations, enabled by unique in-house expertise and equipment.

Key interviews with clients from the private sector and OGDs revealed that they do not have the capacity that the CRC offers in-house. Paying clients consider any alternatives as biased in their research priorities and competitive interests. This is especially true for large government departments who require input on matters of regulation, national security and public safety. In the client survey, the majority of clients indicated that at least part of the work they commissioned the CRC to do may be sourced elsewhere, but comments indicated that issues of quality and shortfall of professional expertise would likely arise without CRC involvement in their work.

Expert panels put forward the importance of Canadian research remaining at the forefront of emerging trends, citing that such work is important for economic development, as well as matters of scientific priority to government. Expert panels also identified niche areas within the organization that may be doing research that is currently of a low degree of importance to the Government of Canada: while the majority of research conducted at the CRC was deemed highly relevant, expert panels noted that some research has outlived its relevance and a refocus of the corresponding research groups was recommended.

Performance

The research being carried out by the CRC is world-class. Of the twenty research programs the organization houses, sixteen boast of winning awards in the past five years. Researchers collectively lay claim to several hundred publications, thousands of citations and spoke at over 200 engagements over the same period of time. The organization attracts clients and collaborators from all over the globe and clients unanimously rated the CRC's work as being of high quality in key interviews. Some of the CRC's research was flagged for refocus by expert panels as the excellence of the work had lapsed over time.

At the time of this assessment, the CRC was engaged in a joint exercise with SITT to articulate objectives and performance measures. As such, it was difficult to conclude on whether or not CRC is meeting its expected outcomes. Nonetheless, the evaluation used a combination of primary data sources gathered as part of the assessment as well as different data sets and studies available from the CRC to assess performance against general outcomes stated in the CRC's 2007–2010 Strategic Plan, along with the Departmental Performance Report (DPR). There is evidence to suggest that the CRC has made progress toward the achievement of their objectives. The CRC is expected to have a more robust performance measurement strategy following the completion of the joint exercise.

Multiple lines of evidence suggest that clients benefit from the CRC's unbiased advice in terms of scientific findings, technical solutions, and IP licensing. The client survey indicated that the majority of clients believe the work the CRC conducts for their organization offers genuine benefits, with 91.2% indicating that the benefits to their organization were either very good or excellent. However, clients do not believe they are making full use of the organization's capacities and indicated that a client development function would enhance their ability to exploit the full suite of offerings the CRC houses.

This assessment was unable to report on remuneration for work conducted for clients. While incoming funds are documented in financial systems, the CRC cannot extract information relating to corresponding time spent on individual research contracts. Without appropriate administrative tracking systems, as available in other federal labs, it is not possible to discern whether or not the CRC is charging appropriately.

The CRC's governance of the Shirleys Bay campus is in line with Treasury Board policies and the organization is maximizing the recovery of costs. However, the current model detracts from the organization's capacity to focus on its research mandate. Further, significant time and energy is directed toward managing the campus and addressing infrastructure issues.

The CRC's corporate costs are relatively high compared with other research centres. In part, this is because some of the organization's corporate services are allocated towards supporting campus operations. Efficiencies may be gained if some functions were centralized, consolidated, or outsourced.

Finally, research activities at the CRC are not currently functioning at optimal levels with respect to efficiency. This is not a by-product of the capacity of researchers, but is rather an artifact of historical management practices and a silo-based organizational structure. Some reconfiguration of research resources is required to optimize the excellence, talent and capacity to achieve inherent in the organization.

Recommendations

The conclusions of the evaluation led to the following recommendations:

  1. The CRC should seek internal efficiencies and respond to the modern realities of ICT research within government.
    1. Consider the recommendations of the expert panels, particularly those which address areas of overlap, inefficiencies and topics of lower strategic value to the organization
    2. Seek more efficient solutions for corporate services
    3. Set regular intervals for the review of research lines in terms of their excellence, relevance and impact
    4. Continue to work toward a governance model that will focus the attention of the organization on its research mandate
  2. The CRC should improve its tracking of resource use on research and work performed for clients.
    1. Researchers should track the time they spend on all projects to determine the true cost of services to clients and to ensure appropriate billing
    2. Systems should be established to ensure that reports on client-based activities by research program can be produced
  3. To better communicate its capabilities, the CRC should strengthen its client/business development interface. Consideration should be given to:
    1. Articulating and promoting of the CRC's capabilities to clients and stakeholders
    2. Collecting market intelligence on new and emerging trends
    3. Actively updating a client feedback mechanism at the conclusion of research projects
    4. Formally acting as a catalyst for a stronger working relationship with SITT
  4. To better evaluate the performance of research, the CRC should continue to work on performance measurement for outputs as well as identifying and measuring immediate, intermediate and ultimate outcomes.

1.0 Introduction

This report presents the results of an evaluation of the Communications Research Centre Canada (CRC) conducted between May and November 2011. The purpose of the evaluation was to assess the relevance and performance of the CRC. The report is organized into four sections:

  • Section 1 provides the profile of CRC;
  • Section 2 presents the evaluation methodology;
  • Section 3 presents the findings pertaining to the evaluation issues of relevance and performance; and,
  • Section 4 summarizes the evaluation's conclusions and provides recommendations for future actions.

1.1 Overview of CRC

Located at Shirleys Bay in Ottawa, the CRC is the federal government's primary laboratory for research and development in advanced telecommunications, and is a centre of excellence in Information and Communications Technologies (ICTs). The origins of the CRC date back to the 1940's, when the National Research Council (NRC) investigated radio waves and the development of electronics equipment in support of Canada's war effort. Following WWII, these research efforts were merged into the Defence Research Telecommunications Establishment (DRTE). In 1962, the DRTE built Canada's first satellite, the Alouette 1, which has been designated one of the ten most outstanding achievements in the first 100 years of engineering in Canada.

In the wake of Alouette's success, the government moved to expand civilian communications with the creation of the Department of Communications and its research arm in 1969 and the CRC was spun out of DRTE. The CRC was designated research institute status in 1992Footnote 1 and was moved to Industry Canada in 1994, and is currently housed in the Spectrum, Information Technologies and Telecommunications (SITT) Sector of the department.

CRC research has a number of clients both internal and external to the federal government. Within Industry Canada, the CRC provides SITT with independent scientific knowledge and insights to help SITT in the development of telecommunications and spectrum management policies, regulations, standards and programs. Other government departments also represent major clients for the CRC, most notably the Department of National Defence (DND) and the Canadian Space Agency (CSA). The organization's relationship with DND goes back to its inception and the long-standing Memorandum of Understanding between the two organizations calls the relationship "mutually beneficial." Recent CRC research projects for DND aim to ensure that the Canadian Forces have state-of-the-art wireless communications when deployed or for remote monitoring. CSA, also a long-standing client, makes use of CRC's technical expertise to ensure effective communications systems for its major satellite projects. The CRC additionally provides scientific advice to other government organizations.Footnote 2

The CRC also seeks to support Canadian industry by conducting research on behalf of clients, anticipating future directions in ICTs, and transferring technologies and Intellectual Property to industry for further development.

In addition to its research activities, the CRC is the landlord of the Shirleys Bay research campus, located about 20km west of downtown Ottawa. Industry Canada owns the land on campus, and CRC manages the campus for the benefit of all government tenants on a cost-recovered basis.

1.2 Mission and Expected Results

The CRC's mission statement reflects the multiple priorities that the organization responds to both at the federal level and within Canadian industry. The CRC articulates its mission as followsFootnote 3:

  1. To be the federal government's centre of excellence for communications research and development (R&D), ensuring an independent source of advice for public policy purposes.
  2. To support government operations led by major clients in selected areas of ICT application such as defence, public safety and space-based communications.
  3. To stimulate the growth and evolution of Canada's communications sector by engaging in industry partnerships, performing technology transfer and working with universities and other organizations.
  4. To advance Canada's global reputation as an ICT research leader by engaging in international collaborations and partnerships.

1.3 CRC Organizational Structure

The CRC is headed by a President, who reports to the Assistant Deputy Minister of Spectrum, Information Technologies and Telecommunications Sector. The organization has an Advisory Board made up of representatives of key clients, who provide management advice as well as advice on technical issues.

CRC has four research branches: Broadband Network Technologies Research; Broadcast Technology Research; Satellite Communications and Radio Propagation; and Terrestrial Wireless Systems. Each of these research branches is headed by a Vice-President, who reports to the CRC President.

In addition to the four research branches, CRC activities include a number corporate services groups and a campus operations group. These groups report to the President of CRC. Further details on campus operations and corporate services are provided in Section 1.5 and Section 1.6.

1.4 Organizational Resources

The CRC's funding for the 2011-12 fiscal year is approximately $52.7 million. This funding is derived from a variety of sources, including Industry Canada's A-base, other government departments, the private sector (contracting-in and revenues from Intellectual Property), and cost recoveries from tenants at Shirleys Bay campus. Table 1 below identifies the source CRC's total funding for 2011–2012 and Figure 1 presents an overview of budget allocation for the same time periodFootnote 4.

Table 1: 2011–2012 Source of Funds (approximate)
Source Funding %
Industry Canada A-Base $32.5M 61.7%
OGDs (Research) $8.9M 16.9%
OGDs (Tenant Services) $8.3M 15.7%
Private Sector (Research) $0.8M 1.5%
IP Revenues $2.2M 4.2%
Total $52.7M 100.0%

Figure 1: Overview of CRC Budget Allocation, 2011-12

Pie chart of Overview of CRC Budget Allocation, 2011-12 (the long description is located below the image)
Description of Figure 1

Figure 1 is a pie chart divided into five slices. The largest slice represents research activities at $30.0 million for the 2011-12 fiscal year and 57% of the total budget. The next largest slice represents campus operations at $14.5 million and 28% of the total. Following that, there is a slice representing corporate services at $6.4 million and 12% of the total. The two smallest slices are the manufacturing and design shop ($1.3 million and 2% of the whole) and IP awards, which represents $0.5 million or 1% of the whole).

The organization allocates approximately 57% of its budget to research activities, 28% to campus operations and 12% to corporate services. An additional 1% is allocated to redistributing earned IP royalties to the CRC's in-house inventors as per the Public Servants Inventions Act, and 2% is allocated to the manufacturing and design shop, which produces prototypes and other materials required by researchers.

The CRC employs about 385 FTEs across its corporate services, campus operations and research activities. The breakdown of employees by these three activities is included in Table 2.

Table 2: CRC Employees by Activity
Business Line FTEs %
Campus Operations 51 13.2%
Corporate Services 59 15.3%
Research 259 67.3%
Manufacturing & Design Shop 16 4.2%
Total 385 100.0%

1.5 Campus Operations

The CRC has been the custodian of the Shirleys Bay campus since 1970. The campus houses laboratories, computer centres, offices, libraries and other facilities that belong to either Industry Canada or one of the department's tenants. The site encompasses 313.2 hectares of land and has 111 permanent and temporary buildings, of which 40 belong exclusively to the CRC and another 24 are shared (e.g. sheds for electrical conduits, the steam plant, the cafeteria, security building, etc.). The majority of the remaining buildings are small, special purpose facilities that are predominantly lab space and storage. There are approximately 20 buildings that are occupied by CRC staff full-timeFootnote 5.

Tenants on the Shirleys Bay campus include DND, CSA and Library and Archives Canada (LAC), all of whom own and retain responsibility for their own buildings and facilities. SITT's Certification and Engineering Bureau (SITT-CEB) is also located on-site. Table 3 outlines major tenants and the proportion of buildings each occupies on campus.

Table 3: Shirleys Bay Buildings
Department # of buildings % of buildings
CRC 40 46.0%
DND 40 46.0%
CSA 5 5.7%
SITT—Certification and Engineering Bureau 1 1.1%
Library and Archives of Canada 1 1.1%
TOTAL 87 100%
SharedFootnote * 24
Total 111

There are 51 FTEs working in campus operations, 40 of which work in building maintenance and 11 of whom work in property management. One Director oversees the maintenance and development of Shirleys Bay campus.

1.6 Corporate Services

Apart from housing the research itself, the CRC plays host to a variety of in-house corporate services that assist the CRC in fulfilling its mission. These services span from financial management to in-house graphic design and include:

  • Creative Visual Services—A full-service graphic design and photography resource that supports the marketing of CRC, including the organization's technology transfer function and research.
  • Business Development—This function represents the CRC or IC on federal committees on R&D management and acts as Canadian liaison to the European Union's scientific collaborations framework.
  • Communications and Promotion—This is the CRC's vehicle for publicity.
  • Technology Transfer Office (TTO)—This office manages CRC's Intellectual Property (IP), negotiates IP licensing agreements and acts a liaison with other research organizations (e.g. NRC). The TTO also oversees the Innovation Centre, CRC's business incubator, which facilitates access to the CRC's in-house expertise and facilities for fledgling businesses.
  • Finance and Material Management—This group is responsible for the management and control of CRC's financial resources as well as financial performance information, contracting and fleet management.
  • IT & IM—This group provides IT supportFootnote 6 as well as information management services to the CRC, including library and records management.

1.7 CRC Research

The CRC's core research competencies include wireless systems, communications networks, radio fundamentals, interactive multimedia and broadcasting, and photonics. Research undertaken within these core competencies employs 259 researchers who support 20 different research programs. A listing of research programs assigned to each core competency and a brief description of their work is provided in Appendix 1.

At the time of this evaluation, the CRC was in the process of finalizing its 2011–2014 Strategic Plan. The plan reflects recent efforts by the CRC to revise its research priorities to ensure that its research remains relevant to its clients' current and emerging needs and is of world-class calibre. As part of this ongoing effort, the plan identifies four strategic priorities that CRC will focus its research efforts on going forward:

  1. Spectrum: Research to support Industry Canada's policy, regulatory, and program delivery decisions for spectrum management.
  2. Defence, Public Safety and Security: Research to support the procurement and deployment of defence communications systems, the improvement public safety response systems, and the identification and mitigation of cyber-security threats.
  3. Emerging Network Technologies: Research and development into new advanced communications technologies to understand their implications for government programs and Canadian industry.
  4. Advanced Communications Technologies Applications and Adoption: Research and development that allows CRC to provide technical advice, prototypes and software, and transfer technologies to industry to support the implementation of new systems and the adoption of new technologies.

CRC's five core research competencies cut across these new priorities. New outcomes and related performance indicators that relate to the new strategic outcomes will be laid out as SITT develops its sector-wide logic model—a process ongoing at the time of this evaluation.


2.0 Evaluation Methodology

This section describes the objectives of the evaluation, the overall approach and the specific questions that were addressed, the data collection methods used, and the limitations of the methodology.

2.1 Context

In recent years, CRC has undertaken extensive review exercises. In 2009, the program undertook a series of four reviews, focusing on the organization's client-based research activities, core research activities, governance, and corporate management services. Following these four studies, SITT management committed to reviewing the CRC's five major research areas in more depth. The five core competencies that represent the dominant lines of research at the CRC are as followsFootnote 7:

  • Photonics
  • Wireless Systems
  • Communications Networks
  • Radio Fundamentals
  • Interactive Multimedia and Broadcasting

The first of these reviews was piloted in 2010, and focused on photonics research at the CRC. This review focused on the excellence, relevance and impact of CRC's research in the photonics field. It identified priority research areas and recommended that a number of photonics research activities be prioritized, realigned and/or cease.

Following the photonics review, SITT and CRC management approached the Audit and Evaluation Branch (AEB) to lead similar reviews of the CRC's other core research areas and to use these reviews as inputs into the overall evaluation of the CRC. Where the photonics review was conducted in advance of this assessment, its methodology was replicated in order to ensure comparability of results across core competencies.

2.2 Approach

The evaluation was conducted by AEB. The expert panels, a critical component of the evaluation, were moderated by an independent expert in ICT research who was tasked with reporting on the findings of the expert panels. This is in line with a best practice identified by Treasury Board Secretariat's 2005 Case Studies on the Uses and Drivers of Effective Evaluations in the Government of Canada. The study found evaluations that engage external expertise to assist with assessments have greater impact with respect to departmental decision-making.

2.3 Evaluation Objective and Scope

The objective of this evaluation was to provide input into senior management decision-making with respect to future priorities and resource allocation.

2.4 Evaluation Questions

The evaluation addressed the following questions:

Relevance

  • To what extent does CRC continue to address a demonstrable need?
  • To what extent do CRC's objectives align with federal government priorities and Industry Canada's strategic outcomes?
  • To what extent does the CRC align with federal government roles and responsibilities?

Performance

  • To what extent is the quality of CRC research excellent?
  • To what extent has CRC achieved its expected outcomes?
  • To what extent do clients benefit from CRC research?
  • To what extent does CRC demonstrate efficiency in its corporate, campus and research operations?

2.5 Data Collection Methods

Multiple lines of evidence, along with the triangulation of data, were used where possible to address all evaluation questions of relevance and performance. Evaluation methods included the following:

  • Document review
  • Literature review
  • Administrative data review
  • Key interviews
  • Client survey
  • Expert panels on core research competencies
  • Review of federal research centres

2.5.1 Document review

A review of CRC program documents was conducted as part of this evaluation, including the organization's strategic plans, corporate reports, previous reviews, economic impact studies and other reports. It also included various Industry Canada corporate reports, CRC research program reviews, and relevant legislation. A list of documents is available in Appendix 3.

2.5.2 Literature review

The literature review sought out general information on the ICT industry as well as the economic impact of ICT R&D to provide context for the study. In addition, it was used to identify potential measures of excellence, again in conjunction with the document review. A list of reviewed literature is available in Appendix 4.

2.5.3 Administrative data review

The administrative database from the Technology Transfer Office (TTO) at the CRC was reviewed to lend insight into the relevance and performance of the organization. The database provided indication of the magnitude of IP licensing and the contracted work for other organizations over time.

2.5.4 Key interviews

Key interviews were conducted to address issues of relevance and performance and to provide additional context to incoming data from other sources. A total of 38 interviews were conducted:

  • CRC executives, including one past executive (7)
  • Other IC staff and executives (12)
  • Other government departments (including DRDC, DND and CSA) (6)
  • Private sector representatives (11)
  • Academia (2)

The above list includes one randomly selected client from each research program reviewed by the expert panelsFootnote 8. The list of individuals interviewed and the interview guide are presented in Appendix 5.

2.5.5 Client survey

A client survey was conducted as part of this evaluation. It was largely based on the survey directed toward photonics clients during the 2010 study to ensure similar data was available to all panels and to keep results comparable. The survey additionally examined issues of both relevance and effectiveness, offering clients the opportunity to indicate their expectations for future involvement with the CRC, the reasons they chose to work with the CRC, as well as the impact the organization made on their work.

For this evaluation, clients were defined as those who had paid the CRC for services rendered. The client list for each research manager, as well as contact information (e.g. phone numbers and e-mail addresses), was provided by the CRC. Data collection was completed using an online application with an option to call AEB for survey completion over the phone, and non-response follow-up activities were conducted via e-mail. The final response rate of the survey was 63% (or 68 of 108). The questionnaire can be viewed in Appendix 6 and summative results of the survey can be viewed in Appendix 7.

2.5.6 Expert panels on core research competencies

The expert panels on core competencies aimed to provide an in-depth analysis of the excellence, relevance and impact of the research conducted by the CRC. Each group consisted of four members (16 in 2011, and four for the 2010 photonics review) plus the expert consultant who led the group. Panel members represented Canadian industry, major clients, academia and foreign industry, where appropriate for that competency.

In preparation for the expert panels, the CRC management team was surveyed by AEB to capture the fundamental elements of each of the organization's research programs. The survey addressed relevance, efficiency and effectiveness via questions relating to the management of research at the CRC. It additionally collected data ranging from an inventory of specialized equipment to the publication activity and awards of researchers working within each research program.

The survey's questionnaire was largely based on what was required from managers for CRC's 2010 photonics review, however, some additions were made to ensure it addressed the core issues of relevance and performance for this assessment. The Managers' survey can be viewed in Appendix 8. Managers of photonics research programs were asked to provide updates via the new template and an update to last year's report was prepared.

The consultant moderating the panels then received the results for early review and preliminary analysis, as did panel members. Following that, expert panels on each core competence were convened at the CRC to review the inputs made available to them as a group (including the managers' survey, client survey, and presentations from each research manager), and to ask questions of the research managers in person. Expert panel members were also afforded the opportunity to walk through labs and to view the CRC's innovations.

2.5.7 Review of federal research centres

In order to assess the efficiency of the CRC's operating model, a small review of Canadian federal labs was undertaken, involving four federal research facilities. The criteria for selection were based on similar budget size and number of FTEs. The review included a set of four key interviews, accompanied with a brief financial questionnaire. The interview guide, financial template, as well as participating organizations are available in Appendix 9.

2.6 Data Limitations

The following were the three main limitations of the evaluation:

Program logic model and performance indicators

The CRC did not have a logic model at the outset of this assessment, a key component of the evaluative process. At the end of the conduct phase, the CRC had landed on a version that was nearing the final stages and was beginning to populate a first year of data. This presented a challenge in identifying the key data and data sources required to measure the program's ability to meet expected outcomes. In order to mitigate this, IC's DPR was used to assess the organization's achievement of outcomes. Additionally, AEB embarked on considerable primary data collection activities (including those outlined in this report, such as the client survey, the managers' survey, the expert panels, etc.) to ensure an adequate information base to assess relevance and performance.

Self reported data sets

Following the photonics review in 2010, multiple programs underwent transformation. In addition, the department undertook a strategic operating review process during the same time period. The perception that this evaluation could result in further organizational changes increased the risk of bias in the responses of staff. Self-reported elements of this evaluation included data gathered from the management survey and the client lists from which the universe for the client survey was created. Wherever possible, additional lines of evidence were consulted to verify findings.

Expert panel membership

Expert panels predominantly represented non-government groups, and while this was important for ensuring the best advice was available to the CRC for future decision-making, it also introduced an intentional bias in results—a risk for an academic or industry-oriented direction within panel results. Panels were informed by the CRC that they were providing advice, not binding recommendations at the outset of their exercises. The CRC will need to assess the recommendations of the panels through the lens of a government entity prior to implementing changes.


3.0 Findings

3.1 Relevance

3.1.1 To what extent does CRC continue to address a demonstrable need?

Key Finding: There is a continued need for CRC, as indicated by the increasing importance of the digital infrastructure and ICT industry as well as reliance on CRC by its client base, including Industry Canada's SITT, other government departments, academia, industry and other players in the international forum. The CRC is the Government of Canada's source of sound and unbiased advice to inform policies and regulations, enabled by unique in-house expertise and equipment. However, some pockets of the organization are less relevant than they were in the past.

The CRC sits at the technological intersection of spectrum management, innovation and regulation in Canada. Evidence suggests that the organization presents the Government of Canada with neutral, unbiased advice on matters ranging from the efficient use of spectrum to public safety and national defence wireless communications needs. The organization also contributes to the development of the ICT industry in Canada, providing access to Intellectual Property developed within the organization, as well as contracting services and business incubation for industry.

In recent years, the ICT sector in Canada has grown faster than the rest of the Canadian economy, save for the 2009 economic downturn. Since that time, the sector has been trending upwards with a cumulative growth of 5.8% since the third quarter of 2009Footnote 9. This compares with a growth of 1.1% for all Canadian industries. The sector presents a major source of employment for a knowledge-intensive workforce and compensates workers with higher than average wagesFootnote 10. CRC's contracting services and business incubator give rise to new and existing businesses in this sector by providing access to expertise and equipment small businesses would not otherwise have.

Interviews with clients revealed they do not have the capacity that the CRC offers in-house. Additionally, they consider any alternatives as biased in their research priorities and competitive interests. This is especially true for large government clients who require input on matters of regulation, national security and public safety. Spectrum is a limited resource and interviews with Industry Canada staff indicated that the organization needs unbiased research to ensure that its policies and regulations provide optimal benefit to the Canadian telecommunications industry and to other users of spectrum in Canada. Additionally, both government and industry indicated that the CRC's international reputation provides credibility to the results of their work in dealing with clients and stakeholders.

The client survey also revealed a need for the CRC: when asked if they would continue to engage the CRC in their future work, 39.7% of clients saw their involvement with CRC as staying about the same while another 48.5% of clients indicated that their involvement would be potentially increasing over the next few years. While a majority of clients indicated that at least part of the work they commissioned CRC to do may be sourced elsewhere, comments indicated that issues of quality and a shortfall of professional expertise would likely arise without CRC involvement in their work.

The total value of contracting from private industry at CRC for the 2010-11 fiscal year was approximately $600,000Footnote 11. Further, the organization saw about $1.4 million in revenues from 75 IP licenses, demonstrating the extensive use government, industry and other players make of the organization. As of July 2011, CRC also had 7 businesses developing in its business incubation facility, the Innovation Centre, which includes access to the CRC's in-house expertise and test beds. Since 1960, the CRC has produced 107 spin off companies of which 54 are still active.Footnote 12,Footnote 13

Expert panels convened to assess the technical value of research programs echo sentiments of the value implicit in the Government of Canada engaging in ICT research. The need for Canada to be at the forefront of many new and developing technologies was identified as important for economic development, as well as matters of scientific priority to government. For example, expert panels underscored the value of one research group that focuses on defining intelligent spectrum access technologies as well as supporting regulatory and policy initiatives undertaken by SITT. The group is seeking to alleviate interference as well as spectrum congestion.

Other research groups are focused on activities that greatly enhance the use of spectrum with respect to military and space applications, while still others are addressing industry needs with respect to new technologies as well as access to software applications. Panels also noted that the CRC makes facilities and equipment such as test beds available to clients and collaborators alike.

However, all expert panels also identified niche areas within the organization that may be doing research that, while excellent, are currently of a lower degree of strategic importance for the Government of Canada. One such program identified by expert panels is producing research of excellent quality; however, a long-term research plan is not evident. The result is selected topics that are academically-oriented and based on individual research interests, with limited relevance and impact.

Such issues as lack of focus, duplication of work being conducted elsewhere, and a diminished value of research in areas that have either been overtaken by newer technologies or are simply more of value to other fields of study were identified in a small number of research projects.

3.1.2 To what extent do CRC's objectives align with federal government priorities and Industry Canada's strategic priorities?

Key Finding: The CRC's high-level activities align with the federal government's priorities, including the Science and Technology Strategy, Mobilizing Science and Technology to Canada's Advantage. The CRC's activities also align with Industry Canada's strategic outcomes, however, the CRC research lines reflect a very broad interpretation of them. This interpretation should be narrowed, and research should be reviewed for relevance more frequently, in order to ensure work carried out by the organization is appropriately aligned.
Alignment with federal government priorities

The CRC's activities align with the Speech from the Throne on June 3, 2011, which states that the government will continue to make targeted investments to promote and encourage research and development in Canada's private sector and in academia in order to improve Canada's productivity, enhance economic competitiveness and increase the standard of living. The Speech goes on to state that the government will look for ways to support innovation while ensuring that federal investment in research and development is effective and maximizes results for Canadians.

The organization's activities also respond to the federal strategic priority innovative and knowledge-based economy and align with the Science and Technology Strategy, Mobilizing Science and Technology to Canada's Advantage (S&T Strategy), which identifies ICT research as one of Canada's four research priority areas. While the organization is not specifically mentioned in the Strategy, its work is represented within the context of the document, as identified in CRC's mission statement. Specifically, the CRC relates to both the S&T Strategy's entrepreneurial advantage and people advantage.

For example, the entrepreneurial advantage targeted by the S&T Strategy aims at fostering a competitive and dynamic business environment, pursuing public-private research and commercialization partnerships, and increasing the impact of federal business R&D assistance programs. The CRC's mission statement asserts that the organization aims "to stimulate growth and evolution of Canada's communications sector by engaging in industry partnerships, performing technology transfer and working with universities and other organizations."

One of the primary focuses of the people advantage, another key component of the S&T Strategy, is increasing the supply of highly qualified and globally connected science and technology graduates. The CRC's mission mandates the organization to work with universities and "to advance Canada's global reputation as an ICT research leader by engaging in international collaborations and partnerships." Further, the managers' survey revealed that research programs habitually take in graduate students for mentorship and knowledge exchange in the course of their work.

Alignment with Industry Canada's strategic priorities

According to the department's Program Activity Architecture, the CRC is aligned with Industry Canada's Program Activity Architecture sub-objective 2.2 Science and Technology, Knowledge, and Innovation are Effective Drivers of a Strong Canadian Economy. CRC is the department's source of ICT-research-related information and supports ICT expertise for regulations, standards and programs; ICT expertise for other federal partners; and innovation and technology transfer.

The organization predominantly responds to the SITT strategic priority Excellence in ICT Research and Innovation. Specifically, the CRC's Strategic Plan for 2011–2014 states that the organization's vision is "to be the federal centre of excellence for ICT research in Canada.

A survey of managers revealed that 100% of the organization's research managers see their work as fitting into the above stated SITT's strategic priority. Most managers additionally listed SITT's other strategic priorities and identified research projects and other activities that directly relate to them. This demonstrates the broad research mandate researchers have become accustomed to, as well as their efforts to accommodate that mandate.

The expert panels found that there was some scope creep occurring in CRC research programs. In a number of cases, programs were undertaking activities that moved beyond their primary objectives, thus potentially overlapping with other CRC programs. In other cases, the research activities themselves fell outside the scope of what was relevant to CRC.

This sense of a broad research mandate is reflected in key interviews as well: interviews with internal clients and senior executives at IC revealed sentiments that the CRC's strategic planning was not always in line with SITT's needs, however, the most recent exercise did engage the SITT senior management cadre at a level that did begin to address alignment needs at that levelFootnote 14. Despite improving this strategic alignment, interviews indicated a sentiment that some research managers continue to pursue their own research interests as opposed to those defined by this processFootnote 15.

The panels unanimously suggested that more frequent reviews may be necessary to keep work focused and relevant. The panels also suggested that annual, rolling updates to the mandates of the organization would "provide more focus and direction to the research objectives."

3.1.3 To what extent does the CRC align with the roles and responsibilities of the federal government?

Key Finding: The CRC is responding to the roles and responsibilities of the federal government as set out in legislation via both its function in ICT research as well as its capacities to provide unbiased technical advice in the policy and regulatory forums.

The work of the CRC directly relates to both the Radiocommunications Act and the Telecommunications Act. The Radiocommunications Act, in referring to the Minister of Industry states:

…the Minister may, taking into account all matters that the Minister considers relevant for ensuring the orderly establishment or modification of radio stations and the orderly development and efficient operation of radiocommunication in Canada…undertake, sponsor, promote or assist in research relating to radiocommunication, including the technical aspects of broadcasting…

For example, the CRC has developed a loudness meter that allows for the monitoring of changes in volume of broadcast television commercials. It has, in fact, been so well recognized within the ICT research community that it has become an international standard.

Also identifying the roles and responsibilities of the federal government, the Telecommunications Act states:

It is hereby affirmed that telecommunications performs an essential role in the maintenance of Canada's identity and sovereignty and that the Canadian telecommunications policy has as its objectives…to stimulate research and development in Canada in the field of telecommunications and to encourage innovation in the provision of telecommunications services…

As an example, in supporting preparations for the switch to digital television (DTV) in 2011, the CRC carried out tests to study over-the-air DTV reception. Using Quebec City as its test site, it was found that small "DTV-ready" antennas sold to consumers of over-the-air television were not very efficient at VHF frequencies. The study was mandated by SITT to investigate possible discrepancies between the analogue and digital TV signal coverage. Moving forward, it may be used by SITT in refining rules and regulations, or in managing the digital television VHF band.

The CRC additionally supports the Minister in carrying out duties related to the Radiocommunications Act and the Telecommunications Act by providing neutral, unbiased scientific advice to regulators and policy makers (such as those housed in SITT), as well as real technological support for specific initiatives. In the managers' survey, the majority of respondents identified SITT as a beneficiary of their work, specifically in that they are equipped and available to provide policy and regulatory advice in their area on an as-needed basis.

3.2 Performance

3.2.1 How do we know this research is excellent?

Key Finding: The work being carried out by the CRC is world-class. This is well recognized by award-granting institutions and is reflected in the number of publications, citations and speaking engagements each program has engaged in over the course of the past five years. There are very few pockets of activity among the research programs that do not meet the organization's high standard of excellence.
Measuring excellence

According to a review of literature, measuring the excellence of engineering research is a field unto itself, but there are commonalities throughout the field that provide a framework through which the excellence of the work carried out at CRC can be assessed. In the fields of science and engineering, the development of intellectual property and the overall impact of the work is often referenced. Publications and subsequent citations of publications are two examples of the more academic-oriented indicators for scientific research. The network of other professionals surrounding the program being assessed also points to excellence.

Excellence at the CRC

Indeed, the excellence of the CRC has been recognized formally by a great many other organizations. According to the managers' survey, among CRC's twenty research programs sixteen boast of winning awards from external organizations within the past five years. For example, the organization has been the recipient of three Emmy AwardsFootnote 16 from the International Academy of Television Arts and Sciences.

Among the achievements recognized by the awards is the contribution of the CRC's Advanced Television Evaluation Laboratory, along with three other organizations, in standardizing the digital television system that is replacing North America's analogue system (this transition took place in Canada August 31, 2011).

The 2006 Excellence in Technology Transfer Award was presented to the CRC by the Federal Partners in Technology Transfer for pioneering research and development in the field of software defined radio. Specifically, the award recognized the development of a system that provides reliable and uninterrupted communication between radios and other devices particularly when they need it most, such as times of natural disasters when communications systems are most vulnerable. Additional examples of recent awards can be viewed in Table 4.

Table 4: Recent Awards for Scientific Excellence
Year Awards
2012 Emmy Award for ground-breaking research in global standardization of loudness metering for use in broadcast audio
2010 Wireless Innovation Forum International Achievement Award in Software Defined Radio
2009 Emmy Award for contribution to standardization of ATSC Digital Television System
2008–2009 Federal Partners in Technology Transfer Award for developing omni-directional antenna that led to creation of one of the world's largest satellite-based mobile asset tracking systems
2008 OCRIFootnote 17 Technology Partnership Commercialization Award for long-standing support to Ottawa's high tech sector
2007 Canada's Telecommunications Hall of Fame Special Recognition Award
2007 Natural Sciences and Engineering Research Council Award for contributions to Agile All-Photonic Networks Research Network

Further, researchers within the organization lay claim to several hundred publications over the past five years, and thousands of citations during the same period of time. They have been invited (and accepted) to speak professionally at over two hundred engagements, and have been asked to chair numerous committeesFootnote 18. Expert panels praised of many of the institution's research lines, and panellists actually commented on the international reputation of some research managers.

The organization attracts clients and collaborators from all over the globe for work ranging from consultation to the development of new technologies. These relationships leverage outside expertise and reduce the cost of doing research at the CRC as well as bolster the organization's capacities and outputs. Additionally, the CRC habitually hires graduate students, which facilitates the development of additional highly-trained professionals within Canada as well as information exchange that benefits the CRC via giving the organization access to new ideas circulating within academia.

In interviews, CRC's clients, including industry, OGDs and those in-house at Industry Canada unanimously rated the CRC's research to be of high quality. Clients who were able to comment on the quality of research conducted by the CRC in comparison to other internationally recognized labs suggested that the CRC quality was on par with other comparable organizations with the CRC being the world leader in certain areas.

In a benchmarking study conducted in 2007Footnote 19, the CRC was ranked as one of the top three comparable ICT labs in the world. The study praises the organization's commercialization efforts as outperforming the other organizations in the study. In addition, the CRC's staff won second place for publications per researcher relative to similar organizations.

The expert panels unanimously commented on the calibre and excellence of the research being done at CRC (and sometimes commented on their own experience with the researchers leading the program being reviewed), and concluded that CRC researchers have earned impressive reputations within their global peer communities. For example, one panel commented that the academic papers of one research manager is so highly regarded that they are often accepted for publication with little challenge. Another program's projects were so timely that it is regularly sought out by international standards bodies for prototype assessment.

That said, the excellence of research conducted by two research programs was questioned by the expert panels. It was concluded that much of one group's capabilities are legacy from earlier activities and that they could now be obtained elsewhere. It was also determined that a different research line is not truly engaging in research at this point in time, rather it was found to be providing services/support capability, working responsively to requests from government entities and providing technical services.

3.2.2 Is the CRC meeting its expected outcomes?

Key Finding: In the absence of a logic model, it is difficult to ascertain whether or not the CRC is meeting its expected outcomes. However, there is evidence that the organization made progress towards the achievement of outcomes stated in the 2007–2010 CRC Strategic Plan and the Departmental Performance Report. Once a fully developed logic model and corresponding performance measurement strategy is implemented, the capacity to assess expected outcomes should improve.

The 2009 Review of the Communications Research Centre, Element 1: Client Based Research, suggested that the CRC required appropriate performance measures that would better articulate the contributions the organization makes to the department. Responding to this, as well as demonstrating increased integration at the strategic level between the CRC and the rest of SITT, a joint logic model is being developed and is expected to be finalized in the coming months.

The CRC's 2007–2010 Strategic Plan identified a number of outcomes for the organization, however, the CRC has not clearly articulated outcomes beyond a general level and has not formally tracked performance indicators. Nonetheless, data collected for this assessment suggest that the organization has made progress toward the achievement of these outcomes. The CRC's 2011–2014 Strategic Plan did not contain a similar set of outcomes, presumably because the organization is engaged in the development of a joint logic model.

In the absence of a full set of performance indicators, this assessment used three data sources to evaluate the CRC's fulfillment of expected outcomes: draft research outputs, as per the draft logic model; the outcomes set out in the 2007–2010 Strategic Plan; and the Departmental Performance Report's expected results for the CRC. It was anticipated that the draft outputs and strategic plan outcomes would provide some indication of what immediate outcomes might be and that the DPR might provide some indication of what ultimate outcomes might be in the future.

Outputs

The draft outputs, presented in Table 5 below, will likely be used as performance indicators for the logic model currently under development when it is approved. While outputs represent a portrayal of what the organization does with its research time, they do not provide insight into the real value and influence of the CRC's work. Presumably, once the program develops a more sophisticated set of outcomes, this will be present in its indicators. Additionally, without stated targets, as is the case with most outputs, it is not possible to provide meaningful analysis on the organization's activities.

Table 5: CRC Research Outputs, 2010-11
Output 10-11 Target
Number of CRC scientific papers (and conference publications)Table note t 287 250
Number of technical briefs, reports and presentationsTable note t 317 Table note *
Number of standard contributions 40 Table note *
Number of patents (active and applications) 239 Table note *
Number of CRC technical inputs provided to federal agencies developing policies and programs related to the information and communications technologies sector. (Excludes DND and CSA)Footnote 20 10 5
Number of service contracts executed in the year Table note * Table note *
Number of new R&D agreements 26 Table note *
Number of new license agreements 30 Table note *
Number of new and ongoing IP licenses 556 Table note *
Number of times technology test beds are used for industrial partners and clients contracts. 29 Table note *
Number of companies in the Innovation Centre 8 Table note *
Outcomes

In its 2007–2010 Strategic Plan, the CRC identified eight strategic outcomesFootnote 21. Presented in Table 6 below, they provide a measure of insight into the achievements of the organization, but not a complete picture. The CRC does not formally track the indicators listed per se, but many were covered by other reporting mechanisms, the draft outputs presented above, as well as the primary data sources used in this assessment. As with the draft outputs, indicators in the strategic plan did not include benchmarks or targets and the scarcity of readily available data precluded time series analysis, making analysis challenging.

Table 6: Summary of 2007–2010 Outcomes
Outcome Indicator(s) Available data
1. Telecommunications policies, regulations and standards will be developed using CRC technical expertise Use of CRC inputs by groups developing policies, programs, regulations and standards related to the telecommunications sector including IC, ITU, IEEE and CRTC 40 standards contributions and 10 technical inputs provided to OGDs in 2010-11
2. Canadian companies in the telecommunications sector will use CRC-developed technology to improve their product lines and competitiveness Number of IP licences issued to Canadian companies 556 new and on-going IP licenses in 2010-11 (Canadian and international)
Sales revenue of Canadian companies in the ICT sector that were formed as a result of their involvement with the CRC Collectively $2.6 billion in 2010Table note **
3. There will be improved decision-making by National Defence on new technologies related to future military communications systems The number of technologies that are adopted which enhance or provide new capabilities for Canadian Forces operations, as well as the number of DND Technology Development Programs (TDP) and NATO and TTCPFootnote 22 Committees led by CRC No data available
4. The Canadian telecommunications sector will have knowledge of, and access to, CRC's intellectual property portfolio Number of research partnerships between the CRC and private sector, academic and national/international research organizations 113 formal agreements of which 83 were R&D Collaborative Agreements in 2010-11
5. More government departments will have knowledge of, and access to, CRC technologies and know-how for ICT-related applications The number of partnerships between the CRC and other federal departments 9 formal agreements in 2010-11
6. CRC will maintain a high standard of research excellenceTable note f Number of scientific publications, conference presentations, patents, technical reports, invited papers, conference chairs and participation in international research consortia
  • Publications: 833
  • Conference presentations: 743
  • Patents: 65
  • Technical reports: Not available
  • Invited papers: 70
  • Conference chairs (invitations): 142
  • Participation in international research consortia: Not available
7. CRC will promote Canadian capabilities in the ICT sector Participation in conferences, collaborations with international organizations and representation in the science and technology community, both nationally and internationally, as ICT experts
  • 154 conferences attended in 2010-11
  • 9 formal collaboration agreements with foreign organizations
8. CRC will continue to improve its national presence The number of partnerships with academia, industry and centres of excellence in the Atlantic region, Quebec, Ontario, the West and the North No data available

Table notes

Table note 4

Data for this indicator covers off the five year period between 2006 and 2011. Note that some double counting may have occurred in cases where researchers from different programs within the CRC collaborated on the same product. For details, please see Appendix 10.

Return to note f referrer

Table note 5

Doyletech updates CRC's economic impact analysis, Eye on Technology, Issue 14, March 2011.

Return to note ** referrer

Overall, the indicators demonstrate that the CRC is working toward its stated outcomes. For example, the organization utilized traditional academic indicators to identify research excellence within its organization. Most telling are such indicators as invited papers, which go beyond showing the acceptance of the researchers, but demonstrates the degree of professional notoriety and recognition of achievement within the ICT research community.

Expected Results

Under IC's DPR, the CRC identifies two expected results as follows:

  • IC and other government organizations receive high quality, research-based technical inputs to develop telecommunications policies, regulations and standards and support government operations.
  • Canadian companies use CRC-developed technologies to enhance their product lines.

The CRC measures IC and other government organizations receive high quality, research-based technical inputs to develop telecommunications policies, regulations and standards and support government operations via client satisfaction, with a target of 80% satisfied clients. According to the survey conducted as part of this evaluation, 91.1% of clients ranked their overall experience with the CRC a four or five out of five, with one being entirely unsatisfactory and five being excellent. It should be noted that future iterations of the DPR will not include a client survey as it will soon be replaced with other indicators.

The second expected result, that Canadian companies use CRC-developed technologies to enhance their product lines is measured by the increase in total sales revenues every five years among Canadian companies with a link to the CRC, compared to market averages. A target of 20% increase was expected and registered at 61% in studies on the economic impact of technology transfer from the CRCFootnote 23. The report noted that despite the downturn of 2008-09, aggregate sales of these businesses in 2010 were significantly higher than in 2005. This was largely due to merger and acquisition activity by larger firms. Smaller firms actually had lower sales in 2010 than in 2005Footnote 24. It is interesting to note that this expected result and indicator is similar to outcome 2 from the 2007–2010 Strategic Plan discussed above.

The CRC is also listed in the DPR as one of three programs fuelling Canada's continued standing in the Innovation IndexFootnote 25 (target met by moving up to 11th place from 12th for 2010-11), the international ranking of Canada in university-industry collaboration in R&D (met by maintaining 2nd place) and in the number of people working in R&D of total employment numbers (exceeded the last time it was measured in 2004-05 at 13.3 in 1,000 employees working in R&D).

While the measures listed above provide a degree of insight, they do not provide a full and complete picture of the CRC's activities and outputs, nor is the full picture of how the stated objectives might be met available through the data.


3.2.3 To what extent do clients benefit from CRC research?

Key Finding: Multiple lines of evidence suggest that clients benefit from the CRC's unbiased, advice in terms of scientific findings, technical solutions, and IP licensing. That said, clients do not believe they are making full use of the organization's capacities and that a client/business development function would enhance their ability to exploit the full suite of offerings the CRC houses.

The CRC attracts its client base from as close to home as other areas of Industry Canada and as far away as India. The list of clients produced by managers in their surveys includes SITT, various other government departments in Canada (OGDs), private industry, academia, foreign governments and other organizations. Main OGD clients include DND (which invests approximately $6 million in the CRC annually), Canadian Space Agency ($0.5 million), and Public Safety Canada. Private industry clients range from fledgling companies in the CRC's business incubation centre to large multinational corporations.

The CRC's service offerings span a wide range. Perhaps most notable because of the financial contributions the organization makes to the CRC, is DND. The defence research organization commissions the CRC with developing state-of-the-art wireless communications for the Canadian Forces when deployed for remote monitoring. The CRC also offers unbiased, advice in terms of scientific findings, technical solutions, and IP licensing. While some solutions can be drawn from existing or past research, other requests require more in-depth testing and research, as well as a longer-term contract with the client.

The client survey indicated that the majority of clients believe the work the CRC conducts for their organization offers genuine benefits, with 91.2% indicating that it was either very good or excellent. It is important to note that no clients reported that they did not perceive benefits for their organization.

Similar to the client survey, interviews conducted for this assessment indicate that the CRC's clients generally report that they benefit from CRC research. The benefits include being able to make decisions based on unbiased, cost-effective, top-notch advice that would not be available through alternate sources. Government clients reported that CRC advice and research allows them to make sound policy and regulatory decisions which are seen to be based on credible evidence which helps them achieve their organization's strategic outcomes. Private sector clients report that technology transfer and technology insights provide them with a competitive advantage.

IC enjoys additional benefits for housing the CRC: perhaps most relevant to SITT is the advancement of unconstrained research in an effort to keep the organization prepared to provide SITT with expert, unbiased technical advice as new developments arise. Unlike other clients who are generally able to approach the CRC with a discrete piece of work or specific objectiveFootnote 26, SITT is not always able to articulate its policy needs at the interval of time required to conduct necessary research: according to a review of literature, much research in ICT takes as long as ten years to reach a point of fruitionFootnote 27.

This creates a challenge for the CRC in fulfilling its role. Key interviews with CRC senior management and IC indicated that the CRC is now prioritizing SITT's requirements more so than in the past, however, there remain some challenges in that SITT must articulate its research needs years in advance in order to have results when required, and this is difficult to do. At times, the CRC already has answers on-hand from research conducted for the private sector or other government departments. The CRC also offers SITT an array of technical services that assist the sector with spectrum management on a more formal, on-going basis.

The unique skills and expertise of staff was identified as a core reason clients choose to work with the CRC, followed by the significant impact the CRC's contributions make to the individual or organization's work. Closely following is the quality of the CRC's results, as demonstrated in Table 7 below.

Table 7: Why clients choose to work with CRC
n=68 # %Table note *
The skills and expertise I require are unique to CRC 46 67.7
The equipment I require is unique to CRC 23 33.8
CRC is geographically convenient 23 33.8
The quality of CRC's results 38 55.9
CRC contributions make a significant impact on my work and/or my organization's work 41 60.3
We have a long-term contract or other arrangement in place 17 25.0
OtherTable note t 10 14.7

Expert panels also identified the value that the CRC has for its stakeholders, and highlighted technology transfer, the expertise and experience offerings of staff, and the organization's international reputation as being major contributors to the positive impact the organization has on ICT in Canada.

In spite of the numerous benefits for clients in engaging the CRC in research, multiple lines of evidence suggest that clients do not believe they have fully grasped the organization's capacities and believe that a client development function would enhance their ability to exploit the full suite of offerings the CRC houses: key interviews pointed to the need, with one interviewee in particular asking for a "translator between researchers and users", and others identifying a need to better understand the CRC's technological offerings. In the client survey, a client or business development function was the second most requested addition to the CRC's current offerings in a comments fieldFootnote 28. The most-requested addition was additional capacity in a subject matter area relatively unique to the client responding to the survey.

Expert panels also put this notion forward, calling unanimously for additional business support for clients via the function of a business exploitation champion or a strengthened business development functionFootnote 29. Such a role (or roles) would not only provide a means of bringing in customers and revenue for CRC technologies and intellectual property, but would also aid researchers by providing updates on the current needs in the client domains (whether industry, military, regulatory, or other).

While clients do reimburse the CRC for work completed, it was not possible for this assessment to determine whether related charges to clients were appropriate. The CRC does not currently have a time tracking system for researchers, nor is there a formal mechanism to ensure the original price point is met as work progresses and is completed. The review of federal research labs revealed that other labs do track the time staff spends on various projects for cost recovery and management purposes. Where research contracts earned the CRC $7.7 million of the $31 million spent on research in the 2010-11 fiscal year from other government departments and a further $0.6 million from the private sector and $1.4 million from IP revenues, there is merit in understanding the true value of services provided.

3.2.4 To what extent does CRC demonstrate efficiency in its operations?

This question was divided into three sub-sections for the purposes of analysis:

  • Campus operations
  • Corporate services
  • Research activities
Campus Operations
Key Finding: The CRC's governance model is in line with Treasury Board policies and the organization is maximizing the recovery of costs. However, the governance model is not ideal as having the CRC manage the campus detracts from the organization's ability to focus on its stated research mission. Further, the campus is in need of major investments to head off mounting health and safety issues that could put the current governance model at risk. Significant time and energy is directed towards addressing these issues.

The CRC Directorate of Campus Operations is the authority responsible for overseeing the maintenance and development of the Shirleys Bay campus. With a forecast budget of approximately $14.5 million for the 2011-12 fiscal year, the organization manages five primary costs associated with this work, as laid out in Table 8.

Table 8: Campus Operation Costs—2011-12 (Forecast)
Costs Tenants CRC Total Cost of Service
$ % $ % $ %
* excludes capital funding of $4M
Operations and Maintenance $4,603,964 60.1 $3,058,836 39.9 $7,662,800 100.0
Real Estate and Property Services $1,211,672 31.8 $2,596,271 68.2 $3,807,943 100.0
Tenant Specific Projects $1,461,000 100.0 $0 0.0 $1,461,000 100.0
Payment in Lieu of Taxes (PILT) $0 0.0 $1,040,574 100.0 $1,040,574 100.0
Campus Stewardship - Direct $486,390 100.0 $0 0.0 $486,390 100.0
Sub-Total Campus $7,763,026 53.7 $6,695,681 46.3 $14,458,707 100.0

The major costs shared by all departments are utilities (water, sewer, electricity, and fuel), security and emergency operations, campus stewardship and property management. Any expansion or major construction undertaken by a tenant, including administrative and overhead costs, are cost recovered as per the tenancy agreements in place.

Shirleys Bay tenants own their own buildings and are responsible for financing related to their upkeep. Of the 111 permanent and temporary buildings on-site, 47 belong to tenants, 24 are shared between tenants and the CRC. The CRC occupies 40 buildings of its own (46% of buildings on campus) and covers 46.3% of the overall costs.

The document review suggests that in the past the CRC has not always maximized its recovery of costs. However, the CRC has modified its policies with tenants and the organization now recovers what costs it is able to, in accordance with Treasury Board policies. These recovered costs assist with upkeep and maintenance of the campus, but do not address the need for larger real property projects over time.

The Shirley's Bay campus is aging and multiple issues relating to both the CRC's and the overall campus' state of repair have emerged. In fact, the CRC has identified eleven health and safety-related real property projects that must be completed in order to reduce liability. Projects include the upgrade of the heating and cooling distribution system and interior plumbing and electrical systems. This assessment found that the CRC is considering multiple options to rectify the situation as the required funds cannot be collected through existing cost recovery policies.

Key interviews indicate that the management of campus is clearly a preoccupation of the senior management cadre of the organization, who are primarily tasked with the oversight of research. The issue was often raised during interviews, even though not specifically prompted by questions. During this evaluation, it was observed that some senior management positions are affected by health and safety incidents on campus, which reduces time available to manage research-related matters.

In interviews, most CRC managers indicated that they believe managing the CRC campus is not the optimal governance model. Dominant among concerns was that funds must be diverted from the research agenda to cover infrastructure deficiencies. Some CRC managers also suggested that IC does not have a large real property management capacity and that the campus would be better maintained, and the research organizations' budget more stable, if Shirleys Bay was managed by an external organization. One key interview indicated that a better solution would be to improve internal capacity and related authorities to address the issues at hand. Regardless, interviews affirmed a tension between the CRC's duties as a landlord and its primary mission as a federal research centre. This tension necessarily detracts from the core research that should be the organization's primary focus.

The existing CRC governance model may be at risk should the proposed projects to improve the state of CRC buildings and campus infrastructure not proceed. Until existing issues are addressed, another organization is unlikely to be willing to take over as landlord of the campus. Further, existing tenants may opt to move to newer facilities rather than repair their own aging infrastructure.

Corporate Services
Key Finding: The CRC's corporate costs are relatively high compared with other federal research centres. In part, this is due to the CRC delivering its own corporate function as well as supporting activities related to campus operations. There may be some efficiencies gained in centralizing, consolidating, or outsourcing some corporate functions.

In order to assess the efficiency of the CRC's corporate services, a review of other labs of a similar size was conducted. A profile of organizations participating in the review is presented in Table 9Footnote 30.

Table 9: Profile of Organizations Participating in Review of Other Labs (2011-12)
Research Centre Total Budget (approx.) FTEs (approx.) Facility Description
CRC $52.7M 385 Main campus with four smaller areas off-site
A $76.5M 429 One campus
B $58.3M 390 Four campuses in various locations
C $24.0M 224 Four facilities, each facility in a different region
D $25.0M 253 One campus

The review examined corporate services provided within these four organizations, the costs associated with the delivery of those services, and how these costs are managed (i.e. by the research centre itself or by the associated federal department).

Overall, the review found that all of the research centres had comparable corporate service functions, with a few exceptionsFootnote 31. However, differences existed in the way these functions were delivered, who delivered the functions, and how they were funded. In some cases multiple functions were consolidated under a single unit, in other cases functions were centralized at departmental headquarters with no cost to the research centre, and in still other cases some functions were centralized, but paid for by the research centre based on usage. Table 10 indicates where similar corporate services were provided internally across the research organizations.

Table 10: Comparison of Corporate Services Provided Internally
Research Centre Creative Visual Services Communications & Promotions Business Development Technology Transfer Office Finance & Material Management IT & IM Footnote 32 Human Resources
CRC X X X X X X X
A X X X X X
B X X XTable note ** X
C N/A N/A X X X
D N/A

Of the four research centres, only Centre A had equivalent services to CRC managed internally, but it merged four services under two units, with the same FTEs performing multiple roles. Centre B also had the same corporate services as CRC. However, two functions (HR and Finance) were delivered centrally with the research centre paying for any services provided. Further, the centre had merged two functions under one unit. Centre C and Centre D utilised centralized corporate services to a greater extent than the other research centres. In these cases, functions were provided by departmental headquarters, which absorbed all associated costs.

Interviews with these centres suggested that centralizing services is an efficient means of delivery. By centralizing some services, the organizations can rely on the expertise and critical mass built within the corporate branch of the department. This model may avoid some duplication between a research centre and the department's corporate function; reduce dependency on specific personnel; and, allow the research centres to focus primarily on the research at hand.

In terms of assessing the efficiency of corporate activities, a percentage split indicator was applied outlining the amount of resources spent on research relative to the amount spent on corporate support. A summary of the percentage splits is outlined in Figure 2.Footnote 33

Figure 2: Percentage Split between Research Expenditures and Corporate Expenditures between Comparable OrganizationsFootnote 34

Bar chart of Percentage Split between Research Expenditures and Corporate Expenditures between Comparable Organizations (the long description is located below the image)
Description of Figure 2

Figure 2 is a bar graph with a y axis that has the values from 0% to 100% represented and an x axis with four labels: CRC, and three letters representing other research centres: A, B and C. Above each label is a bar that spans the whole range from 0 to 100% on the graph. Each bar is split into two sections, with the lower section representing research expenditures and the upper section representing corporate expenditures. On the CRC bar, research expenditures are 83% and corporate expenditures are 17%. On the bar labelled "A," research expenditures are 90% and corporate expenditures are 10%. On the bar labelled "B," research expenditures are 88% and corporate expenditures are 12%. On the bar labelled "C," research expenditures are 86% and corporate expenditures are 14%.

The percentage split indicates that the CRC allocates relatively more resources towards corporate activities to support research than its counterparts. In part, this higher allocation can be explained by CRC shouldering additional responsibility to provide corporate services that are related to campus operations. That said, both Centre A and Centre B achieve some of their efficiencies through consolidating functions within single units or having some functions performed by the home department. The nature of the work at Centre C precludes the need for some corporate functions (creative visual services and business development), and some services are centralized at no cost to the research centre. As such, the percentage allocated to corporate services is understated.

While centralizing and consolidating functions has assisted some research organizations in gaining corporate efficiencies, there are some unique considerations for CRC. For example, the CRC's Technology Transfer Office manages the organization's suite of patents, including the Fiber Bragg Gratings patents that have played a significant role in the CRC's success compared with similar labs world-wideFootnote 35. While the review of other research centres found that the Technology Transfer function is sometimes housed in the managing department, the CRC has a unique need in this respect that likely warrants a separate unit. It would, however, be prudent for the CRC to monitor the level of activity in this office as the Canadian and American Fiber Bragg Gratings patents will both expire during the 2012 calendar year, and this may have a significant impact on the unit's activities and subsequent requirements.

Additionally, with regards to a client/business development function (a common feature in other research centres), as indicated earlier in this report, multiple lines of evidence suggest that the CRC bolster this activity by introducing a function that is more specialized at articulating the organization's capacities and findings, as well as strengthening its working relationship with SITT. A centralized function may not fulfill CRC or clients unique needs in this area.

Finally, any change to a research centre with respect to corporate services must consider the risks associated with reducing the proportion of funds spent on corporate activities. Interviews indicated that past studies have shown that reducing the corporate function beyond a certain critical mass can place an added burden on the research teams to do this work, taking time away from principal responsibilities. While this may result in operational efficiencies, overly aggressive reductions in corporate support could ultimately result in inefficiencies.

Research Activities
Key Finding: Research activities at the CRC are not currently functioning at optimal levels with respect to efficiency. This is not a by-product of the capacity of researchers, but is rather an artifact of legacy programs and silo-based management practices.

Expert panels globally indicated that the organizational structure of the research function at the CRC is based on "historical precedents and legacy responsibilities, rather than using a structure that should more accurately reflect the relationships of technologies they are using today, (and in the future towards which the CRC research is aiming)." Over the course of this assessment the CRC was, in fact, working toward modernizing its management and organizational structure.

The review of other labs conducted as part of this assessment found that other labs manage their research activities somewhat differently than the CRC. For example, organizations informed evaluators that they had detailed process manuals that standardized and streamlined project management, enabling research teams to more efficiently conduct their projects. One organization had ISO certification, demonstrating best practices in the field of project management. At the time of this evaluation, it was not observed that the CRC had any such processes in place.

Multiple lines of evidence pointed to the best mechanism for modernization and renewal of research as being expert reviews: the 2009 Review of the Communications Research Centre recommended the convening of such expert panels as a performance indicator and the review of other labs revealed that it is common practice in other federal research centres to regularly assess research in light of alignment with strategic objectives, departmental priorities and stakeholder needs. Perhaps most significantly, the expert panels convened for this evaluation make a sweeping call for more regular reviews of the research function.

It was observed during this assessment that the research programs of the CRC are largely silo-based. While managers' surveys indicated that some collaboration goes on between research groups, expert panels noted that efficiencies could be gained by select research programs working more closely together.

For example, one panel noted that a number of research programs were working on elements of cognitive radio, but there did not seem to be a coordinated effort within the organization. Another panel noted that one research program in particular is valuable in that it conducts activities that support decisions around the efficient use of spectrum and new spectrum bands within SITT, however, the program was not conducting true research and so was not in the position to support industry in the design and qualification of products. To harness untapped potential within this team, the panel recommended that the program be combined with a more research-oriented team to rationalize any activities in common and meet joint needs.

Similarly, panels noted that high-performing teams may gain capacity by working together. For example, one panel identified two programs that act as test beds and develop prototypes and demonstrations of wireless and satellite systems respectively. It was suggested that these two groups could be combined to a greater effect as a "demonstration development" or "applications" group.

Panels also identified potential efficiencies in the sharing of assets and equipment across research groups. For example, the panels noted that according to equipment lists provided by the 18 groups reviewed, the CRC owns a fleet of 20 equipped motor vehicles (including SUVs, ATVs and vans) for field measurement and transmission characteristics and other data; and an additional major vehicle investment had been requested at the time of the panel. The panel stated that it is possible that greater efficiency could be reached via more sharing of resources, such as vehicles.

Also of note are potential efficiencies in making use of the facilities of other research organizations. For example, one expert panel noted that the CSATemp's David Florida Lab on Shirleys Bay campus offers antenna testing facilities that could be of use to the CRC. Similarly, a panel also noted that the National Research Council (NRC) had recently made an investment in printed electronics and a panel noted that it may be more efficient to forge a collaborative relationship with the NRC rather than make a significant investment in this area.


4.0 Conclusions and Recommendations

Major conclusions reached during the evaluation are summarized. A set of recommendations are presented to improve the ability of the CRC to meet its objectives.

4.1 Conclusions

Regarding relevance, the evaluation determined that:

  • There is a continued need for the CRC, however, some programs no longer respond to the needs of the organization as well as they may have in the past.
  • The CRC is aligned with federal government priorities. There was considerable latitude with respect to how individual research programs fulfill these priorities in the past. A recent CRC strategic planning exercise involving SITT may change this over time, but its effects have not yet filtered down to working-level activities.
  • The CRC aligns with the roles and responsibilities of the federal government.

Regarding performance, the evaluation determined that:

  • The work being carried out by the CRC is world-class. There are very few pockets of activity among the research programs that do not meet the organization's high standard of excellence.
  • In the absence of a logic model, it is difficult to ascertain whether or not the CRC is meeting its expected outcomes. There is some evidence to demonstrate that the CRC is making progress towards its stated objectives. However, the program needs to improve its performance measurement.
  • Multiple lines of evidence suggest that clients benefit from the CRC's unbiased advice in terms of scientific findings, technical solutions and IP licensing. However, a more robust client/business development function is required.
  • It is currently not possible to track expenditures and resource use on individual research projects, largely due to the absence of a time tracking system for researchers. This makes it difficult to determine the true cost of services provided to clients. These systems are in use in other Government of Canada research centres.
  • The CRC's governance model is in line with Treasury Board policies and the organization is maximizing the recovery of costs. However, the governance model is not optimal as it detracts from the organization's ability to focus on its stated research mission. Further, the campus is in need of major investments to head off mounting health and safety issues which may put the governance model at risk.
  • The corporate costs are relatively high compared with other federal research centres, however, some of the organization's corporate services are allocated to supporting tenant services. Efficiencies in corporate costs may be gained if some functions were consolidated, centralized, or outsourced.
  • Research activities at the CRC are not functioning at optimal levels, largely due to an organizational structure that is based on historical precedents and legacy responsibilities.

4.2 Recommendations

The conclusions of the evaluation led to the following recommendations:

  1. The CRC should seek internal efficiencies and respond to the modern realities of ICT research within government.
    1. Consider the recommendations of the expert panels, particularly those which address areas of overlap, inefficiencies and topics of lower strategic value to the organization
    2. Seek more efficient solutions for corporate services
    3. Set regular intervals for the review of research lines in terms of their excellence, relevance and impact
    4. Continue to work toward a governance model that will focus the attention of the organization on its research mandate
  2. The CRC should improve its tracking of resource use on research and work performed for clients.
    1. Researchers should track the time they spend on all projects to determine the true cost of services to clients and to ensure appropriate billing
    2. Systems should be established to ensure that reports on client-based activities by research program can be produced
  3. To better communicate its capabilities, the CRC should strengthen its client/business development interface. Consideration should be given to:
    1. Articulating and promoting of the CRC's capabilities to clients and stakeholders
    2. Collecting market intelligence on new and emerging trends
    3. Actively updating a client feedback mechanism at the conclusion of research projects
    4. Formally acting as a catalyst for a stronger working relationship with SITT
  4. To better evaluate the performance of research, the CRC should continue to work on performance measurement for outputs as well as identifying and measuring immediate, intermediate and ultimate outcomes.

Management Response and Action Plan

Management Response and Action Plan
Recommendation Management Response and Planned Action Management Accountability Action Completion Date
  1. The CRC should seek internal efficiencies and respond to the modern realities of ICT research within government.

Agreed.

   
  1. Consider the recommendations of the expert panels, particularly those which address areas of overlap, inefficiencies and topics of lower strategic value to the organization
  1. Panel recommendations were taken into account when recently launching a research and operations re-organization of CRC. Less relevant research will be abandoned and consolidation of talent and expertise will be done to reduce overlap and increase efficiencies. Twenty research groups will be realigned to be reconfigured to ten new ones.

President

Organization designed by September 30, 2012

  1. Seek more efficient solutions for corporate services
  1. CRC will seek more efficient solutions for corporate services. A plan will be developed to determine which functions would be better centralized or outsourced.

Vice President Business Development

Plan in place by September 30, 2012

  1. Set regular intervals for the review of research lines in terms of their excellence, relevance and impact
  1. CRC will address this on 2 fronts. It will first conduct project reviews 3 times a year to ensure that resources are aligned on relevant work for its clients.

    It will then develop a plan and schedule, likely on a four or five year cycle for review of the research lines in term of their relevance, excellence and impact.

Vice President Business Development

Project reviews implemented by May 30, 2012.

 

Research review plan to be completed by November 30, 2012

  1. Continue to work toward a governance model that will focus the attention of the organization on its research mandate
  1. CRC will invest funds into the campus to address the most pressing health and safety and infrastructure issues.

    CRC is preparing a business case that will identify the projects that pose the most significant health and safety and environmental liability issues, along with the associated financial requirements.

    CRC will continue to meet with its campus partners and senior management to work toward a governance model that will enable it to focus on its research mandate.

President

March 31, 2012

President

March 31, 2012

President

Ongoing

  1. The CRC should improve its tracking of resource use on research and work performed for clients.

Agreed.

 

 

  1. Researchers should track the time they spend on all projects to determine the true cost of services to clients and to ensure appropriate billing
  1. CRC will implement a system to enable researchers to track the time they spend on all projects to determine the true cost of services to clients and to ensure appropriate billing.

Director Finance Materials Management

System implemented and online by September 30, 2012

  1. Systems should be established to ensure that reports on client-based activities by research program can be produced
  1. CRC will implement a system to ensure that reports on client-based activities by research projects can be produced. Program information will be retrieved from rolling up the reports of various projects composing the various research programs.

Director Finance Materials Management

Reports implemented and online by November 30, 2012

  1. To better communicate its capabilities, the CRC should strengthen its client/business development interface. Consideration should be given to:

Agreed.

 

 

  1. Articulating and promoting the CRC's capabilities to clients and stakeholders
  2. Collecting market intelligence on new and emerging trends
  3. Actively updating a client feedback mechanism at the conclusion of research projects
  4. Formally acting as a catalyst for a stronger working relationship with SITT

CRC will create a business development function to consolidate its client interface and market presence. A Vice President Business Development position will be created.

President

Position to be staffed by June 30, 2012

The business development branch will be organized to address points a) through d)

Vice President Business Development

Organization to be defined by September 30, 2012

  1. To better evaluate the performance of research, the CRC should continue to work on performance measurement for outputs as well as identifying and measuring immediate, intermediate and ultimate outcomes.

Agreed

Through the creation of a Business Development function, CRC will align with its clients and stakeholders to define and document how the research outputs contribute to the realization of the programs immediate, intermediate and ultimate outcomes. It will create a research performance measurement framework that will be aligned with the Industry Canada PAA 2.2 and SITT's Performance Management Framework.

Vice President Business Development

Research performance framework in place by February 28, 2013