Evaluation of the Certification and Engineering Bureau

PDF version

631 KB, 30 pages

June 2016

Presented to the Performance Measurement and Evaluation Committee on February 8, 2017

Approved by the Deputy Minister on May 29, 2017

Table of Contents

• Executive Summary
• 1.0 Introduction
  • 1.1 Program profile
  • 1.2 Logic model
• 2.0 Profile
  • 2.1 Evaluation objectives and scope
  • 2.2 Evaluation approach
  • 2.3 Evaluation questions
  • 2.4 Data collection methods
  • 2.5 Limitations
• 3.0 Findings
  • 3.1 Relevance
  • 3.2 Performance
• 4.0 Conclusions and recommendations
  • 4.1 Relevance
  • 4.2 Performance
  • 4.3 Recommendations
• Management Response and Action Plan
• Summary of the Evaluation of the Certification Engineering Bureau (CEB)

List of acronyms

AEB

Audit and Evaluation Branch

ANSI

American National Standards Institute

CB

Certification Body

CBSA

Canada Border Services Agency

CEB

Certification and Engineering Bureau

CRC

Communications Research Centre

CRF

Consolidated Revenue Fund

DGEPS

Engineering, Planning and Standards Branch

DGSO

Spectrum Management Operations Branch

DRS

Directorate Regulatory Standards

DSL

Digital Subscriber Line

FCC

Federal Communications Commission

FYE

Fiscal Year-End

GDP

Gross Domestic Product

GPS

Global Positioning System

IEEE

Institute of Electrical and Electronic Engineers

ISED

Innovation, Science and Economic Development

IT

Information Technology

ITU

International Telecommunication Union

KDB

Knowledge Database

LED

Light Emitting Diode

LG

Life's Good

MURS

Multi-Use Radio Services

OGD

Other Government Departments

RCMP

Royal Canadian Mounted Police

REL

Radio only and hybrid equipment list

RF

Radio Frequency

RSS

Radio Service Standards

SAM-CSI

Spectrum Application Modernization Commercial Software Implementation

SAR

Specific Absorption Rate

SITT

Spectrum, Information Technologies and Telecommunications

TAC

Technical Acceptance Certificate

TAR

Telecommunications Apparatus Registry

TCBC

Telecommunication Certification Body Council

TIA

Telecommunications Industry Association

USA

United States of America

WWRF

Wireless World Research Forum

List of tables

• Table 1: CEB revenue and spending profile
• Table 2: CEB's non-compliance activities, all audits, 2012–13 to 2014–15
• Table 3: CEB's technical non-compliance activities, physical audits, 2012–13 to 2014–15

List of figures

• FIGURE 1: Logic model for CEB
• FIGURE 2: Total number of applications submitted to CEB from 2010–11 to 2014–15

Executive summary

Program overview

The Certification and Engineering Bureau (CEB) is part of the Engineering, Planning and Standards Branch (DGEPS), within the Spectrum, Information Technologies and Telecommunications (SITT) Sector of Innovation, Science and Economic Development (ISED). The Bureau's overall objective is to contribute to a fair and competitive radiocommunication and telecommunications marketplace.

The Bureau works to ensure that radio and telecommunications equipment used in Canada meet the requirements of all applicable standards. CEB certifies and registers equipment pursuant to the Radiocommunication Act and Telecommunications Act; approves and registers laboratory test sites that perform compliance assessments; monitors compliance through market surveillance; and provides support to external clients to comply with standards and ensure timely market accessibility. The Bureau also provides technical expertise and support to DGEPS and participates in standards development committees, which helps to improve regulations/standards development and spectrum planning within the department.

Evaluation purpose and methodology

The purpose of this evaluation was to address the core issues of relevance and performance in accordance with the Policy on Evaluation and the Directive on the Evaluation Function, with a focus on assessing how CEB could improve its efficiency to address an anticipated increase in workload. The evaluation findings and conclusions are based on the analysis of multiple lines of evidence. The methodology included a document review, literature review, program data review, interviews, and an Alternatives, Best Practices and Trends Study.

Findings

Relevance

There is a continued need for CEB. Its importance is reflected in the growing demand for its services and the potential negative economic and social impacts if CEB did not play an active role in the market. Demand for CEB's services is expected to continue to grow given the rapid proliferation of wireless devices and technological developments.

CEB plays a key role in supporting a number of federal government priorities, including improving high-speed broadband access and growing the digital economy, which rely on seamless use of technology. The Bureau's objectives are also consistent with the department's priority to foster a strong investment environment for telecommunication services.

The Bureau's activities align with the roles and responsibilities of the federal government to ensure that radio and telecommunications equipment used in Canada meet all applicable standards, as stated in the Radiocommunication Act and the Telecommunications Act.

Performance

CEB has largely been successful in achieving its short-term outcomes. The Bureau has contributed to improved regulations, standards and spectrum planning within the department, has certified/registered radio and telecommunications equipment in a timely manner, and has helped to ensure compliance with standards via its targeted market surveillance and proactive approach to resolving non-compliance.

The Bureau has also demonstrated success in achieving its long-term outcomes of helping prevent interference, ensuring technologies are safe as per Health Canada's Safety Code 6 requirements^{Footnote 1}, promoting timely market access and contributing to a fair marketplace.

CEB has demonstrated efficiency and economy during the evaluation period and has been planning for an anticipated increase in workload. Further efficiencies could be achieved through increased communication within the department and with industry stakeholders and other administrations, as well as by reducing its pre-market certification activities to focus on assessing newer technologies and testing procedures. The evaluation also identified room for improvement with respect to CEB's performance measurement and reporting.

Recommendations

CEB has been successful in achieving its outcomes to date. One of the key challenges CEB faces is how to meet its objectives in the future given what is expected to be a significant increase in complex wireless products entering the marketplace. While it is difficult to pinpoint which products will enter the marketplace and how soon, it is clear that the wireless world is a moving target and CEB needs to be ready to deal with a rapidly changing environment. More specifically, the Bureau needs to ensure it keeps pace with the latest changes in technology in order to remain agile and adapt its management and testing procedures as technologies evolve. With this in mind, CEB should:

1. Consider reducing its pre-market certification activities and focus on assessing newer technologies.
2. Examine options for increased information sharing within the department to better understand the environment for technological developments, which will enhance CEB planning.
3. Further leverage its partnership with key foreign administrations by establishing information sharing agreements.
4. Consider building on the results of this evaluation by revising its performance measurement and reporting accordingly.

1.0 Introduction

1.1 Program profile

The Certification and Engineering Bureau (CEB) is part of the Engineering, Planning and Standards Branch (DGEPS), within the Spectrum, Information Technologies and Telecommunications (SITT) Sector of Innovation, Science and Economic Development (ISED).

Objectives of CEB

The objective of CEB is to contribute to a fair and competitive radiocommunication and telecommunications marketplace. The Bureau works to ensure that radio and telecommunications equipment used in Canada, whether manufactured in or imported to Canada, meets the requirements of all applicable standards for specific types of equipment.

In the case of telecommunications equipment, technical standards are prescribed to ensure that no harm is caused to the network and that features are properly implemented for the benefit of Canadian consumers as well as Canadian businesses. This encourages a fair marketplace for radio and telecommunications product manufacturers and suppliers and prevents radio frequency interference or harm to the public switched telephone network.

For radio apparatus (e.g. cell phones and other hand-held radiocommunication devices), the technical standards ensure that the limits set forth by Health Canada on the exposure of radio frequencies are met, particularly concerning the limits for Specific Absorption Rate (SAR)^{Footnote 2}. Ensuring that radio equipment is certified therefore reduces possible harm that high exposure to radio frequency electromagnetic fields could cause to the human body.

Description of CEB

CEB consists of laboratory facilities, equipment and a team of professionals that provide certification and regulatory functions. Its key activities are described below.

Equipment Certification and Registration:
CEB certifies and registers wireless and telecom equipment pursuant to the Radiocommunication Act and Telecommunications Act. There are three types of equipment subject to federal regulations:

• Type 1: radio equipment, which requires certification, includes high-powered transmitters such as broadcast transmitters, taxi or police mobile radios, satellite telephones and cellular telephones;
• Type 2: radio equipment, which requires a declaration of conformity, includes low power transmitters and receivers such as remote control toys, wireless mouse and keyboards, Global Positioning System (GPS) receivers, some television satellite receivers and other consumer electronic devices such as microwave ovens, DVD players and computers; and
• Type 3: telecommunication equipment, which requires registration and includes analogue wireline telephones, fax machines, wireline modems, and Digital Subscriber Line (DSL) customer equipment.

Manufacturers or importers must apply for equipment certification by CEB or a recognized certification body (CB)^{Footnote 3} and/or register terminal equipment in Canada. Manufacturers/importers can use ISED's e-filing service to apply and ensure their equipment is either already certified or tested. Equipment is tested for compliance and certified by a recognized CB or CEB. CEB adds certified and/or registered equipment to the Telecom Apparatus Registry (TAR) or radio only and hybrid equipment list (REL) and maintains these published lists. Once the equipment is listed, it can be manufactured, imported, distributed, leased, sold or used in Canada. CEB charges fees for technical expertise provided to external clients, assessment of radio equipment for the purpose of certification, registration in the TAR and listing in the REL.

Test Facility Registration: CEB approves and registers laboratory test sites located worldwide that are used to assess equipment's compliance with federal standards. CEB maintains a database and publishes the lists of all registered companies, certification bodies, and terminal or radio laboratories.

External Client Support: CEB provides support to clients [e.g. Blackberry, Apple, Samsung, Life's Good (LG), Cisco, etc.] to comply with certification standards and ensure timely market accessibility. Support to external clients may include responding to inquiries, providing IT support, providing technical information or conducting measurement studies.

Internal Client Support: CEB provides technical expertise to internal ISED clients. In particular, the Bureau contributes to the development of the standards (e.g. terminal, wireless and SAR) by DGEPS. CEB contributes to both international and domestic standards through participation in various standards development organizations such as the International Electrotechnical Commission (IEC), the Institute of Electrical and Electronic Engineers (IEEE), the American National Standards Institute (ANSI) and the Telecommunications Industry Association (TIA). CEB also provides spectrum and telecommunications measurement services, including specialized lab tests and field studies or surveys in support of spectrum engineering projects and planning exercises or in interference investigation and resolution.

Market Surveillance: CEB conducts market surveillance according to an annual plan, targeting higher risk products and equipment, and subjects selected equipment to either desk audits or physical audits to determine compliance of equipment with standards. CEB provides the documentation on cases of non-compliance to ISED's Spectrum Management Operations Branch (DGSO), who is responsible for enforcement of the standards.

Resources

CEB's revenue and spending profile are outlined in Table 1 below. Total program spending over five years (2010–2011 and 2014–15) has been approximately $2.8 million per year, with salary expenses accounting for approximately 64% of spending ($1.8 million per year). CEB has 24 staff members.

The Bureau receives an average of $597,779 per year in revenues (from fees for service paid by manufacturers and suppliers). These funds are not re-spent on CEB activities, but are returned to the Consolidated Revenue Fund (CRF).

1.2 Logic model

The logic model (on the next page) shows how CEB's activities are expected to lead to certain outputs and various levels of outcomes, and ultimately, to one of ISED's strategic outcomes. The program's logic model was developed in consultation with CEB staff prior to the evaluation.

Figure 1: Logic Model for CEB

2.0 Profile

2.1 Evaluation objectives and scope

The objectives of the evaluation were to address the core issues of relevance and performance in accordance with the Policy on Evaluation and the Directive on the Evaluation Function, with a focus on assessing how CEB could improve its efficiency to address current and future anticipated workload, as requested by DGEPS management. The evaluation covered the five-year period of 2010–11 from 2014–15 and included more current information and data where possible.

2.2 Evaluation approach

As the primary focus of the evaluation was to assess how CEB could improve its efficiency, a key source of information for the evaluation was an independent study examining alternatives/best practices to CEB's current operations that could be considered, as well as trends in technology development that could impact the CEB. Further, given that CEB accounts for a small portion of departmental spending, the assessment of CEB's effectiveness was streamlined, using four data collection methods (outlined further below) in addition to the study, for which only key information and data was collected.

2.3 Evaluation questions

The evaluation sought to address the following questions:

Relevance

1. What is the importance of the CEB? Has it changed over time and how is it expected to change in the near future?
2. Does the CEB align with the priorities of federal government and the strategic outcomes of ISED?
3. Do CEB activities align with the roles and responsibilities of the federal government?

Performance

4. To what extent has the CEB achieved its short-term outcomes?
   • How effective is the CEB in improving regulations, standards and spectrum planning?
   • How effective is the CEB in ensuring all radio and telecommunications equipment in Canada is certified/registered in a timely manner?
   • To what extent has the CEB contributed to ensuring that radio and telecommunications equipment in Canada is compliant with standards?
5. To what extent has the CEB achieved its long-term outcomes?
   • To what extent has the CEB contributed to ensuring that radio equipment in Canada operates without negatively interfering with each other?
   • To what extent has the CEB contributed to ensuring technologies used by Canadian consumers are safe as it relates to safety requirements falling under CEB's purview?
   • To what extent has the CEB contributed to timely market accessibility?
   • To what extent has the CEB contributed to a fair marketplace for radio and telecommunications product manufacturers and suppliers?
6. To what extent does CEB demonstrate efficiency and economy? Can CEB improve efficiency and economy to be able to address current and future workload?

2.4 Data collection methods

Multiple lines of evidence were used to address all evaluation questions. The data collection methods included a document review, literature review, program data review, interviews, and an Alternatives, Best Practices and Trends Study.

Document Review

A document review was conducted to gain an understanding of CEB and provide insight into the relevance and performance of the Bureau. Key documents included relevant legislation and policy documents (i.e., acts, regulations and policies), Departmental Reports on Plans and Priorities, Departmental Performance Reports, as well as CEB procedures, plans and Fiscal Year-End (FYE) Reports.

Literature Review

The literature review addressed the core evaluation issues of relevance (i.e., ongoing need) and some performance issues. With respect to the latter, the evaluation team looked for evidence regarding the extent to which CEB has helped ensure that technologies used by Canadian consumers are safe, and efficiency and economy in terms of CEB's resource allocation and utilization compared to similar programs in other countries.

Program Data Review

Program data was analyzed primarily to assess the effectiveness of the program (e.g., certification and registration statistics, compliance rates, client survey results), as well as efficiency (e.g. client service metrics, processing times, overtime, and financial data).

Interviews

The objective of the interviews was to gather in-depth information for evaluation purposes, including views, explanations and factual information that address the evaluation questions. The interviews were semi-structured in nature and were designed to obtain qualitative feedback from a range of respondents. Interviews were conducted with a total of 31 interviewees and included:

• Current and former CEB staff (6)
• Other ISED staff (8)
• Other Government Departments (OGDs) [Health Canada, Royal Canadian Mounted Police (RCMP) and Canada Border Services Agency (CBSA)] (3)^{Footnote 5}
• Standards setting bodies (3)
• Industry representatives (11)^{Footnote 6}
  • Certification bodies and labs (5)
  • Manufacturers (6)

Alternatives, Best Practices and Trends Study

This study sought to address the evaluation issues related to efficiency and economy through examining alternatives/best practices to CEB's current operations that could be considered, as well as trends in technology development that could impact the Bureau. The study was conducted by two external contractors that are knowledgeable about research and development in radio and telecommunications, the market for these products, and the regulatory environment in which CEB operates.

The study included a limited literature review of technological developments and trends, document review of ISED's and other administrations' information, and data review from foreign administration web sites. In addition, interviews were conducted with current and former staff, industry representatives (the same interviewees identified in the previous section were interviewed for the overall evaluation as well as this study) and foreign administrations. Specific foreign administrations interviewees included U.S., Germany, Europe, Switzerland and China. The study was designed in close collaboration with AEB and CEB staff.

2.5 Limitations

The following were limitations to this evaluation:

Attribution of outcomes to CEB

CEB's performance is influenced by other areas of SITT. DGSO also contributes to ensuring compliance with standards by taking enforcement action when required in the event of non-compliance. The speed at which the Directorate Regulatory Standards (DRS) branch develops standards also contributes to timely market access. This presents challenges in attributing the success of the outcomes achieved specifically to the CEB. The evaluators mitigated this to the extent possible. For example, efforts were made to ensure interviewees were referring specifically to the CEB in their responses.

Performance measurement and reporting

The evaluation relied heavily on CEB's FYE reports, which presented some challenges in assessing CEB's achievement of expected outcomes. For example, these reports did not readily provide information that allowed the evaluators to assess CEB's effectiveness as it relates to interference. Inconsistencies in reporting non-compliance data were also identified. Further, only three years of CEB's non-compliance activity data (2012–13 to 2014–15) was readily available; as such the full five-year evaluation period was not assessed for all outcomes. To mitigate for these limitations, the evaluators worked with CEB staff to re-categorize, collect and analyze appropriate program data in order to better capture CEB's performance. In addition, the evaluation provides a recommendation to strengthen this area.

3.0 Findings

3.1 Relevance

3.1.1 What is the importance of the CEB? Has it changed over time and how is it expected to change in the near future?

The importance of CEB

CEB works to ensure that radio and telecommunications equipment entering into and used in Canada operates within established standards. The Bureau plays a critical role in ensuring that Canadians benefit from timely access to technologically advanced goods and promoting an environment where radio and telecommunications equipment being used are compatible with one another.

The importance of CEB was further echoed by stakeholder interviewees, who indicated that without the CEB there would be significant economic and social consequences for Canadians as devices would enter the market that could cause interference for other users (including those involved in public safety such as emergency services) and expose Canadians to potentially harmful levels of radiofrequency (RF) electromagnetic energy.

Literature suggests that the radio and telecommunications sector that CEB supports is important for the Canadian economy. One article estimates that the wireless communications industry generated a total value of nearly $43 billion for the Canadian economy in 2010^{Footnote 7}. A more recent report estimates that the telecommunications industry contributed close to $32 billion to the Canadian GDP in 2014^{Footnote 8}.

Demand for CEB in the last five years

Program data suggests that demand for CEB's services has generally increased over the last five years. As per Figure 2 below, with the exception of a small decrease in 2014–15, the total number of applications submitted has steadily increased, from 4,220 applications in 2010–11 to 5,272 in 2014–15.

Figure 2: Total number of applications submitted to CEB from 2010–11 to 2014–15

Future demand for CEB

Literature reviewed as part of the Alternatives, Best Practices and Trends study and the evaluation suggest that demand for CEB will continue to increase as the number of devices and appliances are expected to grow significantly in the future. One article predicts that, globally, the total number of mobile devices will grow from 6.8 billion in 2014 to 9.2 billion in 2019^{Footnote 9}. Another article predicts a ten-fold global increase in the number of wireless devices between 2011 and 2022^{Footnote 10}.

Wireless technology developments have also grown dramatically in the last 10 years. The two key contributors of this growth has been fifth generation (5G) services^{Footnote 11} which will support ubiquitous mobility, people and appliances and the Internet of Things (IoT)^{Footnote 12} which will introduce much more interconnectivity and intelligence into appliances^{Footnote 13Footnote 14}. The study points to a number of wireless devices and applications that are being developed that can be embedded in cars, machines, transportation systems, and people's bodies that can connect across different frequency bands and other technologies. "Smart Cities" are being developed that include services such as connected parking, lighting and waste management. This is expected to lead to the development of new and more interconnected devices and appliances, thereby increasing the volume and complexity of equipment to be certified/registered in the future. For example, 5G technologies support a much higher user density, wherein users will expect high speed performance in the presence of a large number of concurrent users in close proximity (e.g., in a stadium, shopping mall and open air festival or in emergencies such as environmental disasters). In this case, CEB will need to understand these technologies and ensure appropriate test procedures are in place to meet standards and to mitigate interference.

The study also points to a number of technology trends that are rapidly emerging in new industries that CEB may be required to interact with in the future. Several examples are provided to support the notion that technology has progressed significantly in different industries (e.g., wearable and implantable devices in the medical field, devices to support self-navigating cars in the automotive sector) and is expected to continue to do so^{Footnote 15Footnote 16Footnote 17}. If these developments materialize, CEB will have to interact with and educate these industries accordingly, and will also need to interact with other government departments (e.g., Health Canada, Transport Canada) in approving these devices and ensuring they do not create interference issues.

3.1.2 Does the CEB align with the priorities of government and the priorities of ISED?

Alignment with Federal priorities

Though not mentioned explicitly, CEB plays an important supportive role in delivering on a number of recent federal government priorities.

Budget 2016 denotes the government's commitment to strengthening the middle class by improving high-speed broadband access for rural communities. CEB plays a supportive role by ensuring that rural communities can use associated devices safely and without interference issues.

The 2015 Mandate Letter to the Minister of Innovation, Science and Economic Development identifies a Ministerial priority to "foster a strong investment environment for telecommunication services" to keep Canada at the leading edge of the digital economy. CEB's efforts to prevent substandard equipment from entering the Canadian marketplace instills confidence in the Canadian marketplace, which in turn encourages investment in telecommunications.

Digital Canada 150, launched in 2014, aims to support Canada in the digital age, stating that "Canada's long-term success and prosperity depends on it." CEB plays a key role in supporting a number of pillars outlined in the initiative (i.e., connecting Canadians online, providing Canadian businesses with opportunities in the global digital economy and leading on digital government) as seamless use of digital technology is required for achieving these objectives.

Given its focus on compliance, CEB activities are also consistent with federal government requirements identified in Treasury Board's Cabinet Directive on Regulatory Management. Section (K) (42) states that "Departments and agencies are responsible for promoting regulatory effectiveness by developing and implementing compliance and enforcement strategies". Further, section 2 (iv) stipulates that when regulating, the government will "promote a fair and competitive market economy that encourages entrepreneurship, investment, and innovation." This aligns with CEB's objective to "maintain a fair, equitable and competitive marketplace in the area of telecommunications and radiocommunication"^{Footnote 18}.

The Bureau's activities align with ISED's strategic outcome of an efficient and competitive marketplace by promoting timely market access to technologically advanced products.

3.1.3 Do CEB activities align with the roles and responsibilities of the federal government?

The Radiocommunication Act and the Telecommunications Act indicate that ISED is the federal department responsible for ensuring that radio and telecommunications equipment used in Canada meet the requirements of all applicable standards for specific types of equipment. CEB is the primary mechanism by which the department fulfills these responsibilities. The Bureau seeks to verify that radio and telecommunications equipment comply with standards and works proactively with its clients to accomplish this by providing IT support and technical information, and working to resolve non-compliance issues to the extent possible when they arise.

ISED's responsibility to carry out market surveillance is linked to section 4 of the Radiocommunication Act and section 69.2 of the Telecommunications Act, which prohibits the manufacture, distribution, lease, offer for sale, sale or importation of any radio or telecommunication equipment for which technical standards have been established under the Act unless the equipment complies with those standards. This aligns with CEB's core activity of conducting market surveillance to determine compliance of radio and telecommunications equipment with standards.

The Minister's powers related to issuing certificates is established in section 5.1 (a) of the Radiocommunication Act and section 69.2 (1) of the Telecommunications Act, and the Minister's powers related to testing radio and telecommunications apparatus for compliance is established in section 5.1 (g) of the Radiocommunication Act and section 69.3(1) of the Telecommunications Act. This aligns with CEB's activity of certifying radio equipment based on compliance to applicable standards, and testing radio and telecommunications equipment for compliance to these standards.

3.2 Performance

3.2.1 How effective is the CEB in improving regulations/standards and spectrum planning?

ISED staff indicate that CEB makes an important contribution to improving regulation, standards and spectrum planning through its technical expertise, as well as by sharing its knowledge of the marketplace. ISED interviewees explained that due to its regular interaction with industry stakeholders, CEB is able to provide valuable feedback on regulations and standards being developed within the department. In 2014–15, the Bureau actively participated in the review of, and contributed to, the development of 16 different ISED regulatory standards and procedures, and led the development process for two ISED regulatory procedures.

Information gleaned from the document review and interviews suggest that CEB is making positive contributions to improved regulations, standards and spectrum planning. For example, a measurement study conducted by CEB in 2014–15 evaluating the risk of interference caused by Multi-Use Radio Services (MURS)^{Footnote 19} devices already certified in the U.S. found that MURS could potentially interfere with existing licensees, including public safety users in the same band. This contributed to a decision to delay the introduction of these devices in Canada. Also in 2014–15, CEB conducted a measurement study on Light Emitting Diode (LED) light bulbs in response to a complaint of interference with police radios that were linked to LED lights. This identified the need to update ISED standards to include LEDs and other lighting systems.

Standards bodies commended CEB's proactive approach to reviewing standards via its participation in international standards development committees, which has also led to improved standards and measurement methodologies within the department. Recent examples include incorporating IEEE 1528 (an international standard SAR measurement procedure) into ISED's Radio Standards Specification (RSS) 102^{Footnote 20} and ANSI C63.26 (an American standard on licensed radio apparatus compliance testing) incorporated into ISED's RSS-General, which outlines the general requirements applicable to radio apparatus used for radiocommunication other than broadcasting.

3.2.2 How effective is the CEB in ensuring all radio and telecommunications equipment is certified/registered in a timely manner?

CEB has consistently certified and registered equipment in accordance with its service standards for the last five years. CEB's targets include assessing radio equipment for the purpose of certification within 14 calendar days at least 90% of the time, and two business days for registering equipment in the TAR/listing equipment in the REL at least 90% of the time. Industry representatives generally agree that CEB is providing timely service. Most industry representatives perceived CEB to be as effective as, or more effective than, other regulators such as the United States (U.S.) Federal Communications Commission (FCC) in this area.

3.2.3 To what extent has the CEB contributed to ensuring that radio and telecommunications equipment in Canada is compliant with standards?

CEB's market surveillance process is comprised of two components: "reactive" market surveillance resulting from the receipt of a complaint regarding suspected equipment non-compliance^{Footnote 21}, and "proactive" market surveillance, wherein the Bureau targets higher risk products and equipment based on factors such as history of non-compliance, the Certification Body/manufacturer's familiarity with CEB's requirements, low product price and popularity/mass deployment of the product^{Footnote 22}.

Desk audits are conducted primarily to verify the quality of the work performed by the Certification Body or test laboratory^{Footnote 23}. CEB conducts two types of desk audits, the most common one being post-market desk audits based on its targeted market surveillance plan. Subject to available resources, the Bureau also conducts desk audits when notifications are received from CBs in an effort to resolve issues prior to having non-compliant products introduced into the Canadian market (i.e., pre-market).

The average non-compliance rate for desk audits (based on the total number conducted) was 27%^{Footnote 24}. The most common type of non-compliance identified involved "CB or test-related" administrative non-compliance. Examples include: missing or incorrect information/documents from/in the product's REL/TAR listing; the product not being listed in the REL (when applicant confirmed product is certified) and/or TAR (when applicant used a CB/test lab to register its product with ISED); and missing test cases/test results or incorrect method of measurement from/in the test report^{Footnote 25}.

Physical audits involve inspecting and testing one sample (or more) of a product to verify its compliance with applicable federal regulatory requirements. The average non-compliance rate for physical audits (based on the total number conducted) in the last five years was 30%. Of these cases, 63% involved technical non-compliance (i.e., test results indicate that the product does not comply with applicable technical requirements). The remaining 37% involved administrative non-compliance and were often "product-related". Examples are similar to those described above (i.e., product not certified and/or not registered, product not listed in the REL /TAR), as well as missing or incorrect label and/or applicable markings/user manual statements.

It is important to note that the amount of non-compliance identified by CEB should not be seen as a reflection of the Bureau's effectiveness. Rather, the Bureau's role is to look for non-compliance issues and to work with industry to resolve non-compliance cases to the extent possible. Also, because CEB targets higher risk equipment in its market surveillance, it may not reflect the actual amount of non-compliance in Canada for all equipment requiring certification/registration.

CEB aims to resolve non-compliance cases by working with the responsible parties without the need for taking enforcement measures. As demonstrated in Table 2 below, CEB successfully resolved the majority (an average of 70% including resolution at both the pre- and post-market stage) of the non-compliant cases identified in its audits in the last three years (2012–13 to 2014–15). "Pre-market" resolution includes cases where CEB sent a submission back to an applicant after a non-compliant desk audit result was identified based on a notification from a CB. "Post-market" resolution typically includes CEB working with the manufacturer or applicant to develop an action plan to resolve the non-compliance issue. An average of 11% were sent to DGSO for enforcement action and 15% were "in process", meaning that CEB was still in discussion with the applicant to address the issues identified. A small percentage of cases were resolved through other branches.

CEB staff indicate that the ease with which non-compliance can be resolved varies depending not only on the cooperation of the applicant/manufacturer, but also the type of non-compliance. Administrative non-compliance can typically be resolved by working with the manufacturer/applicant to develop an action plan to get certification issued in a short timeframe or to ensure that the equipment will be labelled properly^{Footnote 28}. More complicated cases involve physical testing, particularly for those where potential safety (SAR) issues have been identified. In this case, CEB tests multiple samples in order to ensure that it is a true non-compliance and not just one defective unit before contacting the manufacturer to request an action plan. A resolution is normally possible if there is a hardware or firmware fix. An impasse occurs if there is no possible fix and the only solution is to stop sales of the product on the market and possibly a voluntary recall by the manufacturer, both of which are coordinated by the enforcement branch (DGSO).

3.2.4 To what extent has the CEB contributed to ensuring that radio equipment in Canada operate without negatively interfering with each other?

Interviewees indicate that CEB reviews equipment information before it enters the Canadian market, which helps to reduce the likelihood of interference. "Spectrum impact and network harm/access" is also one of the three key risk factors CEB uses for evaluating risk and determining what product types will be targeted for audits^{Footnote 29}. This helps identify equipment that could have interference issues that is already on the market.

While there is no direct quantitative indicator of non-compliance as it relates to interference issues, technical non-compliance identified in physical audits can be used as a proxy-measure as it typically indicates potential interference issues. According to data from CEB's FYE reports, an average of 19% (76 out of 405) of the physical audits CEB conducted identified technical non-compliance in the last five years. As demonstrated in Table 3 below, CEB resolved an average of 51% of these cases. An average of 27% were sent to DGSO for enforcement action, and a small percentage of cases were resolved through other branches (i.e., DRS issued a waiver and DGSO provided the correct testing software).

3.2.5 To what extent has the CEB contributed to ensuring technologies used by Canadian consumers are safe as it relates to safety requirements falling under CEB's purview?

Health Canada, through its Safety Code 6, establishes the safety limits for human exposure to radiofrequency (RF) fields from mobile and other radio emitting devices. This code, last updated in 2015, is the scientific basis for ISED's regulations for radio equipment certification and RF field exposure compliance^{Footnote 31}. According to representatives of manufacturers, CBs and standards bodies, CEB has taken an active role in developing methodology for RF exposure testing and setting international testing standards. The unit of measurement for these tests is the Specific Absorption Rate (SAR), which assesses the RF absorbed by different parts of the body when using a wireless or other device. CEB began working on SAR testing in 2007 and has its own SAR lab dedicated to this type of testing.

"Safety impact" is also one of the three factors CEB uses for evaluating risk and determining what product types will be targeted for audits. Of the average 15 SAR physical audits conducted each year over the last five years, an average of three were found to be non-compliant, the majority of which were resolved by CEB according to data available from the last 3 years. One non-compliance case has been sent to DGSO for enforcement action in the last 3 years.

3.2.6 To what extent has the CEB contributed to timely market accessibility?

According to representatives of manufacturers and certification bodies, CEB works proactively with its clients to ensure compliance, is open to dialogue and works with clients to resolve non-compliance issues in a timely manner. Such an approach fosters timely market access. None of the nine industry representatives interviewed thought that CEB was causing any significant delay in the introduction of equipment into the Canadian market.

Industry representatives also commended CEB for its proactive exploration of more efficient testing methods, a key example being an international study being led by CEB to develop recognized applications of a new fast-SAR technique which could reduce testing time from months to days for technologically complex devices such as smart phones. Further, a letter to ISED from an international association of telecommunications equipment manufacturers in August 2015 noted that "if the full benefits are realized, this will be a quantum shift in SAR compliance testing and hence market access."

CEB's contribution to harmonizing testing standards with the FCC was also praised by standards body representatives as this facilitates the introduction of new products into Canada at the same time as the U.S.

3.2.7 To what extent has the CEB contributed to a fair marketplace for radio and telecommunications product manufacturers and suppliers?

Industry representatives indicate that CEB is contributing to a level playing field for radio and telecommunications product suppliers and manufacturers by consistently applying standards and regulations, and by preventing low cost, substandard equipment from becoming the norm for manufacturers competing in the Canadian marketplace. CB representatives also praised CEB for its impartiality, whereby all companies, regardless of size, get the same service and must meet the same requirements.

The Bureau also targets risk factors that are linked to a fair marketplace, such as low product price compared to equivalent competing products and popularity/mass deployment of the product in its marketplace surveillance activities; this helps to prevent substandard equipment from entering the Canadian marketplace, thereby levelling the playing field for industry.

3.2.8 To what extent does CEB demonstrate efficiency? Can CEB improve efficiency to be able to address current and future workload?

Observations on Efficiency and Economy

Information gleaned from a document review and interviews point to a number of examples demonstrating CEB's efficiency and economy in the last five years, including:

• Cross-training staff to deal with fluctuations in workload and to help develop/retain staff;
• Clearly identifying workflows for each type of non-compliance scenario and articulating the roles and responsibilities of CEB and other areas within the department in its Procedure for Market Surveillance of Radiocommunication and Telecommunications Equipment (April 2015);
• Implementing a number of measures to maximize use of its market surveillance resources (e.g., partial compliance evaluations for physical audits and desk audits, both of which are less resource intensive and allow CEB to conduct more market surveillance).

The Bureau also contributes to larger efficiencies within the department by working to ensure testing standards are harmonized with other administrations, which is seen as cost-efficient compared to developing independent Canadian standards.

The Bureau has continued to meet its objectives despite its O&M budget being cut in half in 2010–11 and has been able to manage excess demand during surge periods in the last 5 years; however this has involved an increasing amount of overtime.

The Bureau has been planning for an anticipated increase in workload. For example, the CEB has developed a long-term plan to modernize its equipment over a 10-year period, and considers new technologies in its market surveillance. The results of the evaluation, notably the Alternatives, Best Practices and Trends Study, suggest that further efficiencies could be achieved to address future workload. These are described in the sections below.

Communication with industry (Information Technology Support)

E-filing

In March 2015, CEB, along with the rest of the SITT sector, launched new software for equipment certification and registration as part of the Spectrum Application Modernization Commercial Software Implementation (SAM-CSI). While SAM-CSI has helped streamline and standardize business processes and replaces legacy systems, representatives of CB's, who are the major users of CEB's E-filing system, identified some issues with the new system. Specifically, CB representatives noted that it asked for too much information, was hard to navigate and substantially increased the amount of time it took to file an equipment certification or registration request. While CEB has worked with industry to provide clarification and guidance as required, there may be an opportunity for the Bureau to further address these issues through other means suggested by interviewees, such as creating a YouTube video or set of frequently asked questions (FAQs) to assist users in navigating the system.

Website

The CEB website is one of the primary ways CEB communicates with its clients^{Footnote 32} and could become increasingly important as CEB interacts with new industries (e.g., automotive, medical and transportation). Industry representatives identified room for improvement with respect to its user-friendliness. For example, the 'certification program' section of the website consists of questions and answers describing the basis of wireless device regulation in Canada but does not explain how to certify equipment in Canada, nor does it provide a link to the 'wireless program' section where this process is explained. CEB may be able to increase its efficiency by improving the user-friendliness of its website.

Information exchange with other administrations

Administrations interviewed as part of the Alternatives, Best Practices and Trends Study points to the importance of exchanging information with their peers to be better informed and to coordinate their activities to get the best results from their individual efforts. This will become particularly important as more devices are expected to enter the market that are increasingly complex. While understanding some of the difficulties in exchanging confidential information with other administrations, many felt they should be able to meet their requirements without jeopardizing national autonomy. There may be an opportunity for CEB to further leverage its partnership with key administrations by establishing information sharing agreements. Given its proximity and activity in market surveillance and enforcement, a logical place to start is likely the FCC.

The Alternative, Best Practices and Trends Study also suggests that further efficiencies could be achieved by coordinating CEB's market surveillance activities with other foreign administrations (e.g., coordinated testing of equipment and review of CB market surveillance results) to ensure compliance with standards. While this may not occur in the near future, information sharing could be the first step as respective administrations further develop relationships over time.

Pre-market certification

CEB conducts two types of pre-market certification activities:

• Full Assessments: Of the applications CEB receives every year, approximately 5% are submitted directly to CEB. These are referred to as "full assessments". In this case, CEB reviews every document including test reports and all data fields to ensure the product complies with all requirements and issues a TAC (Technical Acceptance Certificate).
• Review of CB applications: The remaining applications (about 95%) are received from CBs, in which case CBs have performed the full assessment and have issued a CB certificate. When CEB receives these applications from CBs, they perform a cursory review, which includes a review of all data fields, a selection of key documents and verification that fees were paid and procedures were followed.

As noted in the Alternatives, Best Practices and Trends Study, most administrations have moved away from this pre-market certification as they have become more comfortable with CBs certifying equipment. These efforts also help reduce the administrative burden on CBs and the manufacturing industry, as time does not need to be spent submitting all documentation to the regulator for review.

Industry representatives interviewed as part of this evaluation were satisfied with CEB's efficiency overall. In fact, most ranked CEB as efficient, if not more efficient than other regulators such as the U.S. FCC. However, they did express some dissatisfaction with CEB's pre-market certification process, suggesting that CEB should rely on CBs more and reduce its review of CB documentation. Corresponding data suggests that there is value in CEB conducting these cursory reviews, as the number of issues found increased significantly in 2007 when CBs became responsible for submitting their documents to ISED directly and has continued to increase until 2014^{Footnote 33}. Further, from an efficiency perspective, it allows for identification of issues before equipment enters the Canadian marketplace, which is less taxing on resources compared to the more costly option of addressing non-compliance issues once equipment is already on the market.

CEB staff also indicated that they are responsible for maintaining the integrity of the REL, and that this information, which is impacted by the information they receive from CBs, needs to be accurate in order to conduct quality market surveillance. This is especially important for Canada as the vast majority of equipment is being used in Canada, whereas other administrations are certifying equipment for the purpose of international use.

While it appears to be important that CEB conduct some level of pre-market certification as it relates to CBs, the Bureau could address industry stakeholders concerns by reducing its pre-market certification to focus on select CBs – for example new CBs or ones where there have been issues in the past. Further reinforcing this notion is the fact that, according to data from the last 5 years, there have only been issues with an average of 19% of these applications. Given that most of these applications have not had issues, it may be more efficient for CEB (and less burdensome on CBs) to adopt a risk-based approach and target CBs accordingly.

Assessing newer technologies and testing procedures

According to the Alternatives, Best Practices and Trends study, a consistent theme that arose when discussing the impacts of technological development on CEB is the need for staff to keep up with the latest changes in technology and incorporate them in their management (e.g., training, equipment, staffing and program plans) and testing processes. This includes understanding and anticipating technological changes through ongoing scanning of the environment for technological developments.

There are several ways this can be accomplished, one of which includes building partnerships with other regulators and information sharing as discussed earlier. CEB also reviews reports on consumer developments and its regular interaction with industry and participation in domestic and international standards committees helps the Bureau stay informed. Another important aspect includes close collaboration with other groups in DGEPS (e.g. Directorate Regulatory Standards (DRS), who develop technical standards) and the Communications Research Centre (CRC), the federal government's primary laboratory for research and development in advanced telecommunications. CEB currently obtains information from these groups in primarily an informal, ad hoc fashion. CEB may be better able to scan the environment for technological developments with more proactive information-sharing with these groups.

Also, international organizations such as the International Telecommunications Union (ITU) and 3rd Generation Partnership Projects (3GPP)^{Footnote 34} discuss these technological changes before they are implemented. Similar to the above, CEB appears to be obtaining information on these meetings on an ad hoc basis. There may be advantages to ensuring there is more proactive information-sharing from ISED colleagues that do attend/participate in these meetings, such as setting up or using existing regularly scheduled committees to ensure the Bureau stays informed. This would lead to a better informed and knowledgeable CEB organization, and would help to inform CEB's development of testing procedures.

Observations on the Adequacy and Use of Performance Measurement Data

CEB's Fiscal Year-End reports were identified as a key data source for the evaluation. While these reports provided some performance information, they were primarily designed for internal reporting. As such they tended to be technical in nature. Compliance-related activities and corresponding data were reported on differently than what was needed for the evaluation, or no information or data was available. This made it difficult to assess the achievement of expected outcomes over the past five years.

As SITT works towards developing its own Performance Measurement Strategy, now may be a timely opportunity for CEB to revise its performance measurement and reporting accordingly in order to support the sector in this endeavor. In the future, the Bureau may also wish to make their Fiscal Year-End reports public, thus increasing communication of CEB's objectives and accomplishments with industry stakeholders.

4.0 Conclusions and recommendations

4.1 Relevance

• There is a continued need for CEB. Its importance is reflected in the growing demand for its services and the potential negative impacts if CEB did not play an active role in the market. Demand for CEB's services is expected to continue to grow in the future given the rapid proliferation of wireless devices and technological developments.
• CEB plays a key role in supporting a number of federal government priorities, including improving high-speed broadband access and growing the digital economy, which rely on seamless use of technology. The Bureau's objectives are also consistent with the department's priority to foster a strong investment environment for telecommunication services.
• The Bureau's activities align with the roles and responsibilities of the federal government to ensure that radio and telecommunications equipment used in Canada meet all applicable standards, as stated in the Radiocommunication Act and the Telecommunications Act.

4.2 Performance

• CEB has largely been successful in achieving its short-term outcomes. The Bureau has contributed to improved regulations, standards and spectrum planning within the department, has certified/registered radio and telecommunications equipment in a timely manner, and has helped to ensure compliance with standards via its targeted market surveillance and proactive approach to resolving non-compliance.
• The Bureau has also demonstrated success in achieving its long-term outcomes of helping prevent interference, ensuring technologies are safe as per Health Canada's Safety Code 6 requirements, promoting timely market access and contributing to a fair marketplace.
• CEB has demonstrated efficiency and economy during the evaluation period and has been planning for an anticipated increase in workload. Further efficiencies could be achieved through increased communication within the department and with industry stakeholders and other administrations, as well as by reducing its pre-market certification activities to focus on assessing newer technologies and testing procedures.

4.3 Recommendations

CEB has been successful in achieving its outcomes to date. One of the key challenges CEB faces is how to meet its objectives in the future given what is expected to be a significant increase in complex wireless products entering the marketplace. While it is difficult to pinpoint which products will enter the marketplace and how soon, it is clear that the wireless world is a moving target and CEB needs to be ready to deal with a rapidly changing environment. More specifically, the Bureau needs to ensure it keeps pace with the latest changes in technology in order to remain agile and adapt its management and testing procedures as technologies evolve. With this in mind, CEB should:

1. Consider reducing its pre-market certification activities and focus on assessing newer technologies.
2. Examine options for increased information sharing within the department to better understand the environment for technological developments, which will enhance CEB planning.
3. Further leverage its partnership with key foreign administrations by establishing information sharing agreements.
4. Consider building on the results of this evaluation by revising its performance measurement and reporting accordingly.

Management Response and Action Plan

A - For inclusion in the report

The findings and recommendations of the Evaluation of the Certification and Engineering Bureau (CEB) were presented to SITT management. Management has agreed with the findings included in this report and will take action to address all applicable recommendations by March 2018.

B - For follow-up purposes - Detailed actions to address the recommendations in the report