Evaluation of the Spectrum Management Business Plan Investments (BPI)

This page has been archived on the Web

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Evaluation of the Spectrum Management Business Plan Investments (BPI)
Final Report
Audit and Evaluation Branch
Industry Canada
March 10, 2005

Tabled and approved by DAEC on December 8, 2005

Table of Contents

List of Acronyms and Abbreviations

List of Acronyms and Abbreviations
ALS Assignment and Licensing System
ASD Alternate Service Delivery
BPI Business Plan Investments
CITEL Inter-American Telecommunication Commission
CRA Canada Revenue Agency
DGRB Director General, Radiocommunications and Broadcasting Regulatory Branch,
DGSE Director General, Spectrum Engineering Branch
DGTP Director General, Telecommunications Policy Branch
DGs Director Generals
DTH Direct-to-Home
FCC Federal Communications Commission
GDP Gross Domestic Product
IC Industry Canada
ITU International Telecommunication Union
MRAs Mutual Recognition Agreements
NGOs Non-Governmental Organizations
O&M Operating and Maintenance
OPI Office of Primary Interest
PCS Personal Communications Services
PMP Performance Management Program
PSAT Public Security and Anti-Terrorism
RCMP Royal Canadian Mounted Police
SEED Spectrum E-Services E-Commerce Delivery
SITT Spectrum, Information Technology, and Telecommunications,
TB Treasury Board
WiFi Wireless Fidelity
WRC World Radio Communications

Executive Summary

We find that BPI is an effective means of responding to the evolving, fast-moving environment of the Spectrum / Telecom Program. Industry Canada, in conjunction with central agencies, should pursue an alternative funding mechanism or process to transition activities from BPI that have become operational or ongoing in nature in order to continue to service the evolving needs of the Canadian telecommunications industry.

Background

In 2001, Industry Canada (IC) submitted a request for new and enhanced investments to respond to the dynamic telecommunications environment and increasing demands related to spectrum management. The initial year of funding (2001/02) was $2M, with $9M provided in each subsequent year for a total of $38M. This was a continuation of a series of supplementary funding to the Spectrum/Telecom Program with more than $40M of incremental funding received from 1995/96 through 2000/01.

This evaluation focuses on the areas of relevance, success, and cost effectiveness of the program and provides input into programming decisions regarding the ongoing funding of these types of activities. The evaluation assesses the effectiveness of the program in achieving its objectives, impacts, both intended and unintended, the continued relevance and alternative ways of achieving intended results.

Key Findings

Relevance

  • The need for the incremental investments was clearly articulated at the time, and links were made to the overall mission of the Spectrum / Telecom Program.
  • Investments were appropriate for the identified needs. New investments were approved through the management committee as critical needs arose. Two of these new investments are not linked to the needs – the Telezone legal case and succession planning activities.
  • The Spectrum / Telecom Program mission, spectrum management objective and BPI results statement together provide a clear and relevant vision. The objectives of the program were clearly identified.
  • The activities of the program are aligned with the Spectrum / Telecom Program, Industry Canada and government-wide priorities primarily through the two themes of ensuring a fair and effective regulatory regime and promoting a competitive telecommunications industry.

Success

  • The identified activities and outputs broadly rolled out as expected with the exception of Alternative Service Delivery initiatives, which were particularly slow to implement due to challenges in the public and private sector environments.
  • Management reallocated funds to address significant pressures on the Spectrum / Telecom Program in the areas of compliance activities, network integrity and security and succession planning.
  • Timeliness, efficiency and equity of access have improved, with industry consistently acknowledging the improvements. Industry would like to see continued improvement focused on timeliness of post-auction verification and speed of replacement of bands reassigned.
  • The Spectrum / Telecom Program has a very good understanding of the marketplace, with improvements recognized over the past five to ten years. Industry attributes this more to the level of direct contact, consultation and ease of access as opposed to formal studies or research. The result has been a positive working relationship and positive effects on businesses. An important role for Industry Canada in the future is to continue to represent the smaller businesses that may not have as strong a voice.
  • Progress on many issues in international fora is consistent with Canadian positions and Canada's leadership and influence is considerably greater than what its size would suggest. IC's role on the international stage is highly valued by the private sector and is especially important to manufacturers. Business feels that they have been able to capitalize on the opportunities afforded by the negotiations and agreements. Consultation with industry and incorporation of their input facilitates better results from an industry perspective. There is a desire to see this level of effort continued and, by some, expanded.
  • MRAs are viewed as very successful with progress on harmonization of standards and positive impacts on the private sector in terms of reduced cost and faster time to market. Work must continue on identified next steps including regulations, in order to truly take advantage of the MRAs.
  • Auctions are generally working well and better information is available during the auction process to allow firms to adjust bidding. This creates a more efficient market where companies pay as little as needed to get product. Improvements could be made in the procedure manuals and timeliness of the post-auction verification process. The Spectrum Direct initiative has been widely viewed as successful. Client Satisfaction surveys conducted in the Regions illustrate a high level of satisfaction with services.
  • Productivity and efficiency gains appear to have been achieved however, these conclusions would benefit from more targetted data collection. ASD projects have been slow to implement and the expected gains have not been fully realized.
  • Stakeholders report taking advantage of related business opportunities domestically and internationally. The positive impacts have been muted during this period by the cyclical downturn in the telecommunications industry.
  • Stakeholders generally believe interference has not increased in licensed bands but remained stable. Changes in the operation of the railway industry and mobile radio may be creating an issue that will need to be addressed. There is some indication of growing issues with interference in license exempt bands, including unintended radiators, mobile radio and the railway industry.

Cost Effectiveness

  • The primary forum for management and decision-making was the joint DG Level Committee including the three Branches and the Regions. This DG committee focus was on financial management, with output and outcomes managed individually by the respective DGs. Individual Branches and Regions have independence of action within bounds of BPI objectives. At the project level, there were clear lines of authority for planning, managing and implementing projects.
  • The management was characterized by a balance of stability and an appropriate degree of flexibility to allow the Program to respond to the dynamic environment.
  • Improvements have been noted in terms of management process and documentation.
  • Possible enhancements to management and decision making include more rigorous processes for priority setting and more timely identification of free balances by project as year-end approaches to facilitate reallocation.
  • Mid-term and final performance assessments were conducted for the BPI funds and the majority of the lessons learned from the mid-term assessment were discussed and implemented or rejected.

Funding

  • Broadly, the funding levels were sufficient in that Spectrum / Telecom Program was able to meet its commitments. Instances of funds being rolled over are noted, however, these are within the 5% allowable.
  • The funding model fits well with the project-type activities and does not fit well with the operational activities, limiting good management and creating significant uncertainty at time of renewal.
  • BPI funding in the regions and in headquarters was regularly used as a support for normal operations, which suggests that A Base funding may be insufficient.

Design and Delivery

  • Day-to-day accountability has been implemented appropriately. Improvements could be made in progress reporting, particularly by identifying a small number of attributable outcomes, selecting the right indicators, and regularly collecting and reporting on these.
  • Lines of accountability were split because regional managers for BPI needed to report to regional Executive Directors, who reported to Ottawa, and directly to BPI management personnel in Ottawa.

Recommendations

Recommendation: Develop alternative funding mechanism

We find that BPI is an effective means of responding to the evolving, fast-moving environment of the Spectrum / Telecom Program. Industry Canada should pursue an alternative funding mechanism or process to transition activities from BPI that have become operational or ongoing in nature in order to continue to service the evolving needs of the Canadian telecommunications industry. This should be done in conjunction with central agencies.

Recommendation: Maintain stakeholder consultation with, and ease of access, to IC

Ongoing structured consultation with key stakeholders, including the use of mixed delegations, is seen to be key in the areas of international issues. Ease of access to government representatives is highly valued. These should be maintained, and further institutionalized in the regions, consistent with Federal Government Smart Regulation principles of valuing the perspectives of stakeholders and enhancing access to the federal government. Stakeholders recognised the importance of including large and small businesses in the process. IC should monitor equity of access for small and large businesses.

Recommendation: Develop and implement an appropriate evaluation framework

While no formal evaluation framework was required at the outset of this program, good practice requires this. Should additional project funds be sought, an evaluation framework should be developed and implemented. Specific, measurable and attributable results indicators need to be clearly identified at the outset, and in situations where incremental funding is applied to existing activities, results should be identified to demonstrate the incremental effects. The level of effort for data collection and reporting should be relative to the importance and value of the fund.

Recommendation: Maintain and build on current governance structure

The joint headquarter and regional committee DG has generally been an effective management mechanism. The levels of flexibility and controls were appropriate to the needs of the program. Management should consider a more rigorous process for priority setting such as a business case approach or establishment of criteria to evaluate projects. The practical effects of dual report requirements on the Regions should be monitored.

1.0 Program Description

1.1 Background

The Spectrum/Telecom Program of Industry Canada derives its statutory mandate mainly from the Telecommunications, Radiocommunications and Broadcasting Acts. These three Acts provide the basis for managing Canada's radio spectrum and establishing the policy and regulatory framework for communication services. The Spectrum/Telecom Program is jointly delivered through three Headquarters branches (Radiocommunications and Broadcasting Regulatory Branch (DGRB), the Telecommunications Policy Branch (DGTP), and the Spectrum Engineering Branch (DGSE)) and five regions. Many of its functions are delivered at the local level to clients across Canada.

In 2001, Industry Canada identified a need for new and enhanced investments to respond to the dynamic environment and increasing demands related to spectrum management. The initial year of funding (2001/02) was for $2M, with $9M allocated in each subsequent year for a total of $38M. The program is ending in 2004/05.

This was a continuation of a series of supplementary funding to the Spectrum/Telecom Program with more than $40M of incremental funding received from 1995/96 through 2000/01.

The investment funds activities within the three Headquarters branches of the Spectrum Management and Telecommunications Sector, as well as the regional offices of the Operations Sector. Within Industry Canada, the program is commonly referred to as "Business Plan Investments (BPI)".

1.2 Program Activities

Industry Canada identified seven areas for incremental investment. The list of activities as outlined below is a mix of very discreet activities such as the Mutual Recognition Agreements (MRAs) and very broad categories of supplemental support to existing services, such as improving service delivery.

Facilitate Spectrum Access for New Services

Intensified activity was required to facilitate timely spectrum access for new services such as wireless local loop, wireless internet access, satellite communications services and enhanced mobile telephone service. This was to include updating the Spectrum Release Plan, involving extensive consultations with stakeholders and detailed analysis of issues that emerge. In addition, research and specific studies were required in such areas as user needs, technical feasibility of certain specific proposals and technology applications, and the effectiveness of various consultation fora.

Understanding of the Competitive Marketplace

BPI activities were intended to enhance the understanding of the competitive marketplace by acquiring more knowledge about the business, economic and technical environment facing Canada's telecommunications and wireless sectors, their capabilities, their potential to meet basic service objectives, and relevant policy issues. This activity was to involve intelligence gathering and analysis, and ongoing consultations with sector representatives. This knowledge and expertise was to be used to develop the required rules and procedures for the implementation of new radiocommunication and telecommunication services taking into consideration contentious legal, health and privacy issues.

Representation in International Fora

Increased effort was required to ensure that Canadian spectrum interests are effectively represented in ITU regulatory processes and other international fora, so that Canadians have access to the right spectrum at the right time for new businesses to compete in world markets, and Canadian use of spectrum is protected from foreign terrestrial and space systems.

Mutual Recognition Agreements

BPI activities included the negotiation and development of Mutual Recognition Agreements (MRAs) for telecommunications with countries in the Americas, the European Free Trade Association, the European Union, and APEC; and to continue with the recognition of Conformity Assessment Bodies for testing and certification of telecom products. The desired end goal was to foster Canadian competitiveness and generate business opportunities for Canadian companies in the international environment.

Auction Management System

The investment in this area was to continue to invest additional resources to refine and develop the Auction Management System. Activities included development of a new market driven licence fee model, increasing the use of alternate licensing mechanisms such as spectrum licences, working to create a secondary market for spectrum licenses, and establishing and implementing criteria for spectrum quality determination.

Improve Service Delivery

Service delivery improvements were to include further development of a web-based licensing system and enhancement of current bandwidth capacity to better offer client services. Measures were to include software development, systems hardware, process re-engineering, project management and implementation support, database management, integrating existing operational software, and installing new automated tools to support licensing and spectrum quality activities.

Alternative Service Delivery

BPI activities included a strategy for alternative service delivery mechanisms, facilitating the use of non-government organizations (NGOs) as service providers to improve service delivery. Initiatives were to include the delegation of co-ordinating spectrum use in certain frequency bands, and the examination and certification of radio operators.

The estimated annual requirement for each of the seven activity areas is summarised in the table below, in $ millions:

Estimated annual requirement for each of the seven activity areas, in $ millions
Activity Salary and Benefits
(Inc provision for accommodation)
Non-salary Total
Facilitating spectrum access for new services $0.4M $1.3M $1.7M
Understanding the competitive marketplace 0.5 0.4 0.9
Representation in international for a 0.4 0.4 0.8
Mutual Recognition Agreements 0.2 0.3 0.5
Spectrum auctions and licence tools 0.2 0.5 0.7
Improved service delivery 0.6 1.7 2.3
Alternative service delivery -- 2.1 2.1
Total $2.3M $6.7M $9.0M

In addition, during the 2001 – 2005 period, a number of significant changes in the environment brought about new and emerging pressures on the program. Management responded by reallocating funds from lower priority to higher priority items. Specifically, new activities were funded in the areas of:

  • Network integrity and security;
  • Direct-to-home satellite and communications antennae issues;
  • Litigation; and
  • Succession planning pressures.

These new activities and their associated outputs and outcomes are discussed further throughout the evaluation report.

1.3 Program Objectives

The objectives of the program, described as the expected results of these incremental investments were:

  • Canada will obtain international authorization to use the spectrum it requires to serve Canadian communications needs and create business opportunities; this spectrum will be made available for use in a timely, efficient and equitable manner.
  • Canadian firms in the wireless and telecommunications industry will take advantage of related business opportunities, leading to stronger economic growth and improved communications services.
  • Mutual Recognition Agreements will be put in place with countries in the Americas (United States, Brazil, Argentina, Venezuela, Chile, and Mexico), the European Free Trade Association, and APEC (Hong-Kong, Singapore, China), leading to harmonization of international standards and creating business opportunities for Canadian suppliers.
  • Spectrum licensing and certain other services will be provided to the public on-line, providing better client service and faster decision making.
  • Productivity and efficiency gains will be realized through development of new tools for spectrum access, systems automation, electronic interface with clients, and use of Non-Government Organizations for alternative service delivery.

1.4 Structure of the Report

This report is divided into a further seven sections:

  • Section 2 describes how the evaluation framework was developed and outlines the objective, issues and methodologies;
  • Section 3 provides the analysis and conclusions regarding the issue of program relevance;
  • Section 4 analyzes the success of the program in terms of outputs and outcomes;
  • Section 5 discusses the cost-effectiveness of the program and assessment and use of lessons learned;
  • Section 6 examines at a high level, the issues of adequacy of funding and the funding model;
  • Section 7 reviews the design and delivery of the program; and
  • Section 8 provides some overall lessons learned collected during the evaluation process.

The Annexes contain the Steering Committee Members, the Evaluation Framework, a Summary of Issues Prioritization Exercise, a Document List, a List of Interviewees and the Interview Guides.

In some sections of the evaluation report, the views of key stakeholders are provided in text boxes where of particular interest and emphasis.

2.0 Evaluation Framework

A formal evaluation framework was not required in 2001 when the program received funding. However, the TB Submission did contain expected results and indicated how those results would be measured. Based on the TB Submission, TB Evaluation Policy, a document review, and selected strategic interviews, the consultants developed the following components of the evaluation framework:

  • BPI Logic Model;
  • Outcomes Measurement Strategy, including Indicators and Data Sources;
  • List of Evaluation Issues; and
  • Evaluation Methodology, including Indicators and Data Sources.

Subsequently, a workshop with key members of the Evaluation Steering Committee (for a list of members, see Annex A) was held to discuss, modify and validate the evaluation framework. The final evaluation framework is detailed in Annex B. At that time, a prioritization exercise was conducted to provide input on the relative level of effort that the evaluators should place on the evaluation issues. The results of this prioritization exercise are contained in Annex C. This feedback was incorporated into the evaluation process.

2.1 Evaluation Objective

This evaluation focused on the areas of relevance, success, and cost effectiveness of the program and provides input into programming decisions regarding the ongoing funding of these types of activities. The evaluation assessed the effectiveness of the program in achieving its objectives, impacts, intended and unintended, continued relevance and alternative ways of achieving expected results.

2.2 Evaluation Issues

The list of evaluation issues outlined below was developed based on the TB Submission, TB Evaluation Policy and existing documentation. The issues were then confirmed and refined through consultations and Steering Committee direction.

Relevance

The continued relevance of the BPI, including whether or not it is consistent with IC and government priorities:

  • What need does the BPI satisfy?
  • Are the investments identified appropriate for satisfying this need?
  • Is there a clear and relevant vision and objectives for Spectrum/Telecom Program activities under BPI?
  • Does this program continue to be consistent with Spectrum/Telecom Program, Industry Canada and government-wide priorities?

Success

An assessment of the success of the BPI; to what extent has BPI achieved its objectives:

  • Were the planned activities implemented and did they produce the expected outputs?
  • Have the expected outcomes been achieved through the activities identified?
  • Have there been any unexpected outcomes?

Cost-Effectiveness

  • Are there appropriate management and decision-making structures in place to meet the objectives?
  • Has there been an assessment and use of lessons learned? Where, when and how were lessons used?
  • For identifiable groups of activities, has there been consideration of options/alternatives to increase cost-effectiveness or efficiency in delivery? Were any options found? Which were tried and what was the impact?

Funding

  • Was the funding sufficient?
  • Is the use of program, sunset funding the most appropriate way to fund these activities?

Design and Delivery

The extent to which the design and delivery of the investment initiatives facilitates established objectives.

  • Were the programs and activities designed and delivered with the appropriate management accountability?
  • Could these needs be satisfied through a mechanism other than BPI?

Lessons Learned and Recommendations

  • What are the relevant lessons learned from the BPI and what recommendations can be made for future policy and programming needs?

2.3 Evaluation Methodologies

The evaluation was conducted using a multi-method approach, including:

  • Document and File Review;
  • Review of Web Publications;
  • Key Management Interviews;
  • Central Agency Interviews; and
  • Key Stakeholder Interviews.

How each of these methods was used is discussed in more detail below.

Document and File Review

The document and file review provided a solid understanding of the purpose of the Business Plan Investments in the Spectrum / Telecom Program, including the activities that were undertaken. The information contained in these documents assisted in measuring planned versus actual outputs and outcomes. The BPI documents also were used to provide information on management practices, and lessons learned. Two key documents were an interim and final performance assessment of the BPI program, conducted in 2002 and 2004 respectively. A full list of the documents that were supplied by the Spectrum / Telecom Program and reviewed is listed in Annex D.

Review of Web Publications

A variety of the outputs related to the BPI fund have been published on the IC website. A review of the published documents was conducted. The information contained in these documents assisted in measuring planned versus actual outputs and outcomes.

Central Agency Interviews

A total of 14 key management interviews (see Annex E) were conducted with IC representatives, from Headquarters and the Regions. These interviews focused on the overall program, including the achievement of objectives and results. The interview guides for these interviews is provided in Annex F.

Key Stakeholder Interviews

A total of ten key stakeholder interviews were conducted with key stakeholders external to Industry Canada. This included representatives from federal central agencies, industry associations, private sector clients and partners, manufacturers and testing laboratories. These interviews focused on the achievement of outputs and outcomes. The list of interviewees is included in Annex E and the interview guides are provided in Annex F.

3.0 Relevance

This section discusses the relevance of the BPI and assesses the extent to which there are clearly articulated needs and goals and objectives. It assesses if these needs, goals and objectives are consistent with Program, IC and government-wide priorities. Findings are derived primarily from the document reviews and the key IC management interviews. The perspective of key stakeholders has been included where it is of interest.

3.1 What need does the BPI satisfy?

To examine the need for BPI, it is necessary to review the objective of spectrum management in Canada, namely, to make available the optimal amount of spectrum in an efficient, equitable, and timely manner to facilitate communications among Canadians and generate business opportunities in the telecommunications and wireless sectors. Spectrum management is carried out under the Spectrum / Telecommunications Program of Industry Canada. The mission of the Spectrum / Telecommunications Program as stated in the 2001 timeframe was:

"… facilitating the development and use of world-class communications infrastructure, technologies and services for the express purpose of enhancing Canada's competitiveness, economic growth and the quality of life of all Canadians1".

At the time, a number of key challenges were affecting the Spectrum Management / Telecomm Program's activities in spectrum management, including:

Stakeholders Perspective:

Key stakeholders indicated that their ultimate need is for access to spectrum and international markets. Strong support was voiced for IC activities that facilitate this.

  • Accelerating technological change;
  • Regulatory developments;
  • Convergence of technologies; and
  • Increasing demand for new telecommunications services.

These factors created a critical 'need' for additional up-front investment to manage increased demands on Spectrum / Telecom Program, improve service delivery and respond to technological and regulatory changes.

The need for the incremental investments was clearly articulated at the time, and links were made to the overall mission of the Spectrum / Telecom Program.

3.2 Are the investments identified appropriate for satisfying the stated needs?

A review of the seven investment areas related to the needs identified in Section 3.1 above demonstrates a clear link. This is summarised in the table below. Given the very broad nature of many of the activities, some address more than one need.

A review of the seven investment areas related to the needs identified in Section 3.1
Need

Activities
Accelerating technological change Regulatory developments Convergence of technologies Facilitate the increase in demand for new telecommunications services
Facilitating spectrum access for new services To shift users to new bands so that new technologies that require certain spectrum bands could be introduced     To ensure access to spectrum given increased demands
Understanding the competitive marketplace To understand the potential impact of technological changes on spectrum need Understanding the competitive marketplace to inform regulatory development To understand the potential impact of convergence on spectrum need  
Representation in international fora To influence the direction of change and benefit Canadian companies by helping them introduce products more quickly. To represent Canada on international regulatory developments   To ensure access to spectrum for new services
Mutual Recognition Agreements To allow companies to get their products and services to market more quickly. To allow Canadian companies to source products from more places and as a result introduce new technology more quickly. To implement new regulatory tools   To increase efficiency through MRAs
Spectrum auctions and licence tools Reductions in time and cost to bidders indirectly accelerate technology introduction.     To increase the efficiency and transparency of service delivery
Improved service delivery Reductions in time and cost indirectly accelerate technology introduction     To increase efficiency of service delivery
Alternative service delivery       To move service delivery to private/non-profit sector

In addition, over the period under review, a number of new needs were identified as critical to the Spectrum / Telecom Program. The Spectrum/Telecom Directors General Committee, which provides ongoing BPI oversight, reviewed the identified needs and approved the expenditures as an appropriate use of BPI funding. These have been grouped into the following three categories:

Network Integrity and Security

  • The heightened security concerns since September 2001 drove the need for additional spectrum dedicated to defence and for increased security and reliability of Canada-U.S. border communications.
  • The Canada/US power outage in the summer of 2003 illustrated the need to improve emergency preparedness with respect to communications amongst first responders such as police, ambulance, fire, etc.
  • Concerns over the integrity of the telecommunications system have increased as integration with the Internet creates potential vulnerabilities to computer viruses.

Increased Compliance Activities

  • Additional support to law enforcement agencies was required following the 2002 Supreme Court decision that unauthorized decoding of DTH television signals from communications satellites was illegal under the Radiocommunications Act.
  • The location of communications antennae became a significant issue in the Regions as communications technologies and services increased.
  • Telezone Inc. brought legal action against the Government of Canada, following the 1995 allocation of spectrum for PCS communications through a comparative analysis process. Telezone is now claiming $250M in damages. Ongoing legal activities have been funded.

Stakeholders Perspective:

  • Spectrum access, MRAs, auctions and improved service delivery were identified as highly important, relevant activities.
  • Support expressed for succession planning and an ongoing need for continuity of staff.
  • Demands not abating, ongoing investment needed to build on achievements to date

Succession Planning Pressures

  • A need for additional investment in succession planning became apparent as the retirement numbers for eligible individuals grew. In this highly specialized environment, significant planning and lead time are required to recruit and train replacement staff.

The original set of investments was appropriate for the identified needs. In addition, new investments were approved through the management committee as critical needs arose. Two areas of new investment are less directly linked to the needs — the Telezone legal case and succession planning. The Telezone legal case was in response to a specific action taken against the Government. The succession planning investments are in response to the aging workforce in the public sector.

Stakeholders support investments of the past and believe ongoing investments are needed to capitalize on achievements to date and respond to new/ increasing demands.

3.3 Is there a clear and relevant vision and objectives for BPI activities under the initiative?

As outlined in Section 1.3 above, five specific objectives were identified in the TB Submission for the BPI Activities. Due to fact that this program was designed, in part, as a supplement to ongoing activities, the objectives of BPI reflect this. Two of the objectives reflect more broadly the overall objective of spectrum management and the final three objectives are more specific to discreet BPI activities.

The BPI activities are directly in support of the broader mission of the Spectrum / Telecom Program of IC and in particular support improving competitiveness and economic growth. In addition, the BPI activities are linked to the overall objective of spectrum management in Canada:

" … to make available the optimal amount of spectrum in an efficient, equitable, and timely manner to facilitate communications among Canadians and generate business opportunities in the telecommunications and wireless sectors."

The Spectrum / Telecom Program mission, spectrum management objective and BPI results statement together provide a clear and relevant vision. The objectives of the program were clearly identified.

3.4 Does the program continue to be consistent with Spectrum/Telecom Program, Industry Canada and government-wide priorities?

In 2004, the Spectrum/Telecom Program published a five year vision including strategic directions. The medium term priorities identified in this strategic document are:

  1. Accommodating emerging technologies and services;
  2. Advancing the modernization of our spectrum/telecom management regime;
  3. Protecting the integrity of Canada's spectrum/telecom interests;
  4. Ensuring the continuing relevancy of the Program; and
  5. Securing and managing Program funding.

The activities of the BPI program support the first three of these priorities.

Industry Canada's mandate is to help make Canadians more productive and competitive in the knowledge-based economy, thus improving the standard of living and quality of life in Canada. Through its three strategic objectives: a fair, efficient and competitive marketplace; an innovative economy; and competitive industry and sustainable communities; IC aims to help Canadians contribute to the knowledge economy and improve productivity and innovation performance. A review of the 2004/05 Estimates—Report on Plans and Priorities finds direct references to the activities funded in whole or in part by BPI, as well as priorities where the BPI activities are aligned but not directly described.

A Fair, Efficient and Competitive Marketplace

  • Modernizing Marketplace Framework Instruments:
    • "To foster an innovative Spectrum Management Program that remains responsive to the marketplace, Industry Canada will bring reforms to its Spectrum Policy Framework. This framework contains the essential policy elements of Canadian spectrum management. The Department will release a discussion paper in fall 2004 that will focus on issues such as increased flexibility in the use of spectrum resources, facilitating the introduction of new technologies, smart regulation, access to underutilized spectrum, greater reliance on market-based licensing and availability of license-exempt spectrum.
    • To promote the development and interoperability of e-commerce applications internationally, Industry Canada will continue to collaborate with the Canadian telecommunications industry to ensure that International Telecommunication Union (ITU) worldwide standards for e-commerce applications are in place. Membership in the ITU allows Canada to enhance its partnerships with other governments and the private sector, thus providing for more effective coordination on standard setting and spectrum allocations for global telecommunications networks and services, broadcasting systems, and the continuous development of communications technologies."2
  • Improving Marketplace Programs and Services
    • "The Spectrum, Information Technologies and Telecommunications Sector will explore opportunities to modernize the spectrum management software to increase use, service delivery, spectrum access and post-market surveillance.3"

Competitive industry and sustainable communities

  • Access to Reliable and Modern Digital Infrastructure; and
  • Improving On-Line Service.

In addition, the BPI activities of improving service delivery and succession planning (a new priority area) are aligned with three of the Government-wide themes and management initiatives:

  • Citizen-Focused Service Improvement;
  • Citizen-Centred Service Delivery; and
  • People: Exemplary Workplace—succession planning.

From a Government of Canada perspective, the BPI activities, through the Spectrum / Telecom Program and Industry Canada, are well-aligned with two of Canada's Priorities as described in Canada's Performance 2004: an innovative and knowledge-based economy and a secure and fair marketplace.

The activities of the program are aligned with the Spectrum / Telecom Program, Industry Canada and government-wide priorities primarily through the two themes of ensuring a fair and effective regulatory regime and promoting a competitive telecommunications industry.

3.5 Conclusions

Conclusion #3-1: The need for the incremental investments was clearly articulated at the time, and links were made to the overall mission of the Spectrum / Telecom Program.

Conclusion #3-2: Investments were appropriate for the identified needs. New investments were approved through the management committee as critical needs arose. Two of these new investments are not directly linked to the needs, specifically the Telezone legal case and succession planning.

Conclusion #3-3: External stakeholders believe ongoing investments are needed to capitalize on achievements to date and respond to new/ increasing demands.

Conclusion #3-4: The Spectrum / Telecom Program mission, spectrum management objective and BPI results statements together provide a clear and relevant vision. The objectives of the BPI program were clearly identified in the TB Submission.

Conclusion #3-5: The activities of the program are aligned with the Spectrum / Telecom Program, Industry Canada and government-wide priorities primarily through the two themes of ensuring a fair and effective regulatory regime and promoting a competitive telecommunications industry.


1 Investment in Spectrum Management Treasury Board Submission, 2001 (return to reference 1)

2 Industry Canada 2004/05 Estimates: Report on Plans and Priorities (return to reference 2)

3 Ibid. (return to reference 3)

4.0 Success

This section provides a review of the success of the BPI and the extent to which BPI has achieved its objectives. Findings are derived primarily from the document reviews, web information and key IC management interviews. In addition, key industry representatives were interviewed to provide data on how well the program has met its intended outcomes.

4.1 Were the planned activities implemented and did they produce the expected outputs?

In this section we examine expenditures in the activity categories as well as the outputs produced.

The following table summarizes the actual expenditures under the BPI program by each of the seven activities, as well as the unanticipated new activities. The information contained in the following table was sourced from the December 2004 BPI Performance Assessment. It is noted that the actual percentage of expenditures allocated to each category of activity is an estimate. Some projects can be aligned with more that one activity and in these cases, expenditures were split amongst activity categories based on management judgement. However, this is considered to be a reasonable representation of expenditures.

The following table summarizes the actual expenditures under the BPI program by each of the seven activities, as well as the unanticipated new activities.
Activity Planned %1 Actual %1,2 Variance
(Under)/
Over

Notes:
1 Totals may not add to 100% due to rounding (return to table reference 1)
2 2004/05 Budget estimates were used (return to table reference 2)

Facilitating spectrum access for new services 18.9% 9.5% (9.4)
Understanding the competitive marketplace 10.0% 2.5% (7.5)
Representation in international fora 8.9% 14.5% 5.6
Mutual Recognition Agreements 5.6% 6.0% 0.4
Spectrum auctions and licence tools 7.8% 11.7% 3.9
Improved service delivery 25.6% 36.4% 10.8
Alternative service delivery 23.3% 1.0% (22.3)
Unanticipated program priorities -- 18.3% 18.3

The expenditure allocation demonstrates that activities took place in all categories with some reallocation between activities. However, Alternative Service Delivery (ASD) activities are significantly under-spent. Several factors appear to have contributed to the lower than expected progress on ASD:

  • Changes in the federal environment with respect to ASD resulted in some uncertainty in the level of support and implementation for these types of initiatives. The Auditor General, in the 1999 Report, Chapter 23, was concerned about accountability arrangements in new governance arrangements. The key recommendation from that report was that Treasury Board Secretariat (TBS) should clearly identify and communicate the essential elements of an effective governing framework for new governance arrangements and provide departments with consistent guidance. A new Policy on Alternative Service Delivery came into effect on April 1, 2002, providing a framework to govern and account for the development of federal ASD initiatives. TBS also established an ASD team to provide support to departments;
  • Challenges finding appropriate/willing partners. The burst of the telecommunications bubble in the private sector may have created uncertainty and financial pressures on potential partners; and
  • Complexity of ASD: Experience with the implementation of partnerships featuring the Railway Association of Canada and the Canadian Power and Sail Squadrons demonstrated that administration can be complex.

Through the management process of the BPI funds (further described in Section 5 and Section 7), a series of new and emerging priorities were brought forward, discussed and prioritized, resulting in reallocation of funds. Activities were funded in network integrity and security, compliance activities and succession planning activities. The table below summarizes and aligns examples of the key outputs achieved with those committed in the TB Submission.

The table below summarizes and aligns examples of the key outputs achieved with those committed in the TB Submission
Activity Outputs Key Implementation Examples
1.0 Facilitating spectrum access for new services
  • Annual Spectrum Release Plan
  • Consultations with stakeholders
  • Research and studies—user needs, technical feasibility
  • Technical standards for wireless equipment
  • Spectrum Release Plan 1999, 2001
  • Studies in Prairies and Northern, Pacific, Ottawa
  • Technical/engineering standards and regulations for: telecom equipment used or manufactured in Canada; broadcasting; fixed satellite communications in Canada
  • Communications on release and modification of spectrum (Strategis, Gazette Notices, Press Releases, departmental news and information bulletins)
  • Spectrum in Pacific region assigned to meet new demands
  • Business plans for 8 Ontario communities recommended for funding from the Federal Broadband for Rural and Northern Development Program
  • Spectrum redeployment in congested areas of Northern Ontario
  • Broadcasting certificates for new and modified broadcasting proposals
2.0 Understanding the competitive marketplace
  • Intelligence gathering and analysis (reports and studies)
  • Consultations with sector representatives
  • Business and market reports and/or studies; client consultation
  • Client and market databases and profiles; spectrum utilization studies and monitoring reports; surveys
  • Reports, studies, data bases, presentations, meetings with clients, information management
  • Meetings with clients, reports
3.0 Representation in international fora
  • Position papers
  • Representation and negotiation of Canadian position at meetings
  • Spectrum allotments/approvals
  • Canadian/CITEL position paper and proposals for WRC /ITU meetings on international spectrum issues
  • Analytical and Technical Reports related to use of telecom equipment and broadcasting regulations for WRC /ITU working groups, sub-committees, etc. in preparation for major international meetings
  • Canada-U.S. border agreements for broadcasting and wireless
  • Canadian/CITEL positions and proposals for negotiation of regulatory/procedural matters governing satellite and related terrestrial communications; reduced time for responding to co-ordination requests
  • Regional input as required
4.0 Mutual Recognition Agreements
  • Mutual recognition agreements the Americas (US, Brazil, Argentina, Venezuela, Chile, Mexico), European Free Trade Association, APEC
  • Recognition of Conformity Assessment Bodies for testing/certification
  • Agreements covering 79 countries in place
  • Certification\registration of telecom equipment as meeting Canadian standards and regulations
5.0 Spectrum auctions and licence tools
  • Auction Management System
  • Post Auction verification of licensing requirements
  • Market driven licence fee model
  • Alternate licensing mechanisms
  • Secondary market for spectrum licenses
  • Criteria for spectrum quality determination
  • Spectrum Auction Framework for Canada
  • 52 licences issued in 00–01 auction, $1.5 billion in federal revenue raised; in 03–04, 392 licences raised $11.2 million; January 2005 Auction ongoing; auctions fully automated via Internet
  • Secondary market research / position
  • Enhanced systems and software to manage the licence fee process.
  • Regional perspective on new licence fee regime for radio authorization
6.0 Improved service delivery
  • Web-based licensing system
  • Enhanced bandwidth capacity
  • S/W, H/W, improved processes, project management
  • Automated tools to support licensing and spectrum quality activities
  • National Antenna Tower Policy
  • Increased share of clients using Spectrum Direct on-line services; Better understanding of processes
  • SITT electronic publications with "look and feel" of Strategis
  • Engineering and technical operations systems and related software operational
  • Input to National Antenna Tower Policy Review, meetings, analysis and negotiations on antenna issues
  • Replacement/upgrading of vehicles, instrumentation, equipment, systems hardware and software
  • Client satisfaction surveys completed
  • Pacific Service Improvement Plan
7.0 Alternative service delivery
  • Services outsourced to NGOs
  • Partnerships with non-government organizations in implementing spectrum management programs
  • Management/operation of amateur radio operators centre in Canada under an arrangement with the Amateur Radio Operators Association
New and Emerging Activities
Activity Outputs Implementation
8.0 Compliance Activities
  • Legal counsel and preparation for legal cases
  • Telezone defence activities
  • DTH illegal decoding: information pamphlets, warnings
  • Legal Counsel and related services for SITT initiatives and activities
  • Management of Telezone case
  • Increased awareness of extent of illegal DTH signal decoding
  • Less aggressive marketing by decoding equipment providers
9.0 Network Integrity and Security
  • Harmonized spectrum use for safety services
  • Priority access to telephone system for priority users
  • Vulnerability assessments
  • Priority access to telephone system for designated users during emergencies
  • Information sessions, demonstrations of impact and likelihood of cyber attacks
10.0 Succession Planning and Support
  • Succession Plans
  • Employees in training
  • Knowledge management
  • Employee development programs and succession planning
  • Continuity of service and transfer of knowledge

Both of these tables reflect that the identified activities and outputs broadly rolled out as expected with the exception of Alternative Service Delivery initiatives, which were particularly slow to implement due to challenges in the public and private sector environments. Management reallocated funds to address significant pressures on the Spectrum / Telecom Program in the areas of compliance activities, network integrity and security, and succession planning.

4.2 Have the expected outcomes been achieved through the activities identified?

One of the key recommendations following the mid-term performance assessment in 2002 was to improve the collection, analysis and reporting of BPI results information. Some improvements have been made, including the mid-term and final performance assessments themselves. Regular results information is not routinely or comprehensively collected. Many of the activities under BPI are incremental to core ongoing activities, and separation of the incremental impact has proven to be very difficult.

In the following section, we bring together key information from documents, management interviews and consultations with private sector stakeholders and apply it to each of the outcomes. It is important to note that external stakeholders are not aware of the separation of BPI activities from Spectrum / Telecom Program activities overall and this was reflected in the interviews.

4.2.1 Short-Term Outcomes

Spectrum will be made available in a timely, efficient and equitable manner

Management believes that industry is generally satisfied with reallocating and increasing available spectrum. IC is striving to be a follower of US policy in terms of spectrum release. In particular, it was noted that the auction process has helped reduce bidder application times by more than 50%. Regionally, the trends appear to be headed in the right direction with regional companies able to access.

The Performance Assessment documentation indicates that "...the Spectrum/Telecom Program has been active over the five years in providing spectrum to meet priority needs. For example, significant new spectrum has been made available to meet new PCS (cellular) requirements; to facilitate new broadband access in remote regions in accordance with the Federal Broadband initiative; and to meet the needs of the growing array of short range consumer products for which demand has been strong. Modified use of spectrum has been implemented to meet growing broadcasting needs in Southern Ontario, particularly to meet ethnic broadcasting requirements. Initiatives have been implemented to maintain and further enhance the use of satellite broadcasting facilities to serve Canadians.4"

Stakeholders consistently acknowledged the improvements made in the past four years with respect to the timeliness, efficiency and equity of spectrum allocation. It was noted that IC has moved quickly to release spectrum and is better than FCC and Europe at accommodating new services, while still a bit slower than some Asian countries. Positive examples of this include PCS and WiFi, which were brought to market relatively easily. The development of spectrum auctions for fixed and mobile service spectrum users has been very good, resulting in improvements in efficiency and access that have been positive for Industry Canada and the private sector. Some improvement was also described in access to license exempt bands, such as accommodating the growing market for private family walkie-talkies.

Industry would like to see continued improvements in timeliness, specifically with respect to post auction verification of licensing requirements and replacement of bands reassigned by IC.

Increased understanding and knowledge and of the marketplace for spectrum planning, allocation and identification of issues and opportunities

Management believes that they have achieved an enhanced understanding of the marketplace over the period under review and are better able to anticipate emerging needs. It was noted that this is an on-going, critical activity.

Stakeholders support this view and note that IC understands the marketplace and the issues involved and can act as an effective arbitrator when needed. Again, improvements were noted over the past five to ten years. Stakeholders attribute this in large part to the level, nature and frequency of contact between Spectrum /Telecom Program and the private sector. Particular satisfaction was derived from the accessibility and degree of face-to-face contact with IC. Consultation is much more direct in Canada when compared to the FCC (US) and other agencies, leading to a greater appreciation on the part of Industry Canada of marketplace needs. Overall this level of understanding was judged to have had positive effects on businesses in Canada.

It was noted that some of the smaller Canadian organizations have limited influence and it is important for IC to ensure that smaller players are heard in the Canadian marketplace. Some concerns for the future were raised with respect to continuity of understanding in the face of potential large number of retirements within IC.

Overall, the Spectrum / Telecom Program has a very good understanding of the marketplace, with improvements recognized over the past five to ten years. Industry attributes this to the level of direct contact, consultation and ease of access as opposed to formal studies or research. The result has been a positive working relationship and positive effects on businesses. An important role for Industry Canada in the future is to continue to represent the smaller businesses that may not have a large voice on their own.

Canada will obtain international authorization to use the spectrum it requires to serve Canadian communications needs and create business opportunities

Spectrum / Telecom Program management states that progress on many issues has been consistent with the Canadian position. The representation of Canada at ITU and various working groups and subcommittees has been successful, with Canada's position acknowledged and respected. However, there remain some concerns that resources may not have been adequate for these activities and as a result, regulatory lag may occur in developing new regulations or policy.

The Performance Assessment concludes with: "The measure of success for the Outcome would be whether or not Canadian objectives had been met. This would be determined, by comparing the outcome of the meetings with the Canadian/CITEL proposals and negotiating position going in. This type of analysis is generally provided in Delegation Reports and briefings prepared by participants after the meetings. This type of documentation is readily available in the Spectrum/Telecom Program Branches and a sampling of it related to WRC 2000 and 2003 has been reviewed in the course of undertaking this Performance Assessment. On that basis, it can be concluded that the Shorter Term Outcome from this activity has been positive."

"With regard to the generation of new business opportunities, at an aggregate level the growth rate of output and employment in recent years in the wireless and telecommunications sector has continued to be above the average for national GDP, despite the general slowdown that has occurred in the high-technology and internet related sectors. A more detailed analysis of sector performance is provided in Annex II. At the level of individual companies and organizations, correspondence on file from companies that rely on spectrum such as Motorola, Boeing, MacDonald Dettweiler, Telesat, and Nortel Networks, as well as such organizations as the Canadian Wireless Telecommunications Association, and the Canadian Space Agency, confirm how important the positive outcome of international spectrum negotiations is to their business prospects."

Key stakeholders interviewed state that international representation is a very important role for IC. They believe that Canada has been very effective at gaining access to spectrum internationally and leads other countries in pressing for opening bands for international use. Again, it was noted that the relationship between industry and the Government is a key factor. IC is viewed to work effectively with mixed delegations (public, private, and NGO). The level of consultation and communication is considered good.

Representation at the international fora is a benefit to Canadian service providers and manufacturers, allowing Canada to influence standards to which products will be produced, which in the end, helps Canadian firms achieve market competitiveness.

Stakeholders concluded that Canada's international influence is greater than our economic size would suggest. Even though the Canadian delegation at international meetings is among the smallest, it is often the most successful. They attributed a good portion of the success to the fact that Canadian delegations usually arrive at the meetings with positions for which there is a public and private sector consensus already in place. Extensive communication between the public and private sector puts the delegation in a better position to take the initiative and negotiate rather than having to return to Canada for discussions and new instructions. Industry Canada should and does take the lead in these international fora.

Overall, progress on many issues is consistent with Canadian positions and Canada is seen to lead and influence disproportionately with respect to the small size of our delegations. Industry Canada's role on the international stage is highly valued by the private sector and is especially important to manufacturers. Business feels that they have been able to capitalize on the opportunities afforded by the negotiations and agreements. Consultation with industry and incorporation of their input facilitates better results from an industry perspective. There is a desire to see this level of effort continued and, by some, expanded.

Acknowledging that one can almost never do enough at these types of fora, some respondents believe that additional resources are still required and that under-resourcing been a constraint in the past. In addition, the private sector would like to see Industry Canada focus more on specifications and regulatory issues and less on technical issues and details.

Harmonization of international standards and creation of business opportunities for Canadian suppliers

The key activity contributing to this outcome is the negotiation and signing of MRAs. Work has been very successful in this area. The MRAs facilitate product testing and the approval process for electronic equipment in international markets. Benefits that accrue to industry are cost savings, faster time to market, level playing field for trade, and faster turn-around time for certification of equipment. The Performance Assessment notes: "Preliminary results show that Canadian telecom exporters have achieved savings of up to $100K per product type approved, and up to six months in time to market."

Additional benefits have occurred with progress on the harmonization of regulatory systems and the development of a robust testing certification industry in Canada. The certification of equipment has essentially been privatized as activities that were in IC are now being completed by private sector conformity assessment bodies. The labeling program is being streamlined as well. IC has moved to post-market surveillance, audit and an incident driven system in conjunction with CRA at the borders, instead of testing all the equipment itself.

The private sector sees Canada as a leader in the negotiation and signing of MRAs. The move to standardization is leading to lower costs and more rapid service introduction. It is getting easier for Canadian equipment to get into export markets and for Canadian companies to source a broader range of equipment for Canadian consumers.

Stakeholders would like to see additional effort applied to enable Canadian companies to take advantage of the MRAs. Work needs to continue on aligning regulations with the agreements on harmonization in order to facilitate access to other countries. As new services develop, more effort may be required for international harmonization of standards.

Overall, MRAs are viewed as very successful with considerable progress made on harmonization of standards and positive impacts on the private sector in terms of reduced cost and faster time to market. However, work must continue on the next steps including regulations, in order to truly take advantage of the MRAs.

Spectrum licensing and certain other services will be provided to the public on-line, providing better client services and faster decision making

This section focuses on the outcomes of the spectrum auctions and licensing.

Three successful auctions were held during the study period (one of which was ongoing in January/February 2005). Generally auction participation is good with smaller and regionally based companies able to participate. The first auction resulted in 52 licenses to five bidders for PCS services and the second resulted in 392 licenses for wireless broadband.

Stakeholders conclude that the auctions have worked smoothly. They provide for quicker licensing of spectrum and are efficient from a business perspective. Improvements could be made to simplify and clarify the policy and procedures manuals for the auctions to reduce the need to use the support services at the online sites. Some are concerned that auctions and other licenses are becoming more of a revenue generator then a means to manage the Spectrum.

The Performance Assessment of 2004 describes the success of Spectrum Direct:

"A very significant Spectrum/Telecom product supported in part by BPI funding was the Spectrum Direct Website implemented in 2002 as the means for customers to do business with the program on-line. Available statistics indicate that at present some thirty-five per cent of land mobile radio licence applications and seventy per cent of microwave applications are processed on-line using Spectrum Direct, and the annual percentages are on a growing trend. The Spectrum E-Services and E-Commerce Delivery Project (SEED), which managed the development of Spectrum Direct, was consistent with the Government-on-Line initiative implemented by the Government in 2001 and was also partially paid for from GOL funding.5"

In the Auditor General's report on the GOL initiative in 2003, the following was noted.

"Some departments and agencies are making good progress in providing on-line services and making the critical changes that delivering services electronically entails. One program sector at Industry Canada, Spectrum Management and Telecommunications, has been planning and aggressively pursuing the transformation of its operations for several years. Using the possibilities offered by new technologies, and in consultation with its key clients, it is developing new ways of issuing and managing radio-licences in real time. In the process, Spectrum is not only streamlining its operations but is creating new partnerships with its major clients and preparing and training its staff to provide more and better services to its clients. Spectrum had achieved 60 percent of its expected outcomes by April 2003, and Industry Canada expects to achieve 100 percent by December 2005."6

In the area of new licensing tools, these were seen as good when initially released, however, improvements have not been forthcoming as quickly as industry would like.

A key indicator of the achievement of this outcome is the level of client satisfaction. The Regional Offices have undertaken Client Satisfaction Surveys; with a baseline conducted in 2001 covering the processing of interference complaints and radio licence applicants; a second survey in the fall of 2002; and another survey currently underway. These surveys demonstrate a high level of satisfaction. Areas for improvement were also noted and measures have been taken to address them. Ongoing monitoring of key performance standards was also implemented.

Overall, auctions are generally working well and better information is available during the auction process to allow firms to adjust bidding. This creates a more efficient market where companies pay as little as needed to get product. Improvements could be made in the procedure manuals and timeliness of the post-auction verification process. Regional Directors report that smaller players are able to participate in the auctions. These data are limited and therefore prohibit a conclusion that the auction process works as well for small as large firms.

The Spectrum Direct initiative has been widely viewed as successful. Client Satisfaction surveys conducted in the Regions illustrate a high level of satisfaction with services.

Productivity and efficiency gains will be realized in the Program through the development of new tools for spectrum access, electronic interface with clients and use of NGOs for alternative service delivery

A wide variety of activities have been undertaken in headquarters and the regions to improve service delivery and realize productivity and efficiency. Examples of productivity gains are varied and not well consolidated, and include:

  • Reduced application and dispute resolution times due to implementation of new tools and equipment;
  • Reduced turnaround times on environmental assessments for antennae licenses; and
  • Improvements in licensing and certifications completed on-line (see Spectrum Direct—previous outcome).

Interviews with stakeholders support the improvements in turn-around times and efficiency gains.

ASD progress has been slower than anticipated and hence, the productivity and efficiency gains from these types of initiatives have been limited. The Spectrum/Telecom Program has had successes in the past in partnering with its clients to deliver its services and provide access to the spectrum. The Railway Association of Canada membership recently voted to continue to self-manage its spectrum.

The MRAs discussed above are a type of ASD in that private certification bodies conduct testing and certification that used to be provided by IC. Productivity and efficiency gains appear to have been achieved. The information leading to this observation is limited. Hence, solid conclusions cannot be derived at this time based on limited information. ASD projects have been slow to implement and the expected gains have not been fully achieved.

Enhanced understanding and compliance with standards, regulations, licensing procedures and policies

An assessment of the activities related to DTH satellite issues concluded: "A primary Outcome was the heightened public awareness that resulted, and the activities of illegal equipment vendors are reportedly now less visible. There appears to be a slowdown in the growth of illegal decoding activity and an anticipated longer term Outcome will be a reduction in the level of such activity. This will be determined by monitoring related websites and from reports on RCMP enforcement activity."

Management indicates that the investments in procedures, infrastructure and client education to support compliance monitoring have had some success. Efforts were made in response to pressures and some improvements have been observed.

IC has responded to these pressures with application of these funds. Some stakeholders reported success in increasing public awareness and a slowdown on the growth in the illegal decoding activity.

Enhanced emergency planning, network security and spectrum use for safety services

Stakeholders support work done to improve network integrity and security. The private sector and government are cooperating in this area. However, stakeholders expressed some concern with the level of resources and lack of priority given to these areas. A full assessment of the success of emergency planning activities is scheduled for review during the PSAT formative evaluation in late 2004/05.


4 BPI Initiatives—A Performance Assessment, R. Butler, December 2004 (return to reference 4)

5 BPI Initiatives—A Performance Assessment, R. Butler, December 2004 (return to reference 5)

6 Auditor General of Canada; Report of November, 2003; Chapter 1, the Government-on-Line Initiative; Chapter 1, Paragraph 49. (return to reference 6)

4.2.2 Long-Term Outcomes

Some attempts have been made to assess success in these two areas, primarily through stakeholder views and some limited documentation.

Canadian firms in the wireless and telecommunications industry will take advantage of related business opportunities, leading to stronger economic growth and improved communications services

A recent economic report completed by DGRB7 has outlined the importance of the wireless communications sector to the Canadian economy. Related to BPI more specifically, it discusses how MRAs and timely release of spectrum benefit the economy. Positive economic impacts related to PCS/cellular, spectrum auction, and license exempt spectrum are highlighted.

Stakeholders report that IC has set the stage for the private sector to take advantage of related business opportunities, given the downturn in the telecommunications industry over the past four years. A number of examples were provided of domestic and international opportunities that have been facilitated by Spectrum / Telecom Program activities.

Stakeholders report taking advantage of related business opportunities domestically and internationally. However, the positive impacts have been muted during the recent downturn in the telecommunications industry.

Minimized spectrum interference within Canada and along Canada/US border

Stakeholders generally report a low number of interference cases, and that the level of interference in licensed bands is relatively stable. This is due mainly to improved cross-border coordination efforts. Industry Canada's ability to deal with interference has improved, in that they have helped identify who is assigned what piece of spectrum and whether they are using it. The amount of time needed to address typical interference issues has been reduced from about six months to two months since 2001.

Interference with the United States depends on the service being offered. In the Golden Horseshoe area of Ontario there is more interference due to the high levels of use and location near the United States border. Generally, there is little interference for cell phones due to clearly negotiated bilateral agreements. There will be new potential for interference in the railway industry as travel is expanding in an increasingly north-south direction versus the traditional east to west direction.

Mobile radio has brought about an increasing frequency allocation issues. The growth in the use of mobile radio has been faster in the United States and Canada. This growth rate may soon force Canada to increase its spectrum management activity with respect to mobile radio bands.

In the licensed exempt bands there is thought to be more interference. As Canada and the US move towards greater use of direct satellite TV, there will be greater potential for difficulties. Canada and the US are putting more emphasis on access of spectrum by public safety agencies. Both of these developments suggest that there may be increased concern for interference along the border.

Overall, stakeholders generally believe interference has not increased in licensed bands. Changes in the structure of railway industry and mobile radio may be creating difficulties that will need to be addressed.

There is some indication of growing issues related to interference in license exempt bands.

4.3 Conclusions

Conclusion #4-1: The identified activities and outputs were broadly rolled out as expected with the exception of Alternative Service Delivery initiatives, which were particularly slow to implement due to challenges in the public and private sector environment.

Conclusion #4-2: Management reallocated funds to address significant pressures on the Spectrum / Telecom Program in the areas of compliance activities, network integrity and security and succession planning.

Conclusion #4-3: Timeliness, efficiency and equity of access have improved, with industry consistently acknowledging the improvements. Industry would like to see continued improvement focused on timeliness of post-auction verification and speed of replacement of bands reassigned.

Conclusion #4-4: The Spectrum / Telecom Program has a very good understanding of the marketplace, with improvements observed over the past five to ten years. Industry attributes this to the level of direct contact, consultation and ease of access as opposed to formal studies or research. The result has been a positive working relationship and positive effects on businesses. An important role for Industry Canada in the future is to continue to represent the smaller businesses that may not have as significant a voice.

Conclusion #4-5: Progress on many issues in international fora is consistent with Canadian positions and Canada is seen to lead and influence beyond what its size would suggest. Industry Canada's role on the international stage is highly valued by the private sector and is especially important to manufacturers. Business feels that they have been able to capitalize on the opportunities afforded by the negotiations and agreements. Consultation with and incorporation of industry input facilitates better results from an industry perspective. There is a desire to see this level of effort continued and, by some, expanded.

Conclusion #4-6: MRAs are viewed as very successful with progress on harmonization of standards and positive impacts on the private sector in terms of reduced cost and faster time to market. Work must continue to fully take advantage of the MRAs.

Conclusion #4-7: Auctions are generally working well and better information is available during the auction process to allow firms to adjust bidding. This creates a more efficient market where companies pay as little as needed to get product. Improvements could be made in the procedure manuals and timeliness of the post-auction verification process.

Conclusion #4-8: The Spectrum Direct initiative has been widely viewed as successful. Client Satisfaction surveys conducted in the Regions illustrate a high level of satisfaction with services.

Conclusion #4-9: Productivity and efficiency gains appear to have been achieved. This observation is based on limited information and therefore should be considered a tentative conclusion. Targetted data collection could be improved.

Conslusion #4-10: ASD projects have been slow to implement and the expected gains have not been fully achieved.

Conclusion #4-11: Stakeholders report taking advantage of related business opportunities domestically and internationally. The positive impacts have been muted during the recent downturn in the telecommunications industry.

Conclusion #4-12: Stakeholders generally believe interference has not increased in licensed bands. Changes in the structure of railway industry, mobile radio may be creating an issue that will need to be addressed. There is some indication of growing difficulties with interference in license exempt bands.


7 The Importance of Wireless Communications to the Canadian Economy Macro- and Micro-Economic Indicators, Dennis Ward, DGRB (return to reference 7)

5.0 Cost-Effectiveness

This section examines the management and decision-making structures, the use of lessons learned and consideration of alternatives. Data was collected from key management interviews and file review of planning and reporting documents and meeting minutes.

5.1 Are there appropriate management and decision-making structures in place to meet the objectives?

The primary forum for planning and management of the BPI funds is the Directors General Spectrum / Telecom Committee which includes the DGs of the three Headquarters Branches and representation from each Region. Meetings were held approximately three times a year for overall management of the Spectrum / Telecom Program and include a standing item on the BPI funds. In addition, monthly DG teleconferences included BPI items on an as-and-when required basis. These reviews are described as very active discussions and require substantial time.

The DG Committee uses a consensus model and focuses primarily on planning, funding allocations, shortfalls and reallocations, and the funding formula. An Office of Primary Interest (OPI) was established within DGRB in 2002 in response to recommendations in the mid-term performance assessment to coordinate administration.

Improvements in the planning and review functions have been observed overt the period under review in terms of records of decision, supporting documentation, and developing criteria for setting priorities and accessing funds. Examples of critical issues that were brought forward at this forum for discussion and decision include:

  • Record of discussion in 2001 on reallocating funds to succession planning;
  • Reallocation from BPI to DGRB for a salary and O&M shortfall. Carried through to 2002 and 2003;
  • Record of discussion in 2002 for high priority items: succession planning, security related initiatives, SEED/ALS/SOTT initiatives;
  • Record of decision to reallocate funds to DTH 2003; and
  • Record of decision to reallocate some funds to DGSE for salary shortfalls.

Day-to-day project management occurs at the DG and Regional levels for individual projects and activities. This approach is considered appropriate given the breadth of projects and individual requirements, particularly in the Regions. Each Region and Branch appears to have established internal management processes which vary significantly across the country. Co-ordination between Ottawa and the Regions appears good at the project level. However, the lines of authority to senior management are less clear for the Regions. The Regions have a direct line of reporting on operational activities to the ADM of Operations and for the BPI activities, they also report to ADM of SITT.

Flexibility has been identified as critical to the management of the Spectrum / Telecom Program given the dynamic nature of the environment. The Program strove to balance stability with flexibility through a funding formula. Roughly 80% of the funds were allocated to the Branches and Regions based on an agreed-upon formula. The remaining 20% was then made available for new pressures and priorities. The program did change with the environment and responded to new requirements, with examples already noted in cyber security, succession planning, DTH and antennae issues. Some improvements could be made in the rigor of the process used to establish priorities.

In summary, the primary forum for management and decision-making was administered by consensus through the DG Level Committee. This DG committee focus was on financial management, with output and outcomes managed individually by the DGs. Individual DGs and Regions have independence of action within the bounds of BPI objectives. At the project level, there were clear lines of authority for planning, managing and implementing projects.

Management was characterized by a balance of stability and an appropriate degree of flexibility to allow the Program to respond to the dynamic environment. Improvements have been noted in terms of management process and documentation. Possible enhancements to management and decision-making include more rigorous processes for priority setting and more timely identification of free balances as year-end approaches to facilitate reallocation.

5.2 Has there been an assessment and use of lessons learned?

A comprehensive mid-term review of the BPI program was completed in November 2002. The objective of the review was to determine if the projects and activities being implemented were effectively managed in accordance with the objectives and expected results set out in the Treasury Board Submission. The review was presented at DG Management meetings and recommendations were reviewed. Key recommendations and the actions taken are outlined below.

Mid-Term Review of the BPI Program
Recommendation Action Taken
The establishment of an office of primary interest. Completed. DGRB assumed role and assigned management and administrative leads.
The inclusion of regional representation in the management of the program. Completed. Regional participation at quarterly management reviews and monthly teleconferences.
Revision of the format of the presentation of the financial information to communicate impact of BPI Financial information presented and reviewed regularly at DG meetings by Branch. Not well linked to activities, outputs or outcomes.
Revisit the funding allocation formula to assess need to increase regional allocation. No evidence of discussion of increasing regional allocation. Some revisiting of formula in 2003 to discuss if high priority funds should be increased.
Bring together the available results information and improve reporting with respect to the 6 measures outlined in the TB Submission. IC has met its commitments for five of the six results measurement undertakings. Information remains limited for reporting on the success or failure of individual spectrum recipient organizations.
Consider whether a separate reporting document is required to communicate achievements of BPI activities. Final performance assessment completed. Separate reporting document not implemented.

In addition, a final performance assessment was completed in 2004 with the following aim: "To obtain and analyze performance information and provide a performance assessment of initiatives implemented as Business Plan Investments (BPI) under the five-year funding approved by Treasury Board. The assessment will focus on the achievement of the expected results and the related means of measurement as set out in the Treasury Board Submission prepared to obtain the funding." The results of this assessment were shared within the Program.

Mid-term and final performance assessments were conducted for the BPI funds and the majority of the lessons learned from the mid-term assessment were discussed and implemented/rejected.

5.3 Alternatives

The nature of the BPI activities creates special challenges in assessing the consideration and use of alternatives. First, the BPI activities are broad categories of activities and a diverse number and type of actual projects can be funded under a category. Second, the funding is often incremental to A-Base funding rendering alternatives assessment very difficult without examining the entire Spectrum / Telecom Program.

Notably, an estimated 36% of the funds were directed to the activity grouping "improving service delivery" which is itself targeted at increasing cost-effectiveness and efficiency. In addition, there were isolated instances of structured process improvement initiatives. For example, two Regions conducted a structured review of service delivery compared to best practices based on Baldridge principles. Improvements were targeted and client satisfaction tracked. The other Region has implemented Process Improvement Teams.

5.4 Conclusions

Conclusion #5-1: The primary forum for management and decision-making was the joint DG Level Committee including the three Branches and the Regions. This DG committee focus was on financial management, with output and outcomes managed individually by the DGs. Individual Branches and Regions have independence of action within bounds of BPI objectives. At the project level, there were clear lines of authority for planning, managing and implementing projects.

Conclusion #5-2: Management was characterized by a balance of stability and an appropriate degree of flexibility to allow the Program to respond to the dynamic environment.

Conclusion #5-3: Improvements have been noted in terms of management process and documentation.

Conclusion #5-4: Possible enhancements to management and decision making include more rigorous processes for priority setting and more timely identification of free balances by project as year-end approaches to facilitate reallocation.

Conclusion #5-5: Mid-term and final performance assessments were conducted for the BPI funds and the majority of the lessons learned from the mid-term assessment were discussed and implemented or rejected.

6.0 Funding

This section explores issues of levels of funding and the nature of the funding model. No audit activities were performed. Effort was directed toward reviewing the impact of the funding model on management and planning processes. Data were collected from key management interviews and document review of planning and reporting documents.

6.1 Was the funding sufficient?

Information received during the management interviews indicated that, in a broad sense, funding levels were sufficient in that they were able to meet the commitments described in the TB Submission. Activities and outputs were implemented as committed, though there is recognition of the reallocation from ASD to critical activities during the period under review.

A review of the expenditures for the period 2000 – 2004 finds that some insignificant amounts were rolled over, with the highest being $363,000 in 2003/04 or approximately 4% of the total for the year. The funds rolled over in that year were then applied to priority initiatives in the following year. Guidelines allow up to 5% of the fund to be rolled over into the next fiscal year. Current year projections are to be at a zero balance by fiscal year end.

BPI Year End Balance — 2000/2001 to 2003/2004
Fiscal Year Surplus / (Deficit) Notes
2000/2001 $97,796 This surplus was due to slippage from Telezone funds. The roll-over was applied to initiatives for FY 2001/02
2001/2002 $416 These funds lapsed as roll-over was not an option.
2002/2003 ($15,478) No roll-over
2003/2004 $363,430 This surplus was rolled-over to help address key initiatives such as auctions and SIRR.

6.2 Is the use of program, sunset funding the most appropriate way to fund these activities?

Incremental funding has been provided to the Spectrum / Telecom Program in five year cycles for an extended period of time. While the activities implemented have evolved over time, there has always been a mix of operational, core activities and time-limited, discreet projects. The additional funding is considered essential to the effective delivery of many program activities. The funding model was identified as an issue in mid-term review of 2002 when program managers initially expressed concern that many activities now considered to be ongoing for the foreseeable future are being paid for through funding that has not been approved as part of the Department's continuing A-base budget. This model continues to be of concern in the face of renewal.

From a high level review of the activities and categories of projects, it is estimated that approximately 70 – 80% of the BPI expenditures over the five-year period were related to ongoing, operational type activities. The remaining 20 – 30% was related to discreet, project-type activities. These are estimates were derived from the project descriptions and the documentation in the BPI Initiatives Performance Assessment of 2004. This suggests that the Abase funding for the Spectrum / Telecom Program may be insufficient.

Further, it is noted that many of the project-type activities have now become operational in nature. For example, while the establishment of the auction system, processes and procedures and the execution of the initial auctions was a project, this has become an operational activity of the Spectrum / Telecom Program and will continue into the future.

Reflecting the fact that there are two distinct types of activities funded under BPI: incremental support to core business; and discreet, project type activities; the assessment of the impact of the funding model on management decision-making is quite mixed.

On the positive side, the funding model encourages better accountability in terms of regular planning, reporting, and controls. Regular review of the use of funds is required and there are benefits to taking a close look at the value of activities. Resources are generally kept targeted on needs with sufficient flexibility to respond to changes in the environment. In summary, the funding model works well for the time-limited, project type activities.

Contrary to this, the funding model is not a good match with the core, operational, ongoing activities that are funded in the regions and in headquarters. It is thought to require a significant level of effort and resources to manage, and may be out of proportion for a relatively small fund supporting core activities. There is a high overhead cost in terms of personnel and time with respect to management, reporting, evaluations, and negotiations. The uncertainty in the time leading up to renewal impedes long term planning. For example, negative impacts have been noted on succession planning and training. The funding model limits management's ability to deal with activities that are or will become regular.

In summary, the funding model fits well with the project-type activities and does not fit well with the bulk of the activities, which are operational in nature, limiting good management and creating significant uncertainty at time of renewal.

6.3 Conclusions

Conclusion #6-1: Broadly, the funding levels were sufficient in that Spectrum / Telecom Program was able to meet its commitments. Instances of funds being rolled over were observed, but were recognized to be within the 5% allowable.

Conclusion #6-2: While the funding model fits well with the project-type activities, it does not fit well with the bulk of the activities, which are operational in nature, limiting good management and creating significant uncertainty at the time of renewal.

Conclusion #6-3: BPI funding in the regions and in headquarters was regularly used as a support for normal operations, which suggests that A Base funding may be insufficient.

7.0 Design and Delivery

This section examines the extent to which the design and delivery of the investment initiatives facilitates established objectives. Data sources included management interviews and file review of planning and reporting documents and meeting minutes.

7.1 Were the program and activities designed and delivered with the appropriate management accountability?

Day-to-day project management and output/outcome monitoring was the responsibility of individual DG and managers. This has had the positive effect of placing accountability close to responsibility for implementation. It was noted that BPI elements are often in management PMPs adding another level of accountability.

Progress Reporting has been an area of focus for improvement, with reports of continuous improvement since the '90s. However, much of the focus has been on tracking success of the entire spectrum management program. The ability to demonstrate the success of the incremental benefits of the BPI activities is uneven, again reflecting the two types of activities: core, operational activities; and discreet projects.

With no evaluation strategy identified at the outset of the BPI program, indicators and data required were not specified. The monitoring of outcomes has been challenging with some improvements made following mid-term review. The Regions have made significant efforts to establish and implement performance measurement. Work remains to be completed on the identification of attributable outcomes, the selection of the right indicators of performance and regular collection and reporting of data. It will be necessary to reflect the differences in the regional and national perspectives.

As noted in Section 4.1, financial planning has been based on organizational structure (Branches and Regions) as opposed to the seven activity categories. This has made reporting by these categories challenging as noted in both the mid-term and final performance assessments.

Day-to-day accountability has been implemented appropriately. Improvements could be made in progress reporting, particularly with the identification of a small number of attributable outcomes, selection of the right indicators, and regularly collection and reporting of data.

7.2 Could the needs be satisfied through a mechanism other than the BPI?

Management of telecommunications and the radio spectrum is the legislated responsibility of the Minister of Industry and is managed within the Spectrum / Telecom Program. BPI funds have been instrumental in Spectrum / Telecom's ability to respond to a very fast-moving and evolving environment. Specific needs identified by Spectrum / Telecom Program's activities in spectrum management, including:

  • Accelerating technological change;
  • Regulatory developments;
  • Convergence of technologies; and
  • Increasing demand for new telecommunications services.

Some of the activities are time-limited projects that respond to this fast-moving environment and our findings are that the BPI mechanism remains an effective way to deal with these needs. However, many of the activities funded by the BPI fund have become part of normal operations and funds are being used to conduct ABase type activities. For these type of activities, alternatives to the BPI mechanism should be considered, such as a move to ABase.

7.3 Conclusions

Conclusion #7-1: Day-to-day accountability has been implemented appropriately. Improvements could be made in progress reporting, particularly through identification of a small number of attributable outcomes, selection of the right indicators (before BPI implementation begins), and regular collection and reporting of data during the life of the program.

Conclusion #7-2: The BPI mechanism remains an effective way to deal with activities that are time-limited projects or in response to this fast-moving, evolving environment. However, alternatives to the BPI mechanism should be considered for the activities that have become part of normal operations.

8.0 Recommendations

Recommendation: Develop alternative funding mechanism

We find that BPI is an effective means of responding to the evolving, fast-moving environment of the Spectrum / Telecom Program. Industry Canada should pursue an alternative funding mechanism or process to transition activities from BPI that have become operational or ongoing in nature in order to continue to service the evolving needs of the Canadian telecommunications industry. This should be done in conjunction with central agencies.

Recommendation: Maintain stakeholder consultation with, and ease of access, to IC

Ongoing structured consultation with key stakeholders, including the use of mixed delegations, is seen to be key in the areas of international issues and Mutual Recognition Agreements. Ease of access to government representatives is highly valued. These should be maintained and further institutionalised in the regions, consistent with Federal Government Smart Regulation principles of valuing the perspectives of stakeholders and enhancing access to the federal government. Stakeholders recognised the importance of including large and small business in the process. IC should monitor equity of access for small and large businesses.

Recommendation: Develop and implement an appropriate evaluation framework

While no formal evaluation framework was required at the outset of this program, good management practices require this. Should additional project funds be sought, an evaluation framework should be developed and implemented. A limited set of specific, measurable and attributable results indicators need to be clearly identified at the outset, and in situations where incremental funding is applied to existing activities, results should be identified to demonstrate the incremental effects. The level of effort for data collection and reporting should be relative to the importance and value of the fund.

Recommendation: Maintain and build on current governance structure

The joint headquarter and regional committee DG has generally been an effective management mechanism. The levels of flexibility and controls were appropriate to the needs of the program. Management should consider a more rigorous process for priority setting such as a business case approach or establishment of criteria to evaluate projects. The practical effects of the dual report requirements on Regions should be monitored.

Management Response

Evaluation of the Spectrum Management Business Plan Investments (BPI)—Recommendations and Responses

Recommendation: Develop alternative funding mechanism.

Response: We agree. We are seeking increased reference levels or authority for re-spending some of our auction revenues.

Recommendation: Maintain stakeholder consultation with, and ease of access, to IC.

Response: We agree. The Spectrum/Telecommunication Program (S/TP) remains deeply committed to consultations with all stakeholders (both large and small), through such mechanisms as the Radio Advisory Board of Canada, ITU Preparatory Committees, the Canada Gazette and one-on-one meetings with industry representatives. Our regional and district offices, in particular, play a key role in ensuring this consultation extends to small and medium enterprises, who may not necessarily be represented by industry associations.

Recommendation: Develop and implement an appropriate evaluation framework.

Response: We agree. The S/TP is currently developing a Performance Management Framework that will provide the necessary level of transparency and accountability. We recognize that this approach is an important cultural change and that we have to be cautious to not create and feed a database to answer a need but rather to use the data as a management tool for the S/TP.

Recommendation: Maintain and build on current governance structure.

Response: We agree. In response to the dual reporting situation, the S/TP has created the Spectrum/Telecom Directors General Committee, where program issues and priorities are resolved in a collegial fashion. Their work is supported by three subcommittees, representing both HQ and the regions, that consider issues, analyse and evaluate options, identify risks, and provide recommendations to the senior committee.