ARCHIVED - Evaluation and Responses

WarningThe following information is out of date.

Archived content

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Evaluations and Responses - Evaluation of the SIN Segregation of Duties Pilot

Table of Contents

Executive Summary

Management Response

  1. Introduction 
  2. SIN Quality Management Strategy
    1. Broad SIN Quality Management Pilot 
    2. SIN Segregation of Duties Pilot 
  3. Evaluation Scope
    1. Design and Implementation 
    2. Costs 
    3. Error and Risk Mitigation 
  4. Methodology
    1. Literature and Document Review 
    2. Key Informant Interviews 
  5. Limitations 
  6. Findings
    1. Literature Review 
    2. Design and Implementation 
    3. Costs 
    4. Volumes 
    5. Processing Time 
    6. Error and Risk Mitigation 
  7. Conclusions 
  8. Observations

Executive Summary

The primary objective of the Social Insurance Number (SIN) Quality Management (QM) Strategy developed by Integrity Services Branch, Service Canada is to ensure that proper quality controls and monitoring mechanisms are in place for the SIN issuance and approval process. The Strategy includes two models which were piloted in the Ontario Region:

  • The Broad SIN Quality Management Strategy, which includes training, certification, random sampling and a pop-up screen feature, and
  • SIN Segregation of Duties.

The Broad SIN Quality Management Strategy model was piloted in twenty-one Service Canada Centres across Ontario, three of which were located in the Toronto area (Toronto Centre, Toronto North, and Brampton). The SIN Segregation of Duties was piloted only at the Scarborough Service Canada Centre.

The objective of the evaluation was to examine and assess the effectiveness of the Segregation of Duties pilot. The Segregation of Duties model was examined in the context of the QM strategy and in comparison with the Broad SIN QM model, which was also piloted as part of the Strategy. Specifically, the evaluation looked at issues related to the following:

  • Design and Implementation
  • Costs
  • Error and Risk Mitigation

The evaluation relied primarily on the results of a literature/document review and key informant interviews.

Despite some initial confusion and misunderstanding reported by staff, the evaluation found that both the SIN Segregation of Duties and Broad SIN Quality Management Strategy pilots were successfully implemented. Management and staff understand the importance of protecting the integrity of the SIN and appreciate the need for proper and effective controls.

Over the course of the evaluation neither the Segregation of Duties nor the Broad SIN Quality Management Strategy pilots detected potential fraudulent SIN transactions. The Broad SIN Quality Management Strategy and Segregation of Duties models were comparable in terms of identifying administrative and noncritical errors. The evaluation was unable to determine whether either model represented a more effective internal control due to an absence of preimplementation baseline data for comparison purposes.

The Segregation of Duties model is more resource intensive than the SIN Quality Management Strategy model and broader implementation could have significant resource implications for Service Canada.

The Segregation of Duties Pilot was developed and implemented very rapidly. Ideally, pilot testing of a control mechanism should involve a representative cross-section of offices and include collection of pre-implementation base-line information.

Observations:

  • To the extent possible, national pilot test designs should include a cross-section of offices with varying characteristics and from various regions. This design element would support the extrapolation of pilot results to the regional and national levels.
  • In order to support the evaluation and assessment of actual impacts against intended results, pilot project designs should include provisions for the collection of key pre-implementation baseline information.
  • Communication and consultation with Regional and Service Canada Centre management and staff can be critical to the successful design and implementation of a pilot. Although sometimes difficult due to operational constraints, participation of frontline staff can provide valuable insight into the practical and operational issues associated with the introduction of new processes.

Evaluation of the SIN Segregation of Duties – Ontario Region -- Integrity Services Branch Response

Integrity Services Branch is pleased to have participated in this evaluation and we are committed to using the observations raised to inform our work as we move forward with our various integrity initiatives related to the management of the Social Insurance Number (SIN). It is important to note that the results of both pilots demonstrated significant integrity within the SIN issuance process as highlighted by the fact there were zero critical errors found in either model.

Integrity Services Branch will continue to modify SIN processes and systems to have internal controls that ensure the integrity of the SIN issuance process and the Social Insurance Register. This includes the implementation of other components of the Broad SIN Quality Management Strategy and the implementation of direct real-time validation with provincial and vital events registers, beginning in 2008. These measures will not only mitigate internal risks but will also enhance the overall quality of the SIN insurance process. As concluded in the evaluation report, there is no evidence to support the additional resources required by the Segregation of Duties model, consequently Integrity Services Branch will continue to proceed with the full implementation of the Broad SIN Quality Management Strategy.

The following are Integrity Services Branch’s response to the three observations raised in the evaluation report.

Observation # 1

  • To the extent possible, national pilot test designs should include a crosssection of offices with varying characteristics and from various regions. This design element would support the extrapolation of pilot results to the regional and national levels.

Integrity Services Branch recognizes the value of having pilots designed to include a cross-section of offices. As such, the Broad SIN Quality Management Strategy was designed to include this element. The strategy was piloted in March 2007 in twenty-one Service Canada Centres ranging from large-volume offices to low-volume offices; these offices were considered representative of the types of offices throughout Canada. As a result, the findings were used to develop and modify systems and processes to meet the needs of offices with varying characteristics, including remote-type offices.

It is also recognized that the design and development of the Segregation of Duties pilot only encompassed one Service Canada Centre (Scarborough). As this office receives the largest volume of SIN clients, there are particular risks associated with this office. Based on these risks, and senior management’s interest in exploring the segregation of duties concept in relation to the SIN, this office was selected for the pilot.

Observation # 2

  • In order to support the evaluation and assessment of actual impacts against intended results, pilot project designs should include provisions for the collection of key pre-implementation baseline information.

Integrity Services Branch acknowledges the value of having baseline information included in the pilot project design and therefore one of the objectives was to obtain this type of information. The Broad Quality Management Strategy was a new model being developed and included systematic random sampling. This key component was tested during the pilot in order to obtain data required to determine the baseline for quality and integrity. As it continues, this data is collected on an on-going basis and will be assessed regularly so we can continually improve on the quality of SIN processing.

Observation # 3

  • Communication and consultation with Regional and Service Centre management and staff can be critical to the successful design and implementation of a pilot. Although sometimes difficult due to operational constraints, participation of front-line staff can provide valuable insight into the practical and operational issues associated with the introduction of new processes.

Integrity Services Branch recognizes the value of broad consultation, including front-line staff. As a result, the design and implementation of both models included broad consultation. The Broad Quality Management Strategy was designed using established Business Reengineering processes, including having the Business Process Review and the Laboratory performed by a cross-functional team comprised of individuals from all areas: an SCC front end agent; an SCC team coordinator; an SCC local manager; investigators; regional consultants and employees of Integrity Services Branch. Following the pilot and implementation, formal feedback was solicited from the local and regional employees, and the findings have been instrumental in ensuring that the quality management processes in place are practical and meet the desired objectives.

1. Introduction

The purpose of this report is to present findings from an evaluation of the Social Insurance Number (SIN) Segregation of Duties Pilot implemented at the Scarborough Service Canada Service Centre. The pilot was conducted as part of the SIN Quality Management Strategy developed by the Integrity Services Branch, Service Canada.

The objective of the evaluation was to examine and assess the effectiveness of the Segregation of Duties pilot. The Segregation of Duties model was examined in the context of the QM strategy and in comparison with the Broad SIN QM model, which was also piloted as part of the Strategy. It was also piloted as part of the Strategy. The evaluation attempted to determine the effectiveness of Segregation of Duties in mitigating risks and its impact on services and resource requirements.

2. SIN Quality Management Strategy

The primary objective of the SIN Quality Management (QM) Strategy (developed by Integrity Services Branch, Service Canada) is to ensure that proper quality controls and monitoring mechanisms are in place for the SIN issuance and approval process. As well, the strategy is intended to determine whether the controls and measures are reliable and that data collected and reported on the performance of internal controls is valid and provides a true measure of effectiveness. The QM strategy emphasizes continuous improvement in procedures and performance, risk-based analysis, improved system controls, as well as the mitigation of the risk of fraud in the SIN issuance process.

The Quality Management Strategy includes two models which were piloted in the Ontario Region:

  • The Broad SIN Quality Management Strategy, which includes training, certification, random sampling and a pop-up screen feature.
  • SIN Segregation of Duties.

The Broad SIN Quality Management Strategy model was pilot tested in twenty-one Service Canada Centres across Ontario, three of which were located in the Toronto area (Toronto Centre, Toronto North, and Brampton). The SIN Segregation of Duties pilot was implemented only at the Scarborough Service Canada Centre.

  1. Broad SIN Quality Management Pilot

    In the Broad SIN QM pilot, a client is met by a First-Come First-Served Agent (FCFS) who makes an initial inquiry about the intent of their visit. For example, if the client is at the Service Canada Centre to obtain or replace a SIN, the Agent confirms that the client has the required Proof-of-Identity Documents (PIDs) to obtain a new or replacement SIN card. The client is then asked to take a seat and wait for a call from a certified SIN Agent. The SIN Agent reviews and verifies the client’s PIDs to make sure that they are valid/appropriate in order to process the SIN application and, if so, the required information for SIN issuance is entered in the Social Insurance Register (SIR) system. It should be noted that in rural or smaller locations, responsibility for FCFS and SIN issuance may be vested in one person.

    Under pop-up model, a sample of SIN applications is randomly selected for quality monitoring prior to completion of the SIN issuance process. In these cases, a notification screen “pops-up” prior to the SIN confirmation advising the agent that the case has been chosen for quality monitoring. In response, the agent must photocopy all the client’s identity documents used to process the request and fax them under signature of the Service Canada Centre manager to SIN Management Services in Bathurst New Brunswick.

    SIN Management Services reviews the copies to ensure that proper personal identification documents were submitted and that the information was accurately input to the Social Insurance Register (SIR) system. This check is intended to reduce the risk of fraud as well as to determine the quality of the data on the SIR and to identify trends in SIN processing.

    As an additional control, clients whose applications triggered a random pop-up were asked to respond to several ‘identification questions.’ Responses to these questions were used for identification purposes during call backs conducted 24-48 hours after the transaction to verify that the client had indeed applied for a SIN.

    The QM sampling rate can be varied as factors warrant.

    The SIN Quality Management Strategy has a number of other features including coaching and certification of employees, regular performance feedback and monitoring, and reporting on results.

  2. SIN Segregation of Duties Pilot

    The SIN Segregation of Duties model was designed to ensure that more than one individual is involved in the SIN issuance and approval process.

    When a client comes into the Scarborough Service Canada Centre to apply for a new or replacement SIN, the FCFS Agent logs the client into the FCFS system and confirms that the client has the required PIDs. A photocopy of the documents is made and retained by the FCFS Agent and the originals are returned to the client. The client is then directed to the waiting area to be called by the first available, certified SIN Agent. This Agent processes the SIN request as per standard procedures.

    At predetermined times on a daily basis; the Service Delivery Coordinator (SDC) collects copies of PID documents from the FCFS Agent along with copies of processed applications (SIN action results). The SDC is responsible for matching PID documents to the applications, reviewing results, following up on any identified discrepancies and completing a daily ‘sign off’ report, which is submitted electronically to SIN Management Services in the Bathurst office for validation against the system-generated control reports.

3. Evaluation Scope

As already noted, the evaluation focussed on the Segregation of Duties Pilot component of the SIN QM Strategy. Specifically, the evaluation looked at issues related to the following:

  1. Design and Implementation

    • What steps were followed in developing and implementing the SIN QM pilots?
    • Were there any implementation issues?
  2. Costs

    • What were the incremental resource costs associated with implementing the QM Strategy pilots?
    • How do costs associated with the Segregation of Duties Pilot compare to those of the Broad QM Strategy pilots?
  3. Error and Risk Mitigation

    • Was the Segregation of Duties model effective in reducing error and mitigating the risk of fraud?
    • Does the Segregation of Duties model differ significantly in terms of reducing error and mitigating risk as compared to the Broad SIN QM Strategy model?

The SIN QM Strategy is only one component of a strategy to protect the integrity of SINs. For example, several options exist to enhance security and integrity of the SIN/SIR. Assessment and analysis of these and other options, some of which have already been examined by Integrity Services Branch, are outside the scope of this evaluation.

4. Methodology

The evaluation of the SIN Segregation of Duties relied primarily on the results of a literature/document review and key informant interviews.

  1. Literature and Document Review

    The main goal of the literature review was to provide information and context that would help inform the interpretation and analysis of findings from other lines of evidence. Based on published material, the literature review attempted to document common/best practices in identity and credit card issuance available in both public and private sector organizations within OECD nations. Unfortunately, published information on the subject was found to be limited. To supplement the literature review findings, 29 interviews were conducted with representatives of external organizations including the Canadian Bankers Association.

    Operational data were also reviewed as part of the study. These data included time differentials in processing applications with and without the pop-up screen, number of applications processed, numbers and types of errors.

  2. Key Informant Interviews

    A main line of evidence for this evaluation was key informant interviews with Service Canada managers and staff. Key informants were selected to reflect the range of activities associated with the design, development, and implementation of the SIN Segregation of Duties pilot, as well as to assess its effectiveness and utility as an internal control process to mitigate the risk of error and potential fraud. The key informants consulted at Service Canada included management and staff of the Integrity Services Branch, Regional SIN coordinators and service delivery staff and supervisors at the Scarborough and Toronto North Service Canada Centres.

    Specific interview guides were developed for the key informants within NHQ, Regional and Service Canada Centre offices. Interviews were conducted during June and July 2007. Most interviews were conducted by phone with some interviews conducted in person. A copy of the interview guide was sent to each participant prior to the interview. In total, twelve key informants participated.

5. Limitations

An important limitation on the evaluation was a lack of pre-implementation baseline data. SINce the gathering of this information was not built into the pilot design, it was not possible to assess the post implementation impact of the Broad SIN QM Strategy and Segregation of Duties models with respect to the mitigation of potential fraud.

Processing time reported for SIN transactions are estimates based on information gathered through key informant interviews and volumes as reported in system generated reports. Time and motion studies were not conducted as part of either pilot, nor were they done as part of the evaluation study.

6. Findings

  1. Literature Review

    Many organizations, both public and private, have internal controls in place to safeguard the protection of the information they possess and to prevent fraud and error. These include random verification of transactions, processes to identify forged documents and risk assessments. According to several of the organizations interviewed, segregation of duties is one of many internal controls normally used to prevent errors and mitigate risk.

    The literature review found that segregation of duties is applied differently in various organizations. However, the principle remains the same; no one individual completes the whole transaction. For example for debit and credit cards, financial institutions apply segregation of duties principles to the management of client databases. Similarly, for the issuance of driver’s licenses, the person entering data often does not have access to the photo files or signatures needed to complete the transaction.

    While the organizations contacted were understandably reluctant to discuss specific details of how they apply the segregation of duty control, the literature review identified certain factors as constraints in implementing new and modified segregation of duties procedures as an internal control. These factors included:

    • A perception that existing procedures adequately deal with existing risks.
    • Small number of staff, thereby limiting the feasibility of separating duties.
    • Strong focus on customer satisfaction combined with the perception that introducing segregation of duties would negatively impact service delivery.
    • Difficulties involved in making changes to systems and procedures in place, especially in organisations with a large number of offices and staff.

    None of the organizations contacted for the literature review reported having conducted a formal cost-benefit analysis of the segregation of duties control. The organizations saw no rationale for conducting a cost-benefit analysis for the following reasons:

    • They viewed segregation of duties as functional in nature, representing a logical work flow. They believed that there was little if any additional costs, provided offices could accommodate a segregation of tasks.
    • Segregation of duties is just one part of an organization’s overall risk management strategy.
  2. Design and Implementation

    The March 2007 implementation of the Broad SIN Quality Management Strategy pilots was the culmination of a process that started in March 2006 when the proposal for the development of a QM strategy was first presented. There were five key elements to the project:

    • Development of a project charter.
    • Business process review by a cross-functional ‘improvement team’ that included NHQ, regional and Service Canada Centre representatives.
    • Consultation with regions.
    • Preparation of supporting documents, working tools and regular conference calls.
    • Launch of the SIN Quality Management Strategy pilot in March 2007 in twenty-one Service Canada Centres across Ontario including three in the Toronto area (Toronto Centre, Toronto North, and Brampton).

    In contrast, the Segregation of Duties pilot, although part of the QM Strategy, was developed over a two month period and was piloted in only one office, the Scarborough Service Canada Centre.

    Key informants felt that the pilots were good initiatives. Both management and staff within and outside NHQ fully understand the need to put in place proper controls to protect the security and integrity of a SIN. However, several key informants amongst frontline staff expressed concern that the Segregation of Duties pilot was designed with little or no input from them. They stated that it would have been beneficial during the design and implementation phases, to have harnessed the understanding and practical knowledge of frontline workers.

    Although information about the Segregation of Duties pilot was disseminated within the Scarborough office via e-mail, some frontline staff indicated that they were not initially certain of the purpose of performing these pilots. While conference calls with Scarborough office were conducted to provide details concerning the pilot, most frontline officers could not attend due to operational constraints. Key informants indicated that there were some delays in processing SIN applications early in the pilot as a result of this uncertainty.

    Despite considerable consultations and engagement between the Ontario Region and NHQ, some key informants involved in the Broad SIN Quality Management Strategy pilot reported that the nature and content of the pilot was not effectively communicated to frontline staff. In particular, there was initial confusion concerning the process whereby several extra questions were asked of randomly selected clients for the purpose of call-backs to verify that the client had applied for a SIN. This additional requirement caused confusion for some clients. Delivery staff was sometimes unable to provide a satisfactory explanation as to the purpose of these questions. However, as the pilots progressed, staff gained confidence in following the steps and delays were largely eliminated SINce the staff was able to provide satisfactory answers to questions posed by clients.

  3. Costs

    Segregation of duties as a control requires more resources than the Broad SIN Quality Management Strategy model. For the Scarborough pilot, five additional staff was required. This included three employees from the Scarborough office to implement the segregation of duties control and two full-time equivalents at NHQ for validation of reports, follow-up, quality review and reporting. It was estimated that the Segregation of Duties process added an average of 7.5 minutes to each transaction, due to increased photocopying and fax requirements.

    Two methods are offered to extrapolate the incremental resource implications of a national implementation of SIN segregation of duties as tested in the Scarborough office. The first is based on volumes of applications processed, the second on the incremental time required to process an application.

  4. Volumes

    The Scarborough Service Canada Centre processes about 3.33 percent or 1/30th of all new and replacement cards issued in Canada. Extrapolating this incremental resource requirement noted to the national level, approximately 150 additional full-time-equivalents (FTEs) would be needed to implement this control nationally.

  5. Processing Time

    An alternate means of estimating resource implications of introducing segregation of duties nationally is to calculate incremental costs based on the additional 7.5 minutes of processing time when using this control. As there are approximately 1.5 million SIN transactions per year, segregation of duties would require the dedication of an additional 11.25 million minutes of processing time per year. Based on a 7.5 hour working day (450 minutes) and an average of 220 working days per year, approximately 114 FTEs would be needed to implement segregation of duties.

    It should be noted that these are only rough estimates and do not take into consideration operational constraints. For example, segregation of duties as a control requires the participation of at least two individuals in each office and, as such, might prove impractical to implement in smaller Service Canada Centres and Outreach offices.

    The cost implications for the Broad SIN Quality Management Strategy model are not as high. Managing and administering the pilots required the dedication of three FTEs at National Headquarters. If implemented nationally, the Broad QM Strategy would likely require only a minimal incremental resource commitment.

  6. Error and Risk Mitigation

    Analysis of operational data revealed that 88 per cent of transactions were error free in the Broad SIN QM Strategy pilot sites. Errors found in the remaining 12 per cent of the transactions were non-critical and administrative in nature (e.g., data formatting, minor spelling discrepancies) and did not suggest a potential for fraud.

    A comparable administrative error rate of 14 per cent was reported for SIN transactions processed by the Segregation of Duties pilot site. As in the case of the Broad SIN QM sites, critical errors, or errors of the type indicating a potential for fraud, were not reported.

    While errors detected were limited to non-critical, administrative categories, the evaluation could not conclude that either model reduced the risk of fraud as there were no pre-pilot data (e.g. baseline data) against which to make a comparison. The absence of baseline data for comparison purposes made it impossible to conclude whether the SIN Segregation of Duties model was more, less or equally effective in mitigating potential frauds.

    Although not part of either the Broad SIN Quality Management Strategy or Segregation of Duties pilots, some key informants reported that Service Canada has signed Vital Events data sharing agreements with a number of provinces and that these agreements provide the potential to significantly improve the integrity of the SIN. Agreements are currently in place with the provinces of New Brunswick, British Columbia, Ontario, and Alberta and negotiations are underway with Manitoba, Nova Scotia and Saskatchewan.

    These agreements permit Service Canada to electronically validate, in real time, the Proof-of-Identity presented by the clients during the SIN application process. Besides offering potential to improve the PID authentication process, these agreements are seen as key to the introduction of validation processes to support the integrity of data captured and maintained in the Social Insurance Register.

7. Conclusions

Both the SIN Segregation of Duties and Broad SIN Quality Management Strategy pilots were successfully implemented. Management and staff understand the importance of protecting the integrity of the SIN and appreciate the need for proper and effective controls. Nevertheless, there was some initial confusion and misunderstanding reported by staff.

Key informants and literature review results indicate that segregation of duties can mitigate the risk of potential fraud. However, over the course of the evaluation neither the Segregation of Duties nor the Broad SIN Quality Management Strategy pilots detected potential fraudulent SIN transactions. The evaluation was unable to determine whether either model represented a more effective internal control, as pre-implementation baseline data were not available for comparison purposes. Nevertheless, based on transactional and operational data provided, it would appear that the Broad SIN Quality Management Strategy and Segregation of Duties models are comparable in terms of identifying administrative and non-critical errors.

The Segregation of Duties model is more resource intensive than the Broad SIN Quality Management Strategy model and broader implementation would have significant resource implications for Service Canada. Due to these limitations, the evaluation was unable to assess whether or not this additional cost would be warranted.

The Segregation of Duties Pilot was developed and implemented very rapidly. Ideally, for an organization the size of Service Canada, pilot-testing a control mechanism should involve a representative cross-section of offices of varying characteristics and from a number of regions.

Finally, lack of pre-implementation baseline data made the assessment of the impact of the SIN Segregation of Duties pilot difficult. It is acknowledged that very tight timelines were imposed for the implementation of the pilot and as a result, the design was not able to include baseline data acquisition.

8. Observations

  • To the extent possible, national pilot test designs should include a crosssection of offices with varying characteristics and from various regions. This design element would support the extrapolation of pilot results to the regional and national levels.
  • To support the evaluation and assessment of actual impacts against intended results, pilot project designs should include provisions for the collection of key pre-implementation baseline information.
  • Communication and consultation with Regional and Service Canada Centre management and staff can be critical to the successful design and implementation of a pilot. Although sometimes difficult due to operational constraints, participation of frontline staff could provide valuable insight into the practical and operational issues associated with the introduction of new processes.