Design of a Self-Audit Tool for the Application of Lockout on Machinery in the Province of Quebec, Canada to Control Hazardous Energies

Abstract

Failure to apply lockout procedures for the control of hazardous energies is one of the main causes of machinery-related fatal and serious injuries in North America. The absence of audits of lockout or the lack of proper tools for auditing lockout is prevalent, and thus the application of lockout is often not fully in compliance with standards and regulations. A self-audit tool for the application of lockout procedures for machinery was developed on the basis of the current standards and regulations, and previous research. The tool was then tested for content validity through experts’ opinions and qualitative feedback from six organizations in the province of Quebec in Canada. The developed audit tool defines the actual procedures to audit, as well as the surrounding conditions that are needed and the prerequisites based on standards, regulations, and findings from previous research. The results showed that the tool displayed a high content validity index and demonstrated that the usability, applicability, and comprehensiveness of the tool were adequate. This self-audit tool helps organizations monitor the application of lockout on machinery for the safety of workers and to ensure that the actual practice of controlling hazardous energy is in compliance with relevant standards and regulations.

1. Introduction

1.1. Safety Audit

The international standard ISO 19011 explains that an audit is a systematic, independent, and documented process to obtain objective evidence and evaluate it objectively to determine the extent to which the audit criteria are being fulfilled [1]. The audit can also be internal or external. The standard states that internal audits are conducted by, or on behalf of, the organization itself. On the other hand, external audits are conducted by other individuals outside of the organization or by independent auditing organizations (e.g., governmental agencies) [1]. The internal audit allows the organization to determine if its occupational health and safety (OHS) management system is effectively implemented and maintained, and also whether it is compliant with standards [2,3]. A safety audit can be based on regulatory or other compliance assessments (i.e., compliance-type audits) through a review of the documents, interviews, and observations to determine whether the workplace is safe [1,4]. Occupational Safety and Health Administration (OSHA) recommends regular workplace inspection and compliance audits and mentions self-assessment auditing as an effective method to evaluate and improve safety programs [5]. In spite of this fact, conducting internal audits is less widespread in small companies [6,7], which prefer performing external audits likely because of a shortage of qualified internal auditors or resources [8].

Checklists and questionnaires are prevalent tools/methods for auditing safety management systems [9]. Checklists are helpful to conduct observational audits [10]. Depending on the objectives of the audits of an OHS management system [11], it is important to assess measurement properties (e.g., validity, reliability) of the audit tool and to find the extent to which they are acceptable [12,13]. These authors showed, for example, that the content validity of an audit instrument is the most important test before auditing every OHS management system. Content validity indicates the comprehensiveness of the audit tool and the extent to which it represents OHS system concepts [14]. On the other hand, assessing inter-auditor reliability or the responsiveness of a tool is more important when the measurement of the performance of a safety system is the main objective of an audit [13].

1.2. Audit of Lockout

Lockout comprises the procedures, techniques, and methods to protect personnel from injury from the inadvertent release of hazardous energy [15]. North American standards and regulations address the minimum requirements necessary for the methods which are carried out during the non-production phase (e.g., service, repair, and maintenance) of machinery to control hazardous energies (e.g., electrical, hydraulic, pneumatic, kinetic, potential, chemical, and thermal in nature) [15,16,17,18]. These standards and regulations require a specific lockout procedure for each machine, equipment or process in accordance with the general lockout procedure. The general lockout procedure provides authorized employees with a step-by-step approach to controlling hazardous energies, and requires: (i) preparation for shutdown; (ii) shutdown of machine, equipment or process; (iii) isolating the machine, equipment, or process; (iv) application of lockout devices; (v) dissipating and controlling stored energy (de-energization); and (vi) verification of isolation [15,16,18,19]. Moreover, some standards and regulations require a documented lockout program that establishes the company’s general policies and procedures for implementing lockout [15,16,17]. Lockout programs should comprise the following elements: (i) identification of the hazardous energy covered by the program; (ii) identification of the types of energy isolating devices, (iii) identification of the types of de-energizing devices; (iv) selection and providing of protective devices and hardware; (v) assignment of roles and responsibilities; (vi) the general lockout procedure; (vii) alternative methods; (viii) training; and (ix) audit and inspections [15,16,20,21].

The Canadian standard CSA Z460 [15] and American standard ANSI/ASSE Z244.1 [16] state that compliance with specific hazardous energy control procedures (machine, equipment, or process) is critical. These standards (CSA Z460, clause 7.6.3; ANSI/ASSE Z244.1 clause 6.5.2) explain that “the user shall establish a continual auditing plan that will provide current information on the maintenance of application effectiveness. Auditing shall be conducted at least annually and documentation shall be maintained for at least three years. The application effectiveness audits should be random and address all shifts, days of operation, groups, non-standard work situations and personnel. Knowledgeable personnel should conduct visual observations of authorized individuals performing specific hazardous energy control tasks. These observations should include feedback to the authorized individuals and documentation of the findings and any recommended improvements” [15,16]. Furthermore, audit results can reveal the need for retraining. The audit must be a part of the lockout program in which the frequency of the audit, sample size, auditor’s responsibilities, and documentation of an audit are explained [15,16]. In Quebec, the regulation (ROHS) [18], which is almost in line with North American standards and OSHA 29 CFR 1910.147 [22], requires that employers revise the lockout procedures periodically. However, the audit of the lockout program is not mentioned in the regulation. Similarly, OSHA 29 CFR 1910.147 [17] requires that the employer conduct periodic inspections of lockout/tagout procedures at least annually to ensure that the procedure and the requirements are being followed, and also to correct any deviations or inadequacies identified. However, by contrast to OSHA 29 CFR 1910.147, the Quebec regulation (ROHS) requires that tagout be only utilized as an alternative method to control hazardous energy. According to OSHA 29 CFR 1910.147, the periodic inspection must be a part of the lockout program. The regulation (1910.147(c)(7)(iii)(B)) also explains that “additional retraining shall also be conducted whenever a periodic inspection reveals, there are deviations from or inadequacies in the employee’s knowledge or use of the energy control procedures” [17]. Auditing is imperative in these both regulations. The requirements for auditing lockout in North America are briefly described in

Table 1. Audits of lockout in North American standards and regulations.

Standard/Regulation	Audit of a Lockout Program (i.e., Elements of a Program)	Audit of the Application of Lockout Procedures
CSA Z460	Auditing of program elements (lockout program review). This monitoring and measuring frequency shall be at regular intervals of three years or less.	The user shall be responsible for conducting the auditing plan (e.g., semi-annually) through visual observations of authorized individuals applying specific lockout procedures to verify that complete compliance is occurring.
ANSI/ASSE Z244.1	Auditing of program elements (lockout program review). This monitoring and measuring frequency shall be at regular intervals of three years or less.	The user shall be responsible for conducting the auditing plan (e.g., semi-annually) through visual observations of authorized individuals applying specific lockout procedures to verify that complete compliance is occurring.
Quebec’s ROHS (art. 188.5)	Not mentioned in the regulation	“The procedures must be reviewed periodically, in particular every time a machine is altered or a failure is reported, so as to ensure that the energy control method remains efficient and safe.”
OSHA 29 CFR 1910.147 (1910.147(c)(6)(i)(A); 1910.147(c)(6)(i)(C))	Not mentioned in the regulation	“The periodic inspection shall be performed by an authorized employee other than the ones(s) utilizing the energy control procedure being inspected. The periodic inspection shall include a review, between the inspector and each authorized employee, of that employee’s responsibilities under the energy control procedure being inspected.”

. This table includes requirements from the Canadian and American standards on lockout, OSHA 29 CFR 1910.147, and Quebec’s regulation.

As Table 1 demonstrates, the audit of the application of lockout should at least consist of the verification of lockout procedures and the observation of the application of lockout, which is carried out by authorized employees.

1.3. Deficiencies Related to the Audit and Application of Lockout

The U.S. Bureau of Labor Statistics (BLS) revealed that in the period 2015–2018, a total of 723 fatal work injuries annually (approximately 14% of total fatalities) occurred as a result of contact with objects and equipment [23]. Moreover, OSHA reported that lockout was the fifth most cited OSHA violation in the period 2015–2018. For example, during 2017 and 2018, OSHA issued 3,131 and 2,944 citations, respectively, for violations of the lockout/tagout standard OSHA 29 CFR 1910.147. The lack of documented lockout procedures and the absence of periodical inspections were two of the most-cited sections in the lockout/tagout standard [24,25]. Likewise, in the province of Quebec (Canada) on average, 10% of fatalities occurred annually due to poor or absent lockout procedures [26]. Studies showed that one of the leading causes of lockout-related accidents in North America was the absence of lockout procedures [27,28,29]. Chinniah [28] indicated that relying on the lockout program as a measure of actual lockout practices was not a reliable factor since accidents happened due to failure in the application of lockout procedures in organizations that even have lockout programs.

Several studies have explained the importance of audits on lockout. Kelley [30] indicated that periodic inspections of the application of lockout procedures make it possible to correct observed deficiencies. The auditor must observe a sufficient number of authorized personnel who apply a lockout procedure. The author explained that the audit of the application of a lockout procedure should address the following three main questions: (i) are the steps of the lockout procedure being followed correctly? (ii) do authorized employees understand their responsibilities for lockout?; and (iii) is the lockout procedure adequate to control the energy? For each question, if the answer is negative, the auditor must describe the problem and the corrective measures taken or that are planned. Grund [31] stated that audits allow companies to evaluate actual practices of implementation of lockout and to identify the problems (linked to the adequacy and efficiency of lockout procedures and also employee training) and to correct them. The author recommended that the audit should cover five aspects, including (i) machinery; (ii) lockout procedures; (iii) alternative methods, (iv) training; and (v) worker perception [31]. The worker perception of lockout was discussed through situation awareness (SA) [32].

Karimi, Chinniah, Burlet-Vienney, and Aucourt [22] investigated the actual practice of controlling hazardous energies in deferent organizations and demonstrated several major shortcomings to the control of hazardous energies on machinery, such as missing steps in lockout procedures, neglecting to read the placards before applying lockout, and not having appropriate audit tools or the absence of audits. Their findings were consistent with another study reporting the problems such as (i) the lack of audits; (ii) not always using lockout placard; and (iii) not performing the verification step of lockout procedures during the application of lockout [33]. Furthermore, Karimi, Chinniah, Burlet-Vienney, and Aucourt [22] showed that although organizations conduct internal audits on lockout (e.g., auditing their lockout program or auditing the application of lockout), lockout audits are incomplete due to a lack of appropriate audit tools. To illustrate, they found that 40% of the organizations they studied had no tools for audits of lockout and existing audit tools (of the remainder of the organizations) consisted of only a few questions to verify the application of lockout, thereby deficient in completeness. Additionally, in some of the organizations, the audit of lockout was a small part of general safety audits, and it was incomplete. The authors concluded that there is a need for a comprehensive tool for internal audits of lockout to provide organizations with accurate audit results [22].

To our knowledge, the study of developing a valid and complete self-audit tool for the application of lockout on machinery is new. There are no proposed audit tools in the standards or regulations. Yamin, et al. [34] introduced a self-audit tool (checklist) for lockout in manufacturing workplaces. Whereas their tool showed good inter-rater reliability, the content of the tool does not encompass all of the requirements of the application of lockout procedures carried out by authorized employees. For example, questions (items) on continuity of lockout, external services and assessment of some steps in a general lockout procedure could be included in their tool.

In the present study, the main objective is to develop a valid and proper self- audit tool specifically for the application of lockout procedures on machinery. A tool for the audit of a lockout program already exists [20]. Our novel tool defines the actual procedures to audit, as well as the surrounding conditions that are needed and the prerequisites based on standards, regulations, and findings from previous research. It consists of pre-audit and audit and stages that are absent in existing tools. The new tool is complete, easy to use, serves as a checklist, and is practical in order to help organizations conduct internal audits of the application of lockout for controlling hazardous energies.

2. Materials and Methods

2.1. Developing the Self-Audit Tool

A self-audit tool was developed by the research team on the basis of the findings of Karimi, Chinniah, Burlet-Vienney, and Aucourt [22]; the current standards [15,16] and regulation [18]; and a review of the literature. Indeed, the checklist statements were mostly designed based on our prior study [22] (i.e., shortcomings in in the control of hazardous energies on machinery, related recommendations, and the questionnaire used) and the requirements for the application and audit of lockout procedures from: (i) the American standard ANSI/ASSE Z244.1 (e.g., clauses 6.5.2, 7.2, and 7.6.10–11), (ii) the Canadian standard CSA Z460 (e.g., clauses 7.6.3, 7.3.2.3–7), and (iii) Quebec regulation ROHS (arts. 188.5–8, 188.10–11). The final self-audit tool is presented in Section 3 (Results).

According to the literature, lockout is applied by authorized employees through lockout procedures. Therefore, before the observation of the application of lockout, which is carried out by authorized employees, it is necessary to verify the availability and content of lockout procedures, the availability and condition of lockout materials/hardware, and the training records of authorized employees (using audit records can be useful as well). This verification is important for finding gaps or deficiencies and for correcting them before the evaluation of the application of lockout. Thus, in this paper, the self-audit tool for the application of lockout consisted of the two stages, (i) the pre-audit and (ii) the audit. These two stages of the tool comprised a set of checklist statements that were in fact the pre-requirements and requirements of the application of lockout procedures. For each statement, if the pre-requirement or requirements were met, they could be marked with a check (✔); if not, they could be marked with an x (✕), and if not applicable, they could be marked (N/A). The latter was clarified in related checklist statements. In the pre-audit stage, if each checklist statement was not met, required actions would need to be taken by auditors or organizations. In this regard, the required actions were defined and embedded in the pre-audit stage (on the tool) to help organizations address problems. In the audit stage, the auditors could write their comments in the designated column of the tool. The tool also consisted of general instructions and general information (Appendix A,

Table A1. Self-audit tool: Instructions and General information.

Instructions
This self-audit tool has been developed for the application of lockout. By using the self-audit tool, organizations can evaluate the application of lockout carried out by an authorized employee or an external contractor. The audit should be performed by internal auditors who are experts in lockout. Before performing the audit, the auditors must read these instructions as well as the guidelines provided in the tool. The tool is considered generic as it includes recommendations from the standards CSA Z460 (2013), ANSI/ASSE Z244.1 (2016), and Quebec regulation ROHS (2017).  The audits can be random or planned and can address all shifts, days of operation, groups, non-standard work situations, and individual personnel. Organizations can determine the frequency of monitoring and sample size. Detailed information about audit schedule/planning and documentation usually can be found in lockout programs. •  The tool and process of self-audit  This tool is a comprehensive checklist based on a self-audit process and explains how to carry out the audit of the application of the lockout. The self-audit process consists of the two stages named pre-audit and audit. Before starting, the auditor must select in advance a machine/equipment and/or a task for which lockout is applied.  Pre-audit: consists of four sections (general conditions, content of the procedure, authorized employee, and required lockout hardware). In this stage, the auditor must ensure that the necessary conditions/pre-requirements are met. First, the physical conditions are checked (e.g., existing and up-to-date lockout procedure, available lockout material, adequate and functional energy isolation devices). The auditor then verifies training records of authorized employees and also previous audit/inspection records. The tool provides the required actions linked to the pre-requirements which are not met. If pre-requirements are not met, the organization needs to take action to correct them. In this regard, the lockout program also needs to be verified. The verification of lockout program elements (e.g., general lockout procedure, training, audit, subcontractors) is available on the website of the IRSST (i.e., verifying the content of lockout programs: RF-635).  Audit: consists of two sections (de-energizing steps and re-energizing steps). In this stage, the auditor, along with the previously-checked lockout procedure, observes the actual application of lockout for the targeted equipment or task and checks each requirement thereof in the tool through his/her observation and verified lockout procedure (i.e., the lockout procedure has been checked in the previous stage). The auditor can write his/her notes or comments (in the right column) for each requirement which is not met.
General information
Name of department/section:
Machine, process, or equipment being observed:  Task being observed:
Employee(s) being observed:   Authorized employee(s)   External services (sub-contractor):
Name/title of auditor:   Owner or general manager   Safety director/supervisor   Shop supervisor   An authorized employee other than the one(s) working on this machine, process, or equipment
Signature: -------------------------- Date of pre-audit: ------------/------------/-----------  Date of audit of the application: -------------/-------------/----------

).

2.2. Validation and Analysis of the Tool

After developing the self-audit tool, the tool was evaluated for content validity to ensure that all of the requirements of the standards (i.e., CSA Z460 and ANSI/ASSE Z244.1) [15,16] and Quebec regulation [18] are fully reflected in the tool. The assessment was a two-step process, including a review by a panel of experts and testing by the six recruited organizations in the province of Quebec, Canada. The expert panel in this study comprised six experts in the control of hazardous energy who were the representatives of different sectors including: (i) governmental sector on standard and regulation (i.e., CNESST: Commission des normes, de l’équité, de la santé et de la sécurité du travail); (ii) equipment and machinery fabrication; (iii) pulp and printing industry; (iv) transport and warehousing; (v) hospital sector; and (vi) manufacturing sector.

Before assessing the tool for content validity, it (the content of the tool) was translated from English into French, since the official language in the province of Quebec, Canada is French. In the first step, the tool was presented to the expert panel in a group meeting. The experts were asked to judge the appropriateness and relevance of the checklist statements of the tool and the extent to which they agree with them (a scale ranging from irrelevant to highly relevant). All feedback was collected and then the content validity index (CVI) was assessed through (i) calculation of the content validity index for items (I-CVI), which is computed for each item (i.e., statement) as the number of experts find it relevant or highly relevant, divided by the total number of experts [35]; (ii) calculation of the content validity index for scale (S-CVI), which is computed as the average of the I-CVIs for all items on the scale (i.e., all of the statements in the tool) by adding up the I-CVIs, then dividing them by the total number of items [36].

Subsequently, in the next step, the self-audit tool was tested by six organizations in order to receive their feedback (including more practical points of view) after using this tool for the application of lockout procedures. The research team also sent a short questionnaire about the content, usability, and completeness of the tool (

Table 2. Questions used for testing content validity.

Aspect (Elements Considered)	Questions for Participants Involved in Testing the Tool
Content of the tool	Are the statements clear and understandable? Does the sequence of statements make sense? Is the tool in line with the lockout regulation? To what extent?
Usability of the tool	Is it easy to use the tool? To what extent? Does it meet your expectations? Does the tool meet your needs?
Comprehensiveness and completeness of the tool	Is the tool complete and covers all points for an audit of the application? Is there any statement that needs to be added or completed?

). The organizations could contact the research team (through phone or email) in case they had questions about the tool. The feedback received from each organization were categorized by tabs in a Microsoft Excel© file to facilitate qualitative data analysis [37]. Each tab corresponds to the aspects of the questionnaire listed in Table 2. Each tab includes a number of columns based on the questions and also six rows assigned to the organizations selected. All the feedback of each organization was entered in the related cells.

2.3. Organization Recruitment

For the present study, six organizations in the province of Quebec, Canada were selected and recruited on the basis of convenience sampling [38,39]. However, the organizations recruited were diverse in terms of industry, size, and machinery/equipment (as a heterogeneous group of organizations).

Table 3. General information about the six organizations in the study.

Organization	Sector	Size (Number of Employees: <100; <500; ≥500)	Number of Items of Machinery/Equipment (Approximately)	Existence of a Safety Committee	Experience about Audits of Lockout
A	Chemical industry	<100	125	Yes	Yes
B	Manufacturing	<500	800	Yes	Yes
C	Printing	<500	100	Yes	Yes
D	Municipal	≥500	5000	Yes	Yes
E	Pulp and Paper	<500	4000	Yes	Yes
F	Aerospace	≥500	1300	Yes	Yes

summarizes the list of organizations that were selected. Having at least five years of experience (the main selection criterion) with lockout on machinery was mandatory for the organizations selected. As shown in Table 3, all organizations selected had some experience with internal audits on the application of lockout. However, this was not a criterion for selection.

3. Results

3.1. Self-Audit Tool for the Application of Lockout

The validated self-audit tool is presented in

Table 4. Self-audit tool: Pre-audit of the application of lockout.

Pre-Audit of the Application of Lockout
Guidelines:  Before performing the audit of the application of lockout on the selected equipment/task, it is necessary to check the pre-requirements mentioned below.  Read the requirements carefully. Mark (✔) if a requirement is met; (✕) if not met; (N/A) if not applicable. If lockout is applied by external services/subcontractor, in each statement, “authorized employee” should be replaced with “external services”.
Pre-Requirements (of Pre-Audit)	✔ ✕ N/A	Required Actions (if the Pre-Requirement Is not Met)
General conditions
The equipment/machine/process has a written lockout procedure for the targeted task		Written lockout procedure must be provided and tested (exception: where a machine is unplugged under the exclusive control of the person who uses it, or where the machine has a single energy source and where there remains no residual energy after the machine is unplugged). The audit should be postponed until a written lockout procedure is provided.
The procedure has been updated or modified based on recent changes in the selected machine/equipment/process		If there have been changes in the machinery or tasks, the procedure must be updated and verified. The date of creation, revision, and update of each lockout procedure must be documented.
The equipment/machine is in good condition (operating status, energy cut-off points, control system, guards and safeguards, etc.)		A visual check is necessary to check the general conditions of the equipment. Safety aspects are covered at this point.
The content of the lockout procedure
Identification of the equipment/machine		The missing item must be added to the lockout procedure. ANSI/ASSE Z244.1:16 (7.2.1), CSA Z460-13 (7.3.2.4) and the Quebec regulation (sections 188.6 and 188.7) describe the minimum content of a hazardous energy control procedure.
Identification and location of every control device and every energy source of the equipment/machine
Identification and location of every cut-off point of every energy source of the equipment/machine
Identification of the person responsible for the lockout procedure
The type and quantity of material required for applying lockout
Procedural steps for the application of lockout:
deactivation and complete shutdown of the equipment/machine;
elimination or, if that is impossible, control of any residual or stored energy source;
lockout of the equipment/machine’s energy source cut-off points;
verification of lockout by using one or more techniques making it possible to reach the highest level of efficiency;
safely unlocking and re-operating the equipment/machine
Where appropriate, the required personal protective equipment or any other complementary protection measure
Where appropriate, identification of the transfer of responsibilities or required material to ensure the continuity of lockout during a staff rotation/shift change
Authorized employee/subcontractor
Each authorized employee(s) (who will be audited) has been trained or retrained if needed		Check the training/retraining records specifically if there is a change in the machinery or tasks. In general, retraining must not exceed three years, otherwise provide training and document the records. Training requirements are explained in the lockout program.
In the case of the external services/sub-contractor, a written authorization has been issued for the external services/sub-contractor before undertaking work in the danger zone		The written authorization for external services must be provided before the application of lockout. The lockout program provides more detail on the management of external services.
Required lockout hardware or devices
Appropriate lockout devices/hardware for each type of energy control point of the equipment/machine are available and easily accessible in a lockout station or next to machinery		Visual inspection is required to verify the availability and accessibility of required equipment.
The material required for the application of the procedure is in good condition (Statement was added after re-analysis of the organizations’ feedback)		A visual check is necessary to check the condition of the equipment.
The lockout station is generally in good order (e.g., cleanliness, presence of equipment other than that required for the targeted procedure) (Statement was added after re-analysis of the organizations’ feedback)

and

Table 5. Self-audit tool: Audit of the application of lockout.

Audit of the Application of Lockout
Guidelines: The auditor, along with the previously-checked lockout procedure, observes the application of lockout carried out by the authorized employee(s) or external services (sub-contractors). The auditor can identify gaps quickly by using the tool and the lockout procedure (which has been checked in the pre-audit stage) when observing the authorized employee(s). The actual practice of the application of lockout procedure, including the de-energizing steps, placing lockout hardware and energizing (re-energizing) steps, is observed. Read the requirements carefully. Mark (✔) if a requirement is met; (✕) if not met; (N/A) if not applicable. If lockout is applied by external services/subcontractors, in each statement, “authorized employee” should be replaced with “external services”.
	Requirements (of Audit)	✔ ✕ N/A	Auditor’s Note
De-energizing steps
1	The lockout procedure is easily accessible to the authorized employee (e.g., posted near the equipment, available on the intranet)
2	Authorized employees search for the lockout procedure and read its contents
3	Authorized employees get appropriate lockout equipment and devices (e.g., lockout box, padlocks, hasps) (Statement was moved from pre-audit stage after re-analysis of the experts’ feedback)
4	Affected employees are notified before applying the lockout procedure
5	The authorized employees identify all hazardous energy sources of the equipment/machine to be locked out (as per the procedure)
6	The authorized employees mark off the places (e.g., using warnings and signs) where work is carried out in order to protect any employee who is likely to be exposed to danger (Statement was added after re-analysis of the experts’ feedback)
7	The equipment/machine is shut down by using normal stopping procedures (e.g., putting a switch in the “off” position)
8	The equipment/machine is isolated from every energy source (e.g., close valves, switch off main disconnects, switch off circuit breakers)
9	The authorized employees apply hasps and their personal padlocks and information tags in accordance with the lockout procedure
10	The type of lockout required (e.g., simple or group) is respected as per the procedure (Statement was added after re-analysis of the organizations’ feedback)
11	All potential residual hazardous energies are relieved, disconnected, or restrained (e.g., the hydraulic or pneumatic system purged, trapped pressure relieved, pipe flanges blanked, elevated equipment blocked or supported)
12	The verification step (verification of isolation) is performed according to the established procedure to ensure that the equipment/machine cannot be operated (e.g., push start buttons, turn on disconnects, test circuitry, measure the voltage or hydraulic pressure, visual inspection of measuring instruments by authorized employees or supervisors)
13	In the case of more than one authorized employee working on equipment, all employees affix their own (personal) padlocks according to the established procedure
14	In the event of several authorized employees working on the equipment, all employees have the opportunity to participate in the verification step according to the established procedure (Statement was added after re-analysis of the organizations’ feedback)
15	In the case of personnel or shift change, the authorized employee follows the instruction to ensure the continuity of lockout, as mentioned in the procedure
16	In the event that a change in the type of lockout is required, authorized employees follow the instructions, which are explained in the lockout program (Statement was added after re-analysis of the organizations’ feedback)
Re-energizing steps
17	Before lockout removal, authorized employees verify that the affected employees are safe and away from the equipment
18	The equipment/machine is inspected to ensure that it is ready to return to service. All equipment components are intact and capable of operating properly (e.g., machine guards are in place, etc.) (Statement was added after re-analysis of experts’ feedback)
19	Padlocks are only removed by the authorized employees who applied them
20	In the case of the absence of an authorized employee, the supervisor or employer follows the instructions for removing the padlock(s) of the absent authorized employee (Statement was added after re-analysis of the organizations’ feedback)
21	The equipment/machine is resupplied according to the established procedure
22	The authorized employees start the equipment and check that everything is working properly and that the work is finished (e.g., ensure that all work and interventions are completed)
23	All affected employees are notified of the completion of the intervention
24	The application of the procedure is recorded according to the established procedure (e.g., by archiving or by filing a register, etc.)
Other comments/problems observed (including comments from authorized employees):

, which are the pre-audit and audit stages respectively. A brief guide at the beginning of each table is provided.

The pre-audit (stage) of the application of lockout (Table 4) comprises four sections:

(i)

General conditions: three statements indicate and check the pre-requirements linked to the existence of a written and updated lockout procedure and the characteristics of the equipment selected.

(ii)

Content of the lockout procedure: 12 statements present and verify the pre-requirements ensuring completeness of the content of the procedure.

(iii)

Authorized employee/sub-contractor: two statements indicate and review the pre-requirements linked to the training/retraining records of the related authorized employees as well as the existence of an authorization permit for external services (if applicable).

(iv)

Required lockout hardware/material: three statements indicate and check for the pre-requirements linked to the availability of required lockout hardware in the application of lockout.

The audit (stage) of the application of lockout (Table 5) consists of two sections:

(i)

De-energizing steps: 16 statements represent and verified the requirements linked to the de-energizing equipment/machine step.

(ii)

Re-energizing step: eight statements represent and verify the requirements linked to the step on returning to service or re-energizing equipment/machines.

3.2. Content Validity

The feedback from the six experts in the group meeting demonstrated that they recognized all checklist statements as relevant items, a sign that the experts had reached a consensus. The calculated CVI (content validity index) for each checklist statement and the entire audit tool revealed a high validity rate. In other words, for the pre-audit checklist (Table 4), all of the statements except one had the highest CVI scores (I-CVI = 1). The experts (3/6) found one statement (about getting appropriate lockout equipment and devices) to be an irrelevant statement (I-CVI = 0.5) in the pre-audit stage and proposed moving it to the audit stage (Statement 3 in Table 5). The content validity index for the scale also had a high score (S-CVI = 0.972). Similarly, all checklist statements in the audit stage (Table 5) were found to be relevant or highly relevant, and therefore the I-CVI score for each statement was 1, and content validity index for the audit checklist had the highest score (S-CVI = 1). The details of the calculation of I-CVI and S-CVI for the tool (pre-audit and audit checklists) are presented in Appendix A (

Table A2. Calculation of I-CVI and S-CVI for the initial self-audit tool: Pre-audit of the application of lockout.

Item	Statement	Expert 1	Expert 2	Expert 3	Expert 4	Expert 5	Expert 6	Number in Agreement	I-CVI
1	The equipment/machine/process has a written lockout procedure for the targeted task	*	*	*	*	*	*	6	1.00
2	The procedure has been updated or modified based on recent changes in the selected machine/equipment	*	*	*	*	*	*	6	1.00
3	The equipment/machine is in good condition (operating status, energy cut-off points, control system, guards and safeguards, etc.)	*	*	*	*	*	*	6	1.00
4	Identification of the equipment/machine	*	*	*	*	*	*	6	1.00
5	Identification and location of every control device and every energy source of the equipment/machine	*	*	*	*	*	*	6	1.00
6	Identification of the person responsible for the lockout procedure	*	*	*	*	*	*	6	1.00
7	The type and quantity of material required for applying lockout	*	*	*	*	*	*	6	1.00
8	Deactivation and complete shutdown of the equipment/machine	*	*	*	*	*	*	6	1.00
9	Elimination or, if that is impossible, control of any residual or stored energy source	*	*	*	*	*	*	6	1.00
10	Lockout of the equipment/machine’s energy source cut-off points	*	*	*	*	*	*	6	1.00
11	Verification of lockout by using one or more techniques making it possible to reach the highest level of efficiency	*	*	*	*	*	*	6	1.00
12	Safely unlocking and re-operating the equipment/machine	*	*	*	*	*	*	6	1.00
13	Where appropriate, the required personal protective equipment or any other complementary protection measure	*	*	*	*	*	*	6	1.00
14	Where appropriate, identification of the transfer of responsibilities or required material to ensure the continuity of lockout during a staff rotation/shift change	*	*	*	*	*	*	6	1.00
15	Each authorized employee(s) (who will be audited) has been trained or retrained if needed	*	*	*	*	*	*	6	1.00
16	In the case of the external services, a written authorization has been issued for the external services before undertaking work in the danger zone	*	*	*	*	*	*	6	1.00
17	Appropriate lockout hardware for each type of energy control point of the equipment/machine are available and easily accessible in a lockout station or next to machinery	*	*	*	*	*	*	6	1.00
18	Authorized employees get appropriate lockout equipment and devices (e.g., lockout box, padlocks)	*	*	-	-	-	*	3	0.5
								S-CVI = Mean I-CVI = 0.972
I-CVI, item-level content validity index. S-CVI, scale-level content validity index.
Rating by six experts: items rated 3 or 4 on a 4-point relevance scale (1. irrelevant; 2. somewhat relevant; 3. relevant; and 4. highly relevant).

and

Table A3. Calculation of I-CVI and S-CVI for the initial self-audit tool: Audit of the application of lockout.

Item	Statement	Expert 1	Expert 2	Expert 3	Expert 4	Expert 5	Expert 6	Number in Agreement	I-CVI
1	The lockout procedure is easily accessible to the authorized employee (e.g., posted near the equipment, available on the intranet)	*	*	*	*	*	*	6	1.00
2	Authorized employees search for the lockout procedure and read its contents	*	*	*	*	*	*	6	1.00
3	Affected employees are notified before applying the lockout procedure	*	*	*	*	*	*	6	1.00
4	The authorized employees identify all hazardous energy sources of the equipment/machine to be locked out (as per the procedure)	*	*	*	*	*	*	6	1.00
5	The equipment/machine is shut down by using normal stopping procedures (e.g., putting a switch in the “off” position)	*	*	*	*	*	*	6	1.00
6	The equipment/machine is isolated from every energy source (e.g., close valves, switch off main disconnects, switch off circuit breakers)	*	*	*	*	*	*	6	1.00
7	The authorized employees apply hasps and their personal padlocks and information tags in accordance with the lockout procedure	*	*	*	*	*	*	6	1.00
8	All potential residual hazardous energies are relieved, disconnected, or restrained	*	*	*	*	*	*	6	1.00
9	The verification step (verification of isolation) is performed according to the established procedure to ensure that the equipment/machine cannot be operated	*	*	*	*	*	*	6	1.00
10	In the case of more than one authorized employee working on equipment, all employees affix their own (personal) padlocks according to the established procedure	*	*	*	*	*	*	6	1.00
11	In the case of personnel or shift change, the authorized employee follows the instruction to ensure the continuity of lockout, as mentioned in the procedure	*	*	*	*	*	*	6	1.00
12	Before lockout removal, authorized employees verify that the affected employees are safe and away from the equipment	*	*	*	*	*	*	6	1.00
13	Padlocks are only removed by the authorized employees who applied them	*	*	*	*	*	*	6	1.00
14	The equipment/machine is resupplied according to the established procedure	*	*	*	*	*	*	6	1.00
15	The authorized employees start the equipment and check that everything is working properly and that the work is finished (e.g., ensure that all work and interventions are completed)	*	*	*	*	*	*	6	1.00
16	All affected employees are notified of the completion of the intervention	*	*	*	*	*	*	6	1.00
17	The application of the procedure is recorded according to the established procedure	*	*	*	*	*	*	6	1.00
								S-CVI = Mean I-CVI = 1.00
I-CVI, item-level content validity index. S-CVI, scale-level content validity index.
Rating by six experts: items rated 3 or 4 on a 4-point relevance scale (1. irrelevant; 2. somewhat relevant; 3. relevant; and 4. highly relevant).

). Furthermore, the experts proposed that two items needed to be added to the audit stage: (i) statement 6 (about marking off the places (e.g., using warnings and signs) the places where work is carried out) in the de-energizing steps, and (ii) statement 18 (which is about inspecting the equipment/machine to ensure that it is ready to return to service) in the re-energizing steps, as shown in Table 5. Moreover, all of the experts (6/6) also found the tool to be clear and understandable.

After this first step of validation, the six organizations (shown in Table 3) tested the tool on actual lockout practices. Half of the organizations reported that they tested the tool on several equipment/tasks. One organization tested the tool on the equipment when using group lockout. The qualitative feedback was analyzed and categorized in terms of the content, the usability, applicability, completeness, and comprehensiveness of the self-audit tool. The content of the entire tool was clear and understandable to all of the organizations (6/6). All the six organizations found that the tool was easy to use and applicable. Four organizations (4/6) proposed adding several items to the tool.

Table 6. Main feedback and comments by the organizations on the self-audit tool in terms of content validity, and required modifications to the tool.

Aspect	Feedback and Comment	Required Modifications to the Tool
Content of the tool	One organization proposed for more examples to be provided for statements 11, 12, and 24	Notable examples were added to the related statements
Usability of the tool	No critical comments/feedback	No action needed
Comprehensiveness and completeness of the tool	Several items needed to be added to the tool: -In the pre-audit stage: (i) the integrity of the equipment (the lockout equipment and energy cut-off points are in good condition); (ii) the lockout station is in good order (cleanliness and presence of equipment) -In the de-energizing steps of the audit: appropriate steps to verify when applying a group lockout -In re-energizing steps of the audit: appropriate steps to verify when the equipment requires padlock removal and re-operation	Two checklist statements were added in the section “Required lockout hardware/material” of the pre-audit -Statement 10, 14, and 16 were added in the de-energizing steps of the audit -Statement 20 was added in the re-energizing steps of the audit

shows the main feedback collected from the organizations and the modifications required to address that feedback in the tool. In addition, according to the main feedback/comments, the required changes were made and the tool was updated (as shown in Table 4 and Table 5). Furthermore, since all the organizations (6/6) stated some experience with internal audits of lockout, they were asked to send their available audit tools. Four audit tools for the application of lockout were collected and analyzed by the research team. It was found that their tools were very simple (i.e., only a few questions were used to verify the application of lockout) and were not even in accordance with the Canadian standard and Quebec regulation.

4. Discussion

In this study, to develop the self-audit tool for the application of lockout, all the requirements of the relevant standards [15,16] and regulation in Quebec [18], and recommendations from the literature [28] were considered. Reese [40] stated that a compliance audit tool must be based on legal requirements and regulations.

In the proposed tool, the purpose behind the pre-audit stage was that an audit of the application of lockout would not be effective without the verification of the lockout procedures, equipment/machines, and training records of authorized employees. It should be completed before the audit of the application of lockout takes place. Grund [31] and Kelley [30] stated that—before auditing the application of lockout—a review of procedures, physical condition of equipment, and documents is necessary.

Bigelow and Robson [41] indicated that audit tools, based on an extensive review of accident causation and best practices in safety, may have some evidence of content validity. In testing audit tools for content validity, Robson, et al. [42] found that out of 17 safety audit methods, only five audit methods have been tested for content validity. The authors also demonstrated that the content of those five audit methods was incomplete, or only partially complete, in accordance with relevant safety standards and therefore content validity was very low in those methods [42]. Huang and Brubaker [12] showed that the content validity of the audit tool is important in order to have a reliable tool, for example, a high level of internal consistency and reliability. In the present study, the results showed that the tool had high content validity index scores in terms of both the content validity index for items (I-CVI) and content validity index for the tool (S-CVI). With regards to having excellent content validity, Polit and Beck [43] proposed that both I-CVI and S-CVI be calculated for the scale being judged. The authors recommended that a valid scale requires a minimum I-CVI of 78 (for 6–10 experts as explained by Lynn [35]) for each item and a minimum average S-CVI of 90 for the scale. Our results exceeded these expectations. Lynn [35] recommended two rounds of expert evaluations in the event there is the need to improve upon the items. In our study, the levels suggested for content validity were achieved in the first round. The experts confirmed the relevancy of all statements and also found that the tool was clear and understandable.

Moreover, all the organizations (6/6), specifically their internal auditors, had no difficulty understanding the content of the tool and using the tool on actual lockout practices, since the tool comprises clear and understandable content (checklist statements) and includes examples for some statements in order to reduce incorrect interpretation. They were also positive about using our tool and found it useful and applicable. This was expected, since their tools were incomplete, and did not cover all the requirements in the North American standards and Quebec regulation for the implementation of lockout.

Although this study demonstrated significant results with regards to content validity of the tool, it had some limitations. The primary limitation of this study was that inter-rater reliability (the consistency among auditors) of the tool was not tested. However, general instructions and a brief guide for each stage of the audit (i.e., pre-audit and audit) were prepared in the tool. Moreover, the checklist statements in the tool were clear and there was no need for interpretation. Reliability might be increased by adding some guidance in the audit tool [9], and the subjectivity could be minimized when the content is valid, understandable, and free from judgment and influence [44].

The application of alternative methods to lockout (for the control of hazardous energy) was excluded from this study. In future research, developing a self-auditing tool for the control of hazardous energy, where alternative methods to lockout are included could be developed. Karimi, et al. [45] and Poisson and Chinniah [33] showed that the application of alternative methods was not compliant with the relevant standards and regulations within organizations and they need to increase their knowledge about alternative methods. As such, tests for inter-rater reliability would be important in developing a self-audit tool for the application of alternative methods.

5. Conclusions

According to the literature, despite the importance of periodic audit/inspections of the application of lockout procedures in relevant standards and regulations, numerous organizations have no access to a valid or accurate self-audit tool for conducting audits of the application of lockout on machinery, i.e., without external auditors [22]. In this paper, a specific tool for the self-assessment of the application of lockout is presented. The content validity of the tool was tested via a panel of six experts and the results showed the highest validity index scores. The tool was tested by six organizations on different equipment. The feedback from them demonstrated that the tool covered most of the expectations of the organizations, and the content of the tool was clear and understandable for internal auditors who were knowledgeable about lockout. Furthermore, the usability and applicability of the tool was approved. In summary, the usability, applicability, and completeness and comprehensiveness of the self-audit tool were adequate. This self-audit tool will help organizations assess the application of lockout, find deviations and deficiencies, and take corrective actions related to their lockout program and procedures.