Experts assess the use of the first check lists

3 october 2018

Experts of the Project Management Office for the Reform of the Monitoring and Supervision Activities at the Analytical Center have completed the first phase of a study into the introduction of check lists into the state monitoring and supervision function. The results of the second phase are expected to become available by December 2018.

Check lists as an auditing tool began to be used in early 2018 by 8 monitoring and supervisory bodies (the Emergencies Response Ministry, the Healthcare Supervision Service, the Technical Supervision Service, the Agricultural Supervision Service, the Labor and Employment Service, the Transport Supervision Service, the Consumer Supervision Service).

The goal of the study was to find out ways to improve the existing system of check lists. The study analyzed the forms of the existing check lists as well as filled out check lists. The experts compared the check lists approved by the Project Management Committee of the Priority Program with those registered with the Ministry of Justice of the Russian Federation.

Analysis of the use of the check lists found that there is no standard practice for compiling the findings of audits conducted with the use of check lists. In most cases, audit reports never cite specific questions from a check list that allowed the auditors to identify violations, in other words, check lists are not being used as an auxiliary tool. As a result, there are often discrepancies between the violations mentioned in a check list and those recorded in the audit reports.

The experts found that some monitoring and supervisory bodies check adherence to mandatory requirements that are not included in any approved check lists, which is a gross violation of the law.

As a result for the analysis the experts made recommendations for how the monitoring and supervisory bodies can improve their check lists.

In addition, a sociological survey of inspectors and business representatives was conducted to find out how check lists were being used in practice. 6322 respondents from 8 monitoring and supervision bodies took part in it. Over a third of the respondents were from the EMERCOM (35.5%), while representatives of the Healthcare Supervision Service and Consumer Supervision Service showed the least interest in the survey with 0.8% and just 2 respondents respectively.

The survey found that more than half of the respondents had never conducted audits using check lists. 21.3% of the surveyed inspectors conducted at least one audit using check lists, 10.1% were involved in two audits that featured check lists.

Over a third of the respondents use one check list during audits while 14.08% use three and more check lists. Meanwhile, the majority of the respondents say the questions in the check lists are easy to understand or partially understandable with only 1.08% claiming they couldn't understand the questions at all.

As for the time that audits are taking, almost half the respondents said it takes them no more than one hour to fill out a check list. However, there is still a large number of those (17.19%) who spend more than 4 hours per audit, implying that the check lists they use are probably suboptimal.

With regards to using check lists for self-assessment and submitting the results to the supervisory bodies, opinions were divided. About half the respondents don't believe that self-assessment can reduce the number of scheduled audits, while the other half believes it to be possible as long as some additional conditions are met (such as additional random audits of those who did self-assessments).

At the moment, the vast majority of inspectors (98.33%) have to look at additional documents supplied by the audited organizations to answer a question on a check list. Meanwhile, more than half (59.86%) believe that on-site audits that use check lists take longer because essentially there is one more document that has to be filled out but only 5.15% believe that check lists have made on-site audits harder. Meanwhile, 13.35% of the respondents think check lists have made on-site audits simpler and easier to understand.

As for business representatives, three fourths of them never had to deal with audits that used check lists and one fourth experienced at least one such audit. Most of the respondents from the business side believe that check lists contain all or most of the important mandatory requirements but at the same time they also include excessive, overlapping and outdated requirements as well as some requirements that are not found in any applicable laws.

Almost half of the business representatives surveyed say they can easily understand the questions on the check lists and only 3.33% said they could not understand some of the questions. Meanwhile, most of the respondents (72.22%) believe that the questions on check lists should be worded in the most accessible and easy to understand form so they will not need additional clarification.

The vast majority of the surveyed entrepreneurs are willing to do self assessments (91.12%), especially if there was a convenient digital service for that (38.89%) if audits were less frequent (35.56%) and if check lists only included the basic self assessment requirements (16.67%). In total, half the respondents (50%) are willing to spend no more than one hour on self assessment while 17.78% say they'd be ok with spending a whole business day on it.

On the whole, a lot of the respondents both among the entrepreneurs and among the inspectors wrote in the free form answer section that paper check lists should be replaced with electronic ones.

Photo from open sources