Audit, Assessment, Inspection, Verification and Validation

I know that terminology can drive people crazy. We standards and certification types tend to use lots of it, and it can seem like a secret club if you don’t know what it all means. Good thing that I am here to pull back the curtain…

I have written in the past about audits, but I have not really said what an audit is. To do it justice, we should look at several terms that are used and often get confused. To do this I will take a run at the terms: audit, assessment, inspection, verification and validation. I know that there may be other terms, but usually they are usually a different term that refers substantially to one or a combination of these processes.

I should be clear that I am talking about business processes or management systems and only tangentially about products. There are other terms that are used for the work of testing laboratories - these are the folks that conduct physical tests of products to determine if a physical thing meets specific performance standards. For example, if a helmet is strong enough or if a lamp won’t catch fire. Some of the terms we will discuss in this post are used in similar ways for product testing as for management systems.

All of the four terms are important, and many organizations use some or all of them. That is to say that all these processes play a slightly different role. Just like an internal audit and a management review - maybe I should do a post about these also, but that will have to wait for now.

AUDIT

Audits are normally full evaluations that are conducted by an independent person (or team) and produce findings and a decision (sometimes call a determination). Strictly speaking, an audit evaluates performance against specific measures and thresholds. Findings are the results against each of the requirements, it could be a finding of conformity or non-conformity supported by a record of the evidence found. The decision or determination answers the question: Does the system or process that was audited conform to the requirements or does it not?

ASSESSMENT

Assessments are evaluations that determine the capacity of an organization (or a part of one) to perform a specific function. These are similar to the strict definition of an audit, but it the findings are based to a much larger degree on the experience and training of the assessor. Assessments also produce a finding but rather than being based on a numerical set of requirements it is largely based on the judgment of the assessor.

NOTE: Management system audits include many elements that fit the description of an audit and others that fit the description of an assessment.

INSPECTION

Inspections are evaluations that are conducted by either an independent inspector or by an internal inspector. Inspectors will normally identify weakness or failings to meet a set of requirements but will not produce a decision about overall conformity with a standard.

VERIFICATION

Verifications are evaluations that look a single process or service to determine if it meets its design specification. These can be done by an independent verifier or an internal verifier. Often verifications are undertaken with new processes or products to make sure that what was intended was achieved. Verifications are often done at the design stage to test if all factors have been fully considered. For example, a verification may be undertaken to determine if a batch of a product or an input can be fully traced through all the steps in production.

VALIDATION

Validations are evaluations that look at a single product or service to determine that it meets the need of the user. These can be done by an independent validator or an internal validator. Validations are often conducted once a process or product is up and running to make sure that the result is what was intended. To some degree these are related to quality control, but can extend to cover how the product or service is used in the real world.

As I mentioned above, these terms are used differently in some industries, so the important part is not the term itself, but the concept of what is being done.

For example, audit and assessment are interchangeable for some people as are verification and validation for others.

In the case of third-party certification, the key definition is that of audit. It includes a decision about whether or not there is conformity with a standard backed up by the evidence gathered.

All of these evaluations may be based on a standard, but the need for a formal decision of conformity is less important than in an audit and may not carry as much formal weight since there is no certificate being issued.

Finally, I will note that in some cases, some standard systems will hire an inspector to conduct a full inspection of a client and the decision of conformity is made by the standard system based on the findings of the inspector. In these cases, the certification decision is taken by the standard system, not by the inspector.

If we were to map these terms in a Venn diagram, they would all overlap to some degree or other. The point is that they all have a slightly different use and are all used to some degree or other in a whole range of businesses.

There, now it is clear as mud…right?

What happens to your standard when the auitor goes to work?

So, you are now finished with your standard and all the rules that an audit firm (let's call it a CAB - a Conformity Assessment Body) has to follow.

Before you press send and go live with your new system, maybe it is time to sit back and think through how all of your work will be used in the field. Your system will work best if it fits the needs of the users, the client, CAB, Accreditation Body (AB), stakeholders and anyone else that needs to use your system.

Today, let's consider what happens to your system of requirements when they are used by a CAB. In brief, your system is dis-assembled and re-assembled into a new set of procedures, forms, checklists and requirements that fit the way the CAB is organized... 

  • All the requirements that apply to the CAB, including process and timelines are separated from those that apply to the client. The CAB builds the requirements that it must follow in its own internal systems and procedures.

  • All the requirements that apply to the client are put into checklists and other audit tools that are used to audit clients.

  • The CAB's auditors are then trained to use these checklists and audit tools when auditing a client.

It is important to realize that auditors are not walking into an audit carrying copies of your standard, audit rules or other scheme documents, they are carrying materials that the CAB has developed based on your system requirements that are merged with the CAB’s own procedures. Their goal is to ensure that the audit can be carried out in the most efficient and effective manner possible, at least from their perspective.

CABs conducts audits following a standardized template. Most CABs will use ISO 19011 as the basic template and will add a few tweaks to make it their own. ISO 19011 lays out how an auditor conducts an audit. It includes such wonders as the outline for an opening and a closing meeting and other such innovations. The steps in ISO 19011 mean that the auditor is not just using your standard to evaluate the client, they have a number of steps that they should follow. These steps are necessary to ensure that the auditor does a thorough job and that client knows what is going on and what they have to do.

Before a CAB is allowed to conduct audits, its system must be fully reviewed to ensure that all of your requirements made it into the CAB’s re-assembled system. This is done by the AB and it is a key role that the AB assumes. Unless you have built your own expertise to do this job, you will need a competent AB to fully review the CAB’s system.

If this sounds complicated and a little confusing, it is because it is.

Your standard, scheme requirements, rules for auditors and such are most effective when your system is designed to be taken apart and put back together. It needs to be designed so that an AB can fully check the CABs incorporation of your system into theirs. It needs to take into account how auditors work in the field.

It also needs to take into account time…

The time that an auditor has on site with the client - so as you review your system keep in mind some key questions:

  • How many hours does an auditor have per day to ask questions?

  • How many questions can an auditor ask (and get answered) in an hour?

  • Does the auditor have to travel from one site to another and how much time does that take?

  • Auditors need to eat lunch, so allow time for that.

  • Remember that the auditor has to include an opening and closing meeting with the client to explain what will happen in the audit and what the audit has found.

The time that it takes for each step in the process:

  • For your system to be fully developed;

  • For the CAB to incorporate your system into its own;

  • For the AB to assess the CAB and make sure that your system is fully incorporated; and

  • For the CAB to plan, execute and take a decision on any single client.

What happens during an audit is often as important (if not more important) than the specifics of what is being audited. I do not mean to suggest that what is in the standard is not important, but how the audit is conducted should be considered when your standard, requirements and other rules are developed.

Saving the World, one Checklist at a Time

One day a few years back I heard about some producer who wanted to do a reality TV show about environmentalists. They wanted real life environmenatlists; you know, the people who spend all their time chained to bulldozers to resist pipelines, swinging from construction cranes to hang banners or rescuing whales that are caught in fishing gear. This 'reality' TV idea came up while I was visiting a local environmental organization. Once the concept was explained the room was silent, then someone in the back of the room said: 'They want to film us sitting through endless meetings and while we read multi-volume reports?' Everyone started to laugh.

Needless to say, the show was never produced. 

Just like many other jobs, working in the world of standards, accreditation and certification can be dull. We create standards, guidelines and principles; then we turn them into checklists. With a blank checklist in hand we go through stacks of policies, procedures, practices and all kinds of other files; then we interview people and ask them all the same questions just to fill in the checklist and see if everything has been done and done right.

The work in standard systems is not going to get you a starring role in the hottest new reality show, but it is work that keeps us moving forward. Not just so that things continue to work, but to make them work better. We know that each application of the standard can help save an ecosystem, keep water clean, make sure workers are well treated, keep children in school, strengthen a community and so make the world a better place. 

Transforming markets takes dogged perseverance. It takes work over years to create innovative standards, conduct high quality accreditation, certification and auditing. It takes a hell of a lot of checklists.

So every morning when I start my day, I tell myself that I am saving the world, one checklist at a time.

You are too.

 

Accrediation - What is it & Why bother?

 

accreditation: third-party attestation related to a conformity assessment body conveying formal demonstration of its competence to carry out specific conformity assessment tasks (3.1, ISO 17011:2004)
 

Accreditation is sometime described as 'auditing the auditors'...while that is not really correct it does hint at what an accreditation body (AB) does.

Accreditation is better described as a 'competency assessment' rather than an audit. You may feel that I am splitting hairs, stay with me for a few more paragraphs and I'll lay out why that is important.

First of all, let's focus on what competency means. Some folks think that a qualification to do a job is based on your education; in my case, I spent 2 years earing a masters degree in regional planning and a few more years qualifying for membership in a professional planning organization. As a result, I can call myself a 'Registered Professional Planner'.  I am a qualified planner!  In the twenty-eight years since I completed my degree in planning I have never worked as a land use planner or as a regional planner.  As a result, while I am qualified as a professional planner, I submit that I am no longer competent. Yes, I have the letters after my name and I use many of the skills that I learned in planning, but the degree and the designation does not equate to competency.

Competency is determined by demonstrating that a task or set of tasks can be done to a specified level of performance. In essence, it means the ability to do a job.

To assess the competency of a conformity assessment body (CAB) the AB looks at the systems, process and procedures used as well as how they are executed. Accreditation is granted to a CAB that demonstrates its competency in conducting conformity assessment work. In this context, competency is not a low bar to meet, it is a high bar. 

The areas that an AB would examine include a wide range of elements such as:

  • maintaining professional relationships with clients;
  • ensuring that all the staff of a CAB are competent to do their jobs;
  • the frequency and quality of the training and evaluation of auditors; 
  • planning and executing audits;
  • ensuring that non-conformities are addressed;
  • taking certification decisions; 
  • conducting regular management reviews and internal audits of CAB operations;
  • and a host of other elements.

Conducting an accreditation assessment is a specialized business, it is not just determining if the results of audits against your standard are producing the results you want. 

 

Who does what? - An Assurance Ecosystem

This blog focuses on the role of the scheme owner. But, let’s be clear that while the scheme owner is a central player, there are others that have crucial roles. Previous posts have included some elements related to standards development with some references to auditing. In this post I wanted to look at the big picture of how assurance is managed, in what I think of as an ‘assurance ecosystem’.

Before going any further, I want to be clear that while all well run standard systems include all the elements discussed below, who does what and how oversight is conducted can vary greatly. This can be due to the objectives of the scheme owner, the nature of the industry or in response to the way an industry sector or key resource is managed internationally or in specific countries.

A fully functioning assurance ecosystem includes several players, each with their own responsibilities. The diagram below is a highly simplified map of the players and the primary relationships of each.

Assurance Ecosystem

Kitbag - An Assurance Ecosytem - Page 1.jpeg

 

 For this post, as with the all the posts on this blog, the central player in our assurance ecosystem is the scheme owner.

In most cases the scheme owner is responsible for:

  • the standard itself, including its development, maintenance and interpretation;
  • the rules for the assurance system including the documented policies, processes and procedures that govern its application; and,
  • the day to day management of scheme operations including oversight, monitoring and evaluation, risk management, and business operations such as marketing, finance, personnel, etc.

Some schemes delegate some of these responsibilities to other organizations. However, the scheme owner is always ultimately responsible for the scheme. For example: 

  • In the case of organizations such as MSC and RSPO, they act as scheme owner and they have assumed all these three core responsibilities.
  • In the case of FSC, it established a global set of ‘Principles and Criteria’ that standards must address and works with affiliated national and regional organizations who assume the role of a standards development organization (SDO). FSC, the scheme owner, endorses those standards that conform to its system requirements.
  • In the case of ASC, the standard was developed by a group of semi-autonomous bodies, the aquaculture dialogues that were set up by WWF, and the completed standards were given to ASC who now undertakes all the responsibilities of the scheme owner.

The other three players on the assurance ecosystem map (AB, CAB and client) have their own roles to play.

The accreditation body (AB) has the job of providing "third-party attestation related to a conformity assessment body conveying formal demonstration of its competence to carry out specific conformity assessment tasks (ISO 17011:2004 3.1)". In plain English, its job is to make sure that the CAB has the competence and capacity to carry out audits for a scheme. For some schemes, the accreditation body works in a single country, such as DAkkX in Germany or ANSI in the US, or the accreditation body works internationally, such as IOAS, SAAS or ASI.

The conformity assessment body (CAB) - sometimes referred to as the 'certification body' or the auditor - has the job of carrying out audits against the scheme owner's standard following the policies, practices and procedure specified by the scheme owner. CABs can range from large multinational organisations such as Bureau Veritas, SGS or TÜV to small organisations with only a few staff working in single country or region.

The client, that is the applicant for certification or, when certified, the certificate holder, has the job of making sure that the product, process or service conforms to the standard.

The assurance ecosystem is almost never a single organization that does everything. As a result, the task of assuring that each certificate means that the same minimum performance level has been achieved for each certificate is a challenge that requires skills of facilitation, negotiation, management, collaboration and communication.

In brief, the challenge of being a scheme owner includes the skill and capacity to partner with specialist organization in a range of roles and responsibilities in a way that respects the the scheme owner and their specialist roles so that the whole ecosystem delivers consistently high quality.

 

Planning for the future

There is a popular quote from Wayne Gretzky. He is reported to have said something like: "I skate to where the puck is going, not where it has been.” 

As cheesy as the quote may be, it draws attention to a fundamental problem that decision makers face - how to position yourself to take advantage of something that is going to happen rather than try to go after the rewards of something that is happening now or has already happened.

My cynical mind sees populist leaders of our age expending tremendous amounts of energy to recapture the glory of a bygone era, whether that is Trump seeking to recreate the hegemony of the US in the 1950s, Putin seeking to recapture the glory of the Byzantine Empire, or the Brexiteers seeking to reclaim the primacy of the British Empire, they all seem to me to be ‘skating to where the puck was…’

Business leaders can get caught in the same trap, that is focussing their resources on a recent trend only to be faced with all their competition going after the same, now diminishing opportunity.

This may not seem like it has much to do with certification; but it does, stay with me.

Certification systems face the pressing demand to solve the problems that they face now. This pressure can be overwhelming and can mean that leaders can be caught in the trap of focussing on where the puck is now, without devoting time and energy to where it will be. Limited resources can also truncate the capacity of leaders to think about the future.

The visionaries that were behind the leading social and environmental standards were thinking about the future and how to bend it toward creating a more just and greener world. They looked at long-term trends and tried to figure out how to take advantage of them.

While leaders in certification systems have to devote large amounts of energy into solving today’s problems, they also need to spend time thinking about were the world is going and how they can get out in front of the changes that are now developing.

All this comes down to the fact that certification systems are constantly changing and taking advantage of the opportunity to change can give us the opportunity leverage to broad changes that are occurring and are likely to occur.

These can include:

  • Growth in markets and trade in developing countries
  • The ever-increasing rate of technological change
  • The globalization of markets, especially capital markets
  • Market pressure from activist consumers
  • The growth of the size and importance of global megacities
  • The growing inability of national governments to control markets

I would not counsel adopting my list, but encouage looking to the research being done on future trends. Study the reports, evaluate the trends and ask the question: How could this impact my standard system?

Even if we just consider the growth in global trade and especially the growth in south-south trade, certification systems will need to adapt to a new shape in global markets. Most certification systems are designed to manage north-north and south to north trade. As a result, their efforts to develop markets for certified products in Europe, North America and Japan will not encourage growth in the rapidly growing market share in the developing world. Will your system be positioned to grow in Indonesia, Kenya and Brazil? Will you be able to deliver markets to your certificate holders for those that produce products in Africa and sell into Southeast Asia?
 

How to Build an agreement

Core to any standard system is the need to build a consensus, that is to reach agreements that can be supported by all your key stakeholders.

This sentence is self-evident...and impossible at the same time.

It is self-evident because agreements that have the support of all your key groups of stakeholders are the most durable and form a solid foundation for your system. It is impossible because getting that agreement on everything from everyone is, well, impossible.

Multiparty negotiation is one of the most challenging elements of building a consensus standard. It is necessary to get agreement between a wide range of interests, many of which are contradictory. (For more on what a consensus means see my earlier post: Who Can Write a Standard?)

In my experience multiparty negotiations follow a consistent pattern:

  1. Very quickly, the group can reach an agreement on close to 90% of the issues. This is the case because, despite our differences we all agree on wide common base (I know this does not feel like the case but it is really true!).
  2. Getting agreement on the next 7% of the issues will take a lot of work, trust building and creative thinking but it can be done.
  3. The last 3% is the killer. These are the issues where the most fundamental disagreements reside.

As I state in the title, agreements are built. The popular myths are either that the agreement is a compromise, that is all sides must ‘lose’ something or one side just gets its way at the expense of all the others. While these happen sometimes, the best agreements are built from innovative approaches to old problems. This is important because agreements we can actively support are more durable than those we can just accept.

Back in the early days of FSC, the original Principles and Criteria had a text of principle 9 – it dealt with the issues of ‘old growth forest’ and the parties could not agree to a text. After many tries to get an agreement a proposal was made to create a committee of representatives from the three FSC chambers  who would be charged to develop an agreed text for principle 9 (these included environmental, social and economic representatives from both the developed and developing countries).

To create the group, each of the chambers elected their developed and developing world representatives. To incentivise the group to reach a decsion, the FSC board which approved the process was clear, if this group did not solve the problem, the board would. This last point helped because very few folks liked any proposal that the board had produced to date.

When convened, the group started off with the all the expected concerns raised, the foresters from the economic chamber talked about ‘over-mature trees’, the environmentalist talks about the value of ‘old growth forests’, and the social chamber representatives talked about the need for sustainable employment and preservation of cultural sites in forest landscapes.

About halfway through the time we had together, a new idea emerged. It was proposed that we talk about ‘high conservation values’. That is, the concerns of each group could be mixed up in different ways. The question ceased to be “Is this an old growth forest?” and became “What are the high conservation values for this forest?”

Because of this change in approach a text of principle 9 was proposed that required protection of high conservation values. The text was clear that high conservation values could be found in just about any forest, from plantations to virgin forests and everything in between.

For FSC, the issue of principle 9 was the killer 3%. It took years to build an agreement, but it happened because a new idea was brought into the room, it was explored, and an agreement was built. The solution worked better for everyone’s concerns than any other. When the meeting ended, no one left feeling like they had lost.

A creative solution that allows the parties to think about an issue in a different manner is often the route to building agreements.

Writing Code for Humans

My father used to say: “If all else fails, read the instructions!” I think that this is an accurate way to describe how people function. The first thing we do when we open a box is to toss the instructions aside and try to figure out how to use our new toy. We will puzzle through a problem, press buttons on the new camera, and assemble the IKEA cabinet our own way before we find out that that we wasted time, did it wrong, or worse, broke it.

This natural tendency is at odds with how standards, audits and such are designed and implemented. Tossing the certification instructions aside is not going to ensure that your implementation of a complex system is going to be successful.

Standards, audits, certification, labelling, and other bits that are part of a certification scheme are detail oriented and require clear, step-by-step instructions and guidance for users. 

It really helps if the stuff is clear, easy to read and works with the way humans work. But, let’s be honest, most scheme documentation is confusing, not well organized and hard to follow.

We often think of instructions as a computer program in which each step, no matter how trivial must be written and placed in its proper order in the sequence. Humans, thankfully, are not computers and we are a bit more flexible than computers. With that said, clear instructions designed for humans do help and when properly prepared it is more likely they will be followed.

So how do you write code for humans? Here are a few things to keep in mind:

1.       Understand what your user needs

Find out who is going to use your instructions and guidance; write for them. If it is likely to be an intern or junior staffer, keep that in mind. Your writing may be their first real introduction to standards and certification.

2.       Use words and graphics

Some people read the words and ignore the graphics, others read the graphics and just look at the words as if they are footnotes to the graphics. Whenever you can, give presentations that both can read.

3.       Write in the positive

It is very hard to follow instructions that tell you what not to do. Instructions are so much clearer when you keep this in mind.

4.       Tell them how their work will be evaluated or audited.

Give a clear description of how their work will be used. If it is to be audited, explain the steps in how that works.

5.       Don’t think they are stupid or that they can read your mind.

Write for readers that are smart and want to learn, that is why they are reading your work. At the same time, be aware that what is ‘obvious’ to you may not be to others, so include all the steps in your descriptions.