Becoming more efficient....and more rigorous.

From the moment your scheme is up and running you will be under pressure to be more efficient.

This pressure will take many forms. Mostly, this pressure will be from folks that will want you to take a range of steps to reduce the cost of being certified.

• Some folks will want you to be more targeted in your audits. That is audit the important stuff and stop checking the stuff that they feel does not matter.
• Some folks will want you to conduct fewer audits by reducing the number of surveillance audits.
• Some folks will want you to trust their own internal audits or the audits of other folks as proof of compliance.

In addition to this list you will hear a thousand other ideas for increasing the efficiency of your scheme.

While you are being pressured to become more efficient you will also be pressured to be more rigorous. These voices will call for:

• More frequent and more rigorous audits.
• Stronger sanctions for those found to not be in compliance.
• Public shaming of weak performers.
• Rapidly increasing levels of performance for certificate holders.

Please note that no matter how many of these steps you take both groups will want more. For a number of companies, the best cost is zero. For a number of stakeholders the only assurance level that is acceptable is infinite.

The frustrating part of these two pressures is that both of these groups have a point. In truth, efficiency and rigor are not always at odds.

The central question is how to deliver the greatest assurance at the lowest price. This can be done by first of all working to figure out what your certification means. Are you clear about the market signal that your certification carries. Are you sure about the level of assurance that you need to clearly and confidently assure users that your scheme is doing what you promised you would do?

To address the challenges you could consider the following:

1. Do your certificate holders and other stakeholders understand what you are tying to do? Is the level of assurance that you are providing what they want and need?
2. Are you asking for too much in an audit? Are you just piling on more and more requirements in the belief that it will make your scheme more rigorous?
3. Have you carefully studied your scheme to find out if the requirements are really related to the assurance you are providing?
4. Are you scaling back on requirements in the hope of lowering costs and growing your market share?
5. Are there redundant requirements in your scheme? That is, do you require auditors to check multiple times for the same thing; or do you check for several requirements that are all found to either be in conformity or non-conformity 99% of the time?
6. Are you managing your scheme to grow your market share without considering advancements in science or other research that means you should make changes in your requirements?
7. Are you sensitive to the cost of your scheme for users? 
8. Are you exploring ways to use technology to improve confidence in your certification while reducing costs?

The real challenge in all this is that your scheme will never be static. It will always be under review and you should be constantly looking in every corner to find ways to reduce costs as well as enhance confidence in your scheme. 

Most of the changes you will make will be incremental, that is the change in cost or confidence will be minimal. But taken together you will be able to offer constant improvements to the demands of both groups.

Certification schemes live and die in the market. That means both the supply and demand side. To survive and grow you will need to constantly make your scheme more cost efficient and offer higher and higher levels of assurance. 

Most significantly, if you are constantly working to improve all aspects of your scheme you will be around for a long time.

None of it is easy, but for us certification geeks it sure is interesting.

Think like your clients...Why do they want to be certified?

So now your scheme is up an running. New clients are happily signing up for audits and certificates are being issued.

Do you really understand why they are going to all this trouble?

Sometimes a new idea, technology or product is conceived one way by the producer but used in a completely different way by the user. Understanding how your scheme is actually being used may be an eye-opener. 

It is true that in many cases a certification scheme is adopted because someone further down the value chain requires it. Let's say I am growing blueberries. My client comes and tells me that I have to be certified to the 'Green Blueberry' standard or my client will not buy my berries. Well that is fair enough, clients specify what they want and I either meet their requirements or find a new client.

So lets follow the value chain and ask each the folks at each link why they are specifying your new blueberry certification. Maybe the packer is specifying it because she feels it will increase her market share; she has access to certified berries and the market is demanding them. She may or may not be committed to your philosophy on blueberries but she is motivated to do what she has to do to get and stay certified.

Further down the chain you find a bakery that produces blueberry pies for sale in grocery stores. He may be motivated to add one more reason for his clients to continue to buy from him. His clients are clear to him that they want certified berry pies. For the baker it is one more way he can negotiate longer term contracts to supply pies to stores. His clients are less likely to shift suppliers just for price advantage if he can add the certification that adds value to his product.

Now we are at the grocery store, they want certified berries (and pies) because their customers want to feel they are getting healthy food that is not produced in a way that harms the environment or exploits workers. The grocery store is building a relationship with their customers so that they will choose their store over the completion. Certified blueberry pies are one more way they can do this.

While in this supply chain all the participants are happy, it may not always be the case.

One element of supply chain certification is that you can use the list of certified producers to skip over some folks in the supply chain. The baker for example could find a certified farm to buy from directly, skipping the packer completely. This could be better for the farmer and the baker but the packer looses business. The farmer could sell his berries for a little more and the baker could buy them for a little less than the packer charged and they both could come out ahead.

Disruption in supply chains is a common impact of certification, especially in long or complex supply chains. If the baker needed to source his berries from another country it can be expensive and time consuming to hunt up a supplier. But thanks to an online list of certified producers the job becomes much easier. The baker can purchase certified berries from halfway around the world, directly from a certified producer without having to buy through a broker or wholesaler.

In short, certification is good for those in the supply chain that can take advantage of the opportunities that it presents, it also can harm (or even put out of business) those packers, processors, brokers or others in the supply chain that certification may disadvantage.

Statistics, sampling and other mysteries of the universe....

One challenge of a scheme owner is to determine the most appropriate balance between the intensity of the audit and cost. In short, how many examples must an auditor check to ensure that there is conformity? Keep in mind that every additional thing an auditor is required to do will cost the client more (and therefor everyone else in the value chain).

To be blunt, finding a non-conformity is a bit like looking for your lost keys...once you find them you can stop looking. The only difference is that the auditor has no knowledge of how many 'keys' he is looking for.

If the auditor is auditing a large forestry operation and she finds a non-conformity at the first harvest site she will continue to check the rest of the sites to be audited to determine whether or not the non-conformity she found is a single error or if it occurs at each site. One feller buncher operating too close to a stream may be a minor issue (i.e. an operator that is having a bad day) but all feller bunchers operating too close to many streams can be a major issue (i.e. the company told them to do it).

OK, that sounds simple but how many sites should the auditor select for inspection in the first place? Should the auditor inspect all sites, half of them, or just one? 

The "Square Root Rule"

It is a common practice for auditors to choose the number of sites based on the "square root rule"; this simply means that the number of sites to be audited is the same as the square root of the total number of possible sites (NOTE that this also applies to the number of files, employees to interview or other sets of things to be checked). If there are 16 sites, for example, then 4 should be checked.  If the square root does not give you a whole number then round up the number of sites to the nearest home number. If there are 10 sites (the square root of 10 is 3.16...) the number of sites to be checked should be still be 4.

This is all well and good and the square root rule is easy to follow and easy to calculate (just about every calculator has a built in square root function). The square root rule also gives you a small number of sites to check so that the cost of an audit can be kept low (fewer sites to check means less auditor time equals a lower audit fee). But, what about the reliability of the sample?

Using Statistics to Determine Sample Size

If we change hats and look at the question of sample size using the mind of the statistician we may wish to question the reliability of the square root rule.

If your scheme decides that you want to have a sample that gives you, for example, 95% confidence, plus or minus 5%, then your sample requirement will look much different.

As you can see, the number of sites necessary to audit to achieve the statistical confidence that is like that of academic research is far greater than applying the square root rule. The smaller the sample set the less statistical confidence that can be obtained.

To calculate the number of sites, files, employees or other things to audit for a specific number there are many online tools to do it for you (i.e. http://www.surveysystem.com/sscalc.htm).

The Real Issue

The primary question that you as the scheme owner should consider is: "What level of certainty do I need?"

The answer for this is in understanding the industry sectors that your scheme involves. What do they need? What do their customers need? What do the users of the products and services that they produce need?

What is the trade-off between cost and accuracy - i.e. how much is enough?

It is important to understand what happens in sample selection, both how the sample size is determined and how that number of samples is selected. More is not always better. One way to look a is is to see the best as meeting the needs of your users both in terms of accuracy and cost.

Never ask a question to which you don't know the answer....

The first rule of auditing is really quite simple: Do not ask a question to which you do not know the answer. To get to why, let's first look at the purpose and some methods of auditing.

The purpose of an audit

When developing your scheme it is important to know what happens in an audit - this knowledge will form the basis for a well designed and effective scheme. 

First of all a quick reminder of what audits are not... they are not research, investigation or enforcement. 

An audit is an examination of the system, product or service to determine conformity or nonconformity with stated requirements. To undertake the audit an auditor approaches an audit with the expectation of conformity, after all a client would not reasonable hire a conformity assessment body to conduct an audit if they did not feel that they are already in conformity with the scheme requirements. So, an audit is conducted with the sole purpose of auditing the client's explicit or implied assertion that the system, product or service being audited is in full conformity with the standard. Auditors make determinations of conformity or non-conformity based on the evidence observed in the audit.

Auditors do not punish, conduct research or launch investigations.

Conducting an audit

So...back to the first rule of auditing: Do not ask a question to which you do not know the answer.

An auditor does her job by examining evidence provided by the client with an eye to determine if the evidence demonstrates conformity. The evidence that is evaluated can include documents, interviews,  and visual inspections.

In the audit, the auditor should record the evidence evaluated and note when it is in conformity and when it is not.

When asking questions, the auditor should always ask open-ended questions, ones that do not give an expected answer to the person being asked. (i.e. do not ask "You don't put that in this box, do you?" instead ask "What do you do next?") Auditors should listen to the responses and ask further questions to probe further as needed. 

The auditor should ask for objective evidence to support the answers. (i.e. "Why do you do that?")

What this means for your scheme.

As you work on your scheme, carefully consider what the requirements are that apply to the client. Are your requirements designed to be assessed for conformity by the auditor - or are your asking the auditor to play another role. 

Auditors do conformity assessment only. The requirements that your scheme lays out should be clear to the client and easy (as much as possible) for the auditor to determine, based on objective evidence, whether or not the client conforms.

Who can write a standard?

Almost a year ago I posted an entry titled "What is a Standard?".  In that post I included the ISO definition of a standard:

ISO defines a standard as:  A document established by consensus and approved by a recognized body that provides for common and repeated use, rules, guidelines or characteristics for activities or their results, aimed at the achievement of the optimum degree of order in a given context.  (from ISO/IEC Guide 2:1996, definition 3.2)

This definition includes a couple of bits that can be confusing. It says that a standard is "...established by consensus and approved by a recognized body..."

So, what is meant by "consensus" and "recognized body"?

Consensus

Many times the idea of reaching a decision by consensus is rejected because we understand that it means that everyone must fully agree with everything in a decision. This can be problematic for long and complex documents like standards. If I agree with everything in a standard except one line can I withhold my agreement and prevent approval of the standard?

In most standards organizations that I know of the definition of consensus is 'the absence of sustained opposition'. That means that my opposition may be minor but not something that is critical to my decision. I may want my opposition logged in the decision but it may not be worth the cost of renegotiating the whole document to get what I want. If that is the case then a consensus can be reach, even if there is some opposition, even by several participants.

If on the other hand my opposition is to a part of the standard that presents a major concern to me I can raise my objection and potentially withhold consensus.

Some standards developers set the definition of consensus as a super majority of 75, 85 or 90%. In these cases agreement of the super majority is understood to be a consensus.

A consensus under either of these approaches can be hard to achieve because some participants may be particularly strident in their positions or others can play a game with the voting rules to get what they want over the strong objections of others.

No matter which model is adopted it is important to design the rules for decisions in such a way that everyone has a say and that no sector on the decision making body can dictate a decision to others.

This can occur if a decision making body has 50 members and only 2 represent workers and 48 represent employers. If the rule requires a super majority of 90% it is still possible for all the employer representatives to out vote the labour representatives even if their opposition is very strong.

In the case of consensus defined as the absence of sustained opposition it is possible to give such latitude to the chair or an executive committee that they are able to declare a consensus that favours their position by announcing that the opposition is not sustained no matter what the opposition itself wants. 

No matter which system is used it is important the the underlying principle of consensus be respected by the rules and the participants. Everyone should be willing to support the decision even if they disagree with some elements in the standard.

Recognized body

The issue of a recognized body can be a bit more problematic. In some countries this could be defined in legislation, that is there may be only one 'recognized body' in the country or there may be a defined procedure that an organization must follow to become a recognized body.

Some people may even read the ISO definition as declaring that only ISO and bodies that ISO recognizes may develop standards. I will note that if this was the case that there is no way that ISO has the authority to enforce such a provision. ISO is a not-for-profit corporation based in Switzerland and while national governments may be members the organization has no authority to speak for governments. ISO produces standards that are voluntary. Even if a country adopts an ISO standard as a regulation (yes this does happen) this is the result of a decision by a government, not by ISO or any other standards development organization.

I will propose what I think is a better definition of a recognized body - that is a body that conforms to established norms for standards developers. This includes codes of practice like Annex 3 to the World Trade Organization (WTO) 'Code of good practice for the preparation, adoption and application of standards'; ISO Guide 59 'Code of good practice for standarization' and the ISEAL Alliance 'Standard Setting Code'. There are other guides and norms, some are national and some are specific to an industry (such as the FAO guidelines for certification of wild or farmed seafood).

Anyone developing a standard should understand the guidelines and norms that may apply to them and do their best to conform to them. This is the hallmark of a professional organization.

Claims

One of the common complaints about certification is that users do not always understand what is being certified. This may be caused by a certification claim not lining up with scheme requirements (sometimes accidentally and other times intentionally) or with users not being able to easily understand what a certification mark or claim means.

Sometimes a certification is applied to a single product, and a company may represent the certification as applying to the whole company - or the other way around. If for example, you see a product sold with an ISO 9001 certification claim on the box it can seem that the product is certified. This is not the case since ISO 9001 is a management system certification that applies to the company but does not evaluate individual products. In this case it is the company that is certified, not the products. Using the cetification, the company can make claims about how it is run but not about individual products

The reverse can apply if a company claims that it is certified, for example, to MSC (Marine Stewardship Council). The MSC scheme is designed to certify wild-capture seafood. In this case the source of the seafood is certified, the company is not certified. An MSC certificate means that the company can make claims about products that are or contain MSC certified seafood but not about the how the company is run.

Claims can take many forms, they can be:

• A logo or mark applied to products; 
• A logo that is to be linked to a corporate name;
• A tagline that is printed on a product or used in advertisements;
• A tagline that is associated with a corporate name; or
• Any combination of the above.

Claims can be easily exaggerated - just look a claims made about the healthiness of foods and you can easily see how they can get out of hand.

The challenge in designing a claim that a certificate holder can make about a certification is to make sure that the claim is easily understood and accurate. Currently the ISEAL Alliance is working on developing a Good Practice Guide for Sustainability Claims, it you are working on claims for your scheme this draft will help with your thinking.

One advantage of certification is that the claim can be controlled by the scheme owner through a licencing agreement. The way in which the certification is marketed and a logo is used can be controlled by the signed agreement and enforced by the owner of the scheme.

It should always be the responsibility of the scheme owner to set out the rules about what claims can be made and how a scheme logo may be put on products or corporate mastheads.

The claims that you allow for your scheme should be accurate, clear and easily understood by the user. 

Finally, schemes should always make sure that their rules are being followed and take action when they are not. Your reputation is all that you have and when your scheme is misrepresented it undermines the value of your certifications, trademarks and claims as well as reducing the value of your scheme to users that follow the rules.

Transparency is being generous

The ISEAL Credibility Principles explain that for a certification scheme transparency means:

"Standards systems make relevant information freely available about the development and content of the standard, how the system is governed, who is evaluated and under what process, impact information and the various ways in which stakeholders can engage."

Before a scheme can be transparent the scheme owner must first have documented the key information about how the scheme is run. Having clear documentation is the first challenge because it takes time and resources to write it. In addition, this documentation should be written in clear, simple language that is easy for your stakeholders to understand and use.

Why bother with transparency?

Certification is a trust business. Certificate holders and all those in the value chain rely on your certification to ensure that the products they produce or trade conform to the standard. They rely on the assurance that the scheme is providing to assure their customers and manage their own risk. Consumers and other end users rely on the certification in their specifications and in making purchasing decisions.

Transparency is a means for scheme owners to assure all those that rely on the scheme that they are living up to their obligations in managing and operating the scheme. It is a means of demonstrating to users that the scheme is trustworthy, and as a result there can be confidence in the certification. Also, users, should they choose may examine the scheme to assure themselves that it is meeting their needs.

Transparency means that information is available to anyone that wants it, including governments, users, researchers, advocates, and anyone that is interested (including high school students writing assigned essays).

A minimum list for sharing with the public

The first step to facilitate transparency is to have a package of material available on the web for anyone to download. At a minimum this package should include:

• A procedure that describes how the standard is developed, revised and interpreted. This should include a complete description of the processes including how any interested party can participate, how decisions are made and who makes the decisions. 
• A description of the purpose of the standard and its scope; that is what it is intended to accomplish and when it is applicable.
• An explanation of the governance of the scheme, including the scheme owner's board of directors (listing the members of the board, their terms of office and how they are appointed) and any other technical or advisory boards that play a role in the governance of the scheme.
• A description of the monitoring and evaluation program that is used by the scheme owner to evaluate the quality, consistency, efficiency and other aspects of the performance and operation of the scheme.
• A description of how the scheme evaluates its impacts; that is are the objectives of the scheme being achieved including the procedures used in making the evaluation as well as how conclusions are reached. It is best if in addition to this description that regular reports on performance are published.

• A description of how stakeholders can become involved, this should include involvement in certification audits, standards development, governance and most importantly in commenting, raising a concern or making a formal complaint about a certificate, a certification body or the standard (or any other aspect of the scheme).

In addition to these documented procedures, policies and reports each scheme owner should consider what other information should be made freely available to the public. As a general rule, my advice to scheme owners is to default, wherever possible to being transparent. In other words don't just post the minimum information; only keep confidential the material that is absolutely necessary and make all the rest freely available. 

A guiding principle is to be a generous as possible when it comes to being transparent. While it may mean that more questions come, it also is an opportunity to build trust with key individuals, organizations and institutions. Obviously there are limitations in terms of staff time and available resources but the investment will pay off.

What is a major non-conformity?

In simple terms, a non-conformity is when a applicant or certificate holder (the client) fails to conform to a requirement for certification. All of the requirements that apply to the client should be written in the standard.

Grading non-conformities

One of the responsibilities of the auditor is to grade a non-conformity; that is to determine whether or not it is really serious or a small matter. Most schemes use the terms major and minor to grade non-conformities. Other schemes use terms such as critical to describe non-conformities that are so serious that it means the client automatically fails the audit.

There are two approaches that a scheme may uses to determine how to grade a non-conformity. And just to complicate things there are hybrid approaches that some schemes use which combine elements of these two approaches.

The first approach, and generally the most common, is for the auditor to take a decision about how serious the non-conformity is based on a set of criteria. An example of the criteria used to grade a non-conformity are as follows:

Major Non-conformity    

A Major non-conformity is normally raised when one or more of the following are found:

• The absence or total failure to meet a requirement and the failure has or is likely to result in the client failing to achieve the objective of a requirement.
• The non-conformity is highly likely to result in a breakdown of an requirement or materially reduce the ability to achieve the objective of a requirement.
• A minor non-conformity that is shown to continue over a specified period of time or occurs repeatedly.

Minor Non-conformity    

A minor non-conformity is usually raised when the client does not meet the requirement and the non-conformity does not jeopardize the integrity of the scheme. This includes one or more of the following:

• Where there is a failure to comply with a requirement is not likely to result in the breakdown of a system to meet a requirement and will not risk the integrity of the products or services that are being certified by the scheme. 
• Where the failure  does not meet the definition of a major non-conformity.

In brief an auditor must determine if the products or services that are being certified are likely or not meet the objective of the scheme. An example is for a scheme that requires that daily records be kept. If the auditor finds only a few records on file the non-conformity is likely to be graded as a major. If only a few days of records are missing, the non-conformity is likely to be graded as a minor.

In the second approach, the scheme determines in the standard how a non-conformity for each requirement is to be graded. 

In these schemes the standard is written so that each requirement is graded and for any non-conformity, no matter how serious is graded as specified in the standard. Schemes that use this approach sometimes use the category of 'critical' for some requirements.  A failure of a critical requirement means an automatic failure of the audit. For example for a non-conformity raised against a requirement that daily records be kept which is graded in the standard as a major in the standard would be graded as a major non-conformity even if only one day's records are missing.

For both of these approaches, an auditor may raise an observation for cases when the auditor finds cases where a non-conformity is likely to occur. Observations are intended to be helpful to the client so that they can address problems before a non-conformity occurs.

A Hybrid approach

Some schemes may wish to grade some requirements in their standard as critical or major, and leave the remaining requirements un-graded. The result is a scheme in which the auditor is required to grade certain non-conformities a certain way while giving the decision about grading for the rest to the auditor based on the severity of the impact.

Hybrid approaches are used most commonly for schemes that wish to specify which non-conformities will result in an automatic failure of the audit, that is to grade just the critical requirements and leave the rest to the auditor to grade according to the severity of the non-conformity.

Choosing an approach to grading non-conformities

Most schemes will decide which approach (or hybrid version) they will used based on their own needs. Often this is driven by the question of how the scheme owner chooses to approach an audit.  In cases where the scheme wishes to ensure that all non-conformities are graded identically across all clients, the second approach may be preferred. In other cases, the scheme owner may prefer to focus less on designing the standard and wishes to have a professional auditor take a decision about grading. 

Neither is better, neither is worse, it is a matter of preference.