What does it mean to be 'benchmarked'?

For a number of years some folks have been complaining that there are too many different certification marks, they feel that this confuses the consumer. 

One response to this perception is to 'benchmark' the schemes.

The definition of a benchmark is something like: "a standard or point of reference against which things may be compared or assessed". In short, benchmarking of certification schemes is a comparison of a number of schemes against a single set of criteria.

Three approaches to benchmarking

1. An evaluation that focuses on comparing how each scheme addresses (or does not address) each individual element of the benchmark criteria. Usually this is done in a format that encourages each user to make their own decision on which one they like the best.
2. An evaluation of a number of schemes to determine which one meets a minimum level of performance across all the benchmark criteria. Usually this is done to determine who is 'the best'.
3. The awarding of an approval or endorsement to a scheme (or sometimes schemes) that meet a set level of performance set by the benchmarking organization, usually to say that all those that are endorsed are somehow 'equivalent'.

A note about ALL types of benchmarking systems

Before I look at these three types in depth, it is important to note that a benchmark is a type of a standard. It is applied to a scheme rather than an individual client but it is as much a standard as any other standard - it just looks a little different. Organizations that create a benchmarking system for schemes will often tell you that their system is 'neutral' or somehow without any bias - this is never true. Any selection of criteria for benchmarking reflects the interests and values of the folks that put it together. All benchmarking systems have some bias, some more than others.

I should also note that the more generic a benchmarking system is, the less helpful it is in comparing schemes. If I use the same set of criteria to evaluate a scheme for the responsible manufacturing of toasters with a scheme for the sustainable production of frogs legs; I will not really be able to say much about which one is 'better'. If I do the same thing for two schemes that both certify sustainable turnip production I can learn quite a bit about how the two compare.

Now to some examples

The first type of benchmarking systems do not try to award a gold star to the best one. They break each scheme into many elements (hundreds some times) and allow the user to look at each component of each scheme. I can, for example, see how various elements addressing workers' rights are treated in a number of  agricultural schemes. 

These systems produce lots of detailed information and leave it up to the user to draw their own conclusions. As a result they are rich in data but can be really hard to use. The user may need to spend many hours or days studying how two or more schemes compare. This amount of time makes sense if you are purchasing tonnes of fish for processing but hard to justify if you are doing the family shopping.

The second type of benchmarking systems do award a gold star to the best. These are often used by organizations that advocate for the adoption of a certain set of process or production methodologies by an industry sector. They can be done well when all the schemes are responsibly scored against the benchmark criteria or they can be done poorly by picking the winner before the scoring is done. If I know and trust the Responsible Green Bean Council, and they produce a balanced, well researched benchmarking of all green bean certification schemes, I am likely to accept their recommendation of the best scheme even if their choice is based on their own internally defined criteria.

Finally, the third type of system is often used at a commercial scale. It can be used to determine which schemes comply with a set of benchmark criteria (say food safety in the case of GFSI - the Global Food Safety Initiative) and says nothing about any other elements in the schemes. This enables food producers to know if the companies they are purchasing from have a certification that meets or exceeds a set of minimum requirements for food safety. It is important because the procedures for assuring food safety for dairy are very different from the ones that are used for fresh green beans. As a result the producer of frozen green beans with cheese can be assured that both are safe when green beans and cheese are purchased from companies that are certified using GFSI approved schemes. The GFSI system says nothing about any provisions in the green bean scheme that cover workers' rights or animal welfare provisions in the cheese standard.

What is good about benchmarking...

Benchmarking can be a very useful tool. It can provide information that is valuable to producers, manufacturers, processors and consumers. Even if the information is biased, it can be useful if the benchmarkers are honest and up-front about what they are tying to do.

What can go wrong...

When misapplied, benchmarking can be used a way to create a new certification scheme by trying to force existing schemes to do what the benchmarking group wants. This can be successful if a large enough group of organizations are able to influence a key step in the certification process. 

Becoming more efficient....and more rigorous.

From the moment your scheme is up and running you will be under pressure to be more efficient.

This pressure will take many forms. Mostly, this pressure will be from folks that will want you to take a range of steps to reduce the cost of being certified.

• Some folks will want you to be more targeted in your audits. That is audit the important stuff and stop checking the stuff that they feel does not matter.
• Some folks will want you to conduct fewer audits by reducing the number of surveillance audits.
• Some folks will want you to trust their own internal audits or the audits of other folks as proof of compliance.

In addition to this list you will hear a thousand other ideas for increasing the efficiency of your scheme.

While you are being pressured to become more efficient you will also be pressured to be more rigorous. These voices will call for:

• More frequent and more rigorous audits.
• Stronger sanctions for those found to not be in compliance.
• Public shaming of weak performers.
• Rapidly increasing levels of performance for certificate holders.

Please note that no matter how many of these steps you take both groups will want more. For a number of companies, the best cost is zero. For a number of stakeholders the only assurance level that is acceptable is infinite.

The frustrating part of these two pressures is that both of these groups have a point. In truth, efficiency and rigor are not always at odds.

The central question is how to deliver the greatest assurance at the lowest price. This can be done by first of all working to figure out what your certification means. Are you clear about the market signal that your certification carries. Are you sure about the level of assurance that you need to clearly and confidently assure users that your scheme is doing what you promised you would do?

To address the challenges you could consider the following:

1. Do your certificate holders and other stakeholders understand what you are tying to do? Is the level of assurance that you are providing what they want and need?
2. Are you asking for too much in an audit? Are you just piling on more and more requirements in the belief that it will make your scheme more rigorous?
3. Have you carefully studied your scheme to find out if the requirements are really related to the assurance you are providing?
4. Are you scaling back on requirements in the hope of lowering costs and growing your market share?
5. Are there redundant requirements in your scheme? That is, do you require auditors to check multiple times for the same thing; or do you check for several requirements that are all found to either be in conformity or non-conformity 99% of the time?
6. Are you managing your scheme to grow your market share without considering advancements in science or other research that means you should make changes in your requirements?
7. Are you sensitive to the cost of your scheme for users? 
8. Are you exploring ways to use technology to improve confidence in your certification while reducing costs?

The real challenge in all this is that your scheme will never be static. It will always be under review and you should be constantly looking in every corner to find ways to reduce costs as well as enhance confidence in your scheme. 

Most of the changes you will make will be incremental, that is the change in cost or confidence will be minimal. But taken together you will be able to offer constant improvements to the demands of both groups.

Certification schemes live and die in the market. That means both the supply and demand side. To survive and grow you will need to constantly make your scheme more cost efficient and offer higher and higher levels of assurance. 

Most significantly, if you are constantly working to improve all aspects of your scheme you will be around for a long time.

None of it is easy, but for us certification geeks it sure is interesting.

Think like your clients...Why do they want to be certified?

So now your scheme is up an running. New clients are happily signing up for audits and certificates are being issued.

Do you really understand why they are going to all this trouble?

Sometimes a new idea, technology or product is conceived one way by the producer but used in a completely different way by the user. Understanding how your scheme is actually being used may be an eye-opener. 

It is true that in many cases a certification scheme is adopted because someone further down the value chain requires it. Let's say I am growing blueberries. My client comes and tells me that I have to be certified to the 'Green Blueberry' standard or my client will not buy my berries. Well that is fair enough, clients specify what they want and I either meet their requirements or find a new client.

So lets follow the value chain and ask each the folks at each link why they are specifying your new blueberry certification. Maybe the packer is specifying it because she feels it will increase her market share; she has access to certified berries and the market is demanding them. She may or may not be committed to your philosophy on blueberries but she is motivated to do what she has to do to get and stay certified.

Further down the chain you find a bakery that produces blueberry pies for sale in grocery stores. He may be motivated to add one more reason for his clients to continue to buy from him. His clients are clear to him that they want certified berry pies. For the baker it is one more way he can negotiate longer term contracts to supply pies to stores. His clients are less likely to shift suppliers just for price advantage if he can add the certification that adds value to his product.

Now we are at the grocery store, they want certified berries (and pies) because their customers want to feel they are getting healthy food that is not produced in a way that harms the environment or exploits workers. The grocery store is building a relationship with their customers so that they will choose their store over the completion. Certified blueberry pies are one more way they can do this.

While in this supply chain all the participants are happy, it may not always be the case.

One element of supply chain certification is that you can use the list of certified producers to skip over some folks in the supply chain. The baker for example could find a certified farm to buy from directly, skipping the packer completely. This could be better for the farmer and the baker but the packer looses business. The farmer could sell his berries for a little more and the baker could buy them for a little less than the packer charged and they both could come out ahead.

Disruption in supply chains is a common impact of certification, especially in long or complex supply chains. If the baker needed to source his berries from another country it can be expensive and time consuming to hunt up a supplier. But thanks to an online list of certified producers the job becomes much easier. The baker can purchase certified berries from halfway around the world, directly from a certified producer without having to buy through a broker or wholesaler.

In short, certification is good for those in the supply chain that can take advantage of the opportunities that it presents, it also can harm (or even put out of business) those packers, processors, brokers or others in the supply chain that certification may disadvantage.

Statistics, sampling and other mysteries of the universe....

One challenge of a scheme owner is to determine the most appropriate balance between the intensity of the audit and cost. In short, how many examples must an auditor check to ensure that there is conformity? Keep in mind that every additional thing an auditor is required to do will cost the client more (and therefor everyone else in the value chain).

To be blunt, finding a non-conformity is a bit like looking for your lost keys...once you find them you can stop looking. The only difference is that the auditor has no knowledge of how many 'keys' he is looking for.

If the auditor is auditing a large forestry operation and she finds a non-conformity at the first harvest site she will continue to check the rest of the sites to be audited to determine whether or not the non-conformity she found is a single error or if it occurs at each site. One feller buncher operating too close to a stream may be a minor issue (i.e. an operator that is having a bad day) but all feller bunchers operating too close to many streams can be a major issue (i.e. the company told them to do it).

OK, that sounds simple but how many sites should the auditor select for inspection in the first place? Should the auditor inspect all sites, half of them, or just one? 

The "Square Root Rule"

It is a common practice for auditors to choose the number of sites based on the "square root rule"; this simply means that the number of sites to be audited is the same as the square root of the total number of possible sites (NOTE that this also applies to the number of files, employees to interview or other sets of things to be checked). If there are 16 sites, for example, then 4 should be checked.  If the square root does not give you a whole number then round up the number of sites to the nearest home number. If there are 10 sites (the square root of 10 is 3.16...) the number of sites to be checked should be still be 4.

This is all well and good and the square root rule is easy to follow and easy to calculate (just about every calculator has a built in square root function). The square root rule also gives you a small number of sites to check so that the cost of an audit can be kept low (fewer sites to check means less auditor time equals a lower audit fee). But, what about the reliability of the sample?

Using Statistics to Determine Sample Size

If we change hats and look at the question of sample size using the mind of the statistician we may wish to question the reliability of the square root rule.

If your scheme decides that you want to have a sample that gives you, for example, 95% confidence, plus or minus 5%, then your sample requirement will look much different.

As you can see, the number of sites necessary to audit to achieve the statistical confidence that is like that of academic research is far greater than applying the square root rule. The smaller the sample set the less statistical confidence that can be obtained.

To calculate the number of sites, files, employees or other things to audit for a specific number there are many online tools to do it for you (i.e. http://www.surveysystem.com/sscalc.htm).

The Real Issue

The primary question that you as the scheme owner should consider is: "What level of certainty do I need?"

The answer for this is in understanding the industry sectors that your scheme involves. What do they need? What do their customers need? What do the users of the products and services that they produce need?

What is the trade-off between cost and accuracy - i.e. how much is enough?

It is important to understand what happens in sample selection, both how the sample size is determined and how that number of samples is selected. More is not always better. One way to look a is is to see the best as meeting the needs of your users both in terms of accuracy and cost.

Never ask a question to which you don't know the answer....

The first rule of auditing is really quite simple: Do not ask a question to which you do not know the answer. To get to why, let's first look at the purpose and some methods of auditing.

The purpose of an audit

When developing your scheme it is important to know what happens in an audit - this knowledge will form the basis for a well designed and effective scheme. 

First of all a quick reminder of what audits are not... they are not research, investigation or enforcement. 

An audit is an examination of the system, product or service to determine conformity or nonconformity with stated requirements. To undertake the audit an auditor approaches an audit with the expectation of conformity, after all a client would not reasonable hire a conformity assessment body to conduct an audit if they did not feel that they are already in conformity with the scheme requirements. So, an audit is conducted with the sole purpose of auditing the client's explicit or implied assertion that the system, product or service being audited is in full conformity with the standard. Auditors make determinations of conformity or non-conformity based on the evidence observed in the audit.

Auditors do not punish, conduct research or launch investigations.

Conducting an audit

So...back to the first rule of auditing: Do not ask a question to which you do not know the answer.

An auditor does her job by examining evidence provided by the client with an eye to determine if the evidence demonstrates conformity. The evidence that is evaluated can include documents, interviews,  and visual inspections.

In the audit, the auditor should record the evidence evaluated and note when it is in conformity and when it is not.

When asking questions, the auditor should always ask open-ended questions, ones that do not give an expected answer to the person being asked. (i.e. do not ask "You don't put that in this box, do you?" instead ask "What do you do next?") Auditors should listen to the responses and ask further questions to probe further as needed. 

The auditor should ask for objective evidence to support the answers. (i.e. "Why do you do that?")

What this means for your scheme.

As you work on your scheme, carefully consider what the requirements are that apply to the client. Are your requirements designed to be assessed for conformity by the auditor - or are your asking the auditor to play another role. 

Auditors do conformity assessment only. The requirements that your scheme lays out should be clear to the client and easy (as much as possible) for the auditor to determine, based on objective evidence, whether or not the client conforms.

Who can write a standard?

Almost a year ago I posted an entry titled "What is a Standard?".  In that post I included the ISO definition of a standard:

ISO defines a standard as:  A document established by consensus and approved by a recognized body that provides for common and repeated use, rules, guidelines or characteristics for activities or their results, aimed at the achievement of the optimum degree of order in a given context.  (from ISO/IEC Guide 2:1996, definition 3.2)

This definition includes a couple of bits that can be confusing. It says that a standard is "...established by consensus and approved by a recognized body..."

So, what is meant by "consensus" and "recognized body"?

Consensus

Many times the idea of reaching a decision by consensus is rejected because we understand that it means that everyone must fully agree with everything in a decision. This can be problematic for long and complex documents like standards. If I agree with everything in a standard except one line can I withhold my agreement and prevent approval of the standard?

In most standards organizations that I know of the definition of consensus is 'the absence of sustained opposition'. That means that my opposition may be minor but not something that is critical to my decision. I may want my opposition logged in the decision but it may not be worth the cost of renegotiating the whole document to get what I want. If that is the case then a consensus can be reach, even if there is some opposition, even by several participants.

If on the other hand my opposition is to a part of the standard that presents a major concern to me I can raise my objection and potentially withhold consensus.

Some standards developers set the definition of consensus as a super majority of 75, 85 or 90%. In these cases agreement of the super majority is understood to be a consensus.

A consensus under either of these approaches can be hard to achieve because some participants may be particularly strident in their positions or others can play a game with the voting rules to get what they want over the strong objections of others.

No matter which model is adopted it is important to design the rules for decisions in such a way that everyone has a say and that no sector on the decision making body can dictate a decision to others.

This can occur if a decision making body has 50 members and only 2 represent workers and 48 represent employers. If the rule requires a super majority of 90% it is still possible for all the employer representatives to out vote the labour representatives even if their opposition is very strong.

In the case of consensus defined as the absence of sustained opposition it is possible to give such latitude to the chair or an executive committee that they are able to declare a consensus that favours their position by announcing that the opposition is not sustained no matter what the opposition itself wants. 

No matter which system is used it is important the the underlying principle of consensus be respected by the rules and the participants. Everyone should be willing to support the decision even if they disagree with some elements in the standard.

Recognized body

The issue of a recognized body can be a bit more problematic. In some countries this could be defined in legislation, that is there may be only one 'recognized body' in the country or there may be a defined procedure that an organization must follow to become a recognized body.

Some people may even read the ISO definition as declaring that only ISO and bodies that ISO recognizes may develop standards. I will note that if this was the case that there is no way that ISO has the authority to enforce such a provision. ISO is a not-for-profit corporation based in Switzerland and while national governments may be members the organization has no authority to speak for governments. ISO produces standards that are voluntary. Even if a country adopts an ISO standard as a regulation (yes this does happen) this is the result of a decision by a government, not by ISO or any other standards development organization.

I will propose what I think is a better definition of a recognized body - that is a body that conforms to established norms for standards developers. This includes codes of practice like Annex 3 to the World Trade Organization (WTO) 'Code of good practice for the preparation, adoption and application of standards'; ISO Guide 59 'Code of good practice for standarization' and the ISEAL Alliance 'Standard Setting Code'. There are other guides and norms, some are national and some are specific to an industry (such as the FAO guidelines for certification of wild or farmed seafood).

Anyone developing a standard should understand the guidelines and norms that may apply to them and do their best to conform to them. This is the hallmark of a professional organization.

Claims

One of the common complaints about certification is that users do not always understand what is being certified. This may be caused by a certification claim not lining up with scheme requirements (sometimes accidentally and other times intentionally) or with users not being able to easily understand what a certification mark or claim means.

Sometimes a certification is applied to a single product, and a company may represent the certification as applying to the whole company - or the other way around. If for example, you see a product sold with an ISO 9001 certification claim on the box it can seem that the product is certified. This is not the case since ISO 9001 is a management system certification that applies to the company but does not evaluate individual products. In this case it is the company that is certified, not the products. Using the cetification, the company can make claims about how it is run but not about individual products

The reverse can apply if a company claims that it is certified, for example, to MSC (Marine Stewardship Council). The MSC scheme is designed to certify wild-capture seafood. In this case the source of the seafood is certified, the company is not certified. An MSC certificate means that the company can make claims about products that are or contain MSC certified seafood but not about the how the company is run.

Claims can take many forms, they can be:

• A logo or mark applied to products; 
• A logo that is to be linked to a corporate name;
• A tagline that is printed on a product or used in advertisements;
• A tagline that is associated with a corporate name; or
• Any combination of the above.

Claims can be easily exaggerated - just look a claims made about the healthiness of foods and you can easily see how they can get out of hand.

The challenge in designing a claim that a certificate holder can make about a certification is to make sure that the claim is easily understood and accurate. Currently the ISEAL Alliance is working on developing a Good Practice Guide for Sustainability Claims, it you are working on claims for your scheme this draft will help with your thinking.

One advantage of certification is that the claim can be controlled by the scheme owner through a licencing agreement. The way in which the certification is marketed and a logo is used can be controlled by the signed agreement and enforced by the owner of the scheme.

It should always be the responsibility of the scheme owner to set out the rules about what claims can be made and how a scheme logo may be put on products or corporate mastheads.

The claims that you allow for your scheme should be accurate, clear and easily understood by the user. 

Finally, schemes should always make sure that their rules are being followed and take action when they are not. Your reputation is all that you have and when your scheme is misrepresented it undermines the value of your certifications, trademarks and claims as well as reducing the value of your scheme to users that follow the rules.

Transparency is being generous

The ISEAL Credibility Principles explain that for a certification scheme transparency means:

"Standards systems make relevant information freely available about the development and content of the standard, how the system is governed, who is evaluated and under what process, impact information and the various ways in which stakeholders can engage."

Before a scheme can be transparent the scheme owner must first have documented the key information about how the scheme is run. Having clear documentation is the first challenge because it takes time and resources to write it. In addition, this documentation should be written in clear, simple language that is easy for your stakeholders to understand and use.

Why bother with transparency?

Certification is a trust business. Certificate holders and all those in the value chain rely on your certification to ensure that the products they produce or trade conform to the standard. They rely on the assurance that the scheme is providing to assure their customers and manage their own risk. Consumers and other end users rely on the certification in their specifications and in making purchasing decisions.

Transparency is a means for scheme owners to assure all those that rely on the scheme that they are living up to their obligations in managing and operating the scheme. It is a means of demonstrating to users that the scheme is trustworthy, and as a result there can be confidence in the certification. Also, users, should they choose may examine the scheme to assure themselves that it is meeting their needs.

Transparency means that information is available to anyone that wants it, including governments, users, researchers, advocates, and anyone that is interested (including high school students writing assigned essays).

A minimum list for sharing with the public

The first step to facilitate transparency is to have a package of material available on the web for anyone to download. At a minimum this package should include:

• A procedure that describes how the standard is developed, revised and interpreted. This should include a complete description of the processes including how any interested party can participate, how decisions are made and who makes the decisions. 
• A description of the purpose of the standard and its scope; that is what it is intended to accomplish and when it is applicable.
• An explanation of the governance of the scheme, including the scheme owner's board of directors (listing the members of the board, their terms of office and how they are appointed) and any other technical or advisory boards that play a role in the governance of the scheme.
• A description of the monitoring and evaluation program that is used by the scheme owner to evaluate the quality, consistency, efficiency and other aspects of the performance and operation of the scheme.
• A description of how the scheme evaluates its impacts; that is are the objectives of the scheme being achieved including the procedures used in making the evaluation as well as how conclusions are reached. It is best if in addition to this description that regular reports on performance are published.

• A description of how stakeholders can become involved, this should include involvement in certification audits, standards development, governance and most importantly in commenting, raising a concern or making a formal complaint about a certificate, a certification body or the standard (or any other aspect of the scheme).

In addition to these documented procedures, policies and reports each scheme owner should consider what other information should be made freely available to the public. As a general rule, my advice to scheme owners is to default, wherever possible to being transparent. In other words don't just post the minimum information; only keep confidential the material that is absolutely necessary and make all the rest freely available. 

A guiding principle is to be a generous as possible when it comes to being transparent. While it may mean that more questions come, it also is an opportunity to build trust with key individuals, organizations and institutions. Obviously there are limitations in terms of staff time and available resources but the investment will pay off.