Normative vs. Informative

Certification documentation schemes can contain two types of information; normative and informative. 

Normative elements are those that are prescriptive, that is they are to be followed in order to comply with scheme requirements.

Informative elements are those that are descriptive, that is they are designed to help the reader understand the concepts presented in the normative elements.

Normative elements

Normative elements apply to the organization, or individual to whom they are addressed. For example, normative elements in the standard apply to the certificate holder or applicant for certification. Other normative elements may apply to the certification body, accreditation body or scheme owner.

Normative elements are normally phrased using one of the three terms: shall, should or may.

• Shall:  indicates a requirement
• Should: indicated as recommendation
• May: indicates a permission

All normative elements that are requirements (shall) are to be followed in all cases in order to be in conformity. Elements that are recommended (should) are ones that any organization that wants to be in conformity are encouraged to follow. Finally, elements that grant permission (may) have the option of following if they choose to and if they do then they must follow the requirements as described.

Each of these elements can be a single requirement (i.e. ...all letters shall be written on pink paper) or an entire document (i.e. ...the factory shall conform to ISO 19001).

For the sake of clarity, it is best when all scheme documents use these three keywords. When you mean it as a requirement; say 'shall' and not use other terms such as 'will', 'must' or 'need'. Using these terms consistently means that the reader is not left to figure out what you mean when you say 'the certificate holder will...' --- is that a requirement, an option or a recommendation?

Informative Elements

Informative elements are illustrations, examples or suggestions that explain the meaning and implications of the requirements as well as giving suggestions, examples and case studies on the application of the requirements.

Nothing in the informative elements is or can be mandatory; that includes the accreditation body, the certification body, the certificate holder or the applicant for certification. If guidance is mandatory, that is if it includes a shall, should or may statement, it is not informative, it is normative.

What this means for a scheme

The best certification schemes are clearly written and are easily understood by each user and each type of user. An applicant for certification should be able to understand what is required of them, and by implication what is not required for them to be certified.

One of the best ways to make sure your scheme is easily understood is to use the discipline of writing your requirements as clearly as possible. All users should know what is required and should never be surprised to find that a requirement was not clearly presented to them.

Whenever possible guidance should be provided to help the user understand the implications of your scheme requirements. Informative material should only help the user to understand the thinking behind a requirement or maybe how or why a requirement was set. 

This sounds simple until you try to do it. Clear documentation is only possible when you and everyone involved in designing and writing up the scheme clearly understand and agree on the implications of the requirements.

Guest post by Mike Read: Can you be 100% confident that everything is perfect?

I am happy to add this post from my colleague Mike Read to The Kitbag. Mike and I have worked together on several project and his experience has added much to this work. This post started out as a response to my earlier post "Statistics, sampling and other mysteries of the universe" that I posted on 18 May 2015. Mike can be reached directly through his website: www.mikeread.org

-----------------------------------

Can we be 100% confident that everything is perfect? Although we might all like to think so, the answer - especially when it comes to certification - is almost certainly not! Can we check every auditor and every system on every occasion? Are auditors going to check every site, every day? And will they look at everything? Of course not. That would make certification so expensive no one would buy it. There is an inevitable trade-off between cost and certainty.  But how do we get the best value for money, and what is the minimum level of certainty that is acceptable?  And while we’re at it: ‘certainty of what, exactly?’ These questions are vital to the credibility of certification, and what is certified, but the truth is that answers are routinely fudged.

Most systems mix some kind of sampling into their fudge recipes. But before we get to the joys of how to choose a wise sample, there’s another question that is even more rarely asked. Whose opinion matters?

WHOSE OPINION?

It’s unlikely to be just your certification scheme that bases its reputation on the quality of the assurance you offer.  Maybe a big supermarket or chain of cafés insists that your logo appears on products it sells to ensure its brand retains a good image, and to avoid any nasty surprises and exposés of bad practice. Whether you like it or not, you’re in the business of selling them risk management. And perhaps their customers depend on it too for their own sense of well-being, and would switch to a different supplier if something went wrong.

Maybe you would happily put your logo on products coming from a source where one in 20 of your certification criteria are not being met, if corrective action is in place to remedy any failings. Do you share this tolerance level with your client? Do you know that they find it acceptable? What if they would only accept one in 100? Or maybe you accept that your chain of custody scheme can only ensure that 995 out of every 1,000 labelled products are from properly certified sources, Perhaps your customers expect 999 or even 1,000?

Without knowing your value chain partners’ acceptance of risk as well as their acceptance of cost, how can you properly design your assurance system?  Surely it’s far better to engage them in conversation before something goes wrong, rather than after. But how many schemes are scared of doing so?

CONFIDENCE

And so, now, to sampling. We may be familiar with the idea of sampling giving a certain confidence in the overall result. But when a statistical test provides 95% or 99% ‘confidence’, beware! You need to be sure you can answer ‘confident of what?’ You might be 95% confident that you have identified 100% of non-conformities, or 100% confident that you have identified 95% of non-conformities. Think about it for a moment: these are different things with different potential implications. More realistically you might aim to be 95% confident that you have identified somewhere between 90 and 100% of non-conformities. In other words we need to properly understand the difference between confidence levels and confidence intervals (or limits).

Like any established branch of science, statistics and sampling can be incredibly complicated. Yet almost all of it is based on ‘laboratory conditions’ and ‘probability theory’ and almost none of it works very well in the real world of natural product certification. And in passing, the all-too-familiar ‘square root rule’ has no foundation even in lab conditions, but still hangs around like a bad smell.

Statistics and sampling should only ever be used as guidance rather than direction. This is very much a place for informed common sense.  And be sure you know exactly what you’re trying to find out, as a carefully phrased question is already well on the way to being answered.

RISK-BASED SAMPLING

An approach being adopted by a number of certification schemes sees sampling based in part on assessment of where problems are most likely to occur. In other words you check more often or more intensely in regions, or products, or with suppliers where a comprehensive and dispassionate risk analysis tells you problems are more likely to arise. This can keep costs down and really strengthen your assurance processes. You might also want to combine this with cyclical sampling, making sure that over a set period as many as possible – or even all – elements of the system are checked. How to do this well is perhaps the subject for another blog. 

ENLIGHTEN YOURSELF!

Look at the patterns in your own data, perhaps the way, the number and the location of problems that you find out about. This can tell you a lot about how well your assurance system is working and any sampling that you might be doing. But beware those ‘unknown unknowns’. Maybe your data reveals no problems because you’re not looking in the right place at the right time with the right eyes! As Carl Sagan neatly reminded us ‘absence of evidence is not evidence of absence’.

Mike Read Associates would be very happy to help with your risk assessment, risk management, and sampling strategies. So you can deliver assurance that you and your partners’ clients and customers can rely on, and can afford.

What does it mean to be 'benchmarked'?

For a number of years some folks have been complaining that there are too many different certification marks, they feel that this confuses the consumer. 

One response to this perception is to 'benchmark' the schemes.

The definition of a benchmark is something like: "a standard or point of reference against which things may be compared or assessed". In short, benchmarking of certification schemes is a comparison of a number of schemes against a single set of criteria.

Three approaches to benchmarking

1. An evaluation that focuses on comparing how each scheme addresses (or does not address) each individual element of the benchmark criteria. Usually this is done in a format that encourages each user to make their own decision on which one they like the best.
2. An evaluation of a number of schemes to determine which one meets a minimum level of performance across all the benchmark criteria. Usually this is done to determine who is 'the best'.
3. The awarding of an approval or endorsement to a scheme (or sometimes schemes) that meet a set level of performance set by the benchmarking organization, usually to say that all those that are endorsed are somehow 'equivalent'.

A note about ALL types of benchmarking systems

Before I look at these three types in depth, it is important to note that a benchmark is a type of a standard. It is applied to a scheme rather than an individual client but it is as much a standard as any other standard - it just looks a little different. Organizations that create a benchmarking system for schemes will often tell you that their system is 'neutral' or somehow without any bias - this is never true. Any selection of criteria for benchmarking reflects the interests and values of the folks that put it together. All benchmarking systems have some bias, some more than others.

I should also note that the more generic a benchmarking system is, the less helpful it is in comparing schemes. If I use the same set of criteria to evaluate a scheme for the responsible manufacturing of toasters with a scheme for the sustainable production of frogs legs; I will not really be able to say much about which one is 'better'. If I do the same thing for two schemes that both certify sustainable turnip production I can learn quite a bit about how the two compare.

Now to some examples

The first type of benchmarking systems do not try to award a gold star to the best one. They break each scheme into many elements (hundreds some times) and allow the user to look at each component of each scheme. I can, for example, see how various elements addressing workers' rights are treated in a number of  agricultural schemes. 

These systems produce lots of detailed information and leave it up to the user to draw their own conclusions. As a result they are rich in data but can be really hard to use. The user may need to spend many hours or days studying how two or more schemes compare. This amount of time makes sense if you are purchasing tonnes of fish for processing but hard to justify if you are doing the family shopping.

The second type of benchmarking systems do award a gold star to the best. These are often used by organizations that advocate for the adoption of a certain set of process or production methodologies by an industry sector. They can be done well when all the schemes are responsibly scored against the benchmark criteria or they can be done poorly by picking the winner before the scoring is done. If I know and trust the Responsible Green Bean Council, and they produce a balanced, well researched benchmarking of all green bean certification schemes, I am likely to accept their recommendation of the best scheme even if their choice is based on their own internally defined criteria.

Finally, the third type of system is often used at a commercial scale. It can be used to determine which schemes comply with a set of benchmark criteria (say food safety in the case of GFSI - the Global Food Safety Initiative) and says nothing about any other elements in the schemes. This enables food producers to know if the companies they are purchasing from have a certification that meets or exceeds a set of minimum requirements for food safety. It is important because the procedures for assuring food safety for dairy are very different from the ones that are used for fresh green beans. As a result the producer of frozen green beans with cheese can be assured that both are safe when green beans and cheese are purchased from companies that are certified using GFSI approved schemes. The GFSI system says nothing about any provisions in the green bean scheme that cover workers' rights or animal welfare provisions in the cheese standard.

What is good about benchmarking...

Benchmarking can be a very useful tool. It can provide information that is valuable to producers, manufacturers, processors and consumers. Even if the information is biased, it can be useful if the benchmarkers are honest and up-front about what they are tying to do.

What can go wrong...

When misapplied, benchmarking can be used a way to create a new certification scheme by trying to force existing schemes to do what the benchmarking group wants. This can be successful if a large enough group of organizations are able to influence a key step in the certification process. 

Becoming more efficient....and more rigorous.

From the moment your scheme is up and running you will be under pressure to be more efficient.

This pressure will take many forms. Mostly, this pressure will be from folks that will want you to take a range of steps to reduce the cost of being certified.

• Some folks will want you to be more targeted in your audits. That is audit the important stuff and stop checking the stuff that they feel does not matter.
• Some folks will want you to conduct fewer audits by reducing the number of surveillance audits.
• Some folks will want you to trust their own internal audits or the audits of other folks as proof of compliance.

In addition to this list you will hear a thousand other ideas for increasing the efficiency of your scheme.

While you are being pressured to become more efficient you will also be pressured to be more rigorous. These voices will call for:

• More frequent and more rigorous audits.
• Stronger sanctions for those found to not be in compliance.
• Public shaming of weak performers.
• Rapidly increasing levels of performance for certificate holders.

Please note that no matter how many of these steps you take both groups will want more. For a number of companies, the best cost is zero. For a number of stakeholders the only assurance level that is acceptable is infinite.

The frustrating part of these two pressures is that both of these groups have a point. In truth, efficiency and rigor are not always at odds.

The central question is how to deliver the greatest assurance at the lowest price. This can be done by first of all working to figure out what your certification means. Are you clear about the market signal that your certification carries. Are you sure about the level of assurance that you need to clearly and confidently assure users that your scheme is doing what you promised you would do?

To address the challenges you could consider the following:

1. Do your certificate holders and other stakeholders understand what you are tying to do? Is the level of assurance that you are providing what they want and need?
2. Are you asking for too much in an audit? Are you just piling on more and more requirements in the belief that it will make your scheme more rigorous?
3. Have you carefully studied your scheme to find out if the requirements are really related to the assurance you are providing?
4. Are you scaling back on requirements in the hope of lowering costs and growing your market share?
5. Are there redundant requirements in your scheme? That is, do you require auditors to check multiple times for the same thing; or do you check for several requirements that are all found to either be in conformity or non-conformity 99% of the time?
6. Are you managing your scheme to grow your market share without considering advancements in science or other research that means you should make changes in your requirements?
7. Are you sensitive to the cost of your scheme for users? 
8. Are you exploring ways to use technology to improve confidence in your certification while reducing costs?

The real challenge in all this is that your scheme will never be static. It will always be under review and you should be constantly looking in every corner to find ways to reduce costs as well as enhance confidence in your scheme. 

Most of the changes you will make will be incremental, that is the change in cost or confidence will be minimal. But taken together you will be able to offer constant improvements to the demands of both groups.

Certification schemes live and die in the market. That means both the supply and demand side. To survive and grow you will need to constantly make your scheme more cost efficient and offer higher and higher levels of assurance. 

Most significantly, if you are constantly working to improve all aspects of your scheme you will be around for a long time.

None of it is easy, but for us certification geeks it sure is interesting.

Think like your clients...Why do they want to be certified?

So now your scheme is up an running. New clients are happily signing up for audits and certificates are being issued.

Do you really understand why they are going to all this trouble?

Sometimes a new idea, technology or product is conceived one way by the producer but used in a completely different way by the user. Understanding how your scheme is actually being used may be an eye-opener. 

It is true that in many cases a certification scheme is adopted because someone further down the value chain requires it. Let's say I am growing blueberries. My client comes and tells me that I have to be certified to the 'Green Blueberry' standard or my client will not buy my berries. Well that is fair enough, clients specify what they want and I either meet their requirements or find a new client.

So lets follow the value chain and ask each the folks at each link why they are specifying your new blueberry certification. Maybe the packer is specifying it because she feels it will increase her market share; she has access to certified berries and the market is demanding them. She may or may not be committed to your philosophy on blueberries but she is motivated to do what she has to do to get and stay certified.

Further down the chain you find a bakery that produces blueberry pies for sale in grocery stores. He may be motivated to add one more reason for his clients to continue to buy from him. His clients are clear to him that they want certified berry pies. For the baker it is one more way he can negotiate longer term contracts to supply pies to stores. His clients are less likely to shift suppliers just for price advantage if he can add the certification that adds value to his product.

Now we are at the grocery store, they want certified berries (and pies) because their customers want to feel they are getting healthy food that is not produced in a way that harms the environment or exploits workers. The grocery store is building a relationship with their customers so that they will choose their store over the completion. Certified blueberry pies are one more way they can do this.

While in this supply chain all the participants are happy, it may not always be the case.

One element of supply chain certification is that you can use the list of certified producers to skip over some folks in the supply chain. The baker for example could find a certified farm to buy from directly, skipping the packer completely. This could be better for the farmer and the baker but the packer looses business. The farmer could sell his berries for a little more and the baker could buy them for a little less than the packer charged and they both could come out ahead.

Disruption in supply chains is a common impact of certification, especially in long or complex supply chains. If the baker needed to source his berries from another country it can be expensive and time consuming to hunt up a supplier. But thanks to an online list of certified producers the job becomes much easier. The baker can purchase certified berries from halfway around the world, directly from a certified producer without having to buy through a broker or wholesaler.

In short, certification is good for those in the supply chain that can take advantage of the opportunities that it presents, it also can harm (or even put out of business) those packers, processors, brokers or others in the supply chain that certification may disadvantage.

Statistics, sampling and other mysteries of the universe....

One challenge of a scheme owner is to determine the most appropriate balance between the intensity of the audit and cost. In short, how many examples must an auditor check to ensure that there is conformity? Keep in mind that every additional thing an auditor is required to do will cost the client more (and therefor everyone else in the value chain).

To be blunt, finding a non-conformity is a bit like looking for your lost keys...once you find them you can stop looking. The only difference is that the auditor has no knowledge of how many 'keys' he is looking for.

If the auditor is auditing a large forestry operation and she finds a non-conformity at the first harvest site she will continue to check the rest of the sites to be audited to determine whether or not the non-conformity she found is a single error or if it occurs at each site. One feller buncher operating too close to a stream may be a minor issue (i.e. an operator that is having a bad day) but all feller bunchers operating too close to many streams can be a major issue (i.e. the company told them to do it).

OK, that sounds simple but how many sites should the auditor select for inspection in the first place? Should the auditor inspect all sites, half of them, or just one? 

The "Square Root Rule"

It is a common practice for auditors to choose the number of sites based on the "square root rule"; this simply means that the number of sites to be audited is the same as the square root of the total number of possible sites (NOTE that this also applies to the number of files, employees to interview or other sets of things to be checked). If there are 16 sites, for example, then 4 should be checked.  If the square root does not give you a whole number then round up the number of sites to the nearest home number. If there are 10 sites (the square root of 10 is 3.16...) the number of sites to be checked should be still be 4.

This is all well and good and the square root rule is easy to follow and easy to calculate (just about every calculator has a built in square root function). The square root rule also gives you a small number of sites to check so that the cost of an audit can be kept low (fewer sites to check means less auditor time equals a lower audit fee). But, what about the reliability of the sample?

Using Statistics to Determine Sample Size

If we change hats and look at the question of sample size using the mind of the statistician we may wish to question the reliability of the square root rule.

If your scheme decides that you want to have a sample that gives you, for example, 95% confidence, plus or minus 5%, then your sample requirement will look much different.

As you can see, the number of sites necessary to audit to achieve the statistical confidence that is like that of academic research is far greater than applying the square root rule. The smaller the sample set the less statistical confidence that can be obtained.

To calculate the number of sites, files, employees or other things to audit for a specific number there are many online tools to do it for you (i.e. http://www.surveysystem.com/sscalc.htm).

The Real Issue

The primary question that you as the scheme owner should consider is: "What level of certainty do I need?"

The answer for this is in understanding the industry sectors that your scheme involves. What do they need? What do their customers need? What do the users of the products and services that they produce need?

What is the trade-off between cost and accuracy - i.e. how much is enough?

It is important to understand what happens in sample selection, both how the sample size is determined and how that number of samples is selected. More is not always better. One way to look a is is to see the best as meeting the needs of your users both in terms of accuracy and cost.

Never ask a question to which you don't know the answer....

The first rule of auditing is really quite simple: Do not ask a question to which you do not know the answer. To get to why, let's first look at the purpose and some methods of auditing.

The purpose of an audit

When developing your scheme it is important to know what happens in an audit - this knowledge will form the basis for a well designed and effective scheme. 

First of all a quick reminder of what audits are not... they are not research, investigation or enforcement. 

An audit is an examination of the system, product or service to determine conformity or nonconformity with stated requirements. To undertake the audit an auditor approaches an audit with the expectation of conformity, after all a client would not reasonable hire a conformity assessment body to conduct an audit if they did not feel that they are already in conformity with the scheme requirements. So, an audit is conducted with the sole purpose of auditing the client's explicit or implied assertion that the system, product or service being audited is in full conformity with the standard. Auditors make determinations of conformity or non-conformity based on the evidence observed in the audit.

Auditors do not punish, conduct research or launch investigations.

Conducting an audit

So...back to the first rule of auditing: Do not ask a question to which you do not know the answer.

An auditor does her job by examining evidence provided by the client with an eye to determine if the evidence demonstrates conformity. The evidence that is evaluated can include documents, interviews,  and visual inspections.

In the audit, the auditor should record the evidence evaluated and note when it is in conformity and when it is not.

When asking questions, the auditor should always ask open-ended questions, ones that do not give an expected answer to the person being asked. (i.e. do not ask "You don't put that in this box, do you?" instead ask "What do you do next?") Auditors should listen to the responses and ask further questions to probe further as needed. 

The auditor should ask for objective evidence to support the answers. (i.e. "Why do you do that?")

What this means for your scheme.

As you work on your scheme, carefully consider what the requirements are that apply to the client. Are your requirements designed to be assessed for conformity by the auditor - or are your asking the auditor to play another role. 

Auditors do conformity assessment only. The requirements that your scheme lays out should be clear to the client and easy (as much as possible) for the auditor to determine, based on objective evidence, whether or not the client conforms.

Who can write a standard?

Almost a year ago I posted an entry titled "What is a Standard?".  In that post I included the ISO definition of a standard:

ISO defines a standard as:  A document established by consensus and approved by a recognized body that provides for common and repeated use, rules, guidelines or characteristics for activities or their results, aimed at the achievement of the optimum degree of order in a given context.  (from ISO/IEC Guide 2:1996, definition 3.2)

This definition includes a couple of bits that can be confusing. It says that a standard is "...established by consensus and approved by a recognized body..."

So, what is meant by "consensus" and "recognized body"?

Consensus

Many times the idea of reaching a decision by consensus is rejected because we understand that it means that everyone must fully agree with everything in a decision. This can be problematic for long and complex documents like standards. If I agree with everything in a standard except one line can I withhold my agreement and prevent approval of the standard?

In most standards organizations that I know of the definition of consensus is 'the absence of sustained opposition'. That means that my opposition may be minor but not something that is critical to my decision. I may want my opposition logged in the decision but it may not be worth the cost of renegotiating the whole document to get what I want. If that is the case then a consensus can be reach, even if there is some opposition, even by several participants.

If on the other hand my opposition is to a part of the standard that presents a major concern to me I can raise my objection and potentially withhold consensus.

Some standards developers set the definition of consensus as a super majority of 75, 85 or 90%. In these cases agreement of the super majority is understood to be a consensus.

A consensus under either of these approaches can be hard to achieve because some participants may be particularly strident in their positions or others can play a game with the voting rules to get what they want over the strong objections of others.

No matter which model is adopted it is important to design the rules for decisions in such a way that everyone has a say and that no sector on the decision making body can dictate a decision to others.

This can occur if a decision making body has 50 members and only 2 represent workers and 48 represent employers. If the rule requires a super majority of 90% it is still possible for all the employer representatives to out vote the labour representatives even if their opposition is very strong.

In the case of consensus defined as the absence of sustained opposition it is possible to give such latitude to the chair or an executive committee that they are able to declare a consensus that favours their position by announcing that the opposition is not sustained no matter what the opposition itself wants. 

No matter which system is used it is important the the underlying principle of consensus be respected by the rules and the participants. Everyone should be willing to support the decision even if they disagree with some elements in the standard.

Recognized body

The issue of a recognized body can be a bit more problematic. In some countries this could be defined in legislation, that is there may be only one 'recognized body' in the country or there may be a defined procedure that an organization must follow to become a recognized body.

Some people may even read the ISO definition as declaring that only ISO and bodies that ISO recognizes may develop standards. I will note that if this was the case that there is no way that ISO has the authority to enforce such a provision. ISO is a not-for-profit corporation based in Switzerland and while national governments may be members the organization has no authority to speak for governments. ISO produces standards that are voluntary. Even if a country adopts an ISO standard as a regulation (yes this does happen) this is the result of a decision by a government, not by ISO or any other standards development organization.

I will propose what I think is a better definition of a recognized body - that is a body that conforms to established norms for standards developers. This includes codes of practice like Annex 3 to the World Trade Organization (WTO) 'Code of good practice for the preparation, adoption and application of standards'; ISO Guide 59 'Code of good practice for standarization' and the ISEAL Alliance 'Standard Setting Code'. There are other guides and norms, some are national and some are specific to an industry (such as the FAO guidelines for certification of wild or farmed seafood).

Anyone developing a standard should understand the guidelines and norms that may apply to them and do their best to conform to them. This is the hallmark of a professional organization.