AS 9101D (Draft) – A Review of how AS9100C will be Audited

A draft version of AS9101D is out for final review before publication. It provides tremendous insight as to how the new AS9100C standard will be audited. This is an excellent document which really only describes normal/best auditing practice but hopefully it will revolutionize AS9100 and then probably the rest of the ISO industry. It mandates behavior that is likely to ensure better and more effective auditing – so long as the auditor and registrar community is capable. Training has been mandated for auditing the new AS91000C standard in an attempt to ensure competency and it will to some extent be based on AS9101D – despite the fact that the training is already being developed.

Here is a summary of the key points and a link to the full review of this draft.

Process Approach — Strong emphasis on an understanding and use of a process approach by organizations. If your documents are not process based you should perhaps worry.

Auditing Methodology — Strong emphasis on 6 specified methods of auditing which outline key issues which are “recommended” to be included in an audit.

More audit time needed — It is “expected” that the previously specified “total” audit time is now applied as “on-site” time. Time will be added for off-site activities which are also about to increase. AS9101D specifies lots of extra information needed prior to audits and additional information and activities during the audit which is going to ensure a thorough and effective audit – but also a longer and more intensive audit.

Objective Evidence Record (OER) — although there was talk about removing the laborious checklist that was previously mandated – it is still there in the current draft standard. While this is potentially a barrier to process based auditing (which will cause much confusion to some auditors) it is the only way that anyone will be able to see that the auditor has done a good job of assuring every requirement of the standard has been audited. Without this or an equivalent audits will simply fail.

Process Effectiveness Assessment Report (PEAR) — Auditors are now required to assess the effectiveness of processes in an organization. Not all processes will be subject to this assessment but certainly all important ones. This is an obvious need during an audit but this is going to be beyond the ability of many auditors who just audit to checklists. The way to ensure you stay clear of problems is to do this on behalf of the auditor and make it indisputable.

Perhaps the final note to mention here is that there are rumors that AS9101D publication is delayed. This wont cause a problem to the overall progress of AS9100C release and upgrading as the aforementioned sanctioned training is not expected to be “available” until April. Unfortunately nobody really knows the timescales as the key milestones are dependent upon the results of committees voting.

Here is a more detailed description of what you might have to do and how your audit might be performed.

The Process Approach

The process approach has always been “out there” as a concept when developing management systems. Cavendish Scott, Inc. has been using the process approach since 1985. We didn’t call it that but it just made sense that the right way to build a management system was to structure it around the organization and not copy the requirements of a standard. Unfortunately many organizations saw fit to base their management system on the standard, structuring it around the numbering and titles of the standard and not really describing their own processes. Somehow this was satisfactory for the registrars. In 2000, ISO tried to rectify this by describing the process approach. Then they back-tracked declaring that although a process approach is contained within their model, they were not prescribing any method and that organizations already ISO registered would not need to rewrite their procedures. Currently ISO is in decline (there are less certificates in the US than there were last year) and one of the reasons is that organizations saw little benefit – and so did their customers. Part of the blame for this is that a system based around the requirements of a standard is difficult to maintain and is unlikely to yield benefits. Further, registrars were reluctant to ask organizations to change – for fear of losing business and accreditation agencies too didn’t want to rock the boat. Now for instance the accreditation agencies are getting tough on root cause/corrective actions despite the fact that this has been an issue since the original issue of the standard at in 1987. However, the accreditation agencies still don’t press registrars to insist on a process approach.

Clearly the aerospace industry is not putting up with the inadequacies of the accreditation or registration processes. AS9101D is their attempt to address the process model issue. This is a direct quote from the draft.

n0.2 Auditing Approach

This standard supports the engagement and evaluation of an organization’s quality management system process approach, as required by the 9100-series standards. When evaluating an organization’s quality management system, there are basic questions that should be asked of every process, for example:

Is the process identified and appropriately defined?
Are responsibilities assigned?
Are the processes implemented and maintained?
Is the process effective in achieving the desired results?

This, the new auditing methodology (see details later) and requirements throughout the AS9101D standard are pressing for organizations to understand a process approach. If your system is based on the standards requirements it does NOT demonstrate that you understand the process approach, implemented processes and process based control. Given this it is hard to understand how an auditor will not generate nonconformances on the effectiveness of the system. If there is no procedure (or flowchart or process diagram or something) describing your production process, for instance, then you have not met the 4.1 requirement to define your processes. Whether nonconofrmances get generated remains to be seen. Auditors have accepted poor process definitions for years. Given that without good process definitions, measurement, objectives and improvement are not easy to achieve, these nonconformances are long overdue.

Auditing Methodology

This standard identifies 6 approaches that “CAN” be used to conduct audits. The note following that statement implies that this is expected as a minimum. This probably means that registrars cannot ignore the methods or they will get in trouble. They will have to generate tools to help them ensure the audit utilizes these methods (as a minimum) and generates evidence to prove they have. It is complicated to integrate these methods into the audit approach currently used which almost exclusively relies on the existing AS9100C checklist. Consequently expect some registrars and/or auditors to simply do these methods/approaches in addition to the checklist. Perhaps the mandated training will address integration of these methodologies with the checklist but even so it is a difficult concept.

The following extract includes the introduction to this Audit Methodology and then lists the he six methodologies by title and gives a brief (not verbatim) description including extracts. Some of the methodologies include lengthy descriptions with many items listed. Here they are summarized to give an example of what they are about.

4.1.2 Audit Methodology

The following approaches identify different audit methods that can be used, as appropriate, to conduct audits.

NOTE: The identified methods are not intended to be a complete listing, but represent a significant contribution for auditors to evaluate quality management system conformity and effectiveness. Use of these methods will help transition from clause based auditing and put focus on the actual processes, their effectiveness, and their ability to meet the quality objectives.

4.1.2.1 Customer Focus

“The audit team should assess whether customer satisfaction is adequately evaluated and appropriate actions are taken”. “Customer feedback is a process and should be audited as a process” – these extracts from the section highlight where this audit methodology focuses.

4.1.2.2 Organizational Leadership

There should be an interview(s) with top management to evaluate the establishment of policy, objectives and about commitment. - This is not the only place where an “interview with top management” is discussed. It is hard to believe but perhaps some auditors have not been insisting on meeting top management. It will be hard to avoid now.

4.1.2.3 Quality Management System Performance and Effectiveness

Methodology subjects include a review of the processes for complaints, internal audits, stakeholder feedback, nonconformances, preventive actions, management review meetings and performance to targets. These subjects are going to be addressed in many other areas (the methodology does not say how they will be met). The key will be to ensure they are linked where appropriate and evidence complete to demonstrate the methodology has not been ignored.

4.1.2.4 Process Management

“The audit team should conduct quality management system audits using a method that focuses on process performance and effectiveness; this ensures that priority is given to the following:

a) reviewing the organization’s processes, their sequence and interactions, and performance against requirements …..”

Also mentioned, the auditor is to look for defined measures with a focus on processes that directly impact the customer.

4.1.2.5 Process Performance and Effectiveness

Measurements that indicate or directly measure the effectiveness of processes. The example quoted is KPI’s (Key Performance Indicators). In truth ISO has always stressed the importance of this.

4.1.2.6 Continual Improvement

“The audit team should evaluate the organization’s interrelated processes and activities for continual improvement of the quality management system, its processes, their conformity, and effectiveness in order to:

• ensure focus on issues that are important to the organization, their customers”

Many other items are listed in the continual improvement methodology but this is where the focus is. Note also that improvement is a process and not incidental and unrelated improvements that might have occurred. And the focus is on things that are important.

More Information Required by Registrars

Before the audit, registrars must obtain more information in order to be able to “take into account” results of internal audits, performance measures, previous Management Review results and the proportion of aerospace (and defense) customers compared to total revenue.

All of this information must be obtained before the audit starts so that planning can be conducted appropriately. Registrars will have to spend more time on this activity (and it will have an impact on their customer service, response and probably costs.

During the Stage 1 audit they are required to review risk assessment methods, customer satisfaction data, special processes, preventive programs e.g. FOD, in addition to everything that they normally cover in a stage one (e. g. internal audits). Again more time and perhaps more cost (but this should assure audits that get to stage 2 are more successful).

Stage 2 Audits are required to include a review of all new [aerospace] customers, a review of customer satisfaction, special processes, continual improvement of the QMS and an interview with top Managemnet.

This is not particularly a significant increase over previous “expectations” although some additional effort will be required.

Process Efficiency Assessment Report (PEAR)

This requirement is new. It is hoped that sanctioned training will include substantial training for auditors as judging the effectiveness of a process that you get to see for a matter of minutes (rarely hours) is quite a task. However, the form that records the PEAR provides a simple approach and this is where well organized companies will address this issue before the audit and make the auditors life easy. Choosing the right measures for effectiveness will be important. Inappropriate measures are likely to be dismissed but measures that take too much effort may be unpopular. The following extract outlines the importance of the PEAR.

“The results of effectiveness shall be recorded on the PEAR (see Appendix C) for each audited product realization process. The level of effectiveness for each process shall be recorded on the PEAR (statement of effectiveness level). If the level of effectiveness has been classified as a ‘2’ or a ‘1’, this shall result in a nonconformity being issued against 9100-series standards clauses 4.1.c and f (see clause 4.2.4).”

Note that if you are issued a PEAR with a level 3, it is a reasonable expectation that during the next audit, that process will be at a level 4. “Appropriate” actions should ensure that.

The following is a listing of the key fields in the PEAR form. At the bottom is the classification of the effectiveness level mentioned above.

Process Name:

Process details, including associated process interfaces:

Applicable 9100/9110/9120 clause(s):

Organization’s method for determining process effectiveness:

Auditor observations and comments supporting process effectiveness determination:

Statement of Effectiveness Level:

The process is:

Not implemented; planned results are not achieved.
Implemented; planned results are not achieved and appropriate actions not taken.
Implemented; planned results are not achieved, but appropriate actions being taken.
Implemented; planned results are achieved.

Objective Evidence Record

Although there were rumors that the checklist was going to be removed, it is still included in the current draft although in a different format. This is a good thing – despite what many auditors claim. Without this, there is insufficient evidence to demonstrate what was audited. If any problems occur it would be impossible to follow up and see what the auditor looked at and possibly clear them of blame. Registrars are allowed to have their own checklist so long as it meets the same intent.

Nonconformances

“Recurrence of the same or similar nonconformity found during consecutive audits at a particular site/location shall be considered as a failure of the corrective action process (see 9100-series standards clause 8.5.2) and shall result in a major nonconformity being issued”

Inadequate corrective action will no longer be tolerated in aerospace quality management systems.

Conclusion

Many of the issues introduced here represent a substantial change in auditing content and approach. In many respects this is disappointing because what the standard is calling for is mainly good auditing practice by competent and capable auditors. Auditors are witnessed and it is possible that many auditors will be prohibited to audit if they are unable to demonstrate best practice auditing. This too is a good thing as an industry as important as the aerospace industry deserves good auditors to ensure effective quality and to drive improvement up and costs down. That is not the case at the moment and this standard has the opportunity to make substantial, meaningful and beneficial change throughout the aerospace industry and the ISO industry in general.

Organizations will need to carefully prepare for AS9100C. Not necessarily because the standard has changed much but because the auditing approach is getting more sophisticated and better. This will represent some additional cost and effort for organizations but with the loss from the supplier base of those that didn’t do it, and the potential benefits that can be achieved with ISO/AS with the right focus, companies who put that effort in early are likely to reap the benefits.

Cavendish Scott, Inc. invests time and expertise in ensuring AS9100C consulting projects are successful. We are professional about our understanding and design consulting approaches that guarantee success. For more information about what we might do and how we might help you upgrade your AS system, contact us.