Design Verification The Case for Verification, Not Validation

Transcription

1 Overview: The FDA requires medical device companies to verify that all the design outputs meet the design inputs. The FDA also requires that the final medical device must be validated to the user needs. When there are so many more design inputs and outputs than specific user needs, why do companies spend so little time verifying the device and so much time and money on validation? And what is the role of risk management in determining the amount of testing required? This presentation will demonstrate that if developers conduct more complete verifications of design outputs and risk mitigations, validations can be completed in a shorter time, more reliably, and more successfully. Introduction: In our experience working with medical device manufacturers to improve their Quality Management Systems and to gain regulatory clearance of new devices, we have found that Design Verification is an often underutilized tool for ensuring success during the latter stages of device development efforts. In many cases, limited verification efforts represent a lost opportunity to significantly reduce the scope of validation efforts and thereby reduce time to market and development costs. This paper explores the use and misuse of Design Verification and how device manufactures can get the most out of their verification efforts. But first, some background... Background: Medical device manufacturers have been working to align their design and development systems with the FDA s design control regulations (21 CFR ) since their release in The regulations are structured to ensure that device manufacturers maintain control over their designs throughout the development process and that the marketed device is safe and effective for its intended use. At their most basic level, the regulations require that manufacturers: Clearly state what they intend to produce (Planning and Design Inputs); Develop a design that meets those needs (Design Outputs and Design Review); Confirm that the design meets the original intent (Verification); Confirm that manufactured product can be produced reliably and achieve the desired result (Transfer and Validation); and Page 1 of 12

2 Maintain records of key development activities and decisions (Design Changes and Design History File) It would be hard to argue that any of these steps should be excluded from any development effort. Whether you are building a bridge, a cell phone, or a surgical tool, this combination of steps has a logical flow that most engineers would simply consider to be good practice. The challenge arises when you move from the conceptual world, where product development can be seen as single stream of water flowing down a hill: steadily moving from one point to the next without interuption; to the real world, full of rapids, eddies, branching streams, and (occasionally) deep pools of water that don t seem to be moving at all. In this turbulent environment, it s can be difficult to maintain a disciplined design approach particularly when business schedules demand rapid progress and any delays in moving on to the next development phase risk affecting project milestones and development staff deployment plans. These challenges are multiplied when the device has multiple components and integration points that must be managed throughout the development process. Much has been written about the importance of early stage planning and problem solving to speed time to market and reduce development costs and the value of clear Design and Development Plans and well defined Design Inputs cannot be overstated. An equal amount of attention has been focused on Design Transfer and Validation activities. At this late stage of a development effort, the project scope will have expanded to include more active involvement of Clinical, Operations and Marketing staff. In addition, the cost of validation studies and the make or break aspect of these studies require a great deal of attention and company resources. Problems at the validation stage will have serious implications for the success of the project and especially for small companies could determine the fate of the whole company. Compared to these early and late stage activities, the importance of Design Verification tends to be overlooked. There are several reasons why: Definitions: Many practitioners (and established Quality Systems) continue to have trouble defining exactly what verification is, and how it fits into the design control process. The term is often considered to be synonymous with validation and the development of combined V&V plans that do not establish a clear distinction between the objectives of the two efforts don t help. [Note: when discussing verification with the FDA, it s best not to talk about your V&V plan increasingly, inspectors preferences are to address the two activities separately.] Timeframe: Verification is often an ongoing effort conducted throughout the development of outputs. So, a great deal of time may pass between when the first and last output is verified. Page 2 of 12

3 With this extended timeframe, verification activities can get lost within all of the churning associated with developing final outputs. Approaches: There are a variety of ways to complete the verification for an output, so it can be difficult to communicate that all of these activities, as a group, represent the design verification. The problem with this lack of attention is that it weakens a critical link in the design control chain, affecting the strength of the overall design control effort. Poor design verification can lead to problems during design transfer and validation, and can reduce your ability to track down and correct problems if (and when) they occur. Just as importantly, it may represent a lost opportunity to optimize validation activities, reduce time to market, and increase your overall confidence in the safety and efficacy of your products. Key Concepts: To make sure that we re aligned on the proper use of the term verification, here are a few key points from the FDA s Design Control Guidance For Medical Device Manufacturers: Definition: Verification means confirmation by examination and provision of objective evidence that specified requirements have been fulfilled [ 820.3(aa)]. Types of Verification Activities: Verification activities are conducted at all stages and levels of device design. The basis of verification is a three pronged approach involving: tests, inspections, and analyses. Any approach which establishes conformance with a design input requirement is an acceptable means of verifying the design with respect to that requirement. In many cases, a variety of approaches are possible... the manufacturer should select and apply appropriate verification techniques based on the generally accepted practices for the technologies employed in their products. Table 1 provides some examples of the types of verification activities that device manufacturers often employ. Page 3 of 12

4 Table 1: Types of Verification Activities Tests Inspections Analyses o Material performance / o First article inspection o fatigue tests o Comparison of a design to a o Package integrity tests* previous product having an o Biocompatibility testing of materials* established history of successful use* o Bioburden testing of products and use* to be sterilized* o o o o Worst case analysis of an assembly to verify that components are derated properly and not subject to overstress during handling Thermal analysis of an assembly to assure that internal or surface temperatures do not exceed specified limits* Fault tree analysis of a process or design* Failure modes and effects analysis* Engineering analyses: o Finite Element Analysis (FEA) o Computational Fluid Dynamics (CFD) o Tolerance stack-up * Source: Design Control Guidance For Medical Device Manufacturers (1997) A key distinction between design verification and design validation activities is that verification only requires that a single unit be assessed. What constitutes that single unit will vary depending on the intent of the verification. It might be one batch of raw material (for material performance tests), one machined part (for first article inspection), on one package sample (for integrity tests). The intent of verification is to confirm that the design outputs (i.e., the materials or components specified in design documents) meet the design input requirements. Validations, on the other hand, require that studies be conducted to ensure that the device can be manufactured to meet design specifications on a consistent basis (i.e., process validation), and to ensure that the finished device is safe and effective for its intended purpose (i.e., design validation). For the process validation, multiple devices must be manufactured and evaluated to confirm that the production process is capable of producing devices within specifications on a consistent basis. For the design validation, multiple devices must be used to treat multiple patients to confirm that the treatment is safe and effective. Page 4 of 12

5 The number of devices that need to be produced and the number of patients that need to be treated is a function of the risk associated with the particular aspect of the device and the need to establish confidence in the study results. As we will see, the effective use of risk analysis tools and solid verification results can help to reduce the size of the validation studies without negatively affecting the confidence in the study results. What Goes Wrong? As described above, too often design verification does not receive the amount of attention needed to ensure success in the latter stages of a device development program. While not a perfect representation of what goes on in all device companies, it is interesting to look at the FDA s design control audit findings to see what problems they have found with companies design verification programs. Figure 1, shows the number of warning letters in the FDA s database that include findings related to specific sections of the Design Control regulation (21 CFR ). Figure 1: Design Control Warning Letters Page 5 of 12

6 Interestingly, the most observations were found with regard to the Design Change, Design Validation, and the General design control sub headings (the General findings suggest an overall failure of the company s design control process). We argue that failures in these three areas are largely the result of poor performance in the earlier stages of the program (e.g., validation failures due to poorly structured input requirements, outputs and incomplete verification; and design change failures due to an ineffective system to update verifications and validations when changes are made). Of the remaining categories, most warning letters cite 21CFR (f) Design Verification. In these 64 warning letters, the FDA identified that the manufacturer did not complete all required verifications in nearly 45% of the cases. A lack of (or significant gaps in) verification procedures were identified in about 25% of the warning letters. In addition, out of specification results, a lack of records in the DHF, and a lack of acceptance criteria were found in 16%, 14%, and 14% of these companies, respectively. Figure 2 provides a breakdown of all of the types of violations identified in these warning letters. Figure 2: Types of Design Verification Violations Page 6 of 12

7 Note: 92 separate violations were identified in the 64 warning letters. Why are these failures occurring? One possibility is that by viewing verification as a regulation, device developers are losing sight of the fact that conducting verifications is simply good engineering. For example, it makes no sense to move forward with the development of a component before you are sure that the material properties meet the performance and safety requirements established in the design inputs. This meeting the regulations mindset can lead engineers to view design verification simply as a task that must be checked off by QA instead of valuable tool for building knowledge about their device. Failures occur because sometimes the checkoffs get missed. Even when all the boxes are checked, the regulation mindset can drive engineers to focus only the minimum number of verification activities needed to satisfy the design control requirements. While this approach may help satisfy short term budget or schedule constraints, the development tea m will lose the opportunity to learn potentially important information about the performance and behavior of its device and its components. If that learning is put off to the validation stage, the cost of failure can grow dramatically. What To Do? There are a variety of tools that developers can use to improve the effectiveness of their design verification process. In this paper we discuss three tools that can help ensure that verification activities are appropriate and complete and, if well documented, can provide regulators with sufficient confidence in the design that excessive validation can be avoided. The three tools are Traceabilty Matrix Failure Modes and Effects Analysis (FMEA) Fault Tree of Design Inputs Traceability Matrix: The traceability (trace) matrix is a basic design control tool that all device developers should use throughout the design control process to establish clear linkages between Design Inputs, Outputs, Verifications, Validations and Risk Analyses. Too often the trace matrix is left for QA to complete just in time for the final design review. However, if it is begun as soon as Design Inputs are approved and managed throughout the development effort, the trace matrix provided an excellent roadmap guiding developers through key steps of the design control process and ensuring that required documentation is created and controlled. Page 7 of 12

8 To illustrate how the trace matrix can be used, Table 2 shows what one row in a trace matrix might look like when the matrix is first developed and approved. At this stage of the development effort, the matrix is solely a list of the design inputs (the reference number from the Product Requirements Document, and a description). At this stage there is no information about how that design input will be satisfied, but it provides a roadmap for what questions need to be asked and what documents need to be developed. As the project proceeds through the develop stages, subsequent cells will be filled in, identifying how the design input is being met. Table 2: Traceability Matrix at Design Inputs Stage Traceability Matrix Design Design Input Req. (Requirement) No Can be ETO sterilized Design Output (Specification) Risk Analyses Verification Process Validation Design Validation Comments Table 3 illustrates how a row of a trace matrix might look when being reviewed at the Final Design Review. At this point the design outputs have been approved and the requirement that the material in question is appropriate for ETO sterilization is established in Drawing No The matrix also identifies that identified risks associated with the device have been mitigated (in part) through the use of this material. The references to design FMEA 1.3 and process FMEA 5.4 identify two places where this material is addressed. Verification was achieved by confirming that the specified material is included in the purchasing bill of material (BOM 2.3). The Process Validation column identifies that production of the material was assessed in an operation qualification (OQ Study 3.5) and a performance qualification (PQ Study 4.3). Finally, it shows that a design validation (Sterilization Study 2.1) was conducted to confirm that the final manufactured device could in fact be effectively sterilized using ETO. Table 3: Traceability Matrix at Final Design Review Traceability Matrix Design Design Input Req. (Requirement) No Can be ETO sterilized Design Output (Specification) Material A (Dwg 1.2) Risk Analyses dfmea 1.3 pfmea 5.4 Verification BOM 2.3 Process Validation OQ Study 3.5 PQ Study 4.3 Design Validation Sterilization Study 2.1 Comments None Page 8 of 12

9 The trace matrix now serves as a clear record of the chain of analyses and studies used to ensure that the specific design input is met. All of the documents referenced in the trace matrix are included in your design history file (DHF), so if a question arises regarding any stage in the process, the trace matrix points to the key documentation developed at that stage. While the trace matrix is a necessary and valuable tool, it largely serves just a recordkeeping role. It does nothing to inform the team about how to prioritize verification and validation efforts. For that input, risk analyses are needed. FMEA: Like the trace matrix, risk analyses need to be started early in the design and development process and filled in and enhanced as project proceeds. While this paper will not discuss the details of the application of risk analyses, and FMEAs in particular, we will address the link between FMEAs and verification. Too often, risk analyses are conducted in isolation from the other design control activities. As discussed earlier, the regulation mindset can lead to the perspective that risk analyses are just a required task to be completed and not as tools to support decision making which they are. FMEAs are conducted to provide developers with a shared understanding of the risks associated with their device usually from three perspectives: design, use/application, and processing. The result of this tool, as illustrated in Figure 1, is the classification of risks to people, property, and the environment into three categories: Broadly Acceptable, As Low as Reasonably Practicle (ALARP), and Intolerable. Typically, application of this tool focuses on the Intolerable risks and the development of mitigation strategies to reduce the probability of occurrence for those risks, bringing them into the ALARP region. Page 9 of 12

10 Figure 2: Risk Classification While risk identification and mitigation is clearly the primary objective of the risk management activity, these analyses can also be used to focus verification and validation efforts. For example, consider a device that includes two components, one affects how the device is held by the user and the other comes into contact with the patient. Both components have a similar number of physical dimensions to verify, but risks associated with the user facing component are Broadly Acceptable, while the patientfacing component risks are ALARP. While a First Article inspection of the user facing component may be appropriate, a more thorough analysis of the patient facing component may be warranted. This analysis could include a tolerance analysis or the inspection of multiple pieces to provide a better indication of the potential capability to produce the component within specification. While the additional efforts described above may not be required at the verification stage (conducting a First Article would be sufficient to claim that the verification is complete), the ALARP risk classification is an indication that validation of this component may be challenging. The more you learn about this component during verification, the better prepared you will be once you get to the validation. If fact, if the verification is thorough, it may eliminate the need for some aspects of validation saving time and money later in the development effort. The key point is that incomplete or minimal verifications may not provide a sufficient understanding of the likely performance of a device during validation, and excessive verification can be a waste of resources. To effective focus your verification efforts, risk analyses, and FMEAs in particular, can provide Page 10 of 12

11 a clear rationale for how to focus your time and resources potentially reducing validation requirements. Fault Tree Analysis: While FMEAs are a well accepted risk analysis tool, one weakness of the approach is that each risk is considered in isolation. There is no ability to assess the risk of two independent failures (i.e., dimension A is too short and the user applies too much force). As illustrated in Figure 3, Fault tree analysis (FTA) allows you to consider the risk of two independent failures occurring in parallel (the AND ), and the risk of any one of a series of failures (the OR ). For example, in this Bad Coffee example, adding Old Cream requires the cream to be past its expiration date AND for the coffee drinker not to read the date. Unsatisfactory serving temperature could be due to the coffee being either too hot OR too cold. Figure 3: Fault Tree Analysis While more challenging to structure and assess than an FMEA, the FTA provides a better understanding of the risks associated with the system. Since the performance of the system is the objective of validation activities, FTA is an ideal tool to prepare for validations. With the top of the fault tree representing the intended use of the device, this tool allows device developers to fill out the tree Page 11 of 12

12 identifying that range of design or usage faults that could lead to a problem. Just as the FMEA helped to focus attention on independent risks, the FTA helps to focus on system risks. Once the branches of a tree associated with a significant risk are identified, additional verification resources can be focused on verifying the design of components linked to that risk. Again, the objective of verification and validation activities is to build confidence within your organization and with regulators that your device is going to be safe and effective. A well structure FTA combined with thorough verifications of key components can be instrumental in the identification and resolution of problems early in the development process, allowing your team to enter into the validation phase of the process with confidence in the performance of the device and to focus its validation on those system elements that most directly affect user needs. What are the Benefits? While it can sometimes get lost in the churning of engineering process, verification is a critical element of the design control process. While meeting the threshold requirement of documenting a verification for each design input may help to move the design through the development stages, such an approach does not provide the value that a strong verification effort could provide. By developing a trace matrix to ensure that all design inputs are properly addressed and leveraging risk analysis tools, medical device developers would be better able to focus scarce resources on those verification activities that will provide the greatest benefit. Well structured verification activities provide the foundation for validations. If the verifications are sound, validations can be better focused, helping to reduce the scope of these activities reducing validation costs and accelerating time to market. Page 12 of 12