“Survey says … ”
Contrary to what is advertised on the InHealth website, I do not believe this study meets the modern standards of “credible research and analysis”. It also makes me wonder about the peer-review process at J. Med. Dev. and Stanford University. We are no longer in the dark ages of social science research. The current expectation is that it will be equally rigorous with physical and biological research; otherwise, it has no place being in the public debate on science and policy. Of course we need to support our small and start-up medical device firms; they are engines of invention, innovation, and job-creation. But, given our limited resources, we need to do it effectively and efficiently, based on valid scientific evidence. Since this article suggests implied causality, it is fair to invoke the four fundamental criteria for an honest scientific study: internal validity, conclusion validity, construct validity, and external validity. We’ll take each, one at a time, and consider an example or two (just to keep it short). We won’t even discuss why there are no error bars on the percentage values (preventing us from evaluating whether reported differences are real differences).
INTERNAL VALIDITY refers to the degree to which an experiment shows a cause-effect relationship between independent and dependent variables. An obvious problem is that respondents self-selected and the sample is non-random; this is called sampling bias. This is often the case in social science research, but there are means and methods for mitigating such problems (e.g., propensity scoring, Heckman correction); there is no evidence that the authors even recognized the problem, much less tried to mitigate it. It appears from the text of the article (§2, §4.3) that the authors were in part focused on the “availability of guidance documents” and then conclude (§5) that “Improved agency guidance … can contribute to reducing the duration of the 510(k) process”. This is called confirmation bias and is the distortion of results by experiments intended to prove, rather than disprove, a conjecture. We would all love more guidance documents, but let us not forget that production of more guidance documents by review staff will result in fewer hours spent on reviews … and longer times to market.
CONCLUSION VALIDITY is the degree to which conclusions we reach about relationships in our data appear reasonable. The authors’ state: “Worst case consequences may include delayed patient access to safe and effective devices”. But, they could equally state “Best case consequences may include delayed patient access to unsafe and ineffective devices” (metal on metal joint replacements and mesh products come to mind; all were originally touted as important innovations). They also state “whether these difference are attributable to small companies being less informed about the regulatory process or the FDA being more effective in communicating requirements to large companies versus small companies”. They provide no basis for this apparently arbitrary choice; a more rational conclusion is that small companies do not have on staff the experienced personnel or breadth of scientific knowledge to fully understand the requirements for supporting safety and effectiveness. I think most folks in the FDA would be the first to agree they are far from being maximally effective (are you?) and that we would all benefit from a proliferation of guidance documents. But, readers of the guidance documents do need to really understand the meaning of the scientific terms, be able to relate them to the underlying scientific and engineering concepts, and then apply them correctly. When this has not typically been the case with existing guidance documents and international standards, why should more be the remedy?
CONSTRUCT VALIDITY refers to the degree to which inferences may be legitimately made from the operational definitions to the theoretical constructs. The authors provide no operational definition of their metric “predictability”, so we can assume they did not define it for the participants, in which case “predictability” can mean anything from “inability to include the regulatory process in my project plan” to “the regulators did not do what I expected or wanted them to do”. Undefined metrics have no place in purportedly scientific studies. They also state: “Taken together, all of these responses suggest that certain process inefficiencies may disproportionately impact small companies” (presumably, they mean “adversely”). Their implicit underlying construct is that small and large companies are commensurable, but actually they cannot be measured by the same standard! Just like their high power medical, legal, financial, and accounting talent, large companies have high power regulatory talent. In my experience, most small companies seem to struggle with the concepts of Design Controls, modern Software Development Processes, and quantitative Risk Management. Until recently, Human Factors never even made it onto their radar. My experience is their medical device mechatronics maturity is typically, and not unexpectedly, quite low.
EXTERNAL VALIDITY refers to the degree to which the study results may be generalized to the rest of the medical device industry. The authors state “product developers would gain significant benefit from FDA efforts to increase the transparency of testing requirements”. Since a significant component of testing for 510(k) clearances is based on consensus standards (typically, international consensus standards) and individuals working in the medical device industry (typically, medium to large firms) are very well-represented in these decision-making processes, this conclusion will not generalize to small firms and start-ups. It is highly unlikely that small firms and start-ups will ever have the resources (time, personnel, and funds) to engage in the time-consuming, voluntary participation required. The authors further state “Differences between the responses … the 510(k) process poses particular challenges for small companies”. Small companies face many challenges (access to capital, access to potential customers, getting paid on time, affordable healthcare for employees, transitioning to a medium size company, etc.). Those that voluntarily choose to work in a regulated industry need to factor in the costs of regulation, in addition to the benefits of reduced competition in the medical device space.
For our clinician colleagues, the foregoing is analogous to a differential diagnosis; make your own judgment regarding the scientific merit of this study, before consider its recommendations as a legitimate therapeutic intervention.
GM Samaras Pueblo, CO

By GM Samaras on Jun 19, 2012