The Limits of Statistics: Insights from Deming

If you, like me, are interested in improving safety in your workplace, it's important to understand different types of studies that can be conducted and their limitations. W. Edwards Deming, for instance, already distinguished two main types of studies:
-     An "enumerative study"
-     An "analytic study"

Management by LTI — like driving a car by looking in rear view mirror
An enumerative study involves estimating e.g. the number of accidents or hazards that exist within a specific workplace. Deming wrote about two kinds of errors in statistical inference:
1.   adopting a new process and regretting it later;
2.   holding on to an old process and regretting it later.
Statisticians attempt to minimize net loss from both kinds of mistakes, but they face challenges because it is impossible to predict future environmental conditions and performance with certainty.

Injury rates are often used, also to measure the effectiveness of interventions, but injury rates have no correlation with fatalities, and it requires a large number of people to be harmed before a difference can be observed. It's important to use other measures.

Learning is not compulsory… neither is survival
An analytic study can be used e.g. to identify latent conditions causing harm to develop interventions that will improve future performance.
When measuring the effects of interventions, Deming advised to be really clear about factors such as:
1. "the frame": on what aggregate of tangible, identifiable physical units is the investigation focused? This question is important because most troubles and most possibilities for improvement belong to the system.
2. what environmental conditions may affect the results?

There is no substitute for knowledge
To be able to answer the questions above, Deming emphasized the importance of knowledge of the subject matter to fill in the gap beyond statistical inference.

For the statistician, Deming also had some guidance:
“The statistician in practice must write a report for management or for legal purposes, on the statistical reliability of the results, what they may mean and what they don’t mean. (...) Involvement in a problem means the possibility of facing a board of directors, or facing cross-examination. It means tedious work, such as studying the data in various forms, making tables and charts and re-making them, trying to use and preserve the evidence and to be clear enough to the reader: to endure disappointment and discouragement.”

Let's conclude with a quote: "He that would run his company on visible figures alone will in time have neither company nor figures"

Deming, W.E. (1975), On Probability As a Basis For Action, in: The American Statistician, November 1975, Vol. 29, No.4.
Deming, W.E. (1986), Out of the Crisis, Cambridge: MIT CAES.
Hallowell et al (2020), The Statistical Invalidity of TRIR as a Measure of Safety Performance, Construction Safety Research Alliance.