CSA scoring method has some pretty glaring flaws
The main issues brought up by the NAS report include “some BASICs lack correlation with crash risk, data insufficiency, use of relative rankings, use of non-fault or non-preventable crashes, state variations in inspections and violations, lack of consistency in violation coding, a lack of transparency of the SMS algorithm and the public availability of SMS rankings,” according to GloStone.
The DOT will need to make the SMS metrics more fair and accurate
A point to note from the NAS report is that they believe the premise behind the SMS is fairly solid; it’s the FMCSA’s execution of the program that leaves something to be desired. The DOT will need to make the SMS metrics more fair and accurate when it comes to assessing actual safety risk.
Recommendations for the SMS
Again, the idea of the SMS is sound, the main problem is when it comes to the execution of the SMS. “The Safety Measurement System is used to identify commercial motor vehicle carriers at high risk for future crashes. It’s the heart of the Federal Motor Carrier Safety Administration’s Compliance, Safety, Accountability enforcement regime, known as CSA. After numerous criticisms of the methodology from the industry, Congress called for the review of SMS as part of the Fixing America’s Surface Transportation (FAST) Act of 2015,” according to TruckingInfo.
Congress called for the review of SMS as part of the Fixing America’s Surface Transportation (FAST) Act of 2015
The current SMS metric fails to take into account some variables that play a bigger role in safety practices. Some of these faulty measurements include:
- Using highly variable assessments
- Not accounting for crashes where the motor carrier is not at fault
- Including carriers that have very different tasks in the same peer groups
- Using measures that are sensitive to effects from one or more individual states
- Using measures that are not predictive of a carrier’s future crash frequency
- Using measures that are not reflective of a carrier’s efforts to improve its safety performance over time.
Statistically Principled Approach
It’s easy to see that these oversights can lead to some bigger issues down the road. For that reason, the NAS study suggests that the current system takes a “more statistically principled approach” when it comes to collecting data. The NAS report recommends using latent trait theory or an “item response theory” (IRT) model. The IRT is the same approach used by hospitals for safety and performance rankings and helps to shape policy decisions.
“We have found, for the most part, that the current SMS implementation is defensible as being fair and not overtly biased against various types of carriers, to the extent that data on MCMIS can be used for this purpose,” said the National Academies panel.
SMS implementation is defensible as being fair and not overtly biased against various types of carriers
“However, we believe some features of SMS implementation can be improved upon, and some of the details of the implementation are ad hoc and not fully supported by empirical studies. Many of these details of implementation would be easily addressed if the algorithm currently used were replaced by a statistical model that is natural to this sort of discrimination problem,” they added.
Quality of Data
Another issue mentioned by the report is the poor quality of data. It’s recommended that the FMCSA continues to work with state departments and other agencies to improve the collection of data when it comes to miles traveled and crashes. Unfortunately, as it stands, this data is either missing or is of poor quality. Should the FMCSA be able to improve the quality of their data, the SMS will be able to take other factors such as environmental factors of travel which will give a better understanding of the crash conditions.
Unfortunately, as it stands, this data is either missing or is of poor quality
There are other, more obscured, data points that the report says should be included when collecting data on carriers, including driver turnover, cargo type, as well as method and level of driver pay. The panel suggests that driver pay is an important factor to consider especially when taking into account that better-paid drivers (those who aren’t paid based on miles traveled) tend to have fewer crashes.
What Does this Mean for the Industry
As you can imagine, the trucking industry has been waiting for NAS findings as it highlights all the issues they’ve had with the program for the beginning.
“This report has confirmed much of what we have said about the program for some time,” said American Trucking Associations President and CEO Chris Spear. “The program, while a valuable enforcement tool, has significant shortcomings that must be addressed, and we look forward to working with FMCSA to strengthen the program.”
we look forward to working with FMCSA to strengthen the program
If the FMCSA does decide to implement the suggested changes, then we can expect a more or less total overhaul of the CSA rating system. Now carriers with a mediocre level of safety performance can’t rely on poorer carriers to make them look good. Simply put, everyone is going to have to step up their game and start pulling their weight, safely.
Carriers with a mediocre level of safety performance can’t rely on poorer carriers to make them look good
In addition to providing more accurate and reliable data, carriers would also be able to get a better understanding of their scores as well as how to improve them.
Login