Science or Sciencey [part 2]

Part 2 of a 4-part series examining what happens when science is used for marketing (using brain-training software as the central example).

[part 1 | part 2 | part 3 | part 4]

[Full disclosure: I am a co-PI on federal grants that examine transfer of training from video games to cognitive performance. I am also a co-PI on a project sponsored by a cognitive training company (not Posit Science) to evaluate the effectiveness of their driver training software. My contribution to that project was to help design an alternative to their training task based on research on change detection. Neither I nor anyone in my laboratory receives any funding from the contract, and the project is run by another laboratory at the Beckman Institute at the University of Illinois. My own prediction for the study is that neither the training software nor our alternative program will enhance driving performance in a high fidelity driving simulator.]

In my last post, I linked to a blog post on the Posit Science website that training with the DriveSharp program leads to improvements in real-world driving performance. I originally found the post because it linked to one of my own videos and implied (but left unstated) that their training might also help overcome inattentional blindness. I agree that inattentional blindness likely plays a role in driving and driving accidents, but to my knowledge, no studies have shown that training can reduce the rates of inattentional blindness. The rest of the post was unrelated to inattentional blindness, though. It’s focused instead on the claim that training with DriveSharp software for 10 hours produces remarkable improvements in real-world driving performance.

According to the post, the technologies used in DriveSharp have been tested in clinical studies for 20 years. In 2008, Posit Science acquired another company, Visual Awareness, which owned the UFOV test, so that presumably is the studied “technology” contained in their DriveSharp program (Posit Science has only existed for a few years).

The UFOV is essentially a measure of the breadth of attention that incorporates a speeded component, and the links between the UFOV and driving have been studied extensively over the years by its creators, Karlene Ball and Daniel Roenker and others. So far, so good. The blog post makes a number of specific claims about the DriveSharp software based on those studies. Let’s evaluate each claim in light of the scientific evidence:

“Drivers with poor UFOV performance are twice as likely to get into automobile accidents.”

Accurate enough (depending on how you define poor performance). There are a number of studies showing that people who perform poorly on the UFOV are poorer drivers.

“UFOV performance can be improved substantially by DriveSharp training.”

Likely accurate, but not surprising—presumably DriveSharp incorporates the UFOV, so training with DriveSharp should improve performance on the UFOV. Practicing a task makes you better at that task even if it doesn’t lead to improvements on other tasks.

“Training allows drivers to react faster providing an additional 22 feet of stopping distance at 55 mph.”

Misleading—This claim is based on a statement made in a paper published in 2003 in the journal Human Factors by Roenker and colleagues. They showed that speed of processing training (part of the UFOV) led to faster responses in a choice response time task. Before training, subjects averaged about 2.2 seconds to make this sort of speeded decision, and after training they were 277ms faster. Roenker et al (2003) then converted the 277ms improvement into an estimate of stopping distance on the road: If a driver were traveling at 55mph and could hit the brakes 277ms faster, they would stop 22ft sooner. The study did not show any effect of training on actual stopping distance and these speed improvements were not measured in a driving context — the claim was based entirely on faster performance in a laboratory choice response time task. The analogy to stopping distance was used to illustrate what a 277ms response time difference could mean for driving.

There is no evidence that speed on a computerized choice response time task translates directly into faster responses when actually driving, especially when the need to stop isn’t known in advance. One bit of evidence suggesting that simple computer responses and driving responses are different comes from a study by Kramer et al (2007). When making simple response times or choice responses on a computer, young subjects are much faster than older subjects. However, older subjects respond just as quickly as younger subjects to warning signals in a driving context. A big difference in pure response speed doesn’t necessarily translate to a difference in driving performance. Claiming that these response time differences in a computer task translate directly into faster stopping when driving is misleading.

“Training reduces dangerous driving maneuvers by 36%.”

Accurate, but perhaps not as impressive as it sounds—This statistic also comes from Roenker et al (2003). Dangerous maneuvers were coded by driving instructors in the car with the subject, and it’s not entirely clear from the original article what constituted a dangerous maneuver. One section of the paper defines dangerous maneuvers as those in which the instructor either had to take control of the car or in which other cars had to alter their course to avoid a collision. However, another section suggests that dangerous maneuvers were coded based on the degree of danger felt by the raters at each of 17 locations during a road test. Although the such maneuvers appear to have been judged reliably across observers, the metric has a subjective component (especially for the second definition). In and of itself, the subjectivity of the judgment might not be an issue, but as we’ll see in the next post of this series, such subjectivity could be an issue if the people making the judgments were not entirely blind to the experimental conditions.

In the study, subjects averaged about 1 dangerous maneuver in about an hour of actual, on-the-road driving. The 36% improvement was based on a change from an average of 1.01 dangerous maneuvers before training to an average of 0.65 in a test 18 months later (the average was 0.69 immediately after training). With such a low rate of dangerous maneuvers, it’s possible that most drivers had no dangerous maneuvers at all and that a small subset had a large number of dangerous maneuvers. In other words, we don’t know how many subjects had any dangerous maneuvers at all, and it’s possible that most subjects had none at all either before or after training.

“Training reduces at-fault crash rates by 50%”

Not supported by published data—As best I can tell, no peer-reviewed scientific papers support this claim. The statistic is mentioned on the wikipedia page for the UFOV, where it is sourced to a conference presentation in 2009. To my knowledge, the Roenker et al (2003) paper and one other paper are the only ones in the scientific literature to conduct something approximating a clinical training trial comparing the UFOV to other forms of training, and there were far too few subjects (and accidents) to measure anything like at-fault crash rates. In fact, Roenker et al (2003) note that the rarity of accidents is one reason for measuring so many other aspects of driving performance rather than just looking at the rates of accidents. Conducting a training study and using accidents as an outcome would require a sample of many thousands of subjects to produce a reliable difference in accident rates.

“Benefits last a long time with significant improvements still measurable 5 years after training.”

Unclear or unsupported—It’s hard to determine the source of this claim, but the wikipedia page for the UFOV makes a similar statement and sources it to the ACTIVE trial, a large-scale study of the effects of cognitive training on self-report measures of daily task performance years later. The ACTIVE study did not directly measure driving performance and has produced relatively few documented benefits of cognitive training despite being sufficiently large in scale to find them (it has shown some intermittent effects on self-report measures of daily activities over the years). Again, practice with an arbitrary laboratory task might lead to some long-lasting improvements, especially for performance on that task, but the unsupported implication of the claim in the blog post is that published scientific evidence shows an improvement from 10 hours of training to driving 5 years later.

Summary
So, what is the scientific basis for the bold claim that DriveSharp will improve your driving? First, there is substantial evidence that the UFOV is correlated with driving performance, especially for elderly drivers. Second, there is evidence that UFOV performance improves with training. (The UFOV wikipedia page has a fairly comprehensive list of references for each of these claims, so I won’t duplicate them here.)

Note that these findings alone do not permit any claim of a benefit to driving of training with the UFOV. To see why, consider that ice cream consumption is correlated with the temperature outside. We can certainly inspire you to eat more ice cream, but that won’t change the weather. The UFOV and driving might be related even if the components of the UFOV play no causal role in driving performance. To make a causal claim like the one on Posit Science’s blog, you would need to show a direct benefit from training. Ideally, you would need to do what the Roenker paper attempted to do — contrast training on the UFOV and training on some other plausible task in a double-blind design.

Given the centrality of the Roenker et al (2003) findings for the claims in the DriveSharp blog post, my next post in this series will take a close look at the Roenker et al (2003) paper to see exactly what has been “proven” about transfer of training. After that, I will end the series by discussing the implications of such science-based marketing for the public consumption of science more broadly.

Sources Cited:

Roenker DL, Cissell GM, Ball KK, Wadley VG, & Edwards JD (2003). Speed-of-processing and driving simulator training result in improved driving performance. Human factors, 45 (2), 218-233 PMID: 14529195