Science or Sciencey [part 4]

The final part of a 4-part series examining what happens when science is used for marketing (using brain-training software as the central example).

[part 1 | part 2 | part 3 | part 4]

[Full disclosure: I am a co-PI on federal grants that examine transfer of training from video games to cognitive performance. I am also a co-PI on a project sponsored by a cognitive training company (not Posit Science) to evaluate the effectiveness of their driver training software. My contribution to that project was to help design an alternative to their training task based on research on change detection. Neither I nor anyone in my laboratory receives any funding from the contract, and the project is run by another laboratory at the Beckman Institute at the University of Illinois. My own prediction for the study is that neither the training software nor our alternative program will enhance driving performance in a high fidelity driving simulator.]

The Posit Science post on the effectiveness of DriveSharp emphasized the scientific backing for their training regimen. Over the past few days, I have been examining the claims and the research underlying them. In this final post of the series, I will attempt to draw some broader conclusions from this analysis of science and the use of science in marketing. Next week, I will add an “afterward” discussing some of the comments I receive about the post series and also examining a couple additional papers on the topic.

I greatly admire attempts to examine the ways in which scientific tasks can be brought to bear on real world problems, and the Roenker et al (2003) paper in Human Factors was an important first attempt to test whether training on a basic, laboratory measure of attention and perception (the UFOV) could enhance real-world driving performance. The study was promising, showing what appears to be a reduction in dangerous maneuvers by subjects who were trained to improve their speed of processing. Without such clinical-trial training studies, there is no way to determine whether improvements on laboratory tasks generalize to real-world problems. That is why clinical-trials with double-blind assignment to conditions are the gold-standard for determining the efficacy of any treatment, including new drug therapies and medical interventions. Without meeting those rigorous experimental conditions, claims that a treatment definitively causes an effect are unsupported. Few studies in the brain-training literature even attempt to meet these requirements, and even fewer actually succeed. The Roenker et al (2003) study was a good first attempt, even though it had some shortcomings.

Any clinical trial, of course, has limitations that can affect whether or not causal claims are merited and also whether the results generalize to other situations or populations. This initial study was limited in scope, tested only elderly and already-impaired subjects, and was subject to some possible concerns about coding objectivity and subject motivation. None of those limitations are surprising, but they do raise the need for further study. That’s especially true given that relatively few studies in the training literature show any transfer at all beyond the specific tasks trained, and even fewer studies show generalization from laboratory tasks to practical, real-world performance on untrained tasks.

The problem, here, isn’t with the science. Science is a work in progress and any study has its limitations. The problem comes when an initial, promising result is taken to be definitive “proof” or when speculative claims are treated as scientific fact. The Posit Science blog took these results as proof, without any qualifications or discussions of the limited scope and generalizability of the findings. It also presented what was a speculative analogy illustrating the potential importance of faster responses (faster choice response time translated into shorter stopping distances) as evidence for actually improved stopping distances.

Could training on the UFOV improve driving performance in general? Possibly, but that’s not what the study showed. It showed improvement on just one of many outcome measures for one type of subject. If training has such a big effect on driving, it’s actually somewhat surprising that only one of the measures showed any benefit at all from training. That’s a far cry from proving the benefits of training for driving in general or for the population at large.

I have written about this example of marketing primarily because it comes from a company, Posit Science, that prides itself on backing its programs with science. And, the fact that most of their programs are connected in some way to published research and award-winning researchers lends an air of scientific credibility to their marketing claims. It makes them sciencey. People who lack the training (or time) to examine the underlying science will be inclined to trust the claims of organizations with the imprimatur of scientific credentials in the same way that people tend to believe that studies with pretty pictures of brains are more scientific. The science might eventually back some of the stronger claims, and I truly hope that it does, but unlike science, marketing typically does not mention the limitations, problems, or shortcomings of the studies.

People are already inclined to believe that simple interventions can produce dramatic results (the illusion of potential that we discuss in The Invisible Gorilla), and they are primed to believe claims like “about 10 hours of DriveSharp training will improve your ability to spot potential hazards, react faster and reduce your risk of accidents.” Using claims of scientific proof in marketing this sort of quick fix capitalizes on the illusion and can be particularly persuasive.

If people take these claims of the power of training to heart, the results could actually increase the danger to drivers. Another paper, co-authored by the blog post author Peter Delahunt, found that elderly subjects with cognitive impairments who underwent speed-of-processing training were more likely to still be driving 3 years later (14% of untrained subjects stopped driving but only 9% of trained subjects did). If training actually improved driving performance, that could be a great outcome. But think about what it means if the training didn’t actually improve driving performance. If the subjects in these studies are convinced that the training helped them, that the benefits of cognitive training for driving are proven, they might believe that the training they underwent justifies remaining on the road. People often lack insight into their own driving performance. That’s one reason people keep talking on their phones while driving—when distracted, we don’t realize how poorly we’re driving. If they believe that training improved their driving even if it actually didn’t, they might not notice their driving problems and they might become unjustifiably confident in their ability to drive well, leading them to continue driving longer than they should!

DriveSharp and programs like it might well produce some benefits, and scientific study is the only way to test whether they do. I truly hope that future clinical studies replicate the Roenker et al (2003) result and show sustained benefits of training on driving performance. That would be a boon for drivers and for society. I credit Posit Science with conducting and supporting research that could test the effectiveness of such products (and I am thankful to Peter Delahunt for reading and commenting on this discussion and engaging in a dialog about the link between scientific results and marketing of those results).

Before the evidence of benefits is conclusive, sciencey marketing can be harmful, potentially giving people an unjustified confidence in their own abilities. And, if future studies fail to find benefits of training, it may be hard to counter the firmly held beliefs in the efficacy of training that result from such persuasive marketing. Sciencey marketing conveys a level of certainty that often isn’t merited by the underlying science.

More broadly, sciencey marketing claims contribute to the persistence of the illusion of potential. If people trust the claim that just 10 hours of training on what appears to them be an arbitrary computer task can lead to dramatic improvements on something as important as driving, why shouldn’t they also believe that playing arbitrary brain games can help them remember their friend’s name, that listening to Mozart could increase their IQ, or that they have other hidden powers just waiting to be released by the right arbitrary task. Quick fixes rarely are genuine, and strong scientific claims often must be reigned in later. Yes, tempered claims make for less enticing marketing, and a blog post stating that “preliminary evidence suggests a possible benefit of cognitive training for driving performance in impaired older drivers” might sell fewer products. But that would be a scientific claim rather than a sciencey one.

Sources Cited:

McCabe DP, & Castel AD (2008). Seeing is believing: the effect of brain images on judgments of scientific reasoning. Cognition, 107 (1), 343-352 PMID: 17803985

Edwards JD, Delahunt PB, & Mahncke HW (2009). Cognitive speed of processing training delays driving cessation. The journal of gerontology. Series A, Biological sciences and medical sciences, 64 (12), 1262-1267 PMID: 19726665

Roenker DL, Cissell GM, Ball KK, Wadley VG, & Edwards JD (2003). Speed-of-processing and driving simulator training result in improved driving performance. Human factors, 45 (2), 218-233 PMID: 14529195

Chabris, C., & Simons, D. (2010). The Invisible Gorilla, and Other Ways Our Intuitions Deceive Us. New York: Crown.

Science or Sciencey [part 3]

Part three of a 4-part series examining what happens when science is used for marketing (using brain-training software as the central example).

[part 1 | part 2 | part 3 | part 4]

[Full disclosure: I am a co-PI on federal grants that examine transfer of training from video games to cognitive performance. I am also a co-PI on a project sponsored by a cognitive training company (not Posit Science) to evaluate the effectiveness of their driver training software. My contribution to that project was to help design an alternative to their training task based on research on change detection. Neither I nor anyone in my laboratory receives any funding from the contract, and the project is run by another laboratory at the Beckman Institute at the University of Illinois. My own prediction for the study is that neither the training software nor our alternative program will enhance driving performance in a high fidelity driving simulator.]

In my last post, I examined some of the claims on the Posit Science blog to see what science they were using as the basis for the claims. Posit Science prides itself on being rooted in science, and unlike most claims for brain training, they actually can point to published scientific results in support of some of their claims. The primary evidence used to support the claim that DriveSharp training can improve driving appear to be based on a 2003 paper by Roenker et al that examined the effects of training on the UFOV. Today I will examine what that article actually showed to see what sorts of claims are justified.

First, I should say that the Roenker et al (2003) study is an excellent first attempt to study transfer of training from a simple laboratory task to real-world performance. It used performance measures during real-world driving, and was far more systematic than most road-test studies of this sort. As with any such study, though, it is limited in scope to the experimental conditions and subjects it tested. It also had several methodological shortcomings that somewhat weaken the conclusion that training transfers to untrained driving tasks. Here are some characteristics of the study that you might not know if you relied solely on the description of the scientific findings touted on the Posit Science site:

1. The subjects were all 55 or older and were recruited specifically because they might benefit from training. At least some (we don’t know how many) were recruited because they were involved in crashes. Screening criteria excluded participants who performed normally on the UFOV, so these were older participants with driving problems who had existing impairments on a demanding perception/attention task. Given that this transfer-of-training study tested only impaired older drivers, don’t count on any benefits if you are unimpaired, a good driver, under age 55, etc. The claims on the Posit Science website don’t mention these potentially important limitations.

2. The study involved 3 conditions: (a) the critical “speed of processing” training group, (b) a simulator training group, and (c) a relatively unimpaired control group. Not surprisingly, the two training groups tended to improve on the tasks that were specifically trained. The simulator training was a more standard driver training program, and those subjects showed improvements on the same tasks that were emphasized in the training (e.g., proper turning into a lane and using a signal). The critical “speed of processing” group showed no improvements on signaling or turning. Not surprisingly, though, their UFOV performance improved. That’s effectively what their training task was. Similarly, the speed training group responded faster in a choice response time task. Again, these sorts of task-specific benefits are not surprising because we know that training tends to improve performance on the trained task.

Even if training didn’t improve performance on the trained task, we might still find improvements if people thought the training should help. Subjects in the simulator condition knew that they were being trained to use their signal correctly and to turn into their lane appropriately, so they would be highly motivated to perform well for those aspects of the driving test (and some even said that they worked hard on doing those tasks well in the post-test). Similarly, a group trained to respond quickly would be motivated to respond quickly on other tasks.

There also was a lot of variability in the outcome measures, and in some cases, the speed trained group underperformed the other groups 18 months later (e.g., on the position in traffic composite measure). Given the number of statistical tests involved (3 training conditions, about 10 outcome measures, multiple follow-up tests), some of the statistically significant differences are likely to be spurious in any case.

3. In the pre-training driving segment, the raters were blind to the condition. However, following training, one or both of the coders knew the training condition. Even if the coders weren’t told the training condition, they might well have been able to tell which condition a subject was from anyway. Given that the training subjects were impaired to start with, the differences between them and control subjects might have been apparent in their driving performance. The paper provided no evidence that coders actually were unaware of the condition or that they couldn’t guess the condition. More importantly, the coders apparently were informed that the subject was in either the critical training group or the unimpaired group (that is, they knew the subject wasn’t in the other training condition). Why does that matter? If the coders knew that subjects were in the speed training condition and they believed that the training might improve some aspects of driving performance, then any subjective measures of driving could be affected by their expectations. If the coders were not truly blind to the condition, then their subjective judgments in coding the events might be biased by their knowledge and expectations about the training condition.

4. The one significant benefit of speed training found in the paper was a reduced number of dangerous maneuvers. Recall the claim that training reduced dangerous maneuvers by 36%. As I noted in the second post of this series, the judgment of what is dangerous could be somewhat subjective. It would be interesting to see the data on what constituted a dangerous maneuver – did the raters spot the same dangerous maneuvers or did they just come up with the same overall number of dangerous maneuvers. Were the two raters blind to each other? That is, could they see each other taking notes about what was dangerous? Either of these sorts of factors, in addition to the possibility that they knew the training condition, could lead to a spurious claim of improvement. Given that such events were rare in the study, a slight bias to code something as a dangerous maneuver or to treat it as safe could lead to what looks like a large relative improvement in performance.

These criticisms are not intended to cast aspersions on the Roenker et al (2003) study. I actually found the study to be quite impressive. If I had been a reviewer of this study, I would have raised some of these concerns, but I likely would have recommended publication (after requesting some weakening of the claims). It is an important first attempt to study transfer of training from the laboratory to actual driving, a topic that deserves further study. What I find problematic is not the science itself, but the way in which the science is applied in marketing the effectiveness of training more generally. The DriveSharp post stated the claim that training improves driving, and made no mention of these limitations and qualifications. Someone reading the post or the Posit Science website might conclude that training has a proven effect on driving for all people, when the effects are limited to one measure in an already impaired older population. Untrained readers might not delve into the paper itself to see what other limitations the study had. In the final part of this series, I will return to the DriveSharp blog post and will briefly discuss the possible negative consequences of sciencey marketing.

Sources Cited:

Roenker DL, Cissell GM, Ball KK, Wadley VG, & Edwards JD (2003). Speed-of-processing and driving simulator training result in improved driving performance. Human factors, 45 (2), 218-233 PMID: 14529195

Science or Sciencey [part 2]

Part 2 of a 4-part series examining what happens when science is used for marketing (using brain-training software as the central example).

[part 1 | part 2 | part 3 | part 4]

[Full disclosure: I am a co-PI on federal grants that examine transfer of training from video games to cognitive performance. I am also a co-PI on a project sponsored by a cognitive training company (not Posit Science) to evaluate the effectiveness of their driver training software. My contribution to that project was to help design an alternative to their training task based on research on change detection. Neither I nor anyone in my laboratory receives any funding from the contract, and the project is run by another laboratory at the Beckman Institute at the University of Illinois. My own prediction for the study is that neither the training software nor our alternative program will enhance driving performance in a high fidelity driving simulator.]

In my last post, I linked to a blog post on the Posit Science website that training with the DriveSharp program leads to improvements in real-world driving performance. I originally found the post because it linked to one of my own videos and implied (but left unstated) that their training might also help overcome inattentional blindness. I agree that inattentional blindness likely plays a role in driving and driving accidents, but to my knowledge, no studies have shown that training can reduce the rates of inattentional blindness. The rest of the post was unrelated to inattentional blindness, though. It’s focused instead on the claim that training with DriveSharp software for 10 hours produces remarkable improvements in real-world driving performance.

According to the post, the technologies used in DriveSharp have been tested in clinical studies for 20 years. In 2008, Posit Science acquired another company, Visual Awareness, which owned the UFOV test, so that presumably is the studied “technology” contained in their DriveSharp program (Posit Science has only existed for a few years).

The UFOV is essentially a measure of the breadth of attention that incorporates a speeded component, and the links between the UFOV and driving have been studied extensively over the years by its creators, Karlene Ball and Daniel Roenker and others. So far, so good. The blog post makes a number of specific claims about the DriveSharp software based on those studies. Let’s evaluate each claim in light of the scientific evidence:

“Drivers with poor UFOV performance are twice as likely to get into automobile accidents.”

Accurate enough (depending on how you define poor performance). There are a number of studies showing that people who perform poorly on the UFOV are poorer drivers.

“UFOV performance can be improved substantially by DriveSharp training.”

Likely accurate, but not surprising—presumably DriveSharp incorporates the UFOV, so training with DriveSharp should improve performance on the UFOV. Practicing a task makes you better at that task even if it doesn’t lead to improvements on other tasks.

“Training allows drivers to react faster providing an additional 22 feet of stopping distance at 55 mph.”

Misleading—This claim is based on a statement made in a paper published in 2003 in the journal Human Factors by Roenker and colleagues. They showed that speed of processing training (part of the UFOV) led to faster responses in a choice response time task. Before training, subjects averaged about 2.2 seconds to make this sort of speeded decision, and after training they were 277ms faster. Roenker et al (2003) then converted the 277ms improvement into an estimate of stopping distance on the road: If a driver were traveling at 55mph and could hit the brakes 277ms faster, they would stop 22ft sooner. The study did not show any effect of training on actual stopping distance and these speed improvements were not measured in a driving context — the claim was based entirely on faster performance in a laboratory choice response time task. The analogy to stopping distance was used to illustrate what a 277ms response time difference could mean for driving.

There is no evidence that speed on a computerized choice response time task translates directly into faster responses when actually driving, especially when the need to stop isn’t known in advance. One bit of evidence suggesting that simple computer responses and driving responses are different comes from a study by Kramer et al (2007). When making simple response times or choice responses on a computer, young subjects are much faster than older subjects. However, older subjects respond just as quickly as younger subjects to warning signals in a driving context. A big difference in pure response speed doesn’t necessarily translate to a difference in driving performance. Claiming that these response time differences in a computer task translate directly into faster stopping when driving is misleading.

“Training reduces dangerous driving maneuvers by 36%.”

Accurate, but perhaps not as impressive as it sounds—This statistic also comes from Roenker et al (2003). Dangerous maneuvers were coded by driving instructors in the car with the subject, and it’s not entirely clear from the original article what constituted a dangerous maneuver. One section of the paper defines dangerous maneuvers as those in which the instructor either had to take control of the car or in which other cars had to alter their course to avoid a collision. However, another section suggests that dangerous maneuvers were coded based on the degree of danger felt by the raters at each of 17 locations during a road test. Although the such maneuvers appear to have been judged reliably across observers, the metric has a subjective component (especially for the second definition). In and of itself, the subjectivity of the judgment might not be an issue, but as we’ll see in the next post of this series, such subjectivity could be an issue if the people making the judgments were not entirely blind to the experimental conditions.

In the study, subjects averaged about 1 dangerous maneuver in about an hour of actual, on-the-road driving. The 36% improvement was based on a change from an average of 1.01 dangerous maneuvers before training to an average of 0.65 in a test 18 months later (the average was 0.69 immediately after training). With such a low rate of dangerous maneuvers, it’s possible that most drivers had no dangerous maneuvers at all and that a small subset had a large number of dangerous maneuvers. In other words, we don’t know how many subjects had any dangerous maneuvers at all, and it’s possible that most subjects had none at all either before or after training.

“Training reduces at-fault crash rates by 50%”

Not supported by published data—As best I can tell, no peer-reviewed scientific papers support this claim. The statistic is mentioned on the wikipedia page for the UFOV, where it is sourced to a conference presentation in 2009. To my knowledge, the Roenker et al (2003) paper and one other paper are the only ones in the scientific literature to conduct something approximating a clinical training trial comparing the UFOV to other forms of training, and there were far too few subjects (and accidents) to measure anything like at-fault crash rates. In fact, Roenker et al (2003) note that the rarity of accidents is one reason for measuring so many other aspects of driving performance rather than just looking at the rates of accidents. Conducting a training study and using accidents as an outcome would require a sample of many thousands of subjects to produce a reliable difference in accident rates.

“Benefits last a long time with significant improvements still measurable 5 years after training.”

Unclear or unsupported—It’s hard to determine the source of this claim, but the wikipedia page for the UFOV makes a similar statement and sources it to the ACTIVE trial, a large-scale study of the effects of cognitive training on self-report measures of daily task performance years later. The ACTIVE study did not directly measure driving performance and has produced relatively few documented benefits of cognitive training despite being sufficiently large in scale to find them (it has shown some intermittent effects on self-report measures of daily activities over the years). Again, practice with an arbitrary laboratory task might lead to some long-lasting improvements, especially for performance on that task, but the unsupported implication of the claim in the blog post is that published scientific evidence shows an improvement from 10 hours of training to driving 5 years later.

So, what is the scientific basis for the bold claim that DriveSharp will improve your driving? First, there is substantial evidence that the UFOV is correlated with driving performance, especially for elderly drivers. Second, there is evidence that UFOV performance improves with training. (The UFOV wikipedia page has a fairly comprehensive list of references for each of these claims, so I won’t duplicate them here.)

Note that these findings alone do not permit any claim of a benefit to driving of training with the UFOV. To see why, consider that ice cream consumption is correlated with the temperature outside. We can certainly inspire you to eat more ice cream, but that won’t change the weather. The UFOV and driving might be related even if the components of the UFOV play no causal role in driving performance. To make a causal claim like the one on Posit Science’s blog, you would need to show a direct benefit from training. Ideally, you would need to do what the Roenker paper attempted to do — contrast training on the UFOV and training on some other plausible task in a double-blind design.

Given the centrality of the Roenker et al (2003) findings for the claims in the DriveSharp blog post, my next post in this series will take a close look at the Roenker et al (2003) paper to see exactly what has been “proven” about transfer of training. After that, I will end the series by discussing the implications of such science-based marketing for the public consumption of science more broadly.

Sources Cited:

Roenker DL, Cissell GM, Ball KK, Wadley VG, & Edwards JD (2003). Speed-of-processing and driving simulator training result in improved driving performance. Human factors, 45 (2), 218-233 PMID: 14529195

Science or sciencey [part 1]

Part 1 of a 4-part series examining what happens when science is used for marketing (using brain-training software as the central example).

[part 1 | part 2 | part 3 | part 4]

[Full disclosure: I am a co-PI on federal grants that examine transfer of training from video games to cognitive performance. I am also a co-PI on a project sponsored by a cognitive training company (not Posit Science) to evaluate the effectiveness of their driver training software. My contribution to that project was to help design an alternative to their training task based on research on change detection. Neither I nor anyone in my laboratory receives any funding from the contract, and the project is run by another laboratory at the Beckman Institute at the University of Illinois. My own prediction for the study is that neither the training software nor our alternative program will enhance driving performance in a high fidelity driving simulator.]

Almost all of the programs that tout their ability to train your brain are limited in scope. Most train your ability to perform simple cognitive tasks by having you perform them repeatedly, often adapting the difficulty of the task over time to keep it challenging. Some determine which tasks you perform well and which need improvement and adjust the tasks based on your ongoing performance. The simplest ones, though, simply track how much you improve and inform you that such improvements have made increased the fitness of your brain. Such task-specific training effects can be really useful—if you want to enhance your ability to do Sudoku, by all means practice doing Sudoku. But what pitches for those programs regularly imply is that playing their videogame or using their training will enhance your ability to do other tasks that weren’t specifically trained. For example, this advertisement for Nintendo’s Brain Age implies that by using their game, you will be better able to remember your friend’s name when you meet him on the street.

The idea that playing games can improve your brain is pervasive, and it taps what Chris Chabris and I have called the “illusion of potential.” A common myth of the mind is that we have vast pools of untapped mental resources that can be released with relatively minimal effort. This common intuitive belief underlies the pervasive myth that we only use 10% of our brains, that listening to Mozart can increase our IQ [pdf], and even the belief that some people have “discovered” psychic abilities. We devote the last main chapter of The Invisible Gorilla to this belief and its ramifications, and we recently wrote a column for the NY Times discussing how popular self-help books like The Secret and The Power capitalize on this mistaken belief.

The marketing for some brain training programs taps into this illusion, promising a quick fix for what ails us. (A marketing strategy similar to no-sweat exercise programs or eat-what-you-want diet programs). Maybe you want to remember your friend’s name, to avoid forgetting where you parked your car, or to improve your driving. Hey! Play our game and we’ll lower the age of your brain and improve your life. People are ready to believe that simple interventions can lead to big effects, so such marketing claims effectively separate people from their hard-earned money by instilling hope of less-hard-earned self-improvement.

On occasion, brain training companies try to lend credibility to their products by referencing scientific evidence. Or, in the case of Nintendo, trotting out a certified “brain scientist” in the person of Japanese neuroscientist Dr. Ryuta Kawashima, whose work “inspired” the exercises in their program. Appeals to science make the marketing more effective, even when the science does not entirely support the claims.

As scientists know, strong claims demand strong evidence, and strong evidence is alarmingly lacking from claims that brain training can improve real-world performance. The claims of these companies take advantage of the legitimacy granted by scientific backing without actually having the scientific backing necessary to make their strong claims about the real-world impact of their programs—their appeals to science make their claims sciencey, not scientific.

That’s why I was particularly surprised to run across this post by Peter Delahunt at Posit Science entitled, “DriveSharp: Proven to help keep you safe on the road.” Unlike many other brain training companies, Posit Science, founded by neuroscientist and National Academy of Science member Michael Merzenich, prides and markets itself on providing scientific backing for their training programs. They are among the few training companies that has at least some backing from the scientific literature for their claims of transfer of training (although most of the transfer they find is to other laboratory tasks rather than to real-world performance).

The blog post emphasizes this scientific credibility heavily in making the strong claim that “about 10 hours of DriveSharp training will improve your ability to spot potential hazards, react faster and reduce your risk of accidents.” Here’s how the post backs its claim:

“Posit Science prides itself on providing scientifically validated products. The technology contained in DriveSharp has been evaluated in multiple government funded clinical studies over more than 20 years…Completing the DriveSharp program is one of the best ways to help keep you safe on the road. It is based on sound scientific principles and has been extensively validated in numerous government funded studies. I strongly encourage you to try it out!”

The post represents one of the boldest claims I’ve seen for the direct transfer from training on a laboratory task to improvements in a real-world task like driving. But is the claim supported by the scientific evidence? Over the next few days, I will examine the claim and some of the research underlying it as a case study of the use of science in marketing.

Sources Cited:

Chabris, C. F., & Simons, D. J. (2010). The Invisible Gorilla, and Other Ways Our Intuitions Deceive Us. New York: Crown.

Chabris CF (1999). Prelude or requiem for the ‘Mozart effect’? Nature, 400 (6747) PMID: 10476958

Texters: Please ban what I do...

Mike Fumento passed along a remarkably silly statistic from this Reuters story about a Harris Interactive survey from a couple years ago. According to the article:

89 percent of respondents believe texting while driving is dangerous and should be outlawed.


66 percent of the adults surveyed who drive and use text messaging told pollsters they had read text messages or e-mails while driving. Fifty-seven percent admitted to sending them.

The survey was from 3 years ago, but a more recent Pew Survey reported in this Huffington Post piece found a similar rate. Of those adults who use text messaging, 47% have done so while driving (that works out to a little more than 1/4 of adults who text while driving).

So….at least some of the people who were surveyed know that texting while driving is dangerous and think that it should banned, but they do it anyway. Although these responses seem incoherent, there are two ways to make sense of them:

1) Some of the respondents are oddly sado-masochistic; they know they’re being bad and want to be punished. With a ticket.

2) Some of the respondents know that they are so devoid of will power that they need some external deterrent to avoid doing something they know to be stupid.

I’m not sure which is worse…

Truly invisible monkeys

Apparently Dodge took some heat from PETA for a car advertisement in which they dressed a chimp as Evel Knievel and had her press a plunger to release an explosion of confetti (read about it here).

Here’s the original:

In response to the criticism, Dodge edited the ad to make the chimp invisible without changing anything else. Now, an invisible chimp, wearing an Evel Knieval suit depresses the plunger.

I’m surprised anyone noticed the difference. Brilliant!

Silly ideas about safe texting

This is so wrong that if it were April 1, I would assume it is a joke.

Keyboard on windshield

heads-up keyboard from Wired article

But it comes from a writer with Gizmodo, one of the best tech sites around, and it was published by Wired. Unless proven otherwise, I have to assume it’s just a silly human falling prey to the illusion of attention.

The author asks, “Why isn’t there a better way to text while driving?” He acknowledges (correctly) that texting while driving is dangerous and shouldn’t be done, but he simultaneously admits doing it all the time. That’s disturbing since he is admitting to knowingly putting everyone around him in danger for no good reason. The broader focus of the piece, though, is based on a much bigger misunderstanding. He seems to think that the problem could be solved with technological innovation, making it possible for people to text without looking away from the road. His personal solution:

My own strategy is to hold the phone at the top of the steering wheel while typing in the hope that my brain will still be able to recognize dangers in front of me, even if my vision is focused on a little screen on a much closer plane.

Boo. Hiss. Snarl.

In a moment of unusual clarity, he states:

It’s probably not a very sound theory and I’ve been fortunate to never have really had the opportunity to put it to the test.

Yup. It’s not sound at all. Why not check into that before posting it as a possible fix? Perhaps he was too distracted?

The problem isn’t just with where you are focusing your eyes—just because your eyes are directed at the road doesn’t guarantee that you will consciously see everything important (including me prancing in front of your car). In fact, a head-up display can make people less likely to notice unexpected events right in their field of view (see studies of pilots by Haines). If you falsely believe that you are watching the road while focusing attention on your phone or a keyboard, you might be in even greater danger.

Texting requires you to take your mind off the road, and it’s even more cognitively demanding than talking on a phone. Talking on a phone while driving is roughly equivalent to driving under the influence of alcohol. Texting is much worse. There’s a reason that hands-free phones aren’t any safer than hand-held ones. The problem isn’t with your hands — it’s with your head. Your silly, human head. When you occupy your mind with something like texting, you are not devoting your mind to the road. That means you will miss unexpected events that happen right in front of your eyes. You’ll never catch a gorilla texting while driving.

You can't anticipate everything

A standard approach to safety engineering is to try to define all of the potential risks in advance and to design protocols that, if followed precisely, will avoid all of the known hazards. Such safety-by-protocol is great in principle, but it has a critical failing: The illusion of knowledge. The approach assumes that we can know and anticipate all of the potential risks.

Here’s one example of why that approach doesn’t work (I’m hoping it was a faked scene for a comedy program, but I’m not sure). Watch it all the way until the end:

As Nassim Taleb so rightly notes in The Black Swan, it’s the rare and unexpected events that are the really dangerous ones.

Moreover, it makes no sense to build safety protocols to address all of the really remote and rare cases — we’re not great at maintaining vigilance for things that almost never happen, and doing so would divert attention from what happens all the time. We want doctors to look for the most common diagnoses rather than the one-in-a-million ones, we want safety engineers to stave off the most likely problems, and we want security personnel to look for the most frequent risks. That means we might miss some of the rare cases, but we’re going to miss those anyway…

People are great pattern detectors, and we’re built to notice what happens most often, not what happens rarely. Even when we are actively looking for rare events, we often miss them (see work by Wolfe and colleagues). Signs admonishing us to watch for motorcycles might be useful for the few moments after we see them, but really quickly our expectations reset to what we typically see: cars. For the same reason, if I warned you to watch for gorillas, and hours later showed you a video of people passing basketballs, it’s not likely to increase your chances of spotting the unexpected gorilla.

I frequently see blog posts, columns, and advertisements by consultants who use the gorilla video as a way to promote their wares, promising that their workshop, training, or presentation will help you spot all the opportunities/risks you’re missing–the metaphorical gorillas in your midst. Be wary. No form of training can magically let you notice everything. If you’re devoting all your resources to spotting rare, unexpected events, you’re going to do less well in dealing with all of the problems you face almost daily. Moreover, the very nature of rare events means that you’ll miss some of them — you can’t possibly conceive of all of the one-in-a-million possibilities in advance. Taleb’s black swans are inherently unpredictable.

There are good and bad ways to deal with these limits on our ability to notice the unexpected. The bad way is to try to build each rare event we experience into our safety protocols so that we can spot it the next time it occurs. Yes, you can be prepared to notice the gorilla the next time you try to count people passing basketballs, but you might miss something else as a result:

Anticipating rare events that have already happened is why we now have to take off our shoes at airport security — a rare event (shoe bomber) that couldn’t have been anticipated in advance leads to a silly protocol to avoid that same risk again. It gives the appearance of safety while at the same time ignoring the fact that the next threat is likely to be equally unexpected (fortunately, the next rare event, the underwear bomber, didn’t lead to the same sort of protocol change…).

Even if there are not easy ways to anticipate all the unexpected events, knowing your limits allows you to take steps to increase the odds that you will notice some of them. You’re more likely to notice the gorilla in the basketball game if you’re not focusing attention on counting passes, presumably because you have more of your attention available to pick up other aspects of a scene. Similarly, a passenger in a car should be more likely to spot unexpected events on the road because they aren’t engaged in driving (that’s why you should never complain when a back-seat driver tells you to watch out). When driving, you can turn your phone off and put it in the back seat so you won’t be tempted to use up valuable resources that might help you spot the child running into the street. If you are manning a security checkpoint, it’s a good idea to have someone watching the scene who has no task other than to watch the scene for anything out of the ordinary. If you’re designing a product or protocol, don’t assume that you can anticipate all possible risks. Instead, assume that you can’t and make sure people are as aware of their limits as possible. That won’t let you anticipate everything, but knowing that you can’t anticipate everything at least gives you the chance to maximize your odds of noticing what matters.

Sources mentioned:

Wolfe JM, Horowitz TS, & Kenner NM (2005). Rare items often missed in visual searches. Nature, 435 (7041), 439-40 PMID: 15917795

Simons, D. J. (2010). Monkeying around with the gorillas in our midst: familiarity with an inattentional-blindness task does not improve the detection of unexpected events i-Perception, 1 (1), 3-6 : 10.1068/i0386

Simons, D. J., & Chabris, C. F. (1999). Gorillas in our midst: sustained inattentional blindness for dynamic events Perception, 28 (9), 1059-1074 DOI: 10.1068/p2952

Predicting learning and the illusion of knowledge

This post was chosen as an Editor's Selection for
In The Invisible Gorilla, Chris and I discuss many aspects of the illusion of knowledge, the tendency to think we have a better understanding than we actually do. One aspect of this illusion is that we easily mistake surface understanding for deep understanding, what Leon Rozenblit and Frank Keil called the “illusion of explanatory depth.” That aspect of the illusion of knowledge leads us to think we have a deep understanding when all we really have is knowledge of the surface properties. In a recent op-ed in the LA Times, we argued that this illusion of knowledge, when coupled with technology that presents information in short, surface-level bursts, can lead to a mistaken belief that we actually understand more than we do.

One practical consequence of the illusion of knowledge is the planning fallacy – we almost always assume that new projects will take less time and resources than they actually do. In part, the planning fallacy arises when we fail to take into account all the unpredicted complications that can arise and we assume the simplest possible scenario. A new paper by Darron Billeter, Ajay Kalra, and George Loewenstein presents an interesting twist on the typical course of the illusion of knowledge. In most cases, the illusion of knowledge leads us to think that we’re more skilled or knowledgeable than we actually are and it leads us to underestimate how long it will take us to accomplish our goals. Billeter et al looked at predictions for how long people would take to learn a new skill such as typing on a Dvorak keyboard.

Before using it, they were overconfident in estimating how long it would take them to learn, just as we would expect from overconfidence in our own knowledge. However, as soon as they tried it out and realized that they couldn’t succeed without practice, they over-corrected their expectations and assumed it would take them longer to learn the skill than it actually did. Apparently, people lack insights into how quickly they can acquire new skills even though they initially think they won’t need new skills at all. I do wonder, though, whether these underestimates apply only to people who are capable of learning new skills fairly readily. I wonder if the estimates of old dogs would be better once they try the new product — that is, would their corrections be better calibrated to their actual ability to learn.

hat tip to Cynthia Graber at Scientific American

Source Cited:

Billeter, D., Kalra, A., & Loewenstein, G. (2011). Underpredicting Learning after Initial Experience with a Product Journal of Consumer Research, 37 : 10.1086/655862

more choice blindness videos

Yesterday I posted about a new, in-press study of choice blindness by Lars Hall, Petter Johansson, and colleagues. Their new study extended the phenomenon of choice blindness to real world taste decisions made by shoppers in a market. Read more about it in yesterday’s post entitled “Do you know what you like.”

Here are two new videos they just posted that show how they pulled off the taste-test experiment (the first is from the BBC and the second is from their own footage which looks much nicer):