bjoern.brembs.blog

In a paper published in Nature Neuroscience now over a year ago, the authors claimed to have found a very surprising feature, which was long thought to be a bug. In my blog post covering the hype in the paper and that in an accompanying news-type article, I wrote that today, it has become ever rarer that scientists admit to standing on the shoulders of giants, as we are not rewarded for referring to giants, but only for being giants ourselves. The blog post has triggered some email correspondence between a number of colleagues in or close to this field, at the end of which was the decision that two of us, Nicolas Stergiou and I, would contact the journal and inform them of the missing references in the two articles.

The first reply was that we ought to write a ‘letter to the editor’ instead of informally notifying the journal that there were some crucial references missing. So we sat down and wrote our letter (submitted February 10, 2015):

Dear Sir,

HHMI Investigator Michael Eisen recently wrote that we ought not only to come down hard on people who cheat but also on the far great number of people who overhype their results. He suggests that hyping should be punished just as fraud. This letter to the editor represents such an effort as we would like to report on such a case of hyping.

“Standing on the shoulders of giants” is what scientists say to acknowledge the work they are building on. It is a statement of humility and mostly accompanied by citations to the primary literature preceding the current work. In today’s competitive scientific enterprise, however, such humility appears completely misplaced. Instead, what many assume to be required in the struggle to survive is to convince everyone that they are the giant, the genius, the prodigy who is deserving of the research funds, the next position, tenure. The Nature Neuroscience article “Temporal structure of motor variability is dynamically regulated and predicts motor learning ability” by Wu et al. (doi:10.1038/nn.3616) with its accompanying news-type article “Motor variability is not noise, but grist for the learning mill” by Herzfeld and Shadmehr (doi:10.1038/nn.3633) from earlier this year clearly fall within this category. Both articles claim that the researchers have made the game-changing discovery that something long thought to be a bug in our movement system is actually a spectacular feature. It is argued that this discovery is such a huge surprise, because nobody in their right mind would have ever thought this “unwanted characteristic” to actually serve some purpose.

The problem with this line of argument is that probably most people in the field thought it should be obvious, even to be expected – and not surprising at all. Skinner is largely credited with the analogy of operant conditioning and evolution. This analogy entails that reward and punishment act on behaviors like selection is acting on mutations in evolution: an animal behaves variably and encounters a reward after it initiated a particular action. This reward will make the action now more likely to occur in the future, just as selection will make certain alleles more frequent in a population. Already in 1981, Skinner called this “Selection by Consequences“ (Science Vol. 213 no. 4507 pp. 501-504, DOI: 10.1126/science.7244649). Skinner’s analogy sparked wide interest, e.g. an entire journal issue (Behavioral and Brain Sciences 7(04), 1984), which later appeared in book form (The Selection of Behavior: The Operant Behaviorism of B. F. Skinner: Comments and Consequences. A. Charles Catania, Stevan R. Harnad, Cambridge University Press). Clearly, the idea that reinforcement selects from a variation of different behaviors is not a novel concept at all, but more than three decades old and rather prominent. This analogy cannot have escaped anybody working on any kind of operant/motor learning, except those seriously neglecting the most relevant literature. This interaction of variability and selection is a well-known and not overly complicated concept, based in evolutionary biology and psychology/neuroscience. Consequently, numerous laboratories have been studying various aspects of this interaction for a long time. Skinner’s projection was that increased behavioral variability leads to increased operant learning rates, just like increased mutations rates lead to increased rates of evolutionary change. More than a dozen years ago, Allen Neuringer showed this to be the case in rats (Psychonomic Bulletin & Review 2002, 9 (2), 250-258, doi: 10.3758/BF03196279), but there are studies in humans as well (Shea, J. B., & Morgan, R. B. (1979). Contextual interference effects on the acquisition, retention, and transfer of a motor skill. Journal of Experimental Psychology: Human Learning and Memory, 5, 179–187). That such variability is beneficial, rather than detrimental has been shown even in situations where the variability is so high, that the acquisition rate is reduced, but post-training performance is enhanced (Schmidt RA, Bjork RA (1992): New conceptualizations of practice: Common Principles in Three Paradigms Suggest New Concepts for training. Psychological Science, 3(4): 207-217).

Wu et al. confirm both Skinner’s conjecture as well as previously published reports (some cited above) that indeed the rate of learning in operant conditioning is increased in subjects where the initial variability in the behavior is higher. However, instead of citing the wealth of earlier work, Wu et al. claim that their results were surprising: “Surprisingly, we found that higher levels of task-relevant motor variability predicted faster learning”. Herzfeld and Shadmehr were similarly stunned: “These results provide intriguing evidence that some of the motor variability commonly attributed to unwanted noise is in fact exploration in motor command space.”

We regard it as highly unlikely that none of the seven authors in total should have never heard of Skinner or the work over the last four decades by many human movement scientists that have explored the temporal structure of human movement variability and its relationship with motor learning. The work by senior scientists such as Karl Newell, Michael Turvey, Richard Schmidt, and their students published in books and hundreds of journal articles is completely ignored, just as the work by several younger mid-career scientists such as Nick Stergiou, Jeff Hausdorff, Thurmon Lockhart, Didier Dilignieres, and many others. After a thorough review of this literature the authors may realize that their results are neither new nor novel. If indeed the authors were unaware of this entire section of literature so relevant to their own research, it would be an indictment in its own right.

Hence, in formal corrections of both articles, we would expect all mentions throughout both texts of how surprising these findings were, to be replaced with references including, but not limited to the works cited above.

Yours sincerely,

In hindsight, we probably ought to have mentioned that the research of course has merit and that there are a lot of valuable and exciting results in the paper, but that it is one specific aspect that is really not surprising at all. However, we wanted to make it brief and focused just on the hype. Maybe that was a mistake. Be that as it may, after about two months of peer review, our letter was rejected. Perhaps not so surprisingly, the peer reviewers apparently were the same ones that reviewed the original manuscript (at least one of them):

This is the basis on which I recommended publication, and I do not feel it needs a corrigendum.

I agree that the article’s tone is a little more breathless than strictly required, but this is the style presently in vogue

The letter complains about “over-hyping” about certain claims made in the paper related to previous work. I have some sympathy with the letter writers on this front. However, I also have some sympathy for the authors, who understandably were trying to emphasize the novelty of their work.