Criticisms of mindset research
"[...] anything can be criticized and I think that nothing is above criticism. [...] the growth mindset will also be criticized. No method or concept is final or perfect. The good thing about a rational and scientific approach to any truth claims is that we may find out what is wrong with specific claims so that we can gradually refine and improve them. The growth mindset is a valuable concept but it is not perfect. It is inevitable that we will improve on it by discovering what can be improved in its conceptualization.
Post by Nick Brown about Mueller & Dweck (1998)
Nick
Brown, the person who played a big role in exposing mistakes in Babara Fredrickson’s publication about the positivity ratio, has analyzed a study by Mueller & Dweck (1998). He has written a post about this entitled In which science actually self-corrects, for once. Brown has developed a mathematical technique, GRIM,
with which he can detect irregularities in tables of findings in publications. With this technique he has detected many mistakes in many publications. He has also applied it to the publication by Mueller and Dweck and found inconsistencies in this publication as well.He contacted Dweck and mentions that, to his surprise, he received a reply from Dweck within 24 hours in which she said she would look into it. Within two weeks, he received a 16-page PDF document in which Dweck, with Claudia Mueller and David Yeager had gone through the entire article, line by line, and answered every one of Brown's points.
Some of the inconsistencies could be explained, others turned out to be the result of mistakes. Brown says that "Dweck’s “openness and willingness to address the problems” has been “exemplary”, and that she has done a “thorough job of owning up to the problems” of the paper. She has publicly acknowledged these errors and says she is “considering” writing to the journal to ask them to formally correct the paper." Brown remains skeptical about the concept of mindset but says: "at least I feel confident that the main people researching it are dedicated to doing the most careful reporting of their science that they can."
Buzzfeed article
Another critical article about mindset was published on Buzzfeed. This article mentions Brown but also some other critics of mindset such as Scott Alexander, Timothy Bates, Stuart Richie, and Andrew Gelman.
Alexander, who says he is very skeptical about the mindset concept, is surprised of the big effect sizes in mindset research and wonders whether Dweck has "make a pact with the Devil in which she offered her eternal soul in exchange for spectacular study results?" Bates is currently doing large scale research into mindsets in schools in the UK. Previously he attempted to replicate the Mueller & Dweck (1998) finding and failed. In a response Dweck says that replications are important but they need to be done genuinely and thoughtfully by skilled people (read more). Nick Brown counters: “The question I have is: If your effect is so fragile that it can only be reproduced [under strictly controlled conditions], then why do you think it can be reproduced by schoolteachers?”
The article contains some other critical and interesting issues so I recommend reading it. Brown says that doing statistics properly requires more specialized training than many researchers have had. Bates says the mindset hypothesis should not be thrown out but warns against expecting too much from it.
Gelman: why psychology?
Is psychology such a mess that it is hit by so much criticism? I think the situation is more nuanced than that. Recently a befriended professor told me that similar problems exist in the natural sciences: “[…] people who have once taken a position (often) have great difficulty in acknowledging it may not be entirely right. […] Especially in my area of expertise (simulations of molecules) there are some methods about which not enough questions are asked. To change that a lot of testing and comparing is needed."
Andrew
Gelman, statistics professor at Columbia University, mentions a paradox. He says that some positive characteristics of psychology make this science so suitable for the type of critical reflection that now takes place (read here). Firstly, he explains that psychology is more sophisticated than for example economics and biology in it consideration of topics like validity, reliability, and latent constructs. Secondly, he say that psychology nearly exclusively works with clean and easily comparable methods (experiments and surveys).Thirdly, he says psychology is characterized by a culture of openness and is relatively unaffected by restrictions and big money interested (as medicine and biology are). Fourthly, the current criticism involves the heart of (social) psychology, some of the leaders of the field, whereas in other fields, research controversies typically seem to involve fringe elements in their professions. Fifthly, psychological is interesting and to some extent accessible to the general public.
Discussion
It is a good thing that the demands for good research are becoming stricter. The bar for good psychological research is raised which is necessary for building a robust body of psychological knowledge. It is ironic that the very strengths of psychology of focus on good methods and relative openness make it more vulnerable to criticism than other fields with are less rigorous and open. Other fields may be less transparent but no less problematic.Even more importantly: we are all, on a daily basis, confronted with countless methods and claims which aren't even attempted to be substantiated. Let's not exclusively be skeptical about a field of science which is trying hard to substantiate its findings and is open to criticism.
Improving research into mindset is important. No concept and no researcher is above criticism. For now, I agree with Bates. We must not abandon the concept of mindset but also not have unrealistic expectations of it.
Comments