Growth mindset debunked? Not so much. Simple re-analysis of Li & Bates (2019) shows that Mueller & Dweck (1998) is not debunked, but confirmed
One of the best-known studies in relation to mindsets is the study by Mueller & Dweck (1998). Here I discuss that article in detail. Li & Bates (2019) recently conducted a replication study by Mueller & Dweck and said they had not found the originally found effects. But in a response to this article, Dweck & Yeager (2019) show that Li & Bates' replication study does not meet the requirements that are currently set for replication studies. Moreover, they show that by correcting for some of the simplest deviations from those requirements, Li & Bates' data does not invalidate Mueller & Dweck's conclusions, but corroborates them.
Replication research is important to science, but it is not always done properly
The goal of science is to provide robust, replicable and generalizable knowledge. Replication studies play an important role in science. Well-designed and conducted replication studies can provide important clues about the robustness of effects from original studies. However, as explained in a recent publication by Bryan et al. (2019), replicators often allow themselves too much freedom in the way replication studies are designed and analyzed. This can probably partly be explained by the fact that replicators may have an interest in the failure of replication studies.
The Original Study: Muller & Dweck (1998)
Mueller & Dweck conducted six studies on the effects of intelligence praise versus effort praise after successful task performance on easy task on performance after unsuccessful task performance on a difficult task. The figure below shows how the studies were structured:
Mueller & Dweck showed in four studies that intelligence praise led to poorer performance after failure than effort praise. The results were robust and unaffected by p-hacking.
The Replication by Li & Bates (2019)
In three studies with children in China, Li & Bates followed roughly the same procedure as that of Mueller & Dweck. After easy problems, children got praise and then they got more difficult problems. As a third step, they had less difficult problems. They also introduced a new element.
In Studies 2 and 3, active control condition was added in which a static mindset comment made by a comment about effort: "You can't change your basic aptitude, but you work on things and that's how you get things done."
They also added a fourth step with difficult problems in some studies. In Study 1, Li & Bates found results confirming Mueller & Dweck's conclusions. In studies 2 and 3, they reduced the statistical power and found no difference between the groups.
In Studies 2 and 3, active control condition was added in which a static mindset comment made by a comment about effort: "You can't change your basic aptitude, but you work on things and that's how you get things done."
They also added a fourth step with difficult problems in some studies. In Study 1, Li & Bates found results confirming Mueller & Dweck's conclusions. In studies 2 and 3, they reduced the statistical power and found no difference between the groups.
Commentary and reanalysis by Dweck & Yeager (2019)
The first study had sufficient statistical power and Mueller & Dweck confirmed. The fact that studies Li & Bates reduced the statistical power of studies 2 and 3 goes against guidelines for replication research and may be a reason for the insignificant results.
By analyzing the data from studies 1 to 3 together (which increased the statistical power), Dweck & Yeager did find confirmation of the results of Mueller & Dweck.
A strange interpretation error by Li & Bates was the following: Li & Bates equate effort praise with a growth mindset. For example, they say “The results did not support any effect of growth mindset on children's post-failure performance” (p. 19) and “We found little or no support for the idea that growth mindsets are beneficial for children's responses to failure”. But giving effort praise is not the same as growth mindset manipulation! Both studies did not measure the effect of a growth mindset after failure, but of effort praise after failure.
Having said this, here is another error. This error concerns a serious deviation in the research design of Li & Bates compared to Mueller & Dweck. As shown in the figure above, Mueller & Dweck examined children's responses after failure. They thought that children who received intelligence praise would perceive failure as an indication of a lack of ability. But Li & Bates had not created a real failure trial. As the figure below shows, children in the Li & Bates study scored nearly 2.5 standard deviations higher than the children in the Mueller & Dweck study.
By analyzing the data from studies 1 to 3 together (which increased the statistical power), Dweck & Yeager did find confirmation of the results of Mueller & Dweck.
A strange interpretation error by Li & Bates was the following: Li & Bates equate effort praise with a growth mindset. For example, they say “The results did not support any effect of growth mindset on children's post-failure performance” (p. 19) and “We found little or no support for the idea that growth mindsets are beneficial for children's responses to failure”. But giving effort praise is not the same as growth mindset manipulation! Both studies did not measure the effect of a growth mindset after failure, but of effort praise after failure.
Having said this, here is another error. This error concerns a serious deviation in the research design of Li & Bates compared to Mueller & Dweck. As shown in the figure above, Mueller & Dweck examined children's responses after failure. They thought that children who received intelligence praise would perceive failure as an indication of a lack of ability. But Li & Bates had not created a real failure trial. As the figure below shows, children in the Li & Bates study scored nearly 2.5 standard deviations higher than the children in the Mueller & Dweck study.
This error means that Mueller & Dweck's most important question (how do children react after failure?) cannot be answered by Li & Bates. Unsurprisingly, they found results less consistent and actually surprising that Mueller & Dweck was confirmed at all.
The addition of the active control condition was also an unfortunate choice and based on an interpretation error. First, you are not replicating if you make the conditions in the study different from those in the original study. Second, the active control condition shows their misunderstanding of Mueller & Dweck. Li & Bates linked a fixed mindset comment to a comment about effort ("You can't change your basic aptitude, but you work on things and that's how you get things done"). This was intended as an alternative explanation for the idea that a growth mindset would explain behavior after failure.
But Mueller & Dweck did an attribution intervention (“You must be smart” and “You must have done your best”) and not a mindset intervention (nothing was said about the malleability of capabilities). Mindset was only measured as a dependent variable at Mueller & Dweck. Moreover, the comment also included a comment about effort, and in that sense it was not an alternative explanation.
The addition of the active control condition was also an unfortunate choice and based on an interpretation error. First, you are not replicating if you make the conditions in the study different from those in the original study. Second, the active control condition shows their misunderstanding of Mueller & Dweck. Li & Bates linked a fixed mindset comment to a comment about effort ("You can't change your basic aptitude, but you work on things and that's how you get things done"). This was intended as an alternative explanation for the idea that a growth mindset would explain behavior after failure.
But Mueller & Dweck did an attribution intervention (“You must be smart” and “You must have done your best”) and not a mindset intervention (nothing was said about the malleability of capabilities). Mindset was only measured as a dependent variable at Mueller & Dweck. Moreover, the comment also included a comment about effort, and in that sense it was not an alternative explanation.
Conclusion
Li & Bates missed the opportunity to do an interesting and good replication study. They could have looked to see if Mueller & Dweck's results would be found in a completely different culture, and they could have explored different mediating and moderator variables. Given the sloppy or clumsy design of Li & Bates' study, it is surprising that a small correction (merging the data from studies 1 to 3) allowed the effects of Mueller and Dweck to be replicated.
Comments