By Ranran Li · January 2021
To contribute a small effort toward creating a better research environment — and to keep striving.
In honor of Professor Rao.
For improving the scientific practice culture in China.
During a recent meeting with my supervisor, we briefly discussed recent events in the Chinese academic community. Facing the harsh reality that "bureaucracy and scholar-tyrants hinder the development of good research practices," I felt heartbroken, helpless, and powerless. At the same time, a recent public case about the "misuse of images" sparked wide attention and discussion, which could also be a potential turning point for advancing research integrity. Ten years ago, the prominent Dutch social psychologist Diederik Stapel was exposed for fabricating data. The scandal led to the open science movement in psychology, self-reflection in the discipline, and increasing attention to the transparency and legitimacy of research plans, analytical processes, results, and replication — opening a new chapter for social science research.
We all hope this is not the end of the story. Professor Rao is a rare beacon of light within the system. He may not represent "truth," but he certainly embodies a deep commitment to truth, procedural justice, and transparency in the research process and outcomes.
Two weeks ago, I attended a workshop by Daniel Lakens (one of the key proponents of the open science movement) on trustworthy research practices. He mentioned the structural challenges to promoting rigorous and reliable science: publication bias, tenure-track pressure, and more. He said that as PhD students, we are still junior. The task of reforming systems and policies should fall to senior scholars. Our job, as early-career researchers, is to conduct responsible research and follow good practices. I was moved by this comment. And now, witnessing the recent uproar in the Chinese academic sphere, I realize we junior scholars are not entirely powerless. Besides practicing integrity ourselves, we can also speak out and share knowledge about good vs. questionable practices. Hence, this post.
Narrowly defined, research misconduct includes blatant fraud: plagiarism, data fabrication, falsifying results. These are serious violations. But those who engage in outright fraud are in the minority. It is the more subtle, often-overlooked questionable research practices (QRPs) that most often undermine the replicability and credibility of research. Below, I use the book "The Seven Deadly Sins of Psychology" to illustrate some of the most common QRPs in the social sciences. This book was deeply influential in my early academic training and laid the foundation for my commitment to responsible psychological science.
The empirical research process spans from hypothesis formulation to design, data collection, analysis, interpretation, and publication. At every stage, QRPs can occur. Publication bias is a key driver of many of these behaviors. Journals tend to favor studies with significant, hypothesis-confirming, and novel results ("when the positive and new trumps the negative but true") over boring but truthful findings. As a result, early-career researchers face pressures to graduate or publish for jobs, and senior researchers face tenure/promotion demands. This pressure fuels QRPs and contributes to the reproducibility crisis.
Examples include p-hacking (manipulating the analysis or sample to turn null results significant), HARKing (Hypothesizing After Results are Known: adjusting your theory to fit results), and selective reporting (e.g., running multiple studies or using multiple measures but only reporting the best-looking ones).
Using various statistical models, excluding outliers in a non-transparent way, or selectively reporting findings are all examples of abusing researcher degrees of freedom, which inflates Type I error rates.
Ideally, researchers should conduct an a priori power analysis to determine the required sample size before collecting data. In contrast, collecting data and peeking at results along the way, then deciding whether to stop or continue, is problematic. The more frequently you peek at the data, the higher the chance you get a false positive due to random p-value fluctuations.
Solutions include preregistration, disclosure statements (e.g., "We report how we determined our sample size, all data exclusions (if any), all manipulations, and all measures"), and open data sharing.
Causes of unreliability include disregard for direct replication, lack of statistical power, failure to disclose methods, statistical fallacies, and failure to retract flawed work.
Statistical power reflects the probability of correctly detecting a true effect. Higher power leads to a higher positive predictive value (PPV), meaning reported effects are more likely to reflect true phenomena.
Data transparency has been a central focus of the open science movement.
The most famous case in psychology is Stapel's. Interestingly, some instances of fraud have been exposed using the "law of small numbers." Humans tend to overestimate randomness when generating fake data. For example, in real random sequences, the most frequent number may occur five times. In human-generated fakes, it often appears only three times. This clue helped detect some cases. As the quote goes, "Man is an orderly animal who finds it very hard to imitate the disorder of nature."
In his confessional article, Stapel wrote: "I wanted it so badly. I wanted to belong, to be part of the action, to score... Fraud is a high-reward strategy with low risk of detection that achieved my ambitions and attracted little suspicion." He also commented: "A culture that rewards 'dig and deliver' over 'dig and discover.'"
Two quotes from the book feel especially meaningful in light of recent events:
Prevention strategies include: (a) open data, (b) randomized audits by third parties, (c) harsh penalties for confirmed fraud, and (d) valuing direct replication by independent researchers.
These last two sins concern restricted access publishing and impact factor obsession, so I won't elaborate here.
"Some people succeed at the cost of their conscience. Some find happiness by harming others. May you be confident, proud, self-aware, reflective, and self-redeeming. May your self-respect support a spirit of freedom, autonomy in work, and ease in life. Be the person you respect."
Reference: Chambers, C. (2017). The Seven Deadly Sins of Psychology: A Manifesto for Reforming the Culture of Scientific Practice. Princeton University Press.
← Back to Blog Overview