The ironic effect of significant results on ... [Psychol Methods. 2012. Fixing Science – Systems and Politics : Neuroskeptic. There is increasing concern that the structure of modern science is flawed and that most published research findings may be false. Commonly cited problems with how science works today include: Publication bias and the file drawer problem. “Result fishing”, data dredging etc. – analyzing data in different ways to “get a finding”The privileging of “positive” results over “negative” ones. I have previously argued that, to solve these, problems we need a way to ensure that scientists publicly announce which studies they are going to run, what methods they will use, and how they will analyze the data, before running their studies. We already have such a registration system in place for clinical trials. It’s a good system. It’s not perfect but it’s helped. I’m not sure. Here are some options for systems: There could be a central registry, free and open to the public, where protocols are pre-registered.
How can we actually make this happen? Governments could introduce legislation to force this. Journal impact factor 'distorts science' › News in Science (ABC Science) News in Science Friday, 17 May 2013 Dani Cooper ABC Ranking row An international declaration by more than 150 leading scientists and 75 scientific organisations is demanding a rethink of the role 'journal impact factor' plays in the evaluation of research. The Declaration on Research Assessment , released today, contains 18 recommendations for change and is signed by a coalition of scientists, journal editors, publishers and scientific and funding bodies.
The statement was released to coincide with editorials in scientific journals around the world. More than 10 of Australia's pre-eminent researchers and leading science organisations have signed the declaration, which highlights growing concern at the obsession in world science for publication in journals with a high impact factor. The most influential citation ranking system is the Journal Impact Factor (JIF) that appears annually as part of the Thomson Reuters Web of Knowledge. It is a view shared by the Australian signatories. High impact factors are meant to represent strong citation rates, but these journal impact factors are more effective at predicting a paper’s retraction rate. Journal ranking schemes may seem useful, but Björn Brembs discusses how the Thompson Reuters Impact Factor appears to be a reliable predictor of the number of retractions, rather than citations a given paper will receive. Should academics think twice about the benefits of publishing in a ‘high impact’ journal?
With more than 24,000 scholarly journals in which some piece of relevant research may be published, a ranking scheme seems like a boon: one only needs to read articles from a small, high-ranking subset of journals and safely disregard the low-level chaff. At least this is how one might describe the development of journal ranks in the 1960s and 70s, when scores of new journals began to proliferate.
Today, however, journal rank is used for much more than just filtering the paper deluge. Among the half-dozen or so ranking schemes, one de facto monopolist has emerged which dictates journal rank: Thomson Reuters‘ Impact Factor (IF). High Impact = High Statistical Standards? Not Necessarily So. What are the statistical practices of articles published in journals with a high impact factor? Are there differences compared with articles published in journals with a somewhat lower impact factor that have adopted editorial policies to reduce the impact of limitations of Null Hypothesis Significance Testing?
To investigate these questions, the current study analyzed all articles related to psychological, neuropsychological and medical issues, published in 2011 in four journals with high impact factors: Science, Nature, The New England Journal of Medicine and The Lancet, and three journals with relatively lower impact factors: Neuropsychology, Journal of Experimental Psychology-Applied and the American Journal of Public Health. Figures Citation: Tressoldi PE, Giofré D, Sella F, Cumming G (2013) High Impact = High Statistical Standards? Not Necessarily So. Editor: Robert K. Received: August 23, 2012; Accepted: January 9, 2013; Published: February 13, 2013 Introduction Methods Results.
High-impact journals: where newsworthiness trumps methodology. Criticism continues to mount against high impact factor journals with a new study suggesting a preference for publishing front-page, “sexy” science has been at the expense of methodological rigour. Dorothy Bishop confirms these findings in her assessment of a recent paper published on dyslexia and fears that if the primary goal of some journals is media coverage, science will suffer. More attention should be placed on whether a study has adopted an appropriate methodology. Here’s a paradox: Most scientists would give their eye teeth to get a paper in a high impact journal, such as Nature, Science, or Proceedings of the National Academy of Sciences.
Yet these journals have had a bad press lately, with claims that the papers they publish are more likely to be retracted than papers in journals with more moderate impact factors. It’s been suggested that this is because the high impact journals treat newsworthiness as an important criterion for accepting a paper. Will any damage be done? Feature: Social psychology is primed but not suspect. A series of high-profile research scandals within social psychology have led to unjustified attacks on the whole academic discipline. Wolfgang Stroebe and Miles Hewstone declare that the majority must not suffer for a tiny minority’s misconduct The news in autumn 2011 that Diederik Stapel, the highly respected Dutch social psychologist, had committed scientific fraud on a huge scale came as a shattering blow to the international community of social psychologists.
A recipient of scientific awards from major associations in Europe and the US, Stapel had enjoyed a meteoric rise within our profession. Awarded the social psychology chair at the University of Groningen in 2000, only three years after earning his doctorate, he then moved to Tilburg University, where he became dean of the School of Social and Behavioural Sciences (and a regular tennis partner of the rector). Just as well, you might think, since none of the committee members has any background in social psychology. Click to rate. Www.biomedcentral.com/content/pdf/2050-7283-1-2.pdf.
New APS replication initiative aims to open the file drawer, heralding a positive step for psychological science. | PsySociety. During the past couple of years, psychological science has been in the midst of a PR disaster. Academics have publicly announced that they failed to replicate some of the most classic findings in our field, bringing the original effects themselves — and often the integrity of the original researchers reporting them — into question. These pronouncements and subsequent push to estimate the true effect sizes of various findings led to the even more disturbing realization that it is far too difficult to publish these failed replications — or successful replications, for that matter — in the peer-reviewed, academic journals that serve as our bread and butter.
A new initiative, backed by the Association for Psychological Science and co-headed by Dr. Daniel Simons of the University of Illinois at Urbana-Champaign and Dr. Psychologists have been calling for a widespread replication effort for years. It helps that the initiative is strongly backed by our field’s premiere organization. Like this: Psychology is Starting to Police Itself. I have been in the field of cognitive science for about 25 years now. There are three significant changes that I have seen in the kinds of research that get done in that span.
First, there is tremendous pressure to publish a lot of papers. When I went on the job market for the first time about 20 years ago, I had 4 published papers, and I landed a job on the faculty at Columbia University. Now, many of the job candidates applying for faculty positions here at the University of Texas have more than 10 publications.
Second, the typical length of a psychology paper has gotten shorter. Third, there has been an increased interest in the literature on counterintuitive findings. This combination of factors has had both benefits and costs to the field. On the negative side, though, this openness has led to abuses. More commonly, though, the pressure to publish novel findings has led researchers to cut corners. So, what can be done? Have Your Cake and Eat It Too! Practical Reform in Social Psychology. If you have been following recent headlines in the social sciences then you are aware that the field of social psychology has been in some rough water over the past three years. In this time period, we've had our flagship journal publish a series of studies providing evidence that ESP exists (and then refuse to publish non-replications of these studies).
We've suffered through at least three instances of scientific fraud perpetrated by high profile researchers who engaged in egregious scientific misconduct. We've had an entire popular area of research come under attack because researchers have failed to replicate its effects. And several respected members of the science community have had some harshwords to say about the discipline and its methods. Listing all of these events in succession makes me feel a bit ashamed to call myself a social psychologist.
Clearly our field has been lacking both oversight and leadership if all of this could happen in such a brief period. II. III. IV. V. Power of Suggestion - The Chronicle Review. By Tom Bartlett New Haven, Conn. Mark Abramson for The Chronicle Review John Bargh rocked the world of social psychology with experiments that showed the power of unconscious cues over our behavior. A framed print of "The Garden of Earthly Delights" hangs above the moss-green, L-shaped sectional in John Bargh's office on the third floor of Yale University's Kirtland Hall. By Bosch's standard, it's too much to say the past year has been hellish for Bargh, but it hasn't been paradise either.
Psychology may be simultaneously at the highest and lowest point in its history. At the same time, psychology has been beset with scandal and doubt. Much of the criticism has been directed at priming. What have they found? As in so many other famous psychological experiments, the researcher lies to the subject. The words the subject was asked to rearrange were not random, though they seemed that way (this was confirmed in postexperiment interviews with each subject). It's a cute finding. Six Ways to Clean Up Science. Nfronting the 'sloppiness' that pervades science | Suzi Gage and Pete Etchells | Science. The journal Cortex is pioneering a new publication model that gives the peer review process a much-needed overhaul.
Photograph: public domain Last week, Tilburg University published a damning final report into the actions of the disgraced social psychologist Diederik Stapel, with an accompanying press release that emphasises "a general culture of careless, selective and uncritical handling of research and data" in the field. Commentators have pointed out the need to take great care in making sweeping generalisations on the basis of extreme cases and, indeed, our own recent article somewhat overstated the extent of such extreme cases of fraud.
The general consensus in the ensuing discussion was that, although cases of premeditated fraud are rare, the problems caused by the failure to publish or replicate are important. It was within this framework that our session at the SpotOn London conference took place (you can watch it here). Chambers pointed to several benefits with the model. Brain Box: Bold predictions for good science. Undergraduates are taught proper scientific method. First, the experimenter makes a prediction, then s/he collects data to test that prediction. Standard statistical methods assume this hypothesis driven approach, most statistical inferences are invalid unless this rigid model is followed. But very often it is not. Very often experimenters change their hypotheses (and/or analyses methods) after data collection.
Indeed, students conducting their first proper research project are often surprised by this 'real-world' truth: "oh, that is how we really do it! ". After recent interest in science malpractice, fuelled by revelations of outright fraud, commentators are starting to treat the problem more seriously, especially in psychology and neuroscience. Arguably, scientific journals share much of the responsibility for allowing bad research practices to flourish. This is a bold new initiative, and if successful, could precipitate a major change in the way science is done. Column: When lab leaders take too much control : Naturejobs. The cult of significance testing. I recently found out about a book that was published earlier this year, The Cult of Statistical Significance by Stephen Ziliak and Deidra McCloskey. The subtitle is sure to stir up controversy: How the Standard Error Costs Us Jobs, Justice, and Lives.
From the parts I’ve read it sounds like the central criticism of the book is that statistical significance is not necessarily scientific significance. Statistical significance questions whether an effect exists and is unconcerned with the size or importance of the effect. Significance testing errs in two directions. First, in practice many people believe that any hypothesis with a p-value less than 0.05 is very likely true and important, though often such hypotheses are untrue and unimportant.
Second, many act as if a hypothesis with a p-value greater than 0.05 is “insignificant” regardless of context. Whenever someone raises objections to significance testing the reaction is always “Yes, everyone knows that.” Psychological Science...meet me at camera 3. Table of Contents — November 2012, 7 (6) When Replication Goes Bad. How to ensure that results in psychology (and other fields) are replicated has become a popular topic of discussion recently. There's no doubt that many results fail to replicate, and also, that people don't even try to replicate findings as much as they should.
Yet psychologist Gregory Francis warns that replication per se is not always a good thing: Publication bias and the failure of replication in experimental psychology Among experimental psychologists, successful replication enhances belief in a finding, while a failure to replicate is often interpreted to mean that one of the experiments is flawed. This view is wrong. Because experimental psychology uses statistics, empirical findings should appear with predictable probabilities. Say you took a pack of playing cards and removed half the red cards. Earlier this year, Francis wrote a critical letter about a paper published in PNAS purporting to show that wealthier Americans are less ethical.
Francis G (2012). What counts as a successful or failed replication? « The Hardest Science. Let’s say that some theory states that people in psychological state A1 will engage in behavior B more than people in psychological state A2. Suppose that, a priori, the theory allows us to make this directional prediction, but not a prediction about the size of the effect. A researcher designs an experiment — call this Study 1 — in which she manipulates A1 versus A2 and then measures B. Consistent with the theory, the result of Study 1 shows more of behavior B in condition A1 than A2. The effect size is d=0.8 (a large effect). A null hypothesis significance test shows that the effect is significantly different from zero, p<.05.
Now Researcher #2 comes along and conducts Study 2. The procedures of Study 2 copy Study 1 as closely as possible — the same manipulation of A, the same measure of B, etc. Here’s the question: did Study 2 successfully replicate Study 1? My answer is no. But I have two problems with defining replication in that way. 1. 2. 3. Like this: Like Loading... Paul Meehl on replication and significance testing « The Hardest Science. Nobel laureate challenges psychologists to clean up their act. Theory, and Why It's Time Psychology Got One. A peculiar prevalence of p values just below .05 - The Quarterly Journal of Experimental Psychology - Improving scientific practices in the wake of terrible examples of scientific practices. Crimes and Misdemeanors: Reforming Social Psychology » Random Assignment. How To Fix Science. Science Magazine: Sign In. NeuroChambers: The Dirty Dozen: A wish list for psychology and cognitive neuroscience. SNOOP:a program for demonstrating the cons... [Behav Res Methods. 2006.
Why I don't think publishing more negative results is worth any effort at all.