Guides to pre-registered experiments

Guides to methodological pre-registration for experiments

If you're like me, you've considered doing a pre-registered study but put it off because you weren't sure what to expect or how much paperwork there would be. I've become a big proponent of open science [I'm working on a future blog post on the topic], as I think it's crucially important to make your materials, data, conclusions, etc. available to other researchers and to the wider public!

As Simmons, Nelson, & Simonsohn (2017) wrote, pre-registration and full methodological disclosure are crucial to the credibility of psychological research. If we want to be taken seriously as scientists, we should behave in accordance with the highest standards of scientific integrity...and that includes pre-registering our studies.

Why pre-registration? Two big reasons: 1) it prevents us from fooling ourselves about our own research findings, and 2) it gives us something we can point to and say "Yes, I planned to do that all along!"

And if we didn't actually plan it all along, it forces us to face the facts—which should serve to keep us humble.

Despite what some people may think, increased transparency is good for science. Period. Sanjay Srivastava gives the topic a thoughtful treatment here, and I agree with him. Our priority should be high-quality science first, PR/funding concerns a DISTANT second. If we have good science in the first place, many of the other concerns will evaporate.

So, here are two resources to give you a good overview of the pre-registration process, and to guide you through what's required:

• A pretty comprehensive 22-page overview openly available on the OSF: https://osf.io/sn6by/
• A much briefer blog post by James Bartlett, a PhD student in the UK: https://bartlettje.github.io/2017-03-29-effective-preregistration/

A replacement for SPSS?

Could this program be the end of SPSS?

I have previously recommended JASP as a useful—and free!—statistical software package. I stand by that recommendation (nay, I'm doubling down on it!) as JASP has the following advantages:

1. A slick, easy-to-grasp user interface
2. All of the major types of statistical test, including one-sample, repeated-measures, and independent-samples t tests, ANOVA, ANCOVA, correlation, regression, and even the chi-square test for independence [i.e. the two-variable chi-square]. It even has a module for structural equation modeling, for those who conduct such analyses!
3. Bayesian analogues to each of the above tests
4. A simple, one-click method to run these tests, which makes it an ideal instructional tool (and useful for many basic research needs as well). 
5. It's a no-cost, open-source, cross-platform (Windows, MacOS, Linux) program, so there are zero barriers to personal use.
6. It launches pretty quickly, and runs extremely fast—even on low-powered computers. 

JASP Screenshot from my own personal computer 

This image is freely available for use; just cite http://psychsci.blogspot.com/2017/04/a-replacement-for-spss.html

The recent [March 21, 2017] release of version 0.8.1.1 has rendered JASP is even more useful than it was in the past! Here's the latest major change to the program:

• Data synchronization that (finally!) allows you to edit your data from within the program itself. You can sync a .csv file, .sav file, or .ods [LibreOffice spreadsheet] file.
• By the way, LibreOffice is a great, free alternative to Microsoft Office. I encourage you to check it out and shed the shackles of expensive, closed-source software!

Now that you can edit the data in a window in the statistical program itself (via data synchronization, which can be turned off if you so desire), and since a previous build allowed users to integrate JASP output with their OSF page, I think that JASP has finally become good enough to provide many researchers with all the statistical capability they need!

SPSS can still perform some of the more esoteric/advanced statistical procedures that JASP cannot, such as multi-level modeling. But since such procedures tend to be used relatively infrequently (at least in experimental social science research such as my own), JASP can probably handle the bulk of your analytical load.

Further, as a stats instructor, this tool is my secret weapon! I am encouraging students to use this program for an APA-style paper for which they have to run a handful of analyses to answer different questions.

This semester, I asked my Stats students how they felt about SPSS, and they generally weren't too fond of the program due to its complexity, pickiness, and uninformative error messages (not to mention an appearance that's stuck in the 1990s).

After I demonstrated JASP in class, the students seemed far more impressed with the free and open JASP than they were with the costly SPSS!

EDIT 5/30/2018: Just discovered that JASP is now available as an online resource, according to this blog post on the JASP website and a Tweet by E.J. Wagenmakers. You have to sign up for a RollApp account if you want to use this option.

And, if for some reason you're not a fan of JASP, a similar (and also free!) option called jamovi is under development. You can type data straight into this one, whereas I don't believe that's an option yet in JASP. jamovi lacks some features that JASP incorporates, but it's still a nice stats program for use in the classroom (or for your research)!

jamovi screenshot from my own personal computer

This image is freely available for use; just cite http://psychsci.blogspot.com/2017/04/a-replacement-for-spss.html

Should IBM be worried about SPSS adoption rates? Maybe...

Intrigued? Here is the website; you can download JASP by clicking 

the "Download" tab and selecting the version that's appropriate for your operating system. 
 

Or you can just follow this link instead. Hey, what do you have to lose? 

Try out this free stats program and see if it meets your needs! If it doesn't, you can just uninstall it...

If you'd like to try jamovi, here's the link for that.

Stat-ception II: How to fix statistics in psychology

Stat-ception Part II

I'm a star!

OK, my public speaking skills may not exactly have made me a star (yet!), but I AM on YouTube! I've included a link to my recent (Feb 2017) Cognition Forum presentations, as well as my current thinking about easily--and immediately implementable--solutions to ameliorate those weaknesses.

The first video goes into depth about the issues; the second describes my proposed solutions to those problems.

https://www.youtube.com/playlist?list=PLvPJKAgYsyoKcGOCKEYT2GyzK0yLVXvzN

For your viewing pleasure, I've also embedded the videos here:

 

Any feedback or advice is welcome!

I've also made the slideshows available on Google Drive. Here's the link to the first slideshow, so you can follow along: https://drive.google.com/file/d/0B4ZtXTwxIPrjTktiMGdoQ3JBSHM/view. And here's the link to the slideshow for the second video as well: https://drive.google.com/file/d/0B4ZtXTwxIPrjalZxdFJfUWNKTVU/view?usp=sharing

A draft of my manuscript on the topic (intended for eventual publication) is freely available for download at https://osf.io/preprints/psyarxiv/hp53k/. Since I'm an advocate of the open science movement, it's only right that I make my own work publicly available--hence why I uploaded these videos (and my manuscript) to public repositories.

You may not trust my own take on these issues, in which case I commend you for your skepticism! In the videos, I made numerous references to Ziliak & McCloskey (2009), Gigerenzer (2004), and Open Science Collaboration (2015)--all are worth reading, for anyone who cares about scientific integrity and the research process. All three works were highly influential in my thinking on this topic, though I cited a variety of other papers as well in my aforementioned manuscript.

You may disagree with my recommendations in the second video, and if so, that's okay! How to address the limitations of NHST and fix science is absolutely a discussion worth having; I advance my own ideas in the spirit of jump-starting such a discussion.

So, please put your thoughts in the comments, and share my work with colleagues who may be interested in the topic!

Stat-ception: Everything you think you know about psych stats is wrong!

In the spirit of open science, I have posted a video of a talk on statistical practice that I gave in the Cognition Forum at Bowling Green State University.


This talk was in 2 parts; the first part summarizes many of the common objections to null hypothesis significance testing (NHST) that thinkers have made over the decades, and the second part goes over my current recommendations to tackle the problem.

 

Part I is available at https://youtu.be/JgZZkMJhPvI; Part II is forthcoming! I've also embedded the video right here:

You can view and download the full slideshow at https://drive.google.com/open?id=0B4ZtXTwxIPrjTktiMGdoQ3JBSHM. The free (and very easy-to-use!) statistical program JASP can be found at https://jasp-stats.org/. JASP is useful if you want to run the analysis on the precision-vs-oomph example that I discuss at the end of the video (at the 39:41 mark).

I have already tackled some of the issues with NHST on more than one occasion in prior posts here, and I have also provided a practical guide to psych stats as a freely available educational resource!

There are a variety of excellent papers on the topic of statistical practice in social science fields; my working paper on the subject summarizes them. In the interest of open science, I've made this working paper available at https://osf.io/preprints/psyarxiv/hp53k/. Other great resources on the topic include Gigerenzer (2004) and Ziliak & McCloskey (2009), which are also freely available.