Wednesday, April 19, 2017

...And the winner is



...And the winner is...

In the ever-changing landscape of social media today, have you wondered lately what forms of social media college students are using most often?

The students in my Spring 2017 stats class were wondering this very question! So, as a brief introduction to research (and as an example of the one-way ANOVA, which we had recently covered), I offered my class the option to get a couple extra credit points for surveying 5 of their friends about their social media usage.

Read on for a snapshot of social media usage among college students right now!

Limitations

I intentionally designed the survey with a couple weaknesses, to give the students some practice at identifying those limitations. We used a 7-point Likert-type scale [it's pronounced LICK-ert, by the way! In case that link goes dead, here's a cached version].
  1. Only the endpoints on this scale were labeled: a 1 indicated "I never use this form of social media" and a 7 indicated "I use this form of social media multiple times per day."

    Not having any labels for the intermediate values is a weakness because it introduces an unacceptable amount of error based on how people interpret a particular number—how do we know that you and I interpret a value of "6" the same way?

    Answer: we don't. Hence, this is a weakness. And a rather serious one!
     
  2. Another major weakness is that these students each asked about 5 friends at Bowling Green State University.

    Given that this is a Psych Stats course, many students are Psych majors. Given that fact, they probably have a disproportionately high number of friends who also major in psychology. Are Psych majors representative of all BGSU students, let alone all college students?

    Not necessarily; hence, the sampling procedure is another major limitation of this study.

    For purposes of a class demonstration, this flawed sample is fine. But it severely limits the ability to generalize the results to all BGSU undergraduate students, let alone college students nationwide. Or, at least, the sampling procedure inspires some doubt about generalizibility.
     
  3. A third limitation is that I only included 5 forms of social media, rather than a more complete list. One student suggested including Tumblr, which is defensible—but for simplicity's sake, I shot that idea down.

    Respondents gave self-report data (on the aforementioned 1-7 scale) regarding their usage of: Facebook, Snapchat, Instagram, Twitter, and Pinterest. That's it.

    So, usage of LinkedIn, reddit, tumblr., Google+, flickr, SoundCloud, and other social networking sites were left out of the picture here. Even MySpace has stuck around, as musicians sometimes use it to gain additional exposure for their work. These sites are not captured in this survey.

Nonetheless, some data is better than no data! As far as student engagement goes, this data is also better than made-up data, because we're looking at real responses from real people—even if the survey methodology is less-than-ideal!

Results

The results of the survey are posted in .csv format on my Google Drive, publicly accessible here. I did the analysis in JASP, which I've previously recommended for many use cases (the complete analysis is available here) and in the even newer program jamovi (that analysis is available here).

Here's the [un-editable] graph generated by JASP:

And here are the descriptive stats:

A couple highlights:
  • Snapchat is the clear winner, with the highest mean (5.671) and the lowest SD (1.819)
  • Instagram takes second place, Facebook is a close third, and Twitter lags behind. Pinterest is a distant last place in this sample
  • The F-ratio was 'statistically significant': F(4, 360) = 22.08, p < .001
  • For effect size, I used eta-squared: Eta-squared = 0.197
  • A post-hoc analysis (with Tukey correction) reveals that Pinterest is significantly different from all others (duh!) and Snapchat is significantly different from Twitter. Instagram and Twitter are also significantly different.
     
  • Statisticians will note that Levene's test reveals a violation of the assumption of equality of variance. Strictly speaking, this means that we should not run an ANOVA; instead, we should use a non-parametric alternative like the Kruskal-Wallis H-test.

    In my experience, though, this rarely yields a fundamentally different result. And after you run the H-test, you still need a post-hoc test anyway!

For convenience's sake, I've screencapped the post-hoc test as well. [Click to enlarge image]

I ran the post-hoc test in the brand-new stats program jamovi, which allows you to run the post-hoc test with no correction or with several of the most frequently-used correction procedures. I like how jamovi let me do the analysis both ways, and showed the results side-by-side.

You can see that a post-hoc analysis with no correction for multiple comparisons yields a significant difference for Facebook vs. Snapchat. It also shows that Facebook and Twitter are almost, but not quite, significantly different (p = .055). Should we ignore this result because it didn't meet the sacred .05 criterion?

I'd say that we should consider it in the context of the study. What are we looking for? Patterns in usage of social media among college students (specifically, college students at BGSU).

What are we trying to accomplish? Well, let's suppose I'm trying to advertise a product or service to college students, in which case I want my ad to be seen by as many college students as possible, for as few $$$ as possible.

Even if the difference between Facebook and Twitter usage isn't significant at the conventional alpha level of .05, if we're talking about efficiency of time, effort, and money, it's close enough that I'd certainly consider advertising on Facebook instead of Twitter!

So is Tukey's correction (or another multiple correction procedure) necessary here? It's certainly debatable; I fall on the "no" side of things—after all, if there's a significant ANOVA, then there's clearly a significant difference somewhere, right? Multiple correction procedures reduce power, so if you use a correction like Tukey's test, you could end up with a significant ANOVA but no significant post-hoc results!

And significance is kind of overblown, anyway... 

_________________

Remember, if you're interested in a more nuanced analysis, you can download the .csv file linked above and run the analyses yourself! I suggest using JASP or jamovi, which are both free of cost and open-source!

Friday, April 14, 2017

Guides to pre-registered experiments



Guides to methodological pre-registration for experiments

If you're like me, you've considered doing a pre-registered study but put it off because you weren't sure what to expect or how much paperwork there would be. I've become a big proponent of open science [I'm working on a future blog post on the topic], as I think it's crucially important to make your materials, data, conclusions, etc. available to other researchers and to the wider public!

As Simmons, Nelson, & Simonsohn (2017) wrote, pre-registration and full methodological disclosure are crucial to the credibility of psychological research. If we want to be taken seriously as scientists, we should behave in accordance with the highest standards of scientific integrity...and that includes pre-registering our studies.

Why pre-registration? Two big reasons: 1) it prevents us from fooling ourselves about our own research findings, and 2) it gives us something we can point to and say "Yes, I planned to do that all along!"

And if we didn't actually plan it all along, it forces us to face the facts—which should serve to keep us humble.

Despite what some people may think, increased transparency is good for science. Period. Sanjay Srivastava gives the topic a thoughtful treatment here, and I agree with him. Our priority should be high-quality science first, PR/funding concerns a DISTANT second. If we have good science in the first place, many of the other concerns will evaporate.

So, here are two resources to give you a good overview of the pre-registration process, and to guide you through what's required:

Thursday, April 6, 2017

A replacement for SPSS?



Could this program be the end of SPSS?

I have previously recommended JASP as a useful—and free!—statistical software package. I stand by that recommendation (nay, I'm doubling down on it!) as JASP has the following advantages:
  1. A slick, easy-to-grasp user interface
  2. All of the major types of statistical test, including one-sample, repeated-measures, and independent-samples t tests, ANOVA, ANCOVA, correlation, regression, and even the chi-square test for independence [i.e. the two-variable chi-square]. It even has a module for structural equation modeling, for those who conduct such analyses!
  3. Bayesian analogues to each of the above tests
  4. A simple, one-click method to run these tests, which makes it an ideal instructional tool (and useful for many basic research needs as well). 
  5. It's a no-cost, open-source, cross-platform (Windows, MacOS, Linux) program, so there are zero barriers to personal use.
  6. It launches pretty quickly, and runs extremely fast—even on low-powered computers. 

JASP Screenshot from my own personal computer 

This image is freely available for use; just cite http://psychsci.blogspot.com/2017/04/a-replacement-for-spss.html

The recent [March 21, 2017] release of version 0.8.1.1 has rendered JASP is even more useful than it was in the past! Here's the latest major change to the program:
  • Data synchronization that (finally!) allows you to edit your data from within the program itself. You can sync a .csv file, .sav file, or .ods [LibreOffice spreadsheet] file.
Now that you can edit the data in a window in the statistical program itself (via data synchronization, which can be turned off if you so desire), and since a previous build allowed users to integrate JASP output with their OSF page, I think that JASP has finally become good enough to provide many researchers with all the statistical capability they need!

SPSS can still perform some of the more esoteric/advanced statistical procedures that JASP cannot, such as multi-level modeling. But since such procedures tend to be used relatively infrequently (at least in experimental social science research such as my own), JASP can probably handle the bulk of your analytical load.

Further, as a stats instructor, this tool is my secret weapon! I am encouraging students to use this program for an APA-style paper for which they have to run a handful of analyses to answer different questions.

This semester, I asked my Stats students how they felt about SPSS, and they generally weren't too fond of the program due to its complexity, pickiness, and uninformative error messages (not to mention an appearance that's stuck in the 1990s).

After I demonstrated JASP in class, the students seemed far more impressed with the free and open JASP than they were with the costly SPSS!

EDIT 5/30/2018: Just discovered that JASP is now available as an online resource, according to this blog post on the JASP website and a Tweet by E.J. Wagenmakers. You have to sign up for a RollApp account if you want to use this option.

And, if for some reason you're not a fan of JASP, a similar (and also free!) option called jamovi is under development. You can type data straight into this one, whereas I don't believe that's an option yet in JASP. jamovi lacks some features that JASP incorporates, but it's still a nice stats program for use in the classroom (or for your research)!

jamovi screenshot from my own personal computer
This image is freely available for use; just cite http://psychsci.blogspot.com/2017/04/a-replacement-for-spss.html

Should IBM be worried about SPSS adoption rates? Maybe...

Intrigued? Here is the website; you can download JASP by clicking 
the "Download" tab and selecting the version that's appropriate for your operating system. 
 
Or you can just follow this link instead. Hey, what do you have to lose? 
Try out this free stats program and see if it meets your needs! If it doesn't, you can just uninstall it...

If you'd like to try jamovi, here's the link for that.

ResearcherID