Discussion: View Thread

Test your knowledge - how well do studies replicate?

  • 1.  Test your knowledge - how well do studies replicate?

    Posted 09-20-2019 07:07

    You are invited to join Replication Markets


    Dear Colleagues and Friends,

    Thanks a lot to many of you for participating in our previous prediction market projects, predicting replication outcomes in psychology, economics and other social science studies. We (many collaborators) are now launching a new and way bigger project: Replication Markets is recruiting participant-collaborators for a year-long social and behavioral science replication study. Please see below for more and please share the information below with anyone you think might be interested. Hope you want to participate! 😊

    All the best,

    Anna Dreber and the rest of the Team at Replication Markets


    Replication Markets (RM) invites you to help us predict outcomes of 3,000 social and behavioral science experiments over the next year. We actively seek scholars with different voices and perspectives to create a wise and diverse crowd, and hope you will join us.

    We invite you - your students, and any other interested parties - to join our crowdsourced prediction platform. By mid-2020 we will rate the replicability of claims from more than 60 academic journals. The claims were selected by an independent team that will also randomly choose about 200 for testing (replication).

     RM's forecasters bet on the chance that a claim will replicate and may adjust their assessment after reading the original paper and discussing results with other players. Previous replication studies have demonstrated prediction accuracy of about 75% with these methods.

     RM's findings will contribute to the wider body of scientific knowledge with a high-quality dataset of claim reliabilities, comparisons of several crowd aggregation methods, and insights about predicting replication. Anonymized data from RM will be open-sourced to train artificial intelligence models and speed future ratings of research claims.

     RM's citizen scientists predict experimental results in a play-money market with real payouts totaling over $100K*. Payouts will be distributed among the most accurate of its anticipated 500 forecasters. There is no cost to play the Replication Markets.

    Our project needs forecasters like you with knowledge, insight, and expertise in fields across the social and behavioral sciences. Please share this invitation with colleagues, students, and others who might be interested in participating.

    We hope to see you in the markets!

    On behalf of the Team at Replication Markets,


    Dr. Yiling Chen

    Harvard University

    Boston, MA, USA

    Dr. Thomas Pfeiffer

    Massey University

    Palmerston North, NZ

    Dr. Charles Twardy

    KeyW, a wholly owned subsidiary of Jacobs

    Herndon, VA, USA



    To learn more about the project,visit our homepage
    and sign up for our newsletter

    Ready to join Replication Markets?

    Sign up!


    *Funding provided by DARPA's Systematizing Confidence in Open Research and Evidence (SCORE) project.

    Karl Wennberg
    Linköping University