Mark Steyvers “Combining Human Judgments in General Knowledge and Forecasting Tasks”

    Abstract: In this research, we build on ideas from cognitive science and machine learning to build aggregation models that combine human judgments in general knowledge and prediction tasks. We propose that a successful approach to the aggregation of human judgment requires a cognitive modeling framework that explains how individuals produce their answers, and that also allows for individual differences in skill and expertise of participants. In addition, we argue that it is essential to correct for any systematic distortions in human judgment when aggregating judgments over individuals. We present two case studies that highlight our overall approach. In our first case study, we present preliminary results from the Aggregative Contingent Estimation System (ACES), which is part of a project funded by the Intelligence Advanced Research Projects Activity. The goal is to develop new methods for collecting and combining forecasts of many widely-dispersed individuals in order to increase aggregated forecasts’ predictive accuracy. In the second case study, we apply a cognitive modeling approach to the problem of measuring expertise on ranking tasks involving general knowledge (e.g., ordering American holidays through the calendar year) and forecasting (e.g., predicting the order of football teams at the end of the season). Using a Bayesian model of behavior on this problem that allows for individual differences in knowledge, we are able to infer people’s expertise directly from the rankings they provide and without using any knowledge of the true answer.

    Bio: Mark Steyvers is currently a Professor in the department of Cognitive Science and also has an appointment in the department of Computer Science at UC Irvine. He received his PhD from Indiana University and worked as a postdoctoral associate with Josh Tenenbaum at Stanford University. His recent work is on developing computational models for aggregating human judgments, including probability estimates, rankings, and problem solving behavior. He also works on probabilistic topic modeling, information retrieval, and knowledge representation. For his computational modeling work on understanding human activity, Dr. Steyvers received the New Investigator Award from the American Psychological Association as well as the Society of Experimental Psychologists.


    /groups/cssi/search/index.rss?tag=hotlist/groups/cssi/search/?tag=hotWhat’s HotHotListHot!?tag=hot1/groups/cssi/sidebar/HotListterrieTerrie Kellogg2014-09-25 15:46:50+00:002014-09-25 15:46:50updated5terrieTerrie Kellogg2014-09-25 15:44:37+00:002014-09-25 15:44:37updated4Added tag - hotcscfCSCF2014-09-25 15:44:35+00:002014-09-25 15:44:35addTag3cscfCSCF2014-09-25 14:56:43+00:002014-09-25 14:56:43updated2First createdcscfCSCF2014-09-25 14:55:46+00:002014-09-25 14:55:46created1wiki2014-09-25T15:46:50+00:00groups/cssi/wiki/5f1a4False2013 Archives/groups/cssi/wiki/5f1a4/2013_Archives.htmlTerrie Kellogg5 updates2013 Archives This is a collection of videos of the Cross-Departmental Seminar Series events. February 1, 2013 - Brian F. Schaffner "Inequality and Repr...Falseterrie2014-09-25T15:46:50+00:00hot/groups/cssi/search/index.rss?sort=modifiedDate&kind=all&sortDirection=reverse&excludePages=wiki/welcomelist/groups/cssi/search/?sort=modifiedDate&kind=all&sortDirection=reverse&excludePages=wiki/welcomeRecent ChangesRecentChangesListUpdates?sort=modifiedDate&kind=all&sortDirection=reverse&excludePages=wiki/welcome0/groups/cssi/sidebar/RecentChangesListmodifiedDateallRecent ChangesRecentChangesListUpdateswiki/welcomeNo recent changes.reverse5search