TY - CONF T1 - Conference v2.0: An uncertain version of the OAEI Conference benchmark T2 - 13th International Semantic Web Conference (ISWC 2014) Y1 - 2014 A1 - Michelle Cheatham A1 - Pascal Hitzler ED - Peter Mika ED - Tania Tudorache ED - Abraham Bernstein ED - Chris Welty ED - Craig A. Knoblock ED - Denny Vrandecic ED - Paul T. Groth ED - Natasha F. Noy ED - Krzysztof Janowicz ED - Carole A. Goble KW - benchmark KW - OAEI KW - Ontology Alignment AB - The Ontology Alignment Evaluation Initiative is a set of benchmarks for evaluating the performance of ontology alignment systems. In this paper we re-examine the Conference track of the OAEI, with a focus on the degree of agreement between the reference alignments within this track and the opinion of experts. We propose a new version of this benchmark that more closely corresponds to expert opinion and confidence on the matches. The performance of top alignment systems is compared on both versions of the benchmark. Additionally, a general method for crowdsourcing the development of more benchmarks of this type using Amazon’s Mechanical Turk is introduced and shown to be scalable, cost-effective and to agree well with expert opinion. JF - 13th International Semantic Web Conference (ISWC 2014) PB - Lecture Notes in Computer Science, Springer CY - Riva del Garda, Italy VL - 8797 ER -