submission to preserve the formatting Third author

In this paper we establish a threshold for perceptually ac-ceptable beat tracking based on the mutual agreement of a committee of beat trackers. In the first step we use an existing annotated dataset to show that mutual agreement can be used to select one committee member as the most reliable beat tracker for a song. Then we conduct a lis-tening test using a subset of the Million Song Dataset to establish the threshold which results in acceptable qual-ity of the chosen beat output. For both datasets, we ob-tain a percentage of trackable music of about 73%, and we investigate which data tags are related to acceptable and problematic beat tracking. The results indicate that current datasets are biased towards genres which tend to be easy for beat tracking. The proposed methods provide a means to automatically obtain a confidence value for beat track-ing in non-annotated data and to choose between a number of beat tracking outputs. 1