Automatic subject indexing of Dryad repository datasets: performance evaluation of HIVE and SmartHIVE

HIVE is an automatic metadata application being considered for the Dryad data repository. The goals of this study were to 1) determine if HIVE performance can be improved by subject training and 2) determine HIVE’s effectiveness in automatically generating controlled vocabulary terms that can be used to represent scientific data sets. HIVE performance was evaluated by 1) matching of HIVE-generated keywords with manually generated index terms (precision and recall) and 2) human evaluation of keyword relevancy. Part 1 of the study found that subject training HIVE had no effect on the number of "correct" keywords, precision, or recall. Part 2 found that despite large inter-evaluator inconsistency, the trend was for domain experts to assign a statistically similar number of keywords as "relevant" to both the data (x=4.0) and the articles (x=3.3) suggesting that HIVE may be useful in creating subject metadata for data sets from the full-text article