How do treebank annotation schemes influence parsing results? or how not to compare apples with oranges

In the last decade, the Penn treebank has become the standard data set for evaluating parsers. The fact that most parsers are solely evaluated on this specific data set leaves the question unanswered how much these results depend on the annotation scheme of the tree-bank. In this paper, we will investigate the influence which different decisions in the annotation schemes of treebanks have on parsing. The investigation uses the comparison of similar treebanks of German, NE-GRA and TüBa-D/Z, which are subsequently modi-fied to allow a comparison of the differences. The re-sults show that deleted unary nodes and a flat phrase structure have a negative influence on parsing quality while a flat clause structure has a positive influence.