This work package, WP1, is concerned with the specification of the shallow parsing task, the development of annotated test corpora for the various parsers, and the definition of an evaluation measure to quantify the parsers' performance on the task. This deliverable provides details of the corpus annotation scheme and associated evaluation scheme we have adopted.
The shallow parsers (developed in WP3) will be components of the lexical extraction systems to be constructed, testbeds for the integration of the extracted lexical data, and they will also form part of the demonstrator applications. Therefore, the proposed scheme for annotation of the test corpora and the corresponding evaluation measure are orientated to these tasks. However, in the interests of completeness and extensibility (and also compatibility) with schemes utilised by the wider research community, we present, in §2 and §3, brief surveys of extant corpus annotation and parser evaluation schemes. In §4 we give the background motivation behind the design of our annotation and evaluation schemes, describing them in detail in §5 and §6. The appendices contain examples of annotated sentences from the test corpora, and also give further details of site-specific aspects of the annotation and evaluation. The full versions of the test corpora are on the SPARKLE ftp site.