next up previous contents
Next: Crossing Brackets (Level 2 Only) Up: The SPARKLE Evaluation Scheme Previous: The SPARKLE Evaluation Scheme

Recall and Precision

 

Section §2 briefly discussed the Grammar Evaluation Interest Group evaluation scheme (GEIG; Grishman et al., 1991). This computes the degree of match between unlabelled bracketings derived from corpus treebanks with those derived from parses for the same sentences. More recent work (e.g. Magerman, 1995) has evaluated with respect to labelled bracketings; since the annotation schemes include constituent/functional labels we will use this variant in the SPARKLE scheme.

Recall and precision are defined over either chunks/slabs, phrasal constituents, or grammatical relations, depending on the annotation level. We compute:

CCRs match if they are identical. The case where the functors stand in a subsumption relationship and the arguments are identical is defined as a partial match; for example, an ADJ_C chunk is compatible with an underspecified chunk marked as ADJPART_C, meaning something that can be either an adjective or a past participle. Thus, a parser is able to return an underspecified representation--without being unduly penalised--in situations where it does not have access to information that would help it to resolve the ambiguity. It is likely that a partial match will be assigned a score less than the value of 1 assigned to complete matches, but at this stage we will not attempt to assign this value precisely since it should probably depend on the annotation level and the degree of underspecification.

As an example of how the recall and precision measures work, if the annotated test corpus contains

[N_C the interested watcher]
[FV_C could always observe]
[N_C the stars]
[PART_C visible]
[P_C to the naked eye]
and a parser returns
[N_C the interested watcher]
[N_C could always observe the stars]
[ADJ_C visible]
[P_C to the naked eye]
the recall is 2/5 (40%) and precision is 2/4 (50%), since 2 CCRs match, the annotated corpus contains 5 CCRs, and the parser returns 4.



next up previous contents
Next: Crossing Brackets (Level 2 Only) Up: The SPARKLE Evaluation Scheme Previous: The SPARKLE Evaluation Scheme

Sparkle Project