Semantic accuracy.

An update is a logical formula which can be evaluated against an information state and which gives rise to a new, updated information state. The most straightforward method for evaluating concept accuracy in this setting is to compare the update produced by the grammar with the annotated update. One problem with this approach is the fact that the update language does not provide a simple way to compute equivalence of updates (there is no notion of normal form for update expressions). A further obstacle is the fact that very fine-grained semantic distinctions can be made in the update-language. While these distinctions are relevant semantically (i.e. in certain cases they may lead to slightly different updates of an information state), they often can be ignored by a dialogue manager. For instance, the updates below are semantically not equivalent, as the ground-focus distinction is slightly different. In the first update the feature place is supposed to be ground, whereas in the second update, it is part of the focus.

However, the dialogue manager will decide in both cases that this is a correction of the destination town.

Since semantic analysis is the input for the dialogue manager, we have therefore measured concept accuracy in terms of a simplified version of the update language. Following a somewhat similar proposal in Boros et al. (1996), we translate each update into a set of ``semantic units'', were a unit in our case is a triple $\langle\mbox{\it CommunicativeFunction Slot Value}\rangle$. For instance, the examples above translate as

Both the updates in the annotated corpus and the updates produced by the system are translated into semantic units of the form given above. The syntax of the semantic unit language and the translation of updates to semantic units is defined in van Noord (1997), but note that the translation of updates to semantic units is relatively straightforward and is not expected to be a source of discussion, because the relation is many to one.

Semantic accuracy can now be defined as follows. Firstly, we list the proportion of utterances for which the corresponding semantic units exactly match the semantic units of the annotation ( exact match). Furthermore we calculate precision (the number of correct semantic units divided by the number of semantic units which were produced) and recall (the number of correct semantic units divided by the number of semantic units of the annotation). Finally, following Boros et al. (1996) we also present concept accuracy as

$$\mbox{\it CA} = \left( 1 - \frac{SU_S + SU_I + SU_D}{SU} \right)$$ (5)

where SU is the total number of semantic units in the corpus annotation, and SU_S, SU_I, and SU_D are the number of substitutions, insertions, and deletions that are necessary to make the (translated) update of the analysis equivalent to (the translation of) the corpus update.