User Tools

Site Tools


lennart

The MI workbench: annotation and retrieval of multimodal instructions

In this talk we will present ongoing work on the development of the MI workbench. The MI workbench is a web application which can be used to store, annotate, retrieve and view multimodal instructions (MIs). Multimodal instructions are instructions that combine text and pictures. The MI workbench incorporates a corpus-based coding scheme that allows for a methodical investigation of the use of textual and pictorial information in health care instructions (Van Dijk et al., TABU’16). Annotation of these MIs involves encoding textual phenomena (e.g., length, syntactic and illocutionary properties), pictorial phenomena (e.g., visual and illocutionary properties) and the relations between them. The MI workbench supports certain aspects of this manual annotation task such as the computation of inter-annotator agreement. It also performs state-of-the-art OCR (powered by ABBYY) on MIs that are uploaded in PDF format. Moreover, the MI workbench facilitates corpus analysis as it features search options that allow the retrieval of MIs that match particular conditions informed by the coding scheme. Future work will involve annotation of (parts of) individual pictures in MIs, as well as automatic tagging of MIs based on empirical investigations to determine the relevant feature space of MIs.

lennart.txt · Last modified: 2019/02/06 16:03 (external edit)