An overview of the Alpino Treebank tools

In this document the various tools of the Alpino Treebank will be discussed. These tools include:

Most of the Alpino Treebank tools support two types of annotated corpora. In the simple case, each dependency structure is represented by a single XML file. In addition, the tools support so-called compact annotated corpora, in which collections of XML files are concatenated and compressed. The following tools are relevant for compact corpora in particular:

Some tools are not yet documented:

Use dtsearch to search the corpus. dtsearch uses XPath expressions for querying.

A basic example is:

By using the -v option dtsearch will display the files that match the query using dtview. In this case, the XPath expression matches with any dependency structure which contains a node with category pp.

DTSearch will process directories recursively.

The usage information lists the possible output methods:

When no output method is selected dtsearch will print the filename.

In order to be able to specify relevant XPath queries, it is neccessary to know a little bit about the way in which dependency structures are encoded in XML. This encoding is rather straightforward. Nodes in the dependency structure are encoded by a recursive XML element node. Nodes contain a variety of attributes. The most important attributes are:

There are more attributes, but these are the most important ones.

The following query lists all dependency structures which contain a subject node which itself contains a node with the root form man. Because of the -s option, all matching sentences are displayed, where the matched part of the sentence is given in square brackets.

Further examples are perhaps hard to understand without some knowledge of XPath. There are various tutorials and reference manuals for XPath available on the web. See for example: What is XPath?

List all sentences containing a "krijgen-passive" in the h_suite treebank:

Sometimes it is convenient to get the matched portion of the sentence and nothing more. This is accomplished with the -c option. For instance, the following finds determiner phrases which contain an adverb as a daughter node.

Quantitative information can be obtained with the -l option. For instance, this query finds which relations occur as sisters to the tag relation:

More examples, for the interested reader to try out:

Find topicalized secondary objects with category NP:

Find occurrences of extraposition of comparatives out of topicicalized constituents:

It is also possible to search in the text of the sentence. To do this efficiently, we have developed mg_m_search, that should be used in such cases for very large treebanks. Searching for text can be done as follows:

This works, because there is atmost one sentence element - the string value of a node set in XPath is defined as the string value of the first element of that node set.

In addition to "contains()" there are various other XPATH functions that are useful in this context, such as "starts-with()" and "matches()". The latter is used for regular expression matching in XPath 2 (currently not supported).

In this case, the query will also match in case there are multiple comments (these are indeed allowed).

Use dtview to visualize Alpino Dependency structures on your screen.

DTView can visualize dependency structures and highlight nodes that match an XPath expression. This expression can be specified on the commandline or in the text entry at the top of the window.

The dtxslt tool can be used in a similar fashion as programs such as xsltproc to apply a stylesheet to dependency structures.

An example:

This example applies the print-sentence.xsl stylesheet to every .xml file under Machine/clef. The -r flag ensures dtxslt will recursively walk through the directory structure looking for .xml files and compact corpora.

Look for examples of stylesheets in Alpino/TreebankTools/stylesheets. Another place that shows the use of stylesheets in the Alpino Treebank is the create-sanity-check-stylesheet.py script in the misc-scripts directory.

The dttred program is a simple script which starts the TrEd tree editor, with the appropriate settings for Alpino dependency structures. The program normally takes one or more arguments: the file names you want to edit.

Alpino-specific functionality can be found under the User-defined→Alpino menu.

Documentation for TrEd

The dtedit program is a simple script which starts the Thistle editor, with the appropriate settings for Alpino dependency structures. The program normally takes one or more arguments: the file names you want to edit.

Please use the documentation of Thistle in the local installation directory of Thistle on your machine.

DTEdit is now obsolete. Please use dttred instead.

The Alpino corpora can be stored in what we call "Compact Corpora". They consist of two files per corpus: one file with the compressed data and a seperate file with the filename information. Together they’re in a way similar to .zip files, but their compression ratio (because they typically consist of a lot of small files) is much better. The compression method is based on dictzip (see www.dict.org).

All of our tools work transparently on files in a normal directory structure and those in a compact corpus.

The tool to manage compact corpora is called act, which stands for Alpino Corpus Tool. From the act point of view, a corpus is a directory that contains .xml files.

Let’s assume we have the following directory structure:

We want to create a compact corpus for cdb in the compact_corpora directory. Here are two ways to do this:

The same can be accomplished with:

Both methods will put two new files in the compact_corpora subdirectory:

Compact corpora will only contain .xml files. In our example above, 1.xml~, the CVS directory structure, and Makefile are all ignored.

Consider the following directory structure (only directories shown):

Assume only cdb and the directories starting with “AD” contain .xml files.

The following will create a compact version of the Machine directory structure under compact_corpora/Machine:

The target-directory will be created if necessary.

The default for \--targetdir is the current directory, so the following yields the same result as the previous example:

To replace the Machine subdirectory with a compact equivalent use the following:

The \--remove option will cause the source files to be removed after creating the compact corpus.

When using recursion to create compact corpora, the directories specified should not contain any xml files, i.e. they should (only) contain other directories.

Scenario: there’s a compact corpus and there are several files that have updates. Say, a couple of sentences have been reparsed.

Updating will work on a sparse directory structure, i.e. the directory structure only needs to contain the reparsed files. These files will be merged with the existing compact corpus or compact corpora.

When using \--recursive, the \--targetdir option provides a straight directory to directory mapping. With regular corpus directories (directories that contain xml files) however this is not possible, because when converting to a compact corpus the name of the last directory component has to map to a filename.

Let’s illustrate this with two examples. The cdb directory is a regular corpus directory and (therefore) contains .xml files. The cdb compact corpus will be created/updated below the compact_corpora directory:

The clef directory does not contain any .xml files, it contains other directories that contain the .xml files. In this case we have a direct mapping from directory to directory:

Therefore the directory arguments given to act \--recursive should not contain any .xml files, otherwise the compact corpora may end up in places not intended.

To extract a compact corpus use the \--extract flag. At the moment of writing extracting only works non-recursively.

Extracting creates a new directory with .xml files for every compact corpus specified on the commandline. For example, the following extracts a collection of compact corpora to a specific directory (current directory is the default):

Use \--force to have act overwrite existing directories.

To show the contents of a compact corpus use ‘``dtlist’'. The compact corpus `cdb can be specified as cdb, as cdb.index or as cdb.data.dz.

Note the complete path in the output and the numerical sorting.

DTGrep can be used to search in dependency structures with regular expressions. The dtgrep program basically is a stripped down grep program that knows about Compact Corpora. DTGrep uses Perl-compatible regular expressions.

The dtget tool simply takes a sequence of one or more filenames, and prints the corresponding dependency structures to standard output.