Werkoverleg met GvN.

Taken:

1. EarleyParser

   • In grafiek: ook fail als percentage van zinnen zonder onbekende woorden. Gedaan.

   • Automatic Parser Evaluation.

     • Implementeren. Gedaan.

     • Verschillende manieren van gokken bij onbekende woorden met elkaar vergelijken.
       Probleem: ook zonder gokken zijn er soms behoorlijke verschillen tussen de uitvoer van Alpino en de Earley-parser. Zie: http://www.let.rug.nl/~kleiweg/Earley/cmp/

Probleem met gokken op basis van kansen. Idiomatische uitdrukkingen, speciale constructies, hebben een lagere waarschijnlijkheid dan het algemene geval, omdat elk relatief weinig voorkomt. Maar als je eenmaal de woorden hebt die in zo'n speciale constructie passen, dan is de kans dat het om dat speciale geval gaat juist veel groter dan dat het toch om een algemeen geval gaat.

Meer algemeen:

De kans voor deze regel...

    X → A B

... wordt bepaald door fractie van deze regel t.o.v. andere regels...

    X → C D
    X → E F G

Ik denk dat de kans bepaald zou moeten worden door fractie van deze regel t.o.v. andere regels...

    Y → A B
    Z → A B

Testen met 1000 zinnen van elk minimaal vier woorden, met de Earley-parser, tegen grammatica en lexicon van clef_part0001.

Parses door Alpino en Earley van gewone zinnen gelden als norm (goudstandaard)

De testset bestaat uit dezelfde 1000 zinnen waarbij in in elke zin willekeurig een woord is vervangen door een onbekend woord. Parses vergeleken met de norm.

Test 1

for c := range catlist {
    cat := int32(c)
    count := float64(0)
    tcount := float64(0)
    n := float64(0)
    tn := float64(0)
    for _, r := range rules[cat] {
        count += r.count
        n++
    }
    for _, t := range terms[cat] {
        count += t.count
        tcount += t.count
        n++
        tn++
    }
    if tn > 0 {
        guess[cat] = n / count
    }
}

Resultaten:

Alpino:                                                  Earley:
   Precision          Recall       Crossing brackets       Precision          Recall       Crossing brackets
 Min.   :0.2609   Min.   :0.2400   Min.   :0.00000       Min.   :0.3043   Min.   :0.2593   Min.   :0.00000
 1st Qu.:0.7778   1st Qu.:0.7570   1st Qu.:0.00000       1st Qu.:0.8723   1st Qu.:0.8571   1st Qu.:0.00000
 Median :0.8706   Median :0.8400   Median :0.06092       Median :0.9756   Median :0.9538   Median :0.00000
 Mean   :0.8471   Mean   :0.8200   Mean   :0.07501       Mean   :0.9150   Mean   :0.9040   Mean   :0.03543
 3rd Qu.:0.9489   3rd Qu.:0.9115   3rd Qu.:0.11691       3rd Qu.:1.0000   3rd Qu.:1.0000   3rd Qu.:0.04938
 Max.   :1.0000   Max.   :1.0000   Max.   :0.38636       Max.   :1.0000   Max.   :1.0000   Max.   :0.34783

Test 2

for c := range catlist {
    cat := int32(c)
    count := float64(0)
    tcount := float64(0)
    n := float64(0)
    tn := float64(0)
    for _, r := range rules[cat] {
        count += r.count
        n++
    }
    for _, t := range terms[cat] {
        count += t.count
        tcount += t.count
        n++
        tn++
    }
    if tn > 0 {
        guess[cat] = n * n / count   // VERSCHIL MET TEST 1
    }
}

Resultaten:

Alpino:                                                  Earley:
   Precision          Recall       Crossing brackets        Precision          Recall       Crossing brackets
 Min.   :0.4000   Min.   :0.3462   Min.   :0.00000        Min.   :0.3667   Min.   :0.3548   Min.   :0.00000
 1st Qu.:0.7800   1st Qu.:0.7778   1st Qu.:0.00000        1st Qu.:0.8797   1st Qu.:0.8837   1st Qu.:0.00000
 Median :0.8727   Median :0.8512   Median :0.05556        Median :0.9667   Median :0.9600   Median :0.00000
 Mean   :0.8533   Mean   :0.8355   Mean   :0.06719        Mean   :0.9209   Mean   :0.9199   Mean   :0.02882
 3rd Qu.:0.9467   3rd Qu.:0.9211   3rd Qu.:0.10722        3rd Qu.:1.0000   3rd Qu.:1.0000   3rd Qu.:0.04189
 Max.   :1.0000   Max.   :1.0000   Max.   :0.32653        Max.   :1.0000   Max.   :1.0000   Max.   :0.30000

Test 3

Alles gelijke kans: 0.1

Resultaten:

Alpino:                                                  Earley:
   Precision          Recall       Crossing brackets        Precision          Recall       Crossing brackets
 Min.   :0.2609   Min.   :0.2400   Min.   :0.00000        Min.   :0.3043   Min.   :0.2593   Min.   :0.00000
 1st Qu.:0.7778   1st Qu.:0.7616   1st Qu.:0.00000        1st Qu.:0.8759   1st Qu.:0.8621   1st Qu.:0.00000
 Median :0.8717   Median :0.8423   Median :0.05777        Median :0.9779   Median :0.9574   Median :0.00000
 Mean   :0.8483   Mean   :0.8220   Mean   :0.07361        Mean   :0.9172   Mean   :0.9069   Mean   :0.03408
 3rd Qu.:0.9500   3rd Qu.:0.9130   3rd Qu.:0.11765        3rd Qu.:1.0000   3rd Qu.:1.0000   3rd Qu.:0.04878
 Max.   :1.0000   Max.   :1.0000   Max.   :0.35135        Max.   :1.0000   Max.   :1.0000   Max.   :0.34783

Test 4

for c := range catlist {
    cat := int32(c)
    count := float64(0)
    for _, t := range terms[cat] {
        count += t.count
        n++
    }
    guess[cat] = n / count
}

Resultaten:

Alpino:                                                  Earley:
   Precision          Recall       Crossing brackets        Precision          Recall       Crossing brackets
 Min.   :0.2609   Min.   :0.2400   Min.   :0.00000        Min.   :0.3043   Min.   :0.2593   Min.   :0.00000
 1st Qu.:0.7778   1st Qu.:0.7571   1st Qu.:0.00000        1st Qu.:0.8723   1st Qu.:0.8571   1st Qu.:0.00000
 Median :0.8690   Median :0.8409   Median :0.06024        Median :0.9779   Median :0.9545   Median :0.00000
 Mean   :0.8472   Mean   :0.8203   Mean   :0.07530        Mean   :0.9153   Mean   :0.9045   Mean   :0.03529
 3rd Qu.:0.9495   3rd Qu.:0.9124   3rd Qu.:0.11790        3rd Qu.:1.0000   3rd Qu.:1.0000   3rd Qu.:0.04878
 Max.   :1.0000   Max.   :1.0000   Max.   :0.38636        Max.   :1.0000   Max.   :1.0000   Max.   :0.34783

Test 5

total := 0
for c := range catlist {
    cat := int32(c)
    count := float64(0)
    n := float64(0)
    for _, t := range terms[cat] {
        total += int(t.count)
        count += t.count
    }
    guess[cat] = count
}
for key := range guess {
    guess[key] /= float64(total)
}

Resultaten:

Alpino:                                                  Earley:
   Precision          Recall       Crossing brackets        Precision          Recall       Crossing brackets
 Min.   :0.2273   Min.   :0.2000   Min.   :0.00000        Min.   :0.2273   Min.   :0.1852   Min.   :0.00000
 1st Qu.:0.7748   1st Qu.:0.7555   1st Qu.:0.00000        1st Qu.:0.8603   1st Qu.:0.8541   1st Qu.:0.00000
 Median :0.8660   Median :0.8393   Median :0.06061        Median :0.9730   Median :0.9496   Median :0.00000
 Mean   :0.8440   Mean   :0.8184   Mean   :0.07502        Mean   :0.9113   Mean   :0.9019   Mean   :0.03615
 3rd Qu.:0.9456   3rd Qu.:0.9091   3rd Qu.:0.11691        3rd Qu.:1.0000   3rd Qu.:1.0000   3rd Qu.:0.05263
 Max.   :1.0000   Max.   :1.0000   Max.   :0.40909        Max.   :1.0000   Max.   :1.0000   Max.   :0.40909

Test 6

itotal := int64(0)
pref := make(map[string]int64)
for t, cat := range cathash {
    if t == "" {
        continue
    }
    i := strings.Index(t, "(")
    if i > 0 {
        t = t[:i]
    }
    for _, n := range terms[cat] {
        pref[t] += int64(n.count)
        itotal += int64(n.count)
    }
}
for t, cat := range cathash {
    if t == "" {
        continue
    }
    i := strings.Index(t, "(")
    if i > 0 {
        t = t[:i]
    }
    if pref[t] > 0 {
        guess[cat] = float64(pref[t]) / float64(itotal)
    }
}

Resultaten:

Alpino:                                                  Earley:
   Precision          Recall       Crossing brackets        Precision          Recall       Crossing brackets
 Min.   :0.2609   Min.   :0.2400   Min.   :0.00000        Min.   :0.3043   Min.   :0.2593   Min.   :0.00000
 1st Qu.:0.7727   1st Qu.:0.7600   1st Qu.:0.00000        1st Qu.:0.8778   1st Qu.:0.8750   1st Qu.:0.00000
 Median :0.8667   Median :0.8421   Median :0.06061        Median :0.9740   Median :0.9630   Median :0.00000
 Mean   :0.8442   Mean   :0.8229   Mean   :0.07515        Mean   :0.9160   Mean   :0.9109   Mean   :0.03338
 3rd Qu.:0.9467   3rd Qu.:0.9155   3rd Qu.:0.11662        3rd Qu.:1.0000   3rd Qu.:1.0000   3rd Qu.:0.04290
 Max.   :1.0000   Max.   :1.0000   Max.   :0.38636        Max.   :1.0000   Max.   :1.0000   Max.   :0.34783

CategoryParsing