Utrecht - 5th MOLTO project meeting

Utrecht 21.9.2012

Work group day.

Evaluation discussion

Add flagships to trac

http://tfs.cc/trac

Grammar evaluation in terms of best practices

Inari will make a list of questions for all grammar writers to answer, and write about the results in D9.2, the final evaluation report.

D9.2 is due March 2013, let's set a deadline in February 2013.

D2.3 Best Practices

All MOLTO WPs that have grammars:

Developer's tools

  • Eclipse plugin
  • Test management tools
  • Grammar diagnostic tools

Ongoing projects: Monnet/Be Informed, Query(?),

Already completed grammars: Phrasebook, Museum, ACE(?), Patents(?), MGL(?)

Things to report:

  • use of developer's tools
  • treebanks to automate the testing

Good grammar practices

Things to report:

Abstract syntax design

  • Module structure: base grammar and domain extensions (query grammar as a good example)
  • Language-independentness; good categories and abstractions (how to quantify that?)
  • Manual or automatically derived from ontology (Monnet, Museum)

Concrete syntax implementation

  • Use of functors and modules
  • Linearization types according to Best practices document: no lexical categories
  • Only API functions (mkC), no RGL internal functions or manual concatenation

This applies for all grammars, in progress or complete. Phrasebook was the first MOLTO grammar, in 2010 spring. We could compare it to newer grammars. MGL has been rewritten since WebALT; some reflections on that maybe. How about the evolution of Ontotext query grammar? First it was a bunch of rules to transform SPARQL-GF, only later transformed into a grammar where English, Swedish and SPARQL are all concrete syntaxes.

Evaluation questionnaire

In progress, do not start answering yet!

If some of the terms are completely unfamiliar to you, it doesn't matter. Just answer to those you have used.

1. Use of development tools

Answer yes to all tools that you have used.

a) Text editor + GF shell

if yes, some more questions:

  • Which editor?
  • Which operating system?
  • Give an estimate how large scale was your grammar
  • Any other comments

b) GF-Eclipse plug-in

  • Which OS?
  • How large scale was your grammar?
  • Any other comments

c) Cloud-based IDE (http://cloud.grammaticalframework.org/gfse/)

  • Why? Did you try other tools as well?
  • Any other comments (did it work? did it help your work? ...)

2. Use of diagnostic tools

Answer yes to all tools that you have used.

2.1 compilation diagnostics:

a) verbose (import -v FILE)

b) type check dumps (gf --dump-tc FILES)

c) Other, what?

2.2 grammar display modes:

a) print_grammar (pg) with different options. Examples below.

pg -words

Outputs all terminal tokens in the grammar. Example:

Hammurabi> pg -words
, amahhas amtahas awiilam awiilim awiilum ašaqqal ašebbir aštebir ...... pay shall silver slave slave's slaves slaves' the

pg -missing

Shows what abstract syntax constructions are missing from which concrete syntaxes. Example:

Phrasebook> pg -missing
PhrasebookCat : MMust ObjPlur ThesPlur ThesePlur ThosePlur
PhrasebookEng : ObjPlur ThesPlur ThesePlur ThosePlur

b) abstract_info (ai)

ai EXP

Shows info about an expression. Example:

Phrasebook> ai HowFarFromBy
fun HowFarFromBy : Place -> Place -> ByTransport -> Question ;
Probability: 6.666666666666667e-2

c) show_opers (so)

Shows available opers for a type. Requires first import -retain FILE. Example:

Hammurabi> i -retain HammurabiEng.gf
Hammurabi> so NP
defnt : CN -> NP
indef : CN -> NP
xOf : N2 -> CN -> NP
xOfPron : N2 -> Pron -> NP

d) compute_concrete (cc)

Computes the result of an oper. Requires first import -retain FILE. Example:

> import -retain ParadigmsGer.gf
> compute_concrete -list mkN "Buch" "Bücher" neuter
Buch, Buch, Buch / Buche, Buchs / Buches, Bücher, Bücher, Büchern,
Bücher, ResGer.Neutr

3. Generation and testing

a) treebanks

  • With Eclipse plug-in?
  • With a shellscript?
  • Manually?
  • Other, what?

b) random generation

  • With weights?

4. Resource grammar tools

Answer yes to all tools that you have used.

a) RGL synopsis document http://www.grammaticalframework.org/lib/doc/synopsis.html

b) RGL source browser http://www.grammaticalframework.org/lib/doc/browse/

c) download the source files and read them

  • Why? Was other documentation not helpful?

d) Library browser grammar to construct RGL expressions (LibraryBrowser.pgf here http://cloud.grammaticalframework.org/minibar/minibar.html)

5. Grammar writing

These questions are more "how?" than yes/no. Short answers are enough.

5.1 Starting point for your grammar

a) Test corpus of sentences you want to express in the grammar

  • Where did you get the sentences from?
  • Did you use the test corpus as a treebank later on? (See 3a)
  • Did you use the test corpus as a documentation for the abstract syntax?
  • Other comments

b) Abstract syntax based on an existing formal system

  • What system?
  • Other comments

c) Other method of abstract syntax design

  • What?

5.2 Abstract syntax

5.2.1 Basic unit of the grammar

a) Sentence

b) Paragraph or more

  • Why?

5.2.2 Semantic control

a) No semantic control in abstract syntax (John likes beer and beer likes John equal)

  • Do you have an external way to check?

b) Simple, restrictive abstract syntax (e.g. HowFarBy : Place -> ByTransport -> Question, so how far is John by beer? not allowed.)

c) Dependent types

5.3 Module structure

a) Base module and domain-specific extensions

A way to structure abstract syntax. Different abstract syntax files for base grammar and domain extensions, concrete syntax irrelevant for this question.

  • Specify if you want

b) Functors

A way to structure concrete syntax. It can be used together with base grammar and domain extensions, or with only a single abstract syntax file.

  • Do you use a functor in all levels of your grammar? If you use base grammar+domain extensions, do you have a functor only for the concrete syntax of the base grammar?
  • Other comments

5.4 Concrete syntax

a) lincats only from recommended RGL API categories, that is Text, Utt, S, QS, Cl, CN, NP, AP, Adv, Card (or with good reason also VP, VPSlash)

  • If you have base grammar and domain extensions, are all "hard to understand" categories hidden there and only easy ones visible in domain extensions?

b) lincats from non-recommended categorires: lexical (N, A, V) or structural (Det, Quant, RP).

  • If you have used these categories, why?
  • Other comments

c) lincats from categories that you have created yourself (see pg. 31-32 in Best practices document)

For example, you need a record type such as oper NPNationality : Type = {lang : NP ; country : NP ; prop : A} for all variants that are needed to express someone's nationality.

It is recommended to have user-friendly constructor in a base grammar, such as oper mkAuthor : Str -> Author = \a -> mkNP (mkPN a), and in domain extension use only mkAuthor "Rembrandt" instead of mkNP (mkPN "Rembrandt").

  • Did you use a type synonym and a special constructor?
  • If you have base grammar and domain extensions, is this hidden in the base part?

d) Only API-level functions (ParadigmsXxx, SyntaxXxx) -- no tables, no concatenation, no MorphoXxx opers.

  • If you have done otherwise, why?