Evaluation discussion

Add flagships to trac

http://tfs.cc/trac

Grammar evaluation in terms of best practices

Inari will make a list of questions for all grammar writers to answer, and write about the results in D9.2, the final evaluation report.

D9.2 is due March 2013, let's set a deadline in February 2013.

D2.3 Best Practices

All MOLTO WPs that have grammars:

• WP4 : query grammar -- Ontotext; Maria
• WP6 : MGL -- Jordi, Ares
• WP7 : patents grammar -- Ramona, John
• WP8 : museum grammar -- Ramona, Dana, Aarne
• WP10 : Phrasebook -- Inari etc.
• WP11 : ACE grammar -- Kaarel, John
  • source code: https://github.com/Attempto/ACE-in-GF
  • documentation: http://www.molto-project.eu/biblio/deliverable/ace-grammar-library
  • evaluation wiki: https://github.com/Attempto/ACE-in-GF/wiki/Evaluation
• WP12 : Monnet grammars -- Christina

Developer's tools

• Eclipse plugin
• Test management tools
• Grammar diagnostic tools

Ongoing projects: Monnet/Be Informed, Query(?),

Already completed grammars: Phrasebook, Museum, ACE(?), Patents(?), MGL(?)

Things to report:

• use of developer's tools
• treebanks to automate the testing

Good grammar practices

Things to report:

Abstract syntax design

• Module structure: base grammar and domain extensions (query grammar as a good example)
• Language-independentness; good categories and abstractions (how to quantify that?)
• Manual or automatically derived from ontology (Monnet, Museum)

Concrete syntax implementation

• Use of functors and modules
• Linearization types according to Best practices document: no lexical categories
• Only API functions (mkC), no RGL internal functions or manual concatenation

This applies for all grammars, in progress or complete. Phrasebook was the first MOLTO grammar, in 2010 spring. We could compare it to newer grammars. MGL has been rewritten since WebALT; some reflections on that maybe. How about the evolution of Ontotext query grammar? First it was a bunch of rules to transform SPARQL-GF, only later transformed into a grammar where English, Swedish and SPARQL are all concrete syntaxes.