WP6: Math

The math use case remains as it was, except that the use case may assume that premises requiring encyclopedic knowledge needed to frame word problems are given. Assuming that the math scenario will be embedded in the semantic wiki, the background premises may be given by the author of the problem in the facts database where the problems are formulated.

The mathematics use cases involve a problem author, a student and a teacher. The usability of the scenario is tested with realistic subjects playing each of these roles and the evaluation collected with a questionnaire and/or a journal. In addition, we should try estimate the savings from the system when scaled up to a larger use base and variety of languages, since these are the novelties in the MOLTO solution.

Evaluation for WP6

Diagnostic and progress evaluation for translation quality

WP 6 has developed a treebank based method for doing regression testing on the translations produced by the math grammar. A treebank entry consists of:

• An abstract tree for the gf grammar
• For each language (encoded as ISO 3 letter code), one or more Changesets.

A Changeset has:

• source: The person submitting it;
• revision: An integer equal to the svn revison in which this item is commited
• concrete: The proposed linearization
• and optionally a comment.

A defect is a difference between the actual linearization of an entry and the sample in the last changeset.

The procdure is as follows.

1. Using the gr command, create a list of abstract trees
2. Refine this list by removing or modifying unnatural productions (too deep, too long, too meaningless);
3. Add linearizations for all targeted languages: This makes the initial changeset;
4. Send the pairs (abstract tree, L linearization) to a fluent speaker of language L and ask for corrections;
5. Add the corrections to the treebank as new changesets.
6. Generate a list of defects and tackle them
7. Generate new linearizations, and go to step 4. Cycle until satisfied or out of resources.

See http://www.molto-project.eu/wiki/living-deliverables/d61-simple-drill-grammar-library/5-testing for further discussion.

Use case and usability evaluation