Fred Lakin

Center for the Study of Language and Information, Stanford University
Center for Design Research, Stanford University
Rehabilitation R&D Center, Palo Alto Veterans Hospital
3801 Miranda Ave, Palo Alto, California 94304
    ARPAnet: lakin@csli.stanford.edu

In modern user interfaces, graphics play an important role in the communication between human and computer. When a person employs text and graphic objects in communication, those objects have meaning under a system of interpretation, or visual language. Formal visual languages are ones which have been explicitly designed to be syntactically and semantically unambiguous. The research described in this paper aims at spatially parsing expressions in formal visual languages to recover their underlying syntactic structure. Such spatial parsing allows a general purpose graphics editor to be used as a visual language interface, giving the user the freedom to first simply create some text and graphics, and then later have the system process those objects under a particular system of interpretation. The task of spatial parsing can be simplified for the interface designer/implementer through the use of visual grammars.

For each of the four formal visual languages described in this paper, there is a specifiable set of spatial arrangements of elements for well formed visual expressions in that language. Visual Grammar Notation is a way to describe those sets of spatial arrangements; the context free grammars expressed in this notation are not only visual, but also machine readable, being used directly to guide the parsing.

Contents

1. Visual Languages in Human/Computer Interaction

2. Drawbacks of Special Purpose Visual Language Interfaces

3. Advantages of Parsing Visual Languages in a General Purpose Graphics Editor

4. Spatial Parsing for Visual Languages

4.1 Definitions
4.2 Examples of Visual Languages
4.3 Parsing and Interpretation of Visual Phrases in the User Interface
4.4 Spatial Parsing Versus Image Processing

5. Visual Grammar Directed Parsing of Visual Languages

5.1 Visually Notated Context Free Grammars
5.2 The Spatial Parser

6. Examples of Spatial Parsing using Visual Grammars

6.1 Feature Structures (directed acyclic graph notation used by linguists)
6.2 Sibtran (graphic devices for organizing textual sentence fragments)
6.3 Visual Grammar Notation

7. Theoretical Context: Understanding Conversational Graphics

8. Software Framework

9. Related Work

10. Conclusion

11. Acknowledgments

12. References

A version of this paper first appeared in the proceedings of AAAI-87, the conference of the AMERICAN ASSOCIATION for ARTIFICIAL INTELLIGENCE, Seattle, Washington, July 12-17, 1987, pp 683-688

Return to papers list

(C) Copyright 1994, 1996 PGC