Spatial Parsing for Visual Languages

Fred Lakin

Center for the Study of Language and Information, Stanford University
lakin@csli.stanford.edu
Center for Design Research, Stanford University
lakin@cdr.stanford.edu

Rehabilitation R&D Center, Palo Alto Veterans Hospital
3801 Miranda Ave, Palo Alto, California 94304

Graphics are very important in human/computer interaction. To participate effectively in this kind of interaction, computers must be able to understand how humans use graphics to communicate. When a person employs a text and graphic object in communication, that object has meaning under a system of interpretation, or visual language. A first step toward computer understanding of the visual communication objects used by humans is computer parsing of such objects, recovering their underlying syntactic structure. The research described in this paper combines computer graphics, symbolic computation and textual linguistics to accomplish spatial parsing for visual languages.

Processing of embedded visual languages: the "lumpy oatmeal" analogy

Expressions in formal visual languages are a subset of the images produced in general purpose text graphic sketching.

The expressions in embedded visual languages are like coagulated lumps in oatmeal ... composed of the same "text graphic stuff," just more highly structured.
The structures can be used to guide processing

Parsing has been investigated in four visual languages: VennLISP (a visual programming language based on LISP), VIC (a visual communication system for aphasics), FSA (finite state automaton) diagrams, and SIBTRAN (graphic devices for organizing textual sentence fragments). A parser has been written which can recover the structure for graphic communication objects in the different visual languages, and work has been begun on a new parser which uses visual grammars. In addition, interactive visual communication assistants utilize modules from the parser to actively assist humans using two of the visual languages.

SECTIONS

1. OVERVIEW
2. SPATIAL PARSING IN THE vmacs GRAPHICS EDITOR
    2.1 A Structure from Manipulation for Text-graphic Objects
    2.2 Parsing Visual Communication Objects
    2.3 Spatial Parsing Versus Image Processing
    2.4 Related Work
    2.5 Features in the PAM Framework Suited to Spatial Parsing
    2.6 Separation of Space and Tree in vmacs
3. EXAMPLES OF SPATIAL PARSING
    3.1 VennLISP (A Visual Programming Language Based on Lisp)
    3.2 VIC (A Visual Communication System for Aphasics)
    3.3 FSA (Finite State Automaton) Diagrams
    3.4 SIBTRAN (Graphic Devices for Organizing Textual Sentence Fragments)
    3.5 Informal Conversational Graphics (Corpus from Video Tape of Blackboard Use)
4. COMPARISON OF VISUAL LINGUISTICS AND VERBAL LINGUISTICS
5. SUMMARY
6. ACKNOWLEDGMENTS
7. FOOTNOTES
8. REFERENCES
9. LIST OF FIGURES

A version of this material first appeared as a chapter in VISUAL LANGUAGES, edited by Shi-Kuo Chang, Tadao Ichikawa, and Panos. A. Ligomenides, Plenum Press, 233 Spring Street, NY NY, 1986.

Return to papers list