Sketches are a ubiquitous form of communication in engineering design due to their simplicity and efficiency. However, because of a lack of suitable machine-interpretation techniques, they are virtually unusable with current computer-aided design and engineering tools. The informal nature of sketches and their inherent ambiguity present a number of challenges to the development of such techniques. Here we address one particular challenge, the task of reliably locating and recognizing the intended visual objects from a continuous stream of pen strokes. We present an integrated sketch parsing and recognition approach, based on a novel mark-group-recognize paradigm, which is tailored to the domain of mechanical systems. In the first step of processing, the sketch is examined to identify certain delimiting symbols called “markers.” The remaining pen strokes are then partitioned into distinct clusters, each representing a single symbol. Finally, a trainable symbol recognizer is used to find the best interpretation of each cluster. We have used these techniques to build a sketch-based tool for designing and analyzing vibratory mechanical systems. This tool enables designers to analyze and animate vibratory systems by simply sketching them on a tablet computer. User studies indicate that even first-time users find our tool to be effective.
Issue Section:
Design Theory and Methodology
1.
Shpitalni
, M.
, and Lipson
, H.
, 1995, “Classification of Sketch Strokes and Corner Detection Using Conic Sections and Adaptive Clustering
,” ASME J. Mech. Des.
0161-8458, 119
, pp. 131
–135
.2.
Ullman
, D. G.
, Wood
, S.
, and Craig
, D.
, 1990, “The Importance of Drawing in the Mechanical Design Process
,” Comput. Graph.
0097-8930, 14
(2
), pp. 263
–274
.3.
Schutze
, M.
, Sachse
, P.
, and Romer
, A.
, 2003, “Support Value of Sketching in the Design Process
,” Res. Eng. Des.
, 14
, pp. 89
–97
. 0097-89304.
Yang
, M. C.
, 2003, “Concept Generation and Sketching: Correlations With Design Outcome
,” ASME Design Engineering Technical Conferences and Design Theory and Methodology Conference
, Chicago, IL, September 2–6.5.
Song
, S.
, and Agogino
, A. M.
, 2004, “Insights on Designers’ Sketching Activities in New Product Design Teams
,” ASME Design Engineering Technical Conferences and Computers and Information in Engineering Conference
, Salt Lake City, UT, September 28–October 2.6.
Yang
, M. C.
, and Cham
, J. G.
, 2007, “An Analysis of Sketching Skill and its Role in Early Stage Engineering Design
,” ASME J. Mech. Des.
0161-8458, 129
(5
), pp. 476
–482
.7.
Chusilp
, P.
, and Jin
, Y.
, 2006, “Impact of Mental Iteration on Concept Generation
,” ASME J. Mech. Des.
0161-8458, 128
(1
), pp. 14
–25
.8.
de Silva
, R.
, Bischel
, D. T.
, Lee
, W.
, Peterson
, E. J.
, Calfee
, R. C.
, and Stahovich
, T. F.
, 2007, “Kirchhoff’s Pen: A Pen-Based Circuit Analysis Tutor
,” Eurographics Workshop on Sketch-Based Interfaces and Modeling
, June 11–13, Annecy, France.9.
Fonseca
, M. J.
, Pimentel
, C.
, and Jorge
, J. A.
, 2002, “Cali-an Online Scribble Recognizer for Calligraphic Interfaces
,” AAAI Spring Symposium on Sketch Understanding
, March 25–27, Palo Alto, CA, pp. 51
–58
.10.
Narayanaswamy
, S.
, 1996, “Pen and Speech Recognition in the User Interface for Mobile Multimedia Terminals
,” Ph.D. thesis, University of California at Berkeley, Berkeley, CA.11.
Landay
, J. A.
, and Myers
, B. A.
, 2001, “Sketching Interfaces: Toward More Human Interface Design
,” IEEE Computer
, 34
(3
), pp. 56
–64
.12.
Gennari
, L.
, Kara
, L. B.
, and Stahovich
, T. F.
, 2004, “Combining Geometry and Domain Knowledge to Interpret Hand-Drawn Diagrams
,” AAAI Fall Symposium Series 2004: Making Pen-Based Interaction Intelligent and Natural
, October 21–24, Washington, D.C.13.
Kara
, L. B.
, and Stahovich
, T. F.
, 2004, “Hierarchical Parsing and Recognition of Hand-Sketched Diagrams
,” User Interface Software Technology (UIST)
, October 24–27, Santa Fe, NM.14.
Saund
, E.
, Mahoney
, J.
, Fleet
, D.
, Larner
, D.
, and Lank
, E.
, 2002, “Perceptual Organisation as a Foundation for Intelligent Sketch Editing
,” AAAI Spring Symposium on Sketch Understanding
, March 25–27, Palo Alto, CA, pp. 118
–125
.15.
Jacobs
, D. W.
, 1988, “The Use of Grouping in Visual Object Recognition
,” MIT AI Lab
, Technical Report No. 1023.16.
LaViola
J.
, and Zeleznik
, R.
, 2004, “Mathpad2: A System for the Creation and Exploration of Mathematical Sketches
,” ACM Transactions on Graphics (Proceedings of SIGGRAPH 2004)
, August 8–12, Los Angeles, CA, Vol. 23
, pp. 432
–440
.17.
Notowidigdo
, M.
, and Miller
, R. C.
, 2004, “Off-Line Sketch Interpretation
,” AAAI Fall Symposium Series 2004: Making Pen-Based Interaction Intelligent and Natural
, October 21–24, Washington, D.C.18.
Costagliola
, G.
, and Deufemia
, V.
, 2003, “Visual Language Editors Based on lr Parsing Techniques
,” Proceedings of the Eighth International Workshop on Parsing Technologies (IWPT’03)
, Nancy, France.19.
Sezgin
, T. M.
, and Davis
, R.
, 2005, Hmm-Based Efficient Sketch Recognition, International Conference on Intelligent User Interfaces (IUI’05)
, New York, January 9–12.20.
Yasuda
, H.
, Takahashi
, K.
, and Matsumoto
, T.
, 2000, “A Discrete Hmm for Online Handwriting Recognition
,” Int. J. Pattern Recognit. Artif. Intell.
, 14
(5
), pp. 675
–688
. 0218-001421.
Grimson
W. E. L.
, 1991, “The Combinatorics of Heuristic Search Termination for Object Recognition in Cluttered Environments
,” IEEE Trans. Pattern Anal. Mach. Intell.
0162-8828, 13
(9
), pp. 920
–935
.22.
Shilman
, M.
, Pasula
, H.
, Russell
, S.
, and Newton
, R.
2002, “Statistical Visual Language Models for Ink Parsing
,” AAAI Spring Symposium on Sketch Understanding
, March 25–27, Palo Alto, CA, pp. 126
–132
.23.
Rubine
, D.
, 1991, “Specifying Gestures by Example
,” Comput. Graph.
, 25
, pp. 329
–337
. 0097-893024.
Shilman
, M.
, and Viola
, P.
, 2004, “Spatial Recognition and Grouping of Text and Graphics
,” EU-ROGRAPHICS Workshop on Sketch-Based Interfaces and Modeling
, August 30–31, Grenoble, France.25.
Alvarado
, C.
, and Davis
, R.
, 2005, “Dynamically Constructed Bayes Nets for Multi-Domain Sketch Understanding
,” International Joint Conference on Artificial Intelligence
, July 30–August 5, Edinburgh, Scotland, UK.26.
Kimura
, T. D.
, Apte
, A.
, and Sengupta
, S.
, 1994, “A Graphic Diagram Editor for Pen Computers
,” Software Concepts and Tools
, pp. 82
–95
.27.
Ozer
, O. F.
, Ozun
, O.
, Tuzel
, C. O.
, Atalay
, V.
, and Cetin
, A. E.
, 2001, “Vision-Based Single-Stroke Character Recognition for Wearable Computing
,” IEEE Intell. Syst.
1094-7167, 16
(3
), pp. 33
–37
.28.
Apte
, A.
, Vo
, V.
, and Kimura
, T. D.
, 1993, “Recognizing Multistroke Geometric Shapes: An Experimental Evaluation
,” Proceedings of the UIST 93
, November 3–5, Atlanta, GA, pp. 121
–128
.29.
Fonseca
, M. J.
, and Jorge
, J. A.
, 2000, “Using Fuzzy Logic to Recognize Geometric Shapes Interactively
,” Proceedings of the Ninth International Conference on Fuzzy Systems (FUZZ-IEEE 2000)
, San Antonio, TX, May 2000.30.
Matsakis
, N. E.
, 1999, “Recognition of Handwritten Mathematical Expressions
,” MS thesis, MIT, Cambridge.31.
Hse
, H.
, and Newton
, A. R.
, 2004, “Sketched Symbol Recognition Using Zernike Moments
,” 17th International Conference on Pattern Recognition
, Cambridge, UK, Vol. 1, pp. 367–370.32.
Kara
, L. B.
, and Stahovich
, T. F.
, 2004, “An Image-Based Trainable Symbol Recognizer for Sketch-Based Interfaces
,” AAAI Fall Symposium Series 2004: Making Pen-Based Interaction Intelligent and Natural
, October 21–24, Washington, D.C.33.
Erman
, L. D.
, Hayes-Roth
, F.
, Lesser
, V. R.
, and Raj Reddy
, D.
, 1980, “The Hearsay-II Speech Understanding System: Integrating Knowldge to Resolve Uncertainty
,” ACM Comput. Surv.
0360-0300, 12
(2
), pp. 213
–253
.34.
Duda
, R. O.
, Hart
, P. E.
, and Stork
, D. G.
, 2001, Pattern Classification
, 2nd ed., Wiley
, New York
.35.
Kara
, L. B.
, 2004, “Automatic Parsing and Recognition of Hand-Drawn Sketches for Pen-Based Computer Interfaces
,” Ph.D. thesis, Carnegie Mellon University, Pittsburgh, PA.36.
Stahovich
, T. F.
, 2004, “Segmentation of Pen Strokes Using Pen Speed
,” AAAI Fall Symposium Series 2004: Making Pen-Based Interaction Intelligent and Natural
, October 21–24, Washington, D.C.37.
Domingos
, P.
, and Pazzani
, M. J.
, 1997, “Beyond Independence: Conditions for the Optimality of the Simple Bayesian Classifier
,” Mach. Learn.
, 29
, pp. 103
–130
. 0885-6125Copyright © 2008
by American Society of Mechanical Engineers
You do not currently have access to this content.
