Team Members

• Hanchuan Li, Haichen Shen, Shengliang Xu, Congle Zhang {hanchuan,haichen, slxu,clzhang}@cs.washington.edu

Project Name

Visualizing NLP Annotation for Crowd

Development Process

We design, inplenemted and evaluated two visualizationt tool for crowdsorsing NLP annotations.

Visualizing NLP annotation is useful for the collection of training data for the statistical NLP approaches. Existing toolkits either provide limited visual aid, or introduce comprehensive operators to realize sophisticated linguistic rules. Workers must be well trained to use them. Their audience thus can hardly be scaled to large amounts of non-expert crowd-sourced workers. In this paper, we present CrowdAnno, a visualization toolkit to allow crowd-sourced workers to annotate two general categories of NLP problems: clustering and parsing. Workers can finish the tasks with simplified operators in an interactive interface, and fix errors conveniently. User studies show our toolkit is very friendly to NLP non-experts, and allow them to produce high quality labels for several sophisticated problems. We release our source code and toolkit to spur future research.

System Brainstorming

• All group members work on this together.

System Input Output Implementation:

• Major Responsibility: Congle Zhang
• Minor Responsibility: Shengliang Xu, Haichen Shen

System Graphic & Visualization Implementation:

• Major Responsibility: Haichen Shen Shengliang Xu
• Minor Responsibility: Hanchuan Li, Congle Zhang

System Layout Adjustment & User Evaluation Study:

• Major Responsibility: Hanchuan Li
• Minor Responsibility: Congle Zhang, Haichen Shen, Shehgliang Xu.