KDI (NSF, Annotations)

Annotation: Form, Function and Promise
If you look closely at knowledge oriented work, annotation is likely to figure prominently. If at first this does not seem true it is because annotation is so ubiquitous and used for so many functions that we rarely realize just how often we annotate. Annotations can be spoken comments, written text, graphical elements, and even gestures. And they may be applied to representations and sometimes to non-representational activities. Thus, the thing being annotated may be video, or audio tapes, textual documents obviously, pictures, graphics, animations, and certain ongoing activities, such as sports, which may have running commentaries recorded.

To give a flavor of the diversity of functions which annotation can serve we have informally observed written annotations alone being used to:

In studying the natural history of annotation we will pay particular attention to the diversity of forms they take, the variety of functions they serve, the degree to which there are conventions governing their creation, their capacity to be exploited as new meta-data, and their role as key elements in groupware.

Annotations come in many forms, from the most highly structured or conventional (e.g. author’s marks on galley proofs) to the most free and unconstrained (e.g. doodles, images, wise cracks). If the workflow of documents requires signatures, directives, commentaries and the like, then there are often conventions governing their addition. But equally because annotations are a collection of highly general techniques for posting contextualized or situated content, the meaning being contributing to a document or film, etc., may not be evident to everyone. Thus, although annotations may be in a language, such as English, with well understood syntactic and semantic structure, their meaning may be elusive because not everyone knows the context – the annotational frame of reference – from which the annotational contribution was originally made.

In looking back at our earlier work on history–enriched digital objects [19] it is now clear that the encapsulated history of interaction can be viewed as yet another form of annotation. In a digital medium these annotations can be automatically created as by-products of normal interactions. We will design work materials that capture such interaction histories and explore mechanisms to exploit their use. We think they have the potential to fundamentally change the digital workplace.

For instance, by learning more about the form, function, and conventionality of annotations and the role they play in distributed cognition we expect to be able to design digital media which can help solve the problem of situatedness by automatically storing aspects of the annotator’s context. This has always been an elusive problem in settling on the proper meaning of someone else’s annotation or in determining why someone interacted with a particular object (whose history we are tracking). Here theory can nicely guide design. As we push forward the horizon of digital media we expect annotations to explode in profusion and importance. We also expect that annotations will transform the way we search, filter and collaborate using large knowledge repositories. The reason is that annotations serve as a valuable new source of meta-data, and can be used as the basis for more powerful forms of collaborative filtering and of social interaction.

Here is a simple example. One of the digital ways annotations can be used is to comment on the nature of a link between two documents. Currently in the WWW, a hyperlink between two information elements indicates that there is some unspecified relation between them. Each author who specifies a link, however, usually has some implicit notion of the relevance of the link. In the author’s mind another node often has a rhetorical significance: it may be an elaboration, an exegesis, a different viewpoint, a counterargument, etc. But, at present, there is no official mechanism to share that personal notion, since links are untyped. If we had tools that made it easy for authors to make the personal meaning of their links explicit, then browsers and other tools could provide more suitable interactions to users. The easier this job of annotating links becomes –for readers too – the sooner we can begin to leverage the obvious power of typing links. Searching, browsing and filtering will improve, and we will be one step closer to that elusive goal of finding quality on the web.

The groupware aspects of annotation are of equal importance. Members of a group currently annotate paper documents to indicate what they believe is right or wrong about a document, what is important, what can be ignored, or what is relevant to specific colleagues. As more documents and representations are used in digital form we can exploit more aspects of the context of authorship, more of the historical facts of the annotations, to filter, navigate and visualize contents we are interested in. For instance, at present, teachers use annotations to comment on students’ exercises, and students often annotate readings to identify concepts or aspects they find problematic. If students were to have online access to the annotations of others – the frequently asked questions, the teacher’s replies, the replies of students – then they might study in a different manner. They might appreciate that the areas of the course where they are having problems are ones others also find difficult. Moreover, students will be able to help their fellow students directly. These are merely a few of the virtues digital annotations promise to provide. In the next section, we discuss how we propose to study the general topic of annotation, and how our theoretical results may lead to better mechanisms for coordinating cognition.

General Theory of Annotation
In their most common form annotations resemble moves in a conversation. You begin a story … I interrupt with my objection, my request for clarification, my two cents worth of reinforcement etc. Such conversational interactions become annotations when they are stored in a form that others can view. This description is helpful and marks the way for deeper analysis, but true to a first approximation only. We do not normally call the dialogue in books, plays, correspondence, documentaries and narratives to be annotations. Although these works store conversational moves, they are self contained, whereas annotations are always parasitic on pre-existing documents or representations (or in certain cases activities). This means that annotations, at least the textual or graphical kind, are highly situated by their spatial location in a document or representation, and not just by the author who adds them and the conversational context they come from. Nonetheless, the insights of conversational analysis offer us a good starting point for analyzing annotations, particularly as conversational theory identifies several of the key components constituting the speaker-hearer or author-audience context. Without this sort of deep analysis we cannot expect to design new forms of filters, information visualizations, search engines, and browsers that can take advantage of the valuable information stored in annotations. Following conversational analysis then, to understand an annotation we need to know:

These elements of annotational context, redundant at points, distinct at others, remind us of the contextually salient elements of conversations. Indeed a theory of annotation may open new lines of thought for theories of conversation. But annotations also have other aspects that require an even richer theory. First, as mentioned, annotations have an essentially spatial or temporal position in a document or representation. In simple text an annotation always has a specific 2-D position. It can span contiguous regions of lines of words, or even serve to combine disjoint regions. Moreover, as annotations accumulate, particularly annotations on annotations, their position becomes meaningful relative to each other. In the Talmud, for example, pages are structured so that surrounding a central quote from the Mishnah are comments by the most important early rabbis, and in subsequent pages there are comments on these comments. Where a comment is located is an indication of the historical importance of that comment and its subsequent influence on talmudic thought.

Second, annotations are often multimodal. Words may be used to comment on maps, diagrams or pictures; sketches may be used to comment on prose or timelines; voice-overs may be used to add commentary to films, graphics or text; and almost any modality of annotation can be used to help interpret instructions. For instance, to help interpret instructions explaining how to leave voice annotation in MS Word, we might use animations, graphics, textual call-outs, voice-overs, or even videos. In conversations we do not have to worry about the nature of multimodal interaction, since all conversational interactions are linguistic or gestural. Accordingly, a general theory of annotation must include some discussion of the nature of multimodal interactions – how different modalities can constructively or destructively interfere with each other.

Third, the range of documents or representations which annotations may be attached to is far more diverse than the types of real time conversations that exist. Because annotations can be attached to virtually any representation – conversation being just one type, or family of types, of representations – the range of entities they may be attached to is as broad as the space of possible representations. For instance, textual documents constitute a type of representation distinct from conversation and they range over student exercises, academic readings, syllabi, contact sheets, press releases, letters, hardcopies of PowerPoint presentations (called decks by high-level bureaucrats), travel forms, budget reports, and requisition slips to name only a few. In each case, the type of document and the workflow it is part of constrains the range of plausible annotations one might find, and similarly, the range of plausible interpretations a reader might make upon receipt of the annotated document. This takes us well out of the scope of conversational analysis.

Lastly, annotations are often asynchronous with the representations they are about. In a conversation, there is a clear notion of turn taking. You speak then I reply. You send or read me a document, and I comment on it. A commentator may address several people or positions at once, but the presupposition still holds that these comments are to utterances or to text that temporally precede the speaker’s. Yet in widely distributed electronic systems, particularly those based on hypertext, a speaker or author may leave annotations conditionally lurking, so that when a certain comment is made, a particular annotation or reply is automatically triggered.