Inter-Annotator Agreement (IAA)

Ensuring consistency and identifying conflicts between annotators is crucial for the success of any annotation project. UBIAI’s Inter-Annotator Agreement (IAA) feature allows you to evaluate team performance by distributing the same documents to multiple annotators. This process helps to measure annotation consistency and identify areas of disagreement. UBIAI supports IAA for Named Entity Recognition (NER), Relation Extraction, and Document Classification tasks.

Creating an IAA Task

Follow these steps to set up and manage an IAA task:

• Navigate to the IAA Tab: Access the collaboration menu and select the IAA tab.

• Add a New Task: Click on the "+" sign to create a new IAA task.

• Define Task Details: Enter the following:

  • Task name

  • Selected annotators

  • Annotation guidelines

  • Due date for completion

  • Documents to be distributed for annotation

• Finalize and Distribute: Save the IAA task, and documents will be assigned to the chosen annotators for evaluation.

Tracking IAA Progress

You can monitor the progress of your IAA tasks in real-time:

• View Task Details: Click on "Task Details" to generate the IAA report.

• Project Overview: The first section of the report displays overall task progress, including the status of each annotator.

• Annotator Management: Add or remove annotators from an ongoing IAA task by clicking the "Add Member to IAA Task" button.

Measuring Agreement

UBIAI provides multiple metrics to assess agreement and consistency between annotators:

• Cohen’s Kappa Score: This metric calculates the average kappa score to evaluate the degree of reliability between annotators.

• Annotator-Pair Scores: View pairwise agreement scores to measure compatibility between two specific annotators. Higher scores indicate stronger agreement.

• Document-Level Scores: Click on individual IAA scores to see detailed results for each document, offering granular insights into annotator performance.

Reviewing Conflicts

Conflicts in annotations are automatically highlighted, making it easy to resolve discrepancies:

1. Open the Conflict Report: Select a document from the list of annotated files.

2. Compare Annotations: View annotations from both annotators side-by-side.

3. Highlight Discrepancies: Conflicting annotations are marked in red for quick identification.

Union Merge Feature

UBIAI’s Union Merge feature allows you to merge annotations from multiple annotators into the master project. This ensures a unified dataset.

• Initiate Union Merge: On the IAA page, click the Union Merge button located at the top left of the IAA matrix.

• Select Documents: A prompt will appear asking you to choose the documents you wish to merge.

• Resolve Conflicts: After clicking the Merge Documents button, a new window will display any annotation conflicts.

  • Resolve the conflicts directly within the pop-up window.

  • Once resolved, the annotations will be merged seamlessly into the master project.