Skip to content

Reviewing & Correcting Annotations

This page covers standard and object reviews, which have different interfaces during the review process. For short tutorial videos on either one, click here.

Standard Review

When reviewing annotations or correcting annotations that have been rejected in a standard review, a few more tools than normal are accessible from the canvas. Read below to learn more about each one, or watch the tutorial video here.

Reviewing Annotations

To initiate a review session in the canvas, reviewers can click the 'review' button on the 'My Work' page or annotator interface, or if they have supervisor permissions, reviewers can begin a review from the action menu of a review step in the review tab of a task's details page.

Initiating Review

Rejection Tool

Select the rejection tool in the tool palette and click a specific annotation or a location on the image to reject either one. Reviewers can add comments in the right side panel once a rejection is placed, before clicking 'add' to save each rejection and continue reviewing.

Rejection Tool

Editing Existing Rejections

To view an existing rejection's information in the right side panel, simply click on it; update the comments and click 'update' to save them. To delete an existing rejection, right click it and select 'delete rejection.'

Editing Annotations

Reviewers have access to the full tool palatte for the annotation task they're reviewing, so do have the option to edit or add new annotations to each piece of work they review. Editing an annotation set during the review process does not prohibit a reviewer from accepting or rejecting the annotation set.

Submit Approval

Reviewers can approve work by clicking the normal green 'submit' button in the lower right hand corner, or by holding down Shift and hitting Enter. If any rejections have been made on the annotation set or image, the approve button is disabled. Approved annotation sets either move to the next review step or are marked as complete.

Submit Rejection

To submit a rejected annotation set or image, the reviewer must click the red rejection button in the lower right hand corner. Clicking the rejection button allows the reviewer to add any additional comments - visible to the annotator in the information panel - before clicking 'submit rejection' and sending the annotation set back to the original annotator.

Submitting Rejections

Correcting Rejected Annotations

To begin correcting rejected work, users can click the 'rejections' button on the 'My Work' page or annotator interface.

Reviewing Rejections

Rejected Work Panel

The rejected work panel shows all rejections for the current annotation set to its original annotator, as well as any general comments submitted by the reviewer. Annotators can also toggle the rejection icons on and off to view and edit the annotations more easily, or select an option to have the canvas center on each rejection when selected.

Rejected Work Panel

Correcting Individual Rejections

To correct and view more information about a single rejection, click the '>' icon next to it in the rejected work panel. The rejected work panel will now show any comments specific to this particular rejection. Once the rejection has been corrected, annotators can click the 'I've corrected this rejection' checkbox and click 'continue' to move to the next rejection, click the '<' icon to return to the original rejected work panel, or if there are no outstanding rejections, submit the annotation set for another round of review.

Correcting Individual Rejections

Resubmitting Work

When resubmitting rejected work they've corrected, annotators have the option to provide general comments, shown alongside the reviewer's general comments in the information panel. This allows annotators and reviewers to carry on an annotation-specific dialog throughout the review process.

Resubmitting Work

Reviewing Resubmissions

Reviewing Resubmissions is very similar to the initial annotation review - reviewers have the option to edit, accept, or reject annotations - but includes a bit more information since it's further along in the review process. To view resubmissions in the canvas, click the 'resubmissions' button on the 'My Work' page or annotator interface.

Viewing Resubmissions

Corrected Work Panel

The corrected work panel shows all the corrected annotations from the previous step. Reviewers can click the '>' icon next to each one to see any further details, or view the 'General Comments' section to see the history of comments between the reviewer and annotator. Use the available options to show and hide 'corrected' icons, and to center the canvas on each correction when selected.

Corrected Work Panel

Annotation History

In this stage of review, the annotator and reviewer have created a history for each annotation set, which is available in the information panel. Hover over each item in the 'Task History' section to easily display the differences from one point to the next.

Annotation History

Object Review

Object review occurs in a different interface from the standard review process. Instead of reviewing or correcting one image at a time, reviewers are served many objects from across the associated task, according to the configured selection criteria. To view the tutorial video for object review, click here.

Object Selection

When objects are selected, their class and any metadata is displayed in the right hand panel. If the selected objects have different classes or metadata values, those fields will be blank, and setting them sets the value for all selected items.

Reviewers can select one or multiple objects to review using the standard selection methods from a file browser. Click a second object while holding Shift to select a range, or click multiple objects while holding Ctrl to select them without selecting an entire range.

Selecting Objects

Viewing Objects

Viewing Class Labels

To view or hide class labels, use the associated checkbox at the top of the object review canvas.

Class Toggle

Viewing Metadata

If there is annotation metadata associated with the objects being reviewed, a checkbox will be at the top of the object review canvas allowing reviewers to optionally display object metadata in a tooltip on hover. Metadata can also always be viewed in the right hand 'Review Actions' panel for one or more selected objects.

Adjusting Object Size

To make objects larger or smaller, adjust the 'row size' at the top of the object review canvas; this will automatically fill each row with the number of objects selected.

Editing Objects

Object class and metadata values can be edited simply by changing the values in the right hand 'review actions' panel. When multiple objects are selected and have different values for their class or a given piece of metadata, those fields will be blank in the 'review actions' panel, and changing them will apply those changes to all selected items.

Editing Objects