Skip to content

Reviewing & Correcting Annotations

When reviewing annotations or correcting annotations that have been rejected, a few more tools than normal are accessible from the canvas. Read below to learn more about each one, or watch the tutorial video here.

Reviewing Annotations

To initiate a review session in the canvas, reviewers can click the 'review' button on the 'My Work' page or annotator interface, or if they have supervisor permissions, reviewers can begin a review from the action menu of a review step in the review tab of a task's details page.

Initiating Review

Rejection Tool

Select the rejection tool in the tool palette and click a specific annotation or a location on the image to reject either one. Reviewers can add comments in the right side panel once a rejection is placed, before licking 'add' to save each rejection and continue reviewing.

Rejection Tool

Editing Existing Rejections

To view an existing rejection's information in the right side panel, simply click on it; update the comments and click 'update' to save them. To delete an existing rejection, right click it and select 'delete rejection.'

Editing Annotations

Reviewers have access to the full tool palatte for the annotation task they're reviewing, so do have the option to edit or add new annotations to each piece of work they review. Editing an annotation set during the review process does not prohibit a reviewer from accepting or rejecting the annotation set.

Submit Approval

Reviewers can approve work by clicking the normal green 'submit' button in the lower right hand corner. If any rejections have been made on the annotation set or image, the approve button is disabled. Approved annotation sets either move to the next review step or are marked as complete.

Submit Rejection

To submit a rejected annotation set or image, the reviewer must click the red rejection button in the lower right hand corner. Clicking the rejection button allows the reviewer to add any additional comments - visible to the annotator in the information panel - before clicking 'submit rejection' and sending the annotation set back to the original annotator.

Submitting Rejections

Correcting Rejected Annotations

To begin correcting rejected work, users can click the 'rejections' button on the 'My Work' page or annotator interface.

Reviewing Rejections

Rejected Work Panel

The rejected work panel shows all rejections for the current annotation set to its original annotator, as well as any general comments submitted by the reviewer. Annotators can also toggle the rejection icons on and off to view and edit the annotations more easily.

Rejected Work Panel

Correcting Individual Rejections

To correct and view more information about a single rejection, click the '>' icon next to it in the rejected work panel. The rejected work panel will now show any comments specific to this particular rejection. Once the rejection has been corrected, annotators can click the 'I've corrected this rejection' checkbox and click 'continue' to move to the next rejection, click the '<' icon to return to the original rejected work panel, or if there are no outstanding rejections, submit the annotation set for another round of review.

Correcting Individual Rejections

Resubmitting Work

When resubmitting rejected work they've corrected, annotators have the option to provide general comments, shown alongside the reviewer's general comments in the information panel. This allows annotators and reviewers to carry on an annotation-specific dialog throughout the review process.

Resubmitting Work

Reviewing Resubmissions

Reviewing Resubmissions is very similar to the initial annotation review - reviewers have the option to edit, accept, or reject annotations - but includes a bit more information since it's further along in the review process. To view resubmissions in the canvas, click the 'resubmissions' button on the 'My Work' page or annotator interface.

Viewing Resubmissions

Corrected Work Panel

The corrected work panel shows all the corrected annotations from the previous rejection and correction. Reviewers can click the '>' icon next to each one to see any further details, or view the 'General Comments' section to see the history of comments between the reviewer and annotator.

Corrected Work Panel

Annotation History

In this stage of review, the annotator and reviewer have created a history for each annotation set, which is available in the information panel. Hover over each item in the 'Task History' section to easily display the differences from one point to the next.

Annotation History