Skip to content

Reviewing & Correcting Annotations

When reviewing annotations or correcting annotations that have been rejected, a few more tools than normal are accessible from the canvas. Read below to learn more about each one, or watch the tutorial video here.

Reviewing Annotations

To begin reviewing work, reviewers can click the corresponding 'review' button on the 'My Work' page or annotator interface, or if they have supervisor permissions, reviewers can begin a review from the action menu of a review step within the task's review tab.

Initiating Review

Rejection Tool

To place rejections on specific frames or annotations, use the rejection tool in the tool palette and click a specific annotation or a location in the video to reject either one. Reviewers can add comments in the right side panel once a rejection is placed, before clicking 'add' to save each rejection and continue reviewing. Once placed, rejections will be shown on the annotation timeline.

Rejection Tool

Editing Existing Rejections

To view an existing rejection's information in the right side panel, simply click on it within the video; update the comments and click 'update' to save them. To delete an existing rejection, right click it and select 'delete rejection.'

Editing Annotations

Reviewers have access to the full tool palatte for the annotation task they're reviewing, so do have the option to edit or add new annotations to each piece of work they review. Editing an annotation set during the review process does not prohibit a reviewer from accepting or rejecting the annotation set.

Submit Approval

Reviewers can approve work by clicking the normal green 'submit' button in the lower right hand corner, or by using the hotkey combination Shift + Enter. If there are any rejections, the approve button is disabled. Approved annotation sets move to the next review step if one exists or are marked as complete.

Submit Rejection

To submit a rejected annotation set or video, the reviewer must click the red rejection button in the lower right hand corner. Clicking the rejection button allows the reviewer to add any additional comments - visible to the annotator in the rejected work panel - before clicking 'submit rejection' and sending the annotation set back to the original annotator.

Submitting Rejections

Correcting Rejected Annotations

To begin correcting rejected work, users can click the 'rejections' button on the 'My Work' page or annotator interface.

Reviewing Rejections

Rejected Work Panel

The rejected work panel shows all rejections for the current video to its original annotator, as well as any general comments submitted by the reviewer. Annotators can also toggle the rejection icons on and off to view and edit the annotations more easily.

Rejected Work Panel

Correcting Individual Rejections

To correct and view more information about a single rejection, click the '>' icon next to it in the rejected work panel. The rejected work panel will now show any comments specific to this particular rejection. Once the rejection has been corrected, annotators can click the 'I've corrected this rejection' checkbox and click 'continue' to move to the next rejection, click the '<' icon to return to the original rejected work panel, or if there are no outstanding rejections, submit the annotation set for another round of review.

Correcting Individual Rejections

Resubmitting Work

When resubmitting corrected work, annotators have the option to provide general comments, shown alongside the reviewer's general comments in the corrected work panel. This allows annotators and reviewers to carry on an annotation-specific dialog throughout the review process.

Resubmitting Work

Reviewing Resubmissions

Reviewing Resubmissions is very similar to the initial annotation review - reviewers have the option to edit, accept, or reject annotations - but includes a bit more information since it's further along in the review process. To view resubmissions in the canvas, click the 'resubmissions' button on the 'My Work' page or annotator interface.

Viewing Resubmissions

Corrected Work Panel

The corrected work panel shows all the corrected annotations from the previous rejection and correction. Reviewers can click the '>' icon next to each one to see any further details, or view the 'General Comments' section to see the history of comments between the reviewer and annotator.

Corrected Work Panel

Annotation History

Any annotation and review history is available in the information panel. Hover over each item in the 'Task History' section to see high level information about each historical entry.