Reviewing & Correcting Annotations
This page covers standard and object reviews, which have different interfaces during the review process. For short tutorial videos on either one, click here.
When reviewing annotations or correcting annotations that have been rejected in a standard review, a few more tools than normal are accessible from the canvas. Read below to learn more about each one, or watch the tutorial video here.
To initiate a review session in the canvas, reviewers can click the 'review' button on the 'My Work' page or annotator interface, or if they have supervisor permissions, reviewers can begin a review from the action menu of a review step in the review tab of a task's details page.
Select the rejection tool in the tool palette and click a specific annotation or a location on the image to reject either one. Reviewers can add comments in the right side panel once a rejection is placed, before clicking 'add' to save each rejection and continue reviewing.
Editing Existing Rejections
To view an existing rejection's information in the right side panel, simply click on it; update the comments and click 'update' to save them. To delete an existing rejection, right click it and select 'delete rejection.'
Reviewers have access to the full tool palatte for the annotation task they're reviewing, so do have the option to edit or add new annotations to each piece of work they review. Editing an annotation set during the review process does not prohibit a reviewer from accepting or rejecting the annotation set.
Reviewers can approve work by clicking the normal green 'submit' button in the lower right hand corner, or by holding down Shift and hitting Enter. If any rejections have been made on the annotation set or image, the approve button is disabled. Approved annotation sets either move to the next review step or are marked as complete.
To submit a rejected annotation set or image, the reviewer must click the red rejection button in the lower right hand corner. Clicking the rejection button allows the reviewer to add any additional comments - visible to the annotator in the information panel - before clicking 'submit rejection' and sending the annotation set back to the original annotator.
Correcting Rejected Annotations
To begin correcting rejected work, users can click the 'rejections' button on the 'My Work' page or annotator interface.
Rejected Work Panel
The rejected work panel shows all rejections for the current annotation set to its original annotator, as well as any general comments submitted by the reviewer. Annotators can also toggle the rejection icons on and off to view and edit the annotations more easily, or select an option to have the canvas center on each rejection when selected.
Correcting Individual Rejections
To correct and view more information about a single rejection, click the '>' icon next to it in the rejected work panel. The rejected work panel will now show any comments specific to this particular rejection. Once the rejection has been corrected, annotators can click the 'I've corrected this rejection' checkbox and click 'continue' to move to the next rejection, click the '<' icon to return to the original rejected work panel, or if there are no outstanding rejections, submit the annotation set for another round of review.
When resubmitting rejected work they've corrected, annotators have the option to provide general comments, shown alongside the reviewer's general comments in the information panel. This allows annotators and reviewers to carry on an annotation-specific dialog throughout the review process.
Reviewing Resubmissions is very similar to the initial annotation review - reviewers have the option to edit, accept, or reject annotations - but includes a bit more information since it's further along in the review process. To view resubmissions in the canvas, click the 'resubmissions' button on the 'My Work' page or annotator interface.
Corrected Work Panel
The corrected work panel shows all the corrected annotations from the previous step. Reviewers can click the '>' icon next to each one to see any further details, or view the 'General Comments' section to see the history of comments between the reviewer and annotator. Use the available options to show and hide 'corrected' icons, and to center the canvas on each correction when selected.
In this stage of review, the annotator and reviewer have created a history for each annotation set, which is available in the information panel. Hover over each item in the 'Task History' section to easily display the differences from one point to the next.
Object review occurs in a different interface from the standard review process. Instead of reviewing or correcting one image at a time, reviewers are served many objects from across the associated task, according to the configured selection criteria. To view the tutorial video for object review, click here.
When objects are selected, their class and any metadata is displayed in the right hand panel. If the selected objects have different classes or metadata values, those fields will be blank, and setting them sets the value for all selected items.
Reviewers can select one or multiple objects to review using the standard selection methods from a file browser. Click a second object while holding Shift to select a range, or click multiple objects while holding Ctrl to select them without selecting an entire range.
Viewing Class Labels
To view or hide class labels, use the associated checkbox at the top of the object review canvas.
If there is annotation metadata associated with the objects being reviewed, a checkbox will be at the top of the object review canvas allowing reviewers to optionally display object metadata in a tooltip on hover. Metadata can also always be viewed in the right hand 'Review Actions' panel for one or more selected objects.
Adjusting Object Size
To make objects larger or smaller, adjust the 'row size' at the top of the object review canvas; this will automatically fill each row with the number of objects selected.
Object class and metadata values can be edited simply by changing the values in the right hand 'review actions' panel. When multiple objects are selected and have different values for their class or a given piece of metadata, those fields will be blank in the 'review actions' panel, and changing them will apply those changes to all selected items.