Innotescus provides multi-stage review processes for each task to allow supervisors to configure workflows that ensure high quality annotations. To view the tutorial video for the configuration of these processes, see the tutorial here.
Enabling a Review Process
To begin setting up a review process, simply click the 'enable review process' button, and the controls to configure the primary review step will be displayed. Review processes can be enabled at any point during a task's existence; if annotations have already been completed, they will still be eligible for review once the review process is created.
Configuring the Primary Review Step
The following options are all available to configure the primary review step. While selecting reviewers and a review ratio are required, the other four options are not. Supervisors can use any combination of the selection parameters to sample annotations for review. Click 'save process' to save the primary review step.
Supervisors must designate reviewers for every step of the review process. These reviewers will be the only project members able to accept or reject annotations at this stage of the review process.
Reviewers must also select a proportion of available annotations to send through the review process. At 100%, all annotations that meet the selected criteria will be sent through the review process, but at 50%, only half will be sent through the review process.
Supervisors have the option to select annotators as a criteria for reviewed work, so only annotations completed by selected annotators will be reviewed. This may be useful, for example, if a new annotator requires closer supervision than a more experienced annotator.
Filtering annotations by date range allows supervisors to select annotations completed during a certain time period for review. This may be useful if annotations completed before a certain date did not have access to a recently added class, and need to be corrected.
Filtering by datasets allows supervisors to designate images belonging to a certain dataset for review.
Filtering by class allows supervisors to sample annotations that contain the selected class(es) for review. This option is useful when the specifications for a certain class were unclear during annotation, and each annotation of that class requires closer supervision.
Adding Review Steps
Once the primary review step has been configured, supervisors can add more steps to the review process. For each step, supervisors must select reviewers and a review ratio; after the primary review step, all steps simply sample the designated proportion of annotations from the previous step.
Editing Review Steps
Once a review step has been created, supervisors can return to it to edit the reviewers and the review ratio, as shown below.
Deleting a Review Process
Click the 'delete review process' to delete an existing review process. This action will delete all pending work associated with any step of the review process, but will not revert any of the annotations that have been changed during the review process.
Disabling a Review Step
Use the action menu for a specific review step to disable it; this will stop any new work from being issued to this step of the review process, and will remove this work item from the annotator interface or 'My Work' page once all in=progress work has been completed.
Supervisors can have a full view of their review process using the analytics available in the review tab. Select a single review step and click on any of the available review analytics tabs to identify bottlenecks or gaps in annotator understanding.
View the total number of accepted and rejected annotations for all users during any given review step. Filter by class to view all users' performance on a single class, or filter by date range to identify trends over a specific time period.
View annotation acceptance and rejection statistics for all classes in the selected review step. Filter by annotator to focus in on a specific annotator's per-class performance, or filter by date range to view performance over a certain time period.
View acceptance and rejection statistics for each image that entered the selected review step. While the other two analytics options show statistics on a per-object basis, this metric is calculated on a per-image basis, and helps supervisors gain a high level view of the number of images that have to be sent back for correction.