Innotescus provides multi-stage standard review processes as well as single-step object review processes to allow supervisors to configure workflows that ensure high quality annotations. To view the tutorial videos for configuring and executing these processes, click here.
Adding a New Review Process
To begin setting up a review process, click 'new review' on the review tab of a task. Supervisors can create as many review processes for a single task as necessary.
Configuring the Primary Review Step
The following options are available when setting up a new review process. While selecting the review type, reviewer, and review ratio or batch size (depending on the review type) are required, the other four selection criteria are optional. The optional selection criteria simply allow supervisors to be more targeted in their QA processes.
Users must decide whether to create a standard review or an object review. All options are the same for both standard and object review except for the review ratio (only for standard review processes) and batch size (only for object reviews).
In a standard review, users review one image or video at a time, and can edit, accept, and reject annotations. Rejecting an image or video sends it back to the previous step, while accepting allows it to move to the next step. Standard reviews have a strict locking policy, so once an image or video goes under review, it is 'locked' until it is accepted, rejected, or unassigned.
To see what the standard review process looks like from the canvas, click here.
In an object review, users review multiple objects from different images at the same time; reviewers can edit object classes and metadata, but not boundaries. Instead of working in the image canvas, reviewers are presented with a grid containing a batch of objects that meet the selection criteria. Because object reviews pull from many images, there is a much less 'strict' locking policy.
To see what the object review process looks like from the canvas, click here.
Supervisors must designate reviewers for every step of the review process. These reviewers will be the only project members able to accept or reject annotations at this step of the review process.
Review Ratio (Standard Review Only)
Reviewers must also select a proportion of annotations to send through the review process. At 100%, all annotations that meet the selected criteria will be sent through the review process, but at 50%, only half will be sent through the review process. This ratio applies to all current and future annotations in this task.
Batch Size (Object Review Only)
The batch size determines how many objects can be shown in the grid to the reviewer at once. The batch size can be changed at any time in the object review's life.
Supervisors have the option to select annotators as a criteria for reviewed work, so only annotations completed by selected annotators will be reviewed. This may be useful, for example, if a new annotator requires closer supervision than a more experienced annotator.
Filtering annotations by date range allows supervisors to select annotations completed during a certain time period for review. This may be useful if annotations completed before a certain date did not have access to a recently added class, and need to be corrected.
Filtering by datasets allows supervisors to designate images belonging to a certain dataset for review.
Filtering by class allows supervisors to sample annotations that contain the selected class(es) for review. This option is useful when the specifications for a certain class were unclear during annotation, and each annotation of that class requires closer supervision.
Managing Review Processes
Deleting a Review Process
Use the action menu (⋮) at the process level to delete an existing review process. This action will delete all pending work associated with any step of the review process, but will not revert any of the annotations that have been changed during the review process.
Disabling a Review Process
Use the action menu (⋮) to disable a review process; this will remove any access to the review from the annotator interface and 'My Work' pages, and will disable the review selection on the process action menu as well.
Managing Steps in a Standard Review Process
Only a standard review can have multiple steps. Read below to see how to manage them!
Adding Review Steps
Supervisors can add more steps to a review process by clicking 'add step' within an expanded process. For each new step, supervisors must select reviewers and a review ratio; each step reviews the output of the previous step, except for step 1, which samples from the original annotation task.
Editing Review Steps
Once a review step has been created, supervisors can return to it to edit the reviewers and the review ratio, as shown below. Be sure to save any changes with the 'save' button in the lower left.
Deleting Review Steps
Review steps can be deleted by clicking the action menu next to a review step and selecting 'delete.' Only the final step of a review process can be deleted.
Viewing Review Step Progress
The icons and text next to each review step show its current status. For each step, supervisors can see the amount of work available to review, the number of outstanding rejections, and the number of resubmissions ready for review.
Standard Review Analytics
Supervisors can use the analytics available in the review tab to understand any standard review step at a higher level. Select a single review step and click on any of the available review analytics tabs to identify bottlenecks or gaps in annotator understanding.
View the total number of accepted and rejected annotations for all users during a given review step. Filter by class to view all users' performance on a single class, or filter by date range to identify trends over a specific time period.
View annotation acceptance and rejection statistics for all classes in the selected review step. Filter by annotator to focus in on a specific annotator's per-class performance, or filter by date range to view performance over a certain time period.
View acceptance and rejection statistics for each image that entered the selected review step. While the other two analytics options show statistics on a per-object basis, this metric is calculated on a per-image basis, and helps supervisors gain a high level view of the number of images that have to be sent back for correction.