Rating Tool
How to use the Rating Tool for manual data evaluation and quality assurance.
The Rating Tool is a powerful and flexible node designed for human-in-the-loop tasks within a Manual Queue. It allows you to create custom forms for rating data, classifying items, or performing quality assurance on previous work.
It can be operated in two distinct modes: Standard Mode for initial evaluation and QA Mode for reviewing a previously completed rating.
Standard Mode
In Standard Mode, the tool functions as a form for an initial user to perform an evaluation.
Configuration Fields
- Task Title: A clear, descriptive title for the task (e.g., "Rate Image Similarity").
- Task Description: (Optional) A short description providing context for the task.
- Additional Information: (Optional) Any extra details, guidelines, or context the user might need.
Item to Rate
This section defines the content that will be presented to the user for evaluation.
- Type:
Image
: Displays a single image.Image Comparison
: Displays two images side-by-side.Table
: Displays data in a tabular format.
- Path to Image in Metadata: The dot-notation path to the image URL within the packet's metadata (e.g.,
product.imageUrl
). - Path to Second Image...: (Only for
Image Comparison
) The path to the second image's URL. - Path to Table Data...: (Only for
Table
) The path to an array of objects in the packet's metadata.
Rating Settings
This section defines how the user will submit their rating.
- Rating Title: The title for the rating section (e.g., "How similar are these images?").
- Rating Instructions: Instructions on how to perform the rating.
- Type:
Checkbox
: Presents a list of options. Can be configured to allow single or multiple selections.Scale
: Presents a numerical scale (e.g., from 1 to 5).
- Options: (Only for
Checkbox
) Define thelabel
andvalue
for each checkbox option.
Response Settings
(Optional) Include a free-text field for additional comments.
- Enable Response Section: If checked, a textarea will be shown to the user.
- Response Title: The title for the text input field (e.g., "Additional Comments").
QA Mode
In QA Mode, the tool is used to review, and then approve or reject, a rating that was completed in a previous step.
Configuration Fields
- Enable QA Mode: A toggle to switch the tool into review mode. When enabled, all standard configuration fields are hidden.
- Path to previous rating data: The dot-notation path to the output of the first Rating Tool node.
- If the previous node's output is at the root of the packet's
input
, use a single dot:.
- If the previous node's output is at the root of the packet's
How it Works
When a packet enters a node with the Rating Tool in QA mode, the tool will:
- Load the data from the specified
qaSourcePath
. - Display the original task (instructions, images, etc.) and the first user's rating in a read-only view.
- Provide a new text field for the QA reviewer to add their feedback and notes.
- Show two buttons: Approve and Reject.
The output of the QA node will be an object containing the full original ratingData
and a review
object with the final status
(approved
or rejected
) and any notes
from the reviewer.