Rating Tool

How to use the Rating Tool for manual data evaluation and quality assurance.

The Rating Tool is a powerful and flexible node designed for human-in-the-loop tasks within a Manual Queue. It allows you to create custom forms for rating data, classifying items, or performing quality assurance on previous work.

It can be operated in two distinct modes: Standard Mode for initial evaluation and QA Mode for reviewing a previously completed rating.

Standard Mode

In Standard Mode, the tool functions as a form for an initial user to perform an evaluation.

Configuration Fields

  • Task Title: A clear, descriptive title for the task (e.g., "Rate Image Similarity").
  • Task Description: (Optional) A short description providing context for the task.
  • Additional Information: (Optional) Any extra details, guidelines, or context the user might need.
Item to Rate

This section defines the content that will be presented to the user for evaluation.

  • Type:
    • Image: Displays a single image.
    • Image Comparison: Displays two images side-by-side.
    • Table: Displays data in a tabular format.
  • Path to Image in Metadata: The dot-notation path to the image URL within the packet's metadata (e.g., product.imageUrl).
  • Path to Second Image...: (Only for Image Comparison) The path to the second image's URL.
  • Path to Table Data...: (Only for Table) The path to an array of objects in the packet's metadata.
Rating Settings

This section defines how the user will submit their rating.

  • Rating Title: The title for the rating section (e.g., "How similar are these images?").
  • Rating Instructions: Instructions on how to perform the rating.
  • Type:
    • Checkbox: Presents a list of options. Can be configured to allow single or multiple selections.
    • Scale: Presents a numerical scale (e.g., from 1 to 5).
  • Options: (Only for Checkbox) Define the label and value for each checkbox option.
Response Settings

(Optional) Include a free-text field for additional comments.

  • Enable Response Section: If checked, a textarea will be shown to the user.
  • Response Title: The title for the text input field (e.g., "Additional Comments").

QA Mode

In QA Mode, the tool is used to review, and then approve or reject, a rating that was completed in a previous step.

Configuration Fields

  • Enable QA Mode: A toggle to switch the tool into review mode. When enabled, all standard configuration fields are hidden.
  • Path to previous rating data: The dot-notation path to the output of the first Rating Tool node.
    • If the previous node's output is at the root of the packet's input, use a single dot: .

How it Works

When a packet enters a node with the Rating Tool in QA mode, the tool will:

  1. Load the data from the specified qaSourcePath.
  2. Display the original task (instructions, images, etc.) and the first user's rating in a read-only view.
  3. Provide a new text field for the QA reviewer to add their feedback and notes.
  4. Show two buttons: Approve and Reject.

The output of the QA node will be an object containing the full original ratingData and a review object with the final status (approved or rejected) and any notes from the reviewer.