Staying on top of changes in real-world input data is a key part of any machine learning project. To help with this, you can customize your super.AI projects by choosing who should label your data and tweaking task design.

1. Head to your super.AI dashboard
2. Open the relevant project
3. Click Task design in the left-hand menu

Task design and ground truth data

Changing your project design can invalidate existing ground truth data. Before you save any changes to your project design, you may be asked to confirm that your ground truth data is still relevant to your project. If your ground truth data is not valid, you will be prompted to create a new project from scratch, i.e., without ground truth data and any previously submitted data points.

What is ground truth data?

Ground truth data is labeled data that super.AI knows to be correct. We use ground truth data to measure and predict performance. If your project’s ground truth data is inaccurate or irrelevant, our measurement and performance metrics will also be inaccurate. We create ground truth data for your project automatically, but you are encouraged to submit as much of your own ground truth data as you have available or can create. You can learn more on our Ground truth page.

Where can I find my ground truth data?

You can view your project’s ground truth dataset by opening your project in the super.AI dashboard and clicking Ground truth data in the sidebar.

How do I know if my ground truth data still matches my project design?

If you have changed anything other than your text instructions or a field description/explanation, your ground truth data no longer matches your project. This means that if you have added a new field, removed a field, changed a field name or object, changed any category choices, a label name, etc., your ground truth data does not match your project.

If you have changed your text instructions or a field description/explanation, there is still a chance your ground truth data doesn’t match your project. Generally, there are three cases where you can change your instructions or a field description/explanation and your ground truth data still matches your project:

• Fixing typos and editing grammar or formatting
• Adding or subtracting descriptive text
• Including a previously unseen edge case

Let’s take a look at some examples.

Example: changing the meaning of your instructions

Original instructions: “Label all cats and dogs in the image”
Updated instructions: “Label all seals and manatees in the image”

Your ground truth data is not valid. Any change in the fundamental meaning of your instructions invalidates your ground truth data.

Example: adding detail to your instructions

Original instructions: “Label all cats and dogs in the image”
Updated instructions: “Label all cats and dogs in the image. A dog is any domesticated carnivore descending from a wolf”

Your ground truth data might not be valid. You have added detail and your outputs will be more nuanced e.g., wolves are no longer labeled as dogs. You need to check if there are wolves labeled as dogs in your ground truth data. If there are, your ground truth data is no longer valid. If there are no wolves labeled as dogs, your ground truth remains valid.

Example: adding a previously unseen edge case

Original instructions: “Label all cats and dogs in the image”
Updated instructions: “Label all cats, dogs, and hamsters in the image”

Your ground truth data is valid, providing there are no images of hamsters in your ground truth.

Example: altering grammar or typos

Original instructions: “Label all cats and dosg in those images”
Updated instructions: “Label all cats and dogs in the image”

Your ground truth data is valid. Your changes are superficial, and only intended to make the text easier to read and understand.

Task design options

The design options available to you will depend on the project type you're using. However, one setting that you’ll find in every project is Text instructions for our human annotators.

Text instructions

Sometimes your data will be processed by humans and other times by AI. When it’s humans doing the processing, we need to tell them exactly what to do. This is where instructions come in. Clear, concise text instructions describing how to effectively execute your task are an essential part of any project.

When setting up your project, we give you some pre-filled instructions that you can tweak as you see fit. It’s common to go through several iterations of instructions before arriving at a set that gets you the desired results. If things don’t go perfectly the first time around, you don’t need to be concerned.

You can find extensive tips and guidance on writing and improving instructions on our How to write instructions page.

📘

Changing your task design affects previously submitted data points

Changes to your task design will affect data points that are in the not queued, queued, or paused data point states at the time of the change, as well as any data points submitted subsequently.