Navigating Custom AI Development – Part 1 :: Setting the right metrics

When you’ve determined that custom AI development is necessary (see: The Untold Advantages of Custom AI development), it’s crucial to approach it as a project with well-defined objectives. Let us suppose that we have that as a baseline and let’s now focus on the aspects that make custom AI development projects unique. Setting AI project metrics is due earlier than you would think.

To illustrate each step, we’ll use a simplified, hypothetical AI development project aimed at distinguishing between apples and pears in images. This could serve as the “brain” of an apple and pear sorting machine. While simplified, this example clearly demonstrates the key aspects of an AI development project.

Measuring the objectives

Measurable objectives are generally useful for any project, but for AI projects, they’re indispensable. Take these steps to get well defined targets:

  • Define Development Goals
  • Formalize Objectives
  • Determine Measurement Methodology
  • Create Test and Validation Sets
  1. Define Development Goals: First of all, articulate the project goals in a way that all stakeholders agree upon.
  2. Formalize Objectives: This primarily involves developing Key Performance Indicators (KPIs) that mathematically define how we’ll measure project success. It’s beneficial to have realistic target values for these KPIs, primarily for business purposes.

  3. Determine Measurement Methodology: Establish how the KPIs will be measured, as definition doesn’t always equate to easy measurement.

  4. Create Test and Validation Sets: The test set is used to measure project success at the end, while the validation set helps estimate progress towards project goals during development.

Let’s apply these steps to our apple sorting project:

  1. Project Goal: To accurately sort apples and pears.
  2. KPI: Error rate – the percentage of apples misclassified as pears and vice versa.

  3. Measurement Method: Create test and validation sets of images captured directly by the sorting machine to replicate real-world conditions.

  4. Test and Validation Sets: Run 1000 apples and 1000 pears through the sorting machine, capturing images. Randomly allocate 500 apple images and 500 pear images to each of the test and validation sets.

In this simplified case, defining objectives through creating measurement corpora might take 1-2 weeks. For more complex problems, this process could extend to several months.

Note that this simplified example omits non-functional requirements such as decision-making speed, memory usage, image resolution constraints, tolerance for blurred images, etc.

In Part 2 we will cover the practical steps of model training.

Want to know more? Let us talk!