In our previous post (Navigating Custom AI Development – Part 1 :: Setting the right metrics), we explored the critical decisions to be made before embarking on custom AI development.
Now, let’s dive into the essential steps of training your AI model once you’ve committed to this path.
AI Model Training: Navigating the 4 Key Phases
- Model Selection: The Foundation of Success
- Training Set: Fueling Your AI Engine
- The Model Selection, Fitting, and Validation Cycle
- Testing: The Moment of Truth
AI Model Training: Navigating the 4 Key Phases
1.Model Selection: The Foundation of Success
When opting for an Artificial Neural Network (ANN) approach—the gold standard for modern AI problems—selecting the right model is crucial. For this we need to do extensive research and gather information from literature about suitable models.
While a routine for seasoned AI professionals, this step demands a wealth of experience and in-depth understanding to pinpoint the most suitable models for your specific use case.
2.Training Set: Fueling Your AI Engine
The lifeblood of any AI model is its training data. Here are some cornerstones:
Quantity: Typically we need 10-100-1000 times the size of your test data set (or more)
Potential sources: On-site data, public repositories, or artificially generated datasets
Strategy: Develop a robust plan for data acquisition and preparation
3.The Model Selection, Fitting, and Validation Cycle
Once you have created the training set, you have to follow this iterative process that forms the core of AI model development:
Select a model and its hyperparameters.
Fit the model to the training set, i.e., train its parameters.
Evaluate the completed model on the validation set, then restart the cycle to improve the model
4.Testing: The Moment of Truth
Once you’ve honed your model through multiple iterations, it’s time for the final evaluation:
Use your separate test set to assess performance
Measure how well you’ve achieved your project goals
Real-World Application: The Apple Sorting Project
Let’s see how these phases translate to our sample scenario:
We select a pre-trained model available online as our initial model. This will likely be a CNN-type ResNet network, as these perform well on similar problems.
We create a training set of 20,000 elements using the method described for the test set.
We measure the downloaded network on the validation set, considering this result as the baseline to improve upon. We conduct two experiments:
one fine-tuning the downloaded network’s parameters on our training set, and
another reinitializing the weights and retraining the entire network.
Finally, we measure the best-performing network on the validation set using the test set and document this as the project result.
Timing and Implementation
Basic project cycle: 1-2 weeks
Complex projects: 2-6 months
Ongoing optimization: 2-4 weeks annually to explore new models
Remember, this overview simplifies the AI development process. Additional considerations include:
Integration with existing systems
Potential hardware solutions
Scalability based on project requirements and results
The key takeaway? While setting up your measurement and learning environment is a necessary first step, the true scalability lies in the number and depth of models you explore.
As you embark on your AI development journey, keep these phases in mind to navigate the complexities and unlock the full potential of your custom AI solution.