How to Estimate a Machine Learning Project

To meet this challenge you need to partner with an experienced engineering and project management team to define and estimate the process of custom software development with exceptional precision. Here's an evaluation framework used by Netguru.

Enterprise software development projects are very difficult to estimate. Adding a Machine Learning (ML) modules make them much more challenging. That is why you should keep in mind that you are asking engineers to write a program that will generate a program, which will learn to do something useful for your business. It is complicated.

You bring on a ton of unknowns and then you unload them

On the Neturu blog, we wrote advice on how to estimate a project and avoid problems. One of the key elements is “getting rid of the unknowns”. Well, while Machine Learning end products help us reduce the unknown, the process of building them is long and obscure.

ML projects are not only more complex, they are realized with a untested technology. As I noted in "7 Challenges for Machine Learning Projects", the first version of TensorFlow was released in February 2017, while PyTorch, another popular library, came out in October 2017.

That is why when thinking about AI features, you should look for the most experienced software development company. The Netguru Machine Learning team works on a four-phase estimation framework that includes: Discovery, Exploration, Development, and Improvement.

Phase I: Discovery (from one to two weeks)

The goal is to gather requirements and evaluate whether Machine Learning fits your business goals. You need to confront your vision with the engineers who will inform you what problems can be solved with the use of the current state-of-the-art and what metrics can be used to measure it.

First, metrics and business goals are often different. Users can rate movies by giving them from 1 to 10 stars. Let's say an algorithm can be trained to predict these outcomes with 90% accuracy. Sounds great but, from the business perspective, it may be more helpful to know if a viewer is going to watch the whole movie or switch to something else. It does not have to correlate with their star ratings.

Second, the development team needs to recognize what kind of data you have. Are you gathering it correctly? Or maybe the data needs to be fetched from an outside service?

Third, can we supervise the algorithm? Can we give it the correct answer each time it makes a prediction? It's a critical question, since unsupervised algorithms which have to make the decision on their own are much more difficult to train. It's like making the prediction from the previous paragraph without having the feedback from the review site.

Four, during this stage we estimate the Proof of Concept (PoC), i.e. exactly what we want to achieve, something like: "we want the model to predict whether a user will watch the whole movie and have our prediction be accurate 70% of the time".

As you can imagine, depending on the PoC, the project may be trivial for the ML engineers and bring a substantial business value, or the other way around. That is why smart goals are so important.

Let me give you another example. If we build an algorithm for detecting cats in images, we might demand it to reach 99% accuracy. On the other hand, in an experiment published in the journal Annals of Oncology, a Machine Learning program was able to tell the difference between images showing cancerous skin changes and benign ones with 95% accuracy, while dermatologists were only 86.6% accurate - this was the benchmark to beat to bring revolutionary positive change to predictive medicine.