One of the most important things in machine learning is evaluating how well your model is doing.
Two important metrics for this are accuracy and precision.
Sometimes, a machine learning model might have high accuracy but low precision.
This can be misleading to the machine learning engineer, as a high accuracy score might make you think the model works well when it may need improvement.
High accuracy and low precision mean that the classification algorithm is making a lot of correct predictions. Still, it’s missing predictions on a key group of values, meaning you have a high rate of false positives.
This article will explain what high accuracy and low precision mean and why you shouldn’t outright trust it.
We’ll also walk you through how to improve your machine-learning model process, where you’re not just blindly trusting metrics.
Get ready to take your machine-learning skills to the next level!
What is the Difference Between Accuracy And Precision?
Before jumping right into what high accuracy and low precision mean together, we need to understand what each means individually.
Accuracy is all about how many times the model gets it right.
It’s like a math test in school – if you get 80% of the answers right, then your accuracy is 80%.
Simple, right?
Precision is a little bit different.
Precision is about being specific and getting the right answer for the right thing.
For example, if you’re again taking a math test and get all the geometric answers correct but miss some from another category, you had high precision with geometry but low on that other category.
So in machine learning, precision is slightly different but still uses that same idea.
While precision in the real world can relate to anything, precision within machine learning focuses on one thing and one thing only, false positives.
A high precision rate means you have few to no false positives, but a low precision means you have many false positives.
In short, accuracy is about getting the right answer, and precision is about getting the right answer for the right thing.
What Does High Accuracy Low Precision Mean During Machine Learning?
Simply, High accuracy and low precision mean that the classification algorithm is making a lot of correct predictions. Still, it’s missing predictions on a key group of values, meaning you have a high rate of false positives.
For a dataset to have high accuracy and low precision, it usually means there’s a low amount of true positives within the dataset.
Since your algorithm can predict everything accurately but has a high false positive rate, you need low amounts of true positives relative to the total rows of data to make this happen.
This means you’ll see this type of thing in datasets with a low “hit” rate, like medical diagnosis, error detection, candidate hiring, etc.
Before you freak out about high accuracy and low precision, we’ll review type I and type II errors in the next section.
Understanding Type I and Type II Error
Remember, In machine learning, our model will make some mistakes; it’s part of the game.
However, all errors made aren’t isn’t equal.
There are two different types of mistakes that models can make: Type I and Type II errors.
A Type I error is a false-positive that we’ve been talking about.
It’s when the model says something is wrong, but it’s not. For example, if a fire alarm goes off without fire, that’s a Type I Error.
Type II Error is a false-negative, where we’ve mislabeled an occurring positive event.
For example, if a fire alarm doesn’t go off when there is a fire, that’s a Type II Error.
A well-known secret around the machine learning industry is that these “Errors” are rarely equal.
While it is a bit complicated not to think of errors as equal, it’s easiest to understand with an example.
When Google hires engineers, they spend a ton of money ensuring they don’t hire the wrong candidate. Since a bad hire can corrupt a whole department, they’d rather decline candidates that are on the line (probably good enough) to ensure they don’t accidentally hire any wrong candidates.
In this scenario, google will have many Type II errors (false negatives from declining good candidates), but they’ve already accepted this.
What they’re trying to optimize is having very few Type I errors (hiring the wrong engineer).
With a deep understanding of their business problem, they can utilize errors in a way that allows them to reach their desired outcome.
You need to understand your business problem in the same type of way.
In some scenarios, when you classify, it makes much more sense to optimize on fewer false positives than it does to increase things like accuracy.
How Do We Decide Which Modeling Metric We Want To Improve?
As discussed in the section above, the first step to understanding your modeling metrics is fully understanding your business problem.
Does your business problem call for you to be right about everything, right about a specific subset of information, or not to be wrong about some particular instance of your dataset?
For example, high accuracy is generally useless when building a medical diagnosis model.
This is because the occurrence of an event is so rare; even if you’re machine learning model predicts “No” on everything, it would still be right in the 90%+ percentile! (This is a common interview question in data science interviews, BTW).
Instead, focusing on something like recall, which is the split of samples we could identify in the positive class correctly, would be a much more impactful calculation.
Anyone who tells you there’s a one-fit-all modeling metric is lying, and things like F1 score, precision, recall, and accuracy all have their place when building out machine learning models.
Should I Always Fix Unbalanced Datasets
When calculating these advanced modeling metrics, a tip is to note the split in your dataset.
If you have way more positive events than negative events (1 vs. 0), this is known for leading to some misleading metrics.
Like everything else in this article, there is no strict “rule” regarding unbalanced datasets.
Personally, when it comes to any modeling question, I always leave it up to cross-validation.
Balance out your dataset, test it with cross-validation, and see if that beats not balancing it.
Other Articles In Our Accuracy Series:
Accuracy is used EVERYWHERE, which is fine because we wrote these articles below to help you understand it
- Can Machine Learning Models Give An Accuracy Of 100
- Machine Learning: High Training Accuracy And Low Test Accuracy
- What Is a Good Accuracy Score In Machine Learning?
- How Can Data Science Improve The Accuracy Of A Simulation?
- Data Science Accuracy vs. Precision
- Machine Learning Validation Accuracy
