A Model That Never Predicts Churn Still Scores 73.5% Accuracy: The Class Imbalance Trap

A model (a program that makes predictions from data) that never looks at a single customer's details, and just guesses "will stay" for everyone, scores 73.5% accuracy, meaning 73.5% of its predictions are correct. The task is predicting churn, which means a customer leaving the service, using the public Telco Customer Churn dataset of 7,043 telecom customers. Every score below is measured on a held-out test set: 2,113 customers the models never trained on.

Train an actual model on the same data, so it learns patterns from past examples using facts about each customer (called features), and accuracy climbs to 78.8%. That is 5.3 points for all that work.

Except the trained model catches 41.5% of customers who actually churn, and the guess-the-same-thing baseline catches exactly 0%. Accuracy showed a 5.3-point gap and hid a jump from 0% to 41.5%.

Where the Imbalance Comes From

In the full dataset, 5,174 customers stayed and 1,869 churned, which is 73.5% versus 26.5%. Each customer falls into one of two classes (categories): stayed or churned. Class imbalance means one class is far more common than the other. That split isn't unusual. Fraud, disease screening, equipment failure, customer churn: most of the real-world problems worth building a classifier (a model that sorts things into categories, like "will leave" or "will stay") for have this shape, because the event you actually care about is the rare one.

So why does that make accuracy dangerous? Because a model doesn't need to understand the rare class (the minority class) at all to score well overall. It only needs to be right about the common class (the majority class) most of the time, and the majority class is most of the data by definition.

The Baseline That Never Tries

Here's the laziest possible classifier, used as a baseline (a simple reference model to compare against): always predict the majority class, with no features, no training, and no logic. The code uses pandas, where class_counts holds how many customers fall in each class and y_test holds the true labels of the test customers (1 means churned).

majority_class = class_counts.idxmax()
majority_label = int(majority_class == "Yes")
baseline_preds = pd.Series(majority_label, index=y_test.index)
naive_accuracy = accuracy_score(y_test, baseline_preds)

Isn't a model like this obviously useless? It is, but it still scores 73.5% accuracy on the test set, because 73.5% of those customers really do stay. A model that has learned nothing at all can still look like it's doing most of the job right, purely because of how the classes are split.

What a Real Model Adds — and What It Still Misses

Train a logistic regression (a simple method that learns to turn customer details into a probability of leaving) on tenure (months as a customer), monthly charges, and total charges, and accuracy rises to 78.8%. The data is split first: 70% to train on and a 30% test set held back to check the model on customers it hasn't seen. On paper, an improvement.

model = LogisticRegression()
model.fit(X_train_scaled, y_train)
preds = model.predict(X_test_scaled)

So does a 5.3-point accuracy gain mean the model is 5.3 points better at its actual job? Look at recall, the share of real churners the model correctly flags, instead of accuracy, and the picture changes completely. Of the 561 actual churners in the test set, the trained model caught 233 and missed 328, a recall of 41.5%. The two models look close on accuracy. They are not remotely similar at the one thing a churn model exists to do.

The First Step Is Measuring the Right Thing

Doesn't a more powerful model, or more data, fix this? Possibly, and this post doesn't test it. The model here uses only three numeric columns, so some of the missed churners may come from limited features rather than imbalance alone. What you can do first costs nothing and needs no new data: stop grading with a metric (a number that scores how good a model is) that a model can win by ignoring the class that matters. Changing the metric doesn't catch more churners, but it shows you the problem.

Three measurements say what accuracy cannot, and the test-set numbers show it:

• Recall (of the real churners, how many were caught): 41.5% for the trained model, 0% for the baseline.
• Precision (of the customers flagged as churners, how many really churned): 66.2% for the trained model.
• Confusion matrix (a table counting correct and wrong predictions for each class): 1,433 stayers correctly kept, 119 false alarms, 328 churners missed, 233 churners caught.

Other approaches exist, such as rebalancing the classes in the training data, making mistakes on the rare class count for more while the model learns, or moving the probability cutoff above which the model says "will churn" away from the default 50%. This post does not test them.

Get the Code

The full comparison, with the baseline, the trained logistic regression, the class counts, and the confusion matrix on the Telco Customer Churn dataset. Keep the dataset CSV in the same folder and run it with Python: class_imbalance_demo.py.

Summary

When one class dominates a dataset (73.5% stayed vs. 26.5% churned here), accuracy stops being a trustworthy signal. A model that predicts the majority class every time scores 73.5% on the test set without learning anything.

A real trained model reached 78.8%, only 5.3 points higher, but its recall on actual churners went from 0% to 41.5%, with 328 of 561 churners still missed. Accuracy hides that.

The first step is not a bigger model. It is grading with recall, precision, and a confusion matrix, so the score can't be won by ignoring the class that actually matters.