One-Hot Encoding: Why It Breaks Between Train and Test

Your model works perfectly in testing. You deploy it. It crashes on the first real prediction.

ValueError: The feature names should match those that were passed during fit.
Feature names seen at fit time, yet now missing:
- Delhi

If you've seen an error like this — a feature that's missing, or a mismatch in feature names — it almost always traces back to one decision: how a categorical column got converted into numbers.

What Actually Happened

Somewhere in the pipeline, a column like city — Bangalore, Mumbai, Delhi — got converted into numbers. That conversion is called one-hot encoding. Instead of one column holding text labels, you get one new column per category, each holding a 1 or a 0.

Bangalore becomes city_Bangalore = 1, everything else 0. Mumbai becomes city_Mumbai = 1, everything else 0.

Why Not Just Use 1, 2, 3?

Models only understand numbers. They can't do math on the word "Bangalore." But if you just labeled cities 1, 2, 3, the model would assume Delhi is "greater than" Bangalore — a ranking that doesn't exist. One-hot encoding is how you hand a model categorical data without inventing a fake order.

The One Line That Does It

from sklearn.preprocessing import OneHotEncoder

encoder = OneHotEncoder()
X_train_encoded = encoder.fit_transform(X_train[['city']])

fit_transform does two things at once: it learns every category it sees in the training data, and converts them right there. That's the whole operation.

Where It Actually Breaks

A very common shortcut is pandas' get_dummies instead of scikit-learn's encoder:

pd.get_dummies(train_df['city'])   # → 3 columns
pd.get_dummies(test_df['city'])    # → 2 columns

Call it separately on training and test data, and you can get a different number of columns on each side. Training data had Bangalore, Mumbai, and Delhi — three new columns. Test data only had Bangalore and Mumbai that day — two columns. The model expects three inputs and gets two. Crash. That's the exact error from the start of this post — the real one, not a mockup.

The fix: fit the encoder once, on training data only, and reuse that same encoder — never regenerate columns independently on new data.

And to be precise about what's actually broken here: it's not that pandas is bad and scikit-learn is good. Call fit_transform() a second time on the test set instead of transform(), and scikit-learn breaks the exact same way — fit_transform forgets what it learned and relearns from whatever it's given. The rule isn't "use this library." It's "fit once, transform everywhere, never fit twice."

One More Thing to Watch For

This only works cleanly when there aren't too many categories. A city column with 5 values is fine. A user_id column with 50,000 unique values turns into 50,000 new columns — a dataset that's enormous and mostly zeros. That's usually a sign a different technique is needed entirely.

Get the Code

The full breakdown, including the production crash reproduction: OneHotEncoding.ipynb.

What's Next

One-hot encoding solves the fake-ranking problem, but it introduces its own scaling problem the moment cardinality grows. The next post picks up exactly there — what to do once one column would explode into thousands.

This post is the companion to Video 1 in the preprocessing series: One-Hot Encoding.

Comments

Post a Comment

Popular posts from this blog

READ vs SQL SELECT, A Quick Performance Test

Image
READ vs SQL SELECT With no doubt, SQL is the advanced and efficient way for operations (Read, Write, Update or Delete) on Tables (or Files) on IBM i.  With the use of Embedded SQL programming (SQLRPGLE), it is efficient to use SELECT to read the data from a table compared to using READ.  Here is the result from a quick performance test between READ and SELECT to read One million records from a table. Below are few points to consider about the test before we see the results.  Created Physical File (using DDS) for the use in RPGLE with READ Operation.  Created Table from SQL Session for the use in SQLRPGLE with SQL SELECT Operation. Both the files has same number of fields, length and same number of records. Both Programs are called from a new Interactive Session immediately after login. Let's see the code used to create File/Table and Program.  Physical File (PPTESTFILE): Table (PTESTTABL): Inserted one million records (same data) to both the files. And, Table is...
Read more

All about READ in RPGLE & Why we use it with SETLL/SETGT?

READ READ is one of the most used Opcodes in RPGLE. As the name suggests main purpose of this Opcode is to read a record from Database file. What are the different READ Opcodes? To list, Below are the five Opcodes.  READ - Read a Record READC - Read Next Changed Record READE - Read Equal Key Record READP - Read Prior Record READPE - Read Prior Equal Record We will see more about each of these later in this article. Before that, We will see a bit about SETLL/SETGT .  SETLL (Set Lower Limit) SETLL accepts Key Fields or Relative Record Number (RRN) as Search Arguments and positions the file at the Corresponding Record (or Next Record if exact match isn't found).  SETGT (Set Greater Than) SETGT accepts Key Fields or Relative Record Number (RRN) as Search Arguments and positions the file at the Next Record (Greater Than the Key value). Syntax: SETLL SEARCH-ARGUMENTS/KEYFIELDS FILENAME SETGT  SEARCH-ARGUMENTS/KEYFIELDS FILENAME One of the below can be passed as Search...
Read more

Retrieve list of Spooled files on System from SQL - IBM i

Image
Spooled Files Spooled files are generated when when system writes (prints) the data to an Output Queue. These could be Reports generated by the programs using Printer File (PRTF), System dumps or Job logs.  Traditional way of getting the list of spooled files is by using WRKSPLF command, which by default displays the list of spooled files for the current user.  Same can be retrieved from SQL with the use of table function SPOOLED_FILE_INFO. This is available in QSYS2.  SELECT * FROM TABLE(QSYS2.SPOOLED_FILE_INFO()); Above query is simple and straight forward which returns the list of spooled files for the current user. Information returned is same as WRKSPLF (Work with Spooled files) and API (QGYOLSPL).  Let's have a look at how output of the query looks like.  This query returns a full list of columns (we will see all the columns towards the end) for the current user who is running the query.  We can make use of the parameters of the table function to retr...
Read more