How To Do One-hot Encoding In Several Columns Of A Pandas Dataframe For Later Use With Scikit-learn

Say I have the following data import pandas as pd data = { 'Reference': [1, 2, 3, 4, 5], 'Brand': ['Volkswagen', 'Volvo', 'Volvo', 'Audi', 'Volkswagen'], 'Town': ['Berl

Solution 1:

Consider the following approach.

Demo:

from sklearn.preprocessingimportLabelBinarizerfrom collections import defaultdict

d = defaultdict(LabelBinarizer)

In [7]: cols2bnrz = ['Brand','Town']

In [8]: df[cols2bnrz].apply(lambda x: d[x.name].fit(x))
Out[8]:
BrandLabelBinarizer(neg_label=0, pos_label=1, spars...
TownLabelBinarizer(neg_label=0, pos_label=1, spars...
dtype: objectIn [10]: new = pd.DataFrame({
    ...:     'Reference': [6, 7],
    ...:     'Brand': ['Volvo', 'Audi'],
    ...:     'Town': ['Stockholm', 'Munich']
    ...: })

In [11]: newOut[11]:
   BrandReferenceTown0Volvo6Stockholm1Audi7MunichIn [12]: pd.DataFrame(d['Brand'].transform(new['Brand']), columns=d['Brand'].classes_)
Out[12]:
   AudiVolkswagenVolvo00011100In [13]: pd.DataFrame(d['Town'].transform(new['Town']), columns=d['Town'].classes_)
Out[13]:
   BerlinMunichStockholm00011010

Solution 2:

You could use the get_dummies function pandas provides and convert the categorical values.

Something like this..

import pandas as pd
data = {
    'Reference': [1, 2, 3, 4, 5],
    'Brand': ['Volkswagen', 'Volvo', 'Volvo', 'Audi', 'Volkswagen'],
    'Town': ['Berlin', 'Berlin', 'Stockholm', 'Munich', 'Berlin'],
    'Mileage': [35000, 45000, 121000, 35000, 181000],
    'Year': [2015, 2014, 2012, 2016, 2013]
 }
df = pd.DataFrame(data)

train = pd.concat([df.get(['Mileage','Reference','Year']),
                           pd.get_dummies(df['Brand'], prefix='Brand'),
                           pd.get_dummies(df['Town'], prefix='Town')],axis=1)

For the test data you can:

new_data = {
    'Reference': [6, 7],
    'Brand': ['Volvo', 'Audi'],
    'Town': ['Stockholm', 'Munich']
}
test = pd.DataFrame(new_data)

test = pd.concat([test.get(['Reference']),
                           pd.get_dummies(test['Brand'], prefix='Brand'),
                           pd.get_dummies(test['Town'], prefix='Town')],axis=1)

# Get missing columns in the training test
missing_cols = set( train.columns ) - set( test.columns )
# Add a missing column in test set with default value equal to 0for c in missing_cols:
    test[c] = 0
# Ensure the order of column in the test set is in the same order than in train settest = test[train.columns]