title: How can I combine segmentation and encoding to create a repeatable pipeline using sklearn?
body: <p>I'm trying to build a machine learning model using mixed data types. My data is loaded into a pandas dataframe, and I would like to create a pipeline to encode the data, along the following lines: </p>
  
  <pre><code>import pandas as pd
  import numpy as np
  from sklearn.preprocessing import PowerTransformer, QuantileTransformer, OneHotEncoder
  from sklearn.pipeline import Pipeline
  
  df = pd.DataFrame({'client_id' : range(65),
                     'client_city' : ['Austin']*30+['Dallas']*10+['Tempe']*25, 
                     'client_state' : ['Texas']*40 + ['Arizona']*25,
                     'subscription_type' : ['business']*20 + ['personal']*20 + ['business']*2 + ['personal']*23, 
                     'monthly_fee' : np.random.random(65)*20 + 50})
  
  categorical_features = ['subscription_type']
  categorical_transformer = Pipeline(steps=[
      ('onehot', OneHotEncoder(handle_unknown='ignore'))
  ])
  
  numerical_features = ['monthly_fee']
  numerical_transformer = Pipeline(steps=[
      ('scaler', StandardScaler()) #### Replace this part
  ])
  
  preprocessor = ColumnTransformer(
      transformers=[
          ('cat', categorical_transformer, categorical_features),
          ('num', numerical_transformer, numerical_features)
      ],
      sparse_threshold=0,
      remainder='passthrough'
  )
  </code></pre>
  
  <p>The issue is that in the data, there are regional variations in the <code>monthly_fee</code> by <code>client_city</code> and <code>client_state</code>, so applying <code>StandardScaler</code> on the entire dataset at once would lose significant regional information (e.g., a high monthly payment in Cheap Town, NY would be considered very moderate in New York, New York). At the same time, some cities have sparse data (say &lt; 20 clients), so it makes sense to take averages at the state or national level. </p>
  
  <p>I've been getting around this problem by training encoders on subsets of the data like so:</p>
  
  <pre><code>df.set_index(['client_city','client_state','subscription_type','client_id'], inplace=True)
  df['monthly_fee_scaled'] = np.nan
  grp_by_city = []
  grp_by_state = []
  
  for ii, grp in df.groupby(level=[0,2]):
      N = grp.shape[0] # number of clients in this city
      if N&gt;=20:
          power_tran = PowerTransformer()
          grptmp = grp[['monthly_fee',]].copy()
          grptmp['monthly_fee_scaled'] = power_tran.fit_transform(grp[['monthly_fee',]])
          grp_by_city.append(grptmp)
      else:
          pass
  
  tmp = pd.concat(grp_by_city)
  df.update(tmp)
  
  for ii, grp in df.groupby(level=[1,2]):
      N = grp.shape[0] # number of clients in this state
      if N&gt;=20:
          power_tran = PowerTransformer()
          grptmp = grp[['monthly_fee',]].copy()
          grptmp['monthly_fee_scaled'] = power_tran.fit_transform(grp[['monthly_fee',]])
          grp_by_state.append(grptmp)
      else:
          pass
  
  tmp = pd.concat(grp_by_state)
  df.update(tmp, overwrite=False)
  </code></pre>
  
  <p>This is hacky, but it worked well for an initial proof-of-concept. But now I want to do it right and persist the encoding/model so I can process new data. Ideally, there would be a way to do segmentation+encoding using sklearn pipelines, but I have not been able to google up or figure out a solution. Is there a nice way to handle this?</p>