Feature Engineering

Feature engineering transforms raw data into inputs that expose the signal, often mattering more than the choice of model.

Shaping the inputs

Feature engineering is the craft of turning raw data into representations a model can use well. It includes cleaning, transforming, combining, and encoding variables. On tabular problems, good features frequently improve results more than switching algorithms, because a model can only learn from what the features expose.

Common transformations

• Scaling and normalization so features share a comparable range.
• Encoding categoricals: one-hot, ordinal, or target encoding.
• Nonlinear transforms: log, square root, or Box-Cox for skewed values.
• Interactions and ratios that combine features into more predictive quantities.
• Date/time parts: hour, weekday, seasonality flags.
• Binning continuous values into meaningful ranges.

Domain knowledge is the edge

The most valuable features encode understanding of the problem. A physicist modeling a plasma might build dimensionless ratios (a normalized beta, a confinement figure of merit) rather than feeding raw signals, because those ratios carry the physics and let a simple model generalize. Domain-driven features often beat automatically generated ones.

df['power_per_area'] = df['power'] / df['area']
df['log_density'] = np.log1p(df['density'])
df['is_peak_hour'] = df['hour'].between(9, 17).astype(int)

Guard against leakage

Fit every transformation (scalers, encoders, imputers) on the training data only, then apply it to validation and test data; fitting on the full set leaks future information and inflates scores. Target encoding is especially prone to leakage and needs out-of-fold computation.

Once you have many candidate features, feature selection prunes the ones that add noise rather than signal.