Support Vector Machines
An SVM finds the boundary with the widest margin between classes, using kernels to draw nonlinear separators.
Maximum margin
A support vector machine (SVM) separates two classes with the hyperplane that maximizes the margin, the distance to the nearest points of each class. Those nearest points are the support vectors; they alone define the boundary. A wide margin tends to generalize well, which is the geometric intuition behind the method.
Soft margin
Real data overlaps, so the soft-margin SVM allows some points inside or across the margin, penalized by a parameter C. Large C punishes violations hard (low bias, high variance); small C tolerates them (wider margin, more regularization). C is tuned by cross-validation.
The kernel trick
- Linear kernel: a straight boundary, fast and good for high-dimensional sparse data.
- Polynomial kernel: curved boundaries of a chosen degree.
- RBF (Gaussian) kernel: flexible, local boundaries; the common default.
- Kernels compute inner products in a high-dimensional space without ever forming its coordinates.
See kernel methods for how this implicit feature mapping lets a linear algorithm carve nonlinear boundaries.
from sklearn.svm import SVC
clf = SVC(kernel='rbf', C=1.0, gamma='scale')
clf.fit(X_train_scaled, y_train) # always scale features firstPractical notes
SVMs need feature scaling, and the RBF kernel adds gamma (how far one example's influence reaches) as a second knob tuned with C. They are effective in high dimensions and when the boundary is complex but the dataset is modest; training scales poorly to very large n. For probability outputs, SVM scores need calibration.