K-Nearest Neighbors
k-NN classifies or predicts a point by looking at its k closest training examples, storing all data and computing at query time.
Learning by memorizing
k-nearest neighbors (k-NN) is the simplest non-parametric method: it stores the training set and, to predict for a new point, finds its k nearest examples and returns their majority class (classification) or average target (regression). There is no explicit training phase, all the work happens at query time, which is why it is called a lazy learner.
The choices that matter
- k: small k gives a jagged, low-bias, high-variance boundary; large k smooths it toward the majority class.
- Distance metric: Euclidean is common; Manhattan or cosine suit other data.
- Weighting: closer neighbors can vote more heavily than distant ones.
- Scaling: features must be standardized, or large-scale features dominate the distance.
from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=15, weights='distance')
knn.fit(X_train_scaled, y_train)Strengths and costs
k-NN makes no assumption about the shape of the boundary, so it adapts to complex, nonlinear structure and is trivial to implement. The costs are real: prediction is slow and memory-heavy because every query scans the data, and accuracy collapses in high dimensions where distances lose meaning, an instance of the curse of dimensionality.
Making it practical
Spatial index structures such as KD-trees and ball-trees speed neighbor search in low to moderate dimensions; approximate nearest-neighbor libraries scale it further. Choose k by cross-validation. k-NN is best as a fast, interpretable baseline on small, well-scaled, low-dimensional datasets.