A Naive Bayes Classifier by Hand

Classify with Bayes' rule under a strong independence assumption, worked on a small spam-style example.

Bayes' rule for classification

We want P(class | features). Bayes' rule gives P(class | x) proportional to P(class) * P(x | class). The naive assumption is that features are conditionally independent given the class, so P(x | class) factors into a product of per-feature probabilities - trivial to estimate.

A worked count

Classify short messages as spam or ham from word presence. Estimate each word's probability in each class by counting, apply Laplace smoothing to avoid zeros, then multiply. Work in log space so many small probabilities do not underflow.

import numpy as np
docs=[('buy cheap now',1),('meeting at noon',0),
      ('cheap deal buy',1),('project noon meeting',0)]
vocab=set(w for d,_ in docs for w in d.split())
def counts(c):
    w={}; n=0
    for d,lab in docs:
        if lab==c:
            for tok in d.split(): w[tok]=w.get(tok,0)+1; n+=1
    return w,n
def logp(msg,c):
    w,n=counts(c); V=len(vocab); lp=np.log(0.5)
    for tok in msg.split():
        lp+=np.log((w.get(tok,0)+1)/(n+V))   # Laplace smoothing
    return lp
m='cheap meeting'
print('spam' if logp(m,1)>logp(m,0) else 'ham')

Why naive works

The independence assumption is almost always false - words co-occur - yet naive Bayes classifies well because it only needs the correct class to get the highest score, not accurate probabilities. Its estimates are poorly calibrated but its decision boundary is often good, and it trains in a single pass over the data.

Strengths and limits

Naive Bayes is fast, needs little data, and handles high-dimensional sparse features like text gracefully. It struggles when features are strongly correlated or when calibrated probabilities are required. It remains a strong, hard-to-beat baseline for text classification and a fine sanity check before reaching for heavier models.