Convolutional networks and modern vision

An image is structured: nearby pixels are correlated, edges and textures matter, and the same 'cat ear' looks like a cat ear whether in the top-left or middle of the image. Fully-connected layers don't know any of this — they'd need to *learn* the spatial structure from scratch with millions of parameters per layer.

A convolution bakes the structure in: a small filter (e.g. 3×3) slides over the image producing the same response everywhere it sees the same pattern. This gives you translation equivariance for free and reduces parameter count by orders of magnitude. The same 3×3 filter has 9 weights regardless of image size; a fully-connected layer for a 224×224 image to 128 features has ~6 million.

Four properties of convolutions you should internalize: 1. Local connectivity: each output depends only on a small input neighborhood. 2. Weight sharing: the same filter is applied everywhere. 3. Translation equivariance: shift input → shift output. 4. Hierarchy: stacking convs grows the receptive field; deep nets see whole images at the top.

Given input H, kernel k, padding p, stride s, dilation d:

H_out = floor((H + 2p - d*(k-1) - 1) / s) + 1

For the common case (dilation=1):

H_out = floor((H + 2p - k) / s) + 1

The two design patterns:

• `k=3, p=1, s=1` preserves spatial size — used inside conv blocks.
• `k=3, p=1, s=2` halves spatial size — used at downsampling stages.

A 32×32 input through Conv2d(3, 64, k=3, p=1, s=2): H_out = (32 + 2 - 3)//2 + 1 = 16. Stack a few of these, double the channel count each time, and you go (32×32×3) → (16×16×64) → (8×8×128) → (4×4×256). Memorize this pattern; it's the spine of every CNN.

Dilated (atrous) convolutions insert holes in the kernel — receptive field grows without parameter increase. Used in semantic segmentation (DeepLab) and signal processing models (WaveNet).

class ConvBlock(nn.Module):
    def __init__(self, in_c, out_c, stride=1):
        super().__init__()
        self.conv = nn.Conv2d(in_c, out_c, 3, stride, padding=1, bias=False)
        self.bn   = nn.BatchNorm2d(out_c)
        self.act  = nn.ReLU(inplace=True)

    def forward(self, x):
        return self.act(self.bn(self.conv(x)))

Every CNN is variations on Conv → BN → Activation. Bias is False because BatchNorm has its own bias.

ResNet's basic block adds a residual connection:

class BasicBlock(nn.Module):
    def __init__(self, in_c, out_c, stride=1):
        super().__init__()
        self.conv1 = nn.Conv2d(in_c, out_c, 3, stride, 1, bias=False)
        self.bn1   = nn.BatchNorm2d(out_c)
        self.conv2 = nn.Conv2d(out_c, out_c, 3, 1, 1, bias=False)
        self.bn2   = nn.BatchNorm2d(out_c)
        self.shortcut = nn.Identity() if (stride==1 and in_c==out_c) else \
                       nn.Sequential(nn.Conv2d(in_c, out_c, 1, stride, bias=False), nn.BatchNorm2d(out_c))

    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        return F.relu(out + self.shortcut(x))

This 15-line block, stacked 8-50 times, is ResNet-18 through ResNet-152. The residual out + shortcut(x) solved the vanishing-gradient barrier that capped pre-2015 nets at ~20 layers.

The receptive field of a neuron is the patch of the input image that can influence its value. A 3×3 conv has a 3×3 receptive field; stack two and it's 5×5; add a stride-2 pool in between and it grows fast. By the top of a typical CNN, each neuron sees most or all of the image.

Design principle: early layers learn local features (edges, textures), middle layers learn parts (eyes, wheels), late layers learn whole objects. Mess up the receptive field — too small to see whole objects, or growing so fast that boundaries get blurred — and the model can't represent what it needs to.

U-Net is the canonical architecture for image-to-image tasks (segmentation, denoising, super-resolution): a contracting path (downsamples to small spatial size with many channels) + an expanding path (upsamples back to original resolution), with skip connections at each scale. The skips let the decoder see early-layer fine details that the encoder discarded; this is why U-Net masks have crisp edges. Stable Diffusion's denoiser is a U-Net at heart.

• LeNet-5 (1998): 7 layers, MNIST. The original.
• AlexNet (2012): GPU-trained, ReLU, dropout. Started deep-learning hype.
• VGG (2014): all 3×3 convs, very deep. Memory-hungry.
• GoogLeNet/Inception (2014): parallel multi-scale branches. Less common now.
• ResNet (2015): residual connections. *The* foundation of modern CNNs.
• DenseNet (2017): every layer connects to every later layer.
• MobileNet/EfficientNet (2017-19): depthwise-separable convs + compound scaling. State-of-the-art for mobile/edge.
• Vision Transformer (ViT) (2020): patches → token sequence → transformer. Dominates large-scale image classification when you have lots of data.
• ConvNeXt (2022): a ResNet redesigned with modern tricks (LayerNorm, GELU, larger kernels) to match ViT performance while keeping conv efficiency.
• Swin Transformer (2021): windowed attention + shift, hierarchical like a CNN.
• SAM (Segment Anything) (2023): foundation model for segmentation, ViT backbone.

In 2026, for most production vision work, you'll start with a pretrained model from torchvision.models or timm and fine-tune it. Training from scratch on ImageNet is for research; everyone else stands on the shoulders of someone who already did it.

import timm
model = timm.create_model('convnext_base', pretrained=True, num_classes=10)

Computer vision is more than classification. Each task has its own architectural family on top of a CNN/ViT backbone:

• Object detection: predicts bounding boxes + classes. Two-stage (Faster R-CNN: region proposal network + classifier) or one-stage (YOLO, RetinaNet: dense predictions in one pass). Modern: DETR uses transformers for set prediction.
• Instance segmentation: bounding boxes + per-pixel masks. Mask R-CNN extends Faster R-CNN. Modern: Mask2Former, SAM.
• Semantic segmentation: per-pixel class. U-Net (medical), DeepLab (general), SegFormer (transformer-based).
• Keypoint / pose estimation: per-keypoint heatmaps. OpenPose, HRNet.
• Image generation: GANs (covered later) and diffusion models (also later).

The shared pattern: a powerful pretrained backbone produces a feature pyramid, then a task-specific head (RPN, dense prediction head, decoder) consumes it. Almost all of these can be fine-tuned on your custom data with a few thousand labeled examples plus the right augmentations.