Introduction to Artificial Intelligence - Homework Assignment 07 (5 pts.)

• Name:
• NETID:

Learning to Read the Evidence Due November 9, 2026.

This assignment covers:

• Backpropagation through linear layers, ReLU, and softmax cross-entropy
• Training with minibatch stochastic gradient descent and watching the validation curves
• Training the same network in PyTorch, and fighting overfitting with dropout
• Tuning networks to reach target accuracies, including a convolutional neural network (CNN)
• Pointing the trained models at the real evidence

Task	Description	Points
00	Reopen the archive	0
01	Backpropagate through one layer	0.75
02	Backpropagate through the network; explain the order	1.125
03	Train the scratch and PyTorch networks; interpret the curves	0.75
04	Add dropout and reach the 75% FFN target	0.75
05	Build a CNN, reach the 80% target, and compare on held-out data	1.00
06	Read the recovered note	0.125
07	Report the results to Caulfield	0.50
08	Export the notebook	0
	Total	5.00
Bonus	A CNN built from tensor operations	+1

Run the cells in order and answer the written questions. Unlike HW06, this assignment trains networks, so a few cells take a minute or two. Each training cell prints its progress as it goes.

Two tasks have target accuracies instead of exact expected outputs: your tuned feed-forward network must reach 75% and your tuned CNN 80% on the held-out test set. You'll choose the architecture and training settings yourself, so plan time for a few training runs.

Story Progression

Last week's report hangs on Caulfield's board: a network that computes exactly what it should and reads handwriting about as well as a coin toss. Every one of its 239,919 weights was chosen at random.

This week we teach it. Training needs one new ingredient: for every weight, how much the loss would change if we nudged that weight. Those numbers are the gradient, and the procedure that computes all of them in one sweep backward through the network is backpropagation. With the gradient in hand, training is just HW03's update rule, repeated: take a small step downhill, look at a new batch, and step again.

The forensics lab has a standard, though: it won't accept a character reader that scores below 75% on held-out archive images if it's a feed-forward network, or below 80% if it's a convolutional network. Once our network can learn, we'll tune it to meet that bar, build a second kind of network designed for images, and then finally point both at the cut-out letters from the recovered note. The intern has already printed a certificate that says "Forensic AI Specialist." Caulfield has asked him to wait until the end of the assignment.

Evidence 1: Where HW06 left off: the calculation checks out, but the network can't read the note yet.

Task 00: Reopen the Archive

Task 00-1: Description (0 pts.)

Same Data, Same Split, Same Pieces

This cell repeats HW06's setup so a fresh Colab runtime has everything it needs:

• the same 47-class EMNIST archive, converted to tensors with the same normalization;
• the same 8,460 / 940 training/validation split, using the same seed, so HW06's validation numbers are directly comparable;
• HW06's make_loader, log_softmax, cross_entropy, and evaluate, written exactly as you wrote them.

There are two new pieces of evidence in this week's folder:

File	Contents	When to use it
emnist_balanced_small/emnist_balanced_small_test.npz	2,350 held-out archive images	Once, in Task 05-3, after every model has been chosen
segmented_letter_images/	215 cut-out letter crops from the four recovered pages	Task 06
checked_transcription.txt, note_tools.py	The forensics team's checked reading of the note, and supplied code that prepares and scores the crops	Task 06

Nothing to implement here: run the cell and check the printed summary.

Task 00-1: Code (0 pts.)

import os
import shutil
import subprocess
from pathlib import Path

REPO_URL = 'https://github.com/wtheisen/nd-cse-30124-homeworks.git'

try:
    import google.colab
except ImportError:
    # Reuse a local checkout, including when launched from a homework subfolder.
    DATA_ROOT = next((p for p in [Path.cwd(), *Path.cwd().parents]
                      if (p / '.git').exists() and (p / 'evidence').is_dir()), None)
    if DATA_ROOT is None:
        DATA_ROOT = Path.cwd() / 'nd-cse-30124-homeworks'
else:
    # Leave the old checkout before deleting it, including on a setup-cell rerun.
    os.chdir('/content')
    DATA_ROOT = Path('/content/nd-cse-30124-homeworks')
    # Fetch fresh evidence every time this cell runs in Colab.
    if DATA_ROOT.exists():
        shutil.rmtree(DATA_ROOT)

if not DATA_ROOT.exists():
    subprocess.run(['git', 'clone', REPO_URL, str(DATA_ROOT)], check=True)

if not (DATA_ROOT / '.git').exists():
    raise RuntimeError(f'{DATA_ROOT} exists but is not a Git checkout. Choose a different clone location.')

# All later relative paths now start at the homework repository root.
os.chdir(DATA_ROOT)
DATA_ROOT = Path.cwd()

import json
import math
import time
import numpy as np
import matplotlib.pyplot as plt
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import TensorDataset, DataLoader

torch.set_num_threads(2)
torch.manual_seed(42)

EVIDENCE_DIR = DATA_ROOT / 'evidence' / 'homework07'
required = ['emnist_balanced_small/emnist_balanced_small_train.npz',
            'emnist_balanced_small/emnist_balanced_small_test.npz',
            'emnist_balanced_small/classes.json', 'segmented_letter_images',
            'checked_transcription.txt', 'note_tools.py']
if any(not (EVIDENCE_DIR / name).exists() for name in required):
    raise FileNotFoundError('HW07 evidence is missing. Rerun the setup cell with an internet connection, or check that your local repository contains the evidence/homework07 files.')
os.chdir(EVIDENCE_DIR)

# Supplied from HW06: the same tensors, split, and loaders.
raw = np.load('emnist_balanced_small/emnist_balanced_small_train.npz')
classes = json.loads(Path('emnist_balanced_small/classes.json').read_text())['classes']
images = torch.from_numpy(raw['X'].copy()).float().unsqueeze(1) / 255.0
labels = torch.from_numpy(raw['y'].copy()).long()
order = torch.randperm(len(images), generator=torch.Generator().manual_seed(42))
val_idx, train_idx = order[:940], order[940:]
train_data = TensorDataset(images[train_idx], labels[train_idx])
val_data = TensorDataset(images[val_idx], labels[val_idx])

def make_loader(data, shuffle=False, seed=42, batch_size=128):
    return DataLoader(data, batch_size=batch_size, shuffle=shuffle,
                      generator=torch.Generator().manual_seed(seed), num_workers=0)

val_loader = make_loader(val_data)

# Supplied from HW06: stable log-softmax and mean cross-entropy.
def log_softmax(logits):
    shifted = logits - logits.max(dim=-1, keepdim=True).values
    return shifted - shifted.exp().sum(dim=-1, keepdim=True).log()

def cross_entropy(logits, targets):
    return -log_softmax(logits)[torch.arange(len(targets)), targets].mean()

# Supplied from HW06: example-weighted validation loss and accuracy.
@torch.no_grad()
def evaluate(model, loader):
    model.eval()
    total_loss, correct, count = 0., 0, 0
    for xb, yb in loader:
        logits = model(xb) if isinstance(model, nn.Module) else model.forward(xb)
        total_loss += float(F.cross_entropy(logits, yb, reduction='sum'))
        correct += int((logits.argmax(dim=1) == yb).sum())
        count += len(yb)
    return {'loss': round(total_loss / count, 4), 'accuracy': round(correct / count, 4)}

print('Train / validation:', len(train_data), '/', len(val_data))
print('Classes:', len(classes))
print('Note crops:', len(list(Path('segmented_letter_images').glob('*.png'))))

Task 00-1: Expected Output (0 pts.)

Train / validation: 8460 / 940
Classes: 47
Note crops: 215

Task 01: Backpropagate Through One Layer

Task 01-1: Description (0 pts.)

Your Task

Complete two methods of LinearLayer. forward is HW06's.

1. backward(dY): store self.dW and self.db using the cached self.X, then return $\partial L/\partial X$.
2. update(lr): take one gradient-descent step on self.W and self.b.

Do not divide by the batch size here. The loss is already a mean, so its gradient arrives with the $1/B$ built in (Task 02). Dividing again would make every update $B$ times too small.

Python Help

dY = torch.tensor([[1., 0., -1.], [0., 2., 1.]])
dY.T.shape          # torch.Size([3, 2])
dY.sum(dim=0)       # tensor([1., 2., 0.]): add the rows together
W -= 0.1 * dW       # in-place update: W keeps its identity, only its values change

Task 01-1: Code (0.5 pts.)

class LinearLayer:
    def __init__(self, input_dim, output_dim):
        self.W = torch.randn(output_dim, input_dim) * math.sqrt(2 / input_dim)
        self.b = torch.zeros(output_dim)

    def forward(self, X):
        self.X = X
        return X @ self.W.T + self.b

    def backward(self, dY):
        # TODO: Store the weight and bias gradients, then return the gradient for the layer below.
        self.dW = None
        self.db = None
        return None

    def update(self, lr):
        # TODO: Take one gradient-descent step on W and b.
        pass

toy = LinearLayer(2, 3)
toy.W = torch.tensor([[1., 2.], [-1., 1.], [0., 3.]])
toy.b = torch.tensor([0., 1., -1.])
toy.forward(torch.tensor([[1., 2.], [3., 4.]]))
dX = toy.backward(torch.tensor([[1., 0., -1.], [0., 2., 1.]]))
print('dW =', toy.dW.tolist())
print('db =', toy.db.tolist())
print('dX =', dX.tolist())
toy.update(0.5)
print('W after one step with lr = 0.5:', toy.W.tolist())

Task 01-1: Expected Output (0 pts.)

dW = [[1.0, 2.0], [6.0, 8.0], [2.0, 2.0]]
db = [1.0, 2.0, 0.0]
dX = [[1.0, -1.0], [-2.0, 5.0]]
W after one step with lr = 0.5: [[0.5, 1.0], [-4.0, -3.0], [-1.0, 2.0]]

Task 01-2: Description (0 pts.)

Your Task

Complete ReLU.backward(dY): return the upstream gradient with every entry zeroed where the cached input was not positive. update is supplied: there's nothing to learn.

Python Help

X = torch.tensor([-2., 0.5, 3.])
X > 0                    # tensor([False,  True,  True])
torch.tensor([10., 20., 30.]) * (X > 0)    # booleans act like 0 and 1

Task 01-2: Code (0.25 pts.)

class ReLU:
    def forward(self, X):
        self.X = X
        return X.clamp_min(0)

    def backward(self, dY):
        # TODO: Pass the gradient only where the cached input was positive.
        return None

    def update(self, lr):
        pass  # No trainable parameters.

gate = ReLU()
gate.forward(torch.tensor([-2., 0.5, 3.]))
print('ReLU backward:', gate.backward(torch.tensor([10., 20., 30.])).tolist())

Task 01-2: Expected Output (0 pts.)

ReLU backward: [0.0, 20.0, 30.0]

Figure 1. The forward pass (top) caches each layer's input. The backward pass (bottom) starts at the loss and moves in reverse; each linear layer turns the upstream gradient G into its own dW and db and passes dX down. Every gradient has the shape of the tensor it belongs to.

Task 02: Backpropagate Through the Whole Network

Task 02-1: Description (0 pts.)

A More Flexible Network

ScratchFFN now takes a hidden_sizes argument, so you can change its width and depth in Task 04; the default (256, 128) builds exactly HW06's network. Its dropout argument and its train/eval methods also come into play in Task 04. Nothing else changes.

Your Task

Complete two methods of ScratchFFN. forward is HW06's.

1. backward(logits, targets):
   1. Compute the softmax probabilities from the logits (use log_softmax(...).exp()).
   2. Subtract 1 from each example's correct-class entry (paired indexing, as in HW06's cross_entropy).
   3. Divide by the batch size.
   4. Walk through self.layers in reverse, replacing the gradient with each layer's backward output.
2. update(lr): call every layer's update.

The supplied check prints the logit gradient for the worked example and the gradient shape for every parameter.

Python Help

for layer in reversed(model.layers):   # last layer first
    ...
p = torch.tensor([[0.665, 0.245, 0.090]])
p[torch.arange(1), torch.tensor([0])] -= 1    # subtract 1 at [0, 0]

Task 02-1: Code (1.0 pts.)

class ScratchFFN:
    def __init__(self, hidden_sizes=(256, 128), dropout=0.0):
        # The default builds HW06's network: 784 -> 256 -> 128 -> 47. Dropout arrives in Task 04.
        self.layers = []
        width = 784
        for size in hidden_sizes:
            self.layers += [LinearLayer(width, size), ReLU()]
            if dropout > 0:
                self.layers.append(Dropout(dropout))
            width = size
        self.layers.append(LinearLayer(width, 47))

    def train(self):
        for layer in self.layers:
            if hasattr(layer, 'active'):
                layer.active = True

    def eval(self):
        for layer in self.layers:
            if hasattr(layer, 'active'):
                layer.active = False

    def forward(self, X):
        X = X.flatten(1)
        for layer in self.layers:
            X = layer.forward(X)
        return X

    def backward(self, logits, targets):
        # TODO: Start from (softmax - one-hot) / B, then walk backward through the layers.
        grad = None
        self.dlogits = grad
        return grad

    def update(self, lr):
        # TODO: Update every layer.
        pass

torch.manual_seed(42)
model = ScratchFFN()
logits = torch.tensor([[2., 1., 0.]])
print('Worked example dlogits:', (log_softmax(logits).exp() - torch.tensor([[1., 0., 0.]])).numpy().round(3))

xb, yb = next(iter(make_loader(train_data, batch_size=4)))
logits = model.forward(xb)
model.backward(logits, yb)
print('dlogits shape:', tuple(model.dlogits.shape), '| each row sums to', round(float(model.dlogits.sum(dim=1).abs().max()), 6))
for number, layer in enumerate([l for l in model.layers if isinstance(l, LinearLayer)], 1):
    print(f'Linear {number}: W {tuple(layer.W.shape)} dW {tuple(layer.dW.shape)} | b {tuple(layer.b.shape)} db {tuple(layer.db.shape)}')

Task 02-1: Expected Output (0 pts.)

Worked example dlogits: [[-0.335  0.245  0.09 ]]
dlogits shape: (4, 47) | each row sums to 0.0
Linear 1: W (256, 784) dW (256, 784) | b (256,) db (256,)
Linear 2: W (128, 256) dW (128, 256) | b (128,) db (128,)
Linear 3: W (47, 128) dW (47, 128) | b (47,) db (47,)

Task 02-2: Written Response (0.125 pts.)

1. Why must the backward pass visit the layers in reverse order, and why must it finish before any weights are updated?
   • [ANSWER]
2. Where does the $1/B$ enter the gradient? What would happen to the size of each update if every LinearLayer.backward also divided by $B$?
   • [ANSWER]

Story Progression

The backward pass is written, and the intern asks why we didn't just let PyTorch work out the gradients for us. Caulfield points out that the intern also asked why we couldn't just read the letters by hand. Understanding what the library does is what lets you notice when it is doing the wrong thing.

There's only one test that matters now: if the gradients are right, the network should learn. Time to train.

Evidence 2: The training loop: forward, compare with the label, send the error backward, step downhill, repeat.

Task 03: Train the Networks

Task 03-1: Description (0 pts.)

Your Task

Complete the inner loop of train_scratch:

1. Compute the logits with model.forward.
2. Run model.backward with those logits and the batch's labels.
3. Call model.update with the learning rate.

train_scratch also accepts a batch_size and an optional decay_every (halve the learning rate every that many epochs). The defaults are used here; you'll tune them in Task 04. The supplied code trains for 40 epochs at learning rate 0.1 and prints progress every 5 epochs.

Python Help

for xb, yb in loader:   # one batch of images and labels at a time
    ...

model.forward, model.backward, and model.update are the methods you wrote in Tasks 01 and 02 (and HW06).

Task 03-1: Code (0.375 pts.)

# Supplied: HW06's network in PyTorch, and a helper that copies the scratch network's weights into it.
class TorchFFN(nn.Module):
    def __init__(self):
        super().__init__()
        self.layers = nn.Sequential(nn.Flatten(),
                                    nn.Linear(784, 256), nn.ReLU(),
                                    nn.Linear(256, 128), nn.ReLU(),
                                    nn.Linear(128, 47))

    def forward(self, X):
        return self.layers(X)

def copy_to_torch(scratch, reference):
    scratch_linear = [layer for layer in scratch.layers if isinstance(layer, LinearLayer)]
    torch_linear = [layer for layer in reference.layers if isinstance(layer, nn.Linear)]
    with torch.no_grad():
        for ours, theirs in zip(scratch_linear, torch_linear):
            theirs.weight.copy_(ours.W)
            theirs.bias.copy_(ours.b)

def train_scratch(model, epochs, lr, batch_size=128, decay_every=None):
    loader = make_loader(train_data, shuffle=True, batch_size=batch_size)
    history = []
    for epoch in range(1, epochs + 1):
        if decay_every and epoch > 1 and (epoch - 1) % decay_every == 0:
            lr = lr / 2
        model.train()
        for xb, yb in loader:
            # TODO: Forward, backward, then update.
            pass
        metrics = evaluate(model, val_loader)
        history.append({'epoch': epoch, **metrics})
        if epoch == 1 or epoch % 5 == 0:
            print(f"Scratch epoch {epoch:2d}: val loss {metrics['loss']:.4f}, accuracy {metrics['accuracy']:.4f}")
    return model, history

def peak_epoch(history):
    # Supplied: the epoch with the highest validation accuracy (for judging the epoch budget).
    peak = history[0]
    for row in history:
        if row['accuracy'] > peak['accuracy']:
            peak = row
    return peak

EPOCHS, LR = 40, 0.1
torch.manual_seed(42)
scratch_start = ScratchFFN()
torch_start = TorchFFN()
copy_to_torch(scratch_start, torch_start)   # both networks begin from identical weights

started = time.perf_counter()
scratch_ffn, scratch_history = train_scratch(scratch_start, EPOCHS, LR)
peak = peak_epoch(scratch_history)
print(f"Final scratch model: val accuracy {scratch_history[-1]['accuracy']:.4f} | peak was epoch {peak['epoch']} ({peak['accuracy']:.4f})")
print(f'Training time: {time.perf_counter() - started:.0f} s')

Task 03-1: Reference Output (0 pts.)

The epoch lines should match when you run the cells in order. Training time depends on your machine.

Scratch epoch  1: val loss 3.0444, accuracy 0.1681
Scratch epoch  5: val loss 1.7384, accuracy 0.5117
Scratch epoch 10: val loss 1.9496, accuracy 0.5011
Scratch epoch 15: val loss 1.3074, accuracy 0.6191
Scratch epoch 20: val loss 1.3266, accuracy 0.6457
Scratch epoch 25: val loss 1.2108, accuracy 0.6564
Scratch epoch 30: val loss 1.0979, accuracy 0.6904
Scratch epoch 35: val loss 1.7062, accuracy 0.5872
Scratch epoch 40: val loss 1.1921, accuracy 0.6957
Final scratch model: val accuracy 0.6957 | peak was epoch 26 (0.7245)

Task 03-2: Description (0 pts.)

Your Task

Complete the four lines in the inner loop of train_torch: clear the gradients, compute the loss, backpropagate, and step. The optimizer is created for you. The supplied code then trains the PyTorch network and plots both validation curves.

Useful Links

• Optimizing model parameters (PyTorch tutorial)
• torch.optim.SGD

Task 03-2: Code (0.25 pts.)

def train_torch(model, epochs, lr, optimizer_name='sgd'):
    loader = make_loader(train_data, shuffle=True)
    if optimizer_name == 'adam':
        optimizer = torch.optim.Adam(model.parameters(), lr=lr)
    else:
        optimizer = torch.optim.SGD(model.parameters(), lr=lr)
    history = []
    for epoch in range(1, epochs + 1):
        model.train()
        for xb, yb in loader:
            # TODO: Clear old gradients, compute the loss, backpropagate, and take a step.
            pass
        metrics = evaluate(model, val_loader)
        history.append({'epoch': epoch, **metrics})
        if epoch == 1 or epoch % 5 == 0:
            print(f"PyTorch epoch {epoch:2d}: val loss {metrics['loss']:.4f}, accuracy {metrics['accuracy']:.4f}")
    return model, history

torch_ffn, torch_history = train_torch(torch_start, EPOCHS, LR)
peak = peak_epoch(torch_history)
print(f"Final PyTorch model: val accuracy {torch_history[-1]['accuracy']:.4f} | peak was epoch {peak['epoch']} ({peak['accuracy']:.4f})")

# Supplied: plot both validation histories.
def plot_histories(histories, title):
    fig, axes = plt.subplots(1, 2, figsize=(12, 4))
    for name, history in histories.items():
        epochs = [row['epoch'] for row in history]
        axes[0].plot(epochs, [row['loss'] for row in history], label=name)
        axes[1].plot(epochs, [row['accuracy'] for row in history], label=name)
    axes[0].set_ylabel('validation loss'); axes[1].set_ylabel('validation accuracy')
    for ax in axes:
        ax.set_xlabel('epoch'); ax.grid(alpha=0.3); ax.legend()
    fig.suptitle(title)
    fig.tight_layout()
    plt.show()

plot_histories({'Scratch FFN': scratch_history, 'PyTorch FFN': torch_history},
               'Same start, same batches, same learning rate')

Task 03-2: Reference Output (0 pts.)

PyTorch epoch  1: val loss 3.0444, accuracy 0.1681
PyTorch epoch  5: val loss 1.7408, accuracy 0.5170
PyTorch epoch 10: val loss 1.9982, accuracy 0.4872
PyTorch epoch 15: val loss 1.3124, accuracy 0.6191
PyTorch epoch 20: val loss 1.3096, accuracy 0.6500
PyTorch epoch 25: val loss 1.2220, accuracy 0.6606
PyTorch epoch 30: val loss 1.1059, accuracy 0.6819
PyTorch epoch 35: val loss 1.6854, accuracy 0.5883
PyTorch epoch 40: val loss 1.2064, accuracy 0.6926
Final PyTorch model: val accuracy 0.6926 | peak was epoch 26 (0.7213)

Validation histories of the scratch and PyTorch FFNs. Compare the plot produced by your code with this reference image.

Task 03-3: Written Response (0.125 pts.)

1. Do your two curves track each other? Give one reason they might not be exactly identical even though both implementations are correct.
   • [ANSWER]
2. Describe what validation loss and validation accuracy do in the later epochs. Was 40 epochs a sensible budget? Why must decisions like the number of epochs be made with validation data rather than test data?
   • [ANSWER]

Task 04: Reach the Forensics Lab's Standard

Task 04-1: Description (0 pts.)

Your Task

Complete the two marked parts of Dropout:

1. In forward: build a mask the same shape as X whose entries are $1/(1-p)$ with probability $1-p$ and $0$ otherwise. Store it in self.mask and return X * self.mask.
2. In backward: return the upstream gradient multiplied by the same mask.

The inactive case (evaluation mode, or $p = 0$) is supplied.

Python Help

torch.rand_like(X)              # uniform random numbers in [0, 1), same shape as X
(torch.rand_like(X) >= 0.2)     # True with probability 0.8
(torch.rand_like(X) >= 0.2).float() / 0.8    # 1.25 where kept, 0 elsewhere

Useful Links

• Srivastava et al. (2014), Dropout: A Simple Way to Prevent Neural Networks from Overfitting
• nn.Dropout

Task 04-1: Code (0.25 pts.)

class Dropout:
    def __init__(self, p):
        self.p = p
        self.active = True

    def forward(self, X):
        if not self.active or self.p == 0:
            self.mask = None
            return X
        # TODO: Build a scaled random keep-mask, store it, and apply it.
        self.mask = None
        return None

    def backward(self, dY):
        if self.mask is None:
            return dY
        # TODO: Send the gradient through the same mask.
        return None

    def update(self, lr):
        pass  # No trainable parameters.

torch.manual_seed(0)
drop = Dropout(0.5)
X_demo = torch.full((4, 10000), 3.0)
out = drop.forward(X_demo)
print('Fraction of activations kept:', round(float((out != 0).float().mean()), 3))
print('Mean activation before / after:', 3.0, round(float(out.mean()), 3))
print('Each kept activation becomes:', float(out[out != 0][0]))
print('Backward uses the same mask:', torch.equal(drop.backward(torch.ones_like(X_demo)), drop.mask))
drop.active = False
print('Evaluation mode leaves the input unchanged:', torch.equal(drop.forward(X_demo), X_demo))

Task 04-1: Reference Output (0 pts.)

The kept fraction and mean come from a fixed seed; they should be close to 0.5 and 3.0. The other lines are exact.

Fraction of activations kept: 0.502
Mean activation before / after: 3.0 3.012
Each kept activation becomes: 6.0
Backward uses the same mask: True
Evaluation mode leaves the input unchanged: True

Task 04-2: Description (0 pts.)

Your Task

Edit FFN_CONFIG, run the cell, and repeat until your final validation accuracy is at least 0.75. Leave your final configuration in place when you submit.

Task 04-2: Code (0.5 pts.)

# TODO: Tune these settings until the final validation accuracy is at least 0.75.
FFN_CONFIG = {
    'hidden_sizes': (256, 128),
    'dropout': 0.0,
    'lr': 0.1,
    'epochs': 40,
    'batch_size': 128,
    'decay_every': None,
}

torch.manual_seed(42)
started = time.perf_counter()
tuned_ffn, tuned_history = train_scratch(
    ScratchFFN(FFN_CONFIG['hidden_sizes'], FFN_CONFIG['dropout']),
    FFN_CONFIG['epochs'], FFN_CONFIG['lr'], FFN_CONFIG['batch_size'], FFN_CONFIG['decay_every'])
final, peak = tuned_history[-1], peak_epoch(tuned_history)
print(f"Final tuned model: val loss {final['loss']:.4f}, accuracy {final['accuracy']:.4f} | peak was epoch {peak['epoch']} ({peak['accuracy']:.4f})")
print('Validation goal of 0.75 reached:', final['accuracy'] >= 0.75)
print(f'Training time: {time.perf_counter() - started:.0f} s')

Task 04-2: Reference Output (0 pts.)

This is the reference configuration's run. Yours will differ with your settings; what matters is the last lines.

Scratch epoch  1: val loss 2.4652, accuracy 0.3043
Scratch epoch  5: val loss 1.4147, accuracy 0.5989
Scratch epoch 10: val loss 1.2879, accuracy 0.6266
Scratch epoch 15: val loss 1.0073, accuracy 0.6968
Scratch epoch 20: val loss 0.9586, accuracy 0.7245
Scratch epoch 25: val loss 0.8935, accuracy 0.7330
Scratch epoch 30: val loss 0.8960, accuracy 0.7298
Scratch epoch 35: val loss 0.8591, accuracy 0.7340
Scratch epoch 40: val loss 0.8609, accuracy 0.7457
Scratch epoch 45: val loss 0.8361, accuracy 0.7511
Scratch epoch 50: val loss 0.8324, accuracy 0.7521
Scratch epoch 55: val loss 0.8366, accuracy 0.7521
Scratch epoch 60: val loss 0.8277, accuracy 0.7511
Final tuned model: val loss 0.8277, accuracy 0.7511 | peak was epoch 36 (0.7574)
Validation goal of 0.75 reached: True

Task 05: Build a Network for Images

Task 05-1: Description (0 pts.)

2 × 2 max pooling: each output is the largest value in its colored window.

Your Task

Complete CNN.__init__: build self.layers as one nn.Sequential matching the table above, in order. The supplied code traces the shape after every layer and counts the parameters.

Python Help

nn.Conv2d(in_channels, out_channels, kernel_size, padding=...)
nn.MaxPool2d(2)          # 2 x 2 windows, stride 2
nn.Sequential(nn.Flatten(), nn.Linear(784, 47))   # layers run in order: model(x)
sum(p.numel() for p in model.parameters())   # total parameter count

Useful Links

• nn.Conv2d · nn.MaxPool2d · nn.Sequential
• CS231n notes on convolutional networks

Task 05-1: Code (0.25 pts.)

class CNN(nn.Module):
    def __init__(self):
        super().__init__()
        # TODO: Two convolution + ReLU + pooling blocks, then flatten and two linear layers.
        self.layers = None

    def forward(self, X):
        return self.layers(X)

torch.manual_seed(42)
cnn = CNN()
activations = torch.zeros(4, 1, 28, 28)
print('input'.ljust(10), tuple(activations.shape))
for layer in cnn.layers:
    activations = layer(activations)
    print(type(layer).__name__.ljust(10), tuple(activations.shape))
print('Trainable parameters:', sum(p.numel() for p in cnn.parameters()))

Task 05-1: Expected Output (0 pts.)

input      (4, 1, 28, 28)
Conv2d     (4, 16, 28, 28)
ReLU       (4, 16, 28, 28)
MaxPool2d  (4, 16, 14, 14)
Conv2d     (4, 32, 14, 14)
ReLU       (4, 32, 14, 14)
MaxPool2d  (4, 32, 7, 7)
Flatten    (4, 1568)
Linear     (4, 128)
ReLU       (4, 128)
Linear     (4, 47)
Trainable parameters: 211695

Figure 2. The CNN on the evidence board: small shared filters produce feature maps, pooling shrinks them, and a small classifier reads the result.

Task 05-2: Description (0 pts.)

Your Task

Define TunedCNN and choose CNN_EPOCHS. The supplied code trains it with Adam and reports its final validation accuracy.

Task 05-2: Code (0.5 pts.)

class TunedCNN(nn.Module):
    def __init__(self):
        super().__init__()
        # TODO: Start from your Task 05-1 layers, then improve them to reach the 80% target.
        self.layers = None

    def forward(self, X):
        return self.layers(X)

CNN_EPOCHS = 15

torch.manual_seed(42)
cnn = TunedCNN()
print('Trainable parameters:', sum(p.numel() for p in cnn.parameters()))
started = time.perf_counter()
cnn, cnn_history = train_torch(cnn, epochs=CNN_EPOCHS, lr=0.001, optimizer_name='adam')
final, peak = cnn_history[-1], peak_epoch(cnn_history)
print(f"Final CNN: val loss {final['loss']:.4f}, accuracy {final['accuracy']:.4f} | peak was epoch {peak['epoch']} ({peak['accuracy']:.4f})")
print('Validation goal of 0.81 reached:', final['accuracy'] >= 0.81)
print(f'Training time: {time.perf_counter() - started:.0f} s')

Task 05-2: Reference Output (0 pts.)

This is the reference design's run. Yours will differ with your design; what matters is the last lines.

Trainable parameters: 833967
PyTorch epoch  1: val loss 1.5402, accuracy 0.5702
PyTorch epoch  5: val loss 0.7138, accuracy 0.7702
PyTorch epoch 10: val loss 0.6119, accuracy 0.8043
PyTorch epoch 15: val loss 0.5972, accuracy 0.8191
PyTorch epoch 20: val loss 0.6009, accuracy 0.8213
Final CNN: val loss 0.6009, accuracy 0.8213 | peak was epoch 16 (0.8213)
Validation goal of 0.81 reached: True

Task 05-3: Description (0 pts.)

Opening the Test Set, Once

Every model has now been chosen using validation data only. It's finally time to open the held-out test set and evaluate every model on exactly the same 2,350 images. These numbers are our honest estimate of how each model would do on new archive images, and they are the numbers the lab's targets are checked against.

Your Task

For each (name, model) pair in finalists, evaluate it on test_loader and append a dictionary with its name and metrics to results. The supplied code prints the table and checks both targets.

Python Help

row = {'model': 'Tuned CNN', **{'loss': 0.6, 'accuracy': 0.82}}
# {'model': 'Tuned CNN', 'loss': 0.6, 'accuracy': 0.82}

Task 05-3: Code (0.125 pts.)

heldout = np.load('emnist_balanced_small/emnist_balanced_small_test.npz')
test_data = TensorDataset(torch.from_numpy(heldout['X'].copy()).float().unsqueeze(1) / 255.0,
                          torch.from_numpy(heldout['y'].copy()).long())
test_loader = make_loader(test_data)

finalists = [('Scratch FFN (Task 03)', scratch_ffn), ('PyTorch FFN (Task 03)', torch_ffn),
             ('Tuned scratch FFN', tuned_ffn), ('Tuned CNN', cnn)]
results = []
for name, finalist in finalists:
    # TODO: Evaluate each finalist on the held-out test set.
    pass

print(f'Held-out test set ({len(test_data)} images):')
for row in results:
    print(f"  {row['model']:<22} loss {row['loss']:.4f}  accuracy {row['accuracy']:.4f}")

targets = {'Tuned scratch FFN': 0.75, 'Tuned CNN': 0.80}
for row in results:
    if row['model'] in targets:
        goal = targets[row['model']]
        status = 'met' if row['accuracy'] >= goal else 'NOT met yet'
        print(f"Target for {row['model']}: {goal:.0%} -> {status}")

Task 05-3: Reference Output (0 pts.)

Held-out test set (2350 images):
  Scratch FFN (Task 03)  loss 1.1786  accuracy 0.7034
  PyTorch FFN (Task 03)  loss 1.1849  accuracy 0.7064
  Tuned scratch FFN      loss 0.8043  accuracy 0.7664
  Tuned CNN              loss 0.5977  accuracy 0.8255
Target for Tuned scratch FFN: 75% -> met
Target for Tuned CNN: 80% -> met

Task 05-4: Written Response (0.125 pts.)

Most of your tuned CNN's parameters sit in its first linear layer; its convolutional layers hold only a small fraction of them. Using locality and weight sharing, explain how those few convolutional parameters let the CNN beat the FFN on these images. Then explain why this experiment alone cannot tell us how much of the improvement came from the architecture rather than the optimizer, and describe one extra run that would separate the two.

• [ANSWER]

Task 06: Read the Recovered Note

Task 06-1: Description (0 pts.)

There's no code to write here; this task is about reading the results. The supplied cell reads the note with your tuned FFN and CNN, before and after preparation, and shows a few crops at each stage.

Task 06-1: Code (0 pts.)

import note_tools   # supplied in evidence/homework07: crop loading, preparation, and scoring

crop_paths, raw_batch = note_tools.load_crops('segmented_letter_images')
prepared_batch = note_tools.prepare_crops(crop_paths)
truth = note_tools.load_truth('checked_transcription.txt')
print('Crops:', len(crop_paths), '| prepared batch:', tuple(prepared_batch.shape))

for name, reader in [('Tuned scratch FFN', tuned_ffn), ('Tuned CNN', cnn)]:
    before = note_tools.character_accuracy(note_tools.read_crops(reader, raw_batch, classes), truth)
    after = note_tools.character_accuracy(note_tools.read_crops(reader, prepared_batch, classes), truth)
    print(f'{name:<17} character accuracy: raw crops {before:.3f} -> prepared crops {after:.3f}')

print('CNN reading, raw (first 40):     ', note_tools.read_crops(cnn, raw_batch, classes)[:40])
print('CNN reading, prepared (first 40):', note_tools.read_crops(cnn, prepared_batch, classes)[:40])
print('Checked transcription (first 40):', truth[:40])

note_tools.show_preparation(crop_paths, raw_batch, prepared_batch, cnn, classes)

Task 06-1: Reference Output (0 pts.)

Your accuracies depend on your tuned models; the reference models give:

Crops: 215 | prepared batch: (215, 1, 28, 28)
Tuned scratch FFN character accuracy: raw crops 0.182 -> prepared crops 0.682
Tuned CNN         character accuracy: raw crops 0.192 -> prepared crops 0.799
CNN reading, raw (first 40):      PPQPQQBQQBRBPBBBBBQQPPQBBPBBQMBBMQQQQBGR
CNN reading, prepared (first 40): VYBIROZHWIREVhZZEGURVFRASYBIONGGURRFGNGR
Checked transcription (first 40): VYBIRQZHEQREVATZEGURVFRASYBLQNGGURRFGNGR

The same crops before and after preparation, with the CNN's raw and prepared readings.

Task 06-2: Written Response (0.125 pts.)

Which difference in the table do you think hurt the raw reading most, and how did preparation address it? Then look at the prepared reading's remaining mistakes and give one kind that preparation cannot fix.

• [ANSWER]

Task 07: Report the Results to Caulfield

Task 07-1: Written Response (0.5 pts.)

1. Report the held-out test accuracy of your two tuned models and whether each met its target. Briefly describe what you changed in Tasks 04 and 05 and which change helped most. Which model would you send to the forensics lab, and why?
   • [ANSWER]
2. Your scratch and PyTorch networks trained along nearly the same curves. What does that agreement establish about your implementation, and what doesn't it establish?
   • [ANSWER]
3. Your best model reads archive images far better than it reads the note. Using the term distribution shift, explain to Caulfield why the test-set accuracy overstated how well it would read the evidence, and what that means for how much he should trust the reading.
   • [ANSWER]

Story Progression

The forensics lab returns the reading with a few corrections penciled in. About four letters in five were right. A couple of Qs came back as Os, an L as an I, and one crop turned out to hold two letters glued together. The intern points out, with some satisfaction, that he would never have confused a Q with an O. Caulfield points out that he took four hours to read one page.

Then they read the corrected text aloud: "V YBIRQ ZHEQREVAT ZE ..."

It isn't English. Every letter on the page has been swapped for another one. Our network read the note correctly; the note itself is in code. Reading sequences of characters, and learning to translate them, needs a different kind of network that remembers what came before. That's where HW08 begins.

Evidence 3: The recovered note, read at last, and still in code.

Task 08: Export the Completed Notebook (0 pts.)

1. Complete your code and written responses, and run the cells above in order. Compare your results with the expected and reference outputs.
2. Save your notebook, including its outputs. Set GENERATE = True below when you are ready to export.
3. In Colab, the cell captures the current notebook and its outputs. Locally, it reads the saved notebook at NOTEBOOK_PATH; adjust that path if you saved your copy elsewhere.
4. Download homework07.html and open it in a browser. Check that your name, NETID, code, written answers, and outputs appear.

Submit only homework07.html on Canvas. The exporter does not rerun your code.

Task 08-1: Code (0 pts.)

GENERATE = False
NOTEBOOK_PATH = DATA_ROOT / 'homeworks' / 'homework07' / 'homework07.ipynb'

if GENERATE:
    import json
    import nbformat
    from nbconvert import HTMLExporter
    from pathlib import Path

    try:
        from google.colab import _message, files
    except ImportError:
        current = nbformat.read(NOTEBOOK_PATH, as_version=4)
        html_path = NOTEBOOK_PATH.with_name('homework07.html')
        in_colab = False
    else:
        captured = _message.blocking_request('get_ipynb', timeout_sec=10)
        current = nbformat.reads(json.dumps(captured['ipynb']), as_version=4)
        html_path = Path('/content/homework07.html')
        in_colab = True

    exporter = HTMLExporter()
    exporter.embed_images = True
    report_html, _ = exporter.from_notebook_node(current)
    html_path.write_text(report_html, encoding='utf-8')
    print(f'Exported {html_path.name}. Open it in a browser before submitting.')

    if in_colab:
        files.download(str(html_path))
    else:
        print(f'Saved to: {html_path}')
else:
    print('Export disabled. Set GENERATE = True when ready.')

Task 08-1: Expected Output (0 pts.)

Export disabled. Set GENERATE = True when ready.

Bonus: A CNN Built from Tensor Operations (up to +1 pt.)

This section is optional and is not part of the five required points.

nn.Conv2d and nn.MaxPool2d hide their backward passes inside autograd. Here you'll write both yourself, so that your ScratchFFN machinery can train a CNN with no autograd at all.

Convolution as a matrix product. F.unfold cuts every 3 × 3 patch out of an image and lays each one out as a column: an input of shape (B, C, H, W) becomes (B, C·9, H·W). Flatten the filters into a matrix W of shape (filters, C·9), and the whole convolution is one batched product W @ patches plus a bias, reshaped back to (B, filters, H, W). That means the backward pass is Task 01's linear layer again:

• $dW$: multiply the upstream gradient by the patches, summed over the batch and every position (each position reused the same filter);
• $db$: the upstream gradient summed over the batch and every position;
• $d(\text{patches}) = W^T G$, and then F.fold adds each patch's gradient back into the pixels it came from. A pixel that appeared in nine patches receives nine contributions.

Max pooling backward. Only the winning entry of each 2 × 2 window affected the output, so the whole upstream gradient goes to the winner's position and the other three get zero. Cache which entry won during the forward pass.

Complete the two # BONUS TODO sections. The supplied checks compare your layers with PyTorch's on a small random input. Set RUN_BONUS_TRAINING = True to train the scratch CNN for 10 epochs (this takes several minutes) and report its validation accuracy.

class ConvLayer:
    def __init__(self, in_channels, out_channels, kernel=3, padding=1):
        self.kernel, self.padding = kernel, padding
        fan_in = in_channels * kernel * kernel
        self.W = torch.randn(out_channels, fan_in) * math.sqrt(2 / fan_in)
        self.b = torch.zeros(out_channels)

    def forward(self, X):
        self.input_shape = X.shape
        self.patches = F.unfold(X, self.kernel, padding=self.padding)        # (B, C*k*k, H*W)
        # BONUS TODO: Apply every filter to every patch and add the bias.
        Y = None
        B, _, H, W = X.shape
        return Y.reshape(B, len(self.b), H, W)

    def backward(self, dY):
        G = dY.flatten(2)                                                     # (B, filters, H*W)
        # BONUS TODO: Sum the filter and bias gradients over the batch and positions; fold the patch gradients back.
        self.dW = None
        self.db = None
        return None

    def update(self, lr):
        self.W -= lr * self.dW
        self.b -= lr * self.db

class MaxPool:
    def forward(self, X):
        B, C, H, W = X.shape
        self.input_shape = X.shape
        windows = X.reshape(B, C, H // 2, 2, W // 2, 2).permute(0, 1, 2, 4, 3, 5).reshape(B, C, H // 2, W // 2, 4)
        out, self.winners = windows.max(dim=-1)
        return out

    def backward(self, dY):
        B, C, H, W = self.input_shape
        # BONUS TODO: Send each upstream value to its window's winner; zeros elsewhere.
        windows = None
        return windows.reshape(B, C, H // 2, W // 2, 2, 2).permute(0, 1, 2, 4, 3, 5).reshape(B, C, H, W)

    def update(self, lr):
        pass

class Flatten:
    def forward(self, X):
        self.input_shape = X.shape
        return X.flatten(1)

    def backward(self, dY):
        return dY.reshape(self.input_shape)

    def update(self, lr):
        pass

class ScratchCNN(ScratchFFN):
    def __init__(self):
        self.layers = [ConvLayer(1, 8), ReLU(), MaxPool(),
                       ConvLayer(8, 16), ReLU(), MaxPool(), Flatten(),
                       LinearLayer(16 * 7 * 7, 128), ReLU(), LinearLayer(128, 47)]

    def forward(self, X):
        for layer in self.layers:
            X = layer.forward(X)
        return X

# Supplied checks against PyTorch, in double precision.
torch.manual_seed(0)
conv = ConvLayer(1, 2)
conv.W, conv.b = conv.W.double(), conv.b.double()
x = torch.randn(2, 1, 4, 4, dtype=torch.float64, requires_grad=True)
w = conv.W.reshape(2, 1, 3, 3).clone().requires_grad_()
b = conv.b.clone().requires_grad_()
upstream = torch.randn(2, 2, 4, 4, dtype=torch.float64)
(F.conv2d(x, w, b, padding=1) * upstream).sum().backward()
ours = conv.forward(x.detach())
dx = conv.backward(upstream)
print('Convolution forward matches:', torch.allclose(ours, F.conv2d(x, w, b, padding=1)))
print('Convolution gradients match:', torch.allclose(dx, x.grad) and torch.allclose(conv.dW, w.grad.flatten(1)) and torch.allclose(conv.db, b.grad))

pool = MaxPool()
px = torch.randn(2, 2, 4, 4, dtype=torch.float64, requires_grad=True)
pooled = F.max_pool2d(px, 2)
pool_upstream = torch.randn_like(pooled)
(pooled * pool_upstream).sum().backward()
print('Max pool forward matches:', torch.allclose(pool.forward(px.detach()), pooled))
print('Max pool gradient matches:', torch.allclose(pool.backward(pool_upstream), px.grad))

RUN_BONUS_TRAINING = False
if RUN_BONUS_TRAINING:
    bonus_cnn, bonus_history = train_scratch(ScratchCNN(), epochs=10, lr=0.05)
    print('Scratch CNN validation:', evaluate(bonus_cnn, val_loader))

Bonus: Expected Output

Convolution forward matches: True
Convolution gradients match: True
Max pool forward matches: True
Max pool gradient matches: True