Introduction to Artificial Intelligence - Homework Assignment 03 (5 pts.)

• Name:
• NETID:

Can We Trust the Witness?
Due September 28, 2026.

This assignment covers the following topics:

• Implementing the perceptron learning rule and training loop
• Connecting a weighted score to a decision boundary
• Logistic regression, sigmoid, and binary cross-entropy
• Comparing implementations and choosing a classification threshold

Task	Description	Points
00	Prepare the witness records	0.25
01	Write and train a perceptron	2.00
02	Check the implementation and explain its limits	0.50
03	Write logistic predictions and loss	1.00
04	Compare thresholds on validation records	0.50
05	Evaluate and follow the mansion lead	0.75
06	Export the completed notebook	0
	Total	5.00

Run the cells in order and complete the written responses.

Story Progression

Caulfield has your report on the balloons and gas canisters. Quinn is still a lead, but the police need to look beyond one suspect's party supplies.

Fortunately, everyone at the mansion remembers something. Unfortunately, they do not remember the same something. A delivery, an argument, an envelope in the study—Caulfield has a stack of statements and time to follow only so many of them.

The department has records from earlier investigations. For each statement, officers checked some background details and interviewed the witness twice. Later, separate evidence established whether the main claim was corroborated. Can we learn from those earlier statements to help organize the new leads?

The intern suggests sorting by how confidently each witness spoke. Caulfield quietly takes the spreadsheet away from him.

Evidence 1: Caulfield compares the witnesses’ conflicting accounts.

Task 00: Prepare the Witness Records

Task 00-1: Description (0 pts.)

Task 00-1: Code (0 pts.)

import os
import shutil
import subprocess
from pathlib import Path

REPO_URL = 'https://github.com/wtheisen/nd-cse-30124-homeworks.git'

try:
    import google.colab
except ImportError:
    # Reuse a local checkout, including when launched from a homework subfolder.
    DATA_ROOT = next((p for p in [Path.cwd(), *Path.cwd().parents]
                      if (p / '.git').exists() and (p / 'evidence').is_dir()), None)
    if DATA_ROOT is None:
        DATA_ROOT = Path.cwd() / 'nd-cse-30124-homeworks'
else:
    # Leave the old checkout before deleting it, including on a setup-cell rerun.
    os.chdir('/content')
    DATA_ROOT = Path('/content/nd-cse-30124-homeworks')
    # Fetch fresh evidence every time this cell runs in Colab.
    if DATA_ROOT.exists():
        shutil.rmtree(DATA_ROOT)

if not DATA_ROOT.exists():
    subprocess.run(['git', 'clone', REPO_URL, str(DATA_ROOT)], check=True)

if not (DATA_ROOT / '.git').exists():
    raise RuntimeError(f'{DATA_ROOT} exists but is not a Git checkout. Choose a different clone location.')

# All later relative paths now start at the homework repository root.
os.chdir(DATA_ROOT)
DATA_ROOT = Path.cwd()

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import log_loss

EVIDENCE_DIR = DATA_ROOT / 'evidence' / 'homework03'
required_files = ['historical.csv', 'validation.csv', 'mansion_resolved.csv', 'mansion_leads.csv']
missing_files = [name for name in required_files if not (EVIDENCE_DIR / name).is_file()]
if missing_files:
    raise FileNotFoundError(
        f'HW03 evidence is missing: {missing_files}. '
        'In Colab, rerun setup after the evidence is released; locally, update this checkout with git pull --ff-only. '
    )

# Enter the evidence folder so the remaining cells can use simple filenames.
os.chdir(EVIDENCE_DIR)

FEATURES = ['details_matched', 'details_repeated']
historical_df = pd.read_csv('historical.csv')
validation_df = pd.read_csv('validation.csv')

print(historical_df.head().to_string(index=False))
print(f'Historical statements: {len(historical_df)}; validation statements: {len(validation_df)}')

Task 00-1: Expected Output (0 pts.)

statement_id  details_matched  details_repeated  corroborated
        H001                5                 5             0
        H002                5                 5             1
        H003                7                 7             1
        H004                0                 2             0
        H005                3                 9             1
Historical statements: 120; validation statements: 60

Task 00-2: Description (0 pts.)

[ME]: I've never heard the term "design matrix" before for the input with the column of 1s, where does that come from?

[ChatGPT]: Classical statistics—linear models, regression, and experimental design. The “design” refers to the experimental setup: the matrix encodes which treatments, conditions, or predictor values apply to each observation. It’s also common in econometrics. You’ll encounter it in statistically oriented ML treatments, but feature matrix or input matrix is more typical in modern ML/deep learning.

Build the input matrix in three steps:

1. Select the two feature columns in the order shown above.
2. Scale their counts by dividing by ten, so both range from zero to one.
3. Prepend a column of ones for the bias.

Python Help

features = df[['details_matched', 'details_repeated']].to_numpy(dtype=float)
X = np.column_stack((np.ones(len(features)), features / 10.0))

Task 00-2: Code (0.25 pts.)

def make_design(df):
    # TODO: Select both feature columns and convert the counts to fractions.
    features = None
    # TODO: Add the intercept column, as in HW02.
    return None

X_train = make_design(historical_df)
X_val = make_design(validation_df)

# TODO: Extract the 0/1 labels for both classifiers.
y_train = None
y_val = None

print('Design matrix:', X_train.shape, '| labels:', y_train.shape)
print('Example [8, 7] becomes:', make_design(pd.DataFrame([[8, 7]], columns=FEATURES))[0])

Task 00-2: Expected Output (0 pts.)

Design matrix: (120, 3) | labels: (120,)
Example [8, 7] becomes: [1.  0.8 0.7]

Figure 1. Both labels occur in the same region. Some points even share exactly the same two counts.

Task 01: Write and Train a Perceptron

Task 01-1: Description (0 pts.)

Return one 0/1 prediction per row. np.where(condition, value_if_true, value_if_false) is handy here. A score is not a probability: it can be negative or larger than one.

Task 01-1: Code (0.25 pts.)

def perceptron_predict(X, weights):
    # TODO: Compute scores and turn them into 0/1 predictions.
    scores = None

    return None

example_X = np.array([[1., 0., 0.], [1., .5, .5], [1., 1., 1.]])
example_w = np.array([-1., 1., 1.])

print('Scores:', example_X @ example_w)
print('Predictions:', perceptron_predict(example_X, example_w))

Task 01-1: Expected Output (0 pts.)

Scores: [-1.  0.  1.]
Predictions: [0 0 1]

Task 01-2: Description (0 pts.)

Your function should:

1. Copy the weights so the caller's original array is preserved.
2. Compute the prediction and error, then apply the update.
3. Return the updated weights and a Boolean recording whether an update happened.

Task 01-2: Code (0.5 pts.)

def perceptron_step(x, target, weights, learning_rate):
    new_weights = weights.copy()

    # TODO: Compute the 0/1 prediction and target-minus-prediction error.
    prediction = None
    error = None

    # TODO: Apply the perceptron learning rule, including the bias.
    new_weights += None

    return new_weights, bool(error != 0)

step_w, changed = perceptron_step(np.array([1., .8, .7]), 1, np.zeros(3), .5)

print('New weights:', step_w, '| updated:', changed)
print('New score:', round(np.array([1., .8, .7]) @ step_w, 3))

Task 01-2: Expected Output (0 pts.)

New weights: [0.5  0.4  0.35] | updated: True
New score: 1.065

Task 01-3: Description (0 pts.)

Before training: initialize the weights to zero and create an empty history.

For each epoch:

1. Reset the update counter to zero.
2. Visit the training rows in their given order.
3. Update the weights for each row by calling your update function with that row and its label. Keep the returned weights and increment the counter if an update happened.
4. Evaluate classification error across the training set using the final weights for this epoch.
5. Record the epoch number, update count, and classification error in the history.
6. Stop if the epoch made zero updates; otherwise begin the next epoch, up to max_epochs.

Training settings

• Learning rate: 1.0.
• Maximum epochs: 40.
• Row order: keep the given order; do not shuffle. A fixed order makes your implementation easier to check.

Return: the final weights and a list of history records containing epoch, updates, and error.

Python Help

for x, target in zip(X, targets):
    # x is one row, target is its matching label.
    pass
history.append({'epoch': epoch + 1, 'updates': updates, 'error': error})

Evidence 2: The intern adjusts the dividing line as new statements challenge the previous decision.

Task 01-3: Code (1.25 pts.)

def train_perceptron(X, targets, learning_rate=1.0, max_epochs=40):
    weights = np.zeros(X.shape[1], dtype=float)
    history = []

    for epoch in range(max_epochs):
        updates = 0

        for x, target in zip(X, targets):
            # TODO: Step through examples and labels, keeping each new weight vector.
            weights, changed = None

            updates += int(changed)

        # TODO: Measure classification error after the full pass.
        error = None

        history.append({'epoch': epoch + 1, 'updates': updates, 'error': float(error)})

        # TODO: Stop if a complete pass needed no updates. (~2 lines)
        pass

    return weights, history

perceptron_w, training_history = train_perceptron(X_train, y_train)
history_df = pd.DataFrame(training_history)

print(history_df.iloc[[0, 1, 4, -1]].to_string(index=False))
print('Final weights:', np.round(perceptron_w, 4))
print(f'Validation accuracy: {np.mean(perceptron_predict(X_val, perceptron_w) == y_val):.3f}')

Task 01-3: Reference Output (0 pts.)

epoch  updates    error
     1       50 0.233333
     2       35 0.341667
     5       30 0.183333
    40       36 0.266667
Final weights: [-3.   4.4  1.3]
Validation accuracy: 0.767

Figure 2. The update count and the final error describe different things. Neither is guaranteed to fall every epoch.

Task 02: Check the Perceptron

Task 02-1: Description (0 pts.)

Complete the call to your update function. The six supplied cases cover both mistakes, both correct predictions, and both labels at a score of exactly zero.

Task 02-1: Code (0.25 pts.)

# Supplied hand-calculated cases: (starting weights, target, expected weights, updated).
x_check = np.array([1., .8, .7])

check_cases = [
    ([-1., 0., 0.], 1, [-.5, .4, .35], True),   # missed positive
    ([1., 0., 0.], 0, [.5, -.4, -.35], True),   # false positive
    ([-1., 0., 0.], 0, [-1., 0., 0.], False),  # correct negative
    ([1., 0., 0.], 1, [1., 0., 0.], False),    # correct positive
    ([0., 0., 0.], 1, [.5, .4, .35], True),    # zero score, positive target
    ([0., 0., 0.], 0, [0., 0., 0.], False),   # zero score, negative target
]

for initial, target, expected, expected_changed in check_cases:
    original = np.array(initial)

    # TODO: Run one update for this example with learning rate 0.5.
    actual, changed = None
    np.testing.assert_allclose(actual, expected, atol=1e-12)
    assert changed == expected_changed
    np.testing.assert_array_equal(original, initial)

print('Six hand-calculated update checks passed.')

# Supplied check: these two points can be separated, so training should stop early.
toy_X = np.array([[1., 0.], [1., 1.]])
toy_y = np.array([0, 1])
toy_w, toy_history = train_perceptron(toy_X, toy_y, max_epochs=10)
np.testing.assert_array_equal(toy_w, [0., 1.])
np.testing.assert_array_equal(perceptron_predict(toy_X, toy_w), toy_y)
assert [row['updates'] for row in toy_history] == [1, 1, 0]
assert toy_history[-1]['error'] == 0.0
print('Training-loop and early-stopping checks passed.')

Task 02-1: Expected Output (0 pts.)

Six hand-calculated update checks passed.
Training-loop and early-stopping checks passed.

Task 02-2: Written Response (0.25 pts.)

1. Why does the positive example in Task 01-2 increase its own score after the update?

   • [ANSWER]

2. Would raising max_epochs guarantee zero training error on these witness records? Explain using the two identical inputs.

   • [ANSWER]

Story Progression

Your implementation passes the hand-calculated checks. Caulfield looks pleased until the intern points out that every statement still gets the same two possible answers: yes or no.

Two claims can both land on the positive side of the line even when one is barely across it. Could we get something more informative than a hard decision?

Evidence 3: The intern offers two answers. Caulfield asks for a little more nuance.

Task 03: Logistic Regression

Task 03-1: Description (0 pts.)

Implement the prediction in two steps:

1. Write sigmoid to transform scores into probabilities.
2. In predict_probability, compute X @ weights and pass those scores through your sigmoid function.

Return one probability per statement. We'll interpret it as the model's estimated probability of corroboration—not a measured probability that someone is telling the truth.

For numerical stability, the supplied split handles positive and negative scores separately. For negative $s$, the equivalent expression $e^s/(1+e^s)$ avoids computing a huge $e^{-s}$. Fill the nonnegative-score expression. The negative-score expression, masks, and allocation are supplied.

Task 03-1: Code (0.5 pts.)

def sigmoid(scores):
    scores = np.asarray(scores, dtype=float)
    result = np.empty_like(scores)
    positive = scores >= 0

    # TODO: Apply sigmoid to nonnegative scores.
    result[positive] = None

    exp_scores = np.exp(scores[~positive])

    result[~positive] = exp_scores / (1.0 + exp_scores)

    return result

def predict_probability(X, weights):
    # TODO: Pass the weighted score through your sigmoid function.
    return None

print('Sigmoid([-2, 0, 2]):', np.round(sigmoid(np.array([-2., 0., 2.])), 4))
print('Extreme-score check:', sigmoid(np.array([-1000., 1000.])))

Task 03-1: Expected Output (0 pts.)

Sigmoid([-2, 0, 2]): [0.1192 0.5    0.8808]
Extreme-score check: [0. 1.]

Figure 3. Sigmoid changes the output scale. It does not turn a linear boundary into a curved one.

Task 03-2: Description (0 pts.)

To implement the loss:

1. Clip probabilities to [1e-12, 1-1e-12] so rounding to zero or one does not cause log(0).
2. Compute both terms using np.log, the natural logarithm.
3. Take the negative mean across examples.

Task 03-2: Code (0.5 pts.)

def binary_cross_entropy(y, probabilities):
    p = np.clip(probabilities, 1e-12, 1.0 - 1e-12)

    # TODO: Compute and average both terms of binary cross-entropy.
    return None

print(f'Correct and confident: {binary_cross_entropy(np.array([1]), np.array([.9])):.3f}')
print(f'Wrong and confident:   {binary_cross_entropy(np.array([1]), np.array([.1])):.3f}')
print(f'Always 0.5:            {binary_cross_entropy(y_val, np.full(len(y_val), .5)):.3f}')

Task 03-2: Expected Output (0 pts.)

Correct and confident: 0.105
Wrong and confident:   2.303
Always 0.5:            0.693

Task 03-3: Description (0 pts.)

Create and fit the library model:

1. Construct a LogisticRegression model using all the settings below.
2. Fit it to X_train and y_train before accessing coef_[0].

Required settings:

• solver='lbfgs': an iterative optimizer. You do not need to implement this solver.
• C=np.inf: disables regularization in the supplied scikit-learn version, leaving binary cross-entropy as the objective.
• fit_intercept=False: the intercept is already included inside $X$.
• max_iter=2000 and tol=1e-10: allow sufficient iterations and use the specified stopping tolerance.

This model-fitting step is required setup and carries no separate points.

scikit-learn LogisticRegression documentation

Task 03-3: Code (0 pts.)

# TODO: Construct LogisticRegression with C=np.inf, fit_intercept=False,
# solver='lbfgs', max_iter=2000, and tol=1e-10, then fit it to X_train and y_train.
logistic_model = None

logistic_w = logistic_model.coef_[0]
p_val = predict_probability(X_val, logistic_w)

# Supplied checks for your probability and loss implementations.
np.testing.assert_allclose(
    p_val, logistic_model.predict_proba(X_val)[:, 1], atol=1e-10
)
np.testing.assert_allclose(
    binary_cross_entropy(y_val, p_val), log_loss(y_val, p_val), atol=1e-10
)

print('Probability and loss checks passed.')
print('Logistic weights:', np.round(logistic_w, 4))
print(f'Validation cross-entropy: {binary_cross_entropy(y_val, p_val):.3f}')
print(f'Validation accuracy at 0.5: {np.mean((p_val >= .5) == y_val):.3f}')

Task 03-3: Reference Output (0 pts.)

Probability and loss checks passed.
Logistic weights: [-5.4055  6.8752  4.6096]
Validation cross-entropy: 0.457
Validation accuracy at 0.5: 0.800

Figure 4. Both boundaries are straight lines. These points are validation records; circles were corroborated and crosses contradicted.

Task 04: Choose a Follow-up Threshold

Task 04-1: Description (0 pts.)

Implement the predictions and both error counts. The loop and selection are supplied.

Python Help

count = np.sum((predictions == 1) & (labels == 0))

Parentheses around each comparison are necessary when combining NumPy conditions with &.

Evidence 4: Caulfield weighs a long list of leads against a short list of available interviews.

Task 04-1: Code (0.5 pts.)

def threshold_report(y, probabilities, threshold):
    # TODO: Convert probabilities into 0/1 follow-up decisions.
    predictions = None

    # TODO: Count false positives and false negatives separately.
    fp = None
    fn = None

    return {'threshold': threshold, 'follow_ups': int(predictions.sum()),
            'false_positives': fp, 'false_negatives': fn, 'cost': 2*fn + fp,
            'accuracy': float(np.mean(predictions == y))}

threshold_table = pd.DataFrame([threshold_report(y_val, p_val, t) for t in [.3, .5, .7]])
selected = threshold_table.sort_values(['cost', 'threshold'], ascending=[True, False]).iloc[0]
selected_threshold = float(selected['threshold'])

print(threshold_table.to_string(index=False, float_format=lambda x: f'{x:.3f}'))
print(f'Frozen threshold: {selected_threshold:.1f}')

Task 04-1: Reference Output (0 pts.)

threshold  follow_ups  false_positives  false_negatives  cost  accuracy
     0.300          43               14                1    16     0.750
     0.500          34                8                4    16     0.800
     0.700          26                4                8    20     0.800
Frozen threshold: 0.5

Task 05: Return to the Mansion

Task 05-1: Description (0 pts.)

One Final Check, Then the New Leads

First, evaluate the resolved statements:

1. Open mansion_resolved.csv.
2. Evaluate both models, keeping every weight and the selected threshold fixed.
3. Compare classification accuracy.
4. For logistic regression, also report cross-entropy and the missed/extra follow-ups.

We do not need logistic regression to win every metric for probabilities to be useful.

Then, rank the unresolved leads:

1. Open mansion_leads.csv and compute an estimated probability for each lead.
2. Rank the leads by that probability.
3. Flag leads that meet the frozen follow-up threshold.

These leads have no known answers. The statement text and its ID are for Caulfield to read; neither is a model feature.

Complete the marked prediction lines. The loading and report formatting are supplied. A high rank tells us where the model would start an investigation, not how the investigation will end.

Task 05-1: Code (0.25 pts.)

mansion_df = pd.read_csv('mansion_resolved.csv')
X_mansion = make_design(mansion_df)
y_mansion = mansion_df['corroborated'].to_numpy(dtype=int)

# TODO: Evaluate the fixed perceptron and logistic model on the mansion set.
perceptron_mansion = None
p_mansion = None

print(f'Perceptron mansion accuracy: {np.mean(perceptron_mansion == y_mansion):.3f}')
print(f'Logistic mansion accuracy at 0.5: {np.mean((p_mansion >= .5) == y_mansion):.3f}')
print(f'Logistic mansion cross-entropy: {binary_cross_entropy(y_mansion, p_mansion):.3f}')
print('Frozen-threshold report:', threshold_report(y_mansion, p_mansion, selected_threshold))

leads_df = pd.read_csv('mansion_leads.csv')

# TODO: Score the unresolved leads with the same weights and threshold and find witness follow ups
leads_df['estimated_probability'] = None
leads_df['follow_up'] = None

ranked_leads = leads_df.sort_values(['estimated_probability', 'statement_id'], ascending=[False, True])

print(ranked_leads[['statement_id', 'estimated_probability', 'follow_up']].to_string(index=False, float_format=lambda x: f'{x:.3f}'))
print('Highest-ranked statement:', ranked_leads.iloc[0]['statement'])

Task 05-1: Reference Output (0 pts.)

Perceptron mansion accuracy: 0.750
Logistic mansion accuracy at 0.5: 0.683
Logistic mansion cross-entropy: 0.497
Frozen-threshold report: {'threshold': 0.5, 'follow_ups': 33, 'false_positives': 15, 'false_negatives': 4, 'cost': 23, 'accuracy': 0.6833333333333333}
statement_id  estimated_probability  follow_up
         L02                  0.993       True
         L04                  0.898       True
         L06                  0.736       True
         L05                  0.471      False
         L03                  0.469      False
         L01                  0.066      False
Highest-ranked statement: A catering assistant heard a guest addressed as Professor Dingler. He carried an envelope into the study and left it in the desk drawer.

Figure 5. L02 is the highest-ranked lead. The dashed line is the threshold selected on validation records.

Task 05-2: Written Response (0.50 pts.)

1. Compare thresholds 0.3 and 0.5 on the validation records. How many false positives and false negatives does each produce? Show why their costs tie under $\text{cost}=2\,\text{false negatives}+\text{false positives}$. Which threshold does our tie-breaking rule select, and what tradeoff does that choice make?

   • [ANSWER]

2. Report both models' mansion accuracies at their stated thresholds and logistic cross-entropy. Why would changing settings after seeing those results spoil the held-out check?

   • [ANSWER]

3. Which unresolved lead ranked first? Explain what the probability estimates and name one piece of new evidence Caulfield should seek.

   • [ANSWER]

4. Does sigmoid give logistic regression a nonlinear decision boundary in our two original features? Explain.

   • [ANSWER]

Story Progression

The highest-ranked statement gives Caulfield a name: Professor Dingler.

The catering assistant remembers Dingler carrying an envelope into the study and leaving it in a desk drawer. Caulfield follows the lead and finds the envelope there. Inside are several letters. Now we have a physical piece of evidence to examine, rather than another score in a spreadsheet.

Evidence 5: The lead brings Caulfield to an envelope in the study drawer.

The forensics team scans the pages. The intern volunteers to organize the image files.

By the time Caulfield asks for them, the scans have been scrambled and their colors are wrong. The pixels are still there. The readable letters are not.

That is where HW04 begins: recovering the damaged letter images associated with Professor Dingler. For now, file your witness report—and perhaps keep the intern away from the originals.

Evidence 6: The original letters are safe. The intern’s digital copies need some work.

Task 06: Export the Completed Notebook

Task 06-1: Description (0 pts.)

1. Complete and run your code, and include your name, NETID, and written responses.
2. Save the notebook. For local execution, make sure NOTEBOOK_NAME matches your saved filename in homeworks/homework03/.
3. Set GENERATE = True below and run the export cell.
4. Open the HTML file and check that your answers and outputs are present before submitting it.

In Colab, the cell captures the current notebook and downloads the HTML submission file. Locally, it reads the saved notebook and writes the HTML beside it. Exporting includes existing outputs; it does not rerun your solutions.

Task 06-1: Code (0 pts.)

GENERATE = False
NOTEBOOK_NAME = 'homework03.ipynb'

if GENERATE:
    import json
    from pathlib import Path
    import nbformat
    from nbconvert import HTMLExporter

    try:
        from google.colab import _message, files
    except ImportError:
        notebook_path = DATA_ROOT / 'homeworks/homework03' / NOTEBOOK_NAME
        current = nbformat.read(notebook_path, as_version=4)
        in_colab = False
    else:
        notebook_path = Path('/content') / NOTEBOOK_NAME
        snapshot = _message.blocking_request('get_ipynb', timeout_sec=30)
        current = nbformat.reads(json.dumps(snapshot['ipynb']), as_version=4)
        nbformat.write(current, notebook_path)
        in_colab = True

    # Figures use the course's hosted image URLs.
    notebook_folder = DATA_ROOT / 'homeworks/homework03'
    exporter = HTMLExporter()
    exporter.embed_images = False
    html, _ = exporter.from_notebook_node(
        current, resources={'metadata': {'path': str(notebook_folder)}}
    )
    html_path = notebook_path.with_suffix('.html')
    html_path.write_text(html, encoding='utf-8')
    print(f'Exported {html_path.name}')
    if in_colab:
        files.download(str(html_path))
    else:
        print(f'Saved to: {html_path}')
else:
    print('Export disabled. Set GENERATE = True when ready.')

Task 06-1: Expected Output (0 pts.)

Export disabled. Set GENERATE = True when ready.

With GENERATE = True, the cell reports the exported filename and, in Colab, starts the download.