Introduction to Artificial Intelligence - Homework Assignment 02 (5 pts.)¶

  • Name:
  • NETID:

This assignment covers the following topics:

  • Writing a linear model and finding its weights with ordinary least squares
  • Fitting polynomial models with four different degrees
  • Writing Ridge regression and choosing a model using validation data
  • Applying the chosen model to held-out records from the mansion
Task ID Description Points
00 Load the historical records 0.25
01 Write prediction and loss functions 0.5
02 Find the straight-line weights with OLS 1
03 Check against scikit-learn 0.25
04 Write and compare four polynomial models 0.75
05 Write Ridge regression and choose the best model 1.25
06 Evaluate on the held-out mansion set and explain the result 1
07 Export the completed notebook 0
Total 5

Please complete all sections, including written answers, and run cells in order.

Story Progression¶

Thanks to your work in HW01, Quinn Hynes is the leading match for Mr. Green. He was seen inflating balloons in the kitchen and pantry and mentioned that canisters were stored there. Caulfield would like something a little more convincing than “our classifier said so,” so let's follow up on the balloons.

The police found records of the party preparations, including the balloons filled and the weight of a gas canister before and afterward. The canister got lighter, as you would expect. But did it lose more gas than the balloons should have used?

Apparently our next contribution to the murder investigation is a model of party supplies. We'll learn from earlier filling jobs, check how well we can predict new ones, and then see what the mansion's records tell us.

Evidence 1: Caulfield discovers that the next lead involves party balloons.
Evidence 1: Caulfield discovers that the next lead involves party balloons.

Task 00: Prepare Historical Evidence¶

Task 00-1: Description (0 pts.)¶

Loading the party records. Input balloon_volume_liters, target mass_loss_g, trial_id left out. Three sets: historical (train, 12 jobs) to find weights, validation (12 jobs) to choose a model, mansion (test, 13 jobs) opened once in Task 06.

Your Task¶

The data loading is done for you below, much like HW01. All three files download together, but leave the mansion file closed until Task 06.

Python Help¶

df = pd.read_csv('historical.csv')
print(df.head())

Useful Links¶

  • pandas.read_csv

Task 00-1: Code (0 pts.)¶

In [ ]:
import os
import shutil
import subprocess
from pathlib import Path

REPO_URL = 'https://github.com/wtheisen/nd-cse-30124-homeworks.git'

try:
    import google.colab
except ImportError:
    # Reuse a local checkout, including when launched from a homework subfolder.
    DATA_ROOT = next((p for p in [Path.cwd(), *Path.cwd().parents]
                      if (p / '.git').exists() and (p / 'evidence').is_dir()), None)
    if DATA_ROOT is None:
        DATA_ROOT = Path.cwd() / 'nd-cse-30124-homeworks'
else:
    # Leave the old checkout before deleting it, including on a setup-cell rerun.
    os.chdir('/content')
    DATA_ROOT = Path('/content/nd-cse-30124-homeworks')
    # Fetch fresh evidence every time this cell runs in Colab.
    if DATA_ROOT.exists():
        shutil.rmtree(DATA_ROOT)

if not DATA_ROOT.exists():
    subprocess.run(['git', 'clone', REPO_URL, str(DATA_ROOT)], check=True)

if not (DATA_ROOT / '.git').exists():
    raise RuntimeError(f'{DATA_ROOT} exists but is not a Git checkout. Choose a different clone location.')

# All later relative paths now start at the homework repository root.
os.chdir(DATA_ROOT)
DATA_ROOT = Path.cwd()

EVIDENCE_DIR = DATA_ROOT / 'evidence' / 'homework02'

if not (EVIDENCE_DIR / 'mansion.csv').exists():
    raise FileNotFoundError(
        f'HW02 evidence is missing from {DATA_ROOT}. Rerun setup in Colab after the corrected evidence is published; locally, update this checkout with git pull --ff-only.'
    )

# Enter the evidence folder so the remaining cells can use simple filenames.
os.chdir(EVIDENCE_DIR)

import numpy as np
import pandas as pd

historical_df = pd.read_csv('historical.csv')
quinn_statements = pd.read_csv('hw01_quinn_statements.csv')

FEATURE_COLUMN, TARGET_COLUMN = 'balloon_volume_liters', 'mass_loss_g'
TARGET_UNIT = 'grams'
FEATURE_LOW, FEATURE_HIGH = 20., 96.

def extract_feature(df):
    return df[FEATURE_COLUMN].to_numpy(dtype=float)

print(historical_df.head().to_string(index=False))
print(f'Historical jobs: {len(historical_df)}')

Task 00-1: Expected Output (0 pts.)¶

trial_id  balloon_volume_liters  mass_loss_g
     H01                   20.0     3.114080
     H02                   24.0     4.028029
     H03                   28.0     4.494672
     H04                   32.0     6.266059
     H05                   36.0     4.536902
Historical jobs: 12

Note: The plot below is not part of the expected output, it's just here to help you visualize the data.

Task 00-1 reference figure
The historical jobs: larger balloon volumes generally use more gas, but the points don’t fall perfectly on a line.

Task 00-2: Description (0 pts.)¶

Choosing Our Input and Output¶

First, put the balloon volumes in x_train and the measured weight losses in y_train. Use FEATURE_COLUMN for the input and TARGET_COLUMN for the target, converting both to a NumPy array. Both should have shape (12,), one number per job.

Keep the rows in the same order so each volume still goes with the correct weight. Then look at the supplied output: there's quite a gap between some of our measurements. Keep that in mind when we start drawing curves through them.

Evidence 2: The intern gets the glamorous assignment of reviewing the party records.
Evidence 2: The intern gets the glamorous assignment of reviewing the party records.

Python Help¶

y = df['target'].to_numpy(dtype=float)

Useful Links¶

  • Series.to_numpy()

Task 00-2: Code (0.25 pts.)¶

In [ ]:
#TODO: Load the feature and target columns and convert to numpy floats
x_train = None
y_train = None

print(f'Feature shape: {x_train.shape}; target shape: {y_train.shape}')
print(f'Feature range: {x_train.min():.4f} to {x_train.max():.4f}')

Task 00-2: Expected Output (0 pts.)¶

Feature shape: (12,); target shape: (12,)
Feature range: 20.0000 to 96.0000

Task 01: Implement Prediction and Loss¶

Task 01-1: Description (0 pts.)¶

Predictions are y-hat = X w. A row [1, x] times weights [w0, w1] gives w0 + w1 x; the column of ones carries the intercept. X (12, 2) @ w (2,) gives (12,). Example: features [2, 4], weights [1, 3] give predictions [7, 13]. @ is matrix multiplication; the star operator multiplies element by element.

Your Task¶

Complete create_input by adding that column of ones. Then complete predict_linear using matrix multiplication. In NumPy, @ does matrix multiplication; * multiplies matching entries instead.

Python Help¶

features = np.array([[2.], [4.]])
ones = np.ones(features.shape[0])
X_example = np.column_stack((ones, features))
# [[1., 2.], [1., 4.]]
# @ performs matrix multiplication; * multiplies element by element.

Useful Links¶

  • numpy.column_stack
  • numpy.matmul

Task 01-1: Code (0 pts.)¶

In [ ]:
def create_input(features):
    features = np.asarray(features, dtype=float)

    # TODO: Add a column of 1s to our matrix to represent the y-intercept
    return None

def predict_linear(X, weights):
    # TODO: Perform a matrix multiplication to create our predictions
    return None

example = predict_linear(create_input(np.array([2., 4.])), np.array([1., 3.]))

print(f'Example predictions: {example}')

Task 01-1: Expected Output (0 pts.)¶

Example predictions: [ 7. 13.]

Task 01-2: Description (0 pts.)¶

Residual = measured minus predicted. MSE = (1/n) sum of (y_i - y-hat_i) squared; square before averaging. Residuals [1, -2, 2] give MSE 3.0. RMSE is the square root of MSE, in grams.

Your Task¶

Complete mean_squared_error and check the examples below. Be careful to square the individual errors before averaging them!

Python Help¶

errors = np.array([1., -2., 2.])
print(np.mean(errors**2))  # 3.0

Useful Links¶

  • numpy.mean()

Task 01-2: Code (0.5 pts.)¶

In [ ]:
def mean_squared_error(y_true, y_pred):
    # TODO: Calculate MSE
    residuals = None

    return None

Task 02: Fit the Historical Straight-Line Baseline¶

Task 02-1: Description (0 pts.)¶

OLS: minimize ||Xw - y||^2; setting the gradient 2 X^T (Xw - y) to zero gives X^T X w = X^T y, so w-hat = (X^T X)^-1 X^T y. Compute X^T, X^T X, X^T y, then np.linalg.solve. Intercept first, slope second.

Your Task¶

Complete fit_ols by calculating the transpose, the two matrix products, and the weights. We'll use np.linalg.solve for the last step. It solves the same equation without making us explicitly calculate the inverse. The small example below shows how to use it.

The returned array should contain the intercept first and the slope second. Use your prediction function to see how the fitted line does on the historical jobs. The figure below shows the line and the mistakes it still makes. Is there a pattern in those mistakes?

Python Help¶

A = np.array([[2., 0.], [0., 4.]])
c = np.array([6., 8.])
z = np.linalg.solve(A, c)  # [3., 2.], satisfying A @ z == c
transpose = A.T

Useful Links¶

  • numpy.linalg.solve

Task 02-1: Code (1 pts.)¶

In [ ]:
def fit_ols(X, y):
    # TODO: Calculate the weights via OLS

    return weights

# TODO: generate predictions for X_train with the fit model
X_train = None
linear_weights = None
train_predictions = None

line_train_rmse = np.sqrt(mean_squared_error(y_train, train_predictions))

print(f'Input shape: {X_train.shape}; weight shape: {linear_weights.shape}')
print(f'Intercept w0: {linear_weights[0]:.6f}; slope w1: {linear_weights[1]:.6f}')
print(f'Straight-line historical RMSE: {line_train_rmse:.4f} grams')

Task 02-1: Expected Output (0 pts.)¶

Input shape: (12, 2); weight shape: (2,)
Intercept w0: -3.677931; slope w1: 0.287778
Straight-line historical RMSE: 1.0941 grams

Note: This plot below isn't part of your expected output, it's just here to help you visualize what we're doing.

Task 02-1 reference figure
Our OLS line and the errors it leaves behind. Notice the curved pattern in the residuals.

Task 03: Verify the Implementation¶

Task 03-1: Description (0 pts.)¶

Check against scikit-learn LinearRegression: give it x_train.reshape(-1, 1) without the ones column; coef_[0] is the slope and intercept_ is the intercept. Weights, predictions and MSE should match.

Your Task¶

Fit LinearRegression on the same historical jobs and check its weights, predictions, and MSE against yours. Remember that scikit-learn supplies its own intercept, so don't give it our column of ones. Use x_train.reshape(-1, 1) as its input. Its coef_[0] is the slope and intercept_ is the intercept; the supplied checks take care of rounding. Get all four checks passing before moving on.

Python Help¶

from sklearn.linear_model import LinearRegression
reference = LinearRegression().fit(x_train.reshape(-1, 1), y_train)
predictions = reference.predict(x_train.reshape(-1, 1))

Useful Links¶

  • LinearRegression

Task 03-1: Code (0.25 pts.)¶

In [ ]:
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error as sklearn_mse

# TODO: Use scikit learn to produce predictions on the same data
reference_model = None
reference_predictions = None

checks = {
    'Slope': np.isclose(linear_weights[1], reference_model.coef_[0]),
    'Intercept': np.isclose(linear_weights[0], reference_model.intercept_),
    'Predictions': np.allclose(train_predictions, reference_predictions),
    'MSE': np.isclose(mean_squared_error(y_train, train_predictions), sklearn_mse(y_train, reference_predictions)),
}

for label, value in checks.items(): print(f'{label} matches: {bool(value)}')

Task 03-1: Expected Output (0 pts.)¶

Slope matches: True
Intercept matches: True
Predictions matches: True
MSE matches: True

Story Progression¶

Our line agrees with scikit-learn! Now let's see whether a curve can do a better job. We will try four degrees on the same historical records, then compare their predictions on the validation jobs.

Evidence 3: The intern proposes trying a few more curves. Caulfield requests numbers to go with them.
Evidence 3: The intern proposes trying a few more curves. Caulfield requests numbers to go with them.

Task 04: Write and Compare Polynomial Models¶

Task 04-1: Description (0 pts.)¶

Polynomial models add powers of x as columns: degree d has d + 1 weights including the intercept. Degrees 2, 3, 5 and 9; degree 9 has ten weights for twelve jobs. Fit with np.linalg.lstsq; input scaling is supplied.

Your Task¶

Complete polynomial_features so that degree 3 produces columns $u$, $u^2$, and $u^3$. Leave out the ones column; our create_input function adds it later. Then complete fit_polynomial to find the weights using np.linalg.lstsq(X, y, rcond=None)[0].

Python Help¶

powers = [u**power for power in range(1, degree + 1)]
columns = np.column_stack(powers)

Task 04-1: Code (0.5 pts.)¶

In [ ]:
from sklearn.preprocessing import StandardScaler

def polynomial_features(u, degree):
    # TODO: Compute the polynomial features to add to our input matrix
    return None

def fit_polynomial(X, y):
    # TODO: Compute the weights using np.linalg.lstsq()
    return None

# GIVEN: scale each feature once using historical data only.
def domain_coordinate(x):
    return 2 * (np.asarray(x) - FEATURE_LOW) / (FEATURE_HIGH - FEATURE_LOW) - 1

degrees = [2, 3, 5, 9]
scalers, training_matrices = {}, {}
for degree in degrees:
    raw = polynomial_features(domain_coordinate(x_train), degree)
    scaler = StandardScaler().fit(raw)
    scalers[degree] = scaler
    training_matrices[degree] = create_input(scaler.transform(raw))

def matrix_for(x, degree):
    raw = polynomial_features(domain_coordinate(x), degree)
    return create_input(scalers[degree].transform(raw))

def rmse(y_true, predictions):
    return np.sqrt(mean_squared_error(y_true, predictions))

# A single collection will hold the line, curves, and later Ridge models.
candidates = [{'name': 'straight line', 'degree': 1, 'alpha': 0.,
               'weights': linear_weights}]

for degree in degrees:
    # TODO: find the weights for each degree
    weights = None

    candidates.append({'name': f'degree {degree}', 'degree': degree,
                       'alpha': 0., 'weights': weights})

def predict_candidate(candidate, x):
    X = create_input(x) if candidate['degree']==1 else matrix_for(x, candidate['degree'])

    return predict_linear(X, candidate['weights'])

for candidate in candidates:
    error = rmse(y_train, predict_candidate(candidate, x_train))

    print(f"{candidate['name']}: historical RMSE = {error:.4f} grams")

Task 04-1: Expected Output (0 pts.)¶

straight line: historical RMSE = 1.0941 grams
degree 2: historical RMSE = 0.8472 grams
degree 3: historical RMSE = 0.8075 grams
degree 5: historical RMSE = 0.6787 grams
degree 9: historical RMSE = 0.5890 grams

NOTE: The plots below aren't part of your expected output, but they ARE the actual curves you're fitting

Four degrees fitted to the same historical records. A closer fit is not necessarily a better predictor.
Four degrees fitted to the same historical records. A closer fit is not necessarily a better predictor.

Task 04-2: Description (0 pts.)¶

Validation jobs were not used to fit any model. Historical RMSE falls with degree (1.0941, 0.8472, 0.8075, 0.6787, 0.5890) but validation RMSE does not (1.3153, 1.0508, 1.1031, 1.4186, 1.3416). Fitting old jobs closely but predicting new ones poorly is overfitting.

Your Task¶

Complete the two error calculations below. The resulting table compares the line and all four curves on historical and validation jobs. Lower RMSE means better predictions.

Python Help¶

error = rmse(actual, predictions)

Task 04-2: Code (0.25 pts.)¶

In [ ]:
validation_df = pd.read_csv('case_validation.csv')

# TODO: Read in the val data and convert to numpy float
x_validation = None
y_validation = None

def compare_candidates(candidates):
    rows = []
    for candidate in candidates:
        # TODO: Calculate the rmse for the train and the val datasets
        historical_error = None
        validation_error = None

        rows.append({'model': candidate['name'], 'degree': candidate['degree'],
                     'alpha': candidate['alpha'], 'historical_rmse': historical_error,
                     'validation_rmse': validation_error})

    return pd.DataFrame(rows)

plain_comparison = compare_candidates(candidates)
print(plain_comparison.to_string(index=False, float_format=lambda v: f'{v:.4f}'))

Task 04-2: Reference Output (0 pts.)¶

model  degree  alpha  historical_rmse  validation_rmse
straight line       1 0.0000           1.0941           1.3153
     degree 2       2 0.0000           0.8472           1.0508
     degree 3       3 0.0000           0.8075           1.1031
     degree 5       5 0.0000           0.6787           1.4186
     degree 9       9 0.0000           0.5890           1.3416

NOTE: The plot below is not part of your expected output but is a nice visualization of the above comparison

The validation jobs reveal which curves can predict beyond their training examples.
The validation jobs reveal which curves can predict beyond their training examples.

Story Progression¶

The intern points out that the degree-nine curve fits the old records beautifully. Caulfield points to its predictions on new jobs. We need a way to stop a flexible model from chasing every bump in the old measurements.

Evidence 4: Caulfield asks how the models performed on jobs they had not seen before.
Evidence 4: Caulfield asks how the models performed on jobs they had not seen before.

Task 05: Write Ridge Regression and Choose a Model¶

Task 05-1: Description (0 pts.)¶

Ridge: L(w) = ||Xw - y||^2 + alpha sum over j from 1 to d of w_j^2, leaving the intercept unpenalized. Solve (X^T X + alpha P) w = X^T y, where P is the identity with its first entry set to 0.

Your Task¶

Complete fit_ridge by making that penalty matrix, forming the left and right sides, and solving for the weights. This function actually fits the model; it does not just calculate a score. The supplied check compares your degree-nine fit with scikit-learn.

Python Help¶

P = np.eye(4)  # one diagonal entry per weight
P[0, 0] = 0   # do not penalize the intercept
weights = np.linalg.solve(A, b)

Task 05-1: Code (0.75 pts.)¶

In [ ]:
def fit_ridge(X, y, alpha):
    # TODO: Calculate the weights for a ridge regression
    penalty = None
    penalty[0, 0] = 0

    left = None
    right = None

    return np.linalg.solve(left, right)

# Supplied independent check. The library adds its own intercept.
from sklearn.linear_model import Ridge

X_example = training_matrices[9]
example_weights = fit_ridge(X_example, y_train, alpha=1.)
reference_ridge = Ridge(alpha=1., solver='svd').fit(X_example[:, 1:], y_train)

print('Weights match scikit-learn:', np.allclose(
    example_weights, np.r_[reference_ridge.intercept_, reference_ridge.coef_]))

print('Predictions match scikit-learn:', np.allclose(
    predict_linear(X_example, example_weights), reference_ridge.predict(X_example[:, 1:])))

Task 05-1: Expected Output (0 pts.)¶

Weights match scikit-learn: True
Predictions match scikit-learn: True

Task 05-2: Description (0 pts.)¶

Fit degrees 2, 3, 5 and 9 with alpha 0.001, 0.01, 0.1, 1, 10 and 100: 24 Ridge fits plus 5 earlier models, 29 candidates. Choose the lowest validation RMSE; not historical error and no penalty added. Then keep the chosen model fixed.

Your Task¶

Use your Ridge function to fit each of the four degrees with alpha values of .001, .01, .1, 1, 10, and 100. Every fit uses the historical jobs. The supplied loop adds those models to the same collection as our line and unregularized curves.

We want to add our results to the candidates dict with their: {name, degree, alpha, and weights}.

Now compare all the candidates using validation RMSE and select the row with the lowest error. Store its model name and retrieve that candidate.

Python Help¶

best_row = comparison.loc[comparison['validation_rmse'].idxmin()]
name = best_row['model']

Task 05-2: Code (0.5 pts.)¶

In [ ]:
alphas = [.001, .01, .1, 1., 10., 100.]

# Re-running this cell should replace, rather than duplicate, Ridge candidates.
candidates = [candidate for candidate in candidates if candidate['alpha']==0]

for degree in degrees:
    for alpha in alphas:
        # TODO: Find the weights for our ridge regression
        weights = None

        # TODO: Add our candidate weights to our candidate list
        candidates.append(None)

comparison = compare_candidates(candidates)

# TODO: Get the best model out of the comparison results
best_row = None
selected_name = None

selected_model = next(candidate for candidate in candidates if candidate['name']==selected_name)

def predict_selected(x):
    return predict_candidate(selected_model, x)

print(comparison.sort_values('validation_rmse', kind='stable').to_string(
    index=False, float_format=lambda v: f'{v:.4f}'))

print(f'Chosen model: {selected_name}')

Task 05-2: Expected Output (0 pts.)¶

model  degree    alpha  historical_rmse  validation_rmse
    degree 2 Ridge (alpha=1)       2   1.0000           1.0459           0.6922
    degree 5 Ridge (alpha=1)       5   1.0000           1.2359           0.8551
    degree 3 Ridge (alpha=1)       3   1.0000           1.2798           0.8643
    degree 9 Ridge (alpha=1)       9   1.0000           1.1521           0.9054
  degree 2 Ridge (alpha=0.1)       2   0.1000           0.8497           1.0055
  degree 3 Ridge (alpha=0.1)       3   0.1000           0.8294           1.0161
 degree 2 Ridge (alpha=0.01)       2   0.0100           0.8472           1.0462
degree 2 Ridge (alpha=0.001)       2   0.0010           0.8472           1.0504
                    degree 2       2   0.0000           0.8472           1.0508
 degree 3 Ridge (alpha=0.01)       3   0.0100           0.8078           1.0927
degree 3 Ridge (alpha=0.001)       3   0.0010           0.8075           1.1020
                    degree 3       3   0.0000           0.8075           1.1031
  degree 5 Ridge (alpha=0.1)       5   0.1000           0.7123           1.1697
  degree 9 Ridge (alpha=0.1)       9   0.1000           0.7251           1.1793
               straight line       1   0.0000           1.0941           1.3153
                    degree 9       9   0.0000           0.5890           1.3416
 degree 5 Ridge (alpha=0.01)       5   0.0100           0.6799           1.3716
 degree 9 Ridge (alpha=0.01)       9   0.0100           0.6562           1.4034
degree 5 Ridge (alpha=0.001)       5   0.0010           0.6787           1.4131
                    degree 5       5   0.0000           0.6787           1.4186
degree 9 Ridge (alpha=0.001)       9   0.0010           0.6414           1.4419
   degree 3 Ridge (alpha=10)       3  10.0000           2.9167           1.6994
   degree 5 Ridge (alpha=10)       5  10.0000           2.9197           1.7225
   degree 9 Ridge (alpha=10)       9  10.0000           3.0438           1.8173
   degree 2 Ridge (alpha=10)       2  10.0000           3.6731           2.1761
  degree 9 Ridge (alpha=100)       9 100.0000           5.7772           3.9457
  degree 5 Ridge (alpha=100)       5 100.0000           6.1180           4.1737
  degree 3 Ridge (alpha=100)       3 100.0000           6.4558           4.4204
  degree 2 Ridge (alpha=100)       2 100.0000           7.0095           4.8394
Chosen model: degree 2 Ridge (alpha=1)

NOTE: Again the plot below is not part of your expected output.

Validation error chooses the winner. More regularization does not always mean better predictions.
Validation error chooses the winner. More regularization does not always mean better predictions.

Story Progression¶

We have a winner. Now Caulfield hands over the records from the mansion. No more trying different settings: it is time to see what our chosen model predicts.

Task 06: Open the Held-Out Mansion Set¶

Task 06-1: Description (0 pts.)¶

The mansion set has 13 jobs kept out of fitting and selection. SCENE01 covers the kitchen and pantry canister from 19:55 to 20:45. Difference = measured minus predicted; positive means more gas than expected. Compare with the unregularized degree-nine model, and keep the chosen model fixed.

Your Task¶

Predict the mansion jobs' gas use with your chosen model and calculate its RMSE. The supplied code identifies SCENE01; calculate its measured gas use minus your prediction.

Python Help¶

predictions = predict_selected(volumes)
error = rmse(measured, predictions)
difference = measured - predicted

Task 06-1: Code (0.5 pts.)¶

In [ ]:
# TODO: Load the mansion.csv file
mansion_df = None
x_mansion = None
y_mansion = None

# TODO: Create predictions on our mansion (test) set
mansion_predictions = None
mansion_rmse = None

mansion_results = mansion_df.copy()
mansion_results['predicted_mass_loss_g'] = mansion_predictions

scene_index = mansion_df.index[mansion_df['trial_id']=='SCENE01'][0]
scene_measured = y_mansion[scene_index]
scene_prediction = mansion_predictions[scene_index]
scene_difference = scene_measured - scene_prediction

plain_nine = next(c for c in candidates if c['degree']==9 and c['alpha']==0)
plain_scene = predict_candidate(plain_nine, x_mansion[[scene_index]])[0]

print(mansion_results.to_string(index=False, float_format=lambda v: f'{v:.4f}'))
print(f'Chosen model: {selected_name}')
print(f'Mansion RMSE: {mansion_rmse:.4f} grams')
print(f"Quinn's preparation window: measured {scene_measured:.3f} g; predicted {scene_prediction:.3f} g")
print(f'Chosen-model unexplained difference: {scene_difference:.3f} g')
print(f'Degree-nine model without Ridge: predicted {plain_scene:.3f} g; difference {scene_measured-plain_scene:.3f} g')

Task 06-1: Expected Output (0 pts.)¶

trial_id  balloon_volume_liters  mass_loss_g  predicted_mass_loss_g
     R01                23.0000       5.4765                 4.2235
     R02                34.0000       6.9081                 6.3305
     R03                43.0000       8.5036                 8.2875
     R04                47.0000       9.4346                 9.2246
     R05                51.0000       9.5863                10.2032
     R06                55.0000      11.6934                11.2232
     R07                59.0000      11.8405                12.2846
     R08                63.0000      12.4135                13.3874
     R09                74.0000      16.4033                16.6339
     R10                80.0000      18.9101                18.5368
     R11                86.0000      20.3305                20.5330
     R12                93.0000      23.7009                22.9796
 SCENE01                54.0000      11.0120                10.9643
Chosen model: degree 2 Ridge (alpha=1)
Mansion RMSE: 0.5882 grams
Quinn's preparation window: measured 11.012 g; predicted 10.964 g
Chosen-model unexplained difference: 0.048 g
Degree-nine model without Ridge: predicted 12.301 g; difference -1.289 g

NOTE: The plot below still isn't part of your expected output.

The chosen model meets the held-out mansion records. The marked job covers Quinn’s preparation window.
The chosen model meets the held-out mansion records. The marked job covers Quinn’s preparation window.

Task 06-2: Short Answer Questions (0.5 pts.)¶

What Should We Tell Caulfield?¶

Use a few sentences for each answer.

  1. Which unregularized degree fits the historical jobs most closely? Does it also have the lowest validation error? Explain what this tells you about overfitting.

    • [ANSWER]
  2. Which model won the full validation comparison? Explain what Ridge changed and why we did not simply choose the lowest historical error.

    • [ANSWER]
  3. Report the winner's overall mansion RMSE and the residual for SCENE01. Compare that difference with the unregularized degree-nine prediction. Why must we keep our chosen model fixed after opening this set?

    • [ANSWER]
  4. Does ordinary balloon preparation seem to explain the gas use during Quinn's preparation window? What can these records tell us, and what additional evidence would you want before drawing a conclusion about Quinn?

    • [ANSWER]

Story Progression¶

Time to tell Caulfield what we found!

Evidence 5: Back at the mansion, Caulfield reads the report before drawing any conclusions.
Evidence 5: Back at the mansion, Caulfield reads the report before drawing any conclusions.

Task 07: Export the Completed Notebook¶

Task 07-1: Description (0 pts.)¶

Sending in Your Report¶

Run the notebook from the beginning, save your work, and set GENERATE = True below to export your report. Make sure your written answers are included too! Colab will download the HTML; locally, run the export cell from the folder containing your saved notebook.

Task 07-1: Code (0 pts.)¶

In [ ]:
GENERATE = False

NOTEBOOK_NAME = 'homework02.ipynb'

if GENERATE:
    import json
    from pathlib import Path
    import nbformat
    from nbconvert import HTMLExporter
    try:
        from google.colab import _message, files
    except ImportError:
        notebook_path = Path(NOTEBOOK_NAME)
        current = nbformat.read(notebook_path, as_version=4)
        in_colab = False
    else:
        notebook_path = Path('/content') / NOTEBOOK_NAME
        current = nbformat.reads(json.dumps(_message.blocking_request('get_ipynb', timeout_sec=10)['ipynb']), as_version=4)
        nbformat.write(current, notebook_path)
        in_colab = True
    html, _ = HTMLExporter().from_notebook_node(current)
    html_path = notebook_path.with_suffix('.html')
    html_path.write_text(html, encoding='utf-8')
    print(f'Exported {html_path.name}')
    if in_colab: files.download(str(html_path))
else:
    print('Export disabled. Set GENERATE = True when ready.')

Task 07-1: Reference Output (0 pts.)¶

Export disabled. Set GENERATE = True when ready.