Skip to content

Commit 0849637

Browse files
caalocaalo
committed
update
1 parent 0af919b commit 0849637

11 files changed

Lines changed: 779 additions & 102 deletions

‎01-Problem-Setup scrapped.qmd‎

Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,184 @@
1+
# Problem Set-Up
2+
3+
*In the first week of class, we will go over what machine learning models are good for, what kind of machine learning models are out there and how does one evaluate and pick models, all at a conceptual level. Then, we will get our hands on the NumPy package to prepare our data for our models for the rest of the course.*
4+
5+
Suppose that we are given the [**N**ational **H**ealth **A**nd **N**utrition **E**xamination **S**urvey (NHANES) dataset](https://www.cdc.gov/nchs/nhanes/) and want to build a machine learning model to classify whether a person has hypertension blood pressure based on clinical and demographic variables.
6+
7+
Using algebraic expressions, we formulate the following:
8+
9+
$$
10+
Hypertension=f(Age, BMI, Income)
11+
$$
12+
13+
Where $f(Age, BMI, Income, ...)$ is a machine learning model that takes in the clinical and demographic variables and make a classification on whether someone has $Hypertension$.
14+
15+
```{python}
16+
import pandas as pd
17+
import seaborn as sns
18+
import numpy as np
19+
from sklearn.model_selection import train_test_split
20+
import matplotlib.pyplot as plt
21+
from formulaic import model_matrix
22+
import statsmodels.api as sm
23+
24+
nhanes = pd.read_csv("classroom_data/NHANES.csv")
25+
nhanes['Hypertension'] = (nhanes['BPDiaAve'] > 80) | (nhanes['BPSysAve'] > 130)
26+
27+
y, X = model_matrix("Hypertension ~ BMI", nhanes)
28+
logit_model = sm.Logit(y, X).fit()
29+
30+
plt.clf()
31+
plt.scatter(X.BMI, logit_model.predict(), color="blue", label="Fitted Line")
32+
plt.scatter(X.BMI, y, alpha=.3, color="brown", label="Data")
33+
plt.legend();
34+
plt.show()
35+
```
36+
37+
(This model is not perfect to give the correct prediction, so sometimes there is an "error term" $\epsilon$ (Greek letter epsilon) that captures the imperfectness of the model.)
38+
39+
A machine learning model, such as the one described above, has *two main uses:*
40+
41+
1. **Prediction:** How accurately can we predict outcomes?
42+
43+
- Given a new person's $Age, BMI, Income$ , predict the person's $BloodPressure$ and compare it to the true value.
44+
45+
2. **Inference:** Which predictors are associated with the response, and how strong is the association?
46+
47+
- Suppose the model is described as $BloodPressure = f(Age,BMI,Income)=20 + 3 \cdot Age - .2 \cdot BMI + .00015 \cdot Income$. Each variable has a relationship to the outcome: an increase of $Age$ by 1 will lead to an increase of $BloodPressure$ by 3. This measures the strength of association between a variable and the outcome.
48+
49+
## Population and Sample
50+
51+
The way we formulate machine learning model is based on some fundamental concepts in inferential statistics. We will refresh this quickly in the context of our problem. Recall the following definitions:
52+
53+
**Population:** The entire collection of individual units that a researcher is interested to study. For NHANES, this could be the entire US population.
54+
55+
**Sample:** A smaller collection of individual units that the researcher has selected to study. For NHANES, this could be a random sampling of the US population.
56+
57+
In Machine Learning problems, we often like to take two, non-overlapping samples from the population: the **Training Set**, and the **Test Set**. We **train** our model using the Training Set, which gives us a function $f()$ that relates the predictors to the outcome. Then, for our main use cases:
58+
59+
1. **Prediction:** We use the trained model to predict the outcome using predictors from the Test Set and compare to the true value in the Test Set.
60+
2. **Inference**: We examine the function $f()$'s trained values, which are called **parameters**. For instance, $f(Age,BMI,Income)=20 + 3 \cdot Age - .2 \cdot BMI + .00015 \cdot Income$, the values $20$, $3$, $-.2$, and $.00015$ are the parameters. Because these parameters are derived from the Training Set, they are an *estimated* quantity from a sample, similar to other summary statistics like the mean of a sample. Therefore, to say anything about the true population, we have to use statistical tools such as p-values and confidence intervals.
61+
62+
If the concepts of population, sample, estimation, p-value, and confidence interval is new to you, we recommend do a bit of reading here \[todo\].
63+
64+
## How to evaluate and pick a model?
65+
66+
The little example model we showcased above is an example of a **linear model**, but we will look at several types of models in this course. In order to decide how to evaluate and pick a model, we will need to develop a framework to assess a model. Let's start with the use case of prediction.
67+
68+
### Prediction
69+
70+
Suppose we try to use the single variable $BMI$ to predict $BloodPressure$ using a linear model.
71+
72+
```{python}
73+
import pandas as pd
74+
import seaborn as sns
75+
import numpy as np
76+
from sklearn.model_selection import train_test_split
77+
import matplotlib.pyplot as plt
78+
from formulaic import model_matrix
79+
import statsmodels.api as sm
80+
81+
nhanes = pd.read_csv("classroom_data/NHANES.csv")
82+
nhanes['BloodPressure'] = nhanes['BPDiaAve'] + (nhanes['BPSysAve'] - nhanes['BPDiaAve']) / 3
83+
84+
y, X = model_matrix("BloodPressure ~ BMI", nhanes)
85+
86+
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=42)
87+
model = sm.OLS(y_train, X_train)
88+
results = model.fit()
89+
results.summary()
90+
91+
plt.plot(X_train.BMI, results.fittedvalues, label="fitted line")
92+
plt.scatter(X_train.BMI, y_train, alpha=.3, color="brown", label="training set")
93+
plt.legend();
94+
95+
```
96+
97+
We examine how well our model performs in terms of prediction by seeing how close our model's predicted $BloodPressure$ is to the Training Set's true $BloodPressure$: the **Training Error**. We also take the model to the Testing Set to predict $BloodPressure$ using predictors from the Test Set and compare to the true $BloodPressure$ in the Test Set: the **Testing Error.** We want the model's Training Error to be adequately small on the Training Set, but what we really care about is the Testing Error, because it is a true test of how the model performs on unseen, new data, and allows us to see how generalizeable the model is.
98+
99+
Okay, let's how it does on the Training Set:
100+
101+
```{python}
102+
np.mean((results.fittedvalues - y_train.BloodPressure) ** 2)
103+
```
104+
105+
```{python}
106+
results.mse_resid
107+
```
108+
109+
\[graph here\]
110+
111+
And then on the Test Set:
112+
113+
```{python}
114+
np.mean((results.get_prediction(X_test).predicted_mean - y_test.BloodPressure) ** 2)
115+
116+
```
117+
118+
```{python}
119+
120+
plt.plot(X_test.BMI, results.get_prediction(X_test).predicted_mean, label="fitted line")
121+
plt.scatter(X_test.BMI, y_test, alpha=.3, color="black", label="test set")
122+
plt.legend();
123+
```
124+
125+
We see that the Training Error is fairly high, and the Testing Error is even higher. This is an example of **Underfitting**, where our model failed to capture the complexity of the data in both the Training and Testing Set.
126+
127+
Let's return to the drawing board and fit a new type of model that has more flexibility around complicated patterns of data. Let's see how it does on the Training Set:
128+
129+
```{python}
130+
#y, X = model_matrix("BloodPressure ~ poly(BMI, degree=5)", nhanes)
131+
132+
#X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=42)
133+
#model = sm.OLS(y_train, X_train)
134+
#results = model.fit()
135+
#results.summary()
136+
137+
#plt.plot(X_train.BMI, results.fittedvalues, label="fitted line")
138+
#plt.scatter(X_train.BMI, y_train, alpha=.3, color="brown", label="training set")
139+
#plt.legend();
140+
```
141+
142+
\[graph here\]
143+
144+
And then on the Test Set:
145+
146+
\[graph here\]
147+
148+
We see that the Training Error is low, but the Testing Error is huge! This is an example of **Overfitting**, in which our model fitted the shape of of the training set so well that it fails to generalize to the testing set.
149+
150+
We want to find a model that is "just right" that doesn't underfit or overfit the data. Usually, as the model becomes more flexible, the Training Error keeps lowering, and the Testing Error will lower a bit before increasing. See below:
151+
152+
![Source: An Introduction to Statistical Learning, Ch. 2, by Gareth James, Daniela Witten, Trevor Hastie, Roebert Tibshirani, Jonathan Taylor.](images/testing_error-01.png)
153+
154+
Also see this interactive tutorial: [https://mlu-explain.github.io/bias-variance/](https://mlu-explain.github.io/bias-variance/+)
155+
156+
### Inference
157+
158+
Let's consider how we would evaluate and choose models for Inference.
159+
160+
For models with low number of predictors, there are some plots and metrics one would consider, such as BIC.
161+
162+
For models with high number of predictors, we will talk about it in more detail in weeks 5 & 6.
163+
164+
Besides how flexible a model is, another categorization of machine models is how **interpretable** they are. The more interpretable a model is, the better one can describe how each variable has an predictor of the model. That makes the inference process easier.
165+
166+
Below are some example models mapped to these two dichotomies. The linear model lies very similar as the "Least Squares" models.
167+
168+
![Source: An Introduction to Statistical Learning, Ch. 2, by Gareth James, Daniela Witten, Trevor Hastie, Roebert Tibshirani, Jonathan Taylor](images/flexibility_vs_interpretability.png){width="500"}
169+
170+
## The NumPy Package
171+
172+
### Subsetting
173+
174+
### How to split the data for training and testing
175+
176+
## Linear Regression Preview?
177+
178+
## Appendix: Other terms
179+
180+
Parametric vs. Non-parametric
181+
182+
Bias-Variance trade-off
183+
184+
Supervised vs. Unsupervised

0 commit comments

Comments
 (0)