Prediction Demystified
Jed Ludlow
jedludlow.com
Objectives
New to Prediction?
- Understand the essential terminology and concepts.
- Get ideas about where to go next.
Experienced Data Scientist?
- A few philosophical things to consider.
- Implications of machine learning from an engineering perspective.
Data Science
- A data scientist is a statistician who lives in San Francisco.
- Data science is statistics on a Mac.
- A data scientist is someone who is better at statistics than any software engineer and better at software engineering than any statistician.
(from https://twitter.com/jeremyjarvis/status/428848527226437632/photo/1)
The Original Data Scientist
W. Edwards Deming
(source: Wikipedia)
Control Charting is Prediction
(source: http://blog.minitab.com/blog/michelle-paret/control-charts-subgroup-size-matters)
"Hello, World" Prediction Problem: Concept
Predict the output variables (height) as a function of the input features (age). Accomplish this by fitting a model (linear regression) to the available samples (five boys) in the training data set.
"Hello, World" Prediction Problem: Python
import numpy as np
from sklearn.linear_model import LinearRegression
# Training output samples
height_inches = np.array([[51.0, 56.0, 64.0, 71.0, 69.0]]).T
# Training feature samples
age_years = np.array([[7.8, 10.7, 13.7, 17.5, 20.1]]).T
# Initialize model
model = LinearRegression()
# Train
model.fit(age_years, height_inches)
# Predict
model.predict(15.0)
plot_boys(age_years, height_inches);
plot_boys(age_years, height_inches, test, pred)
plot_boys(age_years, height_inches, test, pred)
Tall babies! Really tall adults!
plot_boys(age_years, height_inches, test, pred)
Look! No training error!
plot_boys(age_years, height_inches, test, pred)
Really tall babies! "Unphysical" heights for adults!
Skillful Prediction
"I've beat this drum a hundred times, but what statistical models should do is to predict what will happen, given or conditioned on the data which came before and the premises which led to the particular model used. Then, since we have a prediction, we wait for confirmatory, never-observed-before data. If the model was good, we will have skillful predictions. If not, we start over."
-William M. Briggs, Statistician to the Stars!
(source: http://wmbriggs.com/post/14532)
Lessons from "Hello, World" Prediction Problem
- Skillful prediction cannot be measured against the training data set. We must test against data not included in the training.
- Reducing the training error is not necessarily the primary objective of model development.
- A flexible model (polynomial regression) is not always better than an inflexible model (linear regression).
- Extrapolation beyond the bounds of the training data can produce unexpected results.
Kaggle: Predicting Survival on the Titanic
- An introductory competition for knowledge sharing.
- Fully worked examples in Excel, Python, and R.
- Upload your solution to Kaggle for scoring.
- A great first example for a classification problem.
Machine Learning Toolboxes
- Very popular open source packages available in many programming languages.
- R
- Python
- Growing number of "Machine Learning as a Service" companies.
- Big companies offering prediction APIs hosted in the cloud.
- Microsoft Azure Machine Learning
- Google Prediction API
Titanic Training Set
train_df = pd.read_csv("train.csv", dtype=dtypes)
train_df
Prediction with Google Prediction API
- Initiate a project through Google Developers Console.
- Enable prediction API and cloud storage for that project.
- Upload training data (specially formatted CSV file) to the storage bucket.
- Install Google API Python Client.
- Get OAuth 2.0 authentication flow working. (Yes, really.)
- Call
insertwith appropriately formatted JSON to configure and train the model. - Call
predictwith similarly formatted JSON to make predictions. - Pay for training and prediction usage. (Free 60-day trial available.)
Luckily, there are good examples here: https://github.com/google/google-api-python-client
Titanic Survival Model Using Google Prediction
- Fit a model to the Kaggle Titanic survival training set after some data cleaning.
- Model is live on Google's cloud services.
- Let's check training error by predicting back against a known survivor from the training set.
pred = model.predict([[
'1', # Fare class
'Spencer Mrs William Augustus Marie Eugenie', # Name
'female', # Gender
20.2, # Age
1, # Number of parents or children aboard
0, # Number of siblings or spouse aboard
146.5208, # Fare price
],])
survived(pred)
We could now make predictions for all the passengers in the test set and upload our solution to Kaggle for scoring, but it might be more fun....
pred = model.predict([[
'1', # Fare class
'Frank Lampard', # Name
'male', # Gender
36.0, # Age
0, # Number of parents or children aboard
0, # Number of siblings or spouse aboard
20.0, # Fare price
],])
survived(pred)
pred = model.predict([[
'1', # Fare class
'Frank Lampard', # Name
'male', # Gender
36.0, # Age
0, # Number of parents or children aboard
0, # Number of siblings or spouse aboard
500.0, # Fare price
],])
survived(pred)
Lessons from Titanic Survival Classifier
- Real data is almost never tidy. Some cleaning is required.
- Some model types are by nature black boxes that cannot be inspected on the inside. You might be able to discern some things about how the model is behaving by feeding it interesting test cases, but that's not the same as having full access to the model internals.
- Choose the machine learning toolbox to match the modeling need and the implementation plan.
Deep Learning
"Deep architectures are composed of multiple levels of non-linear operations, such as in neural nets with many hidden layers..."
-Yoshua Bengio, Learning Deep Architectures for AI
- Input feature vectors are very large. Thousands of features are common.
- The model is very flexible, sometimes employing millions of parameters.
- Training algorithms use sophisticated methods to avoid over fitting.
- Generally require serious hardware (GPU) to train in reasonable time.
Convolutional Neural Networks
(source: http://deeplearning.net/tutorial/lenet.html#lenet)
Records Were Made to Be Broken
(source: https://plus.google.com/113710395888978478005/posts/Gx3w9E14k2P)
(source: http://danielnouri.org/notes/2014/12/17/using-convolutional-neural-nets-to-detect-facial-keypoints-tutorial/)
Validation
What are the consequences of error in your predictions? Is safety paramount?
- Divide the available data into three sets:
- Training set for base model training.
- Cross validation set for model selection and parameter tuning.
- Test set for estimating final accuracy.
For measurements of skillful prediction, "...wait for confirmatory, never-observed-before data."
“In the course of my work in the Potti lab, I discovered what I perceived to be problems in the predictor models that made it difficult for me to continue working in that environment,” he said in an email to The Cancer Letter.
(source: http://www.cancerletter.com/articles/20150109_1)
Good Software Engineering Practice
(source: http://research.google.com/pubs/pub43146.html)
Concluding Thoughts
- Most prediction problems can be described using a common framework of definitions. In that sense, they are not mysterious. But the latest deep learning models are bordering on magical!
- Seek the simplest possible model that attains the required level of prediction performance. Keep a close eye on your "technical debt" credit card bills.
- Validate, validate, and then keep validating.
- Choose your prediction problems wisely.
- The existence of a large data set does not necessarily imply that a valuable model is lurking inside.
- Chances are you don't yet have the data sets for the the most valuable problems. Getting that data will require concerted investment.
Thank You!
Jed Ludlow
jedludlow.com