Data Science and (Unsupervised) Machine Learning with scikit-learn
By at Datacratic
Presented Dec 1, 2014 at Montreal Python
In [216]:
from IPython.display import YouTubeVideo
YouTubeVideo('2lpS6gUwiJQ')
Out[216]:
- Proud to be a Montreal startup
- Specializing in machine learning
- Founded in 2009
- 27 employees downtown (Peel Metro), 1 in Ottawa, 3 in NYC
This talk
- A different way to look at graph analysis and visualization,
- as an introduction to a few cool algorithms: Truncated SVD, K-Means and t-SNE
- with a practical walkthrough using
scikit-learnand friendsnumpyandbokeh, - and finishing off with some more general commentary on this approach to data analysis.
Links:
- One-page HTML version at http://opensource.datacratic.com/mtlpy50
- All materials available at https://github.com/datacratic/mtlpy50
- Video of this presentation at https://www.youtube.com/watch?v=2lpS6gUwiJQ
A map of Reddit
- Reddit is "the front page of the internet"
- Basically a discussion board, with sub-boards called subreddits
- Figure from this paper: Navigating the massive world of reddit: Using backbone networks to map user interests in social media
- http://arxiv.org/pdf/1312.3387v1.pdf
- They took data about who posted to what subreddit
- Two subreddits have a link if enough users posted to both
- Visualization using Gephi with the OpenOrd layout algorithm
- Very cool interactive version
- Seems like a totally natural approach but I want to show another way of doing this
scikit-learn and friends
scikit-learnis becoming the de-facto machine learning library for Python- Works in conjunction with
scipyandnumpy - Part of the PyData toolbox, along with
pandasandbokeh
A first look at the data
File comes from here: http://figshare.com/articles/reddit_user_posting_behavior/874101
In [177]:
%%bash
head reddit_user_posting_behavior.csv
Formatting it a little bit...
In [178]:
import pandas as pd
pd.read_csv("reddit_user_posting_behavior.csv", nrows=10, names=["user"]+range(25)).fillna("")
Out[178]:
Loading the data into a sparse matrix
- We want a matrix: rows are subreddits, columns are users
- cells will be 1 if user posted to subreddit, otherwise 0
- This step was the most technically challenging
- Naïve first few tries: 20 minute run-time
- The code below runs around 10 seconds
In [179]:
%%time
user_ids = []
subreddit_ids = []
subreddit_to_id = {}
i=0
with open("reddit_user_posting_behavior.csv", 'r') as f:
for line in f:
for sr in line.rstrip().split(",")[1:]:
if sr not in subreddit_to_id:
subreddit_to_id[sr] = len(subreddit_to_id)
user_ids.append(i)
subreddit_ids.append(subreddit_to_id[sr])
i+=1
import numpy as np
from scipy.sparse import csr_matrix
rows = np.array(subreddit_ids)
cols = np.array(user_ids)
data = np.ones((len(user_ids),))
num_rows = len(subreddit_to_id)
num_cols = i
# the code above exists to feed this call
adj = csr_matrix( (data,(rows,cols)), shape=(num_rows, num_cols) )
print adj.shape
print ""
# now we have our matrix, so let's gather up a bit of info about it
users_per_subreddit = adj.sum(axis=1).A1
subreddits = range(len(subreddit_to_id))
for sr in subreddit_to_id:
subreddits[subreddit_to_id[sr]] = sr
subreddits = np.array(subreddits)
Unwieldy data
Our adjacency matrix is a bit problematic to deal with as-is:
- It's a very wide: 850,000 columns
- It's very sparse: only about 0.06% full
- It's a binary matrix: only 0's and 1's
Dimensionality reduction
- Family of algorithms for solving this problem
- AKA decomposition, compression, feature extraction
- it's to big matrices what JPEG is to photos, MP3 to music, MPEG for video etc
- The output will be as wide as we ask for
- The wider it is the less lossy the compression will be
- The output matrix will be dense
- The output matrix will have continuous real values
scikit-learnhas a decomposition package
We'll use TruncatedSVD
- TruncatedSVD from
scikit-learn - SVD stands for Singular Vector Decomposition, Truncated because we only want part of the computation
- Good mathy description of how it works in Chapter 11 of online book Mining of Massive Datasets
- Right now let's focus on how to use it and exploring the output
In [180]:
%%time
from sklearn.decomposition import TruncatedSVD
from sklearn.preprocessing import normalize
svd = TruncatedSVD(n_components=100)
embedded_coords = normalize(svd.fit_transform(adj), norm='l1')
print embedded_coords.shape
The output is kind of neat:
- Each row is like a set of coordinates in a 100-dimensional space for a subreddit
- Each column defines one axis of this 100-dimensional space, ordered by how much information they capture
- We can look at how much of the original matrix we captured with the first N dimensions
- The first 2 capture around 25%
- The 100 we will use capture around 60%
In [181]:
%matplotlib inline
pd.DataFrame(np.cumsum(svd.explained_variance_ratio_)).plot(figsize=(13, 8))
Out[181]:
In [190]:
# this function will show you the axes on which a particular subreddit scores the highest/lowest
def pickOutSubreddit(sr):
sorted_axes = embedded_coords[list(subreddits).index(sr)].argsort()[::-1]
return pd.DataFrame(subreddits[np.argsort(embedded_coords[:,sorted_axes], axis=0)[::-1]], columns=sorted_axes)
pickOutSubreddit("soccer")
Out[190]:
A few interesting dimensions
- I went through each column and picked out some interesting dimensions
In [191]:
pd.DataFrame(subreddits[np.argsort(embedded_coords[:,[0, 1, 44,51,84,50,47,40]], axis=0)[::-1]],
columns=[
"0: big - small",
"1: big - small",
"44: soccer - guns",
"51: programming - food",
"84: music - bikes",
"50: osx - books",
"47: wow - starcraft",
"40: male grooming - life hacks"
])
# not shown but also amusing:
# 14: music - pot
# 24: science - porn
Out[191]:
Visualizing these dimensions
- Bokeh makes it easy to get hover-tooltips!
- Each dot in our plot will be a subreddit (mouse over to see which one) scaled by size
- We'll only look at the top 3,500 or so subreddits, for speed
In [185]:
import bokeh.plotting as bp
from bokeh.objects import HoverTool
bp.output_notebook()
row_selector = np.where(users_per_subreddit>100)
In [186]:
bp.figure(plot_width=900, plot_height=700, title="Subreddit Map by Most Informative Dimensions",
x_axis_label = "Dimension 0",
y_axis_label = "Dimension 1",
tools="pan,wheel_zoom,box_zoom,reset,hover,previewsave",
min_border=1)
bp.scatter(
x = embedded_coords[:,0][row_selector],
y = embedded_coords[:,1][row_selector],
radius= np.log2(users_per_subreddit[row_selector])/6000,
source=bp.ColumnDataSource({"subreddit": subreddits[row_selector]})
).select(dict(type=HoverTool)).tooltips = {"/r/":"@subreddit"}
bp.show()
Hm... let's try some other dimensions!
In [192]:
bp.figure(plot_width=900, plot_height=700, title="Subreddit Map by Interesting Dimensions",
x_axis_label = "Guns <–> Soccer (Dimension 44)",
y_axis_label = "Food <–> Programming (Dimension 51)",
tools="pan,wheel_zoom,box_zoom,reset,hover,previewsave",
min_border=1)
bp.scatter(
x = embedded_coords[:,44][row_selector],
y = embedded_coords[:,51][row_selector],
radius= np.log2(users_per_subreddit[row_selector])/6000,
source=bp.ColumnDataSource({"subreddit": subreddits[row_selector]})
).select(dict(type=HoverTool)).tooltips = {"/r/":"@subreddit"}
bp.show()
Can we add some colour?
- Would be nice to colour-code these dots by overall topic
- 20 colours is probably on the upper end of what we can perceive effectively in a single plot
Clustering
- Family of algorithms for grouping items into buckets
scikit-learnhas a clustering package
We'll use KMeans
- KMeans from
scikit-learn - Take K points and try to distribute them in the space so that they sit in the middle of clusters, i.e. close to the mean
In [193]:
%%time
from scipy.stats import rankdata
embedded_ranks = np.array([rankdata(c) for c in embedded_coords.T]).T
from sklearn.cluster import KMeans
n_clusters = 20
km = KMeans(n_clusters)
clusters = km.fit_predict(embedded_ranks)
In [194]:
pd.DataFrame( [subreddits[clusters == i][users_per_subreddit[clusters == i].argsort()[-6:][::-1]] for i in range(n_clusters)] )
Out[194]:
Visualization in Technicolour
- This time we'll colour-code each dot by cluster
In [199]:
colormap = np.array([
"#1f77b4", "#aec7e8", "#ff7f0e", "#ffbb78", "#2ca02c",
"#98df8a", "#d62728", "#ff9896", "#9467bd", "#c5b0d5",
"#8c564b", "#c49c94", "#e377c2", "#f7b6d2", "#7f7f7f",
"#c7c7c7", "#bcbd22", "#dbdb8d", "#17becf", "#9edae5"
])
bp.figure(plot_width=900, plot_height=700, title="Subreddit Map by Interesting Dimensions",
x_axis_label = "Guns <–> Soccer (Dimension 44)",
y_axis_label = "Food <–> Programming (Dimension 51)",
tools="pan,wheel_zoom,box_zoom,reset,hover,previewsave",
min_border=1)
bp.scatter(
x = embedded_coords[:,44][row_selector],
y = embedded_coords[:,51][row_selector],
color= colormap[clusters[row_selector]],
radius= np.log2(users_per_subreddit[row_selector])/6000,
source=bp.ColumnDataSource({"subreddit": subreddits[row_selector]})
).select(dict(type=HoverTool)).tooltips = {"/r/":"@subreddit"}
bp.show()
Can we combine all these dimensions into one plot?
- Just plotting the first two dimensions wasn't very interesting
- We face the same problem map-makers face: making a high-dimensional thing 2-dimensional
- There are lots of ways to make a 2-d map of our 3-d planet!
Manifold learning
- Family of algorithms to transform many dimensions to 2 or 3
- AKA low-dimensional embedding
scikit-learnhas a manifold learning package
We'll use TSNE
- TSNE from
scikit-learn - Original paper is awesome
- Math is fairly involved but basically tries to preserve micro and macro structure, sacrificing some global structure
- Gives very good, human-readable results on real-world datasets, check the examples in the paper
In [197]:
%%time
from sklearn.manifold import TSNE
xycoords = TSNE().fit_transform(embedded_coords[row_selector])
In [198]:
bp.figure(plot_width=900, plot_height=700, title="Subreddit Map by t-SNE",
tools="pan,wheel_zoom,box_zoom,reset,hover,previewsave",
x_axis_type=None, y_axis_type=None, min_border=1)
bp.scatter(
x = xycoords[:,0],
y = xycoords[:,1],
color= colormap[clusters[row_selector]],
radius= np.log2(users_per_subreddit[row_selector])/60,
source=bp.ColumnDataSource({"subreddit": subreddits[row_selector]})
).select(dict(type=HoverTool)).tooltips = {"/r/":"@subreddit"}
bp.show()
Taking a step back
- We loaded a dataset into a big sparse matrix
- We used
TruncatedSVDto make it into a surprisingly informative smaller matrix - We used
KMeansto group our subreddits into some clusters - We used
TSNEto get coordinates for a scatterplot - We used
bokehto make an interactive graphic - So what?
A hidden agenda
- Use these tools on a dataset that you can a priori reason about
- Show that meaningful patterns come out
- Convince you that you can try this on data you can't a priori reason about
- So that you will feel confident that the algo has found something useful
A totally different example
- Latent Semantic Analysis on Twitter data
- Same tricks as before
- load up a sparse matrix of words vs tweets (instead of subreddits vs users)
- Raw data is from here https://archive.org/details/twitter_cikm_2010
- A few million tweets from 2010
- I ran
TruncatedSVDandTSNEas before - Notebook is available at https://github.com/datacratic/mtlpy50
- HTML with interactive graphic is at http://opensource.datacratic.com/mtlpy50/tweets.html
Where to next?
- Notebook is available at https://github.com/datacratic/mtlpy50
- You can use these on your own data (dense or sparse!)
- Big spreadsheets or datasets where each row is an entity are also good candidates for this type of analysis
- Watch patterns appear before your very eyes!
- NB:
TruncatedSVD,KMeansandTSNEare some of my go-to algorightms but each belongs to a family with other options
One more thing...
scikit-learnis nice, but some of these algorithms can be slow and hard to use- Early next year, Datacratic will release an early version of our Machine Learning Database
- It lets you do this type of analysis by calling a REST API and it's lightning-fast!
- Stay tuned :)