INTRODUCTION TO DATA SCIENCE
WFAiS UJ, Informatyka Stosowana I stopień studiów
1
1/12, 8/12, 15/12, 22/12/2020
This lecture is
based on course by E. Fox and C. Guestrin, Univ of Washington
What is retrieval?
1/12, 8/12, 15/12, 22/12/2020
2
What is retrieval?
1/12, 8/12, 15/12, 22/12/2020
3
What is retrieval?
1/12, 8/12, 15/12, 22/12/2020
4
Retrieval applications
1/12, 8/12, 15/12, 22/12/2020
5
What is clustering?
1/12, 8/12, 15/12, 22/12/2020
6
Clustring applications
1/12, 8/12, 15/12, 22/12/2020
7
Clustering applications
1/12, 8/12, 15/12, 22/12/2020
8
Impact of retrieval & clustering
1/12, 8/12, 15/12, 22/12/2020
9
Overwiew of content
1/12, 8/12, 15/12, 22/12/2020
10
1/12, 8/12, 15/12, 22/12/2020
11
Retrieval as
k-nearest neighbor search
1-NN search for retrieval
1/12, 8/12, 15/12, 22/12/2020
12
1-NN search for retrieval
1/12, 8/12, 15/12, 22/12/2020
13
1-NN search for retrieval
1/12, 8/12, 15/12, 22/12/2020
14
1-NN search for retrieval
1/12, 8/12, 15/12, 22/12/2020
15
1-NN algorithm
1/12, 8/12, 15/12, 22/12/2020
16
1-NN algorithm
1/12, 8/12, 15/12, 22/12/2020
17
k-NN algorithm
1/12, 8/12, 15/12, 22/12/2020
18
k-NN algorithm
1/12, 8/12, 15/12, 22/12/2020
19
Critical elements of NN search
1/12, 8/12, 15/12, 22/12/2020
20
Document representation
1/12, 8/12, 15/12, 22/12/2020
21
Document representation
1/12, 8/12, 15/12, 22/12/2020
22
Document representation
1/12, 8/12, 15/12, 22/12/2020
23
Document representation
1/12, 8/12, 15/12, 22/12/2020
24
Distance metrics:
1/12, 8/12, 15/12, 22/12/2020
25
Distance metrics:
1/12, 8/12, 15/12, 22/12/2020
26
Distance metrics:
1/12, 8/12, 15/12, 22/12/2020
27
Distance metrics:
1/12, 8/12, 15/12, 22/12/2020
28
Distance metrics:
1/12, 8/12, 15/12, 22/12/2020
29
Distance metrics:
1/12, 8/12, 15/12, 22/12/2020
30
Distance metrics:
1/12, 8/12, 15/12, 22/12/2020
31
Distance metrics:
1/12, 8/12, 15/12, 22/12/2020
32
Distance metrics:
1/12, 8/12, 15/12, 22/12/2020
33
Distance metrics:
1/12, 8/12, 15/12, 22/12/2020
34
Distance metrics:
1/12, 8/12, 15/12, 22/12/2020
35
Distance metrics
1/12, 8/12, 15/12, 22/12/2020
36
Distance metrics
1/12, 8/12, 15/12, 22/12/2020
37
Distance metrics
1/12, 8/12, 15/12, 22/12/2020
38
Distance metrics
1/12, 8/12, 15/12, 22/12/2020
39
Distance metrics
1/12, 8/12, 15/12, 22/12/2020
40
Distance metrics
1/12, 8/12, 15/12, 22/12/2020
41
Distance metrics
1/12, 8/12, 15/12, 22/12/2020
42
Combining distance metrics
1/12, 8/12, 15/12, 22/12/2020
43
1/12, 8/12, 15/12, 22/12/2020
44
Scaling up k-NN search
by storing data in a KD-tree
Complexity of brute-force search
1/12, 8/12, 15/12, 22/12/2020
45
KD-trees
1/12, 8/12, 15/12, 22/12/2020
46
KD-trees
1/12, 8/12, 15/12, 22/12/2020
47
KD-trees
1/12, 8/12, 15/12, 22/12/2020
48
KD-trees
1/12, 8/12, 15/12, 22/12/2020
49
KD-trees
1/12, 8/12, 15/12, 22/12/2020
50
KD-trees
1/12, 8/12, 15/12, 22/12/2020
51
KD-trees
1/12, 8/12, 15/12, 22/12/2020
52
KD-trees
1/12, 8/12, 15/12, 22/12/2020
53
Nearest neighbor with KD-trees
1/12, 8/12, 15/12, 22/12/2020
54
Nearest neighbor with KD-trees
1/12, 8/12, 15/12, 22/12/2020
55
Nearest neighbor with KD-trees
1/12, 8/12, 15/12, 22/12/2020
56
Nearest neighbor with KD-trees
1/12, 8/12, 15/12, 22/12/2020
57
Nearest neighbor with KD-trees
1/12, 8/12, 15/12, 22/12/2020
58
Nearest neighbor with KD-trees
1/12, 8/12, 15/12, 22/12/2020
59
Nearest neighbor with KD-trees
1/12, 8/12, 15/12, 22/12/2020
60
Nearest neighbor with KD-trees
1/12, 8/12, 15/12, 22/12/2020
61
Nearest neighbor with KD-trees
1/12, 8/12, 15/12, 22/12/2020
62
Nearest neighbor with KD-trees
1/12, 8/12, 15/12, 22/12/2020
63
Nearest neighbor with KD-trees
1/12, 8/12, 15/12, 22/12/2020
64
Nearest neighbor with KD-trees
1/12, 8/12, 15/12, 22/12/2020
65
Complexity for N queries
1/12, 8/12, 15/12, 22/12/2020
66
Complexity for N queries
1/12, 8/12, 15/12, 22/12/2020
67
k-NN with KD-trees
1/12, 8/12, 15/12, 22/12/2020
68
Approximate k-NN with KD-trees
1/12, 8/12, 15/12, 22/12/2020
69
Closing remarks on KD-trees
1/12, 8/12, 15/12, 22/12/2020
70
KD-tree in high dimmensions
1/12, 8/12, 15/12, 22/12/2020
71
Moving away from exact NN search
1/12, 8/12, 15/12, 22/12/2020
72
1/12, 8/12, 15/12, 22/12/2020
73
Locality Sensitive Hashing (LHS)
as alternative to KD-trees
Locality sensitive hashing
1/12, 8/12, 15/12, 22/12/2020
74
Locality sensitive hashing
1/12, 8/12, 15/12, 22/12/2020
75
Locality sensitive hashing
1/12, 8/12, 15/12, 22/12/2020
76
Locality sensitive hashing
1/12, 8/12, 15/12, 22/12/2020
77
Locality sensitive hashing
1/12, 8/12, 15/12, 22/12/2020
78
Locality sensitive hashing
1/12, 8/12, 15/12, 22/12/2020
79
Locality sensitive hashing
1/12, 8/12, 15/12, 22/12/2020
80
Locality sensitive hashing
1/12, 8/12, 15/12, 22/12/2020
81
Locality sensitive hashing
1/12, 8/12, 15/12, 22/12/2020
82
Locality sensitive hashing
1/12, 8/12, 15/12, 22/12/2020
83
LSH: improving efficiency
1/12, 8/12, 15/12, 22/12/2020
84
LSH: improving efficiency
1/12, 8/12, 15/12, 22/12/2020
85
LSH: improving efficiency
1/12, 8/12, 15/12, 22/12/2020
86
LSH: improving efficiency
1/12, 8/12, 15/12, 22/12/2020
87
LSH: improving efficiency
1/12, 8/12, 15/12, 22/12/2020
88
LSH: improving efficiency
1/12, 8/12, 15/12, 22/12/2020
89
LSH recap
1/12, 8/12, 15/12, 22/12/2020
90
LSH: moving to higher dimmensions d
1/12, 8/12, 15/12, 22/12/2020
91
LSH: moving to higher dimmensions d
1/12, 8/12, 15/12, 22/12/2020
92
What you can do now …
1/12, 8/12, 15/12, 22/12/2020
93
1/12, 8/12, 15/12, 22/12/2020
94
Clustering:
An unsupervised learning task
Motivation
1/12, 8/12, 15/12, 22/12/2020
95
Motivation
1/12, 8/12, 15/12, 22/12/2020
96
I dont’t just
like sport!
Motivation
1/12, 8/12, 15/12, 22/12/2020
97
Clustering: a supervised learning
1/12, 8/12, 15/12, 22/12/2020
98
Custering: a supervised learning
1/12, 8/12, 15/12, 22/12/2020
99
Example of
supervised learning
Clustering: an unsupervised learning
1/12, 8/12, 15/12, 22/12/2020
100
An unsupervised
learning task
What defines a cluster ?
1/12, 8/12, 15/12, 22/12/2020
101
Hope for unsupervised learning
1/12, 8/12, 15/12, 22/12/2020
102
Other (challenging!) clusters to discover
1/12, 8/12, 15/12, 22/12/2020
103
Analysed by your eyes
Other (challenging!) clusters to discover
1/12, 8/12, 15/12, 22/12/2020
104
Analysed by clustering algorithms
1/12, 8/12, 15/12, 22/12/2020
105
k-means
clustering algorithm
k-means clustering algorithm
1/12, 8/12, 15/12, 22/12/2020
106
k-means clustering algorithm
1/12, 8/12, 15/12, 22/12/2020
107
k-means clustering algorithm
1/12, 8/12, 15/12, 22/12/2020
108
k-means clustering algorithm
1/12, 8/12, 15/12, 22/12/2020
109
k-means clustering algorithm
1/12, 8/12, 15/12, 22/12/2020
110
k-means as coordinate descent algorithm
1/12, 8/12, 15/12, 22/12/2020
111
K-means as coordinate descent algorithm
1/12, 8/12, 15/12, 22/12/2020
112
Convergence of k-means
1/12, 8/12, 15/12, 22/12/2020
113
Because we can cast k-means as coordinate
descent algorithm we know that we are
converging to local optimum
Convergence of k-mans to local mode
1/12, 8/12, 15/12, 22/12/2020
114
Crosses: initialised centers
Convergence of k-mans to local mode
1/12, 8/12, 15/12, 22/12/2020
115
Crosses: initialised centers
Convergence of k-mans to local mode
1/12, 8/12, 15/12, 22/12/2020
116
Crosses: initialised centers
Assigment to which group has changed
k-means very sensitive to initiased centers
Smart initialisation: k-means++ overwiew
1/12, 8/12, 15/12, 22/12/2020
117
k-means++ visualised
1/12, 8/12, 15/12, 22/12/2020
118
k-means++ visualised
1/12, 8/12, 15/12, 22/12/2020
119
k-means++ visualised
1/12, 8/12, 15/12, 22/12/2020
120
k-means++ visualised
1/12, 8/12, 15/12, 22/12/2020
121
Smart initialisation: k-means++ overwiew
1/12, 8/12, 15/12, 22/12/2020
122
Assessing quality of the clustering
1/12, 8/12, 15/12, 22/12/2020
123
k-means objective
1/12, 8/12, 15/12, 22/12/2020
124
Cluster heterogeneity
1/12, 8/12, 15/12, 22/12/2020
125
What happens to heterogeneity as k increases?
1/12, 8/12, 15/12, 22/12/2020
126
How to choose k?
1/12, 8/12, 15/12, 22/12/2020
127
1/12, 8/12, 15/12, 22/12/2020
128
MapReduce
Counting words on a single processor
1/12, 8/12, 15/12, 22/12/2020
129
Naive parallel word counting
1/12, 8/12, 15/12, 22/12/2020
130
Counting words & merging tabels
1/12, 8/12, 15/12, 22/12/2020
131
MapReduce abstraction
1/12, 8/12, 15/12, 22/12/2020
132
MapReduce – Execution overwiew
1/12, 8/12, 15/12, 22/12/2020
133
Improving performance
1/12, 8/12, 15/12, 22/12/2020
134
Scaling up k-means via MapReduce
1/12, 8/12, 15/12, 22/12/2020
135
Scaling up k-means via MapReduce
1/12, 8/12, 15/12, 22/12/2020
136
Scaling up k-means via MapReduce
1/12, 8/12, 15/12, 22/12/2020
137
Scaling up k-means via MapReduce
1/12, 8/12, 15/12, 22/12/2020
138
Parallel k-means via MapReduce
1/12, 8/12, 15/12, 22/12/2020
139
What you can do now …
1/12, 8/12, 15/12, 22/12/2020
140
1/12, 8/12, 15/12, 22/12/2020
141
Probabilistic approach:
mixture model
Why probabilistic approach?
1/12, 8/12, 15/12, 22/12/2020
142
Why probabilistic approach?
1/12, 8/12, 15/12, 22/12/2020
143
Why probabilistic approach?
1/12, 8/12, 15/12, 22/12/2020
144
Why probabilistic approach?
1/12, 8/12, 15/12, 22/12/2020
145
Mixture models
1/12, 8/12, 15/12, 22/12/2020
146
Application: clustering images
1/12, 8/12, 15/12, 22/12/2020
147
Application: clustering images
1/12, 8/12, 15/12, 22/12/2020
148
Single RGB vector per image
Application: clustering images
1/12, 8/12, 15/12, 22/12/2020
149
Application: clustering images
1/12, 8/12, 15/12, 22/12/2020
150
Application: clustering images
1/12, 8/12, 15/12, 22/12/2020
151
Application: clustering images
1/12, 8/12, 15/12, 22/12/2020
152
We see that they are grouping!
But not easy to distinguish between groups
Application: clustering images
1/12, 8/12, 15/12, 22/12/2020
153
In this dimmension
separable groups!
Model for a given image type
1/12, 8/12, 15/12, 22/12/2020
154
Model for a given image type
1/12, 8/12, 15/12, 22/12/2020
155
Application: clustering images
1/12, 8/12, 15/12, 22/12/2020
156
Application: clustering images
1/12, 8/12, 15/12, 22/12/2020
157
Application: clustering images
1/12, 8/12, 15/12, 22/12/2020
158
Application: clustering images
1/12, 8/12, 15/12, 22/12/2020
159
Mixture of Gaussians
1/12, 8/12, 15/12, 22/12/2020
160
Mixture of Gaussians
1/12, 8/12, 15/12, 22/12/2020
161
Mixture of Gaussians
1/12, 8/12, 15/12, 22/12/2020
162
Mixture of Gaussians
1/12, 8/12, 15/12, 22/12/2020
163
Mixture of Gaussians
1/12, 8/12, 15/12, 22/12/2020
164
Mixture of Gaussians
1/12, 8/12, 15/12, 22/12/2020
165
Mixture of Gaussians
1/12, 8/12, 15/12, 22/12/2020
166
Application: clustering documents
1/12, 8/12, 15/12, 22/12/2020
167
Application: clustering documents
1/12, 8/12, 15/12, 22/12/2020
168
Application: clustering documents
1/12, 8/12, 15/12, 22/12/2020
169
Application: clustering documents
1/12, 8/12, 15/12, 22/12/2020
170
Application: clustering documents
1/12, 8/12, 15/12, 22/12/2020
171
Application: clustering documents
1/12, 8/12, 15/12, 22/12/2020
172
Application: clustering documents
1/12, 8/12, 15/12, 22/12/2020
173
1/12, 8/12, 15/12, 22/12/2020
174
Inferring soft assignments with
expectation maximization (EM)
Inferring cluster labels
1/12, 8/12, 15/12, 22/12/2020
175
1/12, 8/12, 15/12, 22/12/2020
176
1/12, 8/12, 15/12, 22/12/2020
177
1/12, 8/12, 15/12, 22/12/2020
178
1/12, 8/12, 15/12, 22/12/2020
179
1/12, 8/12, 15/12, 22/12/2020
180
1/12, 8/12, 15/12, 22/12/2020
181
1/12, 8/12, 15/12, 22/12/2020
182
Part 1: Summary
1/12, 8/12, 15/12, 22/12/2020
183
1/12, 8/12, 15/12, 22/12/2020
184
Then split into separate tables and consider them independently.
1/12, 8/12, 15/12, 22/12/2020
185
1/12, 8/12, 15/12, 22/12/2020
186
1/12, 8/12, 15/12, 22/12/2020
187
1/12, 8/12, 15/12, 22/12/2020
188
Part 2a : Summary
1/12, 8/12, 15/12, 22/12/2020
189
1/12, 8/12, 15/12, 22/12/2020
190
1/12, 8/12, 15/12, 22/12/2020
191
1/12, 8/12, 15/12, 22/12/2020
192
1/12, 8/12, 15/12, 22/12/2020
193
1/12, 8/12, 15/12, 22/12/2020
194
1/12, 8/12, 15/12, 22/12/2020
195
1/12, 8/12, 15/12, 22/12/2020
196
Part 2b: Summary
Expectation maximization (ME)
1/12, 8/12, 15/12, 22/12/2020
197
Expectation maximization (ME)
1/12, 8/12, 15/12, 22/12/2020
198
Expectation maximization (ME)
1/12, 8/12, 15/12, 22/12/2020
199
Expectation maximization (ME)
1/12, 8/12, 15/12, 22/12/2020
200
