Department of Computer Science, University of Waikato, New Zealand
Eibe Frank
WEKA: A Machine Learning Toolkit
The Explorer
• Classification and Regression
• Clustering
• Association Rules
• Attribute Selection
• Data Visualization
The Experimenter
The Knowledge Flow GUI
Conclusions
Machine Learning with
WEKA
2/22/2011 University of Waikato 2
WEKA: the bird
Copyright: Martin Kramer (mkramer@wxs.nl)
2/22/2011 University of Waikato 3
WEKA: the software
Machine learning/data mining software written in Java (distributed under the GNU Public License)
Used for research, education, and applications
Complements “Data Mining” by Witten & Frank
Main features:
Comprehensive set of data pre-processing tools, learning algorithms and evaluation methods
Graphical user interfaces (incl. data visualization)
Environment for comparing learning algorithms
2/22/2011 University of Waikato 4
WEKA: versions
There are several versions of WEKA:
WEKA 3.0: “book version” compatible with description in data mining book
WEKA 3.2: “GUI version” adds graphical user interfaces (book version is command-line only)
WEKA 3.3: “development version” with lots of improvements
This talk is based on the latest snapshot of WEKA
3.3 (soon to be WEKA 3.4)
2/22/2011 University of Waikato 5
@relation heart-disease-simplified
@attribute age numeric
@attribute sex { female, male}
@attribute chest_pain_type { typ_angina, asympt, non_anginal, atyp_angina}
@attribute cholesterol numeric
@attribute exercise_induced_angina { no, yes}
@attribute class { present, not_present}
@data
63,male,typ_angina,233,no,not_present 67,male,asympt,286,yes,present
67,male,asympt,229,yes,present
38,female,non_anginal,?,no,not_present ...
WEKA only deals with “flat” files
2/22/2011 University of Waikato 6
@relation heart-disease-simplified
@attribute age numeric
@attribute sex { female, male}
@attribute chest_pain_type { typ_angina, asympt, non_anginal, atyp_angina}
@attribute cholesterol numeric
@attribute exercise_induced_angina { no, yes}
@attribute class { present, not_present}
@data
63,male,typ_angina,233,no,not_present 67,male,asympt,286,yes,present
67,male,asympt,229,yes,present
38,female,non_anginal,?,no,not_present ...
WEKA only deals with “flat” files
2/22/2011 University of Waikato 7
2/22/2011 University of Waikato 8
2/22/2011 University of Waikato 9
2/22/2011 University of Waikato 10
Explorer: pre-processing the data
Data can be imported from a file in various formats: ARFF, CSV, C4.5, binary
Data can also be read from a URL or from an SQL database (using JDBC)
Pre-processing tools in WEKA are called “filters”
WEKA contains filters for:
Discretization, normalization, resampling, attribute selection, transforming and combining attributes, …
2/22/2011 University of Waikato 11
2/22/2011 University of Waikato 12
2/22/2011 University of Waikato 13
2/22/2011 University of Waikato 14
2/22/2011 University of Waikato 15
2/22/2011 University of Waikato 16
2/22/2011 University of Waikato 17
2/22/2011 University of Waikato 18
2/22/2011 University of Waikato 19
2/22/2011 University of Waikato 20
2/22/2011 University of Waikato 21
2/22/2011 University of Waikato 22
2/22/2011 University of Waikato 23
2/22/2011 University of Waikato 24
2/22/2011 University of Waikato 25
2/22/2011 University of Waikato 26
2/22/2011 University of Waikato 27
2/22/2011 University of Waikato 28
2/22/2011 University of Waikato 29
2/22/2011 University of Waikato 30
2/22/2011 University of Waikato 31
2/22/2011 University of Waikato 32
Explorer: building “classifiers”
Classifiers in WEKA are models for predicting nominal or numeric quantities
Implemented learning schemes include:
Decision trees and lists, instance-based classifiers, support vector machines, multi-layer perceptrons, logistic regression, Bayes’ nets, …
“Meta”-classifiers include:
Bagging, boosting, stacking, error-correcting output codes, locally weighted learning, …
2/22/2011 University of Waikato 33
2/22/2011 University of Waikato 34
2/22/2011 University of Waikato 35
2/22/2011 University of Waikato 36
2/22/2011 University of Waikato 37
2/22/2011 University of Waikato 38
2/22/2011 University of Waikato 39
2/22/2011 University of Waikato 40
2/22/2011 University of Waikato 41
2/22/2011 University of Waikato 42
2/22/2011 University of Waikato 43
2/22/2011 University of Waikato 44
2/22/2011 University of Waikato 45
2/22/2011 University of Waikato 46
2/22/2011 University of Waikato 47
2/22/2011 University of Waikato 48
2/22/2011 University of Waikato 49
2/22/2011 University of Waikato 50
2/22/2011 University of Waikato 51
2/22/2011 University of Waikato 52
2/22/2011 University of Waikato 53
2/22/2011 University of Waikato 54
2/22/2011 University of Waikato 55
2/22/2011 University of Waikato 56
2/22/2011 University of Waikato 57
2/22/2011 University of Waikato 58
2/22/2011 University of Waikato 59
2/22/2011 University of Waikato 60
2/22/2011 University of Waikato 61
2/22/2011 University of Waikato 62
2/22/2011 University of Waikato 63
2/22/2011 University of Waikato 64
2/22/2011 University of Waikato 65
QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.
2/22/2011 University of Waikato 66
QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.
2/22/2011 University of Waikato 67
QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.
2/22/2011 University of Waikato 68
2/22/2011 University of Waikato 69
2/22/2011 University of Waikato 70
2/22/2011 University of Waikato 71
2/22/2011 University of Waikato 72
2/22/2011 University of Waikato 73
2/22/2011 University of Waikato 74
2/22/2011 University of Waikato 75
Quic k Tim e™ and a TIFF (LZW) dec om pres s or are needed to s ee this pic ture.
2/22/2011 University of Waikato 76
2/22/2011 University of Waikato 77
2/22/2011 University of Waikato 78
2/22/2011 University of Waikato 79
2/22/2011 University of Waikato 80
QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.
2/22/2011 University of Waikato 81
QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.
2/22/2011 University of Waikato 82
2/22/2011 University of Waikato 83
QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.
2/22/2011 University of Waikato 84
2/22/2011 University of Waikato 85
2/22/2011 University of Waikato 86
2/22/2011 University of Waikato 87
2/22/2011 University of Waikato 88
2/22/2011 University of Waikato 89
2/22/2011 University of Waikato 90
2/22/2011 University of Waikato 91
2/22/2011 University of Waikato 92
Explorer: clustering data
WEKA contains “clusterers” for finding groups of similar instances in a dataset
Implemented schemes are:
k-Means, EM, Cobweb, X-means, FarthestFirst
Clusters can be visualized and compared to “true”
clusters (if given)
Evaluation based on loglikelihood if clustering
scheme produces a probability distribution
2/22/2011 University of Waikato 93
2/22/2011 University of Waikato 94
2/22/2011 University of Waikato 95
2/22/2011 University of Waikato 96
2/22/2011 University of Waikato 97
2/22/2011 University of Waikato 98
2/22/2011 University of Waikato 99
2/22/2011 University of Waikato 100
2/22/2011 University of Waikato 101
2/22/2011 University of Waikato 102
2/22/2011 University of Waikato 103
2/22/2011 University of Waikato 104
2/22/2011 University of Waikato 105
2/22/2011 University of Waikato 106
2/22/2011 University of Waikato 107
2/22/2011 University of Waikato 108
Explorer: finding associations
WEKA contains an implementation of the Apriori algorithm for learning association rules
Works only with discrete data
Can identify statistical dependencies between groups of attributes:
milk, butter bread, eggs (with confidence 0.9 and support 2000)
Apriori can compute all rules that have a given
minimum support and exceed a given confidence
2/22/2011 University of Waikato 109
2/22/2011 University of Waikato 110
2/22/2011 University of Waikato 111
2/22/2011 University of Waikato 112
2/22/2011 University of Waikato 113
2/22/2011 University of Waikato 114
2/22/2011 University of Waikato 115
2/22/2011 University of Waikato 116
Explorer: attribute selection
Panel that can be used to investigate which
(subsets of) attributes are the most predictive ones
Attribute selection methods contain two parts:
A search method: best-first, forward selection, random, exhaustive, genetic algorithm, ranking
An evaluation method: correlation-based, wrapper, information gain, chi-squared, …
Very flexible: WEKA allows (almost) arbitrary
combinations of these two
2/22/2011 University of Waikato 117
2/22/2011 University of Waikato 118
2/22/2011 University of Waikato 119
2/22/2011 University of Waikato 120
2/22/2011 University of Waikato 121
2/22/2011 University of Waikato 122
2/22/2011 University of Waikato 123
2/22/2011 University of Waikato 124
2/22/2011 University of Waikato 125
Explorer: data visualization
Visualization very useful in practice: e.g. helps to determine difficulty of the learning problem
WEKA can visualize single attributes (1-d) and pairs of attributes (2-d)
To do: rotating 3-d visualizations (Xgobi-style)
Color-coded class values
“Jitter” option to deal with nominal attributes (and to detect “hidden” data points)
“Zoom-in” function
2/22/2011 University of Waikato 126
2/22/2011 University of Waikato 127
2/22/2011 University of Waikato 128
2/22/2011 University of Waikato 129
2/22/2011 University of Waikato 130
2/22/2011 University of Waikato 131
2/22/2011 University of Waikato 132
2/22/2011 University of Waikato 133
2/22/2011 University of Waikato 134
2/22/2011 University of Waikato 135
2/22/2011 University of Waikato 136
2/22/2011 University of Waikato 137
2/22/2011 University of Waikato 138
Performing experiments
Experimenter makes it easy to compare the performance of different learning schemes
For classification and regression problems
Results can be written into file or database
Evaluation options: cross-validation, learning curve, hold-out
Can also iterate over different parameter settings
Significance-testing built in!
2/22/2011 University of Waikato 139
2/22/2011 University of Waikato 140
2/22/2011 University of Waikato 141
2/22/2011 University of Waikato 142
2/22/2011 University of Waikato 143
2/22/2011 University of Waikato 144
2/22/2011 University of Waikato 145
2/22/2011 University of Waikato 146
2/22/2011 University of Waikato 147
2/22/2011 University of Waikato 148
2/22/2011 University of Waikato 149
2/22/2011 University of Waikato 150
2/22/2011 University of Waikato 151
2/22/2011 University of Waikato 152
The Knowledge Flow GUI
New graphical user interface for WEKA
Java-Beans-based interface for setting up and running machine learning experiments
Data sources, classifiers, etc. are beans and can be connected graphically
Data “flows” through components: e.g.,
“data source” -> “filter” -> “classifier” -> “evaluator”
Layouts can be saved and loaded again later
2/22/2011 University of Waikato 153
2/22/2011 University of Waikato 154
2/22/2011 University of Waikato 155
2/22/2011 University of Waikato 156
2/22/2011 University of Waikato 157
2/22/2011 University of Waikato 158
2/22/2011 University of Waikato 159
2/22/2011 University of Waikato 160
2/22/2011 University of Waikato 161
2/22/2011 University of Waikato 162
2/22/2011 University of Waikato 163
2/22/2011 University of Waikato 164
2/22/2011 University of Waikato 165
2/22/2011 University of Waikato 166
2/22/2011 University of Waikato 167
2/22/2011 University of Waikato 168
2/22/2011 University of Waikato 169
2/22/2011 University of Waikato 170
2/22/2011 University of Waikato 171
2/22/2011 University of Waikato 172
2/22/2011 University of Waikato 173
Conclusion: try it yourself!
WEKA is available at
http://www.cs.waikato.ac.nz/ml/weka
Also has a list of projects based on WEKA
WEKA contributors:
Abdelaziz Mahoui, Alexander K. Seewald, Ashraf M. Kibriya, Bernhard Pfahringer , Brent Martin, Peter Flach, Eibe Frank ,Gabi Schmidberger ,Ian H. Witten , J. Lindgren, Janice Boughton, Jason Wells, Len Trigg, Lucio de Souza Coelho, Malcolm Ware, Mark Hall ,Remco Bouckaert , Richard Kirkby, Shane Butler, Shane Legg, Stuart Inglis, Sylvain Roy, Tony Voyle, Xin Xu, Yong Wang, Zhihai Wang
![„The birth of Athenian democracy. The assembly in the fifth B. C.”, Chester G. Starr, 1990 : [recenzja]](data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACH5BAEAAAAALAAAAAABAAEAAAICRAEAOw==)