Prole management

management Experiments

Datasets Machines Results Conclusions References

Prole management

I The functionality of the prole manager

 decides about the shape of the prole, that is when and how the prole is modied,

 calculates prole similarities for ranking generation,

 manages the knowledge base.

I Example prole manager parameters:

 the number of congurationresult pairs to keep in the prole,

 the strategy of determining the congurations to remove from the prole, when new conguration is provided and the prole has already reached its destination size.

I Needed research:

 What size of the prole is optimal?

 Which congurations should be kept?

 How to measure prole similarity?

25 / 42

Meta-uczenie z analiz¡

proli Krzysztof Gr¡bczewski ML survey

Rankings PBML

Validated rankings Proles The algorithm Prole management Experiments

Datasets Machines Results Conclusions References

Experiments  the knowledge base

I To test the PBML framework, a nontrivial knowledge base had to be available.

I Results from another research task on DT CV Committees (Gr¡bczewski, 2013).

I 21 UCI datasets.

I 13660 machine congurations:

 13560 dierent settings of cross-validation committees.

 100 parameter settings of single DT induction methods I 10×10-fold CV results for all congurations.

Meta-uczenie z analiz¡

proli Krzysztof Gr¡bczewski ML survey

Rankings PBML

Validated rankings Proles The algorithm Prole

Experiments datasets

Dataset classes instances features ordered f.

appendicitis 2 106 7 7

Australian credit 2 690 14 6

breast cancer (Wisconsin) 2 699 9 9

ag 8 194 28 10

glass 6 214 9 9

heart 2 303 13 13

image 7 2310 19 19

ionosphere (trn+tst) 2 351 34 34

iris 3 150 4 4

kr-vs-kp 2 3196 36 0

Ljubjlana breast cancer 2 286 9 1

letter recognition 26 20000 16 16

Pima indians diabetes 2 768 8 8

sonar 2 208 60 60

soybean large 19 307 35 0

splice 3 3190 60 0

thyroid (trn+tst) 3 7200 21 6

vote 2 435 16 0

vowel 6 871 3 3

waveform 3 5000 21 21

wine 3 178 13 13 _{27 / 42}

Meta-uczenie z analiz¡

proli Krzysztof Gr¡bczewski ML survey

Rankings PBML

Validated rankings Proles The algorithm Prole management Experiments

Datasets Machines Results Conclusions References

Experiments  learning machines

13560 DT CV Committees congurations

I 4 DT induction algorithms (Gini index, information gain, QUEST, SSV),

I committee size in the range from 1 to 10 (10-fold CV-based validation),

I 6 DT validation methods: Reduced Error Pruning (REP), cost-complexity (CC), degree-based pruning, OPTimal pruning, Minimum Error Pruning 2 and Depth Impurity, I respecting standard error: 0SE, .5SE, 1SE, and estimated

from sample .5SE and 1SE, I training error factor: 0, 0.5, 1,

I common or separate parameter optimization, I decision making by: proportions, Laplace correction,

m-estimates.

Meta-uczenie z analiz¡

proli Krzysztof Gr¡bczewski ML survey

Rankings PBML

Validated rankings Proles The algorithm Prole management Experiments

Datasets Machines Results Conclusions References

Experiments  learning machines

100 parameter settings of single DT induction methods I 4 DT induction algorithms (Gini index, information gain,

QUEST, SSV),

I 6 DT validation methods: Reduced Error Pruning (REP), cost-complexity (CC), degree-based pruning, OPTimal pruning, Minimum Error Pruning 2 and Depth Impurity, I Respecting standard error: 0SE, .5SE, 1SE, and

estimated from sample .5SE and 1SE,

29 / 42

Meta-uczenie z analiz¡

proli Krzysztof Gr¡bczewski ML survey

Rankings PBML

Validated rankings Proles The algorithm Prole management Experiments

Datasets Machines Results Conclusions References

Experiments  PBML conguration

I Proles of variable size (full proles, up to 100 congurations).

I Proles updated after each 5 congurations.

I Prole similarity measure  Pearson linear correlation coecient (truncated to 0 if negative)

I First ranking on the basis of average p-values (4).

I Weighted p-values, when prole with at least 2 results:

W P V (c)← X

D∈KB

M ax(0, CC(P, D))∗ P V (c, D), (5) where

 D∈ KB means dataset D in the knowledge base,

 CC(P, D)is the Pearson linear correlation coecient,

 P V (c, D)is the p-value obtained in paired t-test.

Meta-uczenie z analiz¡

proli Krzysztof Gr¡bczewski ML survey

Rankings PBML

Validated rankings Proles The algorithm Prole management Experiments

Datasets Machines Results Conclusions References

Experiment 1  PBML vs passive rankings

I PBML algorithm compared with 5 ranking methods:

 completely random,

 average accuracy (1),

 average accuracy dierence in st. deviations (2),

 average ranks (3),

 average p-values (4).

I Leave-one-out procedure for the 21 datasets.

I The most important aspect: what maximum validation accuracy can be obtained in given time.

I Time unit ≈ the number of congurations validated so far.

I Rankings of 100 congurations visualized as:

 maximum accuracy till given time,

 average of 3 maximum accuracies till given time,

 average accuracy dierence,

 average mean accuracy till given time.

31 / 42

Meta-uczenie z analiz¡

proli Krzysztof Gr¡bczewski ML survey

Rankings PBML

Validated rankings Proles The algorithm Prole management Experiments

Datasets Machines Results Conclusions References

Experiment 1 results I

Maxima found till given time

1 100

-0.7999 -0.0896

0.0000

random av. rank

av. accuracy av. p-value av. acc. diff. PBML

Means of 3 maximal results till given time

1 100

-1.0228 -0.1120

0.0000

Meta-uczenie z analiz¡

proli Krzysztof Gr¡bczewski ML survey

Rankings PBML

Validated rankings Proles The algorithm Prole management Experiments

Datasets Machines Results Conclusions References

Experiment 1 results II

Accuracy dierence

1 100

-2.2147 -0.2684

0.0000

Means of all results till given time

1 100

-1.0228 -0.4149

0.0000

33 / 42

Meta-uczenie z analiz¡

proli Krzysztof Gr¡bczewski ML survey

Rankings PBML

Validated rankings Proles The algorithm Prole management Experiments

Datasets Machines Results Conclusions References

Experiment 2  passive vs kNN vs active

I Passive rankings can be signicantly improved by averaging over NNs.

 kNN analysis by landmarking with selected machines of the examined population.

 5NN with Euclidean distance used here.

I As before, 4 ranking measures:

 average accuracy (1),

 average accuracy dierence in st. deviations (2),

 average ranks (3),

 average p-values (4).

I PBML framework also suitable for passive methods with kNN.

I Three versions of each ranking method:

 passive,

 passive with kNN selection,

 active.

Meta-uczenie z analiz¡

proli Krzysztof Gr¡bczewski ML survey

Rankings PBML

Validated rankings Proles The algorithm Prole management Experiments

Datasets Machines Results Conclusions References

Experiment 2 results I

Maxima found till given time

1 100

-0.7999 -0.0625

0.0000

av. accuracy random av. acc. diff. passive av. rank with kNN av. p-value profiles

Means of 3 maximal results till given time

1 100

-1.0228 -0.0737

0.0000

35 / 42

Meta-uczenie z analiz¡

proli Krzysztof Gr¡bczewski ML survey

Rankings PBML

Validated rankings Proles The algorithm Prole management Experiments

Datasets Machines Results Conclusions References

Experiment 2 results II

Accuracy dierence

1 100

-2.2147 -0.1535

0.0000

Means of all results till given time

1 100

-1.0228 -0.2526

0.0000

Meta-uczenie z analiz¡

proli Krzysztof Gr¡bczewski ML survey

Rankings PBML

Validated rankings Proles The algorithm Prole management Experiments

Datasets Machines Results

W dokumencie Warszawa, 13 listopada 2013 (Stron 25-37)