IDSSSeminar,7.12.2010 WojciechJaśkowskiKrzysztofKrawiec WybraneAspektyProblemówOpartychnaTestach

(1)

Wybrane Aspekty Wojciech Jaśkowski, Krzysztof Krawiec Test-Based Problems and Pareto-Coevolution Coordinate System Archive (COSA)

Experiment & Results

Practical Algorithms for “Practical” Problems Experiments on Dimensionality and Pareto Dominance Summary

Wybrane Aspekty Problemów

Opartych na Testach

Wojciech Jaśkowski Krzysztof Krawiec

Institute of Computing Science, Poznan University of Technology, Poland

(2)

Outline

Test-Based Problems and Pareto-Coevolution

Coordinate System Archive (COSA)

Practical Algorithms for “Practical” Problems

Experiments on Dimensionality and Pareto Dominance

(3)

Outline

Coordinate System Archive (COSA) Experiment & Results

(4)

Test-Based Problem

I G = (S,T ,G) that consists of:

I set S of solutions (a.k.a. candidate solutions),

I set T of tests,

I interaction function G : S_{× T → R.} I The codomain of G is a binary set{0,1}.

I G is a game; G is a payoff matrix I Goal: find the best solution in S

(5)

Coevolution and Pareto-Coevolution

(Competitive) Coevolution

I (Sometimes) successful

I (Sometimes) pathologies: cycling, stalling, forgetting, etc.

Pareto-Coevolution

I Treats every test as an objective

I Problem as a multiobjective optimization

t1 t2

s1

s2 s3

(6)

Coevolution and Pareto-Coevolution

(Competitive) Coevolution

I (Sometimes) successful

I (Sometimes) pathologies: cycling, stalling, forgetting, etc.

Pareto-Coevolution

I Treats every test as an objective

I Problem as a multiobjective optimization

t1

t2

s1

s2

(7)

Underlying Objectives

Motivation

I A huge number of objectives (_{|T |)}

I Q: Is it possible to reduce the number of objectives?

I Some tests examine the same skill/aspect of a solution but with different intensity

I Such tests could be grouped and put on a common axis I Axis = underlying objective,

teasytmediumthard

Example

Possible underlying objectives in chess

I How well the player: (i) controls the center of the board, (ii) uses knights, (iii) plays endgames, etc.

(8)

Underlying Objectives

Motivation

I A huge number of objectives (_{|T |)}

I Q: Is it possible to reduce the number of objectives?

I Some tests examine the same skill/aspect of a solution but with different intensity

I Such tests could be grouped and put on a common axis

I Axis = underlying objective,

teasytmediumthard

Example

Possible underlying objectives in chess

I How well the player: (i) controls the center of the board, (ii) uses knights, (iii) plays endgames, etc.

(9)

Coordinate System

Definition

(Bucci et al., 2004) The coordinate system for a game G is a set of axes (Ai), where:

I each axis Ai ⊆ T ,

I each axis Ai is linearly ordered by dominance relation.

I each solution s is placed on P(s).

Definition

The coordinate system is correct iff all relations are preserved, i.e.,s1≤ s2 ⇐⇒ P(s1)≤ P(s2)

(10)

Example: A Coordinate System for Nim [1,3]

t1 t2 t3 t4 t5 t6 t7 t8 t9 s1 1 1 1 1 1 1 s2 s3 1 1 1 1 1 1 1 1 1 s4 1 1 1 1 1 1 s5 1 1 1 s6 1 1 1 I Axes ordered by

increasing difficulty, e.g., (t9<t8<t2)

I All relations preserved

s1<s3=⇒ P(s1)< P(s3) _t9 _t8 _t2 t4 s1 s2 s3 s4 s5 s6

9 objectives/tests compressed to 2 underlying objectives/axes

(11)

Example: A Coordinate System for Nim [1,3]

t1 t2 t3 t4 t5 t6 t7 t8 t9 s1 1 1 1 1 1 1 s2 s3 1 1 1 1 1 1 1 1 1 s4 1 1 1 1 1 1 s5 1 1 1 s6 1 1 1 I Axes ordered by

increasing difficulty, e.g., (t9<t8<t2)

I All relations preserved

s1<s3=⇒ P(s1)< P(s3) _t9 _t8 _t2 t4 s1 s2 s3 s4 s5 s6

9 objectives/tests compressed to 2 underlying objectives/axes

(12)

Outline

(13)

Coordinate System Archive (COSA)

Archive.Submit(Snew, Tnew)

Prepare

I S_{← S}new∪ Sarch

I T _{← T}new∪ Tarch

I Sarch← Tarch← /0

Three steps:

1. Extract a correct coordinate system 2. Put into archive:

2.1 all non-dominated solutions 2.2 one test from each axis

(16)

Coordinate System Archive (COSA)

Archive.Submit(Snew, Tnew)

Prepare

I S_{← S}new∪ Sarch

I T _{← T}new∪ Tarch

I Sarch← Tarch← /0 Three steps:

1. Extract a correct coordinate system

2. Put into archive:

2.1 all non-dominated solutions

2.2 one test from each axis

(17)

Coordinate System Archive

Step 1: Extract a correct coordinate system

I Partition the partially ordered set (T,_{≤) into chains.}

c f a d b e g A1 A2 A3 a c _f b d g e {a<c <f_{} ∪ {}b<d_{} ∪ {}g <e_{} is a minimum chain} partition of (T,≤)

(18)

Coordinate System Archive

Step 2: Put into archive SPareto and Tbase

I Finding SPareto is easy.

I Tbase — take one test from each axis

c f a d b e g A1 A2 A3 a c _f b d g e

I Find an antichain in (T,≤) with 3 properties:

1. its size is equals the number of axes (chains) 2. there exists a solution that solves all its elements 3. it is the greatest antichain that fulfills1. and2.

(19)

Coordinate System Archive

1. its size is equals the number of axes (chains) 2. there exists a solution that solves all its elements 3. it is the greatest antichain that fulfills1. and2.

(20)

Coordinate System Archive

1. its size is equals the number of axes (chains)

2. there exists a solution that solves all its elements 3. it is the greatest antichain that fulfills1. and2.

(21)

Coordinate System Archive

2. there exists a solution that solves all its elements

(22)

Coordinate System Archive

2. there exists a solution that solves all its elements

(23)

Coordinate System Archive

Step 3: Make the archive stable

Archive is stable if all relations are preserved in SPareto and

in Tbase:

I solutions from SPareto must be mutually incomparable

I s1k s2 (in respect to T ) =⇒

s1k s2 (in respect to Tarch), s1, s2∈ SPareto

I tests from Tbase must be mutually incomparable.

I t1k t2 (in respect to S ) =⇒

(24)

Compare-on-one Game

I Artificial numbers game

I Strategies represented as real-number vectors s[1..d] and t[1..d]: I Interaction function: G (s, t) = ( 1 if s[m]_{≥ t[m]} 0 otherwise

I where m = arg maxit[i ]

s1

I Performance measure: the lowest value in the vector representing a solution.

(25)

Experiment Setup

I Comparison with different archives:

I Coordinated System Archive (COSA)

I Iterated Pareto-Coevolutionary Archive (IPCA)

I Layered Pareto-Coevolutionary Archive (LAPCA-3, LAPCA-5, LAPCA-10)

(26)

Results (CompareOnOne, d = 2)

0 10 20 30 P erformance (lo w est dimension) 0 2.5· 105 ₅_{· 10}5 _7.5_{· 10}5 ₁_{· 10}6 Number of interactions COSA IPCA LAPCA-3 LAPCA-5 LAPCA-10

(27)

Results (CompareOnOne, d = 3)

0 5 10 15 P erformance (lo w est dimension) 0 2.5· 105 ₅_{· 10}5 _7.5_{· 10}5 ₁_{· 10}6 Number of interactions COSA IPCA LAPCA-3 LAPCA-5 LAPCA-10

(28)

Results (CompareOnOne, d = 5)

0 1 2 3 4 P erformance (lo w est dimension) 0 2.5· 105 ₅_{· 10}5 _7.5_{· 10}5 ₁_{· 10}6 Number of interactions COSA IPCA LAPCA-3 LAPCA-5 LAPCA-10

(29)

Why it is so good?

1. It ’understands’ the underlying structure of a problem (one test from each objective)

2. Maintains only a small number of useful tests, thus less

interactions are required:

I It can evolve more generations in a given number of interactions. 0 100 200 300 400 Arc hiv e size (solutions) 0 2.5· 105 _{5· 10}5 _7.5_{· 10}5 ₁_{· 10}6 Number of interactions COSA IPCA LAPCA-3 LAPCA-5 LAPCA-10 0 50 100 150 Arc hiv e size (tests) 0 2.5· 105 ₅_{· 10}5 _7.5_{· 10}5 ₁_{· 10}6 Number of interactions COSA IPCA LAPCA-3 LAPCA-5 LAPCA-10

(30)

Outline

(31)

Experiment & Results Practical Algorithms for “Practical” Problems Experiments on Dimensionality and Pareto Dominance Summary

Research questions

I Do algorithms that work for compare-on-one or compare-on-all, work good on real problems?

I How many dimensions do real problems have?

(32)

TicTacToe Problem

Tic Tac Toe:

I ca. 600 different states (250 + 350)

I each state: several possible moves

I strategy encoding: direct (certain move in certain state)

I binary version: treat draw as win

I Goal: a strategy which wins against most other strategies

I TTT-X (looking for best X) vs. TTT-O (looking for best O)

(33)

(Archive) Algorithms

Methods: I Coordinate-system-based I COSA I Pareto-domination-based:

I LAPCA-N (N Pareto-layers of solutions + test to “stabilize”)

I IPCA (Increases its knowledge, with guarantee of monotonicity)

I Simple:

I MaxSolve-N (N best solutions + all tests that are solved by any solution)

(34)

How algorithms work for TTT-X?

0.75 0.8 0.85 0.9 0.95 1 P erformance (exp ected utilit y) 0 2.5· 105 ₅ · 105 _7.5 · 105 ₁ · 106 Number of evaluations LAPCA-2 LAPCA-3 LAPCA-5 LAPCA-10 IPCA COSA IPCA200 Empty Conclusions:

I Archives are neccessary

I LAPCA the best (2 and 3 layers)

I IPCA very bad (archive too big?), COSA very bad (too many dimensions?)

(35)

How algorithms work for TTT-O?

0.4 0.6 0.8 1 P erformance (exp ected utilit y) 0 2.5· 105 ₅_{· 10}5 _{7.5· 10}5 ₁_{· 10}6 Number of evaluations MaxSolve-3 MaxSolve-5 MaxSolve-10 MaxSolve-20 MaxSolve-50 MaxSolve-100 MaxSolve-200 LAPCA-2G LAPCA-3G LAPCA-5G LAPCA-10G LAPCA-2S LAPCA-3S LAPCA-5S LAPCA-10S IPCA COSA IPCA200 Empty

(36)

How algorithms work for TTT-O?

(continued)

Method Expected Utility Number of generations MaxSolve-50 0.837+-0.050 257.920+-17.297 MaxSolve-100 0.778+-0.037 127.040+-14.526 MaxSolve-20 0.702+-0.056 587.240+-13.234 IPCA 0.688+-0.019 192.000+-16.882 MaxSolve-200 0.687+-0.051 84.640+-3.665 LAPCA-10 0.655+-0.050 70.240+-0.907 LAPCA-3 0.651+-0.050 71.720+-0.722 LAPCA-5 0.650+-0.054 70.200+-0.849 LAPCA-2 0.649+-0.042 77.160+-1.405 MaxSolve-10 0.627+-0.072 861.200+-6.776 MaxSolve-5 0.577+-0.052 1036.960+-2.749 MaxSolve-3 0.538+-0.064 1101.520+-2.700 COSA 0.515+-0.027

(37)

How algorithms work for TTT-O?

Conclusions

Conclusions:

I TTT-O is harder than TTT-X

I IPCA > LAPCA (probably, due to more generations)

I MaxSolve significantly> LAPCA

I The simplest MaxSolve is the best (surpise!)

I However, the archive size matters I COSA is the worst

(38)

CESAR

CESAR-MX (Complete Evaluation Set Archive):

I Maintains only X solutions that solve the most tests from the archive (like MaxSolve)

I Maintains all tests that are required to explain any relations between solutions (like LAPCA).

(39)

CESAR on TTT-O

0.4 0.6 0.8 1 P erformance (exp ected utilit y) 0 2.5· 106 ₅_{· 10}6 _7.5_{· 10}6 ₁_{· 10}7 Number of evaluations LAPCA-2S MaxSolve-20 MaxSolve-50 MaxSolve-100 CESAR CESAR-M1 CESAR-M2 CESAR-M5 CESAR-M10 CESAR-M20 CESAR-M50 CESAR-M100

(40)

CESAR on TTT-O

Method ArchExpUtilMax Probab. of 1.0 CESAR-M50 0.995+-0.010 68.0% MaxSolve-100 0.991+-0.011 32.0% CESAR-M10 0.987+-0.018 60.0% CESAR-M2 0.986+-0.025 64.0% CESAR-M5 0.986+-0.019 56.0% CESAR-M1 0.985+-0.030 64.0% CESAR-M100 0.981+-0.027 36.0% CESAR-M20 0.979+-0.028 52.0% MaxSolve-50 0.943+-0.027 0.0% LAPCA-2S 0.753+-0.025 0.0% MaxSolve-20 0.708+-0.063 0.0%

(41)

CESAR on TTT-O

Conclusions:

I CESAR maintains progress well

I Generally, CESAR-M is not sensitive to its parameter.

I CESAR-M is consistently better than MaxSolve especially when take into consideration MaSolve with smaller archives like 20, which simply does not work.

I MaxSolve-50 does quite a good job, but, generally, MaxSolve is very sensitive to its parameter.

I MaxSolve-100 is nearly as good as CESAR-50 in the long run, however it learns more slowly than CESAR.

I Moreover, it is not as consistent as CESAR in holding the best solution that is why its probability of 1.0 is so small.

(42)

More CESARs on TTT-O

Method ArchExpUtilMax Prob of 1.0 CESAR3-10-T50-new 1.000+-0.000 100% CESAR3-10-T50-new150 1.000+-0.000 100% CESAR3-10-T50-new300 1.000+-0.000 100% CESAR3-10-T50 0.805+-0.076 0% CESAR3-10-T100 1.000+-0.000 100%

(43)

Outline

(44)

Dimensionality of a random game

I A random game (n× n payoff matrix, n = 2...360)

0 5 10 15 20 25 30 value 0 50 100 150 200 250 300 350 problem size SmartExactWithTimeout1.0 SmartExact GreedyHeuristics BucciHeuristics y(x) = 4.48 * log(x) - 2.43

(45)

Example: Density Classification Task

0 1 1 0 1 0 0 1 0

1 0 1 1 0 0 0 0 0

2r + 1

n

Density Classification Task

I S — set of all CA rules

I T — set of all initial configurations

I G (s, t) — does s converge to the majority from the initial configuration t?

(46)

Dimensionality of DCT-59-3

I Moderate compression, but dimensino linear w.r.t. size of the problem

(47)

Dimensionality of TTT-X

I (Estimated) dimensionality of TTT-X:12.621∗ ln(2300_{) = 2200.}

(48)

Dimensionality of TTT-X

(49)

Pareto dominance in DCT-59-3

Goal:

I Check whether we have a Pareto dominance in our problems at all.

Method:

I Save all solutions and tests from a single run:

I MaxSolve-20

I Population: 20x20

I _{MaxGenerations = 500 (20*500 = 10000 strategies)} I Create a payoff matrix 10000x10000 and compute

(50)

Results

I #Dominations: 1164644 (4.8% of all possible relations)

(51)

Conclusions

I The number of layers is moderate

I There are a lot non dominated individuals (2619)

I In the Pareto front we have a variety of solutions: expected utility: <0, 0.712>

I Generally the layer number does not mean much in terms of absolute performance

(52)

Pareto dominance in TTT-O (Results)

Layer# #Elements MinExpUtil AvgExpUtil MaxExpUtil

0 1 1 1 1 1 1062 0.295 0.645 0.992 2 4438 0.247 0.685 0.984 3 3293 0.371 0.721 0.982 4 2201 0.327 0.726 0.974 5 1094 0.352 0.736 0.966 6 583 0.424 0.772 0.949 7 178 0.446 0.783 0.932 8 33 0.551 0.79 0.91 9 4 0.487 0.669 0.793

I #Dominations: 89187 (0.11% of all possible)

(53)

Conclusions

I The number of layers is very small. Smaller than in

case of DCT-59-3. Especially when we take into

account the fact that in TTT we got all solutions (from very bad to the optimal one)

I The % of domination is much lower than in DCT-59-3-320. (0.11% compared to 4.8%).

(54)

Outline

(55)

Final Conclusions

I (Currently) no evidence that CS-based coevolutionary algorithms are better than simple algorithms on practical problems.

I The number of axes of a problem depends on the problem itself:

I The dimension for some problems could be managable (log? on TTT-X): a hope for CS-based algorithms,

I but for others (DCT-59-3) not.

I Comparing algorithms on artificial problems such as 2D or 3D compare-on-one/compare-on-all is worthless

I _{a need for new benchmark problems}

I “practical” problems have different properties (COSA vs. LAPCA on TTT vs. CompareOnOne)

I Simple, aggregating algorithms, are currently best (MaxSolve, CESAR)

I The density of domination and #pareto layers depends:

I Quite common on DCT-59-3 (4.8%) (surprise!), 20 layers