Scale-invariant online learning

Scale-invariant online learning

Micha l Kempka Wojciech Kot lowski

IDSS Seminar, 27.11.2018

1 / 22

Online learning example: travel time estimation

• At every timestamp t, navigation software needs to predict travel time y

at a given road segment

• Given feature vector x

∈ R

representing current traffic conditions, predict b y

= x

w

with a linear model

• Observe real y

and measure prediction loss, e.g. (y

− y b

)

• Improve model parameters w

→ w

2 / 22

Online learning example: spam filtering

• At every timestamp t, spam filter needs to classify an incoming email as spam/no-spam (y

∈ {+1, −1})

• Given feature vector x

∈ R

representing email’s body, predict b y

= x

w

with a linear model

• Receive feedback y

from a user and measure prediction loss, e.g. logistic loss log(1 + e

)

• Improve model parameters w

→ w

3 / 22

Online learning with linear models

At each trial t = 1, . . . , T :

Nature reveals input instance x

∈ R

Learner predicts with a linear model y b

= x

w

, where w

∈ R

Nature reveals label y

Learner suffers loss `(y

, b y

)

, convex and L-Lipschitz in y b

revealed before prediction!

L-Lipschitz = (sub)derivative bounded by L Loss function `(y, y) b ∂

by

`(y, y) b L logistic log 1 + e

1+e^yby

1 hinge max{0, 1 − y y} b −y1[y b y ≤ 1] 1 absolute | b y − y| sgn( y − y) b 1

Without loss of generality assume L = 1

No stochastic assumptions on the data sequence (x

, y

) are made Minimize regret relative to oracle weight vector w

∈ R

:

regret

(w

) =

T

X

t=1

`(y

, x

w

) −

T

X

t=1

`(y

, x

w

),

Goal: sublinear regret for any w

and any data sequence (x

, y

)

4 / 22

Online learning with linear models

At each trial t = 1, . . . , T :

Nature reveals input instance x

∈ R

Learner predicts with a linear model y b

= x

w

, where w

∈ R

Nature reveals label y

Learner suffers loss `(y

, b y

)

, convex and L-Lipschitz in y b

revealed before prediction!

L-Lipschitz = (sub)derivative bounded by L Loss function `(y, y) b ∂

by

`(y, y) b L logistic log 1 + e

1+e^yby

1 hinge max{0, 1 − y y} b −y1[y b y ≤ 1] 1 absolute | b y − y| sgn( y − y) b 1

Without loss of generality assume L = 1

No stochastic assumptions on the data sequence (x

, y

) are made Minimize regret relative to oracle weight vector w

∈ R

:

regret

(w

) =

T

X

t=1

`(y

, x

w

) −

T

X

t=1

`(y

, x

w

),

Goal: sublinear regret for any w

and any data sequence (x

, y

)

4 / 22

Online learning with linear models

At each trial t = 1, . . . , T :

Nature reveals input instance x

∈ R

Learner predicts with a linear model y b

= x

w

, where w

∈ R

Nature reveals label y

Learner suffers loss `(y

, b y

), convex and L-Lipschitz in y b

revealed before prediction!

L-Lipschitz = (sub)derivative bounded by L Loss function `(y, y) b ∂

by

`(y, y) b L logistic log 1 + e

1+e^yby

1 hinge max{0, 1 − y y} b −y1[y b y ≤ 1] 1 absolute | b y − y| sgn( y − y) b 1

Without loss of generality assume L = 1

No stochastic assumptions on the data sequence (x

, y

) are made Minimize regret relative to oracle weight vector w

∈ R

:

regret

(w

) =

T

X

t=1

`(y

, x

w

) −

T

X

t=1

`(y

, x

w

),

Goal: sublinear regret for any w

and any data sequence (x

, y

)

4 / 22

Online learning with linear models

At each trial t = 1, . . . , T :

Nature reveals input instance x

∈ R

Learner predicts with a linear model y b

= x

w

, where w

∈ R

Nature reveals label y

Learner suffers loss `(y

, b y

), convex and L-Lipschitz in y b

revealed before prediction!

L-Lipschitz = (sub)derivative bounded by L Loss function `(y, y) b ∂

by

`(y, y) b L logistic log 1 + e

1+e^yby

1 hinge max{0, 1 − y y} b −y1[y b y ≤ 1] 1 absolute | b y − y| sgn( y − y) b 1

Without loss of generality assume L = 1

No stochastic assumptions on the data sequence (x

, y

) are made Minimize regret relative to oracle weight vector w

∈ R

:

regret

(w

) =

T

X

t=1

`(y

, x

w

) −

T

X

t=1

`(y

, x

w

),

Goal: sublinear regret for any w

and any data sequence (x

, y

)

4 / 22

Stochastic Gradient Descent (SGD)

• Fixed learning rate

w

= w

− η∇

, where ∇

= ∇

`(y

, x

w

)

Make a small step along negative gradient of the loss

regret

(w

) ≤ kw

k

2η + η

2

T

X

t=1

k∇

k

(starting at w

= 0)

• Separate fixed learning rate per feature w

= w

− η

∇

, regret

(w

) ≤

d

X

i=1

 w

2η

+ η

2

T

X

t=1

∇



(w

= 0)

• Adaptive learning rate per feature (AdaGrad [Duchi et al., 2011]) w

= w

− η

∇

, where η

= η

q

 + P

j≤t

∇

regret

(w

) ≤

d

X

i=1

 max

|w

− w

|

2η

+η

 s

 + X

t

∇

5 / 22

Stochastic Gradient Descent (SGD)

• Fixed learning rate

w

= w

− η∇

, where ∇

= ∇

`(y

, x

w

) regret

(w

) ≤ kw

k

2η + η 2

T

X

t=1

k∇

k

(starting at w

= 0)

• Separate fixed learning rate per feature w

= w

− η

∇

, regret

(w

) ≤

d

X

i=1

 w

2η

+ η

2

T

X

t=1

∇



(w

= 0)

• Adaptive learning rate per feature (AdaGrad [Duchi et al., 2011]) w

= w

− η

∇

, where η

= η

q

 + P

j≤t

∇

regret

(w

) ≤

d

X

i=1

 max

|w

− w

|

2η

+η

 s

 + X

t

∇

5 / 22

Stochastic Gradient Descent (SGD)

• Fixed learning rate

w

= w

− η∇

, where ∇

= ∇

`(y

, x

w

) regret

(w

) ≤ kw

k

2η + η 2

T

X

t=1

k∇

k

(starting at w

= 0)

Optimal in-hindsight tuning η

= √

P

tk∇_tk²

to minimize the regret (impossible in practice)

• Separate fixed learning rate per feature w

= w

− η

∇

, regret

(w

) ≤

d

X

i=1

 w

2η

+ η

2

T

X

t=1

∇



(w

= 0)

• Adaptive learning rate per feature (AdaGrad [Duchi et al., 2011]) w

= w

− η

∇

, where η

= η

q

 + P

j≤t

∇

regret

(w

) ≤

d

X

i=1

 max

|w

− w

|

2η

+η

 s

 + X

t

∇

5 / 22

Stochastic Gradient Descent (SGD)

• Fixed learning rate

w

= w

− η∇

, where ∇

= ∇

`(y

, x

w

) regret

(w

) ≤ kw

k

s X

t

k∇

k

for η

= kw

k pP

t

k∇

k

• Separate fixed learning rate per feature w

= w

− η

∇

, regret

(w

) ≤

d

X

i=1

 w

2η

+ η

2

T

X

t=1

∇



(w

= 0)

• Adaptive learning rate per feature (AdaGrad [Duchi et al., 2011]) w

= w

− η

∇

, where η

= η

q

 + P

j≤t

∇

regret

(w

) ≤

d

X

i=1

 max

|w

− w

|

2η

+η

 s

 + X

t

∇

5 / 22

Stochastic Gradient Descent (SGD)

• Fixed learning rate

w

= w

− η∇

, where ∇

= ∇

`(y

, x

w

) regret

(w

) ≤ kw

k

s X

t

k∇

k

for η

= kw

k pP

t

k∇

k

• Separate fixed learning rate per feature w

= w

− η

∇

,

Each feature has its own learning rate

regret

(w

) ≤

d

X

i=1

 w

2η

+ η

2

T

X

t=1

∇



(w

= 0)

• Adaptive learning rate per feature (AdaGrad [Duchi et al., 2011]) w

= w

− η

∇

, where η

= η

q

 + P

j≤t

∇

regret

(w

) ≤

d

X

i=1

 max

|w

− w

|

2η

+η

 s

 + X

t

∇

5 / 22

Stochastic Gradient Descent (SGD)

• Fixed learning rate

w

= w

− η∇

, where ∇

= ∇

`(y

, x

w

) regret

(w

) ≤ kw

k

s X

t

k∇

k

for η

= kw

k pP

t

k∇

k

• Separate fixed learning rate per feature w

= w

− η

∇

, regret

(w

) ≤

d

X

i=1

 w

2η

+ η

2

T

X

t=1

∇



(w

= 0)

• Adaptive learning rate per feature (AdaGrad [Duchi et al., 2011]) w

= w

− η

∇

, where η

= η

q

 + P

j≤t

∇

regret

(w

) ≤

d

X

i=1

 max

|w

− w

|

2η

+η

 s

 + X

t

∇

5 / 22

Stochastic Gradient Descent (SGD)

• Fixed learning rate

w

= w

− η∇

, where ∇

= ∇

`(y

, x

w

) regret

(w

) ≤ kw

k

s X

t

k∇

k

for η

= kw

k pP

t

k∇

k

• Separate fixed learning rate per feature w

= w

− η

∇

, regret

(w

) ≤

d

X

i=1

 w

2η

+ η

2

T

X

t=1

∇



(w

= 0)

Optimal in-hindsight tuning η

= √

t∇²_t,i

• Adaptive learning rate per feature (AdaGrad [Duchi et al., 2011]) w

= w

− η

∇

, where η

= η

q

 + P

j≤t

∇

regret

(w

) ≤

d

X

i=1

 max

|w

− w

|

2η

+η

 s

 + X

t

∇

5 / 22

Stochastic Gradient Descent (SGD)

• Fixed learning rate

w

= w

− η∇

, where ∇

= ∇

`(y

, x

w

) regret

(w

) ≤ kw

k

s X

t

k∇

k

for η

= kw

k pP

t

k∇

k

• Separate fixed learning rate per feature w

= w

− η

∇

, regret

(w

) ≤

d

X

i=1



|w

| s

X

t

∇



for η

= |w

| q P

t

∇

Better than the previous bound (single tuning per feature)

• Adaptive learning rate per feature (AdaGrad [Duchi et al., 2011]) w

= w

− η

∇

, where η

= η

q

 + P

j≤t

∇

regret

(w

) ≤

d

X

i=1

 max

|w

− w

|

2η

+η

 s

 + X

t

∇

5 / 22

Stochastic Gradient Descent (SGD)

• Fixed learning rate

w

= w

− η∇

, where ∇

= ∇

`(y

, x

w

) regret

(w

) ≤ kw

k

s X

t

k∇

k

for η

= kw

k pP

t

k∇

k

• Separate fixed learning rate per feature w

= w

− η

∇

, regret

(w

) ≤

d

X

i=1



|w

| s

X

t

∇



for η

= |w

| q P

t

∇

Can we get the optimal SGD regret bound:

d

X

i=1



|w

| s

X

t

∇



with some adaptive tuning strategy?

• Adaptive learning rate per feature (AdaGrad [Duchi et al., 2011]) w

= w

− η

∇

, where η

= η

q

 + P

j≤t

∇

regret

(w

) ≤

d

X

i=1

 max

|w

− w

|

2η

+η

 s

 + X

t

∇

5 / 22

Stochastic Gradient Descent (SGD)

• Fixed learning rate

w

= w

− η∇

, where ∇

= ∇

`(y

, x

w

) regret

(w

) ≤ kw

k

s X

t

k∇

k

for η

= kw

k pP

t

k∇

k

• Separate fixed learning rate per feature w

= w

− η

∇

, regret

(w

) ≤

d

X

i=1



|w

| s

X

t

∇



for η

= |w

| q P

t

∇

• Adaptive learning rate per feature (AdaGrad [Duchi et al., 2011]) w

= w

− η

∇

, where η

= η

q

 + P

j≤t

∇

Tuning the learning rate mimics

the optimal tuning

regret

(w

) ≤

d

X

i=1

 max

|w

− w

|

2η

+η

 s

 + X

t

∇

5 / 22

Stochastic Gradient Descent (SGD)

• Fixed learning rate

w

= w

− η∇

, where ∇

= ∇

`(y

, x

w

) regret

(w

) ≤ kw

k

s X

t

k∇

k

for η

= kw

k pP

t

k∇

k

• Separate fixed learning rate per feature w

= w

− η

∇

, regret

(w

) ≤

d

X

i=1



|w

| s

X

t

∇



for η

= |w

| q P

t

∇

• Adaptive learning rate per feature (AdaGrad [Duchi et al., 2011]) w

= w

− η

∇

, where η

= η

q

 + P

j≤t

∇

regret

(w

) ≤

d

X

i=1

 max

|w

− w

|

2η

+η

 s

 + X

t

∇

5 / 22

Stochastic Gradient Descent (SGD)

• Fixed learning rate

w

= w

− η∇

, where ∇

= ∇

`(y

, x

w

) regret

(w

) ≤ kw

k

s X

t

k∇

k

for η

= kw

k pP

t

k∇

k

• Separate fixed learning rate per feature w

= w

− η

∇

, regret

(w

) ≤

d

X

i=1



|w

| s

X

t

∇



for η

= |w

| q P

t

∇

• Adaptive learning rate per feature (AdaGrad [Duchi et al., 2011]) w

= w

− η

∇

, where η

= η

q

 + P

j≤t

∇

regret

(w

) ≤

d

X

i=1

 max

|w

− w

|

2η

+η

 s

 + X

t

∇

Not there yet: still requires to tune η

depending on unknown w

!

5 / 22

Feature scales

w

= w

− η

∇

By the chain rule ∇

= ∇

`(y

, x

w

) = ∂`(y

, y) b

∂ y b

| {z }

gt

byt=x^>_t wt

x

:

w

= w

− η

g

x

Suppose feature i has a physical unit [X

], while the label and prediction are dimensionless (like in, e.g., classification)

=⇒ i-th weight coordinate w

must have unit 1/[X

]

1/[X

] 1/[X

]

dimensionless [X

]

Learning rate should compensate units on each coordinate! (in fact, the optimal in-hindsight tuning η

=

P

t∇²_t,i

achieves exactly that) Single learning rate is unable to compensate units.

6 / 22

Feature scales

w

= w

− η

∇

By the chain rule ∇

= ∇

`(y

, x

w

) = ∂`(y

, b y)

∂ y b

| {z }

gt

byt=x^>_t wt

x

:

w

= w

− η

g

x

For example, for squared-error loss:

∇

(y

− w

x

)

= 2(y

− w

x

)

| {z }

gt

x

Suppose feature i has a physical unit [X

], while the label and prediction are dimensionless (like in, e.g., classification)

=⇒ i-th weight coordinate w

must have unit 1/[X

]

1/[X

] 1/[X

]

dimensionless [X

]

Learning rate should compensate units on each coordinate! (in fact, the optimal in-hindsight tuning η

=

P

t∇²_t,i

achieves exactly that) Single learning rate is unable to compensate units.

6 / 22

Feature scales

w

= w

− η

∇

By the chain rule ∇

= ∇

`(y

, x

w

) = ∂`(y

, b y)

∂ y b

| {z }

gt

byt=x^>_t wt

x

:

w

= w

− η

g

x

Suppose feature i has a physical unit [X

], while the label and prediction are dimensionless (like in, e.g., classification)

=⇒ i-th weight coordinate w

must have unit 1/[X

]

1/[X

] 1/[X

]

dimensionless [X

]

Learning rate should compensate units on each coordinate! (in fact, the optimal in-hindsight tuning η

=

P

t∇²_t,i

achieves exactly that) Single learning rate is unable to compensate units.

6 / 22

Feature scales

w

= w

− η

∇

By the chain rule ∇

= ∇

`(y

, x

w

) = ∂`(y

, b y)

∂ y b

| {z }

gt

byt=x^>_t wt

x

:

w

= w

− η

g

x

Suppose feature i has a physical unit [X

], while the label and prediction are dimensionless (like in, e.g., classification)

=⇒ i-th weight coordinate w

must have unit 1/[X

] 1/[X

] 1/[X

]

dimensionless [X

]

units do not match!

Learning rate should compensate units on each coordinate! (in fact, the optimal in-hindsight tuning η

=

P

t∇²_t,i

achieves exactly that) Single learning rate is unable to compensate units.

6 / 22

Feature scales

w

= w

− η

∇

By the chain rule ∇

= ∇

`(y

, x

w

) = ∂`(y

, b y)

∂ y b

| {z }

gt

byt=x^>_t wt

x

:

w

= w

− η

g

x

Suppose feature i has a physical unit [X

], while the label and prediction are dimensionless (like in, e.g., classification)

=⇒ i-th weight coordinate w

must have unit 1/[X

] 1/[X

] 1/[X

]

dimensionless [X

]

. . . unless [η

] = 1/[X

]

Learning rate should compensate units on each coordinate! (in fact, the optimal in-hindsight tuning η

=

P

t∇²_t,i

achieves exactly that) Single learning rate is unable to compensate units.

6 / 22

Feature scales

w

= w

− η

∇

By the chain rule ∇

= ∇

`(y

, x

w

) = ∂`(y

, b y)

∂ y b

| {z }

gt

byt=x^>_t wt

x

:

w

= w

− η

g

x

Suppose feature i has a physical unit [X

], while the label and prediction are dimensionless (like in, e.g., classification)

=⇒ i-th weight coordinate w

must have unit 1/[X

] 1/[X

] 1/[X

]

dimensionless [X

]

. . . unless [η

] = 1/[X

]

Learning rate should compensate units on each coordinate! (in fact, the optimal in-hindsight tuning η

=

? i| qP

t∇²_t,i

achieves exactly that) Single learning rate is unable to compensate units.

6 / 22

Feature scales

AdaGrad [Duchi et al., 2011]:

w

= w

− η

q

 + P

j≤t

∇

∇

1/[X

] 1/[X

] [X

]

[X

] 1/[X

] ?

Learning rate still needs to compensate units, but cannot do so for all coordinates at the same time

• Also applies to RMSprop [Tieleman and Hinton, 2012] and Adam [Kingma and Ba, 2014]

• Heuristically solved by Adadelta [Zeiler, 2012]

Motivation: fully adaptive algorithms need to resolve this scaling issue

7 / 22

Feature scales

AdaGrad [Duchi et al., 2011]:

w

= w

− η

q

 + P

j≤t

∇

∇

1/[X

] 1/[X

] [X

]

[X

] 1/[X

] ?

Learning rate still needs to compensate units, but cannot do so for all coordinates at the same time

• Also applies to RMSprop [Tieleman and Hinton, 2012] and Adam [Kingma and Ba, 2014]

• Heuristically solved by Adadelta [Zeiler, 2012]

Motivation: fully adaptive algorithms need to resolve this scaling issue

7 / 22

Feature scales

AdaGrad [Duchi et al., 2011]:

w

= w

− η

q

 + P

j≤t

∇

∇

1/[X

] 1/[X

] [X

]

[X

] 1/[X

] ?

Learning rate still needs to compensate units, but cannot do so for all coordinates at the same time

• Also applies to RMSprop [Tieleman and Hinton, 2012] and Adam [Kingma and Ba, 2014]

• Heuristically solved by Adadelta [Zeiler, 2012]

Motivation: fully adaptive algorithms need to resolve this scaling issue

7 / 22

Feature scales

AdaGrad [Duchi et al., 2011]:

w

= w

− η

q

 + P

j≤t

∇

∇

1/[X

] 1/[X

] [X

]

[X

] 1/[X

] ?

Learning rate still needs to compensate units, but cannot do so for all coordinates at the same time

• Also applies to RMSprop [Tieleman and Hinton, 2012] and Adam [Kingma and Ba, 2014]

• Heuristically solved by Adadelta [Zeiler, 2012]

Motivation: fully adaptive algorithms need to resolve this scaling issue

7 / 22

Scale invariance

A natural symmetry in the linear problems

Rescaling the features followed by the inverse scaling of the weights keep the predictions (and hence losses) invariant:

∀i, t x

7→ a

x

w

7→ a

w

=⇒ x

w 7→ x

w

In particular: if w

is optimal (loss-minimizer) for sequence {(x

, y

)}

, then A

w

is optimal for sequence {(Ax

, y

)}

A learning algorithm is scale-invariant if it returns the same predictions under arbitrary rescaling of the data:

∀t x

7→ Ax

=⇒ w

7→ A

w

=⇒ w

x

7→ w

x

no initial data normalization required!

Motivation: A fully adaptive algorithm needs to be scale-invariant

8 / 22

Scale invariance

A natural symmetry in the linear problems

Rescaling the features followed by the inverse scaling of the weights keep the predictions (and hence losses) invariant:

∀t x

7→ A

x

, w 7→ Aw =⇒ x

w 7→ x

w for any diagonal matrix A

In particular: if w

is optimal (loss-minimizer) for sequence {(x

, y

)}

, then A

w

is optimal for sequence {(Ax

, y

)}

A learning algorithm is scale-invariant if it returns the same predictions under arbitrary rescaling of the data:

∀t x

7→ Ax

=⇒ w

7→ A

w

=⇒ w

x

7→ w

x

no initial data normalization required!

Motivation: A fully adaptive algorithm needs to be scale-invariant

8 / 22

Scale invariance

A natural symmetry in the linear problems

Rescaling the features followed by the inverse scaling of the weights keep the predictions (and hence losses) invariant:

∀t x

7→ A

x

, w 7→ Aw =⇒ x

w 7→ x

w for any diagonal matrix A

In particular: if w

is optimal (loss-minimizer) for sequence {(x

, y

)}

, then A

w

is optimal for sequence {(Ax

, y

)}

A learning algorithm is scale-invariant if it returns the same predictions under arbitrary rescaling of the data:

∀t x

7→ Ax

=⇒ w

7→ A

w

=⇒ w

x

7→ w

x

no initial data normalization required!

Motivation: A fully adaptive algorithm needs to be scale-invariant

8 / 22

Scale invariance

A natural symmetry in the linear problems

Rescaling the features followed by the inverse scaling of the weights keep the predictions (and hence losses) invariant:

∀t x

7→ A

x

, w 7→ Aw =⇒ x

w 7→ x

w for any diagonal matrix A

In particular: if w

is optimal (loss-minimizer) for sequence {(x

, y

)}

, then A

w

is optimal for sequence {(Ax

, y

)}

Example: minimizing squared error loss:

w

=  X

t

x

x



X

t

x

y

A learning algorithm is scale-invariant if it returns the same predictions under arbitrary rescaling of the data:

∀t x

7→ Ax

=⇒ w

7→ A

w

=⇒ w

x

7→ w

x

no initial data normalization required!

Motivation: A fully adaptive algorithm needs to be scale-invariant

8 / 22

Scale invariance

A natural symmetry in the linear problems

Rescaling the features followed by the inverse scaling of the weights keep the predictions (and hence losses) invariant:

∀t x

7→ A

x

, w 7→ Aw =⇒ x

w 7→ x

w for any diagonal matrix A

In particular: if w

is optimal (loss-minimizer) for sequence {(x

, y

)}

, then A

w

is optimal for sequence {(Ax

, y

)}

Example: minimizing squared error loss:

w

7→  X

t

Ax

(Ax

)



X

t

Ax

y

A learning algorithm is scale-invariant if it returns the same predictions under arbitrary rescaling of the data:

∀t x

7→ Ax

=⇒ w

7→ A

w

=⇒ w

x

7→ w

x

no initial data normalization required!

Motivation: A fully adaptive algorithm needs to be scale-invariant

8 / 22

Scale invariance

A natural symmetry in the linear problems

Rescaling the features followed by the inverse scaling of the weights keep the predictions (and hence losses) invariant:

∀t x

7→ A

x

, w 7→ Aw =⇒ x

w 7→ x

w for any diagonal matrix A

In particular: if w

is optimal (loss-minimizer) for sequence {(x

, y

)}

, then A

w

is optimal for sequence {(Ax

, y

)}

Example: minimizing squared error loss:

w

7→ A

 X

t

x

x



A

A X

t

x

y

A learning algorithm is scale-invariant if it returns the same predictions under arbitrary rescaling of the data:

∀t x

7→ Ax

=⇒ w

7→ A

w

=⇒ w

x

7→ w

x

no initial data normalization required!

Motivation: A fully adaptive algorithm needs to be scale-invariant

8 / 22

Scale invariance

A natural symmetry in the linear problems

Rescaling the features followed by the inverse scaling of the weights keep the predictions (and hence losses) invariant:

∀t x

7→ A

x

, w 7→ Aw =⇒ x

w 7→ x

w for any diagonal matrix A

In particular: if w

is optimal (loss-minimizer) for sequence {(x

, y

)}

, then A

w

is optimal for sequence {(Ax

, y

)}

Example: minimizing squared error loss:

w

7→ A

 X

t

x

x



X

t

x

y

= A

w

A learning algorithm is scale-invariant if it returns the same predictions under arbitrary rescaling of the data:

∀t x

7→ Ax

=⇒ w

7→ A

w

=⇒ w

x

7→ w

x

no initial data normalization required!

Motivation: A fully adaptive algorithm needs to be scale-invariant

8 / 22

Scale invariance

A natural symmetry in the linear problems

Rescaling the features followed by the inverse scaling of the weights keep the predictions (and hence losses) invariant:

∀t x

7→ A

x

, w 7→ Aw =⇒ x

w 7→ x

w for any diagonal matrix A

In particular: if w

is optimal (loss-minimizer) for sequence {(x

, y

)}

, then A

w

is optimal for sequence {(Ax

, y

)}

A learning algorithm is scale-invariant if it returns the same predictions under arbitrary rescaling of the data:

∀t x

7→ Ax

=⇒ w

7→ A

w

=⇒ w

x

7→ w

x

no initial data normalization required!

Motivation: A fully adaptive algorithm needs to be scale-invariant

8 / 22

Scale invariance

A natural symmetry in the linear problems

Rescaling the features followed by the inverse scaling of the weights keep the predictions (and hence losses) invariant:

∀t x

7→ A

x

, w 7→ Aw =⇒ x

w 7→ x

w for any diagonal matrix A

In particular: if w

is optimal (loss-minimizer) for sequence {(x

, y

)}

, then A

w

is optimal for sequence {(Ax

, y

)}

A learning algorithm is scale-invariant if it returns the same predictions under arbitrary rescaling of the data:

∀t x

7→ Ax

=⇒ w

7→ A

w

=⇒ w

x

7→ w

x

no initial data normalization required!

Motivation: A fully adaptive algorithm needs to be scale-invariant

8 / 22

Scale invariance

A natural symmetry in the linear problems

Rescaling the features followed by the inverse scaling of the weights keep the predictions (and hence losses) invariant:

∀t x

7→ A

x

, w 7→ Aw =⇒ x

w 7→ x

w for any diagonal matrix A

In particular: if w

is optimal (loss-minimizer) for sequence {(x

, y

)}

, then A

w

is optimal for sequence {(Ax

, y

)}

A learning algorithm is scale-invariant if it returns the same predictions under arbitrary rescaling of the data:

∀t x

7→ Ax

=⇒ w

7→ A

w

=⇒ w

x

7→ w

x

no initial data normalization required!

Motivation: A fully adaptive algorithm needs to be scale-invariant

8 / 22

Past work

Scale-invariant algorithms with bounded predictions [Ross et al., 2013, Orabona et al., 2015]

Assumption: |x

w

| ≤ C for all i, t for some constant C regret

(w

) = O

 d √

C

T



Compare with optimal SGD regret:

d

X

i=1



|w

| s

X

t

∇



C

T =⇒ X

t

(∇

w

)

d =⇒ X

i

[Luo et al., 2016] considers a more general version of scale invariance, but also with bounded predictions

Some more recent work on unconstrained online learning: [McMahan and Streeter, 2010, McMahan and Abernethy, 2013, Orabona, 2013, Cutkosky and Boahen, 2017,

Cutkosky and Orabona, 2018] Prior to this work: [Kot lowski, 2017]

9 / 22

Past work

Scale-invariant algorithms with bounded predictions [Ross et al., 2013, Orabona et al., 2015]

Assumption: |x

w

| ≤ C for all i, t for some constant C regret

(w

) = O

 d √

C

T



Compare with optimal SGD regret:

d

X

i=1



|w

| s

X

t

∇



C

T =⇒ X

t

(∇

w

)

d =⇒ X

i

[Luo et al., 2016] considers a more general version of scale invariance, but also with bounded predictions

Some more recent work on unconstrained online learning: [McMahan and Streeter, 2010, McMahan and Abernethy, 2013, Orabona, 2013, Cutkosky and Boahen, 2017,

Cutkosky and Orabona, 2018] Prior to this work: [Kot lowski, 2017]

9 / 22

Past work

Scale-invariant algorithms with bounded predictions [Ross et al., 2013, Orabona et al., 2015]

Assumption: |x

w

| ≤ C for all i, t for some constant C regret

(w

) = O

 d √

C

T



Compare with optimal SGD regret:

d

X

i=1



|w

| s

X

t

∇



C

T =⇒ X

t

(∇

w

)

d =⇒ X

i

[Luo et al., 2016] considers a more general version of scale invariance, but also with bounded predictions

Some more recent work on unconstrained online learning: [McMahan and Streeter, 2010, McMahan and Abernethy, 2013, Orabona, 2013, Cutkosky and Boahen, 2017,

Cutkosky and Orabona, 2018] Prior to this work: [Kot lowski, 2017]

9 / 22

Past work

Scale-invariant algorithms with bounded predictions [Ross et al., 2013, Orabona et al., 2015]

Assumption: |x

w

| ≤ C for all i, t for some constant C regret

(w

) = O

 d √

C

T



Compare with optimal SGD regret:

d

X

i=1



|w

| s

X

t

∇



C

T =⇒ X

t

(∇

w

)

d =⇒ X

i

[Luo et al., 2016] considers a more general version of scale invariance, but also with bounded predictions

Some more recent work on unconstrained online learning: [McMahan and Streeter, 2010, McMahan and Abernethy, 2013, Orabona, 2013, Cutkosky and Boahen, 2017,

Cutkosky and Orabona, 2018] Prior to this work: [Kot lowski, 2017]

9 / 22

Past work

Scale-invariant algorithms with bounded predictions [Ross et al., 2013, Orabona et al., 2015]

Assumption: |x

w

| ≤ C for all i, t for some constant C regret

(w

) = O

 d √

C

T



Compare with optimal SGD regret:

d

X

i=1



|w

| s

X

t

∇



C

T =⇒ X

t

(∇

w

)

d =⇒ X

i

[Luo et al., 2016] considers a more general version of scale invariance, but also with bounded predictions

Some more recent work on unconstrained online learning:

[McMahan and Streeter, 2010, McMahan and Abernethy, 2013, Orabona, 2013, Cutkosky and Boahen, 2017,

Cutkosky and Orabona, 2018]

Prior to this work: [Kot lowski, 2017]

9 / 22

Past work

Scale-invariant algorithms with bounded predictions [Ross et al., 2013, Orabona et al., 2015]

Assumption: |x

w

| ≤ C for all i, t for some constant C regret

(w

) = O

 d √

C

T



Compare with optimal SGD regret:

d

X

i=1



|w

| s

X

t

∇



C

T =⇒ X

t

(∇

w

)

d =⇒ X

i

[Luo et al., 2016] considers a more general version of scale invariance, but also with bounded predictions

Some more recent work on unconstrained online learning:

[McMahan and Streeter, 2010, McMahan and Abernethy, 2013, Orabona, 2013, Cutkosky and Boahen, 2017,

Cutkosky and Orabona, 2018]

Prior to this work: [Kot lowski, 2017]

9 / 22

Scale-invariant algorithms

Scale Invariant Online Learning, Algorithm 1: ScInOL

Parameter:  = 1

Keep track of data statistics: M

= max

j≤t

|x

|, S

= X

j≤t

∇

, G

= X

j≤t

∇

Maximum value at a given feature

Sum of squared

gradients Sum of gradients and an auxilary variable β

= min{β

,

2

t−1,i+M_t,i² )

x²_t,it

} with β

=  w

= β

sgn(θ

)

2 q

S

+ M



e

− 1 

, where θ

= G

q

S

+ M

1/[X

]

unitless

p[X

]

unitless

[X

]

p[X

]

regret

(w

) =

d

X

i=1

O ˜



|w

| s

max

x

+ X

t

∇

 , where ˜ O(·) hides logarithmic factors

Optimal up to logarithmic terms

10 / 22

Scale-invariant algorithms

Scale Invariant Online Learning, Algorithm 1: ScInOL

Parameter:  = 1

Keep track of data statistics: M

= max

j≤t

|x

|, S

= X

j≤t

∇

, G

= X

j≤t

∇

Maximum value at a given feature

Sum of squared

gradients Sum of gradients and an auxilary variable β

= min{β

,

2

t−1,i+M_t,i² )

x²_t,it

} with β

=  w

= β

sgn(θ

)

2 q

S

+ M



e

− 1 

, where θ

= G

q

S

+ M

1/[X

]

unitless

p[X

]

unitless

[X

]

p[X

]

regret

(w

) =

d

X

i=1

O ˜



|w

| s

max

x

+ X

t

∇

 , where ˜ O(·) hides logarithmic factors

Optimal up to logarithmic terms

10 / 22

Scale-invariant algorithms

Scale Invariant Online Learning, Algorithm 1: ScInOL

Parameter:  = 1

Keep track of data statistics:

M

= max

j≤t

|x

|, S

= X

j≤t

∇

, G

= X

j≤t

∇

Maximum value at a given feature

Sum of squared

gradients Sum of gradients

and an auxilary variable β

= min{β

,

2

t−1,i+M_t,i² )

x²_t,it

} with β

=  w

= β

sgn(θ

)

2 q

S

+ M



e

− 1 

, where θ

= G

q

S

+ M

1/[X

]

unitless

p[X

]

unitless

[X

]

p[X

]

regret

(w

) =

d

X

i=1

O ˜



|w

| s

max

x

+ X

t

∇

 , where ˜ O(·) hides logarithmic factors

Optimal up to logarithmic terms

10 / 22

Scale-invariant algorithms

Scale Invariant Online Learning, Algorithm 1: ScInOL

Parameter:  = 1

Keep track of data statistics:

M

= max

j≤t

|x

|, S

= X

j≤t

∇

, G

= X

j≤t

∇

Maximum value at a given feature

Sum of squared

gradients Sum of gradients

and an auxilary variable β

= min{β

,

2

t−1,i+M_t,i² )

x²_t,it

} with β

= 

w

= β

sgn(θ

) 2

q

S

+ M



e

− 1 

, where θ

= G

q

S

+ M

1/[X

]

unitless

p[X

]

unitless

[X

]

p[X

]

regret

(w

) =

d

X

i=1

O ˜



|w

| s

max

x

+ X

t

∇

 , where ˜ O(·) hides logarithmic factors

Optimal up to logarithmic terms

10 / 22