) a discrete time controlled Markov process on a compact state space E, en- dowed with the Borel σ-field E , with transition kernel P

L. S T E T T N E R (Warszawa)

ERGODIC CONTROL OF PARTIALLY OBSERVED MARKOV PROCESSES WITH EQUIVALENT

TRANSITION PROBABILITIES

Abstract. Optimal control with long run average cost functional of a partially observed Markov process is considered. Under the assumption that the transition probabilities are equivalent, the existence of the solution to the Bellman equation is shown, with the use of which optimal strategies are constructed.

1. Introduction. Let (Ω, F , P ) be a probability space and (x

) a discrete time controlled Markov process on a compact state space E, en- dowed with the Borel σ-field E , with transition kernel P

(x, dz) for v ∈ U , where (U, U ) is a compact space of control parameters. Assume the only observations of x

are R

-valued random variables y

, . . . , y

such that for Y

= σ{y

, . . . , y

} we have

(1) P {y

∈ A | x

, Y

} = P {y

∈ A | x

} = R

A

r(x

, y) dy for n = 0, 1, . . . with r : E×R

→ R

a measurable function, and A ∈ B(R

), the family of Borel subsets of R

.

The Markov process (x

) is controlled by a sequence (a

) of Y

-measur- able U -valued random variables. The best mean square approximation of x

based on the available observation is given by a filtering process π

, defined as a measure valued process such that for A ∈ E ,

(2) π

(A) = P {x

∈ A | Y

} for n = 1, 2, . . . ,

and π

(A) = µ(A)

where µ is the initial law of (x

).

1991 Mathematics Subject Classification: Primary 93E20; Secondary 93E11.

Key words and phrases: stochastic control, partial observation, long run average cost, Bellman equation.

The following lemma gives the most general formula for π

. Its proof, unlike those in [5] and [8], which have more restrictive hypotheses, is not based on the reference probability method.

Lemma 1. Under (1), for n = 0, 1, . . . and A ∈ E we have (3) π

(A) =

R

A

r(z

, y

) R

E

P

(z

, dz

) π

(dz

) R

E

r(z

, y

) R

E

P

(z

, dz

) π

(dz

) .

P r o o f. Denote the right hand side of (3) by M

(y

, π

)(A). Let F : (R

)

→ R be a bounded measurable function, Y

= (y

, . . . , y

) and C ∈ B(R

).

By (1), Fubini’s theorem and properties of conditional expectations we have

R

Ω

M

(y

, π

)(A)χ

(y

)F (Y

) dP

= R

Ω

E[M

(y

, π

)(A)χ

(y

) | x

, Y

]F (Y

) dP

= R

Ω

R

C

M

(y, π

)(A)r(x

, y) dy F (Y

) dP

= R

Ω

R

C

M

(y, π

)(A)E[E[r(x

, y) | Y

, x

] | Y

] dy F (Y

) dP

= R

Ω

R

C

M

(y, π

)(A)E h R

E

r(z, y) P

(x

, dz)    Y

i

dy F (Y

) dP

= R

Ω

R

C

M

(y, π

)(A) R

E

R

E

r(z, y) P

(z

, dz) π

(dz

) dy F (Y

) dP

= R

Ω

R

C

R

A

r(z

, y) R

E

P

(z

, dz

) π

(dz

) dy F (Y

) dP

= R

Ω

R

E

R

A

R

C

r(z

, y) dy P

(z

, dz

) π

(dz

) F (Y

) dP

= R

Ω

E h R

A

R

C

r(z

, y) dy P

(x

, dz

)    Y

i

F (Y

) dP

= R

Ω

E h

E h R

C

r(x

, y) dy χ

(x

)    Y

, x

i    Y

i

F (Y

) dP

= R

Ω

R

C

r(x

, y) dy χ

(x

)F (Y

) dP

= R

Ω

E[χ

(y

) | Y

, x

]χ

(x

)F (Y

) dP

= R

Ω

χ

(y

)χ

(x

)F (Y

) dP = R

Ω

π

(A)χ

(y

)F (Y

) dP.

Therefore, by the definition of conditional expectation, (3) follows.

The class of controls a

= u(π

), where u is a fixed measurable, U -valued function, is of special interest. Namely, we have

Lemma 2. Under (1), if additionally a

= u(π

) with u a fixed measur- able function from the space P(E) of probability measures on E, endowed with the topology of weak convergence, into (U, U ), then π

is a Y

-Markov process with transition operator

(4) Π

(ν, F ) = R

E

R

R^d

F (M

(y, ν))r(z, y) dy R

E

P

(z

, dz) ν(dz

)

where

(5) M

(y, ν)(A) = R

A

r(z, y) R

E

P

(z

, dz) ν(dz

) R

E

r(z, y) R

E

P

(z

, dz) ν(dz

) for v ∈ U , ν ∈ P(E) and F : P(E) → R bounded measurable.

P r o o f. By (1) we easily obtain E[F (π

) | Y

]

= E[F (M

(y

, π

)) | Y

]

= E[E[F (M

(y

, π

)) | Y

, x

] | Y

]

= E h R

R^d

F (M

(y, π

))r(x

, y) dy    Y

i

= E h R

R^d

E[F (M

(y, π

))r(x

, y) | Y

, x

] dy    Y

i

= E h R

R^d

R

E

F (M

(y, π

))r(z, y)P

(x

, dz) dy    Y

i

= R

E

R

R^d

F (M

(y, π

)) R

E

r(z, y) P

(z

, dz) π

(dz

) dy

= Π

(π

, F ) .

Thus (π

) is Markov with transition operator of the form (4).

In this paper we are interested in minimizing the following long run average cost functional:

(6) J

((a

)) = lim sup

n→∞

n

E

n

X

i=0

c(x

, a

) o

over all U -valued, Y

-adapted processes a

, with c : E × U → R

a given bounded measurable cost function.

By the very definition of a filtering process we have (7) J

((a

)) = lim sup

n→∞

n

E

n

X

i=0

R

E

c(z, a

) π

(dz) o

.

The optimal strategies for the cost functional J

are constructed with the use of a suitable Bellman equation, the solution of which is found as a limit of w

(x) = ϑ

(x) − inf

ϑ

(z) as β → 1, where ϑ

is the value function of the β-discounted cost functional. Since our limit results are based on compactness arguments, obtained via the Ascoli–Arzel` a theorem, in Section 2 we show the continuity of ϑ

. Then in Section 3 we prove the uniform boundedness of w

. Using the concavity of w

, obtained from the concavity of ϑ

, proved in Section 2, we get equicontinuity of w

, which allows us to use the Ascoli–Arzel` a theorem.

The discrete time ergodic optimal control problem with partial observa- tion was studied in [1], [2], [3], [6], [8], [9]. In [1] and [8] the observation was corrupted with white noise. In addition, in [1] there was a finite state space and a rich observation structure. In [8] the state space was general but there were some restrictions on controls. The papers [2] and [3] con- tain a general theory but the fundamental example used is a very simple maintenance-replacement model.

In [6] a model with a finite state space and almost steady state transition probabilities was studied. Finally, finite state space semi-Markov decision processes with a completely observable state were considered in [9]. Our paper generalizes [6] in various directions. Namely, we have a general, com- pact state space. Although the techniques to show the boundedness and the equicontinuity of w

follow in some sense the arguments of [6], by a more detailed estimation we obtain the results under the assumptions which are much less restrictive than the corresponding ones in [6], even when E is finite.

2. Discounted control problem. In this section we characterize the value function ϑ

of the discounted cost functional J

defined as follows:

(8) J

((a

))

= E

n X

i=0

β

c(x

, a

) o

= E

n X

i=0

β

R

E

c(z, a

) π

(dz) o

with β ∈ (0, 1).

The theorem below provides a complete solution to the discounted par- tially observed control problem.

Theorem 1. Assume (1) and (A1) c : E × U → R

is continuous,

(H1) for F ∈ C(P(E)), the space of continuous functions on P(E), if µ

⇒ µ, i.e. µ

converges weakly in P(E) to µ, we have

(9) sup

a∈U

|Π

(µ

, F ) − Π

(µ, F )| → 0 as n → ∞, (H2) for F ∈ C(P(E)), if U 3 a

→ a we have

(10) Π

(µ, F ) → Π

(µ, F ).

Then

(11) ϑ

(µ)

= inf

(an)

J

((a

))

is a continuous function of µ ∈ P(E) and is a unique solution to the Bellman equation

(12) ϑ

(µ) = inf

a∈U

h R

E

c(x, a) µ(dx) + βΠ

(µ, ϑ

) i .

There exists a measurable selector u

: P(E) → (U, U ) for which the infimum on the right hand side of (12) is attained. Moreover , we have

(13) ϑ

(µ) = J

((u

(π

))) .

In addition, ϑ

can be uniformly approximated from below by the sequence

(14)

ϑ

(µ) ≡ 0 , ϑ

(µ) = inf

a∈U

h R

E

c(x, a) µ(dx) + βΠ

(µ, ϑ

) i

, and each ϑ

is concave, i.e. for µ, ν ∈ P(E) and α ∈ [0, 1], (15) ϑ

(αµ + (1 − α)ν) ≥ αϑ

(µ) + (1 − α)ϑ

(ν).

P r o o f. We only point out the main steps since the proof is more or less standard (for details see [4], Thm. 2.2).

Define, for ϑ ∈ C(P(E)), T ϑ(µ) = inf

a∈U

h R

E

c(x, a) µ(dx) + βΠ

(µ, ϑ) i

.

By (A1) and (H1), T is a contraction on C(P(E)). Thus, by the Banach

principle there is a unique fixed point ϑ

of T , which is a unique solution to

the Bellman equation (12). Since by (A1) and (H2) the map U 3 a → R

E

c(x, a) µ(dx) + βΠ

(µ, ϑ

)

is continuous, there exists a measurable selector u

. The identity (13) is then almost immediate. Since T is monotonic and contractive, ϑ

is increasing and converges to ϑ

. It remains to show the concavity of ϑ

. We prove this by induction. Clearly, ϑ

≡ 0 is concave. Provided ϑ

is concave, by Jensen’s lemma we have for α ∈ (0, 1),

Π

(αµ + (1 − α)ν, ϑ

) ≥ αΠ

(µ, ϑ

) + (1 − α)Π

(ν, ϑ

) and therefore from (14),

ϑ

(αµ + (1 − α)ν) ≥ αϑ

(µ) + (1 − α)ϑ

(ν) ,

i.e. ϑ

is concave. By induction, ϑ

is concave for each n. The proof of the theorem is complete.

Below we formulate sufficient conditions for (H1) and (H2).

Proposition 1. Assume (A2) r ∈ C(E × R

),

(A3) for fixed a ∈ U , P

(x, · ) is Feller , i.e. for any ϕ ∈ C(E), if x

⇒ x we have

(16) P

(x

, ϕ) → P

(x, ϕ), (H3) if U 3 a

→ a, then for each ϕ ∈ C(E),

(17) sup

x∈E

|P

(x, ϕ) − P

(x, ϕ)| → 0, (A4) for R(z, ψ)

= R

R^d

r(z, y)ψ(y) dy where ψ ∈ C(R

), if E 3 z

→ z we have

(18) R(z

, · ) ⇒ R(z, · ).

Then (H1) and (H2) are satisfied.

P r o o f. Notice first that from (16) and (17), if U 3 a

→ a and µ

⇒ µ, we have

(19) P

(µ

, ϕ)

= R

E

P

(x, ϕ) µ

(dx) → P

(µ, ϕ)

as n → ∞, for ϕ ∈ C(E) . Since U × P(E) is compact, to prove (H1) and (H2) it is sufficient to show that

U × P(E) 3 (a, µ) → Π

(µ, F ) is continuous for F ∈ C(P(E)).

Therefore we shall show that

(20) Π

(µ

, F ) → Π

(µ, F )

for U 3 a

→ a, P(E) 3 µ

⇒ µ and F ∈ C(P(E)). We have (21) |Π

(µ

, F ) − Π

(µ, F )|

≤   

R

E

R

R^d

(F (M

(y, µ

)) − F (M

(y, µ)))r(z, y) dy P

(µ

, dz)    +

R

E

R

R^d

F (M

(y, µ))r(z, y) dy (P

(µ

, dz) − P

(µ, dz))   

= I

+ II

.

From (19), II

→ 0, provided

(22) E 3 z → R

R^d

F (M

(y, µ))r(z, y) dy ∈ C(E).

By (A4), R

3 y → M

(y, µ) ∈ P(E) is continuous. Then, again by (A4), the map (22) is continuous, and consequently II

→ 0.

If

(23) sup

z∈E

R

R^d

(F (M

(y, µ

)) − F (M

(y, µ)))r(z, y) dy    → 0 then clearly I

→ 0.

By (A4), for each ε > 0 there exists a compact set K ⊂ R

such that for any z ∈ E,

(24) R(z, K

) < ε

2kF k . Therefore

R

R^d

(F (M

(y, µ

)) − F (M

(y, µ)))r(z, y) dy   

≤ R

K

|F (M

(y, µ

)) − F (M

(y, µ))|r(z, y) dy + ε

and to obtain (23) it remains to show that

(25) M

(y, µ

)(ϕ) → M

(y, µ)(ϕ) for any ϕ ∈ C(E), uniformly in y ∈ K.

Using the Stone–Weierstrass approximation theorem (see [7], Thm. 9.28,

cf. also the proof of Lemma A.1.2 of [8]) and (19), we obtain

R

E

r(z, y)ϕ(z) R

E

P

(z

, dz) µ

(dz

) − R

E

r(z, y)ϕ(z) R

E

P

(z

, dz) µ(dz

)   

≤   

R

E

r(z, y)ϕ(z)(P

(µ

, dz) − P

(µ, dz))    → 0 uniformly in y ∈ K. Thus, we have uniform convergence of the numerators and denominators in the formula defining M

, and consequently conver- gence of the ratios from which (25) follows.

The proof of Proposition 1 is complete.

R e m a r k 1. (A4) is satisfied when sup

r(z, y) is integrable.

Define

(26) w

(ν) = ϑ

(ν) − ϑ

(µ

) and w

(ν) = ϑ

(ν) − ϑ

(µ

)

where µ

= arg min ϑ

and µ

= arg min ϑ

. Clearly, w

is a solution to the equation

(27) w

(ν) + (1 − β)ϑ

(µ

) = inf

a∈U

h R

E

c(x, a) ν(dx) + βΠ

(ν, w

) i

and w

(ν) → w

(ν) uniformly in ν ∈ P(E). We would like to let β ↑ 1 in (27) and thus obtain a solution w(ν) to the long run average Bellman equation

(28) w(ν) + γ = inf

a∈U

h R

E

c(x, a) ν(dx) + Π

(ν, w) i .

Since we wish to apply the Ascoli–Arzel` a theorem, we have to show the boundedness and the equicontinuity of w

for β ∈ (0, 1), which are studied successively in the next sections.

3. Boundedness of w

. We make the following assumption:

(29) (A5) inf

z,z⁰∈E

inf

a,a⁰∈U

inf

C∈E, P^a(z,C)>0

P

(z

, C) P

(z, C)

def

= λ > 0.

We have

Proposition 2. Under (A5) and the assumptions of Theorem 1, the functions w

(ν) are uniformly bounded for β ∈ (0, 1), ν ∈ P(E).

P r o o f. We improve the proof of Theorem 2 of [6]. Namely, we show by induction the uniform boundedness of w

(ν) for ν ∈ P(E), β ∈ (0, 1), n = 0, 1, . . . For n = 0, w

(ν) ≡ 0.

Assume that for any β ∈ (0, 1), ν ∈ P(E), w

(ν) ≤ L where L ≥ kckλ

.

Let a, a

∈ U be such that for fixed ν ∈ P(E), w

(ν) = R

E

c(x, a) ν(dx) − R

E

c(x, a

) µ

(dx) (30)

+ β[Π

(ν, ϑ

) − Π

(µ

, ϑ

)] . For y ∈ R

, define

m(y)(B) = M

(y, µ

)(B) − λ

M

(y, ν)(B) for any B ∈ E .

By (29) we have

R

B

r(z, y) R

E

P

(z

, dz) µ

(dz

) ≥ λ R

B

r(z, y) R

E

P

(z

, dz) ν(dz

)

= λM

(y, ν)(B) R

E

r(z, y) R

E

P

(z

, dz) ν(dz

)

≥ λ

M

(y, ν)(B) R

E

r(z, y) R

E

P

(z

, dz) µ

(dz

) and therefore m(y)(B) ≥ 0 for B ∈ E .

If λ = 1 we have a stationary, noncontrolled Markov chain with P

(z, C)

= η(C) for any a ∈ U , z ∈ E and some fixed η ∈ P(E), and consequently w

≡ 0 for any n = 0, 1, . . . Therefore we restrict ourselves to the case λ < 1.

Then (1 − λ

)

m(y) ∈ P(E). Since

M

(y, µ

) = λ

M

(y, ν) + (1 − λ

)[(1 − λ

)

m(y)] , by concavity of ϑ

we obtain

(31) ϑ

(M

(y, µ

)) ≥ λ

ϑ

(M

(y, ν)) + (1 − λ

)ϑ

((1 − λ

)

m(y)) and from (30) we have

w

(ν) (32)

≤ kck + β R

E

R

R^d

ϑ

(M

(y, µ))r(z, y) dy

×  R

E

P

(z

, dz) ν(dz

) − λ

R

E

P

(z

, dz) µ

(dz

) 

− β(1 − λ

) R

E

R

R^d

ϑ

((1 − λ

)

m(y))r(z, y) dy

× R

E

P

(z

, dz)µ

(dz

)

= kck + β R

E

R

R^d

(ϑ

(M

(y, µ)) − ϑ

(µ

))r(z, y) dy

×  R

E

P

(z

, dz) ν(dz

) − λ

R

E

P

(z

, dz) µ

(dz

) 

− β(1 − λ

) R

E

R

R^d

(ϑ

((1 − λ

)

m(y)) − ϑ

(µ

))r(z, y) dy

× R

E

P

(z

, dz) µ

(dz

)

≤ kck + βL var  R

E

P

(z

, · ) ν(dz

) − λ

R

E

P

(z

, · ) µ

(dz

)

 .

By (A5) for any B ∈ E ,

(33) R

E

P

(z

, B) ν(dz

) ≥ λ

R

E

P

(z

, B) µ

(dz

) . Thus

(34) w

(ν) ≤ kck + βL(1 − λ

) ≤ L

and the bound L is independent of ν ∈ P(E), β ∈ (0, 1). By induction w

(ν) ≤ L for any ν ∈ P(E), n = 0, 1, . . . , β ∈ (0, 1). Since by the very definition w

(ν) ≥ 0, and for each β, w

(ν) → w

(ν) as n → ∞, we finally obtain w

(ν) ≤ L for ν ∈ P(E) and β ∈ (0, 1).

R e m a r k 2. One can easily see that in the case of a finite state space E, the assumption

(35) (A5

) inf

z,z⁰∈E

inf

a,a⁰∈U

inf

x∈E, P^a(z,x)>0

P

(z

, x) P

(z, x) > 0

also implies the boundedness of w

. Thus Proposition 2 significantly im- proves Theorem 2 of [6]. This was possible because of the choice of µ

in (32) as the argument of minimum of ϑ

.

R e m a r k 3. Assumption (A5) says that the transition probabilities for different controls and initial states are mutually equivalent, with Radon–

Nikodym density bounded away from 0. In particular, in the case when P

(z, C) = R

C

g

(z, x) η(dx) the assumption

(36) inf

z,z⁰∈E

inf

a,a⁰∈U

inf

x∈E, g^a(z,x)>0

g

(z

, x)

g

(z, x) > 0

is sufficient for (A5) to be satisfied.

4. Main theorem. Before we formulate and prove our main result, we show the equicontinuity of w

for β ∈ (0, 1). For this purpose we need an extra assumption:

(A6) If P(E) 3 µ

⇒ µ ∈ P(E) then sup

a∈U

sup

C∈E

|P

(µ

, C) − P

(µ, C)| → 0 with

P

(µ, C)

= R

E

P

(x, C)µ(dx) . We have

Proposition 3. Under (A5), (A6) and the assumptions of Theorem 1, the family of functions w

, β ∈ (0, 1), is equicontinuous, i.e.

(37) ∀

∃

∀

%(µ, µ

) < δ ⇒ ∀

|w

(µ) − w

(µ

)| < ε with % standing for a metric compatible with the weak convergence topology of P(E).

P r o o f. For ν, µ ∈ P(E) let (38) λ(ν, µ)

= inf

a∈U

inf

C∈E,P^a(µ,C)>0

P

(ν, C) P

(µ, C) . From (A5) and (A6), if ν ⇒ µ, then

(39) λ(ν, µ) → 1 and λ(µ, ν) → 1.

By (27) for ν, µ ∈ P(E) we have (40) w

(ν) − w

(µ)

≤ sup

a∈U

R

E

c(x, a)(ν(dx) − µ(dx))  

 + β sup

a∈U

(Π

(ν, w

) − Π

(µ, w

)).

By analogy with the proof of Proposition 2 define

m

(y, µ, ν)(B) = M

(y, µ)(B) − λ(µ, ν)λ(ν, µ)M

(y, ν)(B) for B ∈ E .

Clearly, m

(y, µ, ν)(B) ≥ 0 for B ∈ E , and λ(µ, ν)λ(ν, µ) ≤ 1.

If λ(µ, ν)λ(ν, µ) = 1, then w

≡ 0 for β ∈ (0, 1), and consequently the equicontinuity property is satisfied. Therefore assume λ

= λ(µ, ν)λ(ν, µ)

< 1. Then by the concavity of w

,

(41) w

(M

(y, µ)) ≥ λ

w

(M

(y, ν)) + (1 − λ

)w

((1 − λ

)

m

(y, µ, ν)) .

From (40),

(42) w

(ν) − w

(µ) ≤ sup

a∈U

R

E

c(x, a)(ν(dx) − µ(dx))    + β sup

a∈U

n R

E

R

R^d

w

(M

(y, ν))r(z, y) dy (P

(ν, dz) − λ

P

(µ, dz))

+ R

E

R

R^d

(λ

w

(M

(y, ν)) − w

(M

(y, µ)))r(z, y) dy P

(µ, dz) o

= I + II + III . Now

II ≤ 2kw

k sup

a∈U

sup

B∈E

|P

(ν, B) − λ

P

(µ, B)|

(43)

= 2kw

k(1 − λ(µ, ν)λ(ν, µ)) and using (41) and the nonnegativity of w

we have (44) III

≤ sup

a∈U

R

E

R

R^d

(λ

− 1)w

((1 − λ

)

m

(y, µ, ν))r(z, y) dy P

(µ, dz) ≤ 0 .

Interchanging ν and µ in (40)–(44) we obtain the same estimates and there- fore

|w

(ν) − w

(µ)| ≤ sup

a∈U

R

E

c(x, a)(ν(dx) − µ(dx))   (45) 

+ 2kw

k(1 − λ(µ, ν)λ(ν, µ)) .

Since by the Stone–Weierstrass theorem (Thm. 9.28 of [7]) c(x, a) can be uniformly approximated on E × U by continuous functions of the form P

i=1

c

(x)d

(a), from (39) we obtain

ν⇒µ

lim sup

β∈(0,1)

|w

(ν) − w

(µ)| = 0 . Let us comment on the assumption (A6):

R e m a r k 4. (H3) clearly follows from (A6).

R e m a r k 5. In the case of a finite state space E = {1, . . . , N }, (A6) can be written as

(46) sup

a∈U N

X

k=1

N

X

i=1

(s

− s

)P

(i, k)

 → 0

for s

= (s

, . . . , s

) → s = (s

, . . . , s

), 0 ≤ s

≤ 1, 0 ≤ s

≤ 1, P s

= 1, P s

= 1, and this is satisfied since

sup

a∈U N

X

k=1

N

X

i=1

(s

− s

)P

(i, k)    ≤

N

X

i=1

|s

− s

| → 0 as s

→ s . R e m a r k 6. Assume P

(z, C) = R

C

g

(z, x)η(dx) for C ∈ E and that the mapping

(47) U × E × E 3 (a, z, x) → g

(z, x) is continuous.

Then (A6) is satisfied.

In fact, by the Stone–Weierstrass theorem we can approximate g

uni- formly on U ×E×E by continuous functions of the form P

i=1

b

(a)c

(z)d

(x) and

sup

a∈U

sup

C∈E

|P

(µ

, C) − P

(µ, C)|

≤ sup

a∈U

R

E

R

E

g

(z, x)(µ

(dz) − µ(dz))    η(dx)

≤ ε +

k

X

i=1

sup

a∈U

|b

(a)| R

E

|d

(x)|η(dx)   

R

E

c

(z)(µ

(dz) − µ(dz))    → ε as n → ∞.

Now we can prove our main result:

Theorem 2. Assume (A1)–(A6). Then there exist w ∈ C(P(E)) and a constant γ which are solutions to the Bellman equation

(48) w(µ) + γ = inf

a∈U

h R

E

c(x, a) µ(dx) + Π

(µ, w) i .

Moreover , there exists u : P(E) → U for which the infimum on the right hand side of (48) is attained. The strategy a

= u(π

) is optimal for J

and

(49) J

((u(π

))) = γ .

P r o o f. By Theorem 1, each ϑ

is concave. Therefore w

is concave and w

as limit of w

is also concave. Since by Proposition 2, the w

are uniformly bounded, and by Proposition 3 equicontinuous, from the Ascoli–

Arzel` a theorem the family w

, β ∈ (0, 1), is relatively compact in C(P(E)).

Moreover, |(1 − β)ϑ

(µ

)| ≤ kck. Therefore one can choose a subsequence β

→ 1 such that

(1 − β

)ϑ

(µ

) → γ and

w

→ w in C(P(E)) as k → ∞.

Letting β

→ 1 in (27) we obtain (48).

The remaining assertion of the theorem follows easily from Theorem 3.2.2 of [4].

References

[1] G. B. D i M a s i and L. S t e t t n e r, On adaptive control of a partially observed Markov chain, Applicationes Math., to appear.

[2] E. F e r n a n d e z - G a u c h e r a n d, A. A r a p o s t a t i s and S. J. M a r c u s, Adaptive con- trol of a partially observed controlled Markov chain, in: Stochastic Theory and Adap- tive Control, T. E. Duncan and B. Pasik-Duncan (eds.), Lecture Notes in Control and Inform. Sci. 184, Springer, 1992, 161–171.

[3] —, —, —, On partially observable Markov decision processes with an average cost criterion, Proc. 28 CDC, Tampa, Florida, 1989, 1267–1272.

[4] O. H e r n a n d e z - L e r m a, Adaptive Markov Control Processes, Springer, New York, 1989.

[5] H. K o r e z l i o g l u and G. M a z z i o t t o, Estimation recursive en transmission nu- merique, Proc. Neuvi`eme Colloque sur le Traitement du Signal et ses Applications, Nice, 1983.

[6] M. K u r a n o, On the existence of an optimal stationary J -policy in non-discounted Markovian decision processes with incomplete state information, Bull. Math. Statist.

17 (1977), 75–81.

[7] H. L. R o y d e n, Real Analysis, Macmillan, New York, 1968.

[8] W. J. R u n g g a l d i e r and L. S t e t t n e r, Nearly optimal controls for stochastic ergodic problems with partial observation, SIAM J. Control Optim. 31 (1993), 180–218.

[9] K. W a k u t a, Semi-Markov decision processes with incomplete state observation—

average cost criterion, J. Oper. Res. Soc. Japan 24 (1981), 95–108.

LUKASZ STETTNER

INSTITUTE OF MATHEMATICS POLISH ACADEMY OF SCIENCES P.O. BOX 137

00-950 WARSZAWA, POLAND

E-mail: STETTNER@IMPAN.IMPAN.GOV.PL

Received on 22.6.1992