Stable invariant subspaces for operators on Hilbert space by John B. Conway (Knoxville, Tenn.) and

POLONICI MATHEMATICI LXVI (1997)

Stable invariant subspaces for operators on Hilbert space by John B. Conway (Knoxville, Tenn.) and

Don Hadwin (Durham, N.H.)

Abstract. If T is a bounded operator on a separable complex Hilbert space H, an in- variant subspace M for T is stable provided that whenever {T n } is a sequence of operators such that kT n −T k → 0, there is a sequence of subspaces {M n }, with M n in Lat T n for all n , such that P M

→ P M in the strong operator topology. If the projections converge in norm, M is called a norm stable invariant subspace. This paper characterizes the stable invariant subspaces of the unilateral shift of finite multiplicity and normal operators. It also shows that in these cases the stable invariant subspaces are the strong closure of the norm stable invariant subspaces.

Throughout this paper, H is a separable complex Hilbert space and B(H) is the algebra of bounded operators on H. All notation and terminology used here will be that of [6]. In particular, Lat T denotes the lattice of invariant subspaces of T for any operator T in B(H).

Usually, a closed subspace M of H will be identified with the orthogonal projection onto it, P M . So, in particular, the collection of all closed subspaces of H will be identified with the collection P of all projections in B(H). When the convergence of a sequence of subspaces in the norm or strong topology is mentioned, this is actually a statement about the corresponding convergence of the associated sequence of projections.

If T is a bounded operator on H, an invariant subspace M for T is stable provided whenever {T ⁿ } is a sequence of operators such that kT ⁿ − T k → 0, there is a sequence of subspaces {M n }, with M n in Lat T n for all n, such that P M

→ P ^M in the strong operator topology. The concept needs little justification and has been examined by several previous authors. In the finite-dimensional setting it appears in [4] and [5]. Also, see Appendix 1 in [2] for a characterization of the stable invariant subspace of an operator on a finite-dimensional space. The idea of a stable invariant subspace is also implicit in [10] and [9]. In [1] and [3] a stronger concept of stable invariant

1991 Mathematics Subject Classification: 47A15, 47B15.

Key words and phrases : invariant subspace, stability, normal operators.

[49]

subspace is considered, where, in the definition, it is required that the se- quence of projections converges in norm rather than in the strong operator topology. Here we will call such spaces norm stable.

The concept of stability is also related to the examination of “Lat” as a function and its points of continuity. Because the underlying Hilbert space is separable, the strong operator topology is metrizable on bounded subsets of B(H). Thus (P, SOT) is a metric space. Let C denote the collection of closed subsets of P and furnish C with the Hausdorff metric defined by using the SOT metric. Thus we have a function Lat : B(H) → C. It is a good exercise in the definitions to show that this function is continuous at an operator T if and only if every invariant subspace for T is stable. For a finite-dimensional space, it was proved in [7] that Lat is continuous at T if and only if T is cyclic (or non-derogatory). Later in this paper it will be shown that Lat is continuous at a shift of finite multiplicity.

Similar comments apply if norm stable invariant subspaces are considered and, in the preceding discussion, the norm topology replaces the strong operator topology on the set of projections.

1. Preliminaries. Denote by Lat _s T the collection of stable invariant subspaces of T and by Lat ns T the norm stable invariant subspaces of T . Clearly, Lat ns T ⊆ Lat s T and it is easy to check that Lat ns T contains both {0} and H. The proof of the first lemma is an exercise.

1.1. Lemma. For any operator T , Lat s T is strongly closed and Lat ns T is norm closed.

An obvious question is the following.

1.2. Question. For any operator T , is Lat s T the strong closure of Lat ns T ?

The results of this paper support an affirmative answer. It will be shown that this is the case for the unilateral shift of finite multiplicity and for normal operators. However, this does not constitute sufficient evidence to warrant a conjecture at this time. It is not known, for example, whether the answer to this question is affirmative for the unilateral shift of infinite multiplicity.

1.3. Lemma. If T _n → T (SOT ), P n = cl(ran T _n ), and P = cl(ran T ), then P n P → P (SOT ).

P r o o f. For any vector h,

kP ⁿ T h − T hk = kP ⁿ (T h − T ⁿ h) + T n h − T hk ≤ 2kT h − T ⁿ h k.

Since kP ⁿ k ≤ 1 for all n, for any vector f we have kP ⁿ P f − P fk → 0 as

n → ∞.

1.4. Lemma. Let {E ⁿ } be a sequence of idempotents and assume that kE n − Ek → 0. If P n is the orthogonal projection onto ran E n and P the orthogonal projection onto ran E, then kP n − P k → 0.

P r o o f. We first establish the following.

Claim. If E is an idempotent and f is a continuous function on [0, ∞) with f (0) = 0 and f (t) = 1 for t ≥ 1, then f(EE ^∗ ) is the projection onto ran E.

Indeed, if M = ran E and we write E as a 2 × 2 matrix with respect to the decomposition H = M ⊕ M ^⊥ , then

E =  1 X 0 0

 . Hence

EE ^∗ =  1 + XX ^∗ 0

0 0

 . The claim now easily follows.

If {E n } is a sequence of idempotents and kE n − Ek → 0, then kE n E _n ^∗ − EE ^∗ k→0. If f is a continuous function on [0, ∞), it follows that kf(E ⁿ E _n ^∗ ) − f (EE ^∗ ) k → 0. It is now apparent that the lemma follows from the claim.

When the spectrum of T is not connected, Lat _s T is non-trivial. To econ- omize on words, we adopt the common practice of calling sets clopen if they are simultaneously closed and open.

1.5. Lemma. If U is a relatively clopen subset of σ(T), then the Riesz subspace of T corresponding to U belongs to Lat ns T.

P r o o f. Assume that kT ⁿ − T k → 0. If U and V are open sets in the plane that both meet σ(T ) and satisfy cl U ∩ cl V = ∅, then U ∪ V is an open set containing σ(T ). Thus for all sufficiently large n, σ(T n ) ⊆ U ∪ V . Consequently, U ∩ σ(T ⁿ ) is a relatively clopen set. An examination of the definition of the Riesz idempotent of T n corresponding to U shows that E _n = E(T _n ; U ) → E(T ; U) = E in norm. An application of the preceding lemma shows that E H ∈ Lat ^ns T .

The next lemma says that for surjective operators, the kernel is a norm stable invariant subspace.

1.6. Lemma. If T is a surjective operator and kT n − T k → 0, then kker T n − ker T k → 0.

P r o o f. Since T is surjective, T T ^∗ is invertible; hence T n T _n ^∗ is invert-

ible for all sufficiently large n and (T n T _n ^∗ ) ⁻¹ → (T T ^∗ ) ⁻¹ . Consider the

polar decompositions T = (T T ^∗ ) ^1/2 W and T n = (T n T _n ^∗ ) ^1/2 W n . Thus W n =

(T n T _n ^∗ ) ^{− 1/2} T n → W . But W n W _n ^∗ = ker T n and W W ^∗ = ker T .

It is an elementary exercise that if {P ⁿ } is a sequence of projections and kP n −P k → 0, then for any projection Q ≤ P , there are projections Q n ≤ P n

such that kQ n − Qk → 0. With this the next corollary is immediate.

1.7. Corollary. If T is surjective and M ≤ ker T , then M is a norm stable invariant subspace for T.

The next result constitutes a small diversion, but one with some interest.

1.8. Proposition. If B(H) has the norm topology and P has the strong operator topology, then the following statements are equivalent:

(a) the map X → cl(ran X) is continuous at T ^∗ ; (b) the map X → ker X is continuous at T ; (c) T is either surjective or injective.

P r o o f. (a) ⇒(c). Assume that (c) is false. We will construct a sequence of operators {X n } such that kX n − T ^∗ k → 0 but {cl(ran X n ) } does not converge to cl(ran T ^∗ ) in the strong operator topology. To say that (c) is false is to say that (ran T ^∗ ) ^⊥ = ker T 6= (0) and ran T 6= H.

First suppose that ran T is not dense and choose unit vectors e in ker T and f in (ran T ) ^⊥ . Define X n by X n h = T ^∗ h + n ^{− 1} hh, fie. Because e ⊥ ran T ^∗ , ran X n = ran T ^∗ + Ce and the sequence {X n } is as promised.

Now suppose that ran T is dense but not the whole space. Since ran T is not closed, neither is ran T ^∗ ([6], VI.1.10). Hence 0 is in the left essential spectrum of T ^∗ and so for each integer n ≥ 1 there is an infinite-dimensional space M ⁿ such that sup {kT ^∗ h k : h ∈ M ⁿ and khk = 1} < 1/n ([6], XI.2.3).

Let S n : M ⁿ → M ⁿ be a backward unilateral shift of multiplicity 1 on M ⁿ and let e _n be a unit vector in ker S _n . Fix a unit vector e in ker T and define X n by

X n h =  T ^∗ h when h ∈ M ^⊥ n ,

X n h = T ^∗ S n h + n ^{− 1} hh, e n ie when h ∈ M n .

Thus kX ⁿ − T ^∗ k ≤ 3n ⁻¹ and, since S n ( M ⁿ ⊖ e ⁿ ) = M ⁿ , cl(ran X n ) = cl(ran T ^∗ ) + Ce.

(c) ⇒(b). If we assume that T is surjective, the result is in Lemma 1.6.

If T is injective, assume that {T ⁿ } is a sequence of operators such that kT n − T k → 0 and let P n = ker T n . If Q is any WOT cluster point of {P n }, then T _n P _n → cl T Q (WOT), But T _n P _n = 0 for each n, so T Q = 0. Hence Q = 0 and, as the unique WOT cluster point of {P ⁿ }, Q is the WOT limit of {P n }. Hence we also have P n → 0 (SOT).

(b) ⇒(a). If kX ⁿ − T ^∗ k → 0, then kX n ^∗ − T k → 0 and so, by (b), ker X _n ^∗ → ker T (SOT). Thus (ker X n ^∗ ) ^⊥ → (ker T ) ^⊥ (SOT), whence (a).

2. Shifts of finite multiplicity. In this section we will give a proof

that for a shift S of finite multiplicity, Lat S = Lat s S. This was essentially

obtained in [3], where the norm stable invariant subspaces of any shift S are characterized. That paper shows that if M is a proper invariant subspace for S ^∗ , with S a shift of any multiplicity, then M is norm stable if and only if the spectral radius of S ^∗ |M, written r(S ^∗ |M), is strictly less than 1. If dim M < ∞, then this spectral radius must be less than 1 since S ^∗ has no eigenvalues on the unit circle. Conversely, if it is also assumed that S has finite multiplicity, then the first lemma below shows that the only invariant subspaces of S ^∗ for which the corresponding spectral radius is less than 1 are the finite-dimensional ones. So it follows from [3] that the non-zero norm stable invariant subspaces for a shift of finite multiplicity are the invariant subspaces of finite codimension. As will be shown later in this section, for S having finite multiplicity, the elements of Lat S having finite codimension are strongly dense in Lat S; this shows that such a shift is a point of continuity of Lat.

It is not known whether this is true for a shift of infinite multiplicity.

In particular, if S is a shift of infinite multiplicity and M ∈ Lat S ^∗ , it is unknown whether there is a sequence of subspaces {M ⁿ } in Lat S ^∗ such that r(S ^∗ |M n ) < 1 for all n and P M

→ P ^M (SOT).

2.1. Lemma. If S is a unilateral shift of finite multiplicity, M ∈ Lat S such that M ^⊥ is infinite-dimensional, and P = P M , then kP ^⊥ SP ^⊥ k = 1 = r(P ^⊥ SP ^⊥ ) (the spectral radius of P ^⊥ SP ^⊥ ).

P r o o f. Let T = P ^⊥ SP ^⊥ ; so r(T ) ≤ kT k ≤ 1. Because S has finite multiplicity, dim ker S ^∗n < ∞. Because M ^⊥ is infinite-dimensional, for each n ≥ 1 there is a unit vector g ⁿ in M ^⊥ that is orthogonal to ker S ^∗n . Hence 1 = kS ^{∗ n} g n k = kT ^{∗ n} g n k. So kT ^{∗ n} k = 1 and kT ⁿ k = 1. This implies that r(T ) = lim n kT ⁿ k ^1/n = 1.

2.2. Proposition. If T is any operator on the Hilbert space H with kT k = 1 and M is an invariant subspace for T such that there is a λ in σ(T |M) ∩ σ(P ^⊥ T |M ^⊥ ) with |λ| = 1, then M is not norm stable.

P r o o f. Without loss of generality we may assume that λ = 1. If ε, δ > 0, the fact that 1 belongs to the approximate point spectrum of both T |M and P ^⊥ T |M ^⊥ implies there are unit vectors f in M and g in M ^⊥ such that kT f − fk < ε and kP ^⊥ T g − gk < δ. Now

kP T gk ² = kT gk ² − kP ^⊥ T g k ² ≤ 1 − (kgk − kP ^⊥ T g − gk) ²

< 1 − (1 − δ) ² = 2δ − δ ² .

Thus kT g − gk ² = kP ^⊥ T g − gk ² + kP T gk ² < 2δ. Take δ = ε ² /2 and let e = (f + g)/ √

2.

So for each ε > 0 there is a unit vector e ε such that kT e ^ε − e ^ε k < ε, ke ε − P e ε k = 1/ √

2, and kP e ε k = 1/ √

2. Put L ε = Ce ε and consider the

representation of T as a 2 × 2 matrix with respect to the decomposition H = L ε ⊕ L ^⊥ ε . It follows that there are numbers ε 1 and ε 2 that converge to 0 as ε → 0 and operators B ε and Z _ε with kB ε k ≤ 1 and kZ ε k → 0 as ε → 0 such that

T =  1 + ε 1 Z ε

ε ₂ B _ε

 . Let

A ε =  1 0 0 (1 − ε)B ε

 .

It follows that kT − A ε k → 0 as ε → 0. Since k(1 − ε)B ε k < 1, Lat A ε = {(0), L ε } ⊕ Lat B ε . If it were the case that M is norm stable, then there would be an M ^ε in Lat A ε such that kP − P ^ε k → 0 as ε → 0. But from the splitting property of Lat A ε we deduce that for every ε either L ε ≤ M ε or L ε ≤ M ^⊥ ε . Hence kP ε − P k ≥ k(P ε − P )e ε k ≥ 1/ √

2, a contradiction.

The next corollary is proved by combining the preceding proposition with Lemma 2.1.

2.3. Corollary. If S is a shift of finite multiplicity and M is a norm stable invariant subspace, then dim M ^⊥ < ∞.

Let H ² (D; H) denote the Hardy space of H-valued analytic functions and let H ^∞ (D; B(H)) be the space of bounded analytic B(H)-valued functions.

Say a function Q in H ^∞ (D; B(H)) is inner if Q(z) is an isometry on H for a.e. z in ∂D. If Q is an inner function in H ^∞ (D; B(H)), let E Q be the orthogonal projection of H ² (D; H) onto QH ² (D; H). It is a standard exercise that the linear span of functions of the form z ⁿ e, where n ≥ 0 and e ∈ H, is dense in H ² (D; H)

2.4. Lemma. If Q is an inner function in H ^∞ (D; B(H)) and E Q is the orthogonal projection of H ² (D; H) onto QH ² (D; H), then for any vector e in H and any non-negative integer n,

E Q (z ⁿ e) = Q

n

X

k=0

1

k! z ^n−k Q ^(k) (0) ^∗ e.

P r o o f. If h is the right hand side of the preceding equation, then, since h is clearly in QH ² (D; H), it suffices to show that z ⁿ e −h ⊥ QH ² (D; H). Thus the proof will be accomplished by showing that hz ⁿ e, z ^m Qx i = hh, z ^m Qx i for all m ≥ 0 and all x in H. Now

hz ⁿ e, z ^m Qx i =

\

z ^n−m he, Q(z)xi dm(z).

If f (z) = hQ(z)x, ei, then f ∈ H ² and so

\

f (z)z ^n−m dm(z) =





 1

(n − m)! f ^(n−m) (0) if n ≥ m,

0 if n < m,

=

( 1

(m − n)! hQ ^(n−m) (0)x, e i if n ≥ m,

0 if n < m.

Hence

hz ⁿ e, z ^m Qx i =

( 1

(n − m)! he, Q ^(n−m) (0)x i if n ≥ m,

0 if n < m.

Now

hh, z ^m Qx i =

n

X

k=0

1 k!

\

z ^n−k−m hQ(z)Q ^(k) (0) ^∗ e, Q(z)x i dm(z).

But Q(z) is an isometry a.e. [m], so hh, z ^m Qx i =

n

X

k=0

1 k!

\

z ^n−k−m hQ ^(k) (0) ^∗ e, x i dm(z) = hz ⁿ e, z ^m Qx i.

2.5. Proposition. If dim H < ∞ and Q, Q 1 , Q 2 , . . . are inner functions in H ^∞ (D; B(H)) and Q m (z) → Q(z) in B(H) uniformly for z in compact subsets of D, then E Q

→ E ^Q (SOT ) in B(H ² (D; H)).

P r o o f. Under the hypothesis, for every k ≥ 0, Q ^{(k) m} (0) → Q ^(k) (0) as m → ∞. If {e ¹ , . . . , e d } is a basis for H, then {z ⁿ e j : n ≥ 0, 1 ≤ j ≤ d} is a basis for H ² (D; H). From the preceding lemma,

hE Q

(z ⁿ e j ), z ^p e q i =

n

X

k=0

1 k!

\

z ^n−k−p hQ m (z)Q ^(k) (0) ^∗ e j , e q i dm(z).

It now follows that hE Q

(z ⁿ e j ), z ^p e q i → hE Q (z ⁿ e j ), z ^p e q i for all possible n, j, p and q. Since all the operators here are projections, E _Q

→E Q (SOT) in B(H ² (D; H)).

2.6. Theorem. If S is a unilateral shift of finite multiplicity, then every invariant subspace is stable.

P r o o f. We begin by showing that the invariant subspaces of finite codi-

mension are norm stable. To do this, it is somewhat more convenient to con-

sider T = S ^∗ rather than S. So assume that M is a finite-dimensional invari-

ant subspace of T . Thus σ(T |M) ⊆ σ ^p (T ) ⊆ D. Let σ(T |M) = {λ ¹ , . . . , λ q },

let M 1 , . . . , M q be the corresponding Riesz subspaces for T |M, and let

k i be the smallest positive integer such that (T |M ⁱ − λ ⁱ ) ^k

= 0. Thus

M i ⊆ ker(T − λ i ) ^k

.

Since T is the adjoint of the shift and |λ ⁱ | < 1, (T − λ ⁱ ) ^k is surjective for every positive integer k. So if {T n } is a sequence of operators that converges to T , then

kker(T ⁿ − λ ⁱ ) ^k

− ker(T − λ ⁱ ) ^k

k → 0.

By Corollary 1.7, each M ⁱ is norm stable. The fact that M ⁱ ∩ M ^j = (0) for i 6= j and the linear span of M 1 , . . . , M q is the closed subspace M implies, by a small argument, that M is norm stable.

The proof will be completed by showing that these invariant subspaces are strongly dense in Lat S and invoking Lemma 1.1. Represent S as mul- tiplication by the independent variable on H ² (D; H), where H is a finite- dimensional Hilbert space.

It is well known that the invariant subspaces for S are precisely the subspaces of the form QH ² (D; H), for some B(H)-valued inner function Q.

By a result of Herrero [11], the fact that H is finite-dimensional implies that every inner function in H ^∞ (D; B(H)) is the uniform limit of Blaschke products. But if Q is a Blaschke product, then there is a sequence of Blaschke products such that Q m (z) → Q(z) uniformly on compact subsets of D and each Q m H ² (D; H) has finite codimension in H ² (D; H) (cf. [8]). The result now follows by the preceding proposition.

Note that the preceding theorem gives an affirmative answer to Ques- tion 1.2 when the operator is a unilateral shift of finite multiplicity. If S is a unilateral shift of infinite multiplicity, the norm stable invariant subspaces of S are characterized in [3], but it remains unknown whether Question 1.2 has an affirmative answer in this case.

3. Stable invariant subspaces for normal operators. Throughout this section N will denote a normal operator with spectral decomposition N =

T

z dE(z). From §1 we know that Lat ^s N is strongly closed and contains the spectral subspaces corresponding to the clopen subsets of σ(N ). The main result of this section is that also the converse is true.

3.1. Theorem. For a normal operator N on a separable Hilbert space K, Lat s N consists of those spectral subspaces of N that are strong limits of spaces of the form E(U ) H for U a clopen subset of σ(N).

The norm stable invariant subspaces of a normal operator were charac- terized in [3] as the spectral subspaces corresponding to clopen subsets of σ(N ). This result is not needed here, nor are the results from the preceding section.

For any subset X of the plane, let (X) ε ≡ {z : dist(z, X) < ε}.

3.2. Lemma. Let N be a normal operator , let ε > 0, and let C 1 , . . . , C n be

components of σ(N ) such that σ(N ) ⊆ (C 1 ) ε ∪ . . . ∪ (C n ) ε . If {U 1 , . . . , U n }

is a cover of σ(N ) by open sets with pairwise disjoint closures such that (i) U i ∩ σ(N) is clopen,

(ii) C i ⊆ U ⁱ ⊆ cl U ⁱ ⊆ (C ⁱ ) ε , (iii) σ(N ) ⊆ S n

i=1 U i , then there are operators Q = L n

i=1 Q i and R = L n

i=1 R i on L n

i=1 E(U i ) H that satisfy the following.

(a) kN − Qk < 2ε and kN − Rk < 2ε.

(b) For 1 ≤ i ≤ n, σ(Q i ) and σ(R i ) are single points in U i . (c) For 1 ≤ i ≤ n, Lat Q ⁱ = Lat R ^∗ _i and these lattices are nests.

P r o o f. Put H i = E(U i ) H and N i = N |H i . There are two cases to consider depending on whether H i is finite- or infinite-dimensional. If H i is finite-dimensional, it suffices to consider the case where U i contains a single eigenvalue of N , say λ. So there is a basis for H i with respect to which the matrix of N _i is diagonal with entry λ. In this case we can take for Q _i and R i the matrices

Q i =













λ ε 0 . . . 0 0 λ ε . . . 0 .. . . .. ... .. . 0 0 . . . λ ε 0 0 . . . 0 λ













, R i =













λ 0 0 . . . 0 ε λ 0 . . . 0 .. . . .. ... .. . 0 . . . ε λ 0 0 . . . 0 ε λ











 .

Now assume that H i is infinite-dimensional. Construct a smooth Jordan arc γ i : [0, 1] → U ⁱ such that C i ⊆ (γ ⁱ ) ε .

Claim. There is a diagonal normal operator M i with σ(M i ) = γ i and kN i − M i k < 3ε.

Let D i be a diagonal normal operator with finite spectrum such that σ(D i ) ⊆ σ(N i ), each eigenvalue of D i that belongs to C i has infinite multi- plicity, C _i ⊆ σ(D i ) _ε , and kD i − N i k < ε. Let Γ be a countable dense subset of γ i . Thus each point of Γ is within a distance 2ε of infinitely many eigen- values of D i (counting multiplicities) and every point of σ(D i ) is within 2ε of infinitely many points of Γ . Matching eigenvalues, we can find a diagonal normal operator M i such that kM ⁱ − D ⁱ k < 2ε and whose eigenvalues are precisely the set Γ . This establishes the claim.

Let A _i = γ _i ⁻¹ (M _i ). So A _i is a self-adjoint operator with spectrum [0, 1]

and γ i (A i ) = M i . Let p i be a polynomial with p ^′ _i (0) 6= 0 such that |p(t)−

γ i (t) | < ε for all t in [0, 1]. Thus kp i (A i ) −M i k<ε. Let B i be a quasinilpotent

operator with Lat B i totally ordered and such that kB ⁱ −A ⁱ k and kB ^∗ i −A ⁱ k

are sufficiently small that kp ⁱ (B i ) − p ⁱ (A i ) k < ε and kp ⁱ (B _i ^∗ ) − p ⁱ (A i ) k < ε

(cf. [12]). Put Q i = p i (B i ). Since p ^′ _i (0) 6= 0, it follows that p i is one-to-one

near 0 and so p ⁻ _i ¹ is a well-defined analytic function in a neighborhood of

σ(Q i ) = {p i (0) }. By Runge’s Theorem, B i and Q i generate the same norm

closed algebras, so that Lat B i = Lat Q i . Similarly, if R i = p i (B _i ^∗ ), then Lat R i = Lat B _i ^∗ = Lat Q ^∗ _i and kR i − p i (A i ) k < ε.

Letting Q and R be defined as in the statement of the lemma, we see that kN −Qk < 5ε, kN −Rk < 5ε, and conditions (b) and (c) are satisfied.

3.3. Lemma. If N is a normal operator and M is a stable invariant subspace, then M reduces N.

P r o o f. An application of the Spectral Theorem shows that there is a sequence {N k } of cyclic, reductive normal operators that converges to N in norm. If P is the projection onto M, then there is a sequence of projections {P ^k } such that P ^k ∈ Lat N ^k and P k → P (SOT). Since each N ^k is reductive, N k P k − P k N k = 0. By taking limits we see that N P = P N and so M reduces N .

P r o o f o f T h e o r e m 3.1. We already know from Lemma 1.5 that Lat _s N contains E(U ) H for each clopen subset U of σ(N) and hence the strong limits of such spectral projections (Lemma 1.1). So let M ∈ Lat ^s N and let P be the projection of H onto M.

For each positive integer n, let {C ni : 1 ≤ i ≤ p n } be a collection of components of σ(N ) and let {U ⁿⁱ : 1 ≤ i ≤ p ⁿ } be open sets such that the following are satisfied.

(i) cl U ni ∩ cl U ^nj = ∅ for i 6= j.

(ii) C ni ⊆ U ⁿⁱ ⊆ (C ⁿⁱ ) 1/n . (iii) σ(N ) ⊆ S p

i=1 U ni .

Put E ni = E(U ni ) and H ⁿⁱ = E ni H. According to Lemma 3.2 we can find operators Q n = L p

i=1 Q ni and R = L p

i=1 R ni on L p

i=1 H ni that satisfy the following.

(a) kN − Q n k < 2/n and kN − R n k < 2/n.

(b) For 1 ≤ i ≤ p ⁿ , σ(Q ni ) and σ(R ni ) are single points in U ni . (c) Lat Q ni = Lat R ^∗ _ni for 1 ≤ i ≤ p ⁿ and these lattices are nests.

From Lemma 3.3 we know that P N = N P and so P = P

i P E ni = P

i E ni P . Since P is stable, there are projections A n and B n in Lat Q n

and Lat R n , respectively, such that A n → P and B n → P (SOT). From condition (b) we have A _n = L p

i=1 A _ni and B _n = L p

i=1 B _ni with A _ni and B ni in Lat Q ni and Lat R ni , respectively. Note that by (c), for each i we see that either A ni ≥ E ni − B ni or A ni ≤ E ni − B ni .

Let

E n = X

{E ni : A ni ≥ E ni − B ni }.

By passing to a subsequence if necessary, we may assume that there is an

operator E with 0 ≤ E ≤ 1 such that E n → E in the weak operator topology

(WOT). It will now be shown that E = P . Since each E n is a spectral projection this will imply that E n → P (SOT), proving the theorem.

To do this, first observe that E n A n = A n E n ≥ E ⁿ (1 −B ⁿ ) = (1 −B ⁿ )E n . Now observe that for two commuting sequences of operators, one of which converges in the strong operator topology and the other in the weak operator topology, the product converges in the weak operator topology. Thus taking WOT limits in the above inequalities, we get EP ≥ E(1 − P ). Multiplying both sides by 1 − P and again using commutativity, this implies that 0 ≥ (1 − P )E, which is a positive operator. Thus E = EP = P E. On the other hand,

(1 − E ⁿ )A n = A n

X {E ⁿⁱ : A ni ≤ E ⁿⁱ − B ⁿⁱ } ≤ E ⁿ (1 − B ⁿ ).

Again taking limits we get (1 − E)P ≤ (1 − E)(1 − P ). Multiplying both sides by P we deduce that (1 − E)P = 0 or P = P E. Therefore P = E.

Recall from the introduction that an operator T is a point of continuity of the function Lat : H → C if and only if Lat T = Lat s T .

3.4. Corollary. A normal operator N is a point of continuity of Lat if and only if N is cyclic, reductive, and every spectral projection is the strong operator topology limit of a sequence of spectral projections corresponding to clopen subsets of σ(N ).

P r o o f. If N =

T

z dE(z) is a cyclic, reductive normal operator, then every invariant subspace is the range of a spectral projection. If it is also assumed that for each Borel set ∆ there is a sequence {U n } of clopen subsets of σ(N ) such that E(U _n ) → E(∆) (SOT), then Theorem 3.1 says that Lat N = Lat s N .

Conversely, assume that N is a point of continuity of Lat. It follows from Theorem 3.1 that each invariant subspace for N reduces N and that the projection onto this subspace is a spectral projection. Thus N is cyclic and reductive. The rest of the corollary follows from the theorem.

Note that if a normal operator is a point of continuity of Lat, it is not necessary for it to have totally disconnected spectrum. For example, if K is the compact set consisting of the closed unit interval together with the

“snowflakes” {(1/n, k/n) : n ≥ 1 and 1 ≤ k ≤ n} and µ is any measure assigning positive measure to each snowflake and no measure to the interval, then the normal operator N = M z on L ² (µ) is a point of continuity of Lat and σ(N ) = K.

This paper concludes with a result that touches its subject matter but

has an unusual hypothesis. Recall that an operator T is biquasitriangular if

for each scalar λ such that T − λ is semi-Fredholm, the Fredholm index of

T − λ is 0. Theorem 6 of [9] shows that the closure of the set of unicellular

operators (those for which the lattice of invariant subspaces is a nest) is the set of biquasitriangular operators with connected spectrum and essential spectrum. Thus for any biquasitriangular operator T with connected spec- trum and essential spectrum, Lat s T is linearly ordered. On the other hand, in [3] it was shown that for a biquasitriangular operator T with connected spectrum and essential spectrum, Lat ns T is trivial.

This leads to the possibility that Question 1.2 has a negative answer for a biquasitriangular operator T with connected spectrum and essential spectrum. The final result says that Question 1.2 is related with a some- what better known problem in operator theory. Recall that an operator T is transitive if it has no non-trivial invariant subspaces.

3.5. Proposition. If there exists a transitive operator and T is a biqua- sitriangular operator with connected spectrum and essential spectrum , then Lat s T is trivial.

P r o o f. It is shown in [9] that if there is a transitive operator, then the closure of the set of all transitive operators is the set of biquasitriangular operators with connected spectrum and essential spectrum. This proves the proposition.

Thus if the answer to Question 1.2 is negative for biquasitriangular oper- ators with connected spectrum and essential spectrum, then every operator has a non-trivial invariant subspace. It would be interesting to know whether Question 1.2 is equivalent to the Invariant Subspace Problem.

References

[1] G. T. A d a m s, A nonlinear characterization of stable invariant subspaces, Integral Equations Operator Theory 6 (1983), 473–487.

[2] C. A p o s t o l, L. A. F i a l k o w, D. A. H e r r e r o, and D. V o i c u l e s c u, Approximation of Hilbert Space Operators, Vol. II, Pitman Res. Notes Math. 102, Pitman, Boston, 1984.

[3] C. A p o s t o l, C. F o i a¸s, and N. S a l i n a s, On stable invariant subspaces, Integral Equations Operator Theory 8 (1985), 721–750.

[4] H. B a r t, I. G o h b e r g, and M. A. K a a s h o e k, Stable factorizations of monic matrix polynomials and stable invariant subspaces, ibid. 1 (1978), 496–517.

[5] S. C a m p b e l l and J. D a u g h t r y, The stable solutions of quadratic matrix equa- tions, Proc. Amer. Math. Soc. 74 (1979), 19–23.

[6] J. B. C o n w a y, A Course in Functional Analysis, Springer, New York, 1990.

[7] J. B. C o n w a y and P. R. H a l m o s, Finite-dimensional points of continuity of Lat, Linear Algebra Appl. 31 (1980), 93–102.

[8] Yu. P. G i n z b u r g, The factorization of analytic matrix functions, Dokl. Akad. Nauk SSSR 159 (3) (1964), 489–492 (in Russian).

[9] D. W. H a d w i n, An addendum to limsups of lats, Indiana Univ. Math. J. 29 (1980),

313–319.

[10] P. R. H a l m o s, Limsups of lats, ibid., 293–311.

[11] D. A. H e r r e r o, Inner functions under uniform topology, II , Rev. Un. Mat. Ar- gentina 28 (1976), 23–35.

[12] —, Approximation of Hilbert Space Operators, I , Pitman, London, 1982.

Mathematics Department of Mathematics

University of Tennessee University of New Hampshire

Knoxville, Tennessee 37996-1300 Durham, New Hampshire 03824

U.S.A. U.S.A.

E-mail: conway@novell.math.utk.edu E-mail: don@christa.unk.edu Web: http://www.math.utk.edu/˜conway/

Re¸ cu par la R´ edaction le 23.10.1995