1. Introduction. Let ∆ = {z ∈ C | |z| < 1} be the unit disc in C. The Schwarz lemma in one complex variable is as follows.

POLONICI MATHEMATICI LXVII.3 (1997)

A Schwarz lemma on complex ellipsoids

by Hidetaka Hamada (Kitakyushu)

Abstract. We give a Schwarz lemma on complex ellipsoids.

1. Introduction. Let ∆ = {z ∈ C | |z| < 1} be the unit disc in C. The Schwarz lemma in one complex variable is as follows.

Theorem 1. (i) Let f : ∆ → ∆ be a holomorphic map such that f (0) = 0. Then |f (z)| ≤ |z| for all z ∈ ∆.

(ii) If , moreover , there exists z

∈ ∆ \ {0} such that |f (z

)| = |z

|, or if

|f

(0)| = 1, then there exists a complex number λ of absolute value 1 such that f (z) = λz and f is an automorphism of ∆.

Let D be the unit ball in C

for some norm k · k, and let f : D → D be a holomorphic map such that f (0) = 0. By the Hahn–Banach theorem, we have kf (z)k ≤ kzk for all z ∈ D. As a generalization of part (ii) of the above theorem, Vigu´ e [7] proved the following.

Theorem 2. Let D be the unit ball in C

for some norm k · k, and let f : D → D be a holomorphic map such that f (0) = 0. Assume that every boundary point of D is a complex extreme point of D. If one of the following conditions is satisfied , then f is a linear automorphism of C

.

(H

) There exists a nonempty open subset U of D such that kf (x)k = kxk on U .

(H

) There exists a nonempty open subset U of D such that c

(f (0), f (x)) = c

(0, x) on U , where c

denotes the Carath´ eodory distance on D.

(H

) There exists a nonempty open subset V of T

(D) such that E

(f (0), f

(0)v) = E

(0, v) on V , where E

denotes the infinites- imal Carath´ eodory metric on D.

1991 Mathematics Subject Classification: Primary 32A10.

Key words and phrases: Schwarz lemma, complex ellipsoid, extreme point, balanced domain, Minkowski function, geodesics.

[269]

Moreover, he showed that if there exists a point a ∈ U \ {0} such that f (a) = a, or if the boundary ∂D of D is a real-analytic submanifold of C

, then f is a linear automorphism of D. As a corollary, he proved that if D is the unit ball of C

for the Euclidean norm on C

, then f is a linear automorphism of D. But, in the above results, the conditions (H

) and (H

) are strong, because a point in ∆ is of codimension 1 and an open set in D is of codimension 0. The author [2] announced that Vigu´ e’s results hold under the hypothesis that one of the conditions (H

), (H

) is satisfied for some local complex submanifold of codimension 1 instead of an open subset.

The aim of the present paper is to consider an analogous result on com- plex ellipsoids E (p). However, E (p) is not convex in general. For a bounded balanced convex domain D, the Minkowski function h of D is a norm on C

and D is the unit ball in C

with respect to this norm. Also, c

= e k

and E

= κ

in the convex case (Lempert [4], [5], Royden–Wong [6]), where e k

is the Lempert function and κ

is the the Kobayashi–Royden pseudometric for D. So we use h, e k

and κ

instead of k · k, c

and E

. First we give a theorem on some bounded balanced pseudoconvex domains which corre- sponds to Theorem 2. Then we show that if D = E (p), then f is a linear automorphism of E (p). We also give an example showing that our hypothesis cannot be weakened.

Some ideas of this paper come from Dini–Primicerio [1] and Vigu´ e [7], [8].

2. Main results. The Lempert function e k

and the Kobayashi–Royden pseudometric κ

for a domain D in C

are defined as follows:

e k

(x, y) = inf{%(ξ, η) | ξ, η ∈ ∆, ∃ϕ ∈ H(∆, D) such that ϕ(ξ) = x, ϕ(η) = y}, κ

(z; X) = inf{γ(λ)|α| | ∃ϕ ∈ H(∆, D), ∃λ ∈ ∆ such that

ϕ(λ) = z, αϕ

(λ) = X}, where % is the Poincar´ e distance on the unit disc ∆ and γ(λ) = 1/(1 − |λ|

).

Let D be a balanced pseudoconvex domain with Minkowski function h in C

. Then we have (Propositions 3.1.10 and 3.5.3 of Jarnicki and Pflug [3])

e k

(0, x) = %(0, h(x)) for any x in D, (1)

κ

(0, X) = h(X) for any X in C

. (2)

Let f be a holomorphic map from D to D such that f (0) = 0. By (1) and the distance decreasing property of the Lempert functions, we have

%(0, h(z)) = e k

(0, z) ≥ e k

(0, f (z)) = %(0, h(f (z))).

Since %(0, r) is increasing for 0 ≤ r < 1, we obtain h(f (z)) ≤ h(z). This is a

generalization of part (i) of the Schwarz lemma to balanced pseudoconvex

domains.

A boundary point x of D is said to be an extreme point of D if there is no non-constant holomorphic mapping g : ∆ → D with x = g(0). For example, C

-smooth strictly pseudoconvex boundary points are extreme points (p. 257 of Jarnicki and Pflug [3]).

A mapping ϕ ∈ H(∆, D) is said to be a complex e k

-geodesic for (x, y) if there exist points ξ, η ∈ ∆ such that ϕ(ξ) = x, ϕ(η) = y, and e k

(x, y) =

%(ξ, η).

A mapping ϕ ∈ H(∆, D) is said to be a complex κ

-geodesic for (z, X) if there exist λ ∈ ∆ and α ∈ C such that ϕ(λ) = z, αϕ

(λ) = X, and κ

(z, X) = γ(λ)|α|.

Using (1), (2) and complex e k

-geodesics or κ

-geodesics, we have the following proposition (cf. Vigu´ e [7], [8], Hamada [2]).

Proposition 1. Let D

be bounded balanced pseudoconvex domains with Minkowski functions h

in C

for j = 1, 2, and let f : D

→ D

be a holomorphic map such that f (0) = 0. Let f (z) = P

m=1

P

(z) be the devel- opment of f in vector-valued homogeneous polynomials P

in a neighborhood of 0, where deg P

= m for each m. Let x ∈ D

\{0}. If one of the following conditions is satisfied , then we have P

(x) = 0 for m ≥ 2.

(H

) h

(f (x)) = h

(x) and f (x)/h

(f (x)) is an extreme point of D

. (H

) e k

(f (0), f (x)) = e k

(0, x) and f (x)/h

(f (x)) is an extreme point

of D

.

(H

) κ

(f (0), f

(0)x) = κ

(0, x) and f

(0)x/h

(f

(0)x) is an extreme point of D

.

P r o o f. By (1), the conditions (H

) and (H

) are equivalent. Let ϕ(ζ) = ζ x

h

(x) .

Then ϕ is a complex e k

-geodesic and κ

-geodesic for (0, x). Suppose that (H

) or (H

) is satisfied. Since

e k

(f ◦ ϕ(0), f ◦ ϕ(h

(x))) = e k

(0, f (x)) = e k

(0, x) = %(0, h

(x)), f ◦ ϕ is a complex e k

-geodesic for (0, f (x)). By Proposition 8.3.5(a) of Jarnicki and Pflug [3],

f ◦ ϕ(ζ) = ζ f (x) h

(f (x)) . Since

f ◦ ϕ(ζ) = X P

 ζ x

h

(x)



= X  ζ h

(x)



P

(x)

in a neighborhood of 0, P

(x) = 0 for m ≥ 2.

Suppose that (H

) is satisfied. Since

κ

(0, f

(0)x) = κ

(0, x) = h

(x) and h

(x)(f ◦ ϕ)

(0) = f

(0)x, f ◦ ϕ is a complex κ

-geodesic for (0, f

(0)x). By Proposition 8.3.5(a) of Jarnicki and Pflug [3],

f ◦ ϕ(ζ) = ζe

f

(0)x h

(f

(0)x)

for some θ ∈ R. The rest of the argument is the same as above. This com- pletes the proof.

The following proposition is a key for proving our theorem (Hamada [2]).

Proposition 2. Let U be an open subset of C

. Let M be a complex submanifold of U of dimension n − 1. Assume that there exists a point a in M such that a + T

(M ) does not contain the origin. Then there exists a neighborhood U

of a in C

such that U

⊂ CM = {tx | t ∈ C, x ∈ M}.

P r o o f. To prove this proposition, it is enough to prove the following claim.

Claim. For any x in M , let g(x) be the intersection point of a + T

(M ) and the complex line through x and the origin O. Then g is a biholomorphic map from a neighborhood W

of a in M onto a neighborhood W

of a in a + T

(M ).

Assume the claim is proved. Since there exists an open neighborhood U

of a in C

such that U

⊂ CW

, we obtain U

⊂ CM.

Now we will prove the claim. By an affine coordinate change, we may assume that a = 0, M = {z

= ψ(z

)} with ψ(0) = 0, dψ(0) = 0, where (z

, z

) ∈ C

. Then (z

, ψ(z

)) gives a local parametrization of M at a, a + T

(M ) = {z

= 0} and O = (b

, . . . , b

) with b

6= 0. Let g(z

, ψ(z

)) = (g

(z

), . . . , g

(z

), 0). Since

g

(z

, ψ(z

)) = b

+ b

b

− ψ(z

) (z

− b

) for sufficiently small z

, we have

∂g

∂z

(0) = δ

(1 ≤ i, j ≤ n − 1).

Therefore g is biholomorphic in a neighborhood W

of a. This completes the proof.

From now on, we assume that D is a bounded balanced pseudoconvex domain in C

which satisfies the following condition:

(∗) For any 1 ≤ j

< . . . < j

≤ n (0 ≤ k ≤ n − 1), let

D = D ∩ {z e

= . . . = z

= 0}

be a domain in C

. Then every point of ∂ e D ∩ (C

)

is an extreme point of e D.

By the above two propositions, we have the following theorem.

Theorem 3. Let D be a bounded balanced pseudoconvex domain with Minkowski function h in C

which satisfies the condition (∗), and let f : D → D be a holomorphic map such that f (0) = 0. Let M be a connected complex submanifold of dimension n − 1 of an open subset U of D such that a+T

(M ) does not contain the origin for some a in M. Let V be a connected open subset of T

(D). If one of the following conditions is satisfied , then f is a linear automorphism of C

.

(H

) h(f (x)) = h(x) on M .

(H

) e k

(f (0), f (x)) = e k

(0, x) on M . (H

) κ

(f (0), f

(0)v) = κ

(0, v) on V.

P r o o f. Suppose that (H

) or (H

) is satisfied. We may assume that for any a ∈ M , a + T

(M ) does not contain the origin, the functions f

, . . . , f

do not vanish on M and the functions f

, . . . , f

are identically 0 on M for some k, 1 ≤ k ≤ n. Let

D = D ∩ {z e

= . . . = z

= 0} and f = (f e

, . . . , f

).

Then e D is a bounded balanced pseudoconvex domain in C

with Minkowski function e h = h| e D, and e f is a holomorphic map from D to e D with e f (0)

= 0. Since the functions f

, . . . , f

do not vanish on M , e f (x)/e h( e f (x)) is an extreme point of e D for any x ∈ M . Let

f (z) = e

∞

X

m=1

P

(z)

be the development of e f in vector-valued homogeneous polynomials P

in a neighborhood of 0, where deg P

= m for each m. Since e h( e f (x)) = h(f (x)) = h(x) on M , we have P

(x) = 0 on a nonempty open subset U

of D for m ≥ 2 by Propositions 1 and 2. By the analytic continuation theorem, P

is identically 0 for m ≥ 2. Therefore e f is linear. By Proposition 2, we have e h( e f (x)) = h(x) on U

. We can show that Ker( e f ) = 0 as in Vigu´ e [7].

Then k must be n and f is a linear automorphism of C

.

Suppose that (H

) is satisfied. We may assume that ∂f

(0), . . . , ∂f

(0) are not 0 and ∂f

(0), . . . , ∂f

(0) are 0 for some k, 1 ≤ k ≤ n. Let

D = D ∩ {z e

= . . . = z

= 0} and f = (f e

, . . . , f

).

Then e D is a bounded balanced pseudoconvex domain in C

with Minkowski

function e h = h| e D, and e f is a holomorphic map from D to e D with e f (0) = 0.

We may assume that ∂f

(0) · (P v

∂/∂z

), . . . , ∂f

(0) · (P v

∂/∂z

) do not vanish for any v ∈ V . Then e f

(0)v/e h( e f

(0)v) is an extreme point of e D for any v ∈ V . The rest of the argument is the same as above. This completes the proof.

For p = (p

, . . . , p

) with p

, . . . , p

> 0, let E(p) = n

(z

, . . . , z

)   

n

X

j=1

|z

|

< 1 o .

Then E (p) is a bounded balanced pseudoconvex domain which satisfies the condition (∗) (cf. p. 264 of Jarnicki and Pflug [3]). Let f be a holomorphic map from E (p) to itself which satisfies the condition of Theorem 3. Then f is a linear automorphism of C

by Theorem 3. Moreover, we can show that f is a linear automorphism of E (p) using the idea of Dini and Primicerio [1].

Theorem 4. Let f be a holomorphic map from E (p) to itself such that f (0) = 0. Let M be a connected complex submanifold of dimension n − 1 of an open subset U of E (p) such that a + T

(M ) does not contain the origin for some a in M. Let V be a connected open subset of T

(D). If one of the following conditions is satisfied , then f is a linear automorphism of E (p).

(H

) h(f (x)) = h(x) on M , where h is the Minkowski function of E (p).

(H

) e k

(f (0), f (x)) = e k

(0, x) on M . (H

) κ

(f (0), f

(0)v) = κ

(0, v) on V .

P r o o f. By Theorem 3 and its proof, we may assume that f is a linear automorphism of C

and there exists an open set U

in E (p) such that on U

, the functions z

, . . . , z

, f

, . . . , f

do not vanish and h(f (x)) = h(x).

Then there exists an open connected set U

in C

such that:

1) U

∩ ∂E(p) 6= ∅,

2) the mapping g = (z

, . . . , z

) is well-defined and 1-1 on U

and f (U

),

3) f (U

∩ ∂E(p)) ⊂ ∂E(p).

Then the map F = g ◦ f ◦ g

is holomorphic and 1-1 on g(U

) and F (g(U

) ∩ ∂B

) ⊂ ∂B

. By the proof of Theorem 1.1 and Corollary 1.2 of Dini and Primicerio [1], f is a linear automorphism of E (p).

Corollary 1. Let f be a holomorphic map from D = {(z

, . . . , z

) | kzk

= P

j=1

|z

|

< 1} (q ≥ 1) to itself such that f (0) = 0. Let M be a

connected complex submanifold of dimension n − 1 of an open subset U of

D such that a + T

(M ) does not contain the origin for some a in M. Let V

be a connected open subset of T

(D). If one of the following conditions is

satisfied , then f is a linear automorphism of D.

(H

) kf (x)k

= kxk

on M.

(H

) e k

(f (0), f (x)) = e k

(0, x) on M.

(H

) κ

(f (0), f

(0)v) = κ

(0, v) on V.

Example 1. Let f (z) = (z

, . . . , z

, z

). Then f maps E (p) into itself and f (0) = 0.

(i) Let M = {z

= 0}. We have h(f (z)) = h(z) on M . Since f is not linear, the condition that a + T

(M ) does not contain the origin cannot be omitted.

(ii) For k ≥ 2, let M

= {z

= b, z

= . . . = z

= 0}, where b 6= 0. The complex dimension of M

is n − k, and for any a ∈ M

, a + T

(M ) does not contain the origin. Since h(f (z)) = h(z) on M

and f is not linear, the condition that the complex dimension of M is n − 1 cannot be omitted.

References

[1] G. D i n i and A. S. P r i m i c e r i o, Proper holomorphic mappings between generalized pseudoellipsoids, Ann. Mat. Pura Appl. (4) 158 (1991), 219–229.

[2] H. H a m a d a, A Schwarz lemma in several complex variables, in: Proc. Third In- ternational Colloquium on Finite or Infinite Dimensional Complex Analysis (Seoul, 1995), Kyushu Univ. Co-op., Fukuoka, Japan, 1995, 105–110.

[3] M. J a r n i c k i and P. P f l u g, Invariant Distances and Metrics in Complex Analysis, de Gruyter, Berlin, 1993.

[4] L. L e m p e r t, Holomorphic retracts and intrinsic metrics in convex domains, Anal.

Math. 8 (1982), 257–261.

[5] —, Intrinsic distances and holomorphic retracts, in: Complex Analysis and Applica- tions ’81, Bulgar. Acad. Sci., Sophia, 1984, 341–364.

[6] H. L. R o y d e n and P. M. W o n g, Carath´ eodory and Kobayashi metrics on convex domains, preprint.

[7] J. P. V i g u ´ e, Un lemme de Schwarz pour les domaines born´ es sym´ etriques irr´ educ- tibles et certains domaines born´ es strictement convexes, Indiana Univ. Math. J. 40 (1991), 293–304.

[8] —, Le lemme de Schwarz et la caract´ erisation des automorphismes analytiques, Ast´ erisque 217 (1993), 241–249.

Faculty of Engineering Kyushu Kyoritsu University Jiyugaoka, Yahatanishi-ku Kitakyushu 807, Japan

E-mail: hamada@kyukyo-u.ac.jp

Re¸ cu par la R´ edaction le 29.4.1996

R´ evis´ e le 20.11.1996