A useful theorem for non-linear devices having Gaussian inputs

(1)

1958 _{IRE TRANSACTIONS ON INFORMATION THEORY} ₆₉

A Useful Theorem for Nonlinear Devices

Having Gaussian Inputs*

ROBERT PRICE t SummargIf and only if the inputs to a set of nonlinear,

zero-memory devices are variates drawn from a Gaussian random process, a useful general relationship may be found between certain input and output statistics of the set. This relationship equates partial derivatives of the (high-order) output correlation coefficient taken with respect to the input correlation coefficients,

to the output correlation coefficient of a new set of nonlinear devices

bearing a simple derivative relation to the original set. Application is made to the interesting special cases of conventional cross-correlation and autocross-correlation functions, and Bussgang's theorem is easily proved. As examples, the output autocorrelation functions are simply obtained for a hard limiter, linear detector, clipper, and smooth limiter.

N THE COURSE of in'estigating the asymptotic

frequency behavior of power spectra resulting from the passage of noise through zero-memory nonlinear

devices, an interesting, unique property of Gaussian

processes has been encountered, which does not appear to have been previously reported.

STATEMENT OF THE THEOREM

Assume z1, x2, , z,, to be random variables from a Gaussian process whose nth order joint probability density is given by:'

p(x,, z2, ' . , x,,) = (2ir)'"2

jj

i;

'' (z, -

)(x, -

(1)

where M,, is the determinant of M,, = [p,,] and p,, =

-

= p,, is the correlation coefficient of X and z,,. The means of z, and z,, are and , respectively.

1fr!,. is the cofactor of p,,, in M,,.

Let there be n zero-memory nonlinear devices (linearity of course being included as a special ('ase) specified by the input-output relationship 1(x), i = 1, 2, . , n. Let each r1 be the, single input to a corresponding f1(x), and

designate the nth-order correlation

coefficient

of the

outputs as:

I? ₌

_{fil f (x)}

(2)

where the bar denotes the average taken over all z,. Then, with weak restrictions on the f (x), we have the following theorem for the partial derivatives of R with respect to the input correlation coefficients:

Manuscript, received by the NUT, January 3, 1958. The research in this paper was supported jointly by the Army, Navy, and Air Force under contract with Mass. Inst. 'rech.

t Lincoln Lab., M.I.'l'., Lexington, Mass.

H. Cramér, "Mathematical Methods of Statistics," Princeton lJniversitv Press, Princeton, N. J., sec. 24.2; 1946.

JI

and the C1 and C1_ are appropriate contours. Without assuming any particular form for p(x,, z2, . ,z,,) for the

present,

R =

f

fJ f (x1)p(x,, X2, , z,,) (/X, (/X5 . .. dx,,. ₍₆₎

Substituting (4) in (6) and inverting the order of inte-gration, following Rice's characteristic function method,3

2_{D. V. Widder, "The Laplace Transform," Princeton University}

Press, Princeton, N. J., ch. 6; 1946.

S. O. Rice, "Mathematical analysis of random noise," Bell S'ys. Tech. J., vol. 23, pp. 282-332, July, 1944; and vol. 24, _pp. 46-156; January, 1945. See sec. 4.8.

N

kJ?

(i

₍

-, (xi) (3)

where r,,, and s,,,, m 1, 2,

' ,

N, are integers lying between i and n, inclusive, and are not necessarily distinct. The k,,, are positive integers, with k = k,,,. e,,,, is

the number of times i appears in (r, s,,).

ô,,,,,,

is the

Kronecker ô function, ô,,,,,,,'= i for r,,. = s,,,,O for r,,, s,,,.

The symbol f»(x,) denote,s the qth derivative of /(x),

taken at .r,.

Furthermore, not oniy is the above theorem true for inputs having an nth-order joint Gaussian distribution,

but it holds true only for such inputs if the

f1(x) are allowed to be of general form.

Proof

We now prove that iii order for (3) to be satisfied it is both sufficient and necessary that the x have the joint probability density given by (1). Assume that each f 1(x) can be represented by the sum of two Laplace transforms,2 f (x) = i h1+(u)e" du +

'r

h,_(u)e'""du (4) 2irj

_{Jc_}

where

h,(u)

=

f f1(x)e'' dx

(5) 1i1_(u) = f 1(x)e' dx

(2)

70 IRE TRANSACTIONS ON INFORMATION THEOJ?Y June R

-(2j) '

I

fI

h...(u)O(u,, n2, . ,u) du, du2 . du (7) '-I

where

'

denotes a summation over all possible ±

combinations and O(u,, u2,

characteristic function: O(u1, n2, ,n) ₌

f

-eXp

( Êuix) dx, dz2

... dx,, (S)

withj= V'L

We find a necessary condition for (3) to be satisfied by setting N = i = k = k1. The partial derivative of the left-hand side of (3) is taken on O in the integrand of (7), and the derivatives of the right-hand side are taken using (4). Thus the necessary condition:

/(){ôO(nl

' .

+

u2, , un)} du1 du2 du 0 (9)

is obtained. The term in braces must be zero in order to satisfy (9) for arbitrary f. (x) and hence hi... (u). Integrating

the resulting equation for all

(r,,

s,) (but taking into

account that Pr, P.r),

log O(u1, u2, . , u,)

1

= -

p,.uu. -4- g(v,, u2, . ,

u)

(10)

where g is some function which must now be found. Let p,, = i for all (r, s). Then all the x are completely correlated, and p(x', x2, , x) can be written:

p(x,, x2, , x,,) = p(x1) i[J

ò(x, - x, + x, -

) (11) where ô(x) is the Dirac ô function. Substituting (11) in

(8), 0 is of the form:

u2, - -- ,

u)

= exp

(

Ê

u,)gi(

È a,),

where

g1(u)

=

f

p(x1 -

7)e' d(x1 -

). (13)

Similarly, when PII = i r, p

= Pri

1 foi all

r 1, and p,., = I for all r ot- s 1, then x, .r,, , are completely correlated with (- z,) and we obtain:

O(u1, u2, - , v) = exp

(

Ê

ni)qi(ui

-

Ê

for Pii = 1, Pi

= Pri = -

I for all r i

and Pr, =

i

for all r, s 1. (14)

u,,)

is the nth-order

p(x,, x2, - . ,z,,)

r:

--"i

for all p,., = 1 (12)

Substituting (12) in (10), we find:

g(u,,n2, .-.

,u) ₌ j

Êu+ g2(ui)

(15) where g2(u) =

log g1(u) + u2/2. On the other hand,

substituting (14) in (10) yields

g(u1,n2, ... ii,,)

= i

Êu

+

g2(2u1

-

Eu.).

(16) Since u1 and u, may be considered as independent variables, the only solution which renders (15) and (16) compatible is g2(u) = K, a constant. Thus, finally, we have from (10) and (15) the necessary condition:

O(u,, u2, -. , u,,)

=exp-

i

Pr.Ur

+ j

n7 +

K].

(17)

This is recognized to be the characteristic function of the n-dimensional Gaussian distribution4 of (1) (K = O for proper normalization).

It is now a simple matter to prove the sufficiency of (17), and hence (1), for satisfying (3). Using (17) in (7), and remembering that p,., = p,,.,

N k,,5,

fefc

fc

N

Z.,,,k,,,

fi u,'

h(u)O(u1 ,u2,u,,) du, du2 . . . dun. (18)

By analogy to (6) and (7), and differentiating (4) with respect to x, the right side of (18) is seen to be equal to

2

-I _+,

.+

, (

fIf

' (x1)p(x,, x2, . , x,,) dx, dz2 -.. dx,, (19)

thus yielding (3).

A SPECIAL CASE AND ITS APPLICATIONS

Consider the familiar situation where n =

2, and let

p denote the crosscorrelation coefficient of z, and z2. Then (3) yields

ap =

f(Xj)f(X2).

(20)

Suppose that x, and z2 are values of a stationary Gaussian time series x(t) whose autocorrelation function is p(r).

z1 is taken at time t and z2 at time (t + r). R(r) will

denote the crosscorrelation function between the outputs Cramér, op. Cit., sec. 24.1.

i (_1)k ölcR

(i

(3)

1958 _{Price: A. Useful Theorem for Nonlinear Devices Having Gaussian Inputs}

71 of two zero-memory nonlinear devices whose inputs are

x(t) and x(t + r), respectively.

Taking t.he particular case where x(t) = O, x2(t) = 1,

and using (l),&

Eq. (21)

is particularly simple when the f(s)

are piecewise-polynomial functions and k is sufficiently high. Then the

_f

(s) consist entirely of functions of various orders and the integral can be easily evaluated.

It is often of interest to obtain the derivatives of

a crosscorrelation function with respect to

r. It is

con-venient to break down such r derivatives intoa series of products of derivatives of R(r) with respect to p(r), and p(r) with respect to r, using

(IR(r) aR(r) dp(r)

dr

ôp(r dr

This enables the nonlinear devices to be treated

in-dependently of the shape of the input correlation function p(r), using (21). Similarly, the derivatives of p(r) with

respect to r do not involve the f(x).

As an example, Cohen6 shows that in general, for

autocorrelation functions R(r), the limiting behavior of the corresponding power spectrum (c) is given by:

2 1 dR(r)

limw4,)

2[2(()

₊

I (lJ?(T)

and so on, where the derivatives are with respect to r. Another application of (20) is in deriving Bussgang's interesting result7 that the crosscorrelation function between the input and the output of a nonlinear device

driven by Gaussian noise has the same shape as the

input autocorrelation function. In this case fi(s) =

s

and f7(x) is arbitrary. Then f'(x) is unity, and all higher derivatives of fi(s) are zero. Putting this into (21) and evaluating the integral,

k

I r

f(x) exp (x2/2),1

₌ 3p( r) ir dr

-ld3R(r) âR(r.) (24)

v

Lo

k>1.

(22) (23)

N. Amiantov and V. I. Tikhonov, "The effect of normal fluctuations on typical nonlinear elements," Bull. Acad. Sci. USSR,

pp. 33-42;April, 1956. Here, ân autocorrelatioii caseff,(r) 12(x)] is

studied byexpanding the second-order joint Gaussian probability den-sity of x(t) and x(t + r) in powers of p(r), using Mehier's formula;see

Bateman Manuscript Project, "Higher Transcendental Functions,"

McGraw-Hill Book Co., Inc., New York, N. Y., vol. 2, p. 194, (22);

1953. They then integrate by parts to obtain the output correlation

function in a similar series but do not recognize the simple form (21) for this series. Using this method it is not required that fi(s) be Laplace transformable, rather than our proof.

R. Cohen, "Some Analytical and Practical Aspects of Wiener's

Theory of Prediction," Res. Lab, of Electronics, M.I.T., Cambridge,

Mass., Tech. Rep. No. 69, ch. 4, sec. 2; June 2, 1948.

J. J. Bussgang, "Crosscorrelation Functions of Amplitude-i)istortd Gaussian Signals," Ites. Lab, of Electronics, M.I.T., Camhridge, Mass., Tech. Rep. 216, sec. 3; March 26, 1952.

We find easily R(r) = p(r)

r

xf2(x)e J-

/;

-'/2

dx (25)

akp()

1(k) 'ç(k) f(k)[y(t)Jf(k)[x(t_-f- r)] =

f "

(s,j

(x2) exp { [x

+

x - 2p(r)x,x2]/2(i - p2(r)] dx, dx2. ₍₂₁₎

2ir\/1

-

p2(r) op(T)

thus yielding Bussgang's result. Unlike Bussgang's theorem, (20) cannot be generalized to hold for probability distributions other than Gaussian.8'°

SOME SIMPLE AUTOCORRELATION EXAMPLES [Fon x(t) = 0, x2(t) 1]

Hard Limiter

Van Vleck's well-known result ou the autocorrelation function of the output of a hard limiter" c' 'be derived very simply, using (21). If

f,(x) = f(x) = 1;

°

₍₂₆₎

i-1;

x<0

then f(x) and f»(x) are first-order

functions of area 2, at z = 0.

Substituting in (21) and integrating,

aR(r) 2

ap(r) -

ir V'i -

p2(r)

Whenp(r) = 0, R(r) = 0. Thus

R(r)

2 fP)

dp(r)

-

_2sin'

[p(r)J

ir o

v'lp(r)

ir

which is Van Vieck's result. Linear Detector

Similarly, the autocorrelation function of the output of a linear detector can be easily found. If

f

,(x = f2(X) x

s

0 (29)

0;

_x<O

then f2)_{(s) and fi> (s) are first-Order ö}

functions of area

unity at s

= _{0. Substituting in (21) and integrating:}

a2R(r) i

ap(r)2 - 2irV'l

- p2(r)

Doubly-integrating (30) with the boundary conditions: J. F. Barrett and D. G. Lampard, "An expansion_{for some} second-order probability distributions and its application to noise problems," JItE TaNs. ON INFORMATION THEORY, vol.

_ITI,

pp. 10-15; March, 1955.

J. L. Brown, Jr., "On a cross-correlation property for_stationary random processes," IRE TRANS. ON INFORMATION ThEoRY, _vol.

IT-3, pp. 28-31; March, 1957.

'° A. H. Nuttall, 'Invariance of Correlation _{Functions under} Nonlinear Transformations," Res. 1b. of Electronics, M.I.T.,

Canibridge, Mass., Quart. Progress Rep., p. 63; October 15, 1957. "J. L. Lawson and G. E. tJhlenbeck, "Threshold _Signals," McGraw-Hill Book Co., Inc., New York, N. Y., _{p. 58; 1950.}

(4)

72

r

12 1 R(T)

= Li

o

v'2r

--

(Ix] = -

2 we obtain: =

[p()} +

_-

(32)

which is in agreement with Rice's result." Clipper

The relations derived independently by Robin'3 and Laning and Battin'4 for the autocorrelation function of the output of a clipper may also be found by this method. With a clipper characteristic:

i<x

! < r

1 (33)

x<l

and f(x) and f'(x) each are a pair of first-order

functions at r = 1 and r = i, with areas i and 1,

respectively. Substituting in (21) and integrating.

r

12

i

r

_1'

O'R(r) exp

L1+ p(r)]

exp

L-i

- p(r)

₍₃₄₎

8p(r)2 -

3rV'l -

p'(r

which is Robin's result, for input noise of unit variance.

Smooth Limiter

Finally, Baum's recent interesting

result'5

for the

12 Rice, op. cit., eq. (4.7-5).

"L. Robin, "The autocorrelation function and power spectrum of clipped thermal noise. Filtering of simple periodic signals in this

noise," Ann. Teleconim., vol. 7, pp. 375-387; September, 1952. 14 J H. Laning Jr. and R. H. Batti,,, "Random Processes in Automatic Control," McGraw-Hill Book Co., Inc., New York, N. Y., p. 362, eq. (B-8); 1956.

0 R. F. Baum, "The correlation function of smoothly limited

Gaussian noise," IRE TRANS. ox INFORMATION ThEORY. VOI.

IT-3, pp. 193-197; September, 1957. IRE TRANSACTIONS ON R(r) = fP(r)

[ + j' 2Ï1 +

1 INFORMATION THEORY we have Substituting in (21): 2 2

R(r)l

Jp,p,

äp(r)

2lrJl

E

exp where

t-2

2 2

1 11 p(r)}-+-

fl[l - p(r)]

Pi

-2 2 2 2 l i p (r)] + 1

- p (r)

p(r)[1 p2(r)]

P2

(i [1

-2 2 2 2 p(r)J-f-

fl -

p(r)

The term in braces in (37) must equal unity, since it is the integral

of a second-order Gaussian probability

density. Thus, f rom (38),

¿9R(r)

I

- P2

ap(r) 2ir \/ i p2(r)

2(I + p2)2

-Integrating and using the condition that when p(r) = O, R(r) = O. 11(r) = _{2ir j,}I 1 -/(1' + 1)2 - p2(r) dp( r)

r

₍₄₀₎ 12

= sin_'

_{Li + i}

2ir

which is in agreement with Baum's result. f,(x) = f2(x) =

_ f

e"121' dt (1) f,

t.x) = f't(x)

-[

p,x + p,x -

2p2x,x,1 June (35) (36)

2(p - p)

_{dx, dx,} ₍₃₇₎

2,r/pj'

-(38) (39) ç ...-

-.

-f,(x) = 12(x) i;

r;

I:

-'/2

12 (I(X)e 01?(r)

[T

i

dx'

₄ for p(r) = O (31)

autocorrelation function of the output of a device having an error-function characteristic will be derived. With