cycle constraint ( q, c ) and average power constraint γ can be solved from the following optimization: In the special case of no transition cost, i.e., c = 0 , the result of (93) is equal to that of (5). The mutual information expressed by (85) is hard to optimize, even if the input is restricted to Markov processes. To simply the matter, we introduce a lower bound of I ( µ ) , which is given by L ( µ ) in (11). Property (a): Using the fact that processing reduce relative entropy and µ is specified as a stationary probability distribution, we have Therefore using the fact that the Ces´aro mean of sequence I ( X$_{1}$, X k $_{2}$) is I ( X$_{1}$ ; X ∞ 2 ) . Applying (85), (87) and (97), Thus Property (a) is established. Property (b): For any µ ∈ Λ( γ, q, c ) , which is not Markov in general, its first-order Markov approximation ν is defined by ν$_{X}$$_{1}$$_{,}$$_{···}$$_{,X}$$_{n}$ = µ$_{X}$$_{1}$µ$_{X}$$_{2}$$_{|}$$_{X}$$_{1}$µ$_{X}$$_{3}$$_{|}$$_{X}$$_{2}$ · · · µ$_{X}$$_{n}$$_{|}$$_{X}$$_{n}$$_{−}$$_{1}$. (99) L ( µ ) = I ( X ; Y ) − I ( X$_{1}$ ; X ∞ 2 ) ≤ I ( µ ) ≤ C ( γ, q, c ) . (98) lim n →∞ 1 n D ( P$_{X}$ n ‖ P$_{X}$$_{1}$ × P$_{X}$$_{2}$ ×· · ·× P$_{X}$$_{n}$ ) = I ( X$_{1}$ ; X ∞ 2 ) (97) 1 n D ( P$_{Y}$ n ‖ P$_{Y}$$_{1}$ × P$_{Y}$$_{2}$ ×· · ·× P$_{Y}$$_{n}$ ) ≤ 1 n D ( P$_{X}$ n ‖ P$_{X}$$_{1}$ × P$_{X}$$_{2}$ ×· · ·× P$_{X}$$_{n}$ ) (94) = 1 n$^{n}$$^{−}$$^{1}$ ∑ k =1 D ( P$_{X}$$_{k}$$_{|}$$_{X}$ n k $_{+1}$‖ P$_{X}$$_{k}$ | P$_{X}$ n k $_{+1}$) (95) = 1 n n ∑ k =2 I ( X$_{1}$ ; X k $_{2}$) . (96) maximize P$_{X}$ I ( X ; Y ) subject to P$_{X}$ (0) − 2 cP$_{X}$ (0)(1 − P$_{X}$ (0)) ≤ q, E { X $^{2}$} ≤ γ. (93) C. Proof of Theorem 2