Appl Math Optim (2011) 63: 217-237 DOI 10.1007/s00245-010-9117-6

An Excursion-Theoretic Approach to Stability of Discrete-Time Stochastic Hybrid Systems

Debasish Chatterjee • Soumik Pal

Published online: 25 September 2010 © Springer Science+Business Media, LLC 2010

Abstract We address stability of a class of Markovian discrete-time stochastic hybrid systems. This class of systems is characterized by the state-space of the system being partitioned into a safe or target set and its exterior, and the dynamics of the system being different in each domain. We give conditions for Li-boundedness of Lyapunov functions based on certain negative drift conditions outside the target set, together with some more minor assumptions. We then apply our results to a wide class of randomly switched systems (or iterated function systems), for which we give conditions for global asymptotic stability almost surely and in L1. The systems need not be time-homogeneous, and our results apply to certain systems for which functional-analytic or martingale-based estimates are difficult or impossible to get.

Keywords Stochastic stability ■ Excursion theory ■ Markov process

1 Introduction

Increasing complexity of engineering systems in the modern world has led to the hybrid systems paradigm in systems and control theory [22, 31]. A hybrid system consists of a number of domains in the state-space and a dynamical law corresponding to each domain; thus, at any instant of time the dynamics of the system depends on the

Debasish Chatterjee's research is partially supported by the Swiss National Science foundation grant 200021-122072._

D. Chatterjee (El)

ETL I19, ETH Zürich, Physikstrasse 3, 8092 Zürich, Switzerland e-mail: chatterjee@control.ee.ethz.ch

S. Pal

Department of Mathematics, University of Washington, C-547 Padelford Hall, Seattle, WA 98195, USA

e-mail: soumik@math.washington.edu

domain that its state is in. One would then restrict attention to behavior of the system in individual domains, which is typically a simpler problem. However, understanding how the dynamics in the individual domains interact among each other is necessary in order to ensure smooth operation of the overall system. This article is a step towards understanding the behavior of (possibly non-Markovian) stochastic hybrid systems which undergo excursions into different domains infinitely often. Here we consider the simplest and perhaps the most important hybrid system, consisting of a compact target or safe set and its exterior, with different dynamics inside and outside the safe set. Our objective is to introduce a new method of analysis of systems that are outside the safe set infinitely often in course of their evolution. The analysis carried out here provides a basis for controller synthesis of systems with control inputs—it gives clear indications about the type of controllers to be designed in order to ensure certain natural and basic stability properties in closed loop.

Let us look at two interesting and practically important examples of hybrid systems with two domains—a compact safe set and its exterior, with different dynamics in each. The first concerns optimal control of a Markov process with state constraints. Markov control processes have been extensively studied; we refer the reader to the excellent monographs and surveys [3,4, 14, 15] for further information, applications and references. For our purposes here, consider the canonical example of a linear controlled system perturbed by additive Gaussian noise and having probabilistic constraints on the states. A hybrid structure of the controlled system naturally presents itself in the following fashion. Except in the most trivial of cases, computing the constrained optimal control over an infinite horizon is impossible, and one resorts to a rolling-horizon controller. (Rolling-horizon controllers are considerably popular, for basic definitions, comparisons and references see e.g., [26] in the deterministic context, and [7] and the references therein in the stochastic context.) Computational overheads restrict the size of the window in the rolling-horizon controller, and determine the maximal (typically bounded) region—called the safe set—in which this controller can be active. No matter how good the resulting controller is, the additive nature of the Gaussian noise ensures that the states are subjected to excursions away from the safe set infinitely often almost surely. Once outside the safe set, the rolling-horizon controller is switched off and a recovery strategy is activated, whose task is to bring the states back to the safe set quickly and efficiently. This problem is of great practical interest and a subject of current research, see e.g., [7] and the references in them for possible strategies inside the safe set, and [6] for one possible recovery strategy. Evidently, stability of this hybrid system depends largely on the recovery strategy, since as long as the states stay inside the safe set, they are bounded. However, traditional methods of stability analysis do not work well precisely because of the unlimited number of excursions. Theorem 2.2 of this article addresses this issue, and provides a method of ensuring strong boundedness and stability properties of the hybrid system. Intuitively it says that under the recovery strategy there exists a well-behaved supermartingale until the states hit the safe set, then the system state is bounded in expectation uniformly over time. A complete picture of stability and ergodic properties of a general controlled hybrid system is beyond the scope of the present article, and will be reported elsewhere. We refer the reader to [9, Chap. 3] for earlier work pertaining to stability of a class of hybrid systems, and to [27] for stability of general discrete-time Markov processes.

The second example is one that we shall pursue further in this article, namely, a class of discrete-time Markov processes called iterated function systems [2, 23] (ifs). They are widely applied, for instance, in the construction of fractals [23], in studies on the process of generation of red blood corpuscles [24, 25], in statistical physics [21], and simulation of important stochastic processes [32]. Of late they are being employed in key problems of physical chemistry and computational biology, namely, the behavior of the chemical master equation [33, Chap. 6] (CME), which governs the continuous-time stochastic (Markovian) reaction-kinetics at very low concentrations (of the order of tens of molecules). Invariant distributions, certain finite-time properties, and robustness properties with respect to disturbances of the underlying Markov process are of interest in modeling and analysis of unicellular organisms. It is well-known that the CME is analytically intractable (see [1, 17] for special cases), but the invariant distribution of the Markov process can be recovered from simulation of the embedded Markov chain in a computationally efficient way. This embedded chain is an ifs taking values in a nonnegative integer lattice. From a biological perspective, good health of a cell corresponds to the ifs evolving in a safe region on an average, despite moderate disturbances to the numbers of molecules involved in the key reactions. However, in most cases compact invariant sets do not exist. It is therefore of interest to find conditions under which, even though there are excursions of the states away from a safe set infinitely often, the ifs is stochastically bounded, or some strong stability properties hold. Theorem 2.2 of this article leads to results (in Sect. 3) which address this issue.

This article unfolds as follows. Section 2 contains our main results—Theorems 2.2 and 2.4, which provide conditions under which a Lyapunov function of the states is L1 -bounded. We establish this Li-boundedness under the assumptions that a certain derived process is a supermartingale outside a compact set, and some more minor conditions.1 (The supermartingale condition alone is not enough, as pointed out in [28], where the authors establish variants of our results for scalar, possibly non-Markovian processes having increments with bounded p-th moments for p > 2.) For our results to hold, the underlying process need not be time-homogeneous or Markov-ian. To wit, in Sect. 2.2 we define a class of hybrid processes that switch between two Markov processes depending on whether they inside or outside a fixed set in the statespace, and demonstrate that although the resulting process may be non-Markovian, our results continue to hold. Connections to optimal stopping problems are drawn in Sect. 2.3, which gives a systematic procedure for verifying our assumptions. In Sect. 2.4 we apply the techniques our techniques to a class of sampled diffusion processes. In addition to the cases considered here, the results in Sect. 2 will be of interest in queueing theory, along the lines of the works [5, 13]. Section 3 contains some applications of the results in Sect. 2 to stability and robustness of ifs. The classical weak stability questions concerning the existence and uniqueness of invariant measures of ifs, addressed in e.g., [10, 18, 30], revolve around average contractivity hypotheses of the constituent maps and continuity of the probabilities. In Sect. 3.1 we

1It also seems conceivably possible that relaxed Foster-Lyapunov inequalities as in [11, Condition D(<p ,V,C), p. 1356] arising in the context of subgeometric convergence to a stationary distribution can be employed in the construction of the aforementioned supermartingale; this constitutes future work.

look at stronger stability properties of the ifs, namely, global asymptotic stability almost surely and in expectation, for which we give sufficient conditions. There are no assumptions of global contractivity or memoryless choice of the maps at each iterate; we just require a condition resembling average contractivity in terms of Lyapunov functions with a suitable coupling condition with the Markovian transition probabilities. We mention that although some of the assumptions in [18] resemble ours, the conditions needed to establish existence of invariant measures in [18] are stronger than what we employ; see Sect. 3.1 for a detailed comparison. We also demonstrate in sect. 3.2 that under mild assumptions, iterated function systems possess strong stability and robustness properties with respect to bounded disturbances. In this subsection the exogenous bounded disturbance is not modeled as a random process.

Notations Let N := {1,2,...}, No := {0,1, 2,...}, and R>0 := [0, We let ||-|| denote the standard Euclidean norm on Rd. We let Br denote the closed Euclidean ball around 0, i.e., Br := {y e Rd | ||y || < r}. For a vector v e Rd let vT denote its transpose, and ||v||P denote VvTPv for a d x d real matrix P. The maximum and minimum of two real numbers a and b is denoted by a v b and a A b, respectively.

2 General Results

Before we get into hybrid systems, it will be simpler to follow the arguments if we start by considering a discrete-time Markov chain.

2.1 Obtaining L1 Bound Using Excursions

Let X := (Xt)tsNo be a discrete time Markov chain with a state space S. We denote the transition kernel of this chain by P, i.e., for every x e S, the probability measure Px(-) := P(x, ■) determines the law of Xt+1, conditioned on Xt = x. At this point we only assume the state space S to be any Polish space.

Assumption 2.1 There exists a nonnegative function p : N0 x S ^ R>0 satisfying the following.

(i) There exists a subset K c S such that the process (Yt)teNo defined by Yt = p(t,Xt) is a supermartingale under Px0, for every x0 e S \ K until the first time Xt hits K. To wit, if X0 = x0 e S \ K and we define

tk = inf{t > 0|Xt e K},

then the process (YtAt )teN0 is a supermartingale under Px0.

(ii) There exists a nonnegative measurable real-valued function V : S ^ R and a positive sequence (0(t))teN0 such that

p(t,x) > V(x)/0(t) for all (t,x) e N0 x S,

and C :=£teN0 °(t) <

(iii) S := supxeK V(x) < œ.

Our objective is to prove under the above condition (and another minor assumptions) that there exists a bound on supt Ex0 [V(Xt)] depending on x0.

Theorem 2.2 Consider the setup in Assumption 2.1, and assume that

P := sup E[p(0,X1)1{x1e<s\k}X0 = x<)] < to. (2.1)

x0€K

Let y := sup;sN0 0(t). Then we have

sup Ex0 [V(X) < CP + 8 + YV(0,X0).

t €N0

In the rest of this section we prove the above theorem. Fix a time t € N0, and define two random times

gt := sup{s € N0|s < t, Xs € K} and ht := inf{s € N0|s > t, Xs € K}.

We follow the standard custom of defining supremum over empty sets to be —to, and the infimum over empty sets to be

Note that gt is not a stopping time with respect to the natural filtration generated by the process X, although ht is. The random interval [gt,ht] is a singleton if and only if Xt € K. Otherwise, we say that Xt is within an excursion outside K. Now we have the following decomposition:

E4V(Xt)] = Ex0 [V(Xt)1{gt=—«,}] + £ Ex^V(Xt)1{gt . (2.2)

Our first objective is to bound each of the expectations Ex0 [V(Xt)1{gt =s}].

Before we move on, let us first prove a lemma which follows readily from Assumption 2.1.

Lemma 2.3 Let X0 = x0 € S \ K. Then

Ex0 [V(Xs)1{^>s}] < V(0,x0)6(s) for s € N0, (2.3)

where (0(t))tsN„ is defined in Assumption 2.1.

Proof This is a straightforward application of Optional Sampling Theorem (OST) for discrete-time supermartingales. Applying OST for the bounded stopping time s A tk to the supermartingale (<p(t, Xt))tsN0, in view of p > 0, we have

V(0,x0) > Ex0[v(s A Tk^sAtJ] > Ex0 [P(S,XsU{rK>s}].

Now, by condition (i) in Assumption 2.1, we can write p(s,x) > V(x)/0(s). Thus, substituting back, one has

p(0,x0) > Ex0[V(Xs)1{tK>s}]/e(s). Since (0 (t))tsN0 is positive, we arrive at (2.3). □

We are ready for the proof of Theorem 2.2.

Proof of Theorem 2.2 Let us consider three separate cases:

Case 1. (—to <gt< t). In this case gt can take values {0, 1, 2,...,t — 1}. Now, if s e{0, 1, 2,...,t — 1}, then

Ex<,[ V(Xt)1{gt =s}] = Ex<,[ V(Xt)\{xs eK }1{X;/K,i=s+1,..,t}]

= / Ps(x0, dx)f P(x, dy) Ey [V(Xt—s—1)1{tK>t —s—1}],

JK js\K K

and by Lemma 2.3 it follows that the right-hand side is at most

f Ps(x0, dx) f P(x, dy)p(0,y)9(t — s — 1). K s\K

Thus, one has

Ex0 [V(Xt)1{gt =s}] < e(t — s — 1) f Ps(x0, dx)( P(x, dy) p(0,y)

< 0(t — s — 1) sup Ex[p(0,X1)1{X1Ss\K}]

= 0(t — s — 1)8. (2.4)

Case 2. (gt = t). This is easy, since Xt e K implies V(Xt) < 8. Thus

Ex0[V(Xt)1{gt =t}] < 8Px0(Xt e K) < 8.

Case 3. (gt = —to). This is the case when the chain started from outside K and has not yet hit K, and therefore,

Ex0[V(Xt)1{g,=—to}] = Ex0[V(Xt)1{tK>t}] < p(0,x0)9(t).

Combining all three cases above, we get the bound:

Ex0[V(Xt)] °(t — s — 1)P + 8 + p(0,x0)6(t). (2.5)

Maximizing the right-hand side of (2.5) over t, we arrive at

sup Ex0 [V(Xt)] < pJ2°(s) + 8 + p(0,x0) sup 0(t),

teN0 s=0 t €N0

which is the bound stated in the theorem. □

Often it will turn out that p(t, x) is a function f(t, V(x)) as in the case of the classical Foster-Lyapunov type supermartingales [27]. In that case p(t,x) = eatV(x),

for some positive a. Thus y(t, ■) is a linear function of V(x) for each fixed t, with 0(t) = e-at, which shows that the sequence (0(t))teNo is summable. See also [12] and the references therein for more general Foster-Lyapunov type conditions. For examples which are not linear see Sect. 2.4.

2.2 A Class of Hybrid Processes

The preceding analysis can be extended for processes which switch their behavior depending on whether the current value is within K or not. They constitute a particularly useful class of controlled processes in which a controller attempts to drive the system into a target or safe set K c S whenever the system gets out of K due to its inherent randomness. Below we give a rigorous construction of such a process.

A process X that is (Y, Z)-hybrid with respect to K Consider a pair of Markov chains (Y, Z) where Y is a time-homogeneous Markov chain, and Z is a (possibly) time inhomogeneous Markov chain. We construct a hybrid discrete-time stochastic process X by the following recipe:

Firstly, let the state space for the process be SN0 along with the natural filtration

F C C F2 C ...

generated by the coordinate maps.

Secondly, we define the sequence of stopping times o0 := t0 := —to and r1 < 01 < T2 < 02 <■■■ by

ti := inf{t > oi — 1 | Xt € K} and oi := infjt >Ti | Xt€K}

for i € N0.

Finally, we define the process X as follows: for a measurable B c S,

if Xt = x, 3 i : Ti < t <oi,

I Xt = Yt,

\P(Xt+1 e B F) = P(Y1 e B |Yo = x),

f Y — =1' - t JXt = zt,

if Xt = ^di: <t <T!+1, 1 P(Xt+1 € BFt) = P(Zt+1—Oi € B|Zt—Oi = x).

To wit, the process defined above behaves as the homogeneous chain Y whenever it is inside K. Once the process X exits the set K, a controller alters the behavior of the chain which, until it enters K again, behaves as a copy of the inhomogeneous chain Z starting from a point outside K. The process X is in general non-Markovian due to the possible time inhomogeneity of Z. Nevertheless, it is a natural class of examples of switching systems whose Markovian behavior switches in different regions on the state space. We say that X is (Y, Z)-hybrid with respect to K.

The following generalization of Theorem 2.2 can be proved along lines of the original proof. The only requirement is a slight modification of the condition (2.1) which is needed to alter the second inequality in (2.4).

Theorem 2.4 Consider a stochastic process X that is (Y, Z)-hybrid with respect to a measurable K c S for some homogeneous Markov chain Y and some possibly inhomogeneous Markov chain Z. Suppose Assumption 2.1 holds for the process Z and

3 := sup E[p(0,Yx)1{Y1€S\K}|Y0 = y0] < to. (2.6)

y0€K

If the process X starts from x0 e S \ K, we have

sup Ex0 [V(Xt)] < C/ + 8 + yp(0,x0). (2.7)

It is interesting to note that the right side of above bound is a total of individual contributions by the control (for C), the choice of K (for 8), and the initial configuration (for x0). We stress that the conclusion holds even when X is no longer a Markov chain due to the time inhomogeneity of Z. This is important, especially because operator-theoretic bounds like Foster-Lyapunov, or martingale-based bounds do not work in such a case.

2.3 Connection with Optimal Stopping Problems

Suppose that we are given a Markov chain Z taking values in S, a function V : S ^ R, and a measurable target or safe set K c S. (Alternatively, we may assume that we are given an S-valued process X that is (Y, Z)-hybrid with respect to a measurable K c S.) Our objective is to investigate whether the sequence (V(Xt))teN0 is L1 -bounded. To this end one can follow the two-step procedure of first searching for a function p satisfying Assumption 2.1, followed by an application of Theorem 2.4. A systematic procedure of doing this is given by the following connection with Optimal Stopping problems.

Let (0(t))tsN„ be some positive sequence of numbers such that Y,teN0 @(t) is finite. Define the pay-off or the reward function as

h( )- \ V(x)/0(t) if x e S \ K, t e N0, h(t,x) = ) 0 if x e K, t e N0.

Recall that the Optimal Stopping problem [29, Chap. 1] for the process Z and the reward function h defined above consists of finding a stopping time t* such that

Ex[h(t* A tk,zt*arK)] = ess^sup Ex[h(T A tk,ztaKL (2.8)

where tk is the hitting time to the set K, and ess sup refers to essential supremum over the set of all possible stopping times (see [29, Chap. 1, Lemma 1.3]). Define the value function as

p(n,x0) := ess sup E[h(x, V(Zr))IZn = x^, (2.9)

where Tn is the set of stopping times

{(t v n) A tkIt an arbitrary stopping time}.

Theorem 2.5 Suppose that the value function p(0,x0) is finite for all x0 e S, then

(i) p(t, x0) is finite for all t e N0 and

V(t,x0) > V(x0)/6(t) for all (t,x0) e N0 x (S \ K).

(ii) The process (Yt)tsN„ defined by

Yt := p^ A tK,ZtATk)

is a supermartingale.

Proof The proof follows from the general theory of optimal stopping. See, for example, [8, Chap. 4]. The sequence of rewards is given by the process V(ZtAT )/0(t),

t = 0,1,2,____Applying [8, Theorem 4.1, p. 66] we get

p(n,x0) = (V(x0)/0(n)) V (e[p(w + 1,Zin+1)ArK)\ZnATK = x<>]).

By considering the first of the two terms in the maximum on the right-hand side above we obtain (i), and (ii) follows from the second. □

In other words, the value function p(t, x) defined in (2.9) satisfies the conditions of Theorem 2.4.

Theorem 2.6 Consider an S-valued process X that is (Y, Z)-hybrid with respect to a measurable K c S as in Sect. 2.2. Suppose that for some nonnegative integrable sequence (0(t))teN0 the optimal stopping problem (2.8) has a finite value function p(t, x0). If additionally condition (2.6) is true, then the bound (2.7) holds.

Let us remark that the value function, being the envelope, is the smallest supermartingale (hence the sharpest bound) that can satisfy Theorem 2.4. Several methods of solving optimal stopping problems in the Markovian setting are available and we refer the reader to [29] for a complete review.

Remark 2.7 There is a parallel converse result employing standard Foster-Lyapunov techniques for the verification of f -ergodicity and f -regularity [27, Chap. 14] of Markov processes. The analysis is based on the functional inequality E[V(X\) \ X0 = x] - V(x) < -f(x) + blc(x) for measurable functions V : S ^[0, to] and f : S ^ [1, to[, a scalar b > 0, and a Borel subset C of S; [27, Theorem 14.2.3] asserts that the minimal solution to this inequality, which exists if c is petite (see [27] for precise details), is a "value function" given by Gc(x, f) := E[J^=0 f(Xt)\X0 = x], where ac is the first hitting-time to C. The proof is also based on the existence of a certain supermartingale, and the Markov property is employed crucially.

2.4 A Class of Sampled Diffusions

In the setting of the process X being (Y, Z)-hybrid with respect to a given set K, suppose that the state-space for the Markov chains Y and Z is and the safe set

K is compact. Observe that the only challenge in applying Theorem 2.4 is to find a suitable function p given the Markov chain Z and the function V. In applications, a natural choice for the function V is given by square of the Euclidean norm, i.e., V(x) = J2f=1 xf. For this choice of V, we describe below a natural class of examples of Markov chains for which one can construct a p that satisfies part (i) of Assumption 2.1.

Consider a diffusion with a possibly time-inhomogeneous drift function, given by the d -dimensional stochastic differential equation

dXt = b(t,Xt)dt + dWt, (2.10)

where Wt = (Wt(\), Wt(2), ...,Wt(d)) is a vector of d independent Brownian motions, and b : R>0 x Rd ^ Rd is a measurable function.

We will abuse the notations somewhat and construct a function p : R>0 x R>0 ^ R>0 such that (p(t, V(Xt)))teN0 is a supermartingale outside a compact set K and satisfies p(t, %) > %/0(t) for some nonnegative sequence (0(t))teN0. We define Zi = XiAt for i e N0; Z is the diffusion sampled at integer time points before hitting K. It is clear that Z is a Markov chain such that (p(i, V(Zi)))isN0 is a supermartingale that satisfies the Assumptions 2.1 as long

as teN0 ®(t) < to.

To construct such a p, let us consider a well known family of one-dimensional diffusion, known as the squared Bessel processes (BESQ). This family is indexed by a single nonnegative parameter 8 > 0 and is described as the unique strong solution of the SDE

dYt = 2jYt dbt + 8dt, Y0 = y0 > 0, (2.11)

where b := (bt)teN0 is a one-dimensional standard Brownian motion. We have the following lemma:

Lemma 2.8 Let F : R ^ R>0 be a nonnegative, increasing, and convex function, and fix any terminal time S > 0. Define the function

p(t,y) := E[F(Ys)Y = y], t e[0,S], (2.12)

where Y solves the SDE (2.11). Then p satisfies the following properties:

(i) p is increasing in y,

(ii) p is convex in y, and

(iii) p satisfies the partial differential equation

dp + 8p' + 2yp" = 0, y> 0, t e (0,S), (213)

p(S,y) = F(y).

Note that p' and p" in the statement of Lemma 2.8 refers to the first and second derivatives with respect to the second argument of p.

Proof The proof proceeds by coupling. Let us first show that p is increasing as claimed in (i). Fix S > 0. Consider any two starting points 0 < x < y. Construct on the same sample space two copies of BESQ processes Y(1) and Y(2) such that

both of them satisfy (2.11) with respect to the same Brownian motion b but Y0(1) = x

and Y0( ) = y .It is possible to do this since the SDE (2.11) admits a strong solution (see [20, Chap. 5, Proposition 2.13]). Hence, by [20, Chap. 5, Proposition 2.18], it follows that Yt(1) < Y(2) for all t > 0. Since F is an increasing function, we get

y(t, x) = Ex [F(Y™ )] < Ey [F)] = y(t, y).

This proves that y is increasing in the second argument.

For convexity of y claimed in (ii), we use a different coupling. We follow arguments very similar to the one used in the proof of [16, Theorem 3.1]. Consider three initial points 0 <z < y < x. And let X ,Y ,Z be three independent BESQ processes that start from x, y, and z respectively. Define the stopping times

Tx = inf{u\Yu = X u}, Tz = inf(u\ Yu = Z u}.

Fix a time t e [0,S], and let T = S -1. Define

a = Tx A Tz A T.

Now, on the event a = tx , it follows from symmetry that

E[(Xt - Zt)F(Yt)1{a=Tx}] = E[(Yt - Zt)F(Xt)1{a=Tx}], E[(Xt - YT)F(Zt)1{a=Tx}] = 0.

Similarly, on the event a = tz , we have

E[(Xt - Zt)F(Yt)1{a=Tz}] = E[(Xt - Yt)F(Zt)1{a=Tz}], E[(Zt - yT)F(xT)1{a=Tz}] = 0.

(2.14)

(2.15)

And finally, when a = T, we must have ZT < YT < XT. We use the convexity property of F to get

E[(Xt - zt)F(Yt)1{a=t}] < E[(Xt - yt)F(zt)1{a=t}]

+ E[(Yt - zt)F(xt)1{a=t}]. (2.16) Combining the three cases in (2.14), (2.15), and (2.16) we get

E[(Xt - zt)F(Yt)] < E[(Xt - yt)F(zt)] + E[(Yt - zt)F(xt)]. (2.17)

We now use the fact that X, Y, and Z are independent. Also, it is not difficult to see from the SDE (2.11) that Ex [Xt ] - x = Ey [Yt ]- y = Ez[Zt ] - z = St. Thus, from (2.17) we infer that

(x - z)y(t, y) < (x - y)y(t, z) + (y - z)y(t, x), for all 0 < z < y < x.

This proves convexity of y in its second argument.

Finally, to see (iii), it suffices to observe that (2.13) is the classical generator relation for diffusions, for which we refer to [20, Chap. 5.4]. The transition density of BESQ processes are smooth and have an explicit representation that satisfy (2.13). The general case can be obtained by differentiating under the integral with respect to F. □

Let us return to the multidimensional diffusion given by (2.10). We consider the process (Zt)t€n0, where Zt := p(t, \\Xt ||2), and p is the function in (2.12). Note that, since F is nonnegative, so is p. Additionally, since p is convex, we have

p(t,%) > p(t, 0) + p'(t, 0+)%.

Hence the sequence (6(t))st=0 is given by

0(t) = 1/p'(t, 0+), t = 0,1,...,S.

We have the following theorem:

Theorem 2.9 Suppose that there exists a compact set K c Rd such that the drift function b = (b1,b2,...,bd) in the SDE (2.10) satisfies the sector condition

Y^xibi{t,x)< 0 for (t,x) e K>0 x (S \ K). i = 1

Fix any terminal time T > 0. Define the process (Zt)t eNo := (p(t, \\Xt \\2 ))t eNo, where p is the nonnegative, increasing, convex function defined in (2.12) with

F(y) = \\y\\2 and S = d.

Then, with the set-up as above, the stopped process (Zt At AT)t>0 is a (local) supermartingale. K

Proof Applying Itô's rule to (Zt)tSM>0, we get

dZt = dMt +

— + Lp dt P

(2.18)

where M := (Mt)tsR>0 is in general a local martingale (M is a martingale under additional assumptions of boundedness on the first derivative of p), and L is the generator of X. We compute

+ Lp = B-PP + уьД + i V

+ dt + ^ dx + ? ^

dp 1 d я2

. ч dxi • 2 f-f dx2

I =1 I =1 I

dp I 1

=--+ 2p > biXi +—

2dp' + p"Y^ 4x2

« / \ d

= + dy' + 2ÎJ2 4) + 2y'J2 biXi = 2v'J2 biXi> ^ i ' i = 1 i

where the final equality holds since y satisfies (2.13) at y = J2i xf-

We know that y' > 0 since y is increasing, and, by our assumption, Y,i xibi < 0 whenever x / K. Thus,

^ + Ly < 0 for (t,x) e [0,T] x (S \ K). dt

Now the claim follows from the semimartingale decomposition given in (2.18). □

Note that the supermartingale (Zt )tsn0 has been defined only for a bounded temporal horizon. Thus, to show that Theorem 2.4 holds, some additional uniformity assumptions would be needed.

3 Application to Discrete-Time Randomly Switched Systems

In this section we look at several cases of discrete-time randomly switched systems (or, iterated function systems,) in which Theorem 2.2 of Sect. 2 applies and gives useful uniform L\ bounds of Lyapunov functions. In Sect. 3.1 we give sufficient conditions for global asymptotic stability almost surely and in L1 of discrete-time randomly switched systems. Assumptions of global contractivity in its standard form or memoryless choice of the maps at each iterate are absent; we simply require a condition resembling average contractivity in terms of Lyapunov functions with a suitable coupling condition with the Markovian transition probabilities. In Sect. 3.2 we demonstrate that under mild hypotheses iterated function systems possess strong stability and robustness properties with respect to bounded disturbances that are not modelled as random processes.2

3.1 Stability of Discrete-Time Randomly Switched Systems Consider the system

Xt+1 = fat(Xt), Xo = xo, t e No. (3.1)

Here a : No ^ P := {1,..., N} is a discrete-time random process, the map fi : Rd ^ Rd is continuous and locally Lipschitz, and there are points xf e Rd such that fi (xf) = 0 for each i e P. The initial condition of the system x0 e Rd is assumed to

2Recall the following notation: We let k denote the collection of strictly increasing continuous functions a : R>o ^ R>o such that a(0) = 0; we say that a function a belongs to class-KTO if a e k and limr^TO a(r) = to. A function f : R>o x No ^ R>o belongs to class-K£ if fS(-, n) e k for a fixed n e No, and if f(r,n) ^ 0 as n ^to for fixed r e R>o. Recall that a function f : Rd ^ Rd is locally Lipschitz continuous if for every xo e Rd and open set O containing xo, there exists a constant L > 0 such that Wf(x) — f(xo)W < L ||x — xoll whenever x e O.

be known. Our objective is to study stability properties of this system by extracting certain nonnegative supermartingales.

The system (3.1) can be viewed as an iterated function system: Xt+1 = fat o ••• o fa1 o fa0 (x0). Varying the point x0 but keeping the same maps leads to a family of Markov chains initialized from different initial conditions. The article [10] treats basic results on convergence and stationarity properties of such systems with the process (at)teN0 being a sequence of independent and identically distributed random variables taking values in P, and each map fi is a contraction. These results were generalized in [18] with the aid of Foster-Lyapunov arguments.

The analysis carried out in [18] requires a Polish state-space, and employs the following three principal assumptions: (a) the maps are non-separating on an average, i.e., the average separation of the Markov chains initialized at different points is nondecreasing over time; (b) there exists a set C such that the Markov chains started at different initial conditions contract after the set C is reached; and (c) there exists a measurable real-valued function V > 1, bounded on C, and satisfying a Foster-Lyapunov drift condition QV(x) < XV(x) + b1C(x) for some X e]0, 1[ and b < to, where Q is the transition kernel. Under these conditions the authors establish the existence and uniqueness of an invariant measure which is also globally attractive, and the convergence to this measure is exponential. In particular, this showed that the main results of [10], which are primarily related to existence and uniqueness of invariant probability measures, continue to hold if the contractivity hypotheses on the family {fi }iep are relaxed. In this subsection we look at stronger properties, namely, L1 boundedness and stability, and almost sure stability of the system (3.1) under Assumption 2.1. No contractivity inside a compact set is needed to establish existence of an invariant measure under Assumption 2.1.

Assumption 3.1 The process (at)t€n0 is an irreducible Markov chain with initial probability distribution no and a transition matrix P := [pij]NxN.

It is immediately clear that the discrete-time process (at,Xt)tsN„, taking values in the Borel space P x Rd, is Markovian under Assumption 3.1. The corresponding transition kernel is given by

Q((i,x), P'x B) =£ Pij 1 B(fj(x)) j ep'

for P' c P, B a Borel subset of Rd, and (i, x) e P x Rd.

Our basic analysis tool is a family of Lyapunov functions, one for each subsystem, and at different times we shall impose the following two distinct sets of hypotheses on them.3

3It will be useful to recall here that the deterministic system xt+1 = fi (xt), t e N0, with initial condition x0 is said to be globally asymptotically stable (in the sense of Lyapunov) if (a) for every e > 0 there exists a 8 > 0 such that \\x0 — x*\\ < 8 implies \\xt — x*\\ < e for all t e N0, and (b) for every r,E > 0 there exists a T > 0 such that \\x0 — x*\\ < r implies \\xt — x*\\ < e for all t > T. The condition (a) goes by the name of Lyapunov stability of the dynamical system (or of the corresponding equilibrium point x*), and (b) is the standard notion of global asymptotic convergence to x*.

Assumption 3.2 There exist a family {Vi}iep of nonnegative measurable functions on Rd, functions a\,a2 e K, numbers X0 e]0,1[, r > 0 and i> 1, such that

(V1) ai(\\x - x*\\) < Vi(x) < a2(\\x -x*\\) for all x and i, (V2) Vi(x) < i^Vj(x) whenever \\x \\ > r, for all i, j, and (V3) Vi(fi(x)) < koVi(x) for all x and i.

Assumption 3.3 There exist a family {Vi}iep of nonnegative measurable functions on Rd, functions a1,a2 e K, a matrix [Xij]NxN with nonnegative entries, and numbers r > 0, ix > 1, such that (V1)-(V2) of Assumption 3.2 hold, and

(V3') Vi (fj (x)) < Xij Vi (x) for all x and i, j.

The condition (V1) in Assumption 3.2 is standard in deterministic system theory literature, ensuring, in particular, positive definiteness of each Vi. Condition (V2) stipulates that outside Br the functions {Vi}igp are linearly comparable to each other. The conditions (V1) and (V3) together imply that each subsystem is globally asymptotically stable, with sufficient stability margin—the smaller the number X0, the greater is the stability margin. In fact, standard converse Lyapunov theorems show that (V1) and (V3) are necessary and sufficient conditions for each subsystem to be globally asymptotically stable. The only difference between Assumptions 3.2 and 3.3 is that the latter keeps track of how each Lyapunov function evolves along trajectories of every subsystem.

Let us define p := maxiSN pa and p := max¡jsppij.

Proposition 3.4 Consider the system (3.1), and suppose that either of the following two conditions holds:

(51) Assumptions 3.1 and 3.2 hold, and X0(p + ixp) < 1.

(52) Assumptions 3.1 and 3.3 hold, and i ■ (maxjp-p y)jg-p pijXji) < 1.

Let Tr := inf(i g No| \\Xt || < r} and V!(x) := Vi(x)1Rd\br(x). Suppose that ||xo|| > r. Then there exists a > 0 such that the process (ea(tATr)Vát (Xtatr))tgNo is a nonnegative supermartingale.

Corollary 3.5 Consider the system (3.1), and assume that the hypotheses of Proposition 3.4 hold. Then there exists a constant c > 0 such that supteN0 E[ai(||Xt ||)] < c.

It is possible to derive simple conditions for stability of the system (3.1) from Proposition 3.4. To this end we briefly recall two standard stability concepts.

Definition 3.6 If kerf — id) = {0} for each i e P, the system (3.1) is said to be

o globally asymptotically stable almost surely if (AS1) P(Vs > 0 3S > 0 s.t. supteNo ||XtW < e whenever ||x0|| < S) = 1, (AS2) P(Vr,e' > 0 3 T > 0 s.t. supNo3t>r ||Xt || < e' whenever ||xo|| < r) = 1; o a-stable in Li for some a e K if

(SM1) Ve > 0 3 S > 0 s.t. supteNo E[a(|Xt ||)] < e whenever ||xo|| < S,

(SM2) Vr,e' > 0 3 T > 0 s.t. supN()3t>T E[a(||Xt ||)] < e' whenever ||x0|| < r.

Corollary 3.7 Suppose that kerf - id) = {0} for each i e P, and that either of the hypotheses (S1) and (S2) of Proposition 3.4 holds with r = 0. Then

o there exists a > 0 such that limt^OT E[eat Vat (Xt)] = 0, and o the system (3.1) is globally asymptotically stable almost surely and a1 -stable in L1 in the sense of Definition 3.6.

The proofs of Proposition 3.4, Corollary 3.5 and Corollary 3.7 are given after the following simple lemma; the crude estimate asserted in it resembles the distribution of a Binomial random variable, except that we have p + p > 1. For t e N let the random variable Nt denote the number of times the state of the Markov chain changes on the period of length t starting from 0, i.e., Nt := l{ai-1=a}.

Lemma 3.8 Under Assumption 3.1 we have for s <t, s,t e N0,

P(Nt - Ns = k\as) < { (C ~k)P(-S-k)Pk) A 1 fk = 0 1"--'t - s, 0 else.

Proof Fix s <t, s,t e N0, and let nk(s, t) := P(Nt - Ns = k\as). Then by the Markov property, for k = 0, 1,...,t - s,

nk(s, t) = m(s, t - 1)P(Nt - Ns = k\Nt-i - Ns = k, as)

+ nk-1(s, t - 1)P(Nt - Ns = k\Nt-1 - Ns = k - 1,as) < pnk(s, t - 1) + pnk-1(s, t - 1).

The set of initial conditions m(s,t) = 0 for all i > t - s, follow from the trivial observation that there cannot be more than t - s changes of a on a period of length t - s. This gives a well-defined set of recursive equations, and a standard induction argument shows that nk(s, t) < (t-s)p(t-s-k)pk. This proves the assertion. □

Proof of Proposition 3.4 First we look at the assertion under the condition (S1). Fix s <t, s,t e N0. Given (asAtr,XsAtr), from (V3) we get V^ (X(s+1)Arr) < XoV^Axr(XsMr), and if as+1 = as, we employ (V2) to get V^s+1)Arr(X(s+1)at.) < ^VaSAXr (X(s+1)aTr). Therefore,

V®(s+1)Arr (X(s+1)ar^ < Mo KSAtr (XsATr) if a(s+1)aTr = asATr , and Va(s+1)ATr (X(s+1)aTr ) < V^ (XsaTr ) otherwise.

Iterating this procedure we arrive at the pathwise inequality

V;Arr (XtaTr) < MNtATr-N-<aTr-saTr V^ (XsaTr). (3.2)

Since s a Tr = t a s a Tr, and t a Tr is measurable with respect to Ft as at, , we invoke the Markov property of (at,Xt)tsNoto arrive at

E[v; AJXt aTr)|(asaTr,XsaTr)]

< V^ (XsATrKATr-sATr E[NTr -NsATr |(CTsaTr,XsaTr)].

We now apply the estimate in Lemma 3.8 to get E[HNtATr-nsatr |(ersATr,XsATr)] < Ek=o-saTr CaTr-saTr)P(tATr-sATr-k)PkHk = (P + HP)tATr-sATr, and this leads to

EVt at, (XtATr ) I sATr ,XsATr )] < V^ (XsATr)(Xo(p + HP J)^ ^

Since Xa(p + hp) < 1, letting a' := Xa(p + hp)ea < 1, the above inequality gives E[ea(tATr-sATr)V'tat (XtaT,)Is AT,, XsAT,)]

a t ATr

< aTr rfaTr -SaTr < v^ (x-aTr). (3.3)

This shows that (ea(tATr)V^^tAr (xtAtr))teNo is a nonnegative supermartingale.

Let us now look at the assertion of the proposition under the condition (S2). Fix t e No. Then from (V3' ), Vj (f atMr (Xt ATr)) < j Mr Vj (Xt ATr) for all j e P, and by (V2),

Vff(t+1)Arr (fatATr (XtATr)) < ka(t+1)ATratAZr vff(t+1)Arr (XtATr)

< H^O(t+1) ATr at ATr Vâ,ATr (Xt ATr).

This leads to

E[iXit+DAxMotATr.XtATr)] < H[ m® J2 PiJkJi ) Vat AXr iXt ATr)'

V jep 7

Since by hypothesis there exists a > 0 such that H(maxiep ^ jep pijXji)ea < 1, the last inequality shows immediately that (ea(tATr)V£tAt (XtATr))tsN0 is a supermartingale. This concludes the proof. □

Proof of Corollary 3.5 First observe that since each map fi is locally Lipschitz, the diameter of the set Di := {fi(x) | x e Br} is finite, and since P is finite, so is the diameter of (Jiep Di. Therefore, if Q is the transition kernel of the Markov process (at,Xt)teNo, then employing (V1) and the fact that fi is locally Lipschitz for each i, we arrive at

E[Vai (X1)1{X1eRd\Br}|(or0,X0) = (i,x0)]

= J2 Pij 1Rd\Br (fj (x0))Vj (/j (x0)) j eP

< Y1 Pij 1®d\jjr(/j(x0))a2(y/j(x0)y) < PijL IN II < Lr < œ jep jep

for ||*o II < r, where L is such that sup;g-p r 11^^C^y)M — L ||y||. This shows that condition (2.1) of Theorem 2.2 holds under our hypotheses, and by Proposition 3.4 we know that there exists a> 0 such that CeaCtAtr)Va,Arr (Xtarr)1R\BrCxtATr))teN0 is a supermartingale. Theorem 2.2 now guarantees the existence of a constant C' > 0 such that supteN0 E[VCt(Xt)1Rd\Br(Xt)] — Ct, and finally, from (V1) it follows that there exists a constant c > 0 such that supteN0 E[ai(||Xt ||)] — c < œ, as asserted. □

Proof of Corollary 3.7 We prove almost sure global asymptotic stability and a1 stability in L1 of (3.1) under the condition (S1) of Proposition 3.4; the proofs under (S2) are similar.

First observe that since kerf - id) = {0} for each i e P, i.e., 0 is the equilibrium point of each individual subsystem, Px0 (t{0} < œ) = 0 for x0 = 0, where T{0} is the first time that the process (Xt)teN0 hits {0}. Indeed, since kerf - id) = {0} for each i e P and x0 = 0 we have Q((i,x0), P x {0}) =

ep Pij 1{0}(fj(x0)) = 0, which shows that Qn((i,x0), P x {0}) = 0 whenever x0 = 0. The observation now follows from Px0 (T{0} < œ) — Px0 (Un€N{T{0} — n}) — SneN PX0 (T{0} = n). Therefore, with T{0} = Tr = œ, proceeding as in the proof of Proposition 3.4 above, one can show that (eatVat(Xt))teN0 is a supermartingale for some a > 0. In particular, With s = 0 and Tr = œ in (3.3), we apply (V1) to arrive at lim,^œ E[ea%(Xt)] = lim,^œ E[E[eatVfft(Xt)|(a0,X0)]] — limta2(||x0||)(a/)t = 0. Standard supermartingale convergence results and the definition of T{0} imply that P(limtVat(Xt) = 0) = 1. With s = 0 and Tr = T{0} = œ, the pathwise inequality (3.2) in conjunction with (V1) give Vat(Xt) — a2(Mx0|M)|¿NtXto. The foregoing inequality implies that for almost every sample path (crt,X't)teN0 corresponding to initial condition X0 = x0 with ||x01 < ||x0||, one has

lim Vat(X't) — lim a2(||x0||)Mnat — lim a2(||xo|M)|гNtlt0 = 0,

which proves (AS2). Since the family f }iep is finite, and each f is locally Lip-schitz, there exists L > 0 such that supiep ||f (x)|| — L ||x || whenever ||x| — 1. Fix e > 0. By (AS2) we know that for almost all sample paths there exists a constant T > 0 such that supt>T ||Xt|| < e whenever ||x0|| < 1. Then the choice of S = (eL-T) A 1 immediately gives us the (AS1) property.

It remains to verify (SM1) and (SM2). Both the properties follow from (3.3) in the proof of Proposition 3.4, with s = 0 and Tr = T{0} = 0. Indeed, with these values of s and Tr, (3.3) becomes

E[eata1(||Xt ||)|(c0,X0)] — E[ea%(Xt)|(a0 ,X0)]

— Vc0 (X0)(a')t — a2(||x0||)(a' )t

in view of (V1), where a' = X0 (p + ip)ea < 1. Therefore, given e > 0, we simply choose S < a-1(e) to get (SM1). Given r,e' > 0, we simply choose T = 0 v (ln(a2(r)/e')/ln(a')) to get (SM2). This completes the proof. □

3.2 Robust Stability of Discrete-Time Randomly Switched Systems

Conditions for the existence of the supermartingale (eaCtArк)V(XtAt ))teN0 in Sect. 2 can be easily expressed in terms of the transition kernel Q. However, if Q is not known exactly, which may happen if the model of the underlying system generating the Markov process (Xt)teN0 is uncertain, one needs different methods. We look at one such instance below. Consider the system

Xt+1 = fa,(Xt,wt), Xo = xo, t eNo, (3.4)

where we retain the definition a from Sect. 3.1, f : Rd x Rm ^ Rd is locally Lip-schitz continuous in both arguments with fi(o, 0) = 0 for each i e P, and (wt)teNo is a bounded and measurable Rm-valued disturbance sequence. We do not model (wt)teNo as a random process; as such, the transition kernel of (3.4) is not unique.

Definition 3.9 The system (3.4) is said to be input-to-state stable in L1 if there exist functions x,x' e and f e KL such that Exo [x(||Xt ||)] < f(||xo|| ,t) + supiSNo X'(Hw* ||) for all t e No.

Our motivation for this definition comes from the concept of input-to-state stability iss in the deterministic context [19]. Consider the i-th subsystem of (3.4) xt+1 = fi(xt,wt) for t e No with initial condition xo; note that (xt)teNo is a deterministic sequence. This nonlinear discrete-time system is said to be iss if there exist functions f e KL and x e such that ||xt|| < f(||xo|| ,t) + supisNox(Hws||) for t e No. A sufficient set of conditions (cf. [19, Lemma 3.5]) for iss of this system is that there exist a continuous function V : Rd ^ R>o, a1,a2 e p e K, and a constant X e]o, 1[, such that a1(|x||) < V(x) < a2(|x||) for all x e Rd, and V(fi(x, w)) < XV(x) whenever ||x|| > p(|w|).

in this framework we have the following proposition.

Proposition 3.10 Consider the system (3.4), and suppose that

(i) Assumption 3.1 holds,

(ii) there exist continuous functions Vi : Rd ^ R>o for i e P, a1,a2,p e , a constant ix> 1 and a matrix [Xij]NxN of nonnegative entries, such that

(a) a1(|x|) < Vi(x) < a2(|x ||) for all x and i,

(b) Vi (x) < iVj (x) for all x and i,j, and

(c) Vi(fj(x)) < XijVi(x) whenever ||x|| > p(| w|) and all i, j,

(iii) l(max.ierJ2jeP PijXji) < !.

Then (3.4) is input-to-state stable in L1 in the sense of Definition 3.9.

Proof We define the compact set K := {(i, y) e P x Rd| ||y|| < supseNo p(|ws ||)}, and let tk := inf{t e No|Xt e K}. In this setting we know from the preceding analysis that v(t,\) = eat%, 0(t) = e-at, and C = 1/(1 -e-a). We see from the estimate (2.5)

in the proof of Theorem 2.2 that

Exo[Vo,(Xt)] < y(0,Vao(xo))0(t) + ^^ + 5 < a2(yxo^)e-at + ^^ +

Standard arguments show that there exists some x " e such that j and 5 are each dominated by x"(supseNo ||ws ||), and therefore, there exists some x' e such that j/(1 - e-a) + 5 is dominated by x'(supseNo || ws ||). Applying (ii) (a) on the left-hand side of the last inequality, we conclude that (3.4) is input-to-state stable with x = ai

and ^(r,t) = a2(r)e-at. □

Acknowledgements The authors thank Daniel Liberzon and John Lygeros for helpful comments, Andreas Milias-Argeitis for useful discussions related to the chemical master equation, and the anonymous reviewer for a thorough review of the manuscript, several helpful comments, and drawing their attention to [27, Chap. 14].

References

1. Anderson, D.F., Craciun, G., Kurtz, T.G.: Product-form stationary distributions for deficiency zero chemical reaction networks. arXiv:o8o3.3o42 (2oo8)

2. Barnsley, M.F., Demko, S.G., Elton, J.H., Geronimo, J.S.: Invariant measures for Markov processes arising from iterated function systems with place-dependent probabilities. Ann. Ins. Henri Poincare Probab. Stat. 24(3), 367-394 (1988). Erratum in ibid. 24(4), 589-59o (1989)

3. Bertsekas, D.P., Shreve, S.E.: Stochastic Optimal Control: The Discrete-Time Case. Mathematics in Science and Engineering, vol. 139. Academic Press, San Diego (1978)

4. Borkar, V.S.: Topics in Controlled Markov Chains. Pitman Research Notes in Mathematics Series, vol. 24o. Longman, Harlow (1991)

5. Borodin, A., Kleinberg, J., Raghavan, P., Sudan, M., Williamson, D.P.: Adversarial queuing theory. J. ACM 48(1), 13-38 (2oo1)

6. Chatterjee, D., Cinquemani, E., Chaloulos, G., Lygeros, J.: Stochastic control up to a hitting time: optimality and rolling-horizon implementation. arXiv:o8o6.3oo8 (2oo8)

7. Chatterjee, D., Hokayem, P., Lygeros, J.: Stochastic receding horizon control with bounded control inputs: a vector space approach. arXiv:o9o3.5444 (2oo9)

8. Chow, Y.S., Robbins, H., Siegmund, D.: Great Expectations: The Theory of Optimal Stopping. Houghton Mifflin, Boston (1971)

9. Costa, O.L.V., Fragoso, M.D., Marques, R.P.: Discrete-Time Markov Jump Linear Systems. Probability and Its Applications. Springer, Berlin (2oo5)

10. Diaconis, P., Freedman, D.: Iterated random functions. SIAM Rev. 41(1), 45-76 (1999) (electronic)

11. Douc, R., Fort, G., Moulines, E., Soulier, P.: Practical drift conditions for subgeometric rates of convergence. Ann. Appl. Probab. 14(3), 1353-1377 (2oo4)

12. Foss, S., Konstantopoulos, T.: An overview of some stochastic stability methods. J. Oper. Res. Soc. Jpn. 47(4), 275-3o3 (2oo4)

13. Hastad, J., Leighton, T., Rogoff, B.: Analysis of backoff protocols for multiple access channels. SIAM J. Comput. 25(4), 74o-774 (1996)

14. Hernandez-Lerma, O., Lasserre, J.B.: Discrete-Time Markov Control Processes: Basic Optimality Criteria. Applications of Mathematics, vol. 3o. Springer, New York (1996)

15. Hernandez-Lerma, O., Lasserre, J.B.: Further Topics on Discrete-Time Markov Control Processes. Applications of Mathematics, vol. 42. Springer, New York (1999)

16. Hobson, D.G.: Volatility misspecification, option pricing and superreplication via coupling. Ann. Appl. Probab. 8(1), 193-2o5 (1998)

17. Jahnke, T., Huisinga, W.: Solving the chemical master equation for monomolecular reaction systems analytically. J. Math. Biol. 54(1), 1-26 (2oo7)

18. Jarner, S.F., Tweedie, R.L.: Locally contracting iterated functions and stability of Markov chains. J. Appl. Probab. 38(2), 494-5o7 (2oo1)

19. Jiang, Z.-P., Wang, Y.: Input-to-state stability for discrete-time nonlinear systems. Automatica 37(6), 857-869 (2001)

20. Karatzas, I., Shreve, S.: Brownian Motion and Stochastic Calculus, 2nd edn. Graduate Texts in Mathematics. Springer, Berlin (1991)

21. Kifer, Y.: Ergodic Theory of Random Transformations. Progress in Probability and Statistics, vol. 10. Birkhäuser, Boston (1986)

22. Liberzon, D.: Switching in Systems and Control, Systems & Control: Foundations & Applications. Birkhäuser, Boston (2003)

23. Lasota, A., Mackey, M.C.: Chaos, Fractals, and Noise, 2nd edn. Applied Mathematical Sciences, vol. 97. Springer, New York (1994)

24. Lasota, A., Myjak, J.: On a dimension of measures. Pol. Acad. Sci. Bull. Math. 50(2), 221-235 (2002)

25. Lasota, A., Szarek, T.: Dimension of measures invariant with respect to the Wazewska partial differential equation. J. Differ. Equ. 196(2), 448-465 (2004)

26. Maciejowski, J.M.: Predictive Control with Constraints. Prentice Hall, New York (2001)

27. Meyn, S.P., Tweedie, R.L.: Markov Chains and Stochastic Stability, 2nd edn. Cambridge University Press, Cambridge (2009)

28. Pemantle, R., Rosenthal, J.S.: Moment conditions for a sequence with negative drift to be uniformly bounded in Lr. Stoch. Process. Appl. 82(1), 143-155 (1999)

29. Peskir, G., Shiryaev, A.N.: Optimal Stopping and Free-Boundary Problems. Lectures in Mathematics ETH Zürich. Birkhäuser, Basel (2006)

30. Szarek, T.: Feller processes on nonlocally compact spaces. Ann. Probab. 34(5), 1849-1863 (2006)

31. van der Schaft, A., Schumacher, H.: An Introduction to Hybrid Dynamical Systems. Lecture Notes in Control and Information Sciences, vol. 251. Springer, Berlin (2000)

32. Werner, I.: Contractive Markov systems. J. Lond. Math. Soc. 71(1), 236-258 (2005)

33. Wilkinson, D.J.: Stochastic Modelling for Systems Biology. Chapman & Hall/CRC Mathematical and Computational Biology Series. Chapman & Hall/CRC, London/Boca Raton (2006)