0 EURASIP Journal on Wireless Communications and Networking

a SpringerOpen Journal

RESEARCH Open Access

MMSE-based joint source and relay optimization for interference MIMO relay systems

Khoa Xuan Nguyen*, Yue Rong and Sven Nordholm

Abstract

In this paper, we investigate the transceiver design for amplify-and-forward (AF) interference multiple-input multiple-output (MIMO) relay communication systems when the direct links between the source and destination nodes are taken into consideration. The minimum mean-squared error (MMSE) of the signal waveform estimation at the destination nodes is chosen as the design criterion to optimize the source, relay, and receiver matrices for interference suppression. As the joint source, relay, and receiver optimization problem is nonconvex with matrix variables, a globally optimal solution is computationally intractable to obtain. We propose two iterative algorithms to provide computationally efficient solutions to the original problem through solving convex subproblems. These two algorithms provide efficient performance-complexity trade-off.Simulation results demonstrate that the proposed algorithms converge quickly after a few iterations and significantly outperform existing scheme in terms of the system bit error rate.

Keywords: Interference channel; MIMO relay; Direct link; MSE

1 Introduction

Relay-aided multiple-input multiple-output (MIMO) communication technology has attracted great research interest recently [1,2]. By incorporating relay nodes in a MIMO system, the network coverage and reliability can be significantly improved. In a MIMO relay system, communication between source nodes and destination nodes can be assisted by single or multiple relays equipped with multiple antennas. The relays can either decode-and-forward (DF) or amplify-and-forward (AF) the relayed signals [3]. In the AF scheme, the received signals are simply amplified (including a possible linear transformation) through the relay precoding matrices before being forwarded to the destination nodes. Therefore, in general, the AF strategy has lower complexity and shorter processing delay than the DF strategy.

For single-user two-hop MIMO communication systems with a single relay node, the optimal source and relay precoding matrices have been developed in [4]. For

Correspondence: khoa.x.nguyen@ieee.org

Department of Electrical and Computer Engineering, Curtin University of Technology, Kent Street, Bentley, WA 6102, Australia

a single-user two-hop MIMO relay system with multiple parallel relay nodes, the design of relay precoding matrices has been studied in [5]. Recent progress on the optimization of AF MIMO relay systems has been summarized in the tutorial of [2].

For MIMO interference channel, the idea of interference alignment (IA) [6] was developed for interference suppression by arranging the desired signal and interference into appropriated signal spaces. The idea of IA has been applied in interference MIMO relay systems in [7,8]. However, there is still no general solution for IA as a number of conditions must be met. One main reason is that the number of dimensions required for IA is very large and it depends on the number of independent fading channels. This leads to high computational complexity and infeasi-bility in practical systems. In [9], an iterative algorithm has been proposed to optimize the source beamforming vector and the relay precoding matrices to minimize the total source and relay transmit power such that a minimum signal-to-interference-plus-noise ratio (SINR) threshold is maintained at each receiver. Three iterative transceiver design algorithms to minimize either the matrix-weighted sum mean-squared error (SMSE) or the total leakage have

Springer

© 2015 Nguyen et al.; licensee Springer. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.

been developed in [10]. However, the works in [7,10] did not take the direct source-destination links into consideration.

The direct links between the source and destination nodes provide valuable spatial diversity to the MIMO relay system and should not be ignored. In this paper, we investigate the transceiver design for AF interference MIMO relay communication systems where multiple source nodes transmit information simultaneously to the destination nodes with the aid of multiple relay nodes, and each node is equipped with multiple antennas. The direct source-destination links are taken into account for the design of the transceivers. We aim at optimizing the source, relay, and receiver matrices to suppress the interference and minimize the SMSE of the signal waveform estimation at the destination nodes, subjecting to transmission power constraints at the source and relay nodes. The SMSE criterion is chosen as it provides a good tradeoff between performance and complexity. Since the joint source, relay, and receiver optimization problem is non-convex with matrix variables, a globally optimal solution is computationally intractable to obtain. We propose two iterative algorithms to provide computationally efficient solutions to the original problem through solving convex subproblems. In each iteration of the first algorithm, we first optimize all receiver matrices based on the source and relay matrices from the previous iteration. Then, we optimize all relay matrices using the receiver matrices in this iteration and the source matrices from the previous iteration. Finally, the source matrices are updated.

In the second algorithm, the receiver matrices are optimized in the same way as the first algorithm. However, in contrast to the first algorithm, each source and relay matrix is optimized individually by fixing all other matrices. We show that both proposed algorithms converge. Comparing the two proposed algorithms, the first algorithm has a better MSE and bit error rate (BER) performance, while the second algorithm has a smaller periteration computational complexity. Such performance-complexity trade-off is very useful for practical MIMO relay communication systems. Simulation results demonstrate that the proposed algorithms outperform the existing technique in terms of the system MSE and BER.

We assume that similar to [10], the two proposed algorithms are carried out at a central controlling unit, which can be any node in the system. The controlling unit has knowledge of the global channel state information (CSI). After the convergence of the algorithms, the controlling unit sends the information on the optimal source, relay, and receiver matrices to corresponding nodes.

The rest of this paper is organized as follows. The system model and problem formulation are introduced in Section 2. Two iterative joint source, relay, and receiver matrices design algorithms are developed in Section 3.

Simulation results are presented in Section 4 to demonstrate the performance of the proposed algorithms. Conclusions are drawn in Section 5.

Throughout this paper, scalars are denoted with lower-or uppercase normal letters, and vectors and matrices are denoted with bold-faced lower- and uppercase letters, respectively. Superscripts (-)T, (-)H, and (-)-1 denote the matrix transpose, conjugate transpose, and inverse, respectively; tr() stands for the trace of a matrix; vec() stacks columns of a matrix on top of each other into a single vector; bd() denotes a block-diagonal matrix; <g> represents the Kronecker product, E[ ] denotes the statistical expectation; and In stands for the n x n identity matrix.

2 System model and problem formulation

We consider a two-hop interference MIMO relay communication system where K source-destination pairs communicate simultaneously with the aid of a network of L-distributed relay nodes as shown in Figure 1. The //th source node and the //th destination node are equipped with Ns/ and Ndk antennas, respectively, k = 1, ••• , K, and the number of antennas at the /th relay node is Nri, l = 1, ••• , L.

Using half duplex relay nodes, the communication between source and destination pairs is completed in two time slots. At the first time slot, the / th source node transmits an Nsk x 1 signal vector

xsk = Bk Sk, k = 1, ••• , K

to the relay nodes and the destination nodes, where S/ is the d x 1 information-carrying symbol vector and B/ is the Nsk x d source precoding matrix. The received signal vectors at the 1th relay node and the kth destination node are given by

Yri Hlk*sk + Vrl, l = 1, ••• , L (2)

Yd1k = Y^ TkmXsm + Vdlk, k = 1, ••• , K (3)

where Hik is the Nri x Nsk MIMO fading channel matrix between the kth source node and the ith relay node, Tkm is the Ndk x Nsm MIMO fading channel matrix between the mth source node and the kth destination node, vri is the Nri x 1 additive white Gaussian noise (AWGN) vector at the ith relay node with zero mean and covariance matrix E [vriVi] = ^N, i = 1, ••• , L, and vd1k is the Ndk x 1 AWGN vector at the kth destination node at the first time slot with zero mean and covariance matrix E [vduvf^] =

°dk lNdk, k = 1 ••• ,K •

During the second time slot, the received signal vector at the ith relay node is amplified with the Nri x Nri precoding matrix Fi as

xrl = Fiyrl, l = 1, ••• , L.

Yd2k = GklXrl + Vd2k, k = 1, ••• , K

yd2k yd1k

GklFlHlm

BmSm +

vdk vd1k

Sk = wH Yk,

k = 1, ••• , K

where Wk = [W^2, W^]T is the receiver weight matrix, and Wk1 and Wk2 are the Ndk x d receiver weight matrices for the direct link and the relay link, respectively. In Equations 6 and 7, we have

Sk = [W, WHi]

yd2k yd1k

= ( WH2 E GklFlHlk + wH1TkJ BkSk

desired signal + E (w*2 ^ GklFlHlm + WHTkm) BmSm

m=1,m=k\ l=1 /

interference

+ WfcVdk + WH1 vd1k.

In Equations 1 and 4, the transmission power constraints at the source and relay nodes can be written as

tr (Bk BH) < Psk, k = 1, ••• , K

The precoded signal vector xri is forwarded to the destination nodes. The received signal vector at the kth destination node is given by

where Gki is the Ndk x Nri MIMO channel matrix between the lth relay node and the kth destination node and vd2k is the Ndk x 1 AWGN vector at the kth destination node at the second time slot with zero mean and covariance

matrix E [vd2k= adklNdk,k = ••• ,K.

From Equations 1 to 5, the signal vector received at the kth destination node over two consecutive time slots is

tr (FiE [yriYHA Ff) < Pri, i = 1, ••• , L (10)

where Psk and Pri denote the power budget at the kth source node and the ith relay node, respectively, and E[YriYrf] = Em=1 HimBmBmHfm + lNrf is the covari-ance matrix of the received signal vector at the ith relay node.

In this paper, we aim at optimizing the source precoding matrices {Bk} = {Bk, k = 1, ••• , K}, the relay precoding matrices {Fi} = {Fi, i = 1, ••• , L}, and the receiver weight matrices {Wk} = {Wk, k = 1, ••• , K}, to minimize the sum-MSE of the signal waveform estimation at the destination nodes under transmission power constraints at the source and relay nodes. We would like to mention that minimal MSE (MMSE) is a sensible design criterion based on the links of MSE to other performance measures in MIMO systems such as mutual information and SINR [4,11].

From Equation 8, the MSE of the kth source-destination pair can be calculated as

where Vdk = ^i=\ GkiF(vri + vdik is the total noise vector at the kth destination node at the second time slot.

Due to their simplicity, linear receivers are used at the destination nodes to retrieve the transmitted signals. Thus, the estimated signal vector at the kth destination node can be written as

= tr = tr

MSEk = tr(ß (sk - Sk) (Sk - Skf )

(wHHkk - Id) (wHHkk - Id

+WH CkWk + WH EkWkJ , k = 1, ••• , K (11)

where Hkm is the equivalent MIMO channel matrix from the mth source node to the kth destination node, Ck =

E{ \yTdk, vT1kf [vdk, vd1k]} and Ek are the covariance

matrices of the equivalent noise and the interference at the kth destination node, respectively. For k, m = 1, ••• , K, they are given respectively as

Hkm —

J2 GklFlHlm l=1

Ek = H kmH km

m=1,m= k

E<lGkl FlFH GH + ^ iNdk l=1

'dk lNdk

min y^MSE/

{Wk},{Bk},{F;} j-^ k=1

.t. tr (Bk BH) < Psk, k = 1, ••• , K

min MSEk. Wk

The solution to the problem (Equation 15) is the well-known MMSE receiver [12] given by

(H kk H

+ Ck + Ek) H kk, k = 1, ••• , K. (16)

Let us introduce fi = vec(Fi), l = 1, ••• , L. With given receiver matrices (Wk} and source precoding matrices

{B/}, the sum-MSE SMSE = 1 MSE/ can be rewritten as a function of f = [fj, fT, ••• , fj] as

f1(f) = E

(Ok/f - ok)H (O/kf - ok) + fH Qkf

+ Y^ (°kmf - q/m) (Okmf - q/

+11 (17)

where t1 = Y^k=1 adktr (WHW/) is independent of f and for k, m = 1, ••• , K, l = 1, ••• , L

where Him = HimBm is the equivalent MIMO channel matrix between the mth source node and the ith relay node.

From Equations 9 to 11, the optimal source, relay, and receiver matrix design problem can be written as

°km = [°k Ok,L,m

Qk = bd (Q/1, Q/2, ••• , Q/l)

(18) (19)

tr (Fi£ [yrfyf] Ff) < Pri, l = 1, ••• , L. (14)

3 Proposed source, relay, and receiver matrix design algorithms

The problem (Equations 12 to 14) is highly nonconvex with matrix variables, and a globally optimal solution is intractable to obtain. In this section, we propose two iterative algorithms to solve the problem (Equations 12 to 14) by optimizing (Wk}, (Bk}, and (Fi} in an alternating way through solving convex subproblems.

3.1 Proposed Algorithm 1

In each iteration of this algorithm, we first optimize (Wk} based on (B^} and (Fi} from the previous iteration. Then, we optimize all relay matrices based on (W^} from the current iteration and (B^} from the previous iteration. Finally, we optimize all source matrices using (W^} and (Fi} from the current iteration.

It can be seen from Equation 11 that W^ only affects MSE^. Thus, with given (Fi} and (B^}, the optimal linear receiver matrix which minimizes MSE^ in Equation 11 is the solution to the following unconstrained optimization problem

ok = vec (id - Tkk), q/m = -vec (T/m) (20) °k,l,m = Hm ® G kl, Qkl = or2 iNrf ® (G% G kl) . (21)

Here, G kl = wh2G/i is the equivalent MIMO channel matrix between the lth relay node and the kth destination node and T/m = WHT/mBm is the equivalent direct link MIMO channel matrix between the mth source node and the kth destination node. The detailed proof of Equation 17 is given in Appendix 6.1.

By introducing

Dll =f £ HlmBmB^Hm + a?lIn) ® N, l = 1, ••• , L

\m=1 '

and Dl = bd (Du, D/2, • •• , D/l), where Dly = 0, l = j, the relay transmit power constraints in Equation 10 can be rewritten as

f" ^lf < Prl, l = 1, ••• , L.

In Equations 17 and 23, the relay matrix optimization problem can be written as

min (f)

s.t. fH ID lf < Prl, l = 1, ••• , L.

The optimization problem (Equations 24 to 25) is a quadratically constrained quadratic programming (QCQP) problem [13]. From Equation 21, we can see that Qki, k = 1, ••• , K, i = 1, ••• , L are positive semidefi-nite (PSD) matrices, and thus from Equation 19, Qk, k = 1, ••• , K are PSD matrices. Moreover, it can be seen from Equation 22 that Dg, i = 1, ••• , L are PSD matrices, and thus, Di, i = 1, ••• , L are PSD matrices. Therefore, the QCQP problem (Equations 24 to 25) is convex and can be efficiently solved by the interior-point method [13]. In particular, the problem (Equations 24 to 25) can be solved by the CVX MATLAB toolbox for disciplined convex programming [14].

Let us introduce bk = vec(Bk), k = 1, ••• , K. With given receiver matrices (Wk} and relay matrices (Fi}, the sum-MSE can be rewritten as a function of b = [b[, bT, ••• ,bKf as

$1(b) = £ (Skb - vec(Id)f (Skb - vec(Id)) +bHUb+12. k=1

where t2 = EK=i tr(WHCkWk) can be ignored in the optimization process as it is independent of b and

U = bd(U1, U2, ••• , Uk ) (27)

Sk = [Sk1, Sk2, ••• , SkK] (28)

Skk = Id ® Pkk, Ski = 0, i = k (29)

Uk = Id ®f ]T P^kP. (30)

\m=1,m=k J

Here, Pmk = WaE <miF;Hk + W^T^t. The detailed ¿=1

proof of Equation 26 is given in Appendix 7.1.

Let us introduce Ej = Id ® (HHFHFiHj), E; =

bd (E;1, E;2, ••• , Ek), EEi = bd (EEi1, Ei2, ••• , EEiK), where Eft = IdNs and EEij = 0, i = j. The optimal b can be obtained by solving the following problem

min $1(b) b

s.t. bHEEmb < Psm, m = 1, ••• , K

bHElb < Pri - ar2tr (FlFH), l = 1, ••

(32) , L. (33)

lower bounded by at least zero. Therefore, the proposed Algorithm 1 is guaranteed to converge.

3.2 Proposed Algorithm 2

In the proposed Algorithm 1, all source precoding matrices are optimized together through b, while all relay precoding matrices are updated together through f. Since the dimensions of b and f are EK=1Nskd and EtiN;, respectively, the computational complexity of solving the QCQP problems (Equations 24 to 25 and 31 to 33) using

the interior point method [15] is O ^(EK=i NskdJ ^ and

O ^ (E^iN;) respectively. Therefore, the computational complexity at each iteration of the proposed Algorithm 1 is O ^(EK=1Nsk^3 + (El=1N2)3), which can

be very high for interference MIMO relay systems with a large K and L. To reduce the per-iteration complexity, in this subsection, we develop an iterative algorithm where each source and relay matrix is optimized individually by fixing all other matrices.

Adopting notations from proposed Algorithm 1, with given receiver matrices {Wk}, source precoding matrices {Bk}, and relay precoding matrices Fj, j = 1, ••• , L, j = I, the sum-MSE can be rewritten as a function of F; as

From Equation 30, we can see that Uk, k = 1, ••• , K are PSD matrices, and thus from Equation 27, U is PSD. Moreover, it can be seen that Em, m = 1, ••• , K and E;, / = 1, ••• , L are PSD matrices. Therefore, the problem (Equations 31 to 33) is a convex QCQP problem and can be solved by the CVX MATLAB toolbox [14] for disciplined convex programming.

The steps of applying the proposed Algorithm 1 to optimize {Bk}, {F;}, and {Wk} are summarized in Table 1, where the superscript (n) denotes the variable at the nth iteration, and e is a small positive number up to which convergence is acceptable. Since all subproblems (Equation 15, 24 to 25, and 31 to 33) are convex, the solution to each subproblem is optimal. Thus, the value of the objective function (Equation 12) monotonically decreases after each iteration. Moreover, the value of Equation 12 is

SMSE = J2 (GklF,Hik - Aki) (Gkl№ - A,

klFlHlk - Akl) (,GklFlHlk - Akl

+ J2 "¿G kl FlFH

Gh + ^^ (GklFlHlm - Dk,l,m

l=1 m=1,m=k

x (GklFlHlm -Dk,lm

where for k, m = 1, ••• , K, l = 1, ••• , L

Akl = Id GkjFjHjk - Tkk j=1,j=l L

Dk,l,m = ^ 'GkjFjHjm - Tkm. j=1,j=l

Table 1 Procedure of solving the problem (Equations 12 to 14) by the proposed Algorithm 1

Steps Description

1) Initialize the algorithm with |f(0) J and |b[0) j satisfying Equations 9 and 10; Set n = 0.

2) Obtain {Wf+1)} based on Equation 16 with fixed {F((n)} and {b^}.

3) Update {F(n+1)} through solving the problem (Equations 24 to 25) with given {B<n)} and {w^0

4) Update {B[n+1)} by solving the problem (Equations 31 to 33) with fixed {F((n+1:i} and {wf+^J.

5) If SMSE(n) - SMSE(n+1) < e, then end. Otherwise, let n := n + 1 and go to step 2.

Using the identities in Equations 43 to 45, the SMSE in Equation 34 can be written as

f2(fl) =

T!j=i,j=k bf Eijbj,

(Okikf - aki)H (Okif - aki) + ff Qf + m

min f2(ft) s.t. ff Dfi < Pri.

+bf Uk bk + zk

zk = [(Smmbm - vec (Id))H (Smmbm - vec(Id))

m=1,m=k

+bmUmbm] + t2, k = 1,- • • , K.

By introducing cik = ofa (F;Ff) k = 1, • • • , K, l = 1, • • • , L, the optimal bk can be obtained by solving the following problem for each k = 1, • • • , K

+ (°k,l,mfl - dk,l,m)fOk,l,mfl - dk,l,m)

m=1,m=k

where for k, m = 1, • • • , K, l = 1, • • • , L akl = vec(Akl )

rki = tr I £ o^G k/F/FH Gj + Wk I

\j=i,j=l / dk,i,m = vec (Dk,im) •

Note that since the terms rki in Equation 36 are independent of fi, they can be ignored when optimizing fi. The relay transmit power constraint in Equation 10 can be rewritten as

if Diifi < Pri. (37)

Based on Equations 36 and 37, the optimal fi can be obtained by solving the following problem for each i = 1, ,L

min $2(bk) bk

s.t. bkHbk < Psk

bf Elk bk < Prl - Clk, l = L •

• • , L.

The problem (Equation 38) is a QCQP problem and can be solved effectively using the CVX toolbox.

With given receiver matrices (Wk}, relay precoding matrices (Fi}, and source precoding matrices By, j = 1, • • • , K, j = k, the SMSE can be rewritten as a function of bk as

$2(bk) = (Skkbk - vec(Id))H(Skkbk - vec(Id))

The problem (Equations 39 to 41) is a QCQP problem and can be solved by the CVX MATLAB toolbox [14] for disciplined convex programming. The steps of using the proposed Algorithm 2 to optimize (Bk}, (Fi}, and (Wk} are summarized in Table 2. Similar to the analysis used to the proposed Algorithm 1, since all subproblems (Equations 15, 38, and 39 to 41) are convex, the solution to each subproblem is optimal. Thus, the value of the objective function (Equation 12) monotonically decreases after each iteration. Moreover, the value of Equation 12 is lower bounded by at least zero. Therefore, the convergence of the proposed Algorithm 2 follows directly from this observation.

Since the dimensions of bk and fi are Nskd and N.2, respectively, the computational complexity of solving the QCQP problems (Equations 38 and 39 to 41) is O ((Nskd)3) and O (N6), respectively. Thus, the computational complexity at each iteration of the proposed

Algorithm 2 is O (£K=i(Nskd)3 + EL=iN6i), which is lower than the per-iteration computational complexity of the proposed Algorithm 1. However, we will see through numerical simulations that the proposed Algorithm 1 has a better MSE and BER performance than that of the proposed Algorithm 2. Such performance-complexity tradeoff is very useful for practical interference MIMO relay communication systems.

4 Numerical examples

In this section, we illustrate the performance of the proposed algorithms through numerical simulations. All channel matrices have independent and identically distributed (i.i.d.) complex Gaussian entries with zero mean and unit variance. The noises are i.i.d. Gaussian with zero mean and unit variance. Unless explicitly mentioned, the

Table 2 Procedure of solving the problem (Equations 12 to 14) by the proposed Algorithm 2

Steps Description

1) Initialize the algorithm with |f(0) J and |b[0) j satisfying Equations 9 and 10; Set n = 0.

2) Obtain {W<n+1)} based on Equation 16 with fixed {F((n)} and {B<n)}.

3) For I = 1,- • • ,L, update F(n+1) through solving the problem (Equation 38) with given {b^1}, {w^+^J,andFJ(n),j = 1,- • • ,L,j = I.

4) For k = 1,. • • ,K, update B<n+1) by solving the problem (Equations 39 to 41) with fixed {f^0}, {w<n+1^, and Bj(n), j = 1,. • • , K, j = k.

5) If SMSE(n) - SMSE(n+1) < s, then end. Otherwise, let n := n + 1 and go to step 2.

QPSK constellations are used to modulate the source symbols. For the sake of simplicity, we set d = 2 and assume that all nodes have three antennas, i.e., Nsk = Ndk = Nri = 3, k = 1, ••• , K, l = 1, ••• , L, all source nodes have the same power budget as Psk = 15dB, k = 1, ••• , K, and all relay nodes have the same power budget as Pri = P, l = 1, ••• , L.

For all simulation examples, the simulation results are averaged over 105 independent channel realizations. Unless explicitly mentioned, we assume that there are K = 4 source-destination pairs and L = 5 relay nodes in the interference MIMO relay system. The proposed algorithms are initialized at F^0-1 =

JPrl/tr (Eil (0) _ nr-iw

=1 HikHlk + \Nrl )lNrf, l = 1, ••• , L, and

Bp = VPsk/NskN, k = 1, ••• , K. We would like to mention that when the matrix weight is identity matrix, the performance of the matrix-weighted sum-MSE minimization (WMSE) algorithm without power control in [10] is similar to the proposed Algorithm 2 without considering the direct links.

In the first example, we study the performance of the proposed algorithms at different number of iterations. We also compare the performance of the algorithms when the direct links are ignored. Moreover, the performance of the total leakage minimization (TLM) algorithm in [10] is included as a benchmark. Figure 2 shows the MSE performance of the proposed algorithms versus P at different number of iterations for the first source-destination pair (k = 1). It can be seen from Figure 2 that both proposed algorithms perform better than the TLM algorithm when the direct links are ignored. The performance of both proposed algorithm is significantly improved when the direct links are taken into account. For both proposed

algorithms, the MSE reduces with increasing number of iterations. Moreover, it can be observed that after ten iterations, the decreasing of the MSE is small. Thus, we suggest that only ten iterations need to be carried out in practice to achieve a good performance-complexity tradeoff. It can also be seen from Figure 2 that both proposed algorithms have almost the same MSE performance at convergence.

For this example, the average BER of all source-destination pairs yielded by both proposed algorithms versus P at different number of iterations is shown in Figure 3. It can be clearly seen that the proposed algorithms with direct links yield much smaller BER than the case when the direct links are ignored, especially at high P level. We can also observe from Figure 3 that the proposed Algorithm 1 has a slightly better BER performance than the proposed Algorithm 2. It can also be seen from Figure 3 that when the direct links are ignored, the proposed algorithms perform better than the TLM algorithm.

In the second example, we study the performance of the proposed algorithms with different number of relay nodes. Figure 4 shows the MSE performance of the proposed Algorithm 1 versus P with L = 5 and L = 10. It can be seen that by doubling the number of relay nodes, a power gain of 10 dB is obtained at the MSE of 0.2.

For this example, the BER performance of the proposed Algorithm 1 with L = 5 and L = 10 is illustrated in Figure 5. It can be seen that by increasing the number of relay nodes, the system spatial diversity is increased, and thus, a better BER performance is achieved. In particular, we observe that an 8 dB gain is obtained at the BER of 10-3 by increasing L from 5 to 10.

■ -•- ■ Alg. 2 no DL at 10) -+- Alg. 2 no DL (conv.)

- A - Alg. 2 with DL (It 10)

- ► - Alg. 2 with DL (conv.) —«— Alg. 1 no DL № 10)

Alg. 1 no DL (conv.) Alg. 1 with DL (It 10) Alg. 1 with DL (conv.) TLM algorithm

10 P (dB)

Figure 2 Example 1: MSE versus P at different number of iterations.

-«—Alg. 1 no DL (It 10) 0 Alg. 1 no DL (conv.) -B— Alg. 1 with DL at 10) ^ Alg. 1 with DL (conv.)

■ • - Alg. 2 no DL № 10) + - Alg. 2 no DL (conv.)

■ A - Alg. 2 with DL at 10)

■ ► - Alg. 2 with DL (conv.)

TLM algorithm

10 P (dB)

Figure 3 Example 1: BER versus P at different number of iterations.

-10 relays (It 10) -10 relays (converged) -5 relays (It 10) - 5 relays (converged)

Figure 4 Example 2: MSE versus P for different L

-•— Alg. 1 (16QAM, K=4) —I— Alg. 1 (16QAM, K=3) -A— Alg. 1 (QPSK K=4) —►—Alg. 1 (QPSK, K=3)

- » - Alg. 2(16QAM,K=4)

- « - Alg. 2 (16QAM, K=3)

- B - Alg. 2 (QPSK, K=4)

- ★ - Alg. 2 (QPSK, K=3)

Figure 6 Example 3: BER versus P for different K.

In the next example, we study the performance of the proposed algorithms with different number of source-destination pairs. Figure 6 shows the BER performance of both proposed algorithms versus P. Moreover, the BER of both algorithms using the 16QAM modulation scheme is also illustrated in Figure 6. As expected, the system BER is increased when higher order constellations are used. We can also obverse from Figure 6 that with a smaller number of source-destination pairs, the number of interference channels decreases which yields a better BER performance. Interestingly, the BER difference between the two proposed algorithm becomes bigger when K = 3.

In the last example, we study the performance of the proposed algorithms on the achievable end-to-end sum-rates of all source-destination pairs. It can be seen from Figure 7 that, as expected, with the direct links taken into

account, both proposed algorithms achieve a higher sum-rate. Figure 7 shows that the proposed Algorithm 1 yields slightly better rate than the proposed Algorithm 2.

5 Conclusions

We have investigated the transceiver design for interference MIMO relay systems with direct source-destination links based on the MMSE criterion. Two iterative algorithms have been developed to jointly optimize the source, relay, and receiver matrices under power constrains at each source node and relay node. Numerical simulation results show that the proposed algorithms converge quickly after a few iterations. The system MSE and BER performance can be significantly improved compared with the algorithms without considering the direct links. The proposed Algorithm 1 has a better MSE and BER

performance than the proposed Algorithm 2 at a higher per-iteration computational complexity.

6 Appendix A

6.1 Proof of Equation 17

From Equation 11, we have

= E tr

g kl fi h lk + t kk - id^

(L - \H L I £ G kl FlH lk + T kk -^ I + £ arlG kl Fl fH G

\l=1 / l=1

+a2k WH2 Wk2 + WH1 Wk1

K ( L \( L \H

HGkl Ft Hilm + Tk Il kl Fl H lm + Tkm I I

m=At1 /\l—1 '

Using the identities of [16]

tr (AT b) = (vec (A))T vec(B) tr (AHBAC) = (vec(A))H (ct0 b) vec(A) (44) vec (ABC) = (ct 0 A vec(B) (45)

the SMSE (Equation 42) can be represented as a function of fi, i = 1, • • • , L, as

SMSE =

°k,l,kfl - ok^ °k,l,kfl - ok^ J2 ( J2°k,l,m fl- q/m M J2°k,l,

m=k\l=1 ! \l=1

+ ^ I Z^°k,i ,m fi m=k\i= L

+ £ ff Qki fi +11 i=1

= f1 (f).

7 Appendix B

7.1 Proof of Equation 26

From Equation 11, we have

lm fl- qkm

SMSE = J2tr[(P/kBk-Id) kkBk-Idf

+12. (46)

+ P km Bm BHm pHm

m=1,m=k

Using the identities in Equations 43 to 45, the SMSE function in Equation 46 can be written as SMSE

(S/k bk - vec (Id ))H (S/k bk - vec (Id ))

+ £ bm (Id 0 PHmPkm) bm +12

m=1,m=k K

= J2 (Skk bk - vec (Id ))H(Skk bk - vec (Id)) + bf Uk bk] +12 k=1 = (b).

Competing interests

The authors declare that they have no competing interests.

Acknowledgements

This work was supported under the Australian Research Council's Discovery

Projects funding scheme (project numbers DP110100736 and DP140102131).

Received: 25 August 2014 Accepted: 23 February 2015 Published online: 13 March 2015

References

1. Y Fan, J Thompson, MIMO configurations for relay channels: theory and practice. IEEE Trans. Wireless Commun. 6,1774-1786 (2007)

2. L Sanguinetti, AA D'Amico, Y Rong, A tutorial on the optimization of amplify-and-forward MIMO relay systems. IEEE J. Selet. Areas Commun. 30,1331-1346 (2012)

3. G Kramer, M Gastpar, P Gupta, Cooperative strategies and capacity theorems for relay networks. IEEE Trans. Inf. Theory. 51, 3037-3063 (2005)

4. Y Rong, X Tang, Y Hua, A unified framework for optimizing linear nonregenerative multicarrier MIMO relay communication systems. IEEE Trans. Signal Process. 57,4837-4851 (2009)

5. C Zhao, B Champagne, Joint design of multiple non-regenerative MIMO relaying matrices with power constraints. IEEE Trans. Signal Process. 61, 4861-4873 (2013)

6. M Maddah-Ali, A Motahari, A Khandani, Communication over MIMO X channels: interference alignment, decomposition, and performance analysis. IEEE Trans. Inf. Theory. 54, 3457-3470 (2008)

7. B Nourani, S Motahari, A Khandani, Relay-aided interference alignment for the quasi-static X channel. IEEE International Symposium on Information Theory, 1764-1768 (2009)

8. X Wang, Y-P Zhang, P Zhang, X Ren, Relay-aided interference alignment for MIMO cellular networks. IEEE International Symposium on Information Theory, 2641-2645 (2012)

9. M Khandaker, Y Rong, Interference MIMO relay channel: joint power control and transceiver-relay beamforming. IEEE Trans. Signal Process. 60, 6509-6518(2012)

10. KTTruong, P Sartori, RW Heath, Cooperative algorithms for MIMO amplify-and-forward relay networks. IEEE Trans. Signal Process. 61, 1272-1287 (2013)

11. DP Palomar, JM Cioffi, MA Lagunas, Joint Tx-Rx beamforming design for multicarrier MIMO channels: a unified framework for convex optimization. IEEE Trans. Signal Process. 51,2381-2401 (2003)

12. SM Kay, Fundamentals of Statistical Signal Processing: Estimation Theory. (Prentice Hall, Englewood Cilffs, NJ, 1993)

13. S Boyd, L Vandenberghe, Convex Optimization. (Cambridge University Press, Cambridge, UK, 2004)

14. M Grant, S Boyd, Cvx: Matlab software for disciplined convex programming, version 2.0 beta. http://cvxr.com/cvx, Sep. 2013

15. Y Nesterov, A Nemirovski, Interior Point Polynomial Algorithms in Convex Programming, (Philadelphia, PA, SIAM, 1994)

16. JW Brewer, Kronecker products and matrix calculus in system theory. IEEE Trans. Circuits Syst. 25, 772-781 (1978)