Relay-assisted cooperative communication is an important technique to enhance the transmission reliability and achieve spatial diversity gains in future wireless systems. Its basic idea is to introduce intermediate relay nodes to process and forward signals from the source to the destination. In this thesis, we investigate the design, analysis, and optimization of the wireless relay-assisted cooperative communication systems. Decode-and-forward (DF) is a widely adopted relaying protocol, where the relay node first detects the received signal and then re-encodes it before forwarding. Erroneous detections at the relay may cause error propagation. Therefore, the relay detection error scenarios need to be modeled to obtain near-optimal error performance at the destination. Popularly studied...[ Read more ]

Relay-assisted cooperative communication is an important technique to enhance the transmission reliability and achieve spatial diversity gains in future wireless systems. Its basic idea is to introduce intermediate relay nodes to process and forward signals from the source to the destination. In this thesis, we investigate the design, analysis, and optimization of the wireless relay-assisted cooperative communication systems. Decode-and-forward (DF) is a widely adopted relaying protocol, where the relay node first detects the received signal and then re-encodes it before forwarding. Erroneous detections at the relay may cause error propagation. Therefore, the relay detection error scenarios need to be modeled to obtain near-optimal error performance at the destination. Popularly studied detection schemes usually achieve this by exploiting the knowledge of the instantaneous channel state information (CSI) of the source-relay links. However, due to the inherent distributed nature of the relay networks, it may be practically impossible to acquire the accurate instantaneous CSI of the source-relay links at the destination, especially with a large number of relays or multiple antennas.

We firstly examine the problem of designing efficient near-optimal detectors for a single-source single-destination cooperative network with N parallel DF relays, where the destination is considered to have only the average CSI of the source-relay links and the instantaneous CSI of the source/relay-destination links. In this case, the relay detection error scenarios are modeled exploiting this average CSI. The state-of-the-art detector is called the almost maximum likelihood detector (AMLD), which achieves near-optimal performance with O(M²N) complexity for an M-ary modulation. We first propose an O(MN)-complexity near-optimal detector, which is an accurate approximation of the AMLD. By further exploiting the signal structures of pulse amplitude modulation (PAM) and quadrature amplitude modulation (QAM), we propose an O(1)-complexity near-optimal detector for the single-relay case. The dominant pairwise error probability (PEP) terms of the associated symbol error rate (SER) expression are then characterized for the proposed detectors (with a single relay). In addition, we prove that the achievable diversity orders of both the proposed detectors and the AMLD are exactly ⌈N/2⌉ + 1, which is further shown to be very accurate in various channel scenarios. This suggests that the full diversity order N + 1 may not be achievable with near-optimal detectors, except for the single-relay case.

We further examine the detection and performance analysis problems for the non-coherent counterpart network (with only the average CSI of the source-relay links available at the destination). To reduce the performance gap between this non-coherent DF relay network and its coherent counterpart, we consider the use of a generalized differential modulation (GDM) scheme, in which transmission power allocation over the M-ary phase shift keying (PSK) symbols is exploited when performing differential encoding (DE). In this case, a novel detector at the destination of such a non-coherent DF relay network is proposed. It is an accurate approximation of the state-of-the-art detector, called the (non-coherent) AMLD, but the detection complexity is considerably reduced from O(M²N) to O(MN). By characterizing the dominant error terms, we derive an accurate approximate SER expression. An optimized power allocation scheme for GDM is further designed based on this SER expression. Our simulation demonstrates that the proposed non-coherent scheme can perform close to the coherent counterpart as the block length increases. Additionally, we prove that the diversity order of both the proposed detector and the AMLD is exactly ⌈N/2⌉+1. Extensive simulation results further verify that the diversity expressions are accurate. This suggests that the full diversity N + 1 is not achievable for N > 1. For an extension, we consider a simultaneous wireless information and power transfer (SWIPT) enabled non-coherent DF relay network. For this network, in addition to proposing a new detector with SER analysis, we also develop algorithms to find an optimized power splitting coefficient at the relay node (which minimize the SER).

Besides, inspired by the booming deep learning (DL) technologies that have achieved tremendous successes in various applications including the wireless communication field, we investigate the autoencoder (AE) learning aided design scheme for relay-assisted cooperative communication systems where no CSI of any link is available. We represent the transmitter, relay node, and receiver using neural networks (NNs), such that the entire system can be optimized in a holistic manner. The conventional end-to-end training cannot be applied because the source-relay link information is practically unavailable at the destination, To address this issue, we propose a novel two-stage training approach to indirectly solve the end-to-end training problem by approximating the probability distributions in the loss function. The merits of the proposed scheme are verified via extensive simulation for various channel scenarios.

Furthermore, we investigate the promising cooperative non-orthogonal multiple access (NOMA) technique, which integrates cooperative communication techniques into NOMA and is able to increase spectral efficiency and the communication reliability of users under poor channel conditions. The conventional system design suffers from several inherent limitations and is not optimized from the bit error rate (BER) perspective. Motivated by this, we develop a novel learning-based cooperative NOMA scheme, drawing upon the recent advances in DL. We develop a novel hybrid-cascaded NN architecture such that the entire system can be optimized in a holistic manner. On this basis, we construct multiple loss functions to quantify the BER performance and propose a novel multi-task oriented two-stage training method to solve the end-to-end training problem in a self-supervised manner. The learning mechanism of each NN module is then analyzed based on information theory, offering insights into the explainable NN architecture and its corresponding training method. We also adapt the proposed scheme to handle the power allocation (PA) mismatch between training and inference and incorporate it with channel coding to combat signal deterioration. Simulation results verify its advantages over orthogonal multiple access (OMA) and the conventional cooperative NOMA scheme in various scenarios.