Research

My main research interests target the design and analysis of wireless networked control systems. In such systems the control loops are closed via a communication network. As a result, its study necessitates the interplay between dynamical systems and their communication aspects. Toward this end, my research can be characterized as interdisciplinary because it combines tools from control and estimation theories, communications, and convex optimization. Specifically, my research span the areas of: i) information theory, ii) stochastic control and estimation theory.



Properties of information measures in systems with memory and feedback

In this research we investigate functional and topological properties of directed information and its variants that is known to be a handy information measure that quantifies the information rate in systems with memory and feedback. Directed information from an input process to an output process captures the uncertainty of the latter due to the causal knowledge of the former. In information theory, directed information or its variants are used to characterize capacity of channels with memory and feedback and lossy data compression of causal and zero-delay codes. Moreover, it can be used in network communication systems as a metric for evaluating the capacity of special types of networks, such as, the two-way channel, the multiple access channel, etc. Furthermore, directed information has found usage in a variety of problems subject to causality constraints, such as, gambling, portfolio theory, data compression and hypothesis testing, in biology as an alternative to Granger's measure of causality, and in communication for networked control systems.

MapReduce

Contribution: We derive several functional and topological properties of directed information, defined on general abstract alphabets (complete separable metric spaces), using the topology of weak convergence of probability measures. These include convexity and concavity of directed information, lower semicontinuity of directed information, and under certain conditions continuity. Finally, we derive variational equalities for directed information, including sequential versions. These may be viewed as the analogue of the variational equalities of mutual information utilized in Blahut-Arimoto algorithms to numerically evaluate the capacity and rate distortion function of channels and sources. In summary, we extend the basic functional and topological properties of mutual information to directed information. These properties are fundamental in the sense that they are necessary in order to investigate the extremum problems of directed information which are the optimization problems of interest in communication systems with memory and feedback and in networked control systems.

Selected Publications
Book Chapters:
[3] C. D. Charalambous, P. A. Stavrou, and C. K. Kourtellaris, "Directed information on abstract spaces: Properties and extremum Problems," Coordination Control of Distributed Systems (Edt. by Jan Van Schuppen and Tiziano Villa) in ser. Lecture Notes in Control and Information Sciences, Vol. 456, pp. 307-315, November 2014.
Full Text

Journal papers:
[2] C. D. Charalambous, and P. A. Stavrou, "Directed information on abstract spaces: properties and variational equalities," IEEE Transactions on Information Theory , vol. 62, Issue 11, pp. 6019-6052, November 2016.
Full Text

Conference papers:
[10] P. A. Stavrou and C. D. Charalambous, "Variational equalities of directed information and applications " in proceedings of IEEE International Symposium on Information Theory (ISIT), 2013.
Full Text
[7] C. D. Charalambous and P. A. Stavrou, "Directed information on abstract Alphabets: properties and extremum problems," in proceedings of IEEE International Symposium on Information Theory (ISIT), 2012.
Full Text


Causal and zero-delay processing of information in systems with memory and feedback

In this research we investigate the rate performance and the coding aspects of causal and zero-delay codes (i.e., a zero-delay code is a causal code but not the other way around) in systems with memory and feedback. The information measure that is used to assess these classes of codes is the so-called information nonanticipative or sequential rate distortion function which is a lower bound to the operational causal rate distortion function. An important question in this case is why not to use the operational causal rate distortion function to measure the rate performance and construct efficient coding schemes to convey reliably information instead of using a lower bound? The answer is immediate since by definition the operational causal rate distortion function can be cast as a non-convex optimization problem. As a result, it is extremely difficult to be solved explicitly. On the other hand, information nonanticipative rate distortion is a convex optimization problem and it can be solved explicitly for a variety of systems including memory and feedback. In such systems, the question of interest is as follows. Using this information measure how close we can get to the operational causal rate distortion function? This research aims into providing insights to this question.

Zero-delay source coding system
Fig. An example of zero-delay processing of information.

Contribution: We investigate applications of information nonanticipative rate distortion function in causal and zero-delay source coding problems using analog or uncoded transmission based on average and excess distortion probability. We consider two application examples of sources with memory, the binary symmetric Markov source with parameter p, (BSMS(p)), and, the vector Gauss-Markov source. For the BSMS(p) with Hamming distance distortion, we obtain the closed form expression of information nonanticipative rate distortion function, and we show achievability using joint source-channel coding theory. For the vector Gauss-Markov source with average mean squared-error distortion, we obtain the parametric solution of information nonanticipative rate distortion function and show achievability in the context of joint source-channel coding applications. By realizing the optimal test-channel of the Gaussian information nonanticipative rate distortion function we show achievability via a noiseless coding scheme that makes use of uniform scalar quantization with subtractive dither. Compellingly, using quantization with dithering, it is possible to show that for infinite dimensional Gaussian sources the information nonanticipative rate distortion function and operational causal (and zero-delay) rate distortion functions coincide.

Selected Publications
Book Chapters:
[2] P. A. Stavrou, C. K. Kourtellaris, and C. D. Charalambous, "Information nonanticipative rate distortion function and its applications," Coordination Control of Distributed Systems (Edt. by Jan Van Schuppen and Tiziano Villa) in ser. Lecture Notes in Control and Information Sciences, Vol. 456, pp. 317-324, November 2014.
Full Text

Conference papers:
[18] P. A. Stavrou, J. Østergaard, C. D. Charalambous, and M. Derpich, "An upper bound to zero-delay rate distortion via Kalman filtering for Vector Gaussian sources," in proceedings of IEEE Information Theory Workshop (ITW), 2017 (accepted).
[16] P. A. Stavrou and J. Østergaard, "A lower bound on causal and zero-delay rate distortion for scalar Gaussian autoregressive sources," in proceedings of International Symposium on Information Theory and Signal Processing in Benelux (SITB), 2017.
Full Text
[12] P. A. Stavrou, C. K. Kourtellaris, and C. D. Charalambous, "Applications of information nonanticipative rate distortion function," in proceedings of IEEE International Symposium on Information Theory (ISIT), 2014.
Full Text


Low-delay joint source-channel coding

In this research we investigate the rate performance and the coding aspects of the ultimate communication scenario, that of merging the source dynamics and the channel dynamics in a unified framework. In such joint source-channel communication systems we aim into finding source-channel codes where the source and channel probability distributions, and the distortion measure of the source with the input cost function of the channel are favorably matched. Of particular interest in the recent technological advancements like for example 5G communication systems is the requirement for transmission of information in short data packets reliably with low-delay. This is another target of this research.

Realization
Fig. Probabilistic low-delay lossy joint source-channel coding for sources with memory with respect to average distortion or excess distortion probability.

Contribution: We investigate applications of information causal rate distortion function in zero-delay joint source-channel coding design using uncoded transmission based on average and excess distortion probability. These applications are described using two application examples of stationary ergodic sources with memory; a binary Markov source transmitted over a channel with unit memory with or without feedback, and a vector Gauss-Markov source transmitted over a memoryless vector AWGN channel. We also extend the Gaussian source-channel communication scenario to time-varying processes. In these application examples we assume that the sources and channels operate with matching number of samples/uses (i.e., no bandwidth constraints). Among others, for the time-varying Gaussian setting, we inspect the information rates performance from the aspect of resource allocations when the Gauss-Markov source is probabilistically matched to the noisy channel. Toward this end, we provide an iterative algorithm which can be seen as a joint source-channel resource allocation algorithm that performs the matching over time and space. This algorithm ensures that for each transmission, the end-to-end system operates at the minimum achievable distortion and shows that the probabilistic matching of the source and the channel prohibits the possibility of having the source transmitting data on higher rates than the ones allowed by the nominal capacity of the noisy channel. As a performance measure of the end-to-end dynamical system, we consider and compute the excess distortion probability to upper bound the probability of error at the decoder by showing that this decays exponentially. Moreover, we prove a converse theorem for this dynamical source-channel communication scheme. These operational results ensure that the proposed communication system operates reliably after a finite number of transmissions.

Selected Publications
Book Chapters:
[4] C. D. Charalambous, C. K. Kourtellaris, and P. A. Stavrou, "On Shannon's duality of a source and a channel and nonanticipative communication and communication for control," Coordination Control of Distributed Systems (Edt. by Jan Van Schuppen and Tiziano Villa) in ser. Lecture Notes in Control and Information Sciences, Vol. 456, pp. 291-305, November 2014.
Full Text
[1] C. K. Kourtellaris, C. D. Charalambous, and P. A. Stavrou, "Nonanticipative duality of sources and channels with memory and feedback," Coordination Control of Distributed Systems (Edt. by Jan Van Schuppen and Tiziano Villa) in ser. Lecture Notes in Control and Information Sciences, Vol. 456, pp. 325-335, November 2014.
Full Text

Conference papers:
[12] P. A. Stavrou, C. K. Kourtellaris, and C. D. Charalambous, "Applications of information nonanticipative rate distortion function," in proceedings of IEEE International Symposium on Information Theory (ISIT), 2014.
Full Text


Feedback capacity of channels with memory

In this research we investigate feedback capacity for a class of channels with memory with or without transmission cost constraints. Computing feedback capacity for any class of channel distributions with memory, with or without transmission cost constraints, computing the optimal strategies that achieves feedback capacity, and determining whether feedback increases capacity, are fundamental and challenging open problems in information and communication theories for half a century. Here we aim into determining the closed form expressions of the optimal strategies for a class of channels with memory without assuming a priori any assumptions of stationarity or ergodicity on the given channel.

Feedback Capacity
Fig. Convergence of the binary erasure channel with memory to its feedback capacity after a finite number of transmissions.

Contribution: We derive sequential necessary and sufficient conditions for any channel input probability distribution to maximize directed information for channels and cost functions with memory on the previous channel output symbols. To achieve this goal we apply finite time horizon dynamic programming algorithm and certain known information structures on the capacity achieving distributions. To demonstrate the validity of our results, we apply our framework into several channels with memory with or without transmission cost and we provide closed form expressions of the capacity achieving distributions and the corresponding feedback capacity. For certain channels we observe that feedback does not increase capacity.

Selected Publications
Journal papers:
[3] P. A. Stavrou, C. D. Charalambous and C. K. Kourtellaris, "Sequential necessary and sufficient conditions for capacity achieving distributions of channels with memory and feedback," IEEE Transactions on Information Theory (accepted), June 2017.

Conference papers:
[14] P. A. Stavrou, C. D. Charalambous, and C. K. Kourtellaris, "Sequential necessary and sufficient conditions for optimal channel input distributions of channels with memory and feedback," in proceedings of IEEE International Symposium on Information Theory (ISIT), 2016.
Full Text


Communication and Coding for Networked Control Systems

In this research we investigate the communication aspects of a simple networked control system. In particular, we are interested in the source coding aspects of the realistic possibility that the channel connecting the observer to the controller might be noisy or noiseless. In the case where the controller is separated from the purely communication part of the closed loop system, we wish to design a low-delay optimal communication strategy so that at the output of this system the estimated process obtained based on an optimal linear least squares estimator (Kalman filter) satisfies an end-to-end average fidelity or distortion criterion.

Filtering
Fig. Communicating a possibly unstable source over a noisy channel via a feedback realization scheme.

Contributions: (1) We develop finite-time horizon causal filters using the information causal rate distortion function. We apply the developed framework to design optimal filters for unstable Gaussian processes modeled in state space form with discrete recursions, subject to a mean squared error fidelity constraint. We show that such filters are equivalent to the design of an optimal encoder-channel-decoder, which ensures that the error satisfies a fidelity constraint. Unlike classical Kalman filters, the filter developed is characterized by a reverse-waterfilling algorithm, which ensures that the fidelity constraint is satisfied. (2) We derive bounds on the minimum data rate required to achieve a prescribed closed-loop performance level in a simple networked control system (NCS). In this NCS the plant is assumed to have a single output measurement and a single control output. From a communication point of view, the NCS exchange data between the encoder and the decoder via a noiseless channel with some fixed delay. The performance of the system is assessed using a linear source coding scheme based on entropy coded dither quantizers followed by instantaneous coding.

Selected Publications
Book Chapters:
[4] C. D. Charalambous, C. K. Kourtellaris, and P. A. Stavrou, "On Shannon's duality of a source and a channel and nonanticipative communication and communication for control," Coordination Control of Distributed Systems (Edt. by Jan Van Schuppen and Tiziano Villa) in ser. Lecture Notes in Control and Information Sciences, Vol. 456, pp. 291-305, November 2014.
Full Text

Journals:
[1] C. D. Charalambous, P. A. Stavrou, and N. U. Ahmed, "Nonanticipative rate distortion function and relations to filtering theory," IEEE Transactions on Automatic Control, Vol. 59, Issue 4, pp. 937-952, April 2014.
Full Text

Conference papers:
[17] M. Barforooshan, J. Østergaard, and P. A. Stavrou, "Achievable performance of zero-delay variable-rate coding in rate constrained networked control systems with channel delay," in proceedings of IEEE Conference on Decision and Control (CDC), 2017.
[15] P. A. Stavrou, T. Charalambous, and C. D. Charalambous, "Filtering with fidelity for time-varying Gauss-Markov processes," in proceedings of IEEE 55th Conference on Decision and Control (CDC), 2016.
Full Text
[11] C. D. Charalambous and P. A. Stavrou, "Optimization of directed information and relations to filtering theory," in proceedings of 13th European Control Conference (ECC), 2014.
Full Text
[9] C. D. Charalambous and P. A. Stavrou, "On the relation of nonanticipative rate distortion function and filtering theory," in proceedings of 12th Biannual European Control Conference (ECC), 2013.
Full Text