ECE 64200 - Information Theory and Source Coding

Credits: 3

Areas of Specialization(s):

Communications, Networking, Signal & Image Processing

Counts as:

Normally Offered: Fall - odd years

Catalog Description:
A treatment of the basic concepts of Information Theory. Determination of channel capacity and its relation to actual communication systems. Rate distortion theory is introduced and the performance of various source codes is presented.

Required Text(s):
  1. Elements of Information Theory, 2nd Edition, T.M. Cover and J.A. Thomas, John Wiley & Sons, 2006, ISBN No. 9780471241959.
Recommended Text(s):
  1. Information Theory and Reliable Communication, R. G. Gallager, John Wiley & Sons, ISBN No. 0-471-29048-3.

Lecture Outline:

Lectures Topic
6 1. Introduction A. A Prototypical Communications System (1.0) B. Discrete information source (1.0) C. Typical sequences (2.0) D. Weak Law of Large Numbers (1.0) E. Entropy (1.0)
5 2. Noiseless Source Coding A. Shannon's First Theorem (including converse) (2.0) B. Entropy and its properties (1.0) C. Variable Length Codes (2.0)
10 3. Channel Coding for Discrete Memoryless Channels A. Jointly typical sequences (1.0) B. Mutual information and its properties (2.0) C. Shannon's Second Theorem (including converse) (2.0) D. Combined source and channel coding theorem (1.0) E. Channel capacity (2.0) F. Continuous sources and channels (2.0)
4 4. Source Coding for Discrete Sources with Memory A. Typical sequences (1.0) B. Markov sources (1.0) C. Entropy of Markov sources (1.0) D. Coding theorems revisited (1.0)
16 5. Source Coding with a Fidelity Criterion A. Distortion Measures (1.0) B. The rate-distortion function (3.0) C. Block codes (1.0) D. Quantizers (2.0) E. Transform coding (2.0) F. Sliding block codes (2.0) G. Predictive coding (DPCM) (1.0) H. Universal source coding (Ziv-Lempel Algorithm) (2.0) I. Vector Quantizers (2.0)
3.0 6. Exams
1.0 A. Distortion measures
3.0 B. The rate-distortion function
1.0 C. Block Codes
2.0 D. Quantizers
2.0 E. Transform Coding
2.0 F. Sliding Block Codes
1.0 G. Predictive Coding (DPCM)