Information Theory

Course Information

College	Akashi College	Year	2021
Course Title	Information Theory
Course Code	0048	Course Category	Specialized / Elective
Class Format	Lecture	Credits	School Credit: 1
Department	Electrical and Computer Engineering Electrical Engineering Course	Student Grade	5th
Term	First Semester	Classes per Week	2
Textbook and/or Teaching Materials
Instructor	NAKAI Yuichi

Course Objectives

(1) Understand how the amount of information is defined and how its validity is guaranteed.
(2) Understand the definition of various sources of information and the meaning of entropy in each source, and can derive it.
(3) Understand the types of coding and the conditions that coding should meet, and can derive the average code length and its limits.
(4) Understand Shannon's first theorem and its significance.
(5) What is a communication channel and their types. In addition, understand how they can be expressed.
(6) Understand the significance of Shannon's second theorem.

Rubric

	Ideal Level	Standard Level	Unacceptable Level
Achievement 1	Can accurately explain how the amount of information is defined and how its validity is guaranteed.	Can explain how the amount of information is defined and how its validity is guaranteed.	Cannot explain how the amount of information is defined and how its validity is guaranteed.
Achievement 2	Understand the definition of various sources of information and the meaning of entropy in each source, and hence can derive it accurately.	Understand the definition of various sources of information and the meaning of entropy in each source, and hence can derive it.	Do not understand the definition of various sources of information and the meaning of entropy in each source.
Achievement 3	Understand the types of coding and the conditions that coding should meet, and can derive the average code length and its limits accurately.	Understand the types of coding and the conditions that coding should meet, and can derive the average code length and its limits.	Do not understand the types of coding and the conditions that coding should meet, the average code length and its limits.
	Can accurately explain Shannon's first theorem and its significance.	Can explain Shannon's first theorem and its significance.	Cannot explain Shannon's first theorem and its significance.
	Can specifically explain what is a communication channel and their types, and how they can be expressed.	Can explain what is a communication channel and their types, and how they can be expressed.	Cannot explain what is a communication channel and their types, and how they can be expressed.
	Can accurately explain the significance of Shannon's second theorem.	Can explain the significance of Shannon's second theorem.	Cannot explain the significance of Shannon's second theorem.

Assigned Department Objectives

学習・教育到達度目標 (D) （基礎学力と自主的・継続的学習能力）

学習・教育到達度目標 (H) （多次元的なシステム思考）

Teaching Method

Outline:

The outcome of information theory, which is founded by C.E. Shannon, are indispensable in modern life. In this lecture, we will explain the knowledge necessary to achieve "speedy" and "accurate" information transmission in communication systems. The first half of the course begins with the quantification of information and continues on to the first theorem of Shannon. Shannon's second theorem will be taught in second half after discussing the definition of a communication channel.

Style:

Classes will be held in a lecture style using slides. Students will be given practice questions for assignments, so actively work on them in order to test their understanding.

Notice:

Classes will be given on the premises that students have the knowledge of probability and statistics, so understand these content well beforehand.
Students who miss 1/3 or more of classes will not be eligible for a passing grade.

Characteristics of Class / Division in Learning

Applicable to Remote Class

Course Plan

			Theme	Goals
1st Semester
	1st Quarter
		1st	Model of a communication system	Can explain the model of a communication system which is assumed in information theory and quantify information.
		2nd	Memoryless sources and entropy	Can explain the simplest source of information, the memoryless source, and the entropy that provides clues to the internal structure of the source.
		3rd	Markov source and entropy	Can explain the Markov source, which is close to real-world information sources, and derive its entropy.
		4th	What is coding	Can define coding and explain some conditions that coding must meet.Can explain the definition of average coding length and its limit for instant decryption.
		5th	Shannon's source coding theorem	Can explain Shannon's source coding theorem and its significance.
		6th	Huffman coding	Can construct the Huffman code as a coding scheme that can configure compact coding.
		7th	Communication channel	Can explain the definition of communication channels and how they are represented.
		8th	Midterm exam
	2nd Quarter
		9th	Mutual information	Can explain what is mutual information, which is defined by the information transmitted over the communication channel.
		10th	Various communication channels	Can explain noiseless, deterministic, and longitudinal, and degraded communication channels.
		11th	Communication channel capacity	Can explain the definition of communication channel capacity derived from consideration of the mutual information.
		12th	Improved communication channel reliability	Can explain how to improve reliability in the transmission of information over communication channels.
		13th	Error rates and rules for judging	Can explain the rules for reducing the error rate in communication channels.
		14th	Shannon's noisy-channel coding theorem	Can explain Shannon's noisy-channel coding theorem for a binary symmetric channel.
		15th	Channel coding	Can explain the basic concept of some channel coding schemes.
		16th	Final exam

Evaluation Method and Weight (%)

	Examination	Presentation	Mutual Evaluations between students	Behavior	Portfolio	Other	Total
Subtotal	100	0	0	0	0	0	100
Basic Proficiency	0	0	0	0	0	0	0
Specialized Proficiency	100	0	0	0	0	0	100
Cross Area Proficiency	0	0	0	0	0	0	0