Information Theory

Course Information

College	Tsuyama College	Year	2021
Course Title	Information Theory
Course Code	0147	Course Category	Specialized / Elective
Class Format	Lecture	Credits	Academic Credit: 2
Department	Department of Integrated Science and Technology Electrical and Electronic Systems Program	Student Grade	5th
Term	Year-round	Classes per Week	1
Textbook and/or Teaching Materials	Textbooks : Nakamura Atsuyoshi et al., "Kisokaramanabu jyouhouriron 2nd Ed.(Japanese)"(Muisuri), Reference books : Imai Hideki, "Jyouhouriron(Ohmu)", Murata Noboru,"Jyouhouriron no kiso"(Saiensu)
Instructor	KIKUCHI Yosuke

Course Objectives

Learning purposes :
Students who have taken this course understand basic way of thinking of information theory as base of Information engineering.

Course Objectives :
1. To be able to understand concept and definition of information and calculate it
2. To be able to understand notion and definition of entropy and calculate it
3. To be able to explain model of information source and information source encoding
4. To be able to explain model of communication channel and communication channel encoding

Rubric

	Excellent	Good	Acceptable	Not acceptable
Achievement 1	The students understand the definition of information and can calculate several information. Moreover they can solve advanced applied problems.	The students understand the definition of information and can calculate several information.	The students understand the definition of information and can calculate some information with reference.	The students do not know the definition of information.
Achievement 2	The students understand the definition of entropy and solve advanced applied problems.	The students understand the definition of entropy and calculate several entropy.	The students understand the definition of entropy and calculate several entropy with reference.	The students do not know the definition of entropy.
Achievement 3	The students know kind of information source and can explain them. They can also explain Markov source and state transition diaram. They can calculate stationary distribution of Markov source and its entropy. Moreover they can solve advanced applied problems.	The students know kind of information source and can explain them. They can also explain Markov source and state transition diaram. They can calculate stationary distribution of Markov source and its entropy.	The students know kind of information source. They can explain Markov source and state transition diaram. They can calculate stationary distribution of Markov source.	The students do not know kind of information source. They can not explain Markov source and state transition diaram.
Achievement 4	The students can explain the binary symmetric channel and the binary erasure channel and calculate channel capacities of each channel. Moreover they can calculate channel capacities for several channel. They can also solve advanced applied problem.	The students can explain the binary symmetric channel and the binary erasure channel and calculate channel capacities of each channel. Moreover they can calculate channel capacities for several channel.	The students can explain the binary symmetric channel and the binary erasure channel and calculate channel capacities of each channel.	The students can not explain the binary symmetric channel and the binary erasure channel

Teaching Method

Outline:

※Relationship with practice: Instructor background: This course is provided by a teacher who worked at another institute, IMAI Quantum Computation and Information Project and Quantum Computation and Information Project Solution Oriented Research for Science and Technology. The purpose of this course is understanding the basic idea of information theory as the basis of information engineering, using the instructor's experience. This course is given in the teacher-lecture format.

General or Specialized : Specialized
Foundational academic disciplines : Integrated Disciplines/Informatics/Principles of Informatics
Field of learning : Infromation system・Programming・Network
Relationship with Educational Objectives :This class is equivalent to "(3) Acquire deep foundation knowledge of the major subject area".

MCC Goals(Based on the guideline 4/28/2017 version, number in brackets is MCC level)： V-D-7 Information mathematics・Information theory/Information theory(4)

Relationship with JABEE programs :
The main goal of learning / education in this class are "A",(A-2).

Course outline :
At first, information theory quantifies information and the theory is developed. Entropy and mutual information, dealt with in this course, are used not only in information engineering but also in machine learning, neuroscience and other fields and is an important concept. The purpose of this course is understanding the basis of this theory.

Style:

Course method :
This course provided based on textbook. Sometimes teacher give some quiz to students. This course is given at first semester and each lesson is 2 school hour lesson.

Grade evaluation method :
Exams (100%).
Examinations will be conducted 2 times, equally weighted. Bringing notebooks to examinations is not permitted. Generally, exam retakes are not allowed. If examination is not suitable for evaluating students, then re-examination may execute and the students evaluations may be reconsidered. Regular examinations are based on the evaluation rubric but there is no guarantee that the examinations will cover achievements cited in the rubric.

Notice:

Precautions on the enrollment :
This course frequent use mathematics. If the students are not good at math, they need to do preparation and review.

Course advice : This course deals with abstract concept. Then the students may not understand these concept without preparation and review. To study information theory deeply is recommended to the student who will go on to advanced course or university. The students need to consult with LMS(Blackboard) in advance.

Foundational subjects : Applied Mathematics Ⅰ(4th year)

Attendance advice : This course use knowledge of probability and statistics that were learned at 4th year. The students need to do review. If the students are 30 minutes or more late, they will be treated as absent 2 period. Check and preparing the theme of the lesson before attendance.

Characteristics of Class / Division in Learning

Instructor Professionally Experienced

Ｅｌｅｃｔｉｖｅ　ｍｕｓｔ　ｃｏｍｐｌｅｔｅ　ｓｕｂｊｅｃｔｓ

Course Plan

			Theme	Goals
2nd Semester
	3rd Quarter
		1st	Guidance
		2nd	Probablility, Conditional probability	To check the knowledge of probability
		3rd	Bayes' theorem, random variable	To check the knowledge of probability
		4th	Self-information	The students understand the definition of information and can calculate several information.
		5th	Entropy	The students can calculate entropy. And they understand the notion and definition of informationand calculate them.
		6th	Mutual information	The students can calculate mutual information. And they understand the notion and definition of informationand calculate them.
		7th	KL-divergence	The students can calculate KL-divergence. And they understand the notion and definition of informationand calculate them.
		8th	2nd semester mid-term exam
	4th Quarter
		9th	Return and commentary of exam answers
		10th	Information source	The students understand sampling and quantization of information. The students can explainmodel of information source and encoding of information source.
		11th	Encoding of information source	The students understand compression of information. The students can explainmodel of information source and encoding of information source.
		12th	Error detection and correction	The students understand error detection and correction codes.
		13th	Modeling of communication channel	The students understand channel has noise. The students can explain model and encoding of communication channel.
		14th	Channel capacity, encodig of communication channel	The students can explain model and encoding of communication channel.
		15th	(2nd semester final exam)
		16th	Return and commentary of exam answers

Evaluation Method and Weight (%)

	Examination	Presentation	Mutual Evaluations between students	Behavior	Portfolio	Other	Total
Subtotal	100	0	0	0	0	0	100
Basic Proficiency	0	0	0	0	0	0	0
Specialized Proficiency	100	0	0	0	0	0	100
Cross Area Proficiency	0	0	0	0	0	0	0