## CS103 ‐ Topics in Representation Learning, Information Theory and Control

Caltech, Winter Term 2019

Location: Friday 9-11am in 106 Annenberg

Instructor: Alessandro Achille (achille@cs.ucla.edu)

### Course description

**Prerequisites:** Background in machine learning, probability, Information Theory

**Summary:** This course will cover current topics on information theory,
deep representation learning, and control. We will use an information theoretic
formalisation to study properties of the representation
learned by deep networks, with focus on invariance
(e.g., translations, shape variability) and compositionality of the representation.
We will then see how this representations can be learnt, though implicit or
explicit biases, by deep networks, and exploited to perform efficient
exploration and control.

### Course Details

**First meeting:**Friday Jan 11 at 9am in 106 Annenberg- Lectures on Friday 9-11am in 106 Annenberg
**Office hours:**TBD**Piazza:**Coming soon**Grading:**70% homework assignments, 20% final review of a paper, 10% participation in class and on Piazza

### Instructor

Alessandro Achille (achille@cs.ucla.edu)### Schedule

Weeks | Topics |
---|---|

1 | Introduction: Overview of the class, embodied intelligence (sensing, cognition, action), role of representations |

2-3 | Invariant representations: overview of invariant representations, different formalizations (group invariance/equivariance, contractive representations, statistical independence), deep convolutional representations |

4 | Task-relevant information and information theoretic formalization of invariance: review of information theory, rate-distortion theory, Information Bottleneck, Actionable Information, minimality and invariance |

5-6 | Learning invariant representation: MDL principle, Variational Auto-Encoders, stochastic optimization, SGD as Variational Inference, Kramer's rate |

7 | Life-long learning of compositional representations: Compositionality and disentanglement, current formalizations (total correlation, linear action, causality), β-VAEs |

8-9 | Information and Actions: Variational Inference in Control/Reinforcement Learning, Visual Turing Test, information theoretic duality for control |

### Lectures

Lecture slides will be here, together with links to suggested readings for the class.**Lecture 1:**Slides, Readings:*Steps Towards a Theory of Visual Information*, Section 1