Title: EAI / 08_deep_learning

Q: Why can't an image be encoded as a vector of input pixels?
A: \(n^2\) weights: 1MB \(X \times Y\) RGB image needs \(9 \cdot 10^{12}\) weights

Q: Describe the idea of a CNN.
A: - spatially local connections in intial layers
- kernel/filter: a pattenr of weights replicated across multiple local regions.
in general: multiple (\(d\)) kernels
- convolution \(\mathbf{k} \ast \mathbf{x}\) is applying the kernel \(k\) to the local region \(r\)
- tensor operations are used to describe computation graphs

Q: Name the layers of a typical CNN. (5)
A: - input layer
- convolution layers
- pooling layers
- residual layer (to resolve vanishing gradients)
- full connected layer (to classify)

Q: Define convolution in CNNs.
A: \[\mathbf{z=k \ast x}\]\[\sum_{j=1}^l k_j \ast x_{j+i-(l+1)/s}\]\(\mathbf{x}... \text{input vector}\)
\(\mathbf{k}... \text{vector kernel of size }l\)
\(s... \text{stride}\)

Q: What is a receptive field?
A: The portion of the input affecting neuron's activation (red in figure).

Q: How do you make sure that each layer ends up with the same size after convolution?
A: padding to both ends

Q: What is the formula for calculating the receptive field?
A: layer \(m\ (m>1): m \cdot (l-1) +1\)

Q: What is layer pooling?
A: Summarizing adjacent units from preceding layer

Q: Why is pooling used?
A: Pooling is used to reduce the size of the matrix and help avoid overfitting.

Q: Name two examples of pooling.
A: - average pooling
- max pooling

Q: Describe Recurrent Neural Networks.
A: - Directed cycles in the network
- Delay of input (no instant self-loop), internal state, short term memory
- Can become unstable, oscilating, chaotic
- Example usage: analysis of sequential data

Q: What is the markov assumption?
A: The hidden state \(\mathbf{z_t}\) at time \(t\) suffices to capture the information from all previous inputs.

Q: How does a RNN udpate the hidden state?
A: \(\mathbf{z_t=f_w(z_{t-1},x_t})\)      for some parametrized function \(\mathbf{f_w}\)

Q: What limitation do Feed-Forward Neural Networks have on input?
A: Feed-Forward NNs can only handle fixed-length input sequences (input layer is fixed!).

Q: Describe Long Short-Term Memory (LSTM).
A: LSTM
- are building units to achieve a memory for long(er) time in RNNs
- record values in memory cells \(c\)
- use gating units for decisions on elements of \(c\):
  - forget gate \(g\): carry over or forget (reset to zero)
  - input gate \(i\): update (add new information from input vector)
  - output gate \(o\): transfer to short-term memory cell \(\mathbf{z}\) (akin to hidden state)

Q: Describe LSTM networks.
A: - store context, handle time lags of unknown duration between events
- well-suited for natural language processing, time-series forecasting, handwriting recognition, etc.
- more training effort than Feed-Forward NNs, less parallelization

Q: What is word embedding?
A: Word embedding represents words/tokens by vectors.
- is learned automatically
- similar words are mapped to points close in vectors space
- may be useful for answering analogy questions (is \(\mathbf{A}\)to \(\mathbf{B}\) what \(\mathbf{C}\) is to \(\mathbf{?}\))

Q: What is Feed-Forward Part-of Speech Tagging?
A: Identifies roles of words in a sentence.

Describe Sequence-to-Sequence Models.

Q: Explain Self-Attention.
A: Self Attention: Each sequence of hidden states attends to itself
- is asymmetric
- encodings of wards in \(S\) can be calculted simultaneously

Q: What is the naive realization of self-attention and what issues arise with it?
A: Naive realization: set \(\mathbf{a}=a_{t,t'}\) as a dot product of vectors
Issue: states may be biased to self-attending

Q: What does a transformer do? Name the components.
A: Project input into three different representations:
- query vector \(\mathbf{q_t}\): the one being attended from
- key vector \(\mathbf{k_t}\): the one being attended to
- value vector \(\mathbf{v_t}\): the context being generated

Q: Define the key formula for attention.
A: \[\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V\]

Q: Explain the transformer layer.
A: 1. apply self-attention first, using hidden vectors from previous layer
2. feed through FF layers
3. add two residual connections (avoid vanishing gradients)

Q: What are Generative Adversarial Networks?
A: Generative Adversarial Netwok (GAN) is a pair of networks:
1. generator maps values from \(\mathbf{z}\) to \(\mathbf{x}\), to produce samples from \(\mathbf{P_w(x,z)}\)
2. discriminator classifies \(\mathbf{x}\) as real (drawn from training set) or fake (created from generator
GANs can create new data, which resembles existing data (e.g. deepfake).