Title: EAI / 07_neural_networks

Describe a neuron.

Q: What is the perceptron learning rule?
A: \(w_i \leftarrow w_i + \alpha \cdot Err \cdot g'(in) \cdot x_i\)
where \(\alpha\) is the learning rate

Q: Which boolean functions can be expressed by a perceptron?
A: - AND
- OR
- NOT

Describe 3 network structures.

Q: Describe Feed-Forward Networks.
A: - Uni-directional
- Straight flow of information from input to output\(i\)\(i+1\)\(i-1\)
- Layered networks: node in layer \(i\) is directly connected only to nodes in layer \(i+1\) (and layer \(i-1\))
- Types: single layer perceptron, multi-layer perceptrons
- NO internal state

Q: How should you define the layers of your neural network?
A: deep & narrow

Q: What can you do to avoid slow convergence and local maxima in a neural network?
A: Stochastic gradient descent with minibatches

Q: What is batch normalization?
A: Batch normalization rescales values at internal layer per minibatch

Q: What should you consider to avoid exploding and vanishing gradients?
A: - batch normalization
- non-exponential activation functions
- LSTM for RNN
- residual networks

Explain Residual Networks.

Describe the back-propagation algorithm.

Q: Describe the principle of Reinforcement Learning.
A: Maximize the expected sum of rewards.

Q: Name the pros and cons of deep learning.
A: + easier to develop than statistical methods
+ capability to learn and improve
+ usable for "unstructured" and "difficult" input (e.g. images)
+ inherent parallelism
+ fault tolerance
- choice of parameters even more difficult
- training material: volume and quality (bias in data)
- resulting hypotheses cannto be understood easily
- knowledge implicit
- verification and behaviour prediction difficult (if not impossible)

Q: What do convuolutional neural networks focus on?
A: Adjacency (e.g. in images)

Q: What can recurrent neural networks handle what others can't?
A: input of non-fixed lengths

Q: What allows information to be kept over long time in neural networks?
A: Long-Short Term Memory

Q: Which neural networks are useful for translation?
A: Sequence-to-Sequence models