Generalization

Generalization: Ability to Apply what was learned during Training to new (Test) Data

Reasons for bad generalization

• Overfitting/Overtraining (trained too long)
• Too little training material
• Too many Parameters (weights) or inappropriate network architecture error

Prevent Overfitting

• The obviously best approach: Collect More Data! 💪
• If Data is Limited
  • Simplest Method for Best Generalization: Early Stopping
  • Optimize Parameters/Arcitecture
    • Architectural Learning
    • Choose Best Architecture by Repeated Experimentation on Cross Validation Set
    • Reduce Architecture Starting from Large
    • Grow Architecture Starting from Small

Destructive Methods

Reduce Complexity of Network through Regularization

• Weight Decay

• Weight Elimination

• Optimal Brain Damage

• Optimal Brain Surgeon

Optimal Brain Damage

• 💡Idea: Certain connections are removed from the network to reduce complexity and to avoide overfitting
• Remove those connections that have the least effect on the Error (MSE, ..), i.e. are the least important.
  • But this is time consuming (difficult) 🤪

Constructive Methods

Iteratively Increasing/Growing a Network (construktive) starting from a very small one

• Cascade Correlation
• Meiosis Netzwerke
• ASO (Automativ Structure Optimization)

Cascade Correlation

• Adding a hidden unit

  • Input connections from all input units and from all already existing hidden units
  • First only these connections are adapted
  • Maximize the correlation between the activation of the candidate units and the residual error of the net

• Not necessary to determine the number of hidden units empirically

• Can produce deep networks without dramatic slowdown (bottom up, constructive learning)

• At each point only one layer of connections is trained

• Learning is fast

• Learning is incremental

Dropout

• Popular and very effective method for generalization

• 💡Idea

  • Randomly drop out (zero) hidden units and input features during training
  • Prevents feature co-adaptation

• Illustration

  

• Dropout training & test