Learning and Neural Computation

The course is intended for third-year and Master students. The first part of the course will deal with classical topics in computational learning theory and neural network learning. The second part will be geared more towards applications. In this part, students will apply the acquired methods on some Seismic Data Classification and few other real-world problems. Part of the course will therefore be devoted to topics related to detection in time-series and acoustic signals. This will be done by Dr. Manfred Joswig who is visiting this year and is a world expert on Seismic Data Analysis.

Suggested Reading

• Main Book:  C. M. Bishop (1995) Neural Networks for Pattern Recognition. Oxford University Press.
• N. Intrator (1998) Last Year's Course Outline and Lecture Notes
• M. Orr (1996) Introduction to Radial Basis Functions
• Radial Basis Function (Ph.D. Thesis, 1988)
• Shannon's classic 1948 paper: A Mathematical Theory of Communication
• Entropy in Information and Coding Theory
• Learning and Approximation Theory, Course given by T. Poggio & F. Girosi
• E. Wegman (!996) Data Mining and Statistics
• Neural Smithing Supervised Learning in Feedforward Artificial Neural NetworksMIT Press.
• Pao, Y. H. (1989) Adaptive Pattern Recognition and Neural Networks Addison-Wesley
• Naftali Tishby (1999) Lecture Notes (in Hebrew) (Some overlap)
• Duda and Hart (1973) Pattern Classification and Scene Analysis
• D. Ballard (1996) Neural Computation slides
• T. Mitchell (1997) Machine Learning   Slides
• T. Mitchell (1998) Machine Learning Slides
• B. J. Krose and P. van der Smagt (1993) An introduction to Neural Networks
• B. Ripley (1996) Pattern Recognition and Neural Networks
• V. Honavar (1996) Lecture Notes
• D. A. Stacey (1999) Lecture Notes   Morrell (non parameteric)  
• B. W. Silverman (1986) Density Estimation
• Silvey (1975) Statistical Inference
• Some material about MATLAB
  • Software (Orr)
  • WaveLab  PC Version  Unix Version  Ref Manual  Papers 
  • NETLAB - Radial Basis Functions tool-box from Aston University
  • Matlab Documentation
  • Matlab Primer  FAQ 
  • Reference Manual 
  • Running matlab from batch 
  • Please check the demos in matlab, try to use matlab5 and look also into the optimization and neural network tool-boxes.

Course Outline  

• Some background and notation from Statistics   
  
  Calma1  Calma2  Faces  Mines 

  • Curse of Dimensionality and Over-Fitting
  • Estimation via MSE and Maximum Likelihood
  • Linear regression: Numerical estimation problems
  • Bayes Theorem: Prior & Posterior probabilities.
  • Class Boundaries

  Topics from Non-paramteric statistics

  • Linear discrimination
  • The plug-in method, Bias and Variance of an estimator
  • Kernel estimation and Parzen Windows   (notes)
  • Density Estimation: Naive estimator, Kernel and variable kernel estimator, K-nearest neighbor estimation.
  • Likelihood ratios of nested models
  • Confidence Intervals

  Learning goals: Introduction to the Entropy

  Additional material:

  • T. Cover & J. Thomas (1991) Elements of Information Theory. Wiley publishers.
  • E. T. Jaynes (1999) Probability theory
  • D.J. MacKay (1998) Information Theory  Chapter 1 (Local)

  • Maximization of number of possible configurations
  • Minimization of averaged bit length
  • Motivation from learning and model estimation

  • Error functions
  • Parameter Optimization
  • Maximum likelihood
  • Infomax

  Single Layer Perceptrons 

  • Probabilistic interpretation of Perceptron outputs
  • Perceptron learning rule

  Multi-layer perceptrons

  Introduction to Seismology

  Preprocessing

  • Feature extraction
  • Some properties of Principal components
  • General properties of other orthonormal bases

  Information theory, Projection pursuit and neuronal coding

  • Feature extraction by minimization of entropy
  • Mutual information from MDL perspective
  • Search for non-Gaussian distributions
  • Skewness, Kurtosis, BCM rule
  • Some approximations to the entropy
  • Independent components analysis

  Model complexity via MDL  Introduction  (Hinton's paper)

  • Short discussion about matlab (Project presentation)

  Radial Basis Functions  

  • Description of the EM algorithm and its application to RBF.
  • ORR's code

  Visualization software

  • Linear Multi-Dimensional scaling
  • Non-linear MDS via Neural Networks.
  • Parallel coordinates
  • Hard and soft clustering
  • Mahalanobis clustering

  Prediction confidence

  • Using reconstruction
  • Using variance

  The bias/variance problem & ensemble of experts

  Noise Injection Two-spiral problem

  Exhaustive training

  Combination of Experts based on maximal entropy considerations.

  Bagging, Arcing

  Self Organizing maps and Recurrent networks

  Kohonen LVQ and SOM

  Recurrent Boltzman machines

  Network Ensembles

  Dichotomy between models based on their Entropy minimization or maximization.

  Various ways to measure deviation from maximal entrop .

  Variance error reduction via Ensemble of experts , LVQ

  Super and Sub-Gaussian distributions.

  Examples of Kurtosis, Skewness or BCM optimizations.

Description of the final   

1. Principal components
2. Dimensionality reduction via entropy constraints
3. K-means Clustering
4. Your choice.

1. Reconstruction constraint
2. Kurtosis constraint
3. BCM constraint
4. Minimal entropy constraint
5. Maximal entropy constraint
6. Mixture of Gaussians constraint
7. Your choice

1. Orr's Radial Basis Function code.
2. Bishop's Radial Basis Function code.
3. Neural networks toolbox back-prop code with Levenberg-Marquardt training.
4. Neural networks toolbox RBF code.
5. Your choice.

1. Variance between different experts weighted by their past performance.
2. Weighted reconstruction error.
3. Your choice.

Description of the data, and past projects (from which code can be taken) can be found here.

You will need to supply three functions:

1. [preproc, netarch] = findarch(train_data, train_label, options)
   This function finds an optimal preprocessing and a set of architectures with their corresponding parameters which are best for the given training data. This function calls internally to findpreproc

2. [processed, preproc] = findpreproc(train_data, train_label, procoptsions, preproc)
   This function has two tasks. First it finds an optimal preprocessing and second, it generates the pre-processed data. If preproc is supplied, the program runs the data through the preprocessing methodology which should be completely kept in preproc.

3. [label_pred, confidence] = netpred(data, preproc, netarch, options)
   This function performs the preprocessing on the given data based on the preprocessing scheme that was chosen by findarch and is stored in preproc and then prediction of the class labels based on the architectures that were found by findarch and are stored in netarch. Note that if a collection of several experts is found by findarch, then the algorithm for fusing the experts should also be stored in netarch to be used by netpred.
   You should suggest a way to calculate the confidence and justify in your written report why this confidence method is useful.

Each function should print once on the screen the names of the students and a short description of the methods that are used, the type of preprocessing and the architectures that are found. In the prediction code, again, the names should be displayed as well as the type of expert fusion and the method to calculate the confidence. There should also be a clear description of the possible options to the functions and default values (which were found optimal) in case options are not chosen.
All other functions should be internal to the above three functions.

Note:   You are allowed to use code written by last year's projects or code taken from the web as long as you clearly state it at the beggining of the code, insert it into your own code, make sure that you understand what the code is doing and the code has good documentation.

Project submission

There should be an HTML page describing the project and having a link to the functions. It should include the following:

1. Clear description of the type of algorithms that were used and the way optimal architectures were chosen.
2. Clear justification based on what was learned in the course for your decisions. (This can include graphs.)
3. Clear description and justification of how the confidence is calculated.

Project presentation is scheduled to Tuesday February 8, 2000 at 9am.

Specific-group tasks

1. Project:  Akavia  Gindi  Erez 
   
   • Preprocessing:   clustering and your choice.
   • Training constraints:   Kurtosis and your choice.
   • Architecture and code:   RBF with Orr and Back-prop with LM.
   • Confidence:   Reconstruction and your choice.
   • Data:   Sonl and Sono.
2. Project:  Lifchitz  Weisman  Arnold 
   
   • Preprocessing:   Minimal entropy and clustering.
   • Training constraints:   BCM and Kurtosis.
   • Architecture and code:   Back-prop with LM.
   • Confidence:   Variance and your choice.
   • Data:   Sono and wvlet.
3. Project   Hoffman  Stav 
   
   • Preprocessing:   Maximal entropy and clustering.
   • Training constraints:   Reconstruction and minimal entropy.
   • Architecture and code:   RBF with Orr and Bishop.
   • Confidence:   Variance and your choice.
   • Data:   Sonl and psd.
4. Project   Tobias  
   
   • Preprocessing:   Clustering and your choice.
   • Training constraints: Reconstruction and Kurtosis 
   • Architecture and code:   RBF with Bishop.
   • Confidence:   Reconstruction and your choice.
   • Data:   Sonl and wvlet
5. Project   Sidi,   Bogodlov
   
   • Preprocessing:   clustering and your choice.
   • Training constraints:   BCM and Kurtosis.
   • Architecture and code:   RBF with Orr.
   • Confidence:   Reconstruction and your choice.
   • Data:   Sonl and Sono.