Extreme Learning Machine for digits recognition

General idea behind Extreme Learning Machines (ELMs) is transforming features into higher-dimensional space and using linear model to solve problem. More detailed information about ELMs you can find here.

Although Extreme Learning Machines are far behind Deep Learning methods (at least according to popular CV benchmarks) they have substantial advantage - ability to learn fast caused by removing iterative training. Also as you can read on MLWave principles on which ELM bases can be very powerful.

Here I'll show you how to start to work with ELMs in Python. We'll write a simple digit recognizer.

Data set

For an experiment we will use MNIST data set prepared for Theano's Deep Learning tutorial - as it is in Python-friendly format. Here is the link: http://www.iro.umontreal.ca/~lisa/deep/data/mnist/mnist.pkl.gz

In the archive you can find mnist.pkl file which contains serialized tuple.

(train_obj, validation_obj, test_obj)

Each of these objects is also a tuple in following format:

(inputs, labels)

Where inputs object is a matrix (2d-numpy's array) with every sample in separated row and the labels object is a list.

Code

We will use ELM's implementation already available on GitHub. So you need to clone the repository: https://github.com/dclambert/Python-ELM

Then you can try following code which should achieve 97.33% accuracy on validation set and 97.02% on test set. On my machine learning and evaluation took around 11 minutes. Also if you have less than 8GB of memory it's a good idea to decrease number of hidden units for ELM.

import cPickle
import numpy as np
from elm import ELMClassifier
from sklearn import linear_model


def load_mnist(path='../Data/mnist.pkl'):
    with open(path, 'rb') as f:
        return cPickle.load(f)


def get_datasets(data):
    _train_x, _train_y = data[0][0], np.array(data[0][1]).reshape(len(data[0][1]), 1)
    _val_x, _val_y = data[1][0], np.array(data[1][1]).reshape(len(data[1][1]), 1)
    _test_x, _test_y = data[2][0], np.array(data[2][1]).reshape(len(data[2][1]), 1)

    return _train_x, _train_y, _val_x, _val_y, _test_x, _test_y


if __name__ == '__main__':
    # Load data sets
    train_x, train_y, val_x, val_y, test_x, test_y = get_datasets(load_mnist())
    # Build ELM
    cls = ELMClassifier(n_hidden=7000,
                        alpha=0.93,
                        activation_func='multiquadric',
                        regressor=linear_model.Ridge(),
                        random_state=21398023)
    cls.fit(train_x, train_y)
    # Evaluate model
    print 'Validation error:', cls.score(val_x, val_y)
    print 'Test error:', cls.score(test_x, test_y)

Code is available on GitHub (link). To improve the solution you can use gzip module to read pkl file directly from archive.