Curriculum Learning
I want to go through the Wikipedia series on Machine Learning and Data mining. Data mining is the process of extracting and discovering patterns in large data sets involving methods at the intersection of machine learning, statistics, and database systems.
References
Notes
Curriculum learning is a technique is machine learning in which a model is trained on examples of increasing difficulty, where the definition ofdifficultyay be provided externally or discovered automatically as part of the training process. This is intended to attain good performance more quickly, or to converge to a better local optimum if the global optimum is not found.
More generally, curriculum learning is the technique of successively increasing the difficulty of examples in the training set that is presented to a model over multiple training iterations. This can produce better results than exposing the model to the full training set immediately under some circumstances; most typically, when the model is able to learn general principles from easier examples, and then gradually incorporate more complex and nuanced information as harder examples are introduced, such as edge cases. This has been shown to work in many domains, most likely as a form of regularization.
There are several major variations in how the technique is applied:
- The concept of
difficulty
must be defined - Difficulty can be increased steadily or in distinct epochs, and in a deterministic schedule or according to a probability distribution
- Applications must also decide the schedule for increasing the difficulty. Simple approaches may use a fixed schedule, such as training on easy examples for half of the available iterations and then all examples on the second half
Since curriculum learning only concerns the selection and ordering of training data, it can be combined with man other techniques in machine learning. The success of the method assumes that a model trained for an easier version of the problem can generalize to harder versions, so it can be seen as a form of transfer learning. It is frequently combined with reinforcement learning.
The term curriculum learning
was introduced by Yoshua Bengio et al in 2009, with reference to the psychological technique of shaping in animals and structured education for humans: beginning with the simplest concepts and building on them.
Comments
You have to be logged in to add a comment
User Comments
There are currently no comments for this article.