Algorithms related to information theory.
@author: drusk
Calculates the entropy of a data set.
Entropy is the measure of impurity of the data set. For example, if all the samples have the same classification, the entropy will be 0.
Calculates the information gain of a feature in a data set.
The information gain of a feature is the expected reduction in entropy caused by knowing the value of that feature.