Gradient-based manipulation of nonparametric entropy estimates.
IEEE Trans Neural Netw
; 15(4): 828-37, 2004 Jul.
Article
in En
| MEDLINE
| ID: mdl-15461076
ABSTRACT
This paper derives a family of differential learning rules that optimize the Shannon entropy at the output of an adaptive system via kernel density estimation. In contrast to parametric formulations of entropy, this nonparametric approach assumes no particular functional form of the output density. We address problems associated with quantized data and finite sample size, and implement efficient maximum likelihood techniques for optimizing the regularizer. We also develop a normalized entropy estimate that is invariant with respect to affine transformations, facilitating optimization of the shape, rather than the scale, of the output density. Kernel density estimates are smooth and differentiable; this makes the derived entropy estimates amenable to manipulation by gradient descent. The resulting weight updates are surprisingly simple and efficient learning rules that operate on pairs of input samples. They can be tuned for data-limited or memory-limited situations, or modified to give a fully online implementation.
Search on Google
Collection:
01-internacional
Database:
MEDLINE
Main subject:
Pattern Recognition, Automated
/
Artificial Intelligence
/
Models, Statistical
/
Information Storage and Retrieval
/
Decision Support Techniques
/
Neural Networks, Computer
/
Information Theory
Type of study:
Evaluation_studies
/
Prognostic_studies
/
Risk_factors_studies
Limits:
Humans
Language:
En
Journal:
IEEE Trans Neural Netw
Journal subject:
INFORMATICA MEDICA
Year:
2004
Document type:
Article