Expressive power of ReLU and step networks under floating-point operations.

The study of the expressive power of neural networks has investigated the fundamental limits of neural networks. Most existing results assume real-valued inputs and parameters as well as exact operations during the evaluation of neural networks. However, neural networks are typically executed on computers that can only represent a tiny subset of the reals and apply inexact operations, i.e., most existing results do not apply to neural networks used in practice. In this work, we analyze the expressive power of neural networks under a more realistic setup: when we use floating-point numbers and operations as in practice. Our first set of results assumes floating-point operations where the significand of a float is represented by finite bits but its exponent can take any integer value. Under this setup, we show that neural networks using a binary threshold unit or ReLU can memorize any finite input/output pairs and can approximate any continuous function within an arbitrary error. In particular, the number of parameters in our constructions for universal approximation and memorization coincides with that in classical results assuming exact mathematical operations. We also show similar results on memorization and universal approximation when floating-point operations use finite bits for both significand and exponent; these results are applicable to many popular floating-point formats such as those defined in the IEEE 754 standard (e.g., 32-bit single-precision format) and bfloat16.

Coding Locations Relative to One or Many Landmarks in Childhood.

Cognitive development studies how information processing in the brain changes over the course of development. A key part of this question is how information is represented and stored in memory. This study examined allocentric (world-based) spatial memory, an important cognitive tool for planning routes and interacting with the space around us. This is typically theorized to use multiple landmarks all at once whenever it operates. In contrast, here we show that allocentric spatial memory frequently operates over a limited spatial window, much less than the full proximal scene, for children between 3.5 and 8.5 years old. The use of multiple landmarks increases gradually with age. Participants were asked to point to a remembered target location after a change of view in immersive virtual reality. A k-fold cross-validation model-comparison selected a model where young children usually use the target location's vector to the single nearest landmark and rarely take advantage of the vectors to other nearby landmarks. The comparison models, which attempt to explain the errors as generic forms of noise rather than encoding to a single spatial cue, did not capture the distribution of responses as well. Parameter fits of this new single- versus multi-cue model are also easily interpretable and related to other variables of interest in development (age, executive function). Based on this, we theorize that spatial memory in humans develops through three advancing levels (but not strict stages): most likely to encode locations egocentrically (relative to the self), then allocentrically (relative to the world) but using only one landmark, and finally, most likely to encode locations relative to multiple parts of the scene.