Neural Network Parameter & Memory Calculator

Understanding the neural network parameter and memory calculator — concept and principle

Every connection in a fully connected (dense) neural network has a weight, and every neuron has a bias. This calculator counts those learnable parameters for a stack of dense layers and converts the count into memory: how much space the weights take at a chosen numeric precision, and a rough figure for training with the Adam optimiser. It helps you check whether a model fits on a device.

How it is calculated

• For each pair of consecutive layers: parameters = inputs × outputs + outputs (one bias per output neuron).
• Total parameters = the sum over all layer pairs.
• Weights memory (MB) = parameters × bytes per parameter ÷ 10⁶. FP32 uses 4 bytes, FP16/BF16 2, INT8 1, INT4 0.5.
• Training memory with Adam ≈ parameters × 16 bytes ÷ 10⁶.

Layer sizes are numbers of neurons, listed from the input layer to the output layer.

The principle

A dense layer computes y = Wx + b. The weight matrix W has one entry for every input-output pair, and the bias vector b has one entry per output, which gives in × out + out. Training needs more than the weights: one common accounting is 4 bytes for FP32 weights, 4 for their gradients and 8 for Adam's two running averages (first and second moments), or 16 bytes per parameter. Mixed-precision training rearranges these pieces but also comes to about 16 bytes per parameter. Activations, which depend on batch size, are extra and are not included.

Worked example

1. Layers 784 → 256 → 128 → 10 (for example, a 28 × 28 image classifier with 10 classes).
2. 784 × 256 + 256 = 200,960; 256 × 128 + 128 = 32,896; 128 × 10 + 10 = 1,290.
3. Total = 235,146 parameters.
4. FP32 weights: 235,146 × 4 = 940,584 bytes = 0.940584 MB.
5. Adam training: 235,146 × 16 = 3,762,336 bytes = 3.762336 MB.

How to read the result

Notice that the first layer holds about 85% of the parameters: large input layers dominate the count. Halving the precision halves the weights memory, which is why FP16 and INT8 are used for deployment. MB here means 10⁶ bytes; operating systems often show MiB (2²⁰ bytes), about 4.9% larger.

Common mistakes

• Forgetting the bias terms, or adding a bias to the input layer.
• Applying the dense formula to convolutional layers, whose parameters are kernel height × kernel width × input channels × output channels + output channels.

Frequently asked questions

Do batch-normalisation layers add parameters?

Yes, two learnable values per channel (scale and shift), plus two stored running statistics.

Does quantisation to INT8 change the parameter count?

No. It changes the bytes per parameter, not the number of parameters.

Why does SGD use less memory than Adam?

Plain SGD keeps no moment estimates; SGD with momentum keeps one, Adam keeps two.

Sources: I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016 · D. P. Kingma and J. Ba, Adam: A Method for Stochastic Optimization, ICLR 2015 · S. Rajbhandari et al., ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, SC20, 2020