How Can I Predict The Expected Value And The Variance Simultaneously With A Neural Network?

I'd like to use a neural network to predict a scalar value which is the sum of a function of the input values and a random value (I'm assuming gaussian distribution) whose variance

Solution 1:

You can use dropout for that. With a dropout layer you can make several different predictions based on different settings of which nodes dropped out. Then you can simply count the outcomes and interpret the result as a measure for uncertainty.

For details, read:

Gal, Yarin, and Zoubin Ghahramani. "Dropout as a bayesian approximation: Representing model uncertainty in deep learning." international conference on machine learning. 2016.

Solution 2:

When using dropout to estimate the uncertainty (or any other stochastic regularization method), make sure to also checkout our recent work on providing a sampling-free approximation of Monte-Carlo dropout.

https://arxiv.org/pdf/1908.00598.pdf

We essentially follow ur idea. Treat the activations as random variables and then propagate mean and variance using error propagation to the output layer. Consequently, we obtain two outputs - the mean and the variance.

Solution 3:

Since I've found nothing simple to implement, I wrote something myself, that models that explicitly: here is a custom loss function that tries to predict mean and variance. It seems to work but I'm not quite sure how well that works out in practice, and I'd appreciate feedback. This is my loss function:

defmeanAndVariance(y_true: tf.Tensor , y_pred: tf.Tensor) -> tf.Tensor :
  """Loss function that has the values of the last axis in y_true 
  approximate the mean and variance of each value in the last axis of y_pred."""
  y_pred = tf.convert_to_tensor(y_pred)
  y_true = math_ops.cast(y_true, y_pred.dtype)
  mean = y_pred[..., 0::2]
  variance = y_pred[..., 1::2]
  res = K.square(mean - y_true) + K.square(variance - K.square(mean - y_true))
  return K.mean(res, axis=-1)

The output dimension is twice the label dimension - mean and variance of each value in the label. The loss function consists of two parts: a mean squared error that has the mean approximate the mean of the label value, and the variance that approximates the difference of the value from the predicted mean.