Calculate The Unbiased Standard Deviation Of All Elements In A PyTorch Tensor

First, we import the Python math module for the square root calculation we will later use.

import math

Then we import PyTorch.

import torch

Next, we print the PyTorch version we are using.

print(torch.__version__)

We are using PyTorch 0.3.1.post2.

To start, let’s manually create an example tensor using the PyTorch FloatTensor operation.

pt_tensor_ex = torch.FloatTensor(
[
  [
    [36, 36, 36],
    [36, 36, 36],
    [36, 36, 36]
  ]
  ,
  [
    [72, 72, 72],
    [72, 72, 72],
    [72, 72, 72]
  ]
])

So we do torch.FloatTensor, we pass in our data structure, and we’re assigning it to the Python variable pt_tensor_ex.

We construct the tensor in this way so that it’ll be straightforward to calculate the unbiased standard deviation of the tensor.

Let’s print the pt_tensor_ex Python variable to see what we have.

print(pt_tensor_ex)

We see that it’s a torch.FloatTensor of size 2x3x3, and we see all of the numbers – a matrix of all 36s and a matrix of all 72s.

Now that we have our tensor, let’s calculate the unbiased standard deviation of all elements in a PyTorch tensor by using the torch.std operation.

pt_unbiased_std_ex = torch.std(pt_tensor_ex, unbiased=True)

So we have torch.std.

We pass in our tensor.

We want to make sure that we are using unbiased, so we say unbiased=True.

Then we’re going to assign the value that is returned to the Python variable pt_unbiased_std_ex.

We can then print the pt_unbiased_std_ex Python variable to see what we get.

print(pt_unbiased_std_ex)

The result is 18.52 and a lot more decimal places.

All right, now that we have our result, let’s double check it to make sure that we’re comfortable with what was calculated.

First, let’s reacquaint ourselves with how we calculate the unbiased standard deviation.

So here’s the mathematical formula.

So we’re going to calculate the mean of all the elements in the tensor and then for each element, we’re going to subtract the mean from it, then we’re going to take the square, then we’re going to sum over all of the elements.

Then finally, we’re going to divide the result by n-1.

This is called Bessel's Correction.

Then we take the square root of this final result, and that will be our unbiased estimation of the standard deviation.

Note that you often use the unbiased standard deviation because you’ll be looking at a sample of the data and not the whole population.

So you’ll need to correct for that.

The correction that we use for the unbiased standard deviation calculation is the Bessel’s Correction.

To manually check it, let’s start by calculating the mean of all the elements in the PyTorch tensor.

pt_tensor_mean_ex = torch.mean(pt_tensor_ex)

So we use torch.mean, and we pass in our pt_tensor_ex, and we assign it to the Python variable pt_tensor_mean_ex.

Let’s print the pt_tensor_mean_ex Python variable to see the value.

print(pt_tensor_mean_ex)

We see the mean of all the elements in our example tensor is 54.

With the mean of the tensor, we can subtract that number from all of the elements in the tensor.

pt_tensor_transformed_ex = pt_tensor_ex - pt_tensor_mean_ex

So we say pt_tensor_ex – pt_tensor_mean_ex, and that will mean that every single element will have 54 subtracted from it.

That resulting tensor is going to be assigned to the pt_tensor_transformed_ex Python variable.

When we print the pt_tensor_transformed_ex variable, we get all negative 18s and all positive 18s.

print(pt_tensor_transformed_ex)

That makes sense because 36 minus 54 is negative 18. 72 minus 54 is positive 18.

Now that we have this PyTorch tensor, we’re going to square it.

pt_tensor_squared_transformed_ex = torch.pow(pt_tensor_transformed_ex, 2)

So we’re going to use PyTorch’s pow operation.

So we pass in our pt_tensor_transformed_ex.

And we’re going to raise it to the power of 2, so each element is going to be squared.

And we’re going to assign the resulting tensor to pt_tensor_squared_transformed_ex.

We can then print the result.

print(pt_tensor_squared_transformed_ex)

We see that our tensor is still a FloatTensor of size 2x3x3, and all of the numbers are now 324, which is what we expect.

Negative 18 squared is 324.

Positive 18 squared is 324.

So all of the numbers are 324.

Now that each number has been demeaned and squared, let’s sum all of the elements in the tensor.

pt_tensor_squared_transformed_sum_ex = torch.sum(pt_tensor_squared_transformed_ex)

So we use torch.sum operation.

We pass in our pt_tensor_squared_transformed_ex, and we assign it to the Python variable pt_tensor_squared_transformed_sum_ex.

When we print this result, we see the number 5832.

print(pt_tensor_squared_transformed_sum_ex)

So now that we have this, we can then apply Bessel’s Correction and we’re going to be dividing by n-1.

Remember that n is 18 because we have a 2x3x3 tensor.

So 2*3 = 6 * 3 = 18. So 18-1 = 17.

So the calculation that we do is 5832/17.

5832 / 17

We get 343.0588 and so on.

Then the last thing we do is we take the square root of this expression.

math.sqrt(5832 / 17)

So we use the Python math module dot square root, and we divide 5832/17 and we get 18.52 and a bunch of decimals.

We compare this number to the unbiased standard deviation calculation result we computed earlier:

print(pt_unbiased_std_ex)

And we see that they are the same.

Perfect - we were able to calculate the unbiased standard deviation of all the elements in a PyTorch tensor by using the torch.std operation.