You are right, thank you

Note that the minimal loss is reached when the parameters make neural network predictions the closest to the real labels

Before that, the gradient is non zero generally (except for an very very unlucky local minimum)

You could see the case of the linear regression with least square error as loss to understand better the underlying optimization problem (in one dimension, it is a square function to minimize, so no local minimum)

If all weight are the same (assume 0 to be simple) then the output of the function/neural network is far from the objective/label

The gradient is therefore non zero

And finally the parameters are updated : theta = theta + learning_rate*grad_theta(loss)

And when the parameters are updated the loss is changed

Usually, the parameters are randomly choosen

If the learning rate is zero, the update rule of the params makes the params unchanged

The data balancing does not change the loss (it only changes the overfitting) and same for the regularization strength too low

Bad initialization is rarely a problem (with a lack of chance you could get a local minimum directly but rare event)

3 is the only possible solution