I think you’re underestimating how small we are and just how big and empty space is. Probability of something significant hitting us is extremely low especially when you think of timescale also.

Here’s a map of the Solar System down to a scale https://joshworth.com/dev/pixelspace/pixelspace_solarsystem.html it’s ridiculous