Meditation is not about just being with your breath. It’s about all of the things that pull you away from just being with your breath.
One of the overarching intentions of a meditation practice is cultivating a mind that can be at peace despite inner and outer chaos. So we give ourselves something to settle our attention on (in this case, the breath) and we quickly see that the mind has other plans, and will create all of these reasons why we should do something else, feel a different way, find a different room to meditate, etc. THIS is what we are working with in the meditation. All of these things that create distress, we just notice them arising and passing, and return back to the breath.
Eventually, the mind finds a groundedness and peace within the chaos. And THIS becomes the power of a meditation practice.