Humans are very lightly build and spinly creatures compared to other animals. That's because 1 we are apes and most apes need to be light enough to climb trees. Muscle also makes your movements more stiff and apes need to be flexible to swing through the trees. Also humans are specialized for efficiency and the lighter your body and especially your limbs are the less energy it takes to move them. Horses might have much higher top speeds than humans but they are nowhere near humans when it comes to endurance.
Also horses need to be physically strong enough to defend themselves. Humans don't need that because we have tools to off-set any gap in power.