In school I felt the same. I was very thin, but never could do the things others could. They spent their time watching TV, I spent my time running around, working in garden in summers, swimming and playing active games, but for whatever reason I was still slowest and weakest. I didn't had any particular illness, so I was just called lazy, even though I was trying my best. So I hated sports like crazy. I thought it is not fair I work out but they don't, and if I am still weaker, they get better grades.
But after the school I started to find my ways to be active. I am the same - I can't do push ups, and I always sweat like crazy, and get very hot, so I need temperate temperature where I am working out. I don't know why I am like that, but it doesn't mean people like us can't be active. We just can't be competitive with others, only with ourselves. There are ways to better our health in a moderate and still very good way.