The point here is you aren't elite. That's ok. Stop trying to be elite, unless you're willing to quit you're job and rearrange your life to mimic what the elite do. Let's be honest, you're not willing to do it and even if you did you're still not going to be elite. You'd be starting this journey about 20 years too late.
Yes you're fit..but are you healthy?
I often hear stories about bodybuilders going to the doctor and being told that they are obese. The bodybuilders then go onto their social media sites and blast the doctors for only being able to read BMI charts and not knowing what it takes to achieve elite physical fitness. Then because bodybuilders tend to have lots of bodybuilder friends and followers, the posts get tons of likes and retweets. Is this fair to the doctors?
First off let say that if a doctor is solely relying on BMI charts (or any one "metric" for that matter) then they probably deserve whatever social media skewering that they get. That being said, it's not a doctor's job to judge the size and symmetry of your gluteus maximuses (or is the plural maximi, I don't know). The doc's job is to measure your health status.
This is where it can get confusing: "healthy" and "fit" are often not synonymous. Doctors go to school (for a long time) to learn what "healthy" is. To assist with this baselines have been set: BMI, blood pressure, cholesterol levels and hormone levels to name just a few. Again, hopefully doctors aren't latching on to any one of these and drawing conclusions. Hopefully they're taking a comprehensive approach before giving any medical advice.
I would argue "fitness" can't be a part of this comprehensive approach. Why not? I'll answer that with another question: what would you consider elite fitness? If you're one of the bodybuilders you might say massive symmetrical glutes. On the other hand, if you're a distance runner you might say a sub 3-hour marathon is the gold standard of fitness.
Dan John has the best definition of fitness that I've come across. He says (paraphrasing) fitness is the ability to accomplish a given task. According to that definition both the bodybuilder and the distance runner would have given correct answers based on the events they compete in. This is why if you ask ten different athletes from ten different sports what they consider fit, you could possibly get ten different answers.
I'm sure there would be some overlap but ultimately the answer will depend on the objective of the sport or the position that is played. This ambiguity is why a doctor can't and shouldn't take your "fitness" level at face value. A doctor should however consider your "fitness" along with all objective health measurements that are taking.