I believe equestrian is actually an olympic sport.....so is Bobsledding, tobogganing, baseball....etc
I think some racing is more qualified as being a sports than others, but in the grand scheme of things they are ALL recognized as athlete. If you have ever driven a car in competition, you'll see its actually quite physical, requires a great degree of stamina and hand-eye coordination, which is much like any other traditional sports. In higher level of racing(I guess other than drag), drivers are required to keep in top physical condition in order to perform at their best, they need tremandous upper body strength to deal with the g-forces, they also need insane amount of neck muscle development to hold their head upright to deal with upward to 5 G(in F1's case) of their head+helmet weight in longitudinal and lateral direction, while being able to operate their vehicle.....
Put it another way, I remember a few years back Tiger Woods was asked who he thinks is a better athlete: Himself, Federer, or Michael Schumacher, each were dominating their respective sports at the time, Tiger Woods said Schumacher....
University of Toronto Formula SAE Alumni 2003-2007
Formula Student Championship 2003, 2005, 2006
www.fsae.utoronto.ca