I agree, and frankly if either Man or Sec had gotten enhancement drugs, then surely many others were getting it and should have performed just as well! I don't believe it in either case. Drugs may help a bit but I doubt they turn any given horse into a world-beater!
As far as their builds, Man was clearly of different type. Man was lanky and tight and short-coupled (more square comparing legs to body). Sec had a bruiser build and was longer-coupled (more rectangular).
This has nothing to do w/build per se, but their conformation was quite different too. In fact, I dare say Man's was very unique whereas Sec looks more like an ordinary horse (pretty, but avg). Man looked like he had no chest - the front legs seemed to be hooked to the very front of his chest - in fact they seem atypical in placement vs the withers. He also had a neck that naturally went quite vertical and held his head high and at a greater angle than most horses. Sec didn't; he tended to droop horizontal as do most horses. These features seem very unusual to me - I don't think I've seen any horses that look like Man on the front end. I'm no horse expert, but I've been interested in them long enough to know few have that strange conformation.