As far as I understood from what I read about it, the worst thing are these ligaments, that should function as a perfectly fitting and duely sized natural bra, fixing the breasts to the rib cage and pectoral muscles. They atrophy when not in use. See http://www.breastnotes.com/bc/bc.shtml in the list of references I dropped here earlier. There is a Japanese study that measured the extra sagging caused by wearing a bra, mentioned by Michael G. White. Even the CEO of Tryumph admits that bras do not prevent sagging. They just hide it and replace it by some fake appearance.
Why are that many women ashamed to be mature?
I wonder whether these muscles and ligaments can recover once they have started to atrophy...