I have had a 372 for 3-4 years and often thought of trading for a 587 or 673 when they were newer releases. If I could make the switch for free I probably would, but that's not really possible and with time the desire to trade has kind of faded. At the end of the day you're talking about the MM dial and pencil hands for x thousand more over a 372. Those features are still more desirable to me, but likely just because the grass is always greener on the other side, not because they're really better.
The real struggle I've been facing is the general idea that the 47mm in that case might just be too big for my wrist, which would be the same problem for all of them. This relates to your question in that the Rolex is significantly more practical to wear for most and I'm not sure where you fit into that. If the sizing is not a problem and you have the itch, go for it. You'll never struggle to find another Ceramic LV if you want another down the road. Unless the one you have is sentimental for some reason, move onto the next and see how you like it.
In fairness, I've never owned any Rolex sports model although I've lusted after many and even advised friends on which to buy. I'm not the best person to answer your question on pros/cons between them. As I see it, the PAMs would be more unique and you might have more passion for them because of the brand history etc. The Rolex will probably be a better investment and is more practical for all around daily enjoyment.