Let me know if you are coming in CT, welcome! It's so rare here, only met one other local!
I am wearing my PAM in town everywhere. Few people know Panerai and in general petty crime is really low in the city.
Now with two exceptions: When walking at night, just avoid being drunk on Long Street with your PAM or with your iPhone + if you take the train (but you won't) and that's really it.
Trust the locals, to some extent. We met many privileged South Africans who were overly scared and stressed about security. They live behind high fences and do not mix up with other social groups, making them quite distant and disconnected to the real life. Despite the huge inequality, it's still very quiet.
My hidden gem tip: Go during the week (weekend a bit too crowded) to The Gin Bar, hidden behind Honest Chocolate, 64A Wale Street. And visit the art galleries in Woodstock (Stevenson, Goodman) then go to the Biscuit Mill on a Sturday for brunch.