Would I spend money to remove existing religious writings on public buildings? No. It doesn't bother me that much.
But neither would I want my tax money spent on putting religious writings on such buildings in the future.
I am not anti-religious- you can build all the churches you like on private land- but it's not the business of government.