There is no "should" ... it's a hobby, you get to do it any way you want to. However, if you're looking for advice
on the matter
then I'll tell you the advantaged and disadvantages of both methods that I've found, and let you make up your own mind.
If you mix in paint with the groundwork material, you'll have the advantage that any chipping that may occur will not expose bright spots that detract from the diorama. The down side is that, I discovered the one or two times I tried it, you need to put in a fair amount of paint to make sure the ground doesn't end up a lot lighter than intended once it's dry.