Your mother's concerns certainly do sound valid. No one should have to spend their twilight years feeling stressed out and threatened. The person who need to be helping out more is your sister. Is there anyway to encourage her involvement by putting some pressure on her? For example, your family could return the responsibility of raising her son unless she agrees to become more actively engaged in his parenting. I also would be curious if there's a relationship between your nephew's anger and the fact that his parents seem to want nothing to do with him. How could that not impact his self-esteem and emotional development?
In terms of what can be done to help him, I'd like to suggest you to consider encouraging your meother to get him back ino counseling. If finances are the issue, you may want to find out what help might be provided by your nephew's school.