Okay, so -
1) Harry's mum is muggle-born, right? So I'm assuming that he's either being rude to her by calling her a muggle or talking about her heritage. ie: Her parents were a muggle and a fool. Just my guess, but I am quite sure about the next part which is -
2) Tom Riddle (AKA Voldemort) killed his dad and grandparents. His dad's name was also called Tom Riddle, remember?
Does the above make any sense? If so, well, glad I could be of help. If not, erm...sorry I'm a bit confusing.
Good luck!