House Sparrows haven't hurt my garden very much but they do monopolize my bird feeders... as you say, even eating seed they aren't supposed to like. "My" House Sparrows have figured out how to eat from both my thistle feeder and my peanut feeder. They definitely intimidate my Goldfinches and House Finches. Cardinals seem big enough to handle them, one on one, but they tend to mob the feeders at times and then only the Grackles or a visit hawk can drive them away.
The problem is several of my idiot neighbors "decorate" the sides of their houses with multiple bird houses. These ornate houses are not designed for any particular species so the holes tend to be large and so we end up with sparrow colonies. As they fill up those houses, the House Sparrows then start harassing other birds when they can't even fit into their houses. I've had sparrows try to run off my wrens, when I put up special wren-sized houses with little holes that the sparrows can even enter!
They are a pain. The one good thing they seem to do, however, is fan into my garden in summer and eat bugs. I suspect they help me out by eating aphids on my honeysuckles too, so they are not all bad. I don't have the heart to kill them but I wouldn't mind if they went away!
USDA zone 7a
Heat zone 6
Sunset zone 32