If you are using materials obtained from game animals or other "unprocessed" (meaning those that have not been boiled or otherwise treated for insects) sources, be sure to keep them separate from your good materials. Otherwise you may find your "rare and unusual" to be more unusual than usual...
A little moth crystal in your materials cache will solve a world of infestation problems. Just be sure to wash your hands after handling. Also, this problem can extend to framed fly collections, a friend of mine recently discovered some of his framed salmon flies to be in tatters due to bugs that either hatched in the frame or somehow found their way inside (more likely the former).