"I'se A-Muggin'" originated with Stuff Smith at the Onyx Club on 52nd. Street in Manhattan, New York City.
And the "Musical Numbers Game" was a part of it right from the git-go--or, so it appears.
This "game" is related to the game of "fizz buzz", which is described on its own Wikipedia page (q. v.).
It became so well-known that it was even referenced in "Porky the Wrestler", a Looney Tunes cartoon released at the end of 1936. This cartoon was widely shown on television from the middle 1950's into the middle 1960's. IT circulated again from the late 1960's onward in a cheaply-done "colorized" version.
The "musical numbers game" probably went over well with the semi-schnockered clientele of the Onyx Club, who joined right in.