Take a small dowel or bamboo skewer and round over the top on some sandpaper, Wrap some 320 wet/dry sandpaper over the end and sweep it in and out, angling the round tip so it dips into the port on each pass. The burr will want to tear the paper quickly at first so refresh the abrasive as often as necessary until it stops shredding. Then use some 400 grit in the same manner to smooth it. If the burr is very small, skip the 320 and just use 400.
BTW, I drafted a more comprehensive barrel inspection and accurizing guide here: