I'm not sure, but it sounds like your hands may be starting a little too far out away from your thighs, and when you shift direction from back to forward at the top of the backstroke, there is a slight relaxation of your upper arms when the downstroke begins. This relaxation allows the putterhead to "sink" back towards your feet a little. If this is indeed the problem, the fix is fairly straightforward and perhaps even a bit important.
The fix, essentially, is to pre-set your arms and manage the backstroke so this slight "sink" is not available, biomechanically. This requires that you truly hang the hands naturally and fully at setup and don't grip the putter higher than the hands are hanging; on the backstroke, don't use the hands to start the putter back or otherwise allow the putter's sweetspot to wander away from your feet across the line of the putt. Any of these will generate a putterhead at the top of the backstroke that wants to "sink" if the arms relax for the downstroke. All of this is another way of saying: use only the shoulderframe as a unit to move the "triangle" of arms and putter and move the shoulderframe in a plane of motion that is parallel to or coincident with the line of the putt.
In military sharpshooting training and in target competition shooting, there is an explicit point in the setting of the rifle in aiming to relax into the natural baseline position of the rifle. That is, the rifle rests in a proper position with respect to the body, and is not "held" in a position with tension. The same principle applies in putting during the setup: if the hands are too far away from the thighs and not hanging naturally in a relaxed fashion, or if the grip is higher on the handle than the hands naturally hang, the setup of the putter in the body has not yet settled to its natural state. Part of this is the repeating postures of stance position and distance from the ball and the cant of the shoulders forward and downward at address, but it is mostly hand position. You need to feel that once the stroke starts back, it can only go above the putt's line to the top of the backstroke, and then can only fall forward above the same line coming back. It takes a little experimentation to see what I'm talking about here, but this pre-setting of the hands seems pretty important to a consistent straight stroke.
Another way to view this is to aim the putterface behind the ball and make sure the sole is flat to the surface. Then, bring your body to the putter and assume the address setup positions so that the putterface orientation remains the same. Almost all golfers setup the feet etc. and THEN aim the face, and this strikes me as backwards. First aim the face, and then bring the body to the properly soled putter and hang the arms and hands before taking up the grip. This will usually get rid of the "out there" biomechanics that allow the "sink" at the top of the backstroke, but if not, use the sharpshooting training tip and see if you can't get the stroke working without any "sink" at the beginning of the downstroke.
Let me know if this is what's happening or how the above works for you.
Cheers!
Geoff Mangum
Putting Theorist and Instructor
PuttingZone.com
http://puttingzone.com
Golf's most advanced and comprehensive putting instruction.
Over 45,000 visits monthly and growing strong ....