The Millers dominated much of the first 45 minutes without finding that cutting edge to break down the home rearguard.
Nicky Law, leading by example in midfield, made things tick for Rotherham who created two decent chances in the opening quarter as firstly Tom Pope saw a header loop over from Jamie Green's cross, then Law himself carved out an effort which was well cleared by young Ryan Bennett.
On 23 minutes the Mariners created their first and only substantial effort of the half as Nick Hegerty somehow sidefooted wide of Andy Warrington's post when it looked easier to hit the target.
As Green was forced off with injury and replaced by Marc Joseph at the break, the game sprung to life eight minutes after the restart as ex-Miller Robbie Stockdale handled five yards inside the box and £100,000 signing Le Fondre beat Forecast low to his left infront of jubilant Millers supporters to finally break the deadlock.
Captain Ian Sharps left the field with injury shortly afterwards and was replaced by Micky Cummins as Pablo Mills moved into the back four, but the Millers lead lasted only 10 minutes as Peter Sweeney fired low past Warrington from 18 yards following a slack clearance to level matters.
But parity was soon restored just three minutes later with Law again in the thick of the action.
Another ex-Rotherham player Joe Widdowson brought him down on the right flank, and he picked himself up to deliver the free kick onto the head of substitute Cummins who provided the finish.
Law almost capped a man of the match display in the dying seconds, but his jinking run and sidefoot shot rebounded to safety off the foot of the post, with the Millers more than deserving their first away win of the campaign.