Stockport drafted in two-goal striker Liam Dickinson for his first start of the season, with Elding moving out to the left, while Rotherham replaced midfielder Danny Harrison with Steve Brogan.
Striker Matty McNeil twice fluffed near-post efforts for Stockport, while at the other end goalkeeper Conrad Logan produced a smart save from Chris O'Grady.
A blistering start saw chances galore at both ends, before Stockport eventually went into a deserved 13th minute lead when David Poole's excellent inswinging cross was headed home by the unmarked Elding, for his first goal of the season.
There was then a relative lull in the game, before Rotherham came mightily close on 38 minutes when Dale Tonge's looping cross was headed just wide by Derek Holmes.
Minutes after the restart Holmes was again unlucky, this time after Logan superbly beat away his fierce drive from fully 20 yards.
The referee was the centre of attention as the visitors levelled on 55 minutes when he whistled just as Brogan took a free-kick. The Stockport defence stopped in the confusion and Ian Sharps rose unchallenged to make it 1-1 with a powerful header.
County boss Jim Gannon was subsequently sent to the stands following his protests.
A dramatic spell then produced two goals in five minutes. Substitute Adam Griffin headed Stockport in front direct from Adam Proudlock's corner, then McNeil's own goal levelled it up again as he headed home Brogan's inswinging free kick as the Millers took a point back from Cheshire to maintain their unbeaten start to the season in the league.