The Millers started the evening with only six subs, as Mark Randall was left stranded on the M6, but the team started quickly out of the blocks as Kevin Ellison fired over in the early stages and Taylor headed Tom Newey's set piece just wide.
The home side's Steven Husband was the main livewire, and tested Andy Warrington from 20 yards and Anthony Elding also went close, before two goals in as many minutes sparked the contest into life just past the half hour.
Loanee Ismel Demontagnac raced clear to clip the ball over Warrington and give the hosts the lead, but within a minute Ellison responded at the other end to level affairs with a left foot drive past Williams after being teed up by Adam Le Fondre.
County were then given a penalty by referee Coote, who wasn't the most popular man on the night, before the break as Le Fondre was penalised for handball in the wall and subsequently shown a yellow card.
But Warrington brilliantly saved Elding's spot kick, diving away low to his left to tip the ball to safety.
After the break saw Stockport boss the middle of the park and the Millers struggle to cope with their spare man in midfield.
Husband continued to run the show, as he volleyed in from 25 yards and then sped clear to slot home a third to seemingly give the home crowd all three points. In the meantime Coote was again at the vent of Millers' fans frustrations, ruling out a seemingly good goal by Le Fondre at 2-1 as Taylor was penalised for a foul on Williams.
But Rotherham kept going and with five minutes remaining, Grieve back headed the ball into his own goal in a defensive mix up to give them hope.
In a frantic finale, Taylor powered home a header from Nicky Law's free kick in the last minute as the Millers fans spilled onto the pitch and Taylor celebrated with the crowd.
The unthinkable then almost occured, as sub Tom Pope hit the outside of the post with five minutes of added time elapsing, Taylor saw his point blank header saved and Law fired the rebound wide.
The ref then waved away claims for the evenings second penalty as a shot appeared to be blocked on the line by a Stockport arm in the final scramble of a crazy evening.
Att: 3,612 (711 away)