Dutch winger Etienne Esajas scored twice, but then made the vital miss with his penalty as his night ended in despair.
Esajas thought he had produced a winner three minutes from the end of extra time but Reuben Reid levelled a minute later to signal the shoot-out.
With five minutes gone, Deon Burton pounced on Lee Grant's free-kick as it dropped kindly in the area and snapped off a shot that ricocheted out for a corner.
And when Steve Watson collected a ball on the edge of the box his deft, curling effort had Andy Warrington stretching but it stuck the angle of post and bar.
A Leon Clarke shot drifted wide after good work by Jimmy Smith before 90 seconds of quick-fire action warmed up the crowd.
Wednesday winger Esajas smashed a left-foot shot that deflected off Mark Lynch and Warrington dived at full stretch to parry it away.
Almost immediately, Burton collected the ball wide on the right and drove an astute angled pass that wrong footed the Millers and this time Esajas arrowed his shot just inside the post after 14 minutes.
Wednesday fans were still celebrating the goal when Rotherham went straight to the other end in the 15th minute.
A clever pass from Pablo Mills found Alex Rhodes, who left Richard Wood flat-footed and slotted in a cool shot.
Esajas came within a whisker of restoring Wednesday's lead three minutes before half-time.
The home side won a free kick 30 yards out and when James O'Connor slipped the ball into his path, Esajas smashed a stunning shot that beat Warrington but hit the woodwork.
Two minutes into the second half Esajas - who switched to the right wing - was the dangerman when he cut inside smartly but Warrington made a great point-blank save.
Ten minutes from the end Wednesday were denied again when a Smith shot hit the post.
Then came the extra-time and penalty drama, as Esajas swung home a beautiful free-kick after Marc Joseph was dismissed for hauling down Leon Clarke. The drama continued as Reid pounced to smash home the leveller infront of the Millers fans who spilled onto the bottom tier of Leppings Lane in celebration, after Danny Harrison hit the bar and a goalmouth scramble ensued.
Esajas then turned to villain from the spot, hitting the post from 12 yards leaving Reid to seal the win infront of the kop end by slotting home his kick into the corner.