After the 90 minutes ended 1-1, Kyle Vassell hit the post for the Millers and Joe Mattock's kick was saved by man of the match Vaclav Hladky as the home side secured a perfect start to the season.
Salford thought they had secured the win in normal time when Ian Henderson's late penalty squirmed under on-loan Chelsea keeper Jamal Blackman after he had been adjudged to have felled substitute Tom Elliott. But an injury-time scramble saw substitute Matt Crooks' deflected effort cross the line to send the tie to penalties where Hladky was the hero. The former St Mirren keeper had kept the home side in the game with a string of first-half saves on his debut.
Rotherham's advances continued to be rebuffed after the break, Hladky tipping a Wes Harding drive over and blocking well from Chiedozie Ogbene The Ammies grew into the game and finally started to threaten around the hour mark, but were unable to convert two dangerous free-kicks.
Ash Hunter volleyed well wide with City's first effort on goal before referee James Oldham pointed to the spot as Salford took the lead. Ash Eastham headed against his own bar in the final minute before Crooks' effort found its way in to send the tie to penalties.