It took City 81 minutes to finally finish off a well-organised Rotherham side, who had plenty of chances to claim all three points themselves.
After a poor opening spell, the game sprung into life on 22 minutes when McAllister delivered a deep cross from the right which Dean Gordon did well to keep in.
The ball came back to Richie Partridge and his cross found John Eustace whose header was well saved by Mike Pollitt.
Rotherham then came within inches of snatching a goal when both Richie Barker and Alan Lee slid in to meet a Chris Sedgwick centre, but both failed to connect in front of a gaping goal.
United took a shock lead on 38 minutes when Lee's header from Rob Scott's cross forced an excellent save from Morten Hyldgaard, but the Danish keeper could do nothing about Nick Daws' follow-up.
But the lead lasted under a minute as Coventry burst forward from the restart and Youssef Safri, not enjoying his best game for City, made amends for a shaky performance by picking out McAllister in the box.
The Sky Blues boss delayed, dummied and skipped past Scott and Darren Garner before rifling a low shot past Pollitt with his left foot.
Rotherham continued to cause their hosts plenty of problems after the break as Lee came close twice.
After he found himself one-on-one with Hyldgaard on 63 minutes but rolled his shot wide, he was unlucky to see his header saved on the line by Calum Davenport four minutes later.
Then, with nine minutes to play, a cross from the right found Lee Mills at the back post and his header landed at the feet of McAllister who stabbed home from six yards.