And a quick-fire double strike in the opening 11 minutes from winger Reuben Reid and striker Drewe Broughton went a long way to securing a sixth win of the season.
Andy Warrington, saved an 80th minute Andy Bishop penalty after Nick Fenton had fouled sub striker and ex-Miller Chris O'Grady.
Bishop saw his spot-kick saved superbly by Warrington, diving away to his left.
Lively Reid opened the scoring after just two minutes when he latched onto an Alex Rhodes through ball to side foot home past Wayne Brown.
Broughton doubled the advantage in the 11th minute when his powerful header from 12 yards out flew inside the right-hand upright past Brown infront of a strong vocal away following.
But Bury came back into the game strongly when unmarked captain Paul Scott glanced home a Michael Jones left-wing corner before the break.
The hosts, who could have gone top with a win, have now suffered consecutive home defeats.
Bury were looking all set for a point when defender Fenton felled O'Grady in the box, but it was not to be courtesy of Warrington's heroics.
Former Rotherham striker O'Grady, who is on loan at Gigg Lane from League One local rivals Oldham, suffered a tirade of judas chants from Millers fans and saw a second-half goal disallowed for offside.