Andy Barcham scored the only goal of the game to move the Gills back up to seventh spot with two games remaining, six points better off than Scott's side.
Scott conceded that the Millers can forget about securing a play-off spot and gave a frank assessment of the situation he has inherited.
He said after the game: "We had several chances but we let them off the hook. For my first three games we have been reasonably good for the first hour and then we are tailing off and haven't got the energy levels and the ability to continue for 95 minutes.
"We lacked the finishing touch and a little bit of zip inside the box and the fitness levels in the players are miles off what they should be.
"We have players of good individual ability here but collectively we are not good enough, collectively we are not fit enough and collectively we are not mentally strong enough.
"The players that are left next season will have the toughest pre-season of their lives and I'll be bringing my own men in to make sure things get better."
Source: PA