It was because the rogues shot at them right as he fired.
Taking a direct quote from the book:
"However, we had no luck; for just as Trelawney fired, down he stooped, the ball whistled over him, and it was one of the other four who fell."
Basically, the rogues shot, so Trelawney had to stoop down to avoid a shot from the rogues, which threw off his aim.
