It means he will pretend to be mad, in order to plot his revenge and act upon it. "Antic" used to mean "madcap", "bizarre", which suits Hamlet because he doesn't want to be taken seriously until he proves to himself that Claudius is really his father's murderer, and Gertrude complicit. Hamlet goes on with his plan and does it so well that we, as spectators, are never 100% sure whether he is faking his madness or if it really grabbed hold of him.
