I'll remind everyone that the "magic" in the magic wall is to keep the white walkers out. The magic doesn't do anything against normal humans (or giants.)
As for their general halfassed-ness at defending the wall, there's only like a hundred of them (divided up to defend both sides of the wall), they're not particularly disciplined, and until Snow took over, they weren't really getting orders worth a shit either.