Sure he was a powerful duelist, but a group of others could take him down.

I didn’t read that even slightly as if he was talking about magical power; I read it as he was talking about Dumbledore being extremely influential.