Formalization of numbers.

I always felt that there was something strange about the way the concept of numbers are formalized by set theory or category theory. Although the concept of numbers do have some problems, unless we deal with infinities there does not seem to be an urgent need for formalization, as far as calculations are concerned, including complex numbers.

Humans seem to have an intuitive understanding of numbers, on par with qualia. By trying to formalize numbers by set or category or other theories, something extra is added, and there are consequently some "evil questions" that shouldn't be asked, as numbers represented by these formalisms have something external to our intuitive understanding of numbers. 

I am not saying that attempts at formalization are meaningless. I just feel that numbers should be treated on the same footing as qualia. Just as attempts at formalizing qualia do not go well, formalization of numbers is bound to be unsatisfactory, because such an approach is not addressing the essential nature of numbers directly.