There's a great deal of confusion on what is mathematics. Recently Helen posted a reference to a Jeeva Anandan paper that tried to phrase physics in fundamental symmetry groups (or symmetry fields as defined by Anandan) that are responsible for quantum probabilities and dynamical laws of the universe. Dick has recently posted a reply that furthers his argument that mathematics is a means by which one can make some fundamental argument about the nature of the world, or at least our limitations in that interaction of the world.
What is mathematics has actually a very straightforward answer. Mathematics is a series of human abstractions that make our interactions with nature more convenient and more meaningful (i.e., comprehensible in terms of things that surround us). By utilizing these abstractions (mathematical objects), we can advance science (roughly the study of how nature can best be conceived as working and having worked in the past and working in the future along with how nature has/is/will work in the very small, the very big, the very distant, the very quick moment, etc) and thereby extending our comprehension of nature far beyond our immediate spatial and temporal surroundings. These abstractions represent a formalization of our observations and thereby can be agreed upon by agreed upon standards with everyone far and near.
The confusion comes in when people start abstracting for abstraction's sake, and then things get a whole lot more complicated. Instead of abstracting to solve particular problems and conceptual questions as they somehow relate to the here and now, these abstractions take on a life of their own and this becomes a self-gratifying exercise with myriads of interpretations, few having any real bearing on what is actually being accomplished.
When talking about Anandan's fundamental symmetries causing quantum probabilities or Dick's "can represent anything with numbers" the abuses of mathematical abstraction reach the point to where mathematical abstractions (whether it by symmetries or representing things with numbers) become a liability to our clear problem solving abilities more than they clear up matters on how the universe works.
Take Anandan's approach for example (and many physicists share this view) on the fundamental symmetries. It is ridiculous in speaking of symmetries as 'existing' much in the same way that it is ridiculous in speaking of a moment 'existing'. One moment flows into another moment and we use the term 'moment' to refer a referencable timeframe that we would like to designate for practical reasons (e.g., a moment ago I was saying that mathematical abstractions are human constructions). I don't mean to say that there was actually a 'moment' that had a clear and precise boundary. No, defining the 'moment' is based on context, what you have in mind, what I have in mind, etc. If we get down to it, there is no real objective 'moment', it's all a matter of perception, inaccurate communication, etc. However, to communicate we don't need to be exact in our communication, all we need is to understand to the point to where it is useful and meaningful for all practical purposes.
Similarly, symmetries of physics fall into a very similar category. They are identifiable in a rather formal manner, but not so formal that we fall into endless disagreement about what we mean by a specific symmetry. To be so formal would require an exact definitions which would require an overarching theory of everything that would perfectly define what we mean by every physical designation. Such exact definitions even in physics is impossible, and any kind of manner to try and define physical objects (e.g., defining it by how we test it, or defining it by how we discover it, or defining it by how theory treats it, etc) are all defeatable when you put such notions to philosophical analysis. So, when we speak in terms of symmetries, we aren't speaking very formal, but we are speaking in more loose talk about things that are formal enough on a practical level such that we can meaningfully talk about such items in a progressive (pragmatic) manner. This is until someone gets confused and starts referring to the symmetries themselves 'as if' they are objects that exist instead of how they were originally constructed as abstractions that allow us to make pragmatic progress to some goal of making the world immediately around us more practical and comprehensible.
This is when mathematics and logical abstractions become an actual hinderance to our deeper understanding (or lack thereof) of the universe. We begin to imagine that we understand the universe better than we do, and in the end we actually lose an opportunity to better understand the universe as the mystery that it deserves to continually be construed by humans.
This is not to say that it is a terrible mistake to try and 'see through' the mystery of the universe and look for metaphysical structures that we might, at least, be able to glimpse with our crude analogies that our abstractions provide (e.g., symmetries, numbers, groups, fields, etc). The trick here is that we can certainly extend our abstractions to the metaphysical level, but we must be very careful to do this. We must realize first, that we are speculating, and that second, there is no way to confirm our speculation. At the same time, as humans we are forced to speculate because we seek deeper meaning in the universe than what is apparent without projecting our abstractions into the universe, so we are actually forced to make some sort of sense of the world using our abstractions otherwise it is not an enjoyable or satisfiable experience to do otherwise. This is the source of human myth, and contrary to popular view, humans have not ceased from creating myths, even in science. Myth doesn't mean false or disillusionment, it means rather (roughly) a metaphysical conceptual structure that amounts to some form of speculation in a manner that makes our interaction with nature comprehensible and satisfiable.
This is where religion comes in. The goal of religion is the same as those who misconstrue mathematics with the intent of finding secrets to the universe that just aren't assessible to us mere humans. Our ties with religion are myth making in that it furthers the meaning and satisfaction that humans gain from reflecting on their existence. You can remove religion, then our science is lame as Einstein mentioned. The reason this is true is because humans crave meaning, and without a myth attached to our science, we cannot make sense of the meaning of that science - hence it becomes lame. Similarly, myth making without science falls into the pit of all kinds of ridiculous myths that have little to do with our scientific knowledge and experiences, hence it has our myths are blind.
Here science and education must learn and accept man as the myth maker and become more confortable with the notion that we are myth makers and story tellers. This is our heritage passed onto us by our ancestors who emerged from the forests of Africa to migrate across the world.
Anandan, Dick, and many others like them who use mathematics to create myths are all welcome to do so, in fact they should do so since they too are myth makers as members of the human species. However, they are truth makers and as much as they'd like to think otherwise, this is our human limitation with all of our abstractions and all of our science cannot change.