Mathematics and language are not distinct entirely; rather, I view mathematics as an absolutely precise language with unambiguous grammar and (alas) a limited vocabulary. The vocabulary of mathematics speaks only about mathematics, and not about oranges or stones falling. Physics is, in part, an attempt to make mathematical models having a one-to-one correlation with our observations of the physical world. We attempt to build a map of a territory, so to speak; but no matter how accurate the map, it is only a map, and is not the "true" territory.
The maps are useful. One of the major separating factors between modern science and Fundamentalist religious explanations of the world is that our mathematical maps extend to territories we've never visited. We can test the accuracy of our maps by saying (if you follow my analogy) that across the sea 250 miles is an island of a particular shape and size, as yet unseen by any man. We build a ship (design experiments) to test the assertion--the island exists, or it does not, or there is instead a huge coral reef of the same shape and size that the island was supposed to be. We modify the maps based upon what we find, willingly discarding faulty maps along the way, and our sailing becomes safer as a result.
Fundamentalist religion, by contrast, has a map that can never be modified. Any unexpected island must somehow be forced to fit the only map, for example by claiming that the island did not exist when the map was made.
As for reality, my belief is that perception is the only certain reality. It serves me well, in a practical sense, to assume that my visual and tactile perceptions at this moment correlate to externally existing physical objects that I call a monitor and a keyboard. But that is merely map-making. I live according to a huge map, constantly being revised, that allows me to connect diverse perceptions (reality) in ways that are predictable and profoundly comforting. Only seldom (such as now) do I stop to wonder whether external physical reality exists; the map works so incredibly well in linking my perceptions together that I would be lost without it. But I can never know whether a physical universe lies behind the perceptions.
That said, I choose to believe in the physical reality, and in ethics and morality, because existence without those things would be depressing and devoid of purpose. And I choose to believe in a God who, if He exists at all and if his Creation exists, is truthful, and who has devised a physical reality that does not lie to us in its wonderfully predictable physical laws.