The student of physics cannot help but notice, and perhaps resent, the central place of mathematics in the curriculum. Doing physics is nearly synonymous with writing equations, and the centrality and sophistication of mathematical methods in physics have only grown over time.
Why is this? Why does mathematics seem to be the language of physics, and even of science as a whole? Is it an accident, something which might have been otherwise? Is it something even to be regretted - a bad joke played upon us by the creator, who could have created a more-fun universe but chose not to?
I believe it is not an accident. In the following I will argue that any possible universe must have a rigorous mathematical basis, just as we have discovered in our own.
Observe first that a universe must have rules. Donkeys don't fly, horses don't turn into supernovae, bugs don't become buggies...an endless list of constraints operates at all times in our universe, and it seems clear that a similar list must apply in any universe. Any universe must contain things which are distinct from other things, and every distinction implies some kind of rule - thing A doesn't spontaneously change into thing B. No rules implies no things, i.e., no-thingness - nothing.
So where do these rules come from? Could we design a universe be by listing out each of them in "plain-english" form as above? You can try but - good luck.
For starters, you would need an infinite number of rules. More seriously, it isn't even possible to define any of the terms used in such rules. What, for example, is a bug? If a frog eats a bug, is it still a bug? When does it stop being a bug? Is a partially developed larva a bug? Is an animation of a bug a bug? Is a mutant or genetically altered bug a bug? Virtually all plain-english concepts are impossible to define, and therefore not adequate for specifying a universe.
For example, we said a bug can't become a buggy. If we define a bug as, say, something which has more than 4 legs and an exoskeleton, then we have not ruled out the possibility that a mutant bug with 3 legs and no exoskeleton could turn into a buggy. We also have not ruled out the possibility that a bug could turn into a Lincoln Continental; indeed, we can't even start this discussion without first defining the terms "leg" and "exoskeleton", which we will find impossible to do.
Philosophers have wrestled with these problems of definition for millenia, and there is no solution. The higher-level concepts embodied in plain-language terms cannot be fully defined, but are inherently fuzzy and subjective.
Now, why in reality does a bug not become a buggy? Obviously it is because the bug is made of atoms, and the atoms don't spontaneously rearrange themselves or change their characteristics. And why is this? Is it because we have some fundamental rules governing atoms, rules like "sodium can't spontaneously turn into chlorine"? No - because atoms are not fundamental building blocks either, and cannot be rigorously defined any more than bugs (is an ionized atom still an atom? Is an unstable atom still an atom?)
Atoms act the way they do because they are made from electrons, protons, and neutrons. Protons and neutrons, in turn, act the way they do because they are made from quarks.
And now we are getting someplace, because both electrons and quarks are fundamental, mathematical objects (at least in current theories). In other words, they can be defined completely. We can write down by means of equations exactly what they are and what rules they obey, under all circumstances, with no caveats or gaps. These are the kind of rules on which a universe can be based, and they are called the laws of physics.
So I would argue that the universe is built on mathematical objects because these are the only objects which can be comprehensively defined. No other kinds of objects can exist except as aggregated constructs of underlying mathematical building blocks (e.g., a bug is built from electrons and quarks). The underlying laws of physics are mathematical because no other kinds of laws exist. In creating a universe, the choice is not whether to base it on mathematics, but only which mathematics to use.
None of these arguments are original to me, of course, although I haven't heard them expressed in quite this form. The essential ideas, including my discussion of things not changing to other things, go all the way back to the original Materialists, as recorded by Lucretius. The original Materialists, interestingly enough, based their Materialism not on "scientific evidence", as is the custom today, but rather on exactly the philosophical arguments outlined above. Of course neither philosophy, nor science, nor any other technique can ever prove anything definitively, so we don't claim to prove that math must underly everything; however, we do claim that the case is pretty strong.
Recognition of the primacy of mathematics allows us to formulate a different conception of science and the scientific method, one which frames the debate with Creationists and other pseudoscientists in a different light. Science is the study of the underlying mathematical laws of the universe and the effort to connect all observed phenomena to them. The "scientific method" is nothing but common sense applied to this effort. There is no single method of science, just as there is no single type of argument in a legal case; however, there is a single goal to the endeavor of science, and it is by reference to this goal that we, in fact, distinguish science from pseudoscience. Science proposes explanations which are potentially connected to an underlying mathematical order; pseudoscience proposes explanations which are not. The concept of "refutability", which is very slippery to define in general, becomes crystal clear from this perspective: non-scientific theories are "irrefutable" because they can't be founded on mathematics and therefore do not follow any definable rules - and that which is not bound by rules can never be refuted.
We also find a new perspective on the concept of Materialism. Materialism is inseparable from Mathematics, and the "material" to which it refers can only be a mathematical construct - because no other construct is possible. This "Mathematical Materialism" is the necessary foundation of any universe. In doing science, we don't "discover" that the universe is mathematical, but merely what kind of mathematics it employs.