You say that the rules of math exist 'out there'. Okay, let's figure what that means:
1) The formal system that comprises math (axioms, theorems, definitions, etc) isn't just a methodology practiced by mathematicians, but it also describes the workings of reality.
2) This formal system is built upon axioms (using rules of inference, definitions, etc) to construct theorems. Hence, reality itself is 'built' upon the same principle. That is, statements of reality depend on axioms, which is why statements of reality are to be considered true. If the axioms of reality are wrong, then the statements are also wrong.
3) The axioms in a formal system (e.g., math) are *not* proven, they are merely accepted as true by convention. Translating this in terms of reality, there exists a problem as to how deductive statements of reality are true if the axioms are selected by convention. Your view has no means to substantiate the claim of an axiom. That is, it is true because it is obvious to you by viewing the universe, but it is the universe's existence that we are trying to explain here by saying 'math exists'. So, you have a Catch 22. If you say 'math exists', then you are dependent on axioms being true (even though they are not proven in math but are only selected by convention based on the obviousness of how they appear (in the universe)). However, if you accept those axioms as true, then you have no reason to say they are true without injecting the very the thing you are trying to explain - that being the universe. This is a logical fallacy known as 'circulus in demonstrando' (aka circular reasoning).
You have to give account for the meaning of axioms in your model if you want to use a 'math world' as your metaphysical foundation.
Warm regards, Harv