Philosophers are divided on this question. I myself am agnostic on the issue, but I am somewhat skeptical of physical infinities (e.g., infinite past, infinite distance, etc). The reason is that infinity is a mathematical concept based on certain axioms. However, the physical world is not a mathematical concept.
From what we can tell, infinities only exist from the perspective outside of a set (e.g., there are infinite number of even numbers). We are outside the set of even numbers, and therefore we can say there are an infinite number of evens. But, inside the set you cannot reach infinity. Counting to infinity is a finite operation of adding X qty's to a previous number, and such a finite operation will always arrive at a finite result no matter how much time or computing power you spend. You can never reach infinity.
However, in case of the universe, we are saying that there is nothing outside of the universe, so how can you get 'outside' the universe for there to be infinity (e.g., infinite years ago)?
The only way I know that it could be is either there is some Law or Mind that requires the universe to be infinite (in which case the Law or Mind is 'outside' the universe), but in which case that destroys the idea of a completely uncaused material universe. The other idea is brute fact, which would mean that even though there is no 'outside' of the material universe, it is still infinite if one could be on the 'outside' of the universe. However, that seems unreasonable since a totally material universe you can't be on the 'outside', so the brute fact is much ado about nothing.
Yet, I can still imagine a world having an infinite past without there being an 'outside' to it. But, in that case I think I'm applying the mathematical infinite set approach to the universe, and just ignoring that I have done so. That is, first I think of non-ending time as 'existing' (which I visualize as an infinite set), and then I remove the set from my mind and think in terms of just a road that never ends inside the universe. Without the infinite set, the whole notion appears to violate logical causality. That is, an infinite set is defined by its members and the properties of the set which make the members all members with each other. If the set properties imply an infinite collection of set members (e.g., all even numbers), then it is easy to see how infinities come about. But, from inside the set the matter is entirely different. It is the operation itself which determines the next nearest member. If the operation is a physical operation, then it must have physical contact with the next member in order to be a coherent physical structure. However, if the operation is physical, then it must follow some kind of causal rules of contact. That is problematic if the distant contact is infinite distance since such kind of contact is undefined in a physical way. Therefore, it seems that such a physical operation is undefined (i.e., not physically possible) for infinite cases. I would lean to the notion that we have simply invented the concept based on our intuitive notions of counting physical things, and then re-apply the concept to the physical world.
Obviously we cannot be sure. So, there is a possibility for physical infinities, but I have to agree with the constructivists on this one - at least in terms of physical infinities. Logical infinities (e.g., laws, Mind, etc) is a different situation. I see no problem of having those kind of infinities except the problems associated with Cantor's infinite sets.
Warm regards, Harv