Yes, time is not a dimension (a degree of freedom to move back and forth) it is a coordinate. Space is not only a coordinate but a dimension too (degree of freedom).
Notice that in all GR and QM equations time is an imaginary coordinate (comes with i in front).
Time travel contradicts to fundamentals of logic, namely to mutually exclusive states. Like a question: is there a life after death? has a simple answer: no, because absence of life is what we label as "death" (you know, instead wasting time on saying "this body has lack of life" we shorten: "this body is dead". Both statements are exactly equivalent.)
So, if you assume that any object in our Universe can either exist or not, but not both at the same time, then the time travel is prohibited by this statement. Example: you travel back in the past, find and kill yourself (or your ansestors, etc.) -should you exist upon return or should you die immediately as you arrive back to the present?
Another logical problem: when you go say 1 year back to the past, where in space should you "land"? Earth is no longer there, Sun neither, nor Milky Way. So, should you find yourself somewhere in a vacuum? Well, where exactly?
Even communication with "the past" is strictly prohibited: not a one bit can be sent "back" (cause it may be the bit to order to kill yourself, for example).
Same logical problems with "the future". You go "there" and find yourself dead of car accident on 10/10/2012, 8:55 a.m. (rushing to new work). Then you go back - and now we have 2 futures: one (real) with you dead, the other with you pushing brakes a little earlier - which one is then "more real" one?
Or you "go fture" and learn in the year 2050 that there is a great discovery made - gravity is finally explained, for example. You pick the paper, return back and publish it. Now: future scientists will get it from you, but you got it from future scientists. So, where does this new knowledge about gravity come from?
And this kind of logical mess pops out of time travel all over the idea.
So, the past is simply no longer "there", and the future is not "there" yet.
Moreover, even what you may call "the present" has zero "length" on a "timeline", which strongly suggests that time is absolutely not what our imagination stretches to an infinite line with "floating zero" at the present.