As scientists concluded long ago, gravity is one of the four forces. It is the force that bends space-time. You logic is flawed.
The most basic physics says that there is always a cause and effect. You say the cause is matter. At that point you are correct. You are admitting that there is some sort of reaction between matter and space-time. That reaction is usually known as gravitational force.
Now, on the other hand, if you say that there is no gravitational force(the interaction between matter and space-time), then how would matter affect space-time? How would everyone know the more matter an object has(and also how dense the matter is), the bigger the distortion of space-time?

I hope this has explained my point better.

Chris