That volume is not the volume of mass a or b. The volumes of the masses in not expanding, just the volume of the distance between them.

So what do you mean by gravitational mass.

We know that as the distance between the masses which we are calling D, when that decreases, the force of gravity increases, yet the volume you call gravitational mass is decreasing. It does not make sense.