During the early stages of the big bang, there was a continuing back and forth reaction with matter-antimatter pairs on one side and phton pairs on the other. As a result of the slight assymetry in the decay processes, there eventually was an excess of matter which survived. I don't know what you mean by a "kick-back". In any case as things cooled down during expansion, photon to stuff reactions were no longer energetically possible, leaving the universe with whatever matter survived.