Recent asteroid flew by Earth at the distance about 0.6 million kilometers - almost twice the distance to the Moon.
Let's estimate how many times (in the average) do such asteroids have to fly by, before some hit Earth? Of course ANY of them may hit Earth if it happen to move directly to it, but let's estimate how rare is such event.
Earth radius is about 6400 km, so the cross-section area of Earth is piR^2 = 128 million square kilometers = 1.3x10^8 km^2. If you randomly shoot an asteroid into the circle with the radius 0.6 million km (that is how close the recent one flew), what are chances that it hits tiny Earth? Not hard to calculate: probability is simply the ratio of the area (cross-section) of Earth to the area of that circle. The area of the circle with the radius 0.6 millions km is about 1.1 x 10^12 square kilometers, so the 128 million kilometers of the "Earth share" is only small fraction of it: it is numerically about 1.3x10^8/1.1x10^12 = 1/8500. So, we have to watch a few southand of such fly-by asteroids till one will not miss the Earth.
Because close passes of large asteroids are quite rare (once in a few tens of years), we have to wait about a few tens of thousand years to witness such catastrophic impact.
Yet it more likely will hit an ocean, as 2/3 of Earth surface is water. And if it hits the land, it is also unlikely to directly hit densely populated area. So we have to wait a few hundreds of thousand of years for really catastrophic hit.
Indeed, scars on Earth show that similar hits took place quite rarely - once in a few million years. Larger (global catasrophe) impacts are even much more rare.