The Wide-field Infrared Survey Explorer (WISE) telescope has confirmed the fears of many people. This instrument sees in the infrared, and has recently been used by NASA to pinpoint the locations of near Earth asteroids. It turns out there are many of them. Some are too small to cause anything other than mischief, but a small number are truly dangerous.
Full story here.
Most of the debris zooming around the solar system is either too small to cause much damage, or is confined to a specific orbit. For example, a golf ball sized meteorite strikes the Earth quite often, but is burned up in the atmosphere. On the other end of the spectrum, the asteroid Vesta would likely kill all life on Earth during a collision. However, Vesta has a very specific orbit. It would take an incredible amount of energy to move Vesta into a position where it could threaten us.
Most of the near Earth asteroids are small, but a few are large enough to cause damage. Of the tens of thousands thought to be lurking out there, only an estimated 4,700 are larger than 100m across. A 100m rock will survive a trip through the atmosphere and can devastate a city, but is not large enough to cause a global event.
To cause a truly catastrophic impact, you need a much larger rock. The dinosaurs endured many impacts until a six mile wide asteroid came and wiped them out. An asteroid of that size releases enough energy upon impact to have a profound effect on the entire globe. Luckily, rocks that big only seem to come around every billion years or so.
Tracking potentially threatening asteroids is the first step in fighting them. There is no doubt that an asteroid will devastate the Earth; through sheer probability it must happen, given enough time. In the near run, it is certain that a 'city killer' rock will eventually strike the earth. It might not be for one thousand, ten thousand, or a million years. Or it could be tomorrow. We won't know until we can track all the objects that may be out there.