If you take a rope with say, a can of beer tied to it and revolve it around, you exert a force on the can (via rope) directed toward you (along the rope), and this force is called centripetal force. Where is a force, there is an acceleration a=F/m, so instead of moving in a straight line, the can's velocity will change in the direction of force, thus the can will "go around and around" as long as you keep providing centripetal force If the rope breaks and no force is longer there, the can will fly off in a sraight line in the direction of velocity vector - tangential to the circle).
Also, according to momentum conservation law, the revolving around can will exert equal and opposite by direction force on your hand - called reaction force - and you feel this force by trying to keep rope which pulls your hand strongly in radial direction. Action and reaction forces are always applied to different objects (in this case you on can and can on you), thus they can not cancel out each other.
Now let's review the same circular motion of the can of beer but in a rotating with the can system of reference. In such (non-inertial by the way) system the can will appear non-moving. But if a line breaks the can starts moving radially (deviating more back in a spiral manner in rotating reference system).
To "explain" accelerated radial motion of a can (which actually is nothing more than a straight line motion but viewed from the rotating- thus accelerating- system) someone long ago coined the term "centrifugal force" which does not actually exist. It is not mystical centrifugal force which slides stuff off the dashboard of your car every time you make a turn, it is your car which moves away from the straight inertial motion of the stuff, so for you this straight constant line motion appears to be different.
So, if the Earth would not pull nearby stuff by its gravity, we (as well as air, water and Earth itself) would indeed fly off in a straight line tangent to Earth surface and would soon scatter all over surrounding space. Fortunately, Earth pull is strong enough to hold us and air, etc. on the surface.
If Earth would not spin, we would not feel any much difference - instead 9.80 m/sec^2 acceleration of free fall would be 9.83 m/sec^2, because centripetal acceleration of Earth is very small.
On your question why Earth is not falling on Sun -it actually is, because there is no "centrifugal" or any other forces to counterbalance Sun's gravitational pull. So, Earth is constantly falling toward Sun - instead going along a perfectly straight line it's velocity vector constantly rotates toward Sun making what we call orbital motion.
Question about Sun rotation - Sun does spin too as most of celestial bodies (about ones in 25-27 days).
Black holes move around the same way any other mass does. What happens when 2 black holes meet? Two black holes orbit each other the same way any other two objects do. Because they move with high acceleration, they generate strong gravitational waves radiating their kinetic energy away. As a result, instead of following stable elliptical orbit, they spiral onto each other with increasing orbital velocity. You can even "hear" increasing pitch of their gravitational waves (if you were nearby) before their crash - something like:
There is about 5 million sun mass black hole in our Milky Way galaxy center and about 30 million Sun mass -in the center of nearby big neighbor Andromeda galaxy. (By the way, Andromeda is sucking Milky way by its strong gravity, so Milky way will "soon" end up being torn apart and "digested" by Andromeda galaxy - in about 5-7 billion years from now).
You can see some fun movies about travelling to a black hole here: