Dear Megan, Nelson is right - chaotic systems can be very simple - for example, all systems in unstable equilibrium are chaotic. Mathematicians know such systems for centuries (mathematical description of their behavior in the case of limited information about initial conditions is called chaos theory).
(Non-linear systems are different than chaotic systems, by the way.)
To demonstrate simple example of chaotic system, take a pendulum with the length l and let it go from the position of equilibrium x=0. The equation describing its motion (in simplified linearized form) is the following differential equation: d2x/dt2+xg/l=0. Thus the solution is x = xosin(sqrt(g/l)t + phi) which is called harmonic function (sin and cos functions are labeled "harmonic" functions my mathematicians). So, the solution as you can see describes change of x in a periodic manner - oscillations back and forth around the position of equilibrium (x=0) with the amplitude xo and angular frequency sqrt(g/l).
Now take the same pendulum and place it upside down (we assume that the pendulum arm is rigid - simplest example is a pen placed on its tip). Now you have exactly the same equation describing motion of inverted pendulum, but with "-" sign instesd "+" (restoring force had changed direction): d2x/dt2 - xg/l=0. Now, the solution of this equation is exactly the same but now we have sin(sqrt(-g/l)t + phi) - notice minus sign under square root. Sin of imaginary number is exponent, so the solution is thus x = xoexp(sqrt(g/l)t). Now notice that if initial deviation from position of equilibrium xo is zero, the pen will not fall - will continue to stay on its tip forever. But even slightest deviation from zero to the right (small positive xo will eventually result in increasing deviation to the right. Exactly the same way, taking a look at the solution you may notice that small deviation to the left (negative xo) results in motion to the left (because exp is always positive function). So, depending on initial conditions the pen will fall to the left or to the right. In 3-dimensional case it will move in the direction of initial deviation.
On this example you may see that mathematics of chaotic systems is no different than mathematics of stable systems - the very same equations.
Why there is more attention to such systems?
We are just more fascinated with systems in which small deviations lead not to the return, but to further deviation from the previous behavior - like children are more fascinated with the top not when it is spinning fast and thus precessing slowly and in a stable manner (because the precession angular velocity is inversely proportional to the rotational angular velocity), but rather when the top loses its angular velocity and starts to "seemingly unpredictable" nutate before it falls - dancing on a strange manner, which actually is dictated by the solution of its equation of motion, so it actually behaves all the time in full agreement with math.