Placing the ball on the flat portion of the track (at left) shows neutral equilibrium. Placing the ball in the trough shows stable equilibrium. Placing the ball on the crest of the track (at right) shows unstable equilibrium.
This demonstration illustrates the three types of equilibrium in which an object (a rigid body, that is, one whose constituent parts do not move with respect to each other) can be as it sits in a gravitational field. These three types of equilibrium arise from the different ways in which the object’s potential energy can change when the object is displaced from its position. To express this change in potential energy, U, in relation to gravity, we can write Fx = -∂U/∂x, Fy = -∂U/∂y and Fz = -∂U/∂z. For each direction, if this change in potential energy equals zero, the object is in translational equilibrium in that direction. If we consider a particular coordinate axis, we can imagine three cases in which this partial derivative equals zero.
One is if the potential energy is constant. That is, a displacement in either direction does not change the potential energy of the object. When the object is displaced, then, it experiences no force that either increases its displacement or brings it back to its original position. It is said to be in neutral equilibrium. When you place the ball on the horizontal portion of the track, at left, however you displace it, it experiences no sideways force in either direction. This illustrates neutral equilibrium.
Another way is if the potential energy is at a minimum. In this case, displacing the object to either side raises its potential energy, which results in a (gravitational) force that moves the object back towards its original position. It is in stable equilibrium. Since displacing the object raises its potential energy, any force that would displace the object from its original position must do work on it to do so. The ball resting in the center of the track illustrates stable equilibrium. If you diplace the ball to either side, it rolls back to the middle.
The third way is if the potential energy is at a maximum, in which case displacing the object to either side lowers its potential energy. This results in a (gravitational) force that moves the object further from its original position. It is in unstable equilibrium. Since displacing the object lowers its potential energy, gravity does the work necessary to displace the object from its original position. The right end of the track, which has an upwards-facing curve, allows you to show unstable equilibrium. If you place the ball at the top of the curve, any slight displacement to either side causes it either to roll down into the well, or to roll off the end of the track.
References:
1) Resnick, Robert and Halliday, David. Physics, Part One, Third Edition (New York: John Wiley and Sons, 1977), pp. 291-3.