Riderless bicycles (hand launched) tend to self-stabilise whilst in forward motion.
The physics behind this stability is 'poorly understood'. Various theories have been put forward - including front-wheel spin angular momentum (gyroscopic), or 'trailing caster' effects - but these are disputed. Full details here
More history of the study of bicycle self-stability here
For a recent (2016) article, see The bicycle problem that nearly broke mathematics in Nature, 535(7612): 338-341.