If you want proof by induction, then you do want to use well-ordered sets, but you need the correct definition of ‘well-ordered’. Of course we can't require that any non-empty subset have a minimal element; then Excluded Middle follows. But requiring any inhabited subset to have a minimal element is still too strong. Requiring a total order is also too strong (although at least the natural numbers have that). Instead, we define a well order on a set $ X $ to be a binary relation $ \prec $ that is transitive, extensional, and well-founded (and define ‘well-founded’ to mean that induction works, not anything about minimal elements).
You know what transitivity means: if $ x \prec y \prec z $, then $ x \prec z $. It will follow from well-foundedness that the relation is irreflexive, so we have a (strict) partial order (sometimes called a quasi-order).
Extensionality is complicated to define in general, but since $ \prec $ is well-founded, it's enough to require weak extensionality: if for each $ t $ in $ X $, $ t \prec x $ iff $ t \prec y $, then $ x = y $.
(From this, using well-foundedness below, you can prove strong extensionality: if for every $ \prec $-bisimulation $ \sim $ on $ X $, if $ x \sim y $, then $ x = y $. Here, a $ \prec $-bisimulation is a relation $ \sim $ such that: if $ x \sim y $, then for each $ s $ in $ X $, if $ s \prec x $, then for some $ t $ in $ X $, $ t \prec y $ and $ s \sim t $; and if $ x \sim y $, then for each $ t $ in $ X $, if $ t \prec y $, then for some $ s $ in $ X $, $ s \prec x $ and $ s \sim t $.)
Finally, well-foundedness means that for each subset $ S $ of $ X $, if $ S $ is $ \prec $-inductive, then $ S = X $. Here, $ S $ is $ \prec $-inductive if, for each $ x $ in $ X $, if (for each $ t $ in $ X $, if $ t \prec x $, then $ x \in S $), then $ x \in S $. So if $ S $ is the only inductive subset of $ X $, then you prove properties of $ X $ by induction on $ \prec $.
It's handy to say that $ t $ is a predecessor of $ x $ if $ t \prec x $. Then the definitions are easier to state in words:
- transitivity: a predecessor of a predecessor is a predecessor;
- weak extensionality: if two elements of $ X $ have the same predecessors, then they are equal;
- (bisimulation: if two elements are related, then each predecessor of either is related (in the same direction) to some predecessor of the other;)
- inductive subset: if every predecessor of $ x $ belongs, then so does $ x $ itself.
This stuff is on the nLab, which may not count as a reference since I mostly put it there. I got the ideas from Paul Taylor, and there are references to his work in the other answer, so that's probably what you want to look at. But note that the plump ordinals are only some of the ordinals by this definition; much less ‘plump’ sets like $ \{ 0 , 1 , 2 \} $ and $ \mathbb N $ are also ordinals (just not plump ones).