The first response to any question about "the" ordinals is what you want to do with them. Regrettably, that is most often to find the fixed point of some construction within some already-known object, even though Kazimierz Kuratowski explained eloquently, but unfortunately not persuasively, that this can usually be done using closure operators.
In situations, common in algebraic subjects, where not all (binary) joins are available, you need Dito Pataraia's 1997 (constructive) fixed point theorem, or rather my adaptation of it that I have discussed on MO. There was a classical argument, called the Bourbaki--Witt theorem but actually due to Ernst Zermelo in 1908, which should have been a fundamental part of the curriculum long ago.
If you are trying to construct something by transfinite iteration of functors that goes beyond the logic of a an elementary topos or Zermelo set theory then set theorists will say that you need the axiom-scheme of replacement. However, I made a proposal using fibred category theory in the final pages of my book that I intend to write up properly in the near future.
Proof theorists use ordinals and their arithmetic in a more combinatorial way. I don't currently understand that, but hope that my research programme will throw some light on it.
However, Simon Henry is an accomplished categorist and will want a genuinely categorical answer.
The motivation, as I see it, for re-formulating the structures of set theory categorically is that they provide partial models of the non-existent free algebra for the covariant powerset functor. It would be useful to generalise this to other functors and other ways of presenting mathematical structures such as type theories.
Experienced categorists know that a superior understanding of a topic is not obtained by wrapping an existing symbolic argument in a few diagrams. That means that Homotopy Type Theory is not going to throw any more light on the subject than set theory does.
Starting from the beginning, Gerhard Osius represented any binary relation $(X,\prec)$ as a coalgebra$\alpha:X\to{\mathcal P} X$ for the powerset functor, with$$ \alpha(x) \;=\; \{y|y\prec x\} \quad\mbox{or}\quad y\prec x\iff y\in\alpha(x). $$Then $(\prec)$ is extensional iff $\alpha$ is mono and we can also define well-foundedness and recursion for coalgebras.
In my book, I generalised this to functors that preserve inverse images and relaxed this to just preserving monos in my recent Well Founded Coalgebras and Recursion.
This is where we can start making serious use of categorical tools, in particular replacing "monos" with a factorisation system.
That paper examined what was required of the category and factorisation system to do this. Ordinals as Coalgebras is a worked example in which we replace the powerset with the poset${\mathcal D}(X)$ of lower subsets and three different factorisation systems are considered. I wrote the Introduction of this paper this week and posted a version that is not exactly finished but more-or-less all there on my website.
The best behaved notion of ordinal, from a categorist's point of view, is the plump one. The categorical definition is that the coalgebra $\alpha$ is well founded and embeds $X$ as a lower subset of ${\mathcal D}(X)$. This means that every lower subset of $\alpha(x)\subset X$ with respect to the poset order is $\alpha(y)$ for some $y\leq x\in X$. This is much simpler than the recursive symbolic definition that I gave in Intuitionistic Sets and Ordinals in JSL in 1996.
Yes, it is true that the plump ordinals are very plump, as I demonstrate in the final section of the recent paper.
However, that was not the point of the exercise. The reason for using category theory is that the techniques are transferable to other very different situations.
So, I invite Simon and others to think of some other categories, endofunctors and factorisation systems and reproduce my results in those.
As for Homotopy Type Theory, its purpose is also to generalise the equality relation, by analogy with geometric paths. If the object that you're studying is provably a "set" (in the HoTT sense), so its notion of equality is the naive one, then you're not exploiting this powerful intuition and just messing around with symbols. So I recommend to the HoTT people that they wait to see what comes out of my categorical approach, and then devise some radically new notion of homotopy-ordinal.
All of my own work mentioned above is on my website: http://www.paultaylor.eu/ordinals/