This page describes how to build proper udebs. Since I have only done this once, and I haven't even tested the result, there may be many errors here. Please correct them!

debian/control

You need to define that you're building the udeb using debian/control. It looks very much like any other binary package. Special things are:

the rest

The rest is the same as a normal package. Except that for udebs, you will probably want to strip a lot of stuff (documentation, themes, that sort of things). Udebs don't have to follow policy in many respects, but it is important that they are small.