These sources are maintained (contrary to popular belief) not by Linus Torvalds himself, but by Marcelo Tosatti. Linus is the leader of active kernel development, but as he is only one man, he passes off the maintenance of the stable kernel branch to someone he can trust to handle it once it has stabilized. Thus, Alan Cox became the maintainer of the Linux-2.2 kernel series and Marcelo Tosatti became the maintainer of the Linux-2.4 kernel series. This is what all the other patch sets in the 2.4 series are based on. Marcelo has been doing an outstanding job with its maintenance and it can be counted on for stability and up-to-date (if not bleeding edge) hardware support.
Vanilla sources are probably the most stable sources available since they are the most tested and all possible kernel sources are based on them. If you don't need any of the extras that the other kernels supply then the vanilla-sources are your thing.