This post on lists.ubuntu.com got me going in the right direction:
[ /etc/pam.d/gdm ] #%PAM-1.0 auth requisite pam_nologin.so auth required pam_env.so auth sufficient pam_listfile.so sense=allow file=/etc/passwordless item=user @include common-auth @include common-account session required pam_limits.so @include common-session @include common-password
[ /etc/passwordless ] <username>
This worked for me. Except for the fact that the line is put into /etc/pam.d/gdm it isn't a GDM specific tweak. You should be able to do the same with any service that uses PAM. I'm only using it for GDM and have GDM configured to not allow remote logins.
You may also want to disable the 'lock' feature of your screen saver for the passwordless gdm login accounts.