By default SSHD will try to resolve the DNS name for any clients that are connecting. You’ll notice that it hangs for a bit while its connecting. While could be many things, more often than not its caused by DNS. This is because SSHD looks up the reverse DNS entries of any connection clients in order to improve its logging.
If your SSH server runs in an environment where the connecting client might not have a DNS name, or the DNS server is slow to respond you will want to disable this lookup. To do this change the line that reads:
#UseDNS yes
to
UseDNS no
Then restart SSH. Next time you SSH in it should connect much faster.