Initial SSH connection after deploy is slow

After each deploy, if I ssh into the server for the first time, it takes up to 10 seconds…
After that, it’s less than a second each time.
Why does the first SSH connection take so much longer?
Is there any way to speed it up?