typo error on my behalf,
answer you have to use policy based routing on your server (if its linux its relatively easy)
on your client i meant..
search for policy based routing on linux,
basic you use kernels multiple routing table feature and using iptables mangle table
you mark the packets you want to go outside the vpn,in your case
the responses to incoming ssh sessions
Michael.