Cpu can be a limiter if you are using it on say
CPU almost free at server side.
Your connection to the actual vpn can affect speed too.
When I try to download a file through PPTP speed was "2.8 or 2.5MB/sec" but when i tried the same thing from same location via OpenVPN speed was 190.4 or 184KB/sec it is big difference.
Regards,
Jebran.