have an application on windows server 2008 R2 that sends data to my PC through port 1600, my ISP blocks this port and almost all ports,
if i installed OpenVpn on the server and my PC as client, would that create virtual IP tunnel and the ports will go through software layer and no longer go through ISP?
This depends completely on how the server is configured; do you control the OpenVPN sever? If not, please go ask the "provider" as there are countless ways the addressing and firewall can be setup, and since none of us work for your VPN "provider" we obviously cannot know this.
In particular, with RFC1918 IP space, NAT is required to open of "port forward" these ports since your VPN client does not have a globally routable IP address (RFC1918 address, for all proper purposes, "do not exist" on the Internet.) Since this would need to be handled uniquely server-side, you can assume it will not work unless you make special arrangements with your VPN "provider."
Unless you manage the server, you will need to get in touch with the company that has sold you RFC1918 connectivity to the Internet and inquire with them.
[Edited to add]
Please note that if you are establishing client-initiated connectivity and using OpenVPN's
--redirect-gateway option, that this will route your IP traffic through the server's VPN endpoint; when combined with either a public IP held by the VPN client, or a NAT overload ("source nat") setup, this can masquerade your traffic as originating from the server's address space.
