Page 1 of 1

Unable to SSH to AWS instance after new installation

Posted: Fri Dec 28, 2018 6:25 pm
by ohadbp
Hey,

I installed OpenVpn AS 2.6.1 on an aws instance (Didn't use the OpenVPN AMI) -
I am able to connect to the server but can't ssh to any of the instances on my private subnets nor ssh using private IP to an instance in a public subet (ssh using public DNS in public subnet works, but i don't need VPN for that...)

Steps i did:
  • Disable source/dest check on AWS instance
  • Security Groups - I cleared all rules of security groups setting permission for any protocol on any port, open to the world just to make sure this isn't the issue
  • I checked ifconfig and validated i have the right ip as defined in vpn.server.group_pool.0 .
Any suggestion on what to check in the OpenVPN level or the AWS instance level will be appreciated.

Thanks.

Below is my current configuration:
{
"admin_ui.https.ip_address": "all",
"admin_ui.https.port": "11943",
"aui.eula_version": "2",
"auth.ldap.0.min_ssl": "tls1_2",
"auth.ldap.0.name": "Google Secure LDAP",
"auth.ldap.0.sasl_external": "true",
"auth.ldap.0.server.0.host": "ldap.google.com:636",
"auth.ldap.0.ssl_auth_cert": "/etc/ssl/certs/gldap.crt",
"auth.ldap.0.ssl_auth_key": "/etc/ssl/certs/gldap.key",
"auth.ldap.0.ssl_ciphers": "ECDHE-RSA-AES128-GCM-SHA256",
"auth.ldap.0.ssl_verify": "internal",
"auth.ldap.0.timeout": "4",
"auth.ldap.0.uname_attr": "uid",
"auth.ldap.0.use_ssl": "always",
"auth.ldap.0.users_base_dn": "OU=Users, DC=example, DC=com",
"auth.module.type": "ldap",
"auth.pam.0.service": "openvpnas",
"auth.radius.0.acct_enable": "false",
"auth.radius.0.name": "My Radius servers",
"cs.cws_proto_v2": "true",
"cs.https.ip_address": "all",
"cs.https.port": "943",
"cs.prof_sign_web": "true",
"cs.ssl_method": "SSLv3",
"cs.ssl_reneg": "false",
"cs.tls_version_min": "1.1",
"host.name": "my_hostname",
"hostname": "my_hostname",
"sa.compression_warning_shown": "displayed",
"sa.initial_run_groups.0": "web_group",
"sa.initial_run_groups.1": "openvpn_group",
"sa.ssl_lib": "openssl",
"vpn.client.basic": "false",
"vpn.client.cipher": "AES-256-CBC",
"vpn.client.config_text": "cipher AES-128-CBC",
"vpn.client.routing.inter_client": "false",
"vpn.client.routing.reroute_dns": "true",
"vpn.client.routing.reroute_gw": "false",
"vpn.client.routing.superuser_c2c_access": "false",
"vpn.daemon.0.client.netmask_bits": "24",
"vpn.daemon.0.client.network": "client_network_ip",
"vpn.daemon.0.listen.ip_address": "all",
"vpn.daemon.0.listen.port": "443",
"vpn.daemon.0.listen.protocol": "tcp",
"vpn.daemon.0.server.ip_address": "eth0",
"vpn.general.osi_layer": "3",
"vpn.server.cipher": "AES-256-CBC",
"vpn.server.config_text": "cipher AES-128-CBC",
"vpn.server.daemon.enable": "true",
"vpn.server.daemon.tcp.n_daemons": "1",
"vpn.server.daemon.tcp.port": "443",
"vpn.server.daemon.udp.n_daemons": "1",
"vpn.server.daemon.udp.port": "1194",
"vpn.server.dhcp_option.disable_nbt": "false",
"vpn.server.dhcp_option.nbt": "1",
"vpn.server.duplicate_cn": "false",
"vpn.server.group_pool.0": "group_pool_ip",
"vpn.server.port_share.enable": "true",
"vpn.server.port_share.ip_address": "1.2.3.4",
"vpn.server.port_share.port": "1234",
"vpn.server.port_share.service": "client",
"vpn.server.routing.gateway_access": "true",
"vpn.server.routing.private_access": "nat",
"vpn.server.routing.private_network.0": "my_subnet_ip1",
"vpn.server.routing.private_network.1": "my_subnet_ip2",
"vpn.server.routing.private_network.2": "my_subnet_ip3",
"vpn.server.tls_auth": "true",
"vpn.server.tls_version_min": "1.1",
"vpn.tls_refresh.do_reauth": "true",
"vpn.tls_refresh.interval": "360"
}

My SSH log:

OpenSSH_7.4p1, LibreSSL 2.5.0
debug1: Reading configuration data /Users/ohad/.ssh/config
debug1: /Users/ohad/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolving "ec2-XX-YYY-YYY-YYY.compute-1.amazonaws.com" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to ec2-XX-YYY-YYY-YYY.compute-1.amazonaws.com [XX.XXX.XXX.XXX] port 22
ssh: connect to host ec2-XX-YYY-YYY-YYY.compute-1.amazonaws.com port 22: Operation timed out

Re: Unable to SSH to AWS instance after new installation

Posted: Fri Dec 28, 2018 6:54 pm
by novaflash
When you have a problem like this, you should run packet capturing. Here's a quick start:

install tcpdump on your access server system.
run a ping from a connected vpn client to a target system that SHOULD be reachable through your access server.
run tcpdump and see if you see the pings going through.
for example
tcpdump -eni ani icmp

If you see a ping REQUEST coming in, but nothing more, the problem is that your access server is either not configured to give you access to that subnet (see vpn settings > access to private subnets > yes, using NAT, and try putting the subnet in there and clicking save settings AND update running servers).

If you see a ping REQUEST coming in, and leaving your server, then the openvpn client and openvpn server are fine and the issue is that the packet gets lost in the network behind the access server, or the response to it back to your access server gets lost somehow.

Re: Unable to SSH to AWS instance after new installation

Posted: Sat Dec 29, 2018 4:38 pm
by ohadbp
Thanks novaflash.

So i tried what you suggested and oddly enough I can't see any ping request coming in to the AS from my machine.

I can ping & ssh to the AS instance.
After connecting using my client i can see the following is added to my ifconfig:
utun1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1500
inet ip_address_assigned_bby_the_as --> ip_address_assigned_bby_the_as netmask 0xffffff00

but when i ping any instance from my machine i can't see any ping request coming in the AS.

AWS security group for the AS instance are as follow and seem to allow any connection:
Ports Protocol Source
11943 tcp 0.0.0.0/0
80 tcp 0.0.0.0/0
1194 udp 0.0.0.0/0
22 tcp 0.0.0.0/0
943 tcp 0.0.0.0/0
443 tcp 0.0.0.0/0
-1 icmp 0.0.0.0/0
All All 0.0.0.0/0, ::/0

Any thought?

Thanks.

Re: Unable to SSH to AWS instance after new installation

Posted: Sat Dec 29, 2018 4:58 pm
by novaflash
If there's no traffic from your VPN client going to the VPN server when you ping a target system from the VPN client, then you probably have a routing conflict, or there's no instruction being sent or implemented on the VPN client side to send that traffic to the VPN server. Check the client side log file and check what instructions it is getting. There's usually a line in there with the word "push" in it that contains a whole list of instructions that the VPN server is sending to the VPN client. In there it will specify what routes it has to implement.

Re: Unable to SSH to AWS instance after new installation

Posted: Sat Dec 29, 2018 5:51 pm
by ohadbp
So i can see the PUSH command as follow:
Sending PUSH_REQUEST to server...
Sat Dec 29 19:19:47 2018 OPTIONS:
0 [explicit-exit-notify]
1 [topology] [subnet]
2 [route-delay] [5] [30]
3 [dhcp-pre-release]
4 [dhcp-renew]
5 [dhcp-release]
6 [route-metric] [101]
7 [ping] [12]
8 [ping-restart] [50]
9 [auth-token] ...
10 [comp-lzo] [yes]
11 [redirect-private] [def1]
12 [redirect-private] [bypass-dhcp]
13 [redirect-private] [autolocal]
14 [redirect-private] [bypass-dns]
15 [route-gateway] [group_default_ip_address]
16 [route] [dynamic_ip_address] [255.255.255.0]
17 [route] [my_subnet_ip] [255.255.252.0]
18 [route] [my_subnet_ip2] [255.255.248.0]
19 [dhcp-option] [NBT] [1]
20 [block-ipv6]
21 [ifconfig] [192.168.254.10] [255.255.255.0]
22 [peer-id] [0]
23 [cipher] [AES-256-GCM]

I can't see any error in the log but two strange things:
1) I configured 3 subnets but only two are shown in the [route] options (#17 & #18)
2) I do have the following message "Unknown pushed DHCP option: [dhcp-option] [NBT] [1]" - Can't find anything about this in google, no i touched this setting , it must be default. Is this seem like an issue?

Re: Unable to SSH to AWS instance after new installation

Posted: Sat Dec 29, 2018 5:55 pm
by novaflash
It would help a lot if you let us know what subnets are actually in use instead of anonymizing them. Now I can't tell if there's a conflict.

Looks like your client is trying to implement those routes. Not sure if it's successful. Check the routing table before and after connecting to see if the routes are there.

If the routes are there, and when you ping an IP in the subnet that this route covers, and traffic doesn't show up on the Access Server, then the problem is entirely on the VPN client side, probably a firewall then.

Re: Unable to SSH to AWS instance after new installation

Posted: Sat Dec 29, 2018 8:03 pm
by ohadbp
So I made a few changes in the server config and finally could see ping requests coming in the AS but nothing else.

my tcpdump:
20:00:56.901899 IP ip-192-168-254-4.ec2.internal > ip-172-25-128-75.ec2.internal: ICMP echo request, id 11819, seq 9, length 64
20:00:57.901276 IP ip-192-168-254-4.ec2.internal > ip-172-25-128-75.ec2.internal: ICMP echo request, id 11819, seq 10, length 64

* I verified my private subnets are configured properly
* I test to verify i can ping & ssh from my AS to the instance on the private subnet - Both passed successfully.
* Security groups as mentioned before allow access between machines on all ports and protocols.

What else can cause this?

btw, the changes i made to fix the origin problem was :

1) Deleting two configs from AS configuration:
"vpn.server.dhcp_option.disable_nbt": "false",
"vpn.server.dhcp_option.nbt": "1",
2) Adding "vpn.server.nat.masquerade" = true to the AS config.
3) Running the command "iptables -t nat -A POSTROUTING -o eth0 -s 10.8.0.0/24 -j MASQUERADE"

Re: Unable to SSH to AWS instance after new installation

Posted: Sat Dec 29, 2018 8:44 pm
by novaflash
The NBT stuff, I believe this is only for netbios/WINS stuff, so doesn't affect this. NAT masquerade is on by default so doesn't affect this. Manually adding iptables rules is not supported, and should not have been necessary in any case. But, you do run your own Linux, and not the official image, so maybe there's some firewall program in the way that would normally not be an issue on the official image. In any case, glad to see the traffic is now coming in on the Access Server.

Do you see the ping coming in, and then going out through the appropriate network interface on your server? If not, check your settings again and remove any iptables firewalls that could possibly be interfering. At least the problem is now no longer on the client side.

Re: Unable to SSH to AWS instance after new installation

Posted: Sat Dec 29, 2018 9:24 pm
by ohadbp
Halleluja! I got it all working.
Problem was in the route table configuration of the private subnet. I needed to add a rule from 192.168.254.0/24 to the network interface of my AS instance.

Many thanks for all the help, It's greatly appreciated.