No push notifications with Seamless tunnel during sleep

anatoli

Hi!

All devices under my administration experience the issue that with redirect-gateway and seamless tunnel, push notifications are not working as expected: instead of almost immediate delivery of push notifications via an open channel with APN servers, when sleeping the notifications are polled from the servers with 10-15 min intervals.

Needless to say, this completely breaks real-time communications (VoIP) and severely impacts semi-realtime communications (messengers, urgent mails, etc.). From the user PoV everything appears broken and I get a lot of complaints, some users even turn the VPN off. At the same time we can't disable Seamless tunnel as this would allow a lot of connections to leak outside the tunnel and would defeat the VPN purpose if not entirely, at least partially.

We've already discussed this issue with ordex. As an immediate workaround the recommendation was to try to add a route to APN servers with "route x.x.x.x y.y.y.y net_gateway". I've tried it:

route 17.0.0.0 255.0.0.0 net_gateway

and it didn't work (about the APN IPs: [1], [2], [3]).

Why this approach doesn't work? How can I debug it? The OpenVPN log shows:

NIP: adding (excluded) IPv4 route 17.0.0.0/8

but push still not working (polled with 10 min intervals).

Then, I'd like to also ask if a "proper" solution could be implemented? The problems with the workaround described above is that on the 17.0.0.0/8 network there are a lot of other services, apart from APN, like Apple Ad services and many more that we'd prefer not to route outside the VPN. And finding the exact IPs (or at least a narrow block like /24) is quite difficult as the IPs change on each DNS request and depend on the region where the device is located.

For a robust solution I see 3 possibilities (all 3 would have an option in the settings so the users can decide):

Request a permission from Apple to always stay awake (for the crypto component, as the GUI is not needed for the tunnel to work), like some internet music players and other classes of apps. In this case the tunnel would be always available (I guess this would be the easiest and the most stable solution).
Somehow detect from within the app the exact routes that should be added and add them automatically on every reconnect. Probably Apple exposes some APIs for that, or maybe there's some API that instructs push to go directly.
Implement VPN functionality as a network driver so no app would have to be running for the tunnels to work during sleep.

I don't know if it's technically feasible, especially the 3rd option. If there's nothing existing in the new VPN framework to solve the issue, probably you could contact Apple to discuss possible solutions?

Please let me know I can contribute economically to help solve this issue.

Thanks,
Anatoli

[1]: https://support.apple.com/en-us/HT203609
[2]: https://developer.apple.com/library/content/technotes/tn2265/_index.html#//apple_ref/doc/uid/DTS40010376-CH1-TNTAG41
[3]: https://stackoverflow.com/questions/10688852/ip-address-ranges-for-apns-servers

ordex

anatoli wrote We've already discussed this issue with ordex. As an immediate workaround the recommendation was to try to add a route to APN servers with "route x.x.x.x y.y.y.y net_gateway". I've tried it:
route 17.0.0.0 255.0.0.0 net_gateway
and it didn't work (about the APN IPs: [1], [2], [3]).

Why this approach doesn't work? How can I debug it? The OpenVPN log shows:
NIP: adding (excluded) IPv4 route 17.0.0.0/8
but push still not working (polled with 10 min intervals).

Have you tried sniffing your traffic to see if packets to that IP class are actually going out of the tunnel?
Should be easier to check when running the VPN over WiFi (because you can sniff the traffic on your own router).
On top of that, I guess that APNs IPs might change over time.

anatoli wrote Then, I'd like to also ask if a "proper" solution could be implemented? The problems with the workaround described above is that on the 17.0.0.0/8 network there are a lot of other services, apart from APN, like Apple Ad services and many more that we'd prefer not to route outside the VPN. And finding the exact IPs (or at least a narrow block like /24) is quite difficult as the IPs change on each DNS request and depend on the region where the device is located.

For a robust solution I see 3 possibilities (all 3 would have an option in the settings so the users can decide):
Request a permission from Apple to always stay awake (for the crypto component, as the GUI is not needed for the tunnel to work), like some internet music players and other classes of apps. In this case the tunnel would be always available (I guess this would be the easiest and the most stable solution).

The "VPN process" is already treated differently as it has the permission to continue working in background.
However, it is not granted permission to stay alive during sleep because, in my opinion, it would basically defeat the purpose of the sleep: to stay functional the entire device network stack should stay always on, thus draining the battery (instead of sleeping).

I wonder if it is possible to configure the device to never sleep when locked (?). Maybe that would be an even easier workaround.

Unfortunately this is not something easy to change, but a discussion with Apple can surely be opened, as it is their responsibility to enhance the user experience.

anatoli wrote [*]Somehow detect from within the app the exact routes that should be added and add them automatically on every reconnect. Probably Apple exposes some APIs for that, or maybe there's some API that instructs push to go directly.

Unfortunately there is nothing about push notifications in the NetworkExtension framework.

anatoli wrote [*]Implement VPN functionality as a network driver so no app would have to be running for the tunnels to work during sleep.
[/list]

Apple allows VPN extensions to be implemented only using the NetworkExtension framework. Therefore no other approach can be taken (regardless of the feasibility).

One question: have you conducted your experiments while the mobile data was ON? I know in the past iOS forced push notifications to *always* go through the WWAN, no matter where the default route was set.

anatoli

ordex, thanks for your quick reply.

ordex wrote Have you tried sniffing your traffic to see if packets to that IP class are actually going out of the tunnel?

Following your recommendation, I analyzed the packets going via WiFi and VPN, and also investigated a bit the push mechanism.

Basically, there 2 push mechanisms: the old one (UserNotifications framework) and the new one (PushKit framework, mainly intended as a low-latency interactions mechanism for messengers and VoIP + some other classes, details: [1], [2]). In both cases, the phone opens a TCP connection to APN servers and maintains it. With the 1st mechanism, when a push notification is to be delivered, APN server sends it to the phone via the TCP connection and the phone gives the user some feedback.

But, unlike user notifications, PushKit notifications are never presented to the user - they don't present badges, alerts, or sounds. PushKit notifications wake up the app (or launch it if it's not running) and give it runtime to process the notification, even if it's running in the background or the phone is "sleeping".

With the 1st mechanism, whitelisting the APN servers' IPs is enough, but for the 2nd mechanism to work, the app servers' IPs should be whitelisted too.

So, for WhatsApp I had to whitelist (on top of the 17.0.0.0/8 block) 31.0.0.0/8 and 169.0.0.0/8 networks. With these routes, WhatsApp notifications started to work normally. But only WhatsApp's. Telegram was not working with these routes, logically.

As a conclusion, for thousands of apps to have push working with Seamless Tunnel one would have to whitelist the entire internet. Needless to say, this completely defeats the VPN purpose.

At the same time, without Seamless Tunnel what actually happens is that the apps communicate freely in the background without VPN. When the user responds to the notification and a VPN tunnel is established, the already opened TCP connections of the apps stay open and go outside the tunnel. Only when the apps reestablish their connections, only then the connections start flowing via the VPN. And this, basically, defeats the purpose of redirect-gateway.

Summing it up: for redirect-gateway to work as intended, Seamless Tunnel should be turned on, and with this setup for push notifications to work VPN should be always available.

ordex wrote However, it is not granted permission to stay alive during sleep because, in my opinion, it would basically defeat the purpose of the sleep: to stay functional the entire device network stack should stay always on, thus draining the battery (instead of sleeping).

I don't know the internals of iOS networking so I may be completely wrong here, but I guess for a TCP connection to the APN servers to stay open during sleep the entire network stack should stay always on in any case. The only thing that would be added on top of this is that for each packet that goes via this APN connection the VPN encryption would have to be added/maintained. Again, I don't have a deep understanding of iOS internals, but I guess that this won't significantly impact the battery.

And, after all, even with heavy impact on the battery, the user may prefer to charge the phone more often than not be able to receive calls and messages or have to lessen the security/privacy by leaking connections with Seamless Tunnel turned off.

With wireless charging (on newer models), battery impact, no matter how heavy, is almost non-perceivable nowadays. Even if you say that with VPN always-on the battery would last only 5hs in standby without recharging, this would be an excellent solution in any case. Even with 2hs it would be an acceptable solution, but I know it'll last much longer as I can watch videos on YouTube via VPN for 2hs and the battery won't discharge more than 50% (I have an iPhone 6S).

Please let me know if I can sponsor the always-on feature development or help with it in other ways.

[1] https://developer.apple.com/documentation/pushkit
[2] https://www.sinch.com/tutorials/ios8-apps-and-pushkit/

ordex

anatoli wrote ....

Summing it up: for redirect-gateway to work as intended, Seamless Tunnel should be turned on, and with this setup for push notifications to work VPN should be always available.

Thanks for the detailed analysis. Just to come back to my original question: does your conclusion imply that even when mobile data is enabled push requests are always sent over the VPN?

anatoli wrote
ordex wrote However, it is not granted permission to stay alive during sleep because, in my opinion, it would basically defeat the purpose of the sleep: to stay functional the entire device network stack should stay always on, thus draining the battery (instead of sleeping).
I don't know the internals of iOS networking so I may be completely wrong here, but I guess for a TCP connection to the APN servers to stay open during sleep the entire network stack should stay always on in any case. The only thing that would be added on top of this is that for each packet that goes via this APN connection the VPN encryption would have to be added/maintained. Again, I don't have a deep understanding of iOS internals, but I guess that this won't significantly impact the battery.

Unfortunately the way you describe the mechanism is overly simplistic. Keeping the VPN alive would mean way more than just "encrypt/decrypt" packets as the VPN itself needs other mechanisms to be in place in order to keep its state.

However, iOS internals are not public therefore we can speculate on the "impact" of such change, but only Apple can assess what this really means.

anatoli wrote And, after all, even with heavy impact on the battery, the user may prefer to charge the phone more often than not be able to receive calls and messages or have to lessen the security/privacy by leaking connections with Seamless Tunnel turned off.

With wireless charging (on newer models), battery impact, no matter how heavy, is almost non-perceivable nowadays. Even if you say that with VPN always-on the battery would last only 5hs in standby without recharging, this would be an excellent solution in any case. Even with 2hs it would be an acceptable solution, but I know it'll last much longer as I can watch videos on YouTube via VPN for 2hs and the battery won't discharge more than 50% (I have an iPhone 6S).

I am not against any of your arguments, but we can't do much about this. This is not something that we can fix/change in the OpenVPN Connect app.

The only thing I can do is to try opening a discussion with Apple and see what they will provide in response. Maybe they can find a way to tweak this behaviour and implement a change in a future iOS release.

anatoli

ordex wrote Just to come back to my original question: does your conclusion imply that even when mobile data is enabled push requests are always sent over the VPN?

It's somehow mixed. I can't sniff mobile traffic at the moment (would require setting up own cell station) so I can't confirm if no traffic goes via mobile, but notifications don't work when both mobile and WiFi are turned on with VPN in redirect-gateway + ST mode (without the whitelisting). At the same time, I don't receive on the server incoming VPN connections from a mobile IP for the same client, so my conclusion is that the phone tries to send everything via VPN on WiFi.

Another observation is that even with the 3 whitelisted network segments, when both WiFi and mobile are turned on and the phone enters sleep, after some time the WiFi connection is turned off and the notifications stop working. My conclusion here is that the 'route x net_gateway' only adds routes for WiFi connection so when it's dropped, there are no whitelist routes on the mobile connection (or maybe they are, but they don't work as the gateway/interface changed) and the traffic is blocked.

If only mobile is turned on, the whitelisting works as expected: the routes are added for the mobile connection/interface/gateway and they stay functional indefinitely.

So, pursuing the whitelisting approach, I guess when there's a 'route x net_gateway' directive and the phone has both WiFi and mobile networks, routes should be added on both of them with different priorities to avoid conflicts when both are available.

ordex wrote Unfortunately the way you describe the mechanism is overly simplistic. Keeping the VPN alive would mean way more than just "encrypt/decrypt" packets as the VPN itself needs other mechanisms to be in place in order to keep its state.

Yeah, I simplified it on purpose, but the concept I believe is the same. If I understand it correctly, VPN on iOS works as a proxy and all packets are routed via the interface it provides, i.e. similarly to how a desktop client works, right? In this case the only *processor time* needed when the phone has no network activity is to keep the tunnel up with pings and similar low-intensity tasks. I mean, there's no constant processing in the VPN component when the phone doesn't perform network activity, just periodic pings that require light memory I/O and some crypto processing. I believe all this activity could be even counted in processor ticks, i.e. some 5K ticks every 5-10 seconds?

To reduce it even further, something more conservative could be applied. For example, today I have 5s pings with 10s restarts. In sleep mode these numbers could be increased to 60s and 75s respectively.

Compare this to listening to internet music: the entire network stack is up and running all the time, audio decoding, powering the earphones, etc. I just measured internet music reproduction via VPN on my iPhone 6S: 3% of battery per hour, i.e. 36% of battery consumption in 12hs of constant processing, full VPN (processing 40Mb/hr), powering earphones + some periodic background checks like fetching mail, etc. If I can make my day (12hs) without recharging, I'd be more than happy.

ordex wrote I am not against any of your arguments, but we can't do much about this. This is not something that we can fix/change in the OpenVPN Connect app.

Won't we achieve the intended behavior if the VPN component is granted the permission to stay always on? It could detect the moment when the phone turns off WiFi for power saving and reconnect on a mobile connection.

ordex wrote The only thing I can do is to try opening a discussion with Apple and see what they will provide in response. Maybe they can find a way to tweak this behaviour and implement a change in a future iOS release.

That would be great!

ringcaptcha

I have faced the same issue while developing my app: http://www.ringcaptcha.com/ (react-native-phonecall npm). When my phone moved from wifi to the cellular network, the connection gets lost but, it could receive the SIP calls. When I switch from cellular layer network to the internet network, I needed to re-register to the application. You can shut down the wifi setting and then try to start the process. Another problem can also be the VPN. As the voice over internet protocol runs on the internet layer!