Link: https[://]mullvad[.]net/en/blog/why-we-still-dont-use-includeallnetworks
Our users often ask why we do not use the includeAllNetworks to fix all possible leaks on iOS. This blog post aims to explain why this currently is not possible.
As per Apple's documentation and several vulnerability reports (e.g. TunnelCrack) , setting includeAllNetworks to true (and possibly excludeLocalNetworks too) will prevent traffic from leaking from the tunnel. These flags tell iOS that the VPN app expects all traffic to be routed through it. On other platforms, this would normally be achieved by using the system firewall and, to improve UX, by changing the routing table - superficially setting just one flag seems like a great improvement to the developer experience. The documentation for this flag explains what type of traffic will and will not be excluded, but lacks any further detail.
The reason as to why have we not set this flag in our iOS app is because it does not quite work. It breaks various behaviors the app was relying upon - for some things we have found workarounds, but there is an especially bad one that we cannot work around.
What follows is a deeply technical walkthrough of our challenges with the includeAllNetworks flag. If you care not for the technical details, the short answer is - if we were to enable the flag today, the app would work fine until it would be updated via the AppStore, at which point the system would lose all network connectivity. The most intuitive way of fixing this is to restart the device. As far as we know, there is no way for our app to detect and in any way help work around this behavior.
The beginnings of includeAllNetworks
Our iOS app, much like all of our other VPN client applications, uses ICMP packets to establish whether a given tunnel configuration is working or not. When using DAITA or quantum-resistant tunnels, the app will also need to establish a TCP connection to a host only reachable through the tunnel. Both of these two network connections are done by the tunnel process - on iOS the VPN connection is managed by a separate process from the one that users interact with. In the ICMP case, we use a regular socket() syscall to create an ICMP socket to our gateway at 10.64.0.1. For the TCP connection, we initially used a now deprecated NWTCPConnection. To not leak this traffic outside of the tunnel, we attempt to bind these sockets to the tunnel interface. These work as expected when includeAllNetworks is not in use, but when we set the flag, they just stopped working. No errors were reported from sendmsg, the best feedback we got was that the NWTCPConnection's state never updated away from waiting. When experiencing misbehavior like this, it is almost always a sure bet to assume that we are misusing whatever interface we are trying to use. Apple is not guaranteeing that regular BSD sockets will just work, and since we're trying to reach 10.64.0.1 via the in tunnel TCP connection, maybe it has some weird behavior if it's a 10/8 address?
Could we do without ICMP and TCP traffic from the tunnel process?
Yes, we can change our code to not rely on ICMP and TCP, even if it just to run our experiments. So, when we choose to just not send ICMP traffic and assume that the tunnel is always working, the VPN connection just works. You can open up Safari and browse the internet, watch videos, browse social media, send pings to 10.64.0.1 via a terminal emulator. Hold that thought - when connected via our app, the device is capable of sending ICMP traffic to our gateway via other applications. But our own app is not able to do so.
Holding it harder
We have established that we cannot send ICMP traffic the usual way from the packet tunnel process, and we cannot use the NWTCPConnection from the Network Extension framework to send TCP traffic from the tunnel, a class specifically created to facilitate VPN processes to send traffic inside their own tunnels. We could feasibly come up with a different strategy of inferring whether a given WireGuard relay is working without ICMP, but we do need TCP for negotiating ephemeral peers for DAITA and quantum-resistance. In iOS 18, one can construct a NWConnection with NWParameters with requiredInterface set to the virtualInterface of the packet tunnel - this should create a working connection from within the tunnel process. It does as long as includeAllNetworks flag is set to false. Otherwise, we are observing the exact same behavior as before. This would only make the app work on iOS 18, so it is not an entirely viable solution to our woes, at the time of writing, we are trying to support iOS 15.
What even is a packet tunnel?
There are various different Network Extensions that an iOS app can provide - the one we are using is a Packet Tunnel provider. It provides a way for a developer to read all user traffic to then encrypt it and send it off, and conversely, to write back packets received from the tunnel. To start one, the main app has to create a VPN profile - the profile contains the configuration object where includeAllNetworks can be set. The configuration can be updated with a tunnel running, but the tunnel needs to be shut down and restarted for changes to take effect. Once the VPN process is started, it must signal to the system that it is up and then, to actually move traffic, it should start reading user traffic via packetFlow or, as most VPN applications using WireGuard in the wild do, directly from the utun file descriptor.
In practice, when an app on the device tries sending something on the network, an app implementing a Packet Tunnel provider will end up reading the traffic. When our VPN process is trying to send traffic inside the tunnel, it is essentially trying to write some data into one pipe (NWConnection) and expecting to see it come out of the packet tunnel. We configure our packet tunnel provider with includeAllNetworks = true we are not seeing that traffic coming through. We can see that other processes are able to send traffic to those same hosts. We have to conclude that something is preventing our VPN process from reading traffic that it itself is trying to send.
Holding it even harder
When the VPN process is trying to send traffic to a host within the tunnel, it feels redundant to put something into a pipe to then turn around and read it back out. Could we not just construct the packets ourselves and handle them the same way we would handle them if they were read out from the packet tunnel? Yes we can, we already do this for UDP traffic for multihop, and we can trivially do this for ICMP too. Supporting TCP is a lot more complicated than just adding a header to a payload, but, we already are using WireGuard and the canonical WireGuard implementation on iOS is wireguard-go, which, for testing, already pulls in a userspace networking stack. Since we need at most 2 TCP connections per tunnel connection, performance is not a concern, we can rely on gvisor's gonet package to give us a lovely Go interface for creating TCP connections in userspace. We can then mux between the real tunnel device and our virtual networking stack. After all of that, we can reach a TCP service hosted inside our tunnel from our own tunnel process. This works, and we have tested this for quite some while. We are already using this mechanism in our released app, the TCP and ICMP traffic is already sent via the userspace networking stack. Yet we still are not using the includeAllNetworks flag. Why not?
Locking in an app version
When regular applications use NWConnections, they should wait until their NWConnection's state is set to ready. When a VPN profile is active and it has been configured with includeAllNetworks = true, the connections will only become ready when the VPN process signals to the system that it is up. When a user clicks the connect button in our application to, we start our VPN tunnel, but we also configure it to be started on-demand so that if the device reboots or if the packet tunnel crashes for whatever reason, it should be started up again as soon as any traffic is trying to reach the internet.
The behavior described above intersects horribly with app updates. We have not done a deep investigation to understand the details of an update process, but superficially we can observe the following. When includeAllNetworks = false, the process goes like this:
- Update is initiated (by user or automatically, Xcode or App Store)
- Old packet tunnel process is sent a SIGTERM
- New app is downloaded
- New app is installed
- New pac...
Content cut off. Read original on https://old.reddit.com/r/mullvadvpn/comments/1jk8nfo/why_we_still_dont_use_includeallnetworks_blog/
