A Deeper Look

A Deeper Look

How Google Cloud IAP Uses WebSockets for SSH Tunneling

A deep dive into the IAP Relay protocol, how we can use it, and, teaching you how to build a custom SSHSocket adapter along the way!

—

Contents

📚 Intro

I truly believe that Google's Identity Aware Proxy (Cloud IAP) is one of the coolest products it has created. The concept, at its core, is simple - let us make TCP connections on any port tunnelled through HTTP(s) instead. Why is that useful? Well, consider this…

🔥 Beware! Firewalls About!

Have you ever needed to SSH into a Google Cloud Virtual Machine (VM) that doesn’t have a public IP address? Maybe it’s sitting securely in a private VPC, shielded from the scary dangers of the internet. In the past, you might have set up bastion hosts, used a service like Guacamole or managed fiddly VPN connections.

While those all work, they introduce their own management overhead and potential security risks. So then, what if you could securely connect directly from your machine, using your standard Google identity, without opening any firewall ports on your VM?

SSH to Private GCP VMs_ Implementing IAP Tunnels via WebSockets - Page 2.png

We can’t just connect to a service in Google Cloud - GCP gives us defence-in-depth and blocks this at many different layers. So what do we do then?

This is exactly where Google Cloud’s Identity-Aware Proxy (IAP) TCP Forwarding comes in. It acts as a secure gateway, authenticating and authorising your connection before relaying it to your private VM.

SSH to Private GCP VMs_ Implementing IAP Tunnels via WebSockets - Page 2 (1).png

But how does it actually work, especially when you click the SSH via Browser button in the Cloud Console, use gcloud compute ssh --tunnel-through-iap, or Google’s own iap-desktop.

It turns out, under the hood, it often uses WebSockets! But getting SSH and other protocols, which expect a raw TCP stream, to work over the message-based WebSocket protocol requires some clever adaptation.

In this post, we’ll peel back the layers:

  1. 🧠 The Magic Postbox: A simple analogy for secure relaying.

  2. ✅ The Core Problem & Solution: Why we need IAP, WebSockets, and custom code.

  3. 🧑‍🔬 The Deep Dive: How we can implement our own custom SSH socket in an example using a Dart/Flutter SSH client via the IAP Relay protocol.

Let’s tunnel in…

🪄 Level 1: The Magic Postbox - SSH Without Opening Doors

Imagine your computer is you, and the private Google Cloud VM is your friend living in a super-secure apartment building (a private network) with no publicly listed address or doorbell. You want to send them secret coded messages (SSH).

⚠️ The Problem

They have no address, no front door, they have no mailbox, they have no clear way to communicate with them. You can’t just walk up and knock (no public IP, firewalls block you). So how do we send them a nice little letter?

SSH to Private GCP VMs_ Implementing IAP Tunnels via WebSockets - Page 2 (2).png

How do you send a message to a friend with no front door, no visible mailbox and no street address?

Well, we know that our friend exists, and that they live inside a really safe building surrounded by many secure walls and security guards. This super safe building has a special way for outsiders to communicate with the people inside. We call this The Magic Postbox (Identity Aware Proxy - or, IAP for short).

Google provides a special, super-secure Magic Postbox (IAP) at the building’s main gate. Only residents/known visitors (authorised Google users) with a special key (your Google login/OAuth Token) can use it. The way they use is through a special Tube.

The Tube (also known as: WebSockets) lets you connect a private, always-open tube (in this case a secure HTTPS protocol known as a WebSocket) from your computer directly to the magic postbox.

  1. Why a tube? It stays connected for fast back-and-forth messages, perfect for getting our message through and getting responses back. It also uses standard connections (port 443) that easily pass through most secure walls (firewalls).

  2. How is it secure? It uses HTTPS encryption so no one can read messages in the tube (encryption in-transit).

So we start passing our tube through the wall, but, next up the security guards make sure you’re allowed to send messages to your friend at the Gate Security Check (IAP Authentication & Authorisation). The postbox guard checks your key (access token) to make sure it’s really you (Authentication) and then checks a list (IAM) to ensure you have permission to send messages specifically to your friend’s apartment (Authorisation).

“Ready!” Signal (IAP Handshake/SID). If all checks pass, the guard sends a signal back: “Okay, path is open! Your delivery code is [SID]”.

  1. Why the signal? Tells your app it’s safe to start sending because the tube’s been passed all the way through.

  2. Why the code? Helps reconnect quickly if the tube has a breakage anywhere.

But we have a problem… We actually want to send our friend a glass of water as he’s really thirsty. This message won’t fit in the tube as it only allows messages (WebSocket channel packets) and absolutely no liquids (SSH streams).

We need to split our big glass of water into small little sealed Packages (IAP Relay Protocol - DATA/ACK).

SSH to Private GCP VMs_ Implementing IAP Tunnels via WebSockets - Page 4.png

Sending streams of water in our tube is not only logistically tricky its also very much frowned upon…

So no we pour our water into a labeled, sealed vials/packages (DATA messages). The label printed on each package says “This is water (SSH data)” and how much water in each package in millilitres (packet size).

  1. Why packages/vials? These tubes send messages (WebSocket packets), not continuous streams of water.

  2. Why labels? Keeps things organised.

  3. Why small? Keeps interactive typing fast). It also helps the guard sends back receipts (ACK messages) confirming which packages arrived safely.

  4. Why receipts? Ensures no part of your message gets lost.

In order to get these glasses of water into vials/packages without spilling anything we need to make use of The Funnel (Custom Socket Adapter). This funnel is able to take our glass of water (a “continuous message stream” of SSH) and pour it into evenly sized vials.

This is a very special custom made funnel that we’ve had to build ourselves (A custom SSH socket called IapSshTransportSocket).

It sits between your glass of water and the tube. It takes the stream, puts it into packages/vials (DATA), sends them, but is also able to receives packages/vials and unpack them with receipts (ACKs), and is able to give the stream back to your glass of water.

Why the funnel? Our glass of water only understands pouring and we can’t just pour into the tube (we need to hide some of the complex delivery system from our SSH program).

SSH to Private GCP VMs_ Implementing IAP Tunnels via WebSockets - Page 4 (1).png

The Funnel takes my glass of water and makes sure that it gets broken down into a way that is both allowed in the tube and fits into the tube.

So what are the security guards doing through all of this? Well the tube doesn’t go straight to your friend, the tube first lands in the mailroom and the guards handle The Mailroom Relay (IAP). The guard at the postbox takes your package, removes the (SSH) message piece, and sends it via the building’s internal mail (TCP port 22) to your friend’s apartment (a VM hosting an SSH server). Also, they do the reverse for replies.

SSH to Private GCP VMs_ Implementing IAP Tunnels via WebSockets - Page 5.png

Happy campers - I can send streams of water to my friend using my custom built funnel!

✅ Level 2: Relaying SSH - Why WebSockets, and Why Custom Code?

Okay, let’s get a little more technical with our terms.

The standard/classic SSH protocol has stood the test of time, but it relies on a direct TCP (Transmission Control Protocol) connection stream — a reliable, ordered, two-way byte stream between the client and server (usually on port 22). But our private VM blocks direct TCP connections from the internet.

SSH to Private GCP VMs_ Implementing IAP Tunnels via WebSockets - Page 5 (3).png

🚪 Enter IAP TCP Forwarding

  • What is it? A managed Google Cloud service acting as a secure proxy or gateway. (Reference: IAP Concepts).

  • Why use it? It allows users to connect to internal TCP services (like SSH) without exposing those services directly. Access is controlled by Google’s identity system, not network rules alone.

  • How (Authentication/Authorisation)?

    • Your app authenticates to Google using OAuth 2.0 (typically via Google Sign-In). It gets an access token.

    • When connecting to IAP, this token is sent in the Authorization: Bearer <token> HTTP header.

    • IAP verifies the token with Google’s identity service.

    • IAP checks Cloud IAM to see if the verified user has the roles/iap.tunnelResourceAccessor permission on the target VM or project.

🧪 Why Use WebSockets for the Tunnel?

Once authenticated and authorised, IAP needs to establish a communication channel back to your app.

  • The protocol that allows for this is WebSocket (RFC 6455)

    • This standard protocol provides a way to establish a persistent, full-duplex (bidirectional) message channel over a single TCP connection, initiated via an HTTP/S handshake. Its features include:

      1. Firewall Traversal - It uses standard HTTPS (port 443), which is almost always allowed through firewalls. Trying to make IAP listen on custom ports for different users would be a network security nightmare.

      2. Persistent & Bidirectional - Unlike standard HTTP requests, the connection stays open, allowing the server (IAP) to push data back to the client efficiently, crucial for the interactive nature of SSH.

      3. TLS Security - Running over wss:// means the entire WebSocket communication is encrypted using TLS (Transport Layer Security), providing confidentiality and integrity.

❌ The Adaptation Challenge

SSH to Private GCP VMs_ Implementing IAP Tunnels via WebSockets - Page 6 (1).png

There’s a mismatch!

  • The protocol of SSH - and for examples in this blog here we are using dartssh2 - expects a Socket-like object providing a continuous byte stream.

  • A WebSocketChannel provides a message stream (discrete frames/messages).

  • Furthermore, IAP requires a specific framing protocol inside the WebSocket messages to manage the SSH data and control information (like acknowledgements).

🏗️ Why the Custom Socket Adapter (IapSshTransportSocket)?

We can solve this however with a custom socket that takes our bytestream and converts it into distinct network packets, and vice-versa.

This class is the crucial Adapter Pattern implementation.

  • It Implements

    • SSHSocket

    • This presents the interfaces that dartssh2 and other SSH sockets expects (stream, sink, done, disconnect/destroy).

  • It Wraps

    • WebSocketChannel

    • Internally, it manages the real network connection over the dialled WebSocket to Google’s IAP proxy relay.

  • It Translates

    • It converts the byte stream dartssh2 tries to write into framed IAP Relay v4 DATA messages sent over the WebSocket. It receives IAP Relay v4 messages from the WebSocket, decodes them, extracts the SSH payload from DATA messages, and feeds it into the stream dartssh2 reads. It handles IAP ACK messages internally.

SSH to Private GCP VMs_ Implementing IAP Tunnels via WebSockets - Page 6 (2).png

🏃‍♂️ Why the IAP Relay Protocol (Framing & ACKs)?

Simply dumping raw SSH bytes into WebSocket messages wouldn’t work reliably or allow session management.

IAP uses a subprotocol (negotiated via the Sec-WebSocket-Protocol: relay.tunnel.cloudproxy.app header) with specific message types:

  • DATA Messages: Package chunks of the SSH byte stream with a type Tag (4) and a Length indicator. This framing is necessary for the message-based WebSocket transport. The 16KB limit balances overhead vs. latency.

  • ACK Messages: Provide end-to-end (app-to-IAP) acknowledgement of received payload bytes (Tag 7). This ensures both sides know what data has arrived successfully, essential for IAP’s state management and its ability to resume connections if the WebSocket drops temporarily.

🔬 Level 3: Deep Dive - Implementing the IAP SSH Relay in Dart

Now let’s get a little bit deeper again and look at the specific implementation details within our IapSshTransportSocket, comparing its methods to a standard TCP socket wrapper.

This should also provide a framework on how to build your own Socket adapter. Below is a flow diagram of the broad strokes dataflow of this process:

SSH to Private GCP VMs_ Implementing IAP Tunnels via WebSockets - Page 8 (1).png

1️⃣ Establishing the Connection (connect method)

  • Standard TCP Socket

    • Would call Socket.connect(vmIp, 22), perform TCP handshake directly with the VM. Future completes on TCP ACK. SSH auth follows over the raw stream.

  • Our Implementation IapSshTransportSocket.connect

    • Builds the: wss://tunnel.cloudproxy.app/v4/connect?project=...&zone=...&instance=...&port=22... URL.

    • Uses dart:io’s WebSocket.connect to initiate the TLS handshake and HTTP upgrade request. (Reference: WebSocket class)

    • Crucially passes specific headers:

      headers.dart
      final headers = {
        'Authorization': 'Bearer $_accessToken', // OAuth2 Authentication (RFC 6750)
        // Origin header removed/nulled after debugging 'invalid origin'
        'User-Agent': 'IAP-Flutter-Client/1.0', // Client identification
        }; // And
      protocols: final protocols = ['relay.tunnel.cloudproxy.app']; // Subprotocol Negotiation (RFC 6455 Sec 4)

    • Waits for IAP Handshake

      • Listens on the resulting WebSocketChannel for the first message. It must be an IAP Relay v4 CONNECT_SUCCESS_SID message (Tag=1).

    • Why SID first?

      • This confirms IAP-level Authentication, IAM Authorization, and successful backend TCP connection before the adapter signals readiness to dartssh2.

    • // Conceptual logic inside _handleWebSocketMessage for handshake:

    logic.dart
      if (!_sidReceivedCompleter.isCompleted) {
        if (tag == sshRelayMessageTag.CONNECT_SUCCESS_SID) {
           _sid = SshRelayFormat.decodeConnectSuccessSid(messageBytes);
           print('IAP <<< CONNECT_SUCCESS_SID ($_sid)');
           _sidReceivedCompleter.complete(); // Signal connect() can finish
        }
        else {
          _closeWithError(Exception('Expected CONNECT_SUCCESS_SID, got $tag'), "Handshake Error");
        } return;
      }
    • Only completes its own _connectCompleter (and thus the connect() future) after the SID is successfully received.

2️⃣ Sending SSH Data (sink getter -> _handleOutgoingSshData)

  • Standard TCP Socket

    • sink.add(bytes) writes raw bytes directly to the OS socket buffer for TCP processing.

  • Our ImplementationIapSshTransportSocket._handleOutgoingSshData

    • Triggered when dartssh2 calls _outgoingSshDataController.sink.add(sshBytes).

    • ACK Logic

      • First, it might send an IAP ACK for data received from IAP (using _sendAckIfNeeded).

    • IAP DATA Encoding

      • Encodes the sshBytes into the IAP Relay v4 format:

        ssh_relay_format.dart
        static Uint8List
        encodeData(Uint8List payload) { // ... checks size <= 16KB ...
          var message = BytesBuilder(); // Tag (2 bytes, Big Endian)
          message.add(encodeTag(SshRelayMessageTag.DATA).buffer.asUint8List()); // Length (4 bytes, Big Endian) - Payload length only var
          lenBuffer = ByteData(_lenLength); lenBuffer.setUint32(0,payload.length, Endian.big);
          message.add(lenBuffer.buffer.asUint8List()); // Payload
          message.add(payload); return message.toBytes();
        }

      • Why Big Endian?

        • Essential Network Byte Order (RFC 1700) standard for cross-platform compatibility. Using Endian.little would break communication with Google’s server. This is a necessary implementation detail. Using Dart’s ByteData with Endian.big is the standard way to achieve this. (Reference: Endian class)

      • Why Tag=4, Length=4B?

        • Defined by the (unfortunately undocumented) IAP Relay v4 protocol. Necessary for IAP to parse the message. Length field ensures receiver knows payload boundary.

    • WebSocket Send then sends the entire encoded IAP message via _channel.sink.add().

    • State Update is in charge of increments _bytesSent (tracking payload bytes).

3️⃣ Receiving SSH Data (stream getter -> _handleWebSocketMessage)

  • Standard TCP Socket

    • stream emits raw bytes received from the TCP connection.

  • Our Implementation IapSshTransportSocket._handleWebSocketMessage

    • Listener for _channel.stream.

    • IAP Message Decoding

      • Receives a WebSocket binary frame (an IAP message). Decodes the Tag.

    • DATA Processing (Tag=4)

      • Decodes the 4-byte Length.

      • Extracts the Payload bytes.

      • Pushes only the Payload onto the _incomingSshDataController (which feeds the stream getter). dartssh2 receives these bytes.

      • Updates _bytesReceived.

      • Triggers sending an ACK back to IAP.

      // Conceptual logic: case SshRelayMessageTag.DATA: final payload = SshRelayFormat.decodeDataPayload(messageBytes); _bytesReceived += payload.length; _incomingSshDataController.add(payload); // Forward ONLY payload _sendAckIfNeeded(); // Acknowledge receipt to IAP break;

    • ACK Processing (Tag=7)

      • Decodes the 8-byte AckValue.

      • Updates _lastAckReceived.

        • A Full implementation will prune internal resend buffer.

      • Crucially, does NOT forward anything to dartssh2.

      case SshRelayMessageTag.ACK: final ackValue =
      SshRelayFormat.decodeAckValue(messageBytes); _lastAckReceived =
      ackValue; print('IAP <<< ACK ($ackValue bytes)'); // DO NOTHING MORE FOR SSH CLIENT break;

      • Why 8 bytes for ACK?

        • Prevents counter wraparound over long sessions, vital for IAP’s internal state management and reconnect reliability. Necessary design choice for robustness.

  • Why the Difference?

    • Must de-frame the IAP protocol, separate control (ACK) from data (DATA), and only provide the application-level data (SSH bytes) to the dartssh2 library via the stream.

4️⃣ Closing (disconnect / destroy methods)

  • Standard TCP Socket

    • Initiates TCP FIN handshake via OS calls (socket.close()). destroy() might close abruptly.

  • Our Implementation IapSshTransportSocket:

    • Sets _isClosing flag (standard practice for idempotency).

    • Calls _channel.sink.close(code, reason) to perform the WebSocket Closing Handshake (RFC 6455 Sec 5.5.1). This involves sending a WebSocket CLOSE frame. Necessary for clean WebSocket teardown.

    • Cancels subscriptions, closes StreamControllers (standard Dart resource management).

    • Completes _closeCompleter (signals done).

    • destroy added for compatibility with dartssh2, simply calls disconnect.

5️⃣ Signaling Done (done getter)

  • Standard TCP Socket: Future tied to OS socket closure.

  • **Our Implementation

IapSshTransportSocket:** Returns _closeCompleter.future.

  • Why?

    • This provides explicit control.

    • The adapter isn’t “done” until the WebSocket is closed and internal resources (controllers, subscriptions) are cleaned up.

    • Using a Completer is the standard Dart way to manage the lifecycle of such asynchronous operations. Necessary for dartssh2 to know when the transport is fully terminated.

🎥 The End

Connecting SSH via IAP WebSockets isn’t just a simple pipe or tube. It requires bridging the gap between the stream-based expectations of an SSH library (dartssh2) and the message-based, authenticated, custom-protocol reality of the IAP WebSocket tunnel.

SSH to Private GCP VMs_ Implementing IAP Tunnels via WebSockets - Page 7.png

Our new IapSshTransportSocket acts as this vital adapter. It faithfully implements the SSHSocket interface for dartssh2 while internally handling:

  1. Secure WebSocket connection establishment (WSS, TLS, Auth Headers, Subprotocol).

  2. The initial IAP handshake (CONNECT_SUCCESS_SID).

  3. Encoding SSH data into IAP Relay v4 DATA frames (Tag/Length/Payload, Big Endian).

  4. Decoding IAP DATA frames and extracting the SSH payload.

  5. Managing the IAP ACK mechanism for reliability and synchronisation.

  6. Proper WebSocket closure.

Understanding these layers — why WebSockets are used (network traversal, security), why a custom protocol exists inside (framing, ACKs, session management), and why the adapter pattern is necessary (interface mismatch) — is key to building and debugging clients like this Flutter application. While

Google’s specific IAP Relay v4 protocol isn’t publicly documented so this has been a journey to get here, but, by observing official clients and handling errors, we can and have successfully implemented compatible TCP-over-WebSocket connections.