Add Apple Screen Sharing authentication for VNC (security type 30)

Apple's macOS Screen Sharing server doesn't speak standard VNC
Authentication -- it uses a Diffie-Hellman key exchange followed by
AES-128-ECB-encrypted credentials, security type 30. Apple never
published this scheme (it's not part of RFC 6143); this implements
the well-established reverse-engineered wire format: the server sends
a generator, prime, and its own DH public key; the client generates
an ephemeral keypair, derives the shared secret, MD5-hashes it into
an AES key, and sends back its public key plus a 128-byte encrypted
username+password buffer.

The DH/AES math lives in new src/vnc_apple_dh_auth.h/.cpp as a pure,
socket-free helper (mirroring vncAuthResponse()'s shape for standard
VNC Auth), built entirely on modern EVP_PKEY-based OpenSSL 3.0 APIs --
no deprecated low-level DH_* calls, unlike VNC Authentication's
necessary use of classic DES. Reuses Profile::username (already a
shared field) since Apple's scheme needs an actual macOS account name,
unlike password-only VNC Authentication.

Security-type preference when multiple are offered is now None >
AppleDH > VNCAuth, since DH+AES is strictly stronger than static-
challenge DES. Adds a DH round-trip test (generates a real 512-bit
group at test time, computes the response, then independently
re-derives the shared secret as the server would and decrypts the
credentials back out -- proving self-consistency without needing a
hand-computed expected value), a fake-server integration test for the
full RFB 3.8 handshake sequencing, and a preference-order test.

Not yet live-verified against a real macOS Screen Sharing server --
that's next.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-09-15 21:27:17 -06:00
co-authored by Claude Sonnet 5
parent 2fe2022182
commit 1f026dde70
7 changed files with 579 additions and 16 deletions
+2
View File
@@ -132,6 +132,8 @@ set(ORBITHUB_SOURCES
src/vnc_session_backend.h
src/vnc_pixel_codecs.cpp
src/vnc_pixel_codecs.h
src/vnc_apple_dh_auth.cpp
src/vnc_apple_dh_auth.h
src/unsupported_session_backend.cpp
src/unsupported_session_backend.h
)