Add Apple Screen Sharing authentication for VNC (security type 30)

Apple's macOS Screen Sharing server doesn't speak standard VNC
Authentication -- it uses a Diffie-Hellman key exchange followed by
AES-128-ECB-encrypted credentials, security type 30. Apple never
published this scheme (it's not part of RFC 6143); this implements
the well-established reverse-engineered wire format: the server sends
a generator, prime, and its own DH public key; the client generates
an ephemeral keypair, derives the shared secret, MD5-hashes it into
an AES key, and sends back its public key plus a 128-byte encrypted
username+password buffer.

The DH/AES math lives in new src/vnc_apple_dh_auth.h/.cpp as a pure,
socket-free helper (mirroring vncAuthResponse()'s shape for standard
VNC Auth), built entirely on modern EVP_PKEY-based OpenSSL 3.0 APIs --
no deprecated low-level DH_* calls, unlike VNC Authentication's
necessary use of classic DES. Reuses Profile::username (already a
shared field) since Apple's scheme needs an actual macOS account name,
unlike password-only VNC Authentication.

Security-type preference when multiple are offered is now None >
AppleDH > VNCAuth, since DH+AES is strictly stronger than static-
challenge DES. Adds a DH round-trip test (generates a real 512-bit
group at test time, computes the response, then independently
re-derives the shared secret as the server would and decrypts the
credentials back out -- proving self-consistency without needing a
hand-computed expected value), a fake-server integration test for the
full RFB 3.8 handshake sequencing, and a preference-order test.

Not yet live-verified against a real macOS Screen Sharing server --
that's next.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
ksmithandClaude Sonnet 5 committed 2026-09-15 21:27:17 -06:00
1 parent 2fe2022182
commit 1f026dde70
7 files changed
+579 -16

No files matched your search

+44
View File
@@ -0,0 +1,44 @@
#ifndef ORBITHUB_VNC_APPLE_DH_AUTH_H
#define ORBITHUB_VNC_APPLE_DH_AUTH_H
#include <QByteArray>
#include <QString>
// Apple's Screen Sharing authentication scheme (RFB security type 30):
// Diffie-Hellman key exchange followed by AES-128-ECB-encrypted
// credentials. Apple never published this officially -- it's not part of
// RFC 6143 -- so this implements the well-established reverse-engineered
// wire format used by several independent VNC clients, not a primary
// spec. Kept as a pure, state-free helper (no socket access) so it's
// unit-testable without a live connection, mirroring
// VncSessionBackend::vncAuthResponse()'s shape for standard VNC
// Authentication.
namespace VncAppleDhAuth {
struct Response {
// Same byte length as the server's prime, big-endian, zero-padded.
// Empty on any failure (malformed input, an OpenSSL operation
// failing) -- callers should treat an empty clientPublicKey as "could
// not compute a response" rather than send a degenerate one.
QByteArray clientPublicKey;
// Always exactly 128 bytes on success (16 AES blocks): a 64-byte
// NUL-padded/truncated username followed by a 64-byte NUL-padded/
// truncated password, AES-128-ECB encrypted (no padding, since the
// plaintext is already an exact multiple of the block size) with a
// key derived as MD5(sharedSecret).
QByteArray encryptedCredentials;
};
// Computes the DH keypair, the shared secret, the derived AES key, and
// the encrypted credential blob, using modern EVP-based OpenSSL 3.0 APIs
// throughout (no deprecated low-level DH_*/legacy calls -- unlike VNC
// Authentication's classic DES usage, none of MD5/AES-ECB/the EVP_PKEY DH
// APIs are deprecated, so no compatibility pragma is needed here).
// Returns a Response with an empty clientPublicKey on any failure.
Response computeResponse(const QByteArray& generator, const QByteArray& prime,
const QByteArray& serverPublicKey, const QString& username,
const QString& password);
}
#endif