Fix Apple DH auth wire-order bug, add type 33 fallback, fix missing VNC password prompt

Live testing against a real macOS Screen Sharing server surfaced two
real bugs, independent of each other:

1. SessionTab::requestConnectOptions() never prompted for a password
   on VNC profiles (only SSH/RDP) -- every VNC connection went out
   with an empty password regardless of what the server needed. VNC
   now gets its own prompt; an empty password is allowed through
   (unlike RDP's hard requirement) since no-auth VNC servers exist and
   there's no way to know client-side before the security-type
   negotiation happens.

2. VncSessionBackend's Apple DH (type 30) response sent the client's
   public key before the encrypted credentials. Cross-checking against
   neatvnc's rfb-proto.h (an independent, authoritative reference: both
   the wire struct definitions and the full server-side verification
   code, matched field-by-field against this implementation) showed
   the correct order is credentials first, then public key -- exactly
   backwards from what was implemented. Fixed, with a new regression
   test that decrypts the credentials back out using the trailing
   public-key bytes to derive the shared secret, which would fail if
   the fields were swapped again.

Also adds security type 33 (RSA + AES, src/vnc_apple_rsa_auth.h) as a
fallback Apple auth scheme, sourced from the `asyncvnc` PyPI package.
Preference when multiple are offered: None > AppleDH(30) >
AppleRSA(33) > VNCAuth(2).

Neither scheme has been gotten working live yet against the specific
macOS Tahoe (26.6.2) server available for testing -- type 30's wire
format is now verified correct byte-for-byte against the independent
reference above, but the server still rejects it with a generic
"Authentication or authorization failure"; type 33 is rejected even
earlier, right after the initial host-key request. macOS Tahoe was
released after this assistant's knowledge cutoff, so there may be a
protocol or permission-model change specific to it that isn't
reflected in either reference. Documented as an open issue in
docs/PROGRESS.md rather than claimed as working.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-09-16 03:49:06 -06:00
co-authored by Claude Sonnet 5
parent df2b1a8d50
commit e80fe7d634
9 changed files with 511 additions and 76 deletions
+51
View File
@@ -0,0 +1,51 @@
#ifndef ORBITHUB_VNC_APPLE_RSA_AUTH_H
#define ORBITHUB_VNC_APPLE_RSA_AUTH_H
#include <QByteArray>
#include <QString>
// Apple Screen Sharing's RSA-based authentication scheme (RFB security
// type 33, sometimes called "MacAuthentication" or "ARD authentication").
// Distinct from security type 30 (Diffie-Hellman + AES, see
// vnc_apple_dh_auth.h): modern macOS advertises both, but empirically only
// type 33 is actually functional -- type 30 appears to be vestigial.
// Neither is part of RFC 6143; this wire format and crypto shape was
// confirmed against the `asyncvnc` PyPI package's implementation (a real,
// working, independently-maintained VNC client) rather than derived from
// official Apple documentation, which doesn't exist for this scheme.
//
// Scheme: the server hands the client its RSA public key (DER-encoded
// X.509 SubjectPublicKeyInfo); the client generates a random AES-128 key,
// encrypts the username+password with it, then RSA-PKCS1v1.5-encrypts
// that AES key with the server's public key and sends both back.
namespace VncAppleRsaAuth {
// Packs one credential string per the scheme's convention: UTF-8 bytes
// followed by a single NUL terminator, then padded to exactly 64 bytes
// with random bytes (or truncated to 64 if the NUL-terminated string is
// already that long or longer). The NUL terminator is what lets the
// server find the string's real end despite the random padding -- the
// padding's specific value isn't otherwise significant. Exposed publicly
// so it's independently unit-testable.
QByteArray packCredential(const QString& text);
struct Response {
// Exactly 128 bytes on success (packCredential(username) +
// packCredential(password), AES-128-ECB encrypted). Empty on failure.
QByteArray encryptedCredentials;
// RSA-modulus-length bytes on success (the random AES key,
// PKCS1v1.5-encrypted with the server's public key). Empty on
// failure.
QByteArray encryptedAesKey;
};
// Computes the full type-33 response from the server's DER-encoded RSA
// public key and the credentials to authenticate with. Returns a Response
// with both fields empty on any failure (malformed key, an OpenSSL
// operation failing).
Response computeResponse(const QByteArray& hostKeyDer, const QString& username,
const QString& password);
}
#endif