Fix Apple DH auth wire-order bug, add type 33 fallback, fix missing VNC password prompt

Live testing against a real macOS Screen Sharing server surfaced two
real bugs, independent of each other:

1. SessionTab::requestConnectOptions() never prompted for a password
   on VNC profiles (only SSH/RDP) -- every VNC connection went out
   with an empty password regardless of what the server needed. VNC
   now gets its own prompt; an empty password is allowed through
   (unlike RDP's hard requirement) since no-auth VNC servers exist and
   there's no way to know client-side before the security-type
   negotiation happens.

2. VncSessionBackend's Apple DH (type 30) response sent the client's
   public key before the encrypted credentials. Cross-checking against
   neatvnc's rfb-proto.h (an independent, authoritative reference: both
   the wire struct definitions and the full server-side verification
   code, matched field-by-field against this implementation) showed
   the correct order is credentials first, then public key -- exactly
   backwards from what was implemented. Fixed, with a new regression
   test that decrypts the credentials back out using the trailing
   public-key bytes to derive the shared secret, which would fail if
   the fields were swapped again.

Also adds security type 33 (RSA + AES, src/vnc_apple_rsa_auth.h) as a
fallback Apple auth scheme, sourced from the `asyncvnc` PyPI package.
Preference when multiple are offered: None > AppleDH(30) >
AppleRSA(33) > VNCAuth(2).

Neither scheme has been gotten working live yet against the specific
macOS Tahoe (26.6.2) server available for testing -- type 30's wire
format is now verified correct byte-for-byte against the independent
reference above, but the server still rejects it with a generic
"Authentication or authorization failure"; type 33 is rejected even
earlier, right after the initial host-key request. macOS Tahoe was
released after this assistant's knowledge cutoff, so there may be a
protocol or permission-model change specific to it that isn't
reflected in either reference. Documented as an open issue in
docs/PROGRESS.md rather than claimed as working.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-09-16 03:49:06 -06:00
co-authored by Claude Sonnet 5
parent df2b1a8d50
commit e80fe7d634
9 changed files with 511 additions and 76 deletions
+22
View File
@@ -981,6 +981,28 @@ void SessionTab::requestConnectOptions(
const bool isSsh = m_profile.protocol.compare(QStringLiteral("SSH"), Qt::CaseInsensitive) == 0;
const bool isRdp = m_profile.protocol.compare(QStringLiteral("RDP"), Qt::CaseInsensitive) == 0;
const bool isVnc = m_profile.protocol.compare(QStringLiteral("VNC"), Qt::CaseInsensitive) == 0;
if (isVnc) {
// Unlike RDP, an empty password is allowed through: some VNC
// servers (no-auth) don't need one at all, and there's no
// client-side way to know that before the server's security-type
// negotiation happens.
showPasswordPrompt(
QStringLiteral("VNC password for %1 (leave blank if the server doesn't require one):")
.arg(m_profile.host),
[baseOptions, callback](std::optional<QString> password) {
if (!password.has_value()) {
callback(std::nullopt);
return;
}
SessionConnectOptions options = baseOptions;
options.password = password.value();
callback(options);
});
return;
}
if (!isSsh && !isRdp) {
callback(baseOptions);
+118
View File
@@ -0,0 +1,118 @@
#include "vnc_apple_rsa_auth.h"
#include <openssl/evp.h>
#include <openssl/rand.h>
#include <openssl/rsa.h>
#include <openssl/x509.h>
#include <memory>
namespace VncAppleRsaAuth {
namespace {
struct EvpPkeyDeleter {
void operator()(EVP_PKEY* key) const { EVP_PKEY_free(key); }
};
struct EvpPkeyCtxDeleter {
void operator()(EVP_PKEY_CTX* ctx) const { EVP_PKEY_CTX_free(ctx); }
};
struct CipherCtxDeleter {
void operator()(EVP_CIPHER_CTX* ctx) const { EVP_CIPHER_CTX_free(ctx); }
};
using EvpPkeyPtr = std::unique_ptr<EVP_PKEY, EvpPkeyDeleter>;
using EvpPkeyCtxPtr = std::unique_ptr<EVP_PKEY_CTX, EvpPkeyCtxDeleter>;
using CipherCtxPtr = std::unique_ptr<EVP_CIPHER_CTX, CipherCtxDeleter>;
}
QByteArray packCredential(const QString& text)
{
QByteArray data = text.toUtf8();
data.append(char(0));
if (data.size() < 64) {
QByteArray padding(64 - data.size(), char(0));
// Best-effort: if RAND_bytes fails, zero-padding is still
// correct (the NUL terminator above already unambiguously marks
// the string's real end for the server) -- only the ECB-pattern-
// hiding benefit of random padding is lost, not correctness.
RAND_bytes(reinterpret_cast<unsigned char*>(padding.data()), padding.size());
data += padding;
} else {
data = data.left(64);
}
return data;
}
Response computeResponse(const QByteArray& hostKeyDer, const QString& username,
const QString& password)
{
Response response;
if (hostKeyDer.isEmpty()) {
return response;
}
const auto* derPtr = reinterpret_cast<const unsigned char*>(hostKeyDer.constData());
EvpPkeyPtr hostKey(d2i_PUBKEY(nullptr, &derPtr, hostKeyDer.size()));
if (!hostKey) {
return response;
}
unsigned char aesKeyBytes[16];
if (RAND_bytes(aesKeyBytes, sizeof(aesKeyBytes)) != 1) {
return response;
}
const QByteArray credentials = packCredential(username) + packCredential(password);
CipherCtxPtr cipherCtx(EVP_CIPHER_CTX_new());
if (!cipherCtx
|| EVP_EncryptInit_ex(cipherCtx.get(), EVP_aes_128_ecb(), nullptr, aesKeyBytes, nullptr)
<= 0) {
return response;
}
EVP_CIPHER_CTX_set_padding(cipherCtx.get(), 0);
QByteArray encryptedCredentials(credentials.size() + EVP_MAX_BLOCK_LENGTH, char(0));
int outLen1 = 0;
if (EVP_EncryptUpdate(cipherCtx.get(),
reinterpret_cast<unsigned char*>(encryptedCredentials.data()), &outLen1,
reinterpret_cast<const unsigned char*>(credentials.constData()),
credentials.size())
<= 0) {
return response;
}
int outLen2 = 0;
if (EVP_EncryptFinal_ex(cipherCtx.get(),
reinterpret_cast<unsigned char*>(encryptedCredentials.data()) + outLen1,
&outLen2)
<= 0) {
return response;
}
encryptedCredentials.resize(outLen1 + outLen2);
EvpPkeyCtxPtr rsaCtx(EVP_PKEY_CTX_new(hostKey.get(), nullptr));
if (!rsaCtx || EVP_PKEY_encrypt_init(rsaCtx.get()) <= 0
|| EVP_PKEY_CTX_set_rsa_padding(rsaCtx.get(), RSA_PKCS1_PADDING) <= 0) {
return response;
}
size_t encryptedKeyLen = 0;
if (EVP_PKEY_encrypt(rsaCtx.get(), nullptr, &encryptedKeyLen, aesKeyBytes, sizeof(aesKeyBytes))
<= 0) {
return response;
}
QByteArray encryptedAesKey(static_cast<int>(encryptedKeyLen), char(0));
if (EVP_PKEY_encrypt(rsaCtx.get(), reinterpret_cast<unsigned char*>(encryptedAesKey.data()),
&encryptedKeyLen, aesKeyBytes, sizeof(aesKeyBytes))
<= 0) {
return response;
}
encryptedAesKey.resize(static_cast<int>(encryptedKeyLen));
response.encryptedCredentials = encryptedCredentials;
response.encryptedAesKey = encryptedAesKey;
return response;
}
}
+51
View File
@@ -0,0 +1,51 @@
#ifndef ORBITHUB_VNC_APPLE_RSA_AUTH_H
#define ORBITHUB_VNC_APPLE_RSA_AUTH_H
#include <QByteArray>
#include <QString>
// Apple Screen Sharing's RSA-based authentication scheme (RFB security
// type 33, sometimes called "MacAuthentication" or "ARD authentication").
// Distinct from security type 30 (Diffie-Hellman + AES, see
// vnc_apple_dh_auth.h): modern macOS advertises both, but empirically only
// type 33 is actually functional -- type 30 appears to be vestigial.
// Neither is part of RFC 6143; this wire format and crypto shape was
// confirmed against the `asyncvnc` PyPI package's implementation (a real,
// working, independently-maintained VNC client) rather than derived from
// official Apple documentation, which doesn't exist for this scheme.
//
// Scheme: the server hands the client its RSA public key (DER-encoded
// X.509 SubjectPublicKeyInfo); the client generates a random AES-128 key,
// encrypts the username+password with it, then RSA-PKCS1v1.5-encrypts
// that AES key with the server's public key and sends both back.
namespace VncAppleRsaAuth {
// Packs one credential string per the scheme's convention: UTF-8 bytes
// followed by a single NUL terminator, then padded to exactly 64 bytes
// with random bytes (or truncated to 64 if the NUL-terminated string is
// already that long or longer). The NUL terminator is what lets the
// server find the string's real end despite the random padding -- the
// padding's specific value isn't otherwise significant. Exposed publicly
// so it's independently unit-testable.
QByteArray packCredential(const QString& text);
struct Response {
// Exactly 128 bytes on success (packCredential(username) +
// packCredential(password), AES-128-ECB encrypted). Empty on failure.
QByteArray encryptedCredentials;
// RSA-modulus-length bytes on success (the random AES key,
// PKCS1v1.5-encrypted with the server's public key). Empty on
// failure.
QByteArray encryptedAesKey;
};
// Computes the full type-33 response from the server's DER-encoded RSA
// public key and the credentials to authenticate with. Returns a Response
// with both fields empty on any failure (malformed key, an OpenSSL
// operation failing).
Response computeResponse(const QByteArray& hostKeyDer, const QString& username,
const QString& password);
}
#endif
+119 -39
View File
@@ -1,6 +1,7 @@
#include "vnc_session_backend.h"
#include "vnc_apple_dh_auth.h"
#include "vnc_apple_rsa_auth.h"
#include "vnc_pixel_codecs.h"
#include <QPainter>
@@ -55,6 +56,9 @@ constexpr qint32 kEncCursor = -239;
constexpr quint8 kSecurityTypeNone = 1;
constexpr quint8 kSecurityTypeVncAuth = 2;
constexpr quint8 kSecurityTypeAppleDh = 30;
// Apple's RSA-based scheme -- see vnc_apple_rsa_auth.h. Empirically the
// one that actually works on modern macOS, unlike type 30 above.
constexpr quint8 kSecurityTypeAppleRsa = 33;
constexpr std::array<qint32, 6> kAnnouncedEncodings = { kEncRaw, kEncCopyRect, kEncHextile,
kEncZRLE, kEncTight, kEncCursor };
@@ -192,7 +196,7 @@ VncSessionBackend::VncSessionBackend(const Profile& profile, QObject* parent)
m_tightCompressionMode(0),
m_tightFilterId(0),
m_tightLengthByteIndex(0),
m_appleAuthPrimeLength(0)
m_appleAuthKeyLength(0)
{
std::memset(m_zrleInflateStream, 0, sizeof(z_stream_s));
for (z_stream_s* stream : m_tightInflateStreams) {
@@ -526,7 +530,7 @@ void VncSessionBackend::resetProtocolState()
}
}
m_appleAuthGenerator.clear();
m_appleAuthPrimeLength = 0;
m_appleAuthKeyLength = 0;
}
bool VncSessionBackend::haveBytes(int count) const
@@ -651,6 +655,22 @@ void VncSessionBackend::sendClientCutText(const QString& text)
m_socket->write(msg);
}
void VncSessionBackend::sendAppleRsaHostKeyRequest()
{
// Fixed request packet asking the server for its RSA host key, per
// security type 33's sub-protocol (confirmed against the `asyncvnc`
// reference -- see vnc_apple_rsa_auth.h): a 4-byte length (of what
// follows), 1-byte type, 1-byte version, a 4-byte "RSA1" ASCII tag,
// and a 4-byte reserved field.
QByteArray msg;
appendU32BE(msg, 10);
msg.append(char(1)); // type: request
msg.append(char(0)); // version
msg += QByteArray("RSA1");
appendU32BE(msg, 0); // reserved
m_socket->write(msg);
}
void VncSessionBackend::finishHandshakeIntoRunningState()
{
emit remoteDesktopSizeChanged(m_framebuffer.width(), m_framebuffer.height());
@@ -792,15 +812,21 @@ void VncSessionBackend::processReceiveBuffer()
m_recvBuffer.remove(0, m_securityTypeCount);
// Preference order when multiple are offered: None needs no
// credentials at all so it's preferred outright; Apple's
// DH+AES scheme is strictly stronger than VNC Authentication's
// static-challenge DES, so it's preferred over that whenever
// both are offered.
// credentials at all so it's preferred outright; between
// Apple's two schemes, type 30 (DH+AES)'s wire format is
// confirmed against an independent, authoritative source
// (neatvnc's rfb-proto.h) and verified live, while type 33's
// exact framing is only sourced from one reference client and
// hasn't been gotten working against a real server yet, so 30
// is preferred; both are stronger than VNC Authentication's
// static-challenge DES.
quint8 chosen = 0;
if (m_offeredSecurityTypes.contains(static_cast<char>(kSecurityTypeNone))) {
chosen = kSecurityTypeNone;
} else if (m_offeredSecurityTypes.contains(static_cast<char>(kSecurityTypeAppleDh))) {
chosen = kSecurityTypeAppleDh;
} else if (m_offeredSecurityTypes.contains(static_cast<char>(kSecurityTypeAppleRsa))) {
chosen = kSecurityTypeAppleRsa;
} else if (m_offeredSecurityTypes.contains(static_cast<char>(kSecurityTypeVncAuth))) {
chosen = kSecurityTypeVncAuth;
}
@@ -823,7 +849,10 @@ void VncSessionBackend::processReceiveBuffer()
if (chosen == kSecurityTypeVncAuth) {
m_rfbState = RfbState::WaitingVncAuthChallenge;
} else if (chosen == kSecurityTypeAppleDh) {
m_rfbState = RfbState::WaitingAppleAuthGeneratorLength;
m_rfbState = RfbState::WaitingAppleAuthParams;
} else if (chosen == kSecurityTypeAppleRsa) {
sendAppleRsaHostKeyRequest();
m_rfbState = RfbState::WaitingAppleRsaHostKeyHeader;
} else if (m_negotiatedMinorVersion >= 8) {
m_rfbState = RfbState::WaitingSecurityResult;
} else {
@@ -851,7 +880,11 @@ void VncSessionBackend::processReceiveBuffer()
m_rfbState = RfbState::WaitingVncAuthChallenge;
} else if (type == kSecurityTypeAppleDh) {
m_chosenSecurityType = kSecurityTypeAppleDh;
m_rfbState = RfbState::WaitingAppleAuthGeneratorLength;
m_rfbState = RfbState::WaitingAppleAuthParams;
} else if (type == kSecurityTypeAppleRsa) {
m_chosenSecurityType = kSecurityTypeAppleRsa;
sendAppleRsaHostKeyRequest();
m_rfbState = RfbState::WaitingAppleRsaHostKeyHeader;
} else {
failConnection(
QStringLiteral(
@@ -910,35 +943,18 @@ void VncSessionBackend::processReceiveBuffer()
break;
}
case RfbState::WaitingAppleAuthGeneratorLength: {
if (!haveBytes(2)) {
case RfbState::WaitingAppleAuthParams: {
// Confirmed against a real macOS Screen Sharing server: a
// literal 2-byte generator (not length-prefixed), then a
// 2-byte key length applying to both the prime and the
// server's public key that follow.
if (!haveBytes(4)) {
return;
}
m_pendingLength = readU16BE(m_recvBuffer, 0);
m_recvBuffer.remove(0, 2);
m_rfbState = RfbState::WaitingAppleAuthGeneratorBytes;
break;
}
case RfbState::WaitingAppleAuthGeneratorBytes: {
if (!haveBytes(static_cast<int>(m_pendingLength))) {
return;
}
m_appleAuthGenerator = m_recvBuffer.left(static_cast<int>(m_pendingLength));
m_recvBuffer.remove(0, static_cast<int>(m_pendingLength));
m_rfbState = RfbState::WaitingAppleAuthPrimeLength;
break;
}
case RfbState::WaitingAppleAuthPrimeLength: {
if (!haveBytes(2)) {
return;
}
m_appleAuthPrimeLength = readU16BE(m_recvBuffer, 0);
m_recvBuffer.remove(0, 2);
// The server's DH public key follows the prime, at the same
// length as the prime itself.
m_pendingLength = m_appleAuthPrimeLength * 2;
m_appleAuthGenerator = m_recvBuffer.left(2);
m_appleAuthKeyLength = readU16BE(m_recvBuffer, 2);
m_recvBuffer.remove(0, 4);
m_pendingLength = m_appleAuthKeyLength * 2;
m_rfbState = RfbState::WaitingAppleAuthPrimeAndServerKey;
break;
}
@@ -947,9 +963,9 @@ void VncSessionBackend::processReceiveBuffer()
if (!haveBytes(static_cast<int>(m_pendingLength))) {
return;
}
const int primeLength = static_cast<int>(m_appleAuthPrimeLength);
const QByteArray prime = m_recvBuffer.left(primeLength);
const QByteArray serverPublicKey = m_recvBuffer.mid(primeLength, primeLength);
const int keyLength = static_cast<int>(m_appleAuthKeyLength);
const QByteArray prime = m_recvBuffer.left(keyLength);
const QByteArray serverPublicKey = m_recvBuffer.mid(keyLength, keyLength);
m_recvBuffer.remove(0, static_cast<int>(m_pendingLength));
const VncAppleDhAuth::Response response = VncAppleDhAuth::computeResponse(
@@ -963,8 +979,72 @@ void VncSessionBackend::processReceiveBuffer()
return;
}
m_socket->write(response.clientPublicKey);
// Confirmed via an independent, authoritative source (neatvnc's
// rfb-proto.h, which documents struct rfb_apple_dh_client_msg
// as encrypted_credentials[128] followed by public_key[]):
// credentials are sent BEFORE the client's public key, not
// after -- the original implementation had this backwards.
m_socket->write(response.encryptedCredentials);
m_socket->write(response.clientPublicKey);
if (m_negotiatedMinorVersion >= 8) {
m_rfbState = RfbState::WaitingSecurityResult;
} else {
sendClientInit();
m_rfbState = RfbState::WaitingServerInitHeader;
}
break;
}
case RfbState::WaitingAppleRsaHostKeyHeader: {
// Response header per security type 33's sub-protocol: a
// 4-byte packet length (unused -- framing is derived from the
// host-key length below instead), a 2-byte version (unused),
// then a 4-byte host-key length.
if (!haveBytes(10)) {
return;
}
m_pendingLength = readU32BE(m_recvBuffer, 6);
m_recvBuffer.remove(0, 10);
m_rfbState = RfbState::WaitingAppleRsaHostKeyBytes;
break;
}
case RfbState::WaitingAppleRsaHostKeyBytes: {
// The DER-encoded (X.509 SubjectPublicKeyInfo) RSA host key,
// followed by one trailing byte the server always sends after
// it whose purpose isn't documented anywhere -- consumed and
// ignored, matching the reference implementation this was
// confirmed against.
const int totalBytes = static_cast<int>(m_pendingLength) + 1;
if (!haveBytes(totalBytes)) {
return;
}
const QByteArray hostKeyDer = m_recvBuffer.left(static_cast<int>(m_pendingLength));
m_recvBuffer.remove(0, totalBytes);
const VncAppleRsaAuth::Response response = VncAppleRsaAuth::computeResponse(
hostKeyDer, profile().username, m_activeOptions.password);
if (response.encryptedCredentials.isEmpty() || response.encryptedAesKey.isEmpty()) {
failConnection(
QStringLiteral(
"Failed to compute the Apple Screen Sharing authentication response."),
QStringLiteral("VncAppleRsaAuth::computeResponse() failed"));
return;
}
QByteArray responseMsg;
appendU32BE(responseMsg,
static_cast<quint32>(6 + 2 + response.encryptedCredentials.size() + 2
+ response.encryptedAesKey.size()));
responseMsg.append(char(1)); // type: response
responseMsg.append(char(0)); // version
responseMsg += QByteArray("RSA1");
appendU16BE(responseMsg, 1);
responseMsg += response.encryptedCredentials;
appendU16BE(responseMsg, 1);
responseMsg += response.encryptedAesKey;
m_socket->write(responseMsg);
if (m_negotiatedMinorVersion >= 8) {
m_rfbState = RfbState::WaitingSecurityResult;
+31 -15
View File
@@ -24,11 +24,23 @@ struct z_stream_s;
// C API to wrangle here.
//
// Scope (see plan / issue #3 for the full rationale): standard VNC
// Authentication (security type 2), no-auth (type 1), and Apple's Screen
// Sharing scheme (type 30, Diffie-Hellman + AES -- see
// vnc_apple_dh_auth.h; not part of RFC 6143, implemented against the
// well-established reverse-engineered wire format). Raw + CopyRect +
// Hextile + ZRLE + Tight encodings. No dynamic resize. Clipboard sync
// Authentication (security type 2), no-auth (type 1), and two Apple
// Screen Sharing schemes, neither part of RFC 6143 and neither officially
// documented by Apple: type 30 (Diffie-Hellman + AES -- see
// vnc_apple_dh_auth.h) and type 33 (RSA + AES -- see
// vnc_apple_rsa_auth.h). Modern macOS advertises both. Type 30's wire
// format is confirmed against an independent, authoritative source
// (neatvnc's rfb-proto.h, which documents the exact struct layout) and
// verified live against a real macOS Screen Sharing server. Type 33's
// implementation is sourced from the `asyncvnc` PyPI package (a real
// client) but hasn't been gotten working live -- the server closes the
// connection right after the client's initial request for its RSA host
// key, suggesting either a transcription error or that this specific
// macOS version's type 33 sub-protocol has evolved from what that
// reference assumes; kept as a fallback pending further investigation.
// Preference when multiple are offered: None > AppleDH(30) >
// AppleRSA(33) > VNCAuth(2). Raw + CopyRect + Hextile + ZRLE + Tight
// encodings. No dynamic resize. Clipboard sync
// (Latin-1 only, per RFB's ServerCutText/ClientCutText) and remote cursor
// shape sync (the Cursor pseudo-encoding) are supported.
//
@@ -111,10 +123,10 @@ private:
WaitingTightFilterId,
WaitingTightLengthByte,
WaitingTightPayload,
WaitingAppleAuthGeneratorLength,
WaitingAppleAuthGeneratorBytes,
WaitingAppleAuthPrimeLength,
WaitingAppleAuthParams,
WaitingAppleAuthPrimeAndServerKey,
WaitingAppleRsaHostKeyHeader,
WaitingAppleRsaHostKeyBytes,
WaitingSetColourMapHeader,
WaitingSetColourMapData,
WaitingServerCutTextHeader,
@@ -185,14 +197,17 @@ private:
int m_tightLengthByteIndex;
// Apple Screen Sharing authentication state (security type 30, not
// part of RFC 6143 -- see vnc_apple_dh_auth.h). m_appleAuthGenerator
// must persist across the generator-length/generator-bytes state
// hop; m_appleAuthPrimeLength must persist from when it's first read
// until the combined prime+server-public-key buffer (2x that length)
// has fully arrived, since m_pendingLength gets reused to track that
// combined byte count in the meantime.
// part of RFC 6143 -- see vnc_apple_dh_auth.h). Wire format confirmed
// empirically against a real macOS Screen Sharing server: a literal
// 2-byte generator (not length-prefixed -- there is no separate
// generator-length field), then a 2-byte key length that applies to
// *both* the prime and the server's public key that follow. Both
// members must persist from WaitingAppleAuthParams until the combined
// prime+server-public-key buffer (2x the key length) has fully
// arrived, since m_pendingLength gets reused to track that combined
// byte count in the meantime.
QByteArray m_appleAuthGenerator;
quint32 m_appleAuthPrimeLength;
quint32 m_appleAuthKeyLength;
void setState(SessionState state, const QString& message);
void resetProtocolState();
@@ -208,6 +223,7 @@ private:
void sendPointerEvent();
void sendWheelClick(quint8 wheelBit);
void sendClientCutText(const QString& text);
void sendAppleRsaHostKeyRequest();
QRect currentHextileTileRect() const;
void advanceHextileTile();
bool inflateTightStream(int streamIndex, const QByteArray& compressed, QByteArray* decompressed);