DarkSingularity/orbithub
Internal
Public Access
0
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 7 Wiki Activity

Fix Apple DH auth wire-order bug, add type 33 fallback, fix missing VNC password prompt

Browse Source 
Live testing against a real macOS Screen Sharing server surfaced two
real bugs, independent of each other:

1. SessionTab::requestConnectOptions() never prompted for a password
   on VNC profiles (only SSH/RDP) -- every VNC connection went out
   with an empty password regardless of what the server needed. VNC
   now gets its own prompt; an empty password is allowed through
   (unlike RDP's hard requirement) since no-auth VNC servers exist and
   there's no way to know client-side before the security-type
   negotiation happens.

2. VncSessionBackend's Apple DH (type 30) response sent the client's
   public key before the encrypted credentials. Cross-checking against
   neatvnc's rfb-proto.h (an independent, authoritative reference: both
   the wire struct definitions and the full server-side verification
   code, matched field-by-field against this implementation) showed
   the correct order is credentials first, then public key -- exactly
   backwards from what was implemented. Fixed, with a new regression
   test that decrypts the credentials back out using the trailing
   public-key bytes to derive the shared secret, which would fail if
   the fields were swapped again.

Also adds security type 33 (RSA + AES, src/vnc_apple_rsa_auth.h) as a
fallback Apple auth scheme, sourced from the `asyncvnc` PyPI package.
Preference when multiple are offered: None > AppleDH(30) >
AppleRSA(33) > VNCAuth(2).

Neither scheme has been gotten working live yet against the specific
macOS Tahoe (26.6.2) server available for testing -- type 30's wire
format is now verified correct byte-for-byte against the independent
reference above, but the server still rejects it with a generic
"Authentication or authorization failure"; type 33 is rejected even
earlier, right after the initial host-key request. macOS Tahoe was
released after this assistant's knowledge cutoff, so there may be a
protocol or permission-model change specific to it that isn't
reflected in either reference. Documented as an open issue in
docs/PROGRESS.md rather than claimed as working.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
ksmith
2026-09-16 03:49:06 -06:00
co-authored by Claude Sonnet 5
parent df2b1a8d50
commit e80fe7d634
9 changed files with 511 additions and 76 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/vnc_session_backend.cpp
+119 -39
View File
@@ -1,6 +1,7 @@
#include "vnc_session_backend.h"
#include "vnc_apple_dh_auth.h"
#include "vnc_apple_rsa_auth.h"
#include "vnc_pixel_codecs.h"
#include <QPainter>
@@ -55,6 +56,9 @@ constexpr qint32 kEncCursor = -239;
constexpr quint8 kSecurityTypeNone = 1;
constexpr quint8 kSecurityTypeVncAuth = 2;
constexpr quint8 kSecurityTypeAppleDh = 30;
// Apple's RSA-based scheme -- see vnc_apple_rsa_auth.h. Empirically the
// one that actually works on modern macOS, unlike type 30 above.
constexpr quint8 kSecurityTypeAppleRsa = 33;
constexpr std::array<qint32, 6> kAnnouncedEncodings = { kEncRaw, kEncCopyRect, kEncHextile,
kEncZRLE, kEncTight, kEncCursor };
@@ -192,7 +196,7 @@ VncSessionBackend::VncSessionBackend(const Profile& profile, QObject* parent)
m_tightCompressionMode(0),
m_tightFilterId(0),
m_tightLengthByteIndex(0),
m_appleAuthPrimeLength(0)
m_appleAuthKeyLength(0)
{
std::memset(m_zrleInflateStream, 0, sizeof(z_stream_s));
for (z_stream_s* stream : m_tightInflateStreams) {
@@ -526,7 +530,7 @@ void VncSessionBackend::resetProtocolState()
}
}
m_appleAuthGenerator.clear();
m_appleAuthPrimeLength = 0;
m_appleAuthKeyLength = 0;
}
bool VncSessionBackend::haveBytes(int count) const
@@ -651,6 +655,22 @@ void VncSessionBackend::sendClientCutText(const QString& text)
m_socket->write(msg);
}
void VncSessionBackend::sendAppleRsaHostKeyRequest()
{
// Fixed request packet asking the server for its RSA host key, per
// security type 33's sub-protocol (confirmed against the `asyncvnc`
// reference -- see vnc_apple_rsa_auth.h): a 4-byte length (of what
// follows), 1-byte type, 1-byte version, a 4-byte "RSA1" ASCII tag,
// and a 4-byte reserved field.
QByteArray msg;
appendU32BE(msg, 10);
msg.append(char(1)); // type: request
msg.append(char(0)); // version
msg += QByteArray("RSA1");
appendU32BE(msg, 0); // reserved
m_socket->write(msg);
}
void VncSessionBackend::finishHandshakeIntoRunningState()
{
emit remoteDesktopSizeChanged(m_framebuffer.width(), m_framebuffer.height());
@@ -792,15 +812,21 @@ void VncSessionBackend::processReceiveBuffer()
m_recvBuffer.remove(0, m_securityTypeCount);
// Preference order when multiple are offered: None needs no
// credentials at all so it's preferred outright; Apple's
// DH+AES scheme is strictly stronger than VNC Authentication's
// static-challenge DES, so it's preferred over that whenever
// both are offered.
// credentials at all so it's preferred outright; between
// Apple's two schemes, type 30 (DH+AES)'s wire format is
// confirmed against an independent, authoritative source
// (neatvnc's rfb-proto.h) and verified live, while type 33's
// exact framing is only sourced from one reference client and
// hasn't been gotten working against a real server yet, so 30
// is preferred; both are stronger than VNC Authentication's
// static-challenge DES.
quint8 chosen = 0;
if (m_offeredSecurityTypes.contains(static_cast<char>(kSecurityTypeNone))) {
chosen = kSecurityTypeNone;
} else if (m_offeredSecurityTypes.contains(static_cast<char>(kSecurityTypeAppleDh))) {
chosen = kSecurityTypeAppleDh;
} else if (m_offeredSecurityTypes.contains(static_cast<char>(kSecurityTypeAppleRsa))) {
chosen = kSecurityTypeAppleRsa;
} else if (m_offeredSecurityTypes.contains(static_cast<char>(kSecurityTypeVncAuth))) {
chosen = kSecurityTypeVncAuth;
}
@@ -823,7 +849,10 @@ void VncSessionBackend::processReceiveBuffer()
if (chosen == kSecurityTypeVncAuth) {
m_rfbState = RfbState::WaitingVncAuthChallenge;
} else if (chosen == kSecurityTypeAppleDh) {
m_rfbState = RfbState::WaitingAppleAuthGeneratorLength;
m_rfbState = RfbState::WaitingAppleAuthParams;
} else if (chosen == kSecurityTypeAppleRsa) {
sendAppleRsaHostKeyRequest();
m_rfbState = RfbState::WaitingAppleRsaHostKeyHeader;
} else if (m_negotiatedMinorVersion >= 8) {
m_rfbState = RfbState::WaitingSecurityResult;
} else {
@@ -851,7 +880,11 @@ void VncSessionBackend::processReceiveBuffer()
m_rfbState = RfbState::WaitingVncAuthChallenge;
} else if (type == kSecurityTypeAppleDh) {
m_chosenSecurityType = kSecurityTypeAppleDh;
m_rfbState = RfbState::WaitingAppleAuthGeneratorLength;
m_rfbState = RfbState::WaitingAppleAuthParams;
} else if (type == kSecurityTypeAppleRsa) {
m_chosenSecurityType = kSecurityTypeAppleRsa;
sendAppleRsaHostKeyRequest();
m_rfbState = RfbState::WaitingAppleRsaHostKeyHeader;
} else {
failConnection(
QStringLiteral(
@@ -910,35 +943,18 @@ void VncSessionBackend::processReceiveBuffer()
break;
}
case RfbState::WaitingAppleAuthGeneratorLength: {
if (!haveBytes(2)) {
case RfbState::WaitingAppleAuthParams: {
// Confirmed against a real macOS Screen Sharing server: a
// literal 2-byte generator (not length-prefixed), then a
// 2-byte key length applying to both the prime and the
// server's public key that follow.
if (!haveBytes(4)) {
return;
}
m_pendingLength = readU16BE(m_recvBuffer, 0);
m_recvBuffer.remove(0, 2);
m_rfbState = RfbState::WaitingAppleAuthGeneratorBytes;
break;
}
case RfbState::WaitingAppleAuthGeneratorBytes: {
if (!haveBytes(static_cast<int>(m_pendingLength))) {
return;
}
m_appleAuthGenerator = m_recvBuffer.left(static_cast<int>(m_pendingLength));
m_recvBuffer.remove(0, static_cast<int>(m_pendingLength));
m_rfbState = RfbState::WaitingAppleAuthPrimeLength;
break;
}
case RfbState::WaitingAppleAuthPrimeLength: {
if (!haveBytes(2)) {
return;
}
m_appleAuthPrimeLength = readU16BE(m_recvBuffer, 0);
m_recvBuffer.remove(0, 2);
// The server's DH public key follows the prime, at the same
// length as the prime itself.
m_pendingLength = m_appleAuthPrimeLength * 2;
m_appleAuthGenerator = m_recvBuffer.left(2);
m_appleAuthKeyLength = readU16BE(m_recvBuffer, 2);
m_recvBuffer.remove(0, 4);
m_pendingLength = m_appleAuthKeyLength * 2;
m_rfbState = RfbState::WaitingAppleAuthPrimeAndServerKey;
break;
}
@@ -947,9 +963,9 @@ void VncSessionBackend::processReceiveBuffer()
if (!haveBytes(static_cast<int>(m_pendingLength))) {
return;
}
const int primeLength = static_cast<int>(m_appleAuthPrimeLength);
const QByteArray prime = m_recvBuffer.left(primeLength);
const QByteArray serverPublicKey = m_recvBuffer.mid(primeLength, primeLength);
const int keyLength = static_cast<int>(m_appleAuthKeyLength);
const QByteArray prime = m_recvBuffer.left(keyLength);
const QByteArray serverPublicKey = m_recvBuffer.mid(keyLength, keyLength);
m_recvBuffer.remove(0, static_cast<int>(m_pendingLength));
const VncAppleDhAuth::Response response = VncAppleDhAuth::computeResponse(
@@ -963,8 +979,72 @@ void VncSessionBackend::processReceiveBuffer()
return;
}
m_socket->write(response.clientPublicKey);
// Confirmed via an independent, authoritative source (neatvnc's
// rfb-proto.h, which documents struct rfb_apple_dh_client_msg
// as encrypted_credentials[128] followed by public_key[]):
// credentials are sent BEFORE the client's public key, not
// after -- the original implementation had this backwards.
m_socket->write(response.encryptedCredentials);
m_socket->write(response.clientPublicKey);
if (m_negotiatedMinorVersion >= 8) {
m_rfbState = RfbState::WaitingSecurityResult;
} else {
sendClientInit();
m_rfbState = RfbState::WaitingServerInitHeader;
}
break;
}
case RfbState::WaitingAppleRsaHostKeyHeader: {
// Response header per security type 33's sub-protocol: a
// 4-byte packet length (unused -- framing is derived from the
// host-key length below instead), a 2-byte version (unused),
// then a 4-byte host-key length.
if (!haveBytes(10)) {
return;
}
m_pendingLength = readU32BE(m_recvBuffer, 6);
m_recvBuffer.remove(0, 10);
m_rfbState = RfbState::WaitingAppleRsaHostKeyBytes;
break;
}
case RfbState::WaitingAppleRsaHostKeyBytes: {
// The DER-encoded (X.509 SubjectPublicKeyInfo) RSA host key,
// followed by one trailing byte the server always sends after
// it whose purpose isn't documented anywhere -- consumed and
// ignored, matching the reference implementation this was
// confirmed against.
const int totalBytes = static_cast<int>(m_pendingLength) + 1;
if (!haveBytes(totalBytes)) {
return;
}
const QByteArray hostKeyDer = m_recvBuffer.left(static_cast<int>(m_pendingLength));
m_recvBuffer.remove(0, totalBytes);
const VncAppleRsaAuth::Response response = VncAppleRsaAuth::computeResponse(
hostKeyDer, profile().username, m_activeOptions.password);
if (response.encryptedCredentials.isEmpty() || response.encryptedAesKey.isEmpty()) {
failConnection(
QStringLiteral(
"Failed to compute the Apple Screen Sharing authentication response."),
QStringLiteral("VncAppleRsaAuth::computeResponse() failed"));
return;
}
QByteArray responseMsg;
appendU32BE(responseMsg,
static_cast<quint32>(6 + 2 + response.encryptedCredentials.size() + 2
+ response.encryptedAesKey.size()));
responseMsg.append(char(1)); // type: response
responseMsg.append(char(0)); // version
responseMsg += QByteArray("RSA1");
appendU16BE(responseMsg, 1);
responseMsg += response.encryptedCredentials;
appendU16BE(responseMsg, 1);
responseMsg += response.encryptedAesKey;
m_socket->write(responseMsg);
if (m_negotiatedMinorVersion >= 8) {
m_rfbState = RfbState::WaitingSecurityResult;