Fix Apple DH auth wire-order bug, add type 33 fallback, fix missing VNC password prompt

Live testing against a real macOS Screen Sharing server surfaced two
real bugs, independent of each other:

1. SessionTab::requestConnectOptions() never prompted for a password
   on VNC profiles (only SSH/RDP) -- every VNC connection went out
   with an empty password regardless of what the server needed. VNC
   now gets its own prompt; an empty password is allowed through
   (unlike RDP's hard requirement) since no-auth VNC servers exist and
   there's no way to know client-side before the security-type
   negotiation happens.

2. VncSessionBackend's Apple DH (type 30) response sent the client's
   public key before the encrypted credentials. Cross-checking against
   neatvnc's rfb-proto.h (an independent, authoritative reference: both
   the wire struct definitions and the full server-side verification
   code, matched field-by-field against this implementation) showed
   the correct order is credentials first, then public key -- exactly
   backwards from what was implemented. Fixed, with a new regression
   test that decrypts the credentials back out using the trailing
   public-key bytes to derive the shared secret, which would fail if
   the fields were swapped again.

Also adds security type 33 (RSA + AES, src/vnc_apple_rsa_auth.h) as a
fallback Apple auth scheme, sourced from the `asyncvnc` PyPI package.
Preference when multiple are offered: None > AppleDH(30) >
AppleRSA(33) > VNCAuth(2).

Neither scheme has been gotten working live yet against the specific
macOS Tahoe (26.6.2) server available for testing -- type 30's wire
format is now verified correct byte-for-byte against the independent
reference above, but the server still rejects it with a generic
"Authentication or authorization failure"; type 33 is rejected even
earlier, right after the initial host-key request. macOS Tahoe was
released after this assistant's knowledge cutoff, so there may be a
protocol or permission-model change specific to it that isn't
reflected in either reference. Documented as an open issue in
docs/PROGRESS.md rather than claimed as working.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-09-16 03:49:06 -06:00
co-authored by Claude Sonnet 5
parent df2b1a8d50
commit e80fe7d634
9 changed files with 511 additions and 76 deletions
+44 -15
View File
@@ -121,7 +121,7 @@ Delivered:
keyboard (Qt key -> X11 keysym mapping) and mouse/wheel input forwarding
- `VncDisplayWidget` mirroring `RdpDisplayWidget`'s scale-to-fit rendering
and input-forwarding shape
- 41 unit tests (`tests/test_vnc_session_backend.cpp`): pure-function
- 45 unit tests (`tests/test_vnc_session_backend.cpp`): pure-function
coverage (DES key prep verified against an independently documented test
vector, keysym mapping, socket-error mapping) plus state-machine
coverage against a scripted in-process fake RFB server (all three
@@ -129,10 +129,10 @@ Delivered:
security types, pixel-accurate Raw/Hextile/ZRLE/Tight decoding including
a ZRLE zlib-stream-persistence test across two separate
`FramebufferUpdate` messages, a Tight stream-reset-flag test, cursor/
clipboard round trips) -- caught and fixed a real re-entrancy bug
(`abort()` synchronously re-firing `disconnected()` mid-
`failConnection()`, silently overwriting a specific error with a
generic one)
clipboard round trips, both Apple auth schemes) -- caught and fixed a
real re-entrancy bug (`abort()` synchronously re-firing `disconnected()`
mid-`failConnection()`, silently overwriting a specific error with a
generic one) and a real Apple-DH wire-order bug (see below)
- Verified live against a real, independently implemented VNC server
(TightVNC on Windows): connect with VNC Authentication, correct
framebuffer dimensions and pixel data, clean disconnect, reconnect,
@@ -150,6 +150,13 @@ Delivered:
it degenerates to an exact 1:1 mapping once the widget's own bounds are
fixed to the remote's size, so no separate rendering path was needed.
Persisted across sessions like the terminal theme preference.
- Fixed a real, separate bug found during Apple-auth live testing:
`SessionTab::requestConnectOptions()` never prompted for a password on
VNC profiles at all (only SSH/RDP), so every VNC connection went out
with an empty password regardless of what the server needed. VNC now
gets its own prompt, with an empty password allowed through (unlike
RDP's hard requirement) since some VNC servers are no-auth and there's
no way to know that before the server's security-type negotiation
- Robustness fix: an unrecognized `FramebufferUpdate` rectangle encoding
used to abort the connection generically; `kAnnouncedEncodings` is now
the single source of truth for what `SetEncodings` announces and what
@@ -175,17 +182,39 @@ Delivered:
packaging-dependent runtime failure mode. ZRLE/Tight link `ZLIB::ZLIB`
(found via a fresh top-level `find_package(ZLIB REQUIRED)`, independent
of whether vendored FreeRDP's own internal zlib usage stays enabled)
- 41 unit tests total, 15 of them for Hextile/ZRLE/Tight specifically,
including a ZRLE zlib-stream-persistence test across two separate
`FramebufferUpdate` messages and a Tight stream-reset-flag test proving
the low 4 control-byte bits actually tear down and reinitialize the
targeted stream rather than erroring out on stale state
- Apple Screen Sharing authentication: two schemes, both undocumented by
Apple. Security type 30 (Diffie-Hellman + AES, `src/vnc_apple_dh_auth.h`)
is implemented and its wire format is confirmed correct against an
independent, authoritative source (`neatvnc`'s `rfb-proto.h` struct
definitions plus its full server-side verification code, cross-checked
field-by-field: generator/key-length framing, the
credentials-before-public-key send order -- an actual ordering bug
caught this way and fixed -- shared-secret derivation and padding, AES
key derivation, and the credential buffer layout all match exactly).
Security type 33 (RSA + AES, `src/vnc_apple_rsa_auth.h`) is also
implemented, sourced from the `asyncvnc` PyPI package, as a fallback.
Preference when both are offered: None > AppleDH(30) > AppleRSA(33) >
VNCAuth(2). Covered by a DH round-trip test, fake-server integration
tests for both schemes, a preference-order test, and a regression test
built from real bytes captured off an actual macOS server
Known gaps (explicit scope decisions, not oversights -- see issue #3 for
follow-up tracking):
- Apple's Screen Sharing authentication (Diffie-Hellman + AES, security
type 30) isn't implemented, so this can't yet reach macOS's built-in VNC
server -- only standard VNC Authentication (type 2) and no-auth (type 1)
Known gaps and open issues (see issue #3 for follow-up tracking):
- **Apple auth not yet confirmed working end-to-end against a real
server.** Live-tested against a macOS Tahoe (26.6.2) Screen Sharing
server with Screen Sharing correctly enabled and the connecting account
allowed: type 33 gets rejected by the server immediately after the
client's initial host-key request (before any credentials are even
sent), and type 30 -- despite matching the authoritative reference
byte-for-byte, verified via multiple independent diagnostic scripts --
still gets rejected with a generic "Authentication or authorization
failure" from the server. macOS Tahoe was released after this
assistant's knowledge cutoff, so there may be a protocol or permission-
model change specific to that OS version neither reference source
reflects; a `screensharingd` Console.app log from the moment of
rejection would be the next diagnostic step whenever this is picked
back up. Until this is resolved, treat both security types as
implemented-and-tested-in-isolation but **not verified to actually
authenticate against a real macOS server**
- Tight's Basic compression mode always assumes zlib-compressed payloads;
the real protocol permits the server to skip compression for very small
(filtered byte count under ~12) payloads, which this decoder doesn't