Compare Unity performance runs on your PC (pilot)

Use this opt-in pilot to inspect one repeatable scenario on a PC you control. Framedash stores the summaries and compares their evidence. Start with a manual comparison; CI and automatic release criteria are separate steps. No map is required for this non-spatial comparison.

Prerequisites

1. Unity 2022.3 or later with Framedash Unity SDK 0.1.8 and Node.js 20 or later with CLI 0.1.11. Earlier Unity 0.1.7 / CLI 0.1.10 releases lack these APIs.
2. A Framedash project, an events:write ingest key for the player, and a separate analytics:read key for comparison. Keep keys in environment variables or private files; do not commit them.
3. A non-COPPA organization. COPPA redaction removes required attributes, so this endpoint returns 403. Keep required protection settings; local capture or an HTTP acknowledgement does not establish eligibility.

Install the Unity package by its pinned Git URL in Package Manager. Install the CLI separately:

https://github.com/crane-valley/framedash-unity-sdk.git#v0.1.8

npm install --global @framedash/cli@0.1.11

1. Hold conditions constant

Run the same baseline build twice, then run the candidate. Give every execution a fresh lowercase UUID v4, including retries. Keep the scenario, hardware profile, graphics settings, resolution, configuration, engine/platform, SDK version and warm-up/frame count identical. The unchanged repeat must also keep the baseline build ID and commit. Record driver, power/thermal policy, VSync/FPS cap and other relevant settings with the profile; matching labels do not prove the machine followed them.

2. Capture the scenario

Call this from your player on the main thread after setup, using the actual build/commit and configuration. Treat labels as configuration names, never device serials or personal identifiers. Labels must be nonblank, at most 128 UTF-16 code units, with no ASCII control characters. Preserve the printed run ID and stop the measurement attempt if started is false.

var sdk = Framedash.TelemetrySDK.Initialize(
    apiKey: System.Environment.GetEnvironmentVariable("FRAMEDASH_API_KEY"),
    buildId: "build-a");
var options = new Framedash.PerformanceRunOptions
{
    RunId = System.Guid.NewGuid().ToString("D"),
    Scenario = "route-a",
    Hardware = "lab-pc-a",
    Graphics = "high-vsync-off",
    Resolution = "1920x1080",
    Configuration = "release-dx12-driver-profile-a",
    Commit = "commit-sha",
    Branch = "main",
    WarmupFrames = 120,
    TargetFrames = 3600,
};
bool started = sdk.BeginPerformanceRun(options);
UnityEngine.Debug.Log("Performance run: " + options.RunId + ", started=" + started);

Execute the scenario while Unity’s player loop runs. This example excludes 120 warm-up intervals, then collects 3,600 intervals; it does not promise a fixed duration. Choose a repeatable scenario long enough to cover the window and bound process wall time separately. After scenario completion, call on the main thread:

bool complete = sdk.EndPerformanceRun();
bool acknowledged = sdk.FlushBlocking(5000);
UnityEngine.Debug.Log("complete=" + complete + ", acknowledged=" + acknowledged);

Require both booleans to be true, then verify the server report. complete is local evidence: a missing marker, buffer overflow, dropped samples or an early end cannot pass. acknowledged confirms an HTTP response, not durable storage. Abort with EndPerformanceRun(completed: false). A crash or SDK shutdown with an active run remains incomplete. run-profile-test does not start this capture automatically.

3. Read the comparison

Set the project and the three run IDs from your logs. The CLI reads only the last seven days. Omit --repeat when no unchanged repeat is available, but do not infer measurement stability from that omission:

framedash run-diff --project-id "$FRAMEDASH_PROJECT_ID" \
  --api-key-file analytics-read.key \
  --baseline "$BASELINE_RUN_ID" --candidate "$CANDIDATE_RUN_ID" \
  --repeat "$REPEAT_RUN_ID" --format json

Interpret the evidence

The default JSON includes run conditions, timestamps, accepted/dropped/warm-up counts, duration, P50/P95/P99 intervals, and exact counts/rates above 1,000/60, 1,000/30, 50 and 100 ms. Table/CSV is a compact quantile view.

These are wall-clock intervals between SDK Update callbacks, not GPU or display timing. The first callback is excluded. P50/P95/P99 are histogram-bin bounds with an exclusive upper bound; delta bounds subtract candidate minus baseline conservatively. A positive difference indicates longer frame intervals. These ranges and the optional unchanged-repeat variation are not statistical confidence intervals. One repeat and the 1,000-frame minimum do not establish noise or statistical reliability.

Exit	Meaning
0	Comparable evidence, including candidates that got slower. No regression verdict.
2	Inconclusive: inspect missing/incomplete evidence or condition mismatches.
1	Command/API error, including an invalid response.

Troubleshooting

• Missing records: check the ingest key/project, run IDs and seven-day window, then allow bounded ingestion time. A successful flush alone is insufficient.
• Incomplete: inspect the end result, frame budget and dropped counts. Invalid, nonpositive or >=32,768 ms intervals consume the frame budget as dropped samples. Warm-up can be 0..60,000; the target can be 1,000..1,000,000. Retry the whole scenario with a new ID.
• Mismatch: correct declared and actual conditions. Never relabel different machines or builds to force comparability. The repeat must use the baseline build/commit.
• Conflicting/too many records: do not reuse an ID; reads are bounded to 16 raw records per run before duplicate handling.
• 403: check project access, analytics:read, account entitlement and COPPA eligibility. Do not weaken privacy settings to use the pilot.

Inspect the comparison with a teammate, record the next investigation and setup time, then repeat after the fix. This report does not attribute root cause or certify release safety. Existing CI performance gates keep their current behavior. See API overview and Unity SDK setup.