• CREDIT-BACKTEST-PARQUET-V1
  • CALIBRATED PD → POLICY
  • EVALUATED 2026-07-17

A score is not a policy.

The same calibrated score column supports any portfolio you like — the decision lives in the threshold below it. The figure recomputes approval share against realized defaults from the published backtest rows; no model runs here, and the scores are offline and frozen.

72k/24k/24ktrain / calibration / backtest
.1593brier · baseline isotonic
.6746auc · baseline
.6753auc · challenger
120,000applications
0.218backtest default rate
EXHIBIT 01 · THE POLICY FRONTIER — APPROVAL SHARE VS REALIZED DEFAULTCOMPUTED FROM PUBLISHED PREVIEW ROWS · 624 BACKTEST APPLICATIONS
ThresholdApprovedDefault rate
approve everyone100%19.2%
pd ≤ .1531.1%7.2%
pd ≤ .2050.6%10.8%
pd ≤ .3081.7%13.9%

Baseline logistic and challenger XGBoost tie at .1593 Brier; AUC moves .6746 → .6753. The model race shifts the third decimal — the threshold above moves the whole portfolio.

granted-loan-only offline association · rejected applicants absent
no production, causal, regulatory, or fairness claim

POLICY-CONTRACT-V2 · APPROVE → MANUAL REVIEW → DECLINE

One threshold decides
who gets a loan.

Every scored application falls into exactly one of three bands — approve, manual review, or decline — by comparing its calibrated PD against two published thresholds. Approved exposure is judged on expected loss (selected_pd * lgd * ead), where LGD is a disclosed 45% assumption, not a modeled output.

Approveselected_pd <= approve_threshold
Manual reviewapprove_threshold < selected_pd <= review_threshold
Declineselected_pd > review_threshold

Same 624 backtest rows, three approve-thresholds

Approve whenShare approvedDefault ratevs. approve everyone
pd ≤ .1531.1%7.2%0.38×
pd ≤ .2050.6%10.8%0.56×
pd ≤ .3081.7%13.9%0.72×

Tightening the approve threshold trades approval share for a lower default rate among the applications that remain — the frontier in exhibit 01 is this exact same trade, drawn continuously instead of at three points.

BASELINE ISOTONIC · CHALLENGER ISOTONIC · 24,000 BACKTEST ROWS

The challenger doesn’t win.

MetricBaseline (isotonic logistic)Challenger (isotonic XGBoost)Wins
Brier.159280.159253challenger
Log loss.493403.494320baseline
ROC AUC.674615.675262challenger

NEGATIVE RESULT

The 240-tree XGBoost challenger was expected to pull away from the calibrated logistic baseline on the 24,000 later-backtest rows. It didn't: the challenger wins 2 of 3 metrics, and every gap is far too small to act on — Brier .159280 vs .159253, log loss .493403 vs .494320, ROC AUC .674615 vs .675262. The model race shifts the third decimal; it is not a basis for choosing between them.

LIMITATION

Expected loss on this page assumes a flat 45% loss-given-default for every approved loan — a disclosed simplification, not a modeled LGD. The backtest default rate itself (21.8%) describes granted-loan-only offline historical association; rejected applicants are absent; no production, causal, regulatory, or fairness claim.

SOURCE · RECEIPTS

Every number opens
the same file.

What was verified
The time-ordered credit backtest ties calibrated scores and policy comparisons to the committed loan artifact.
Evidence class
Pipeline-derived granted-loan records, separate from fictional applications used in the synthetic policy fixture.
Boundary
Rejected applicants are absent and LGD is assumed. The policy comparison establishes no causal effect, production decision, regulatory validation or fairness result.
Files, hashes and methods
public/case-studies/credit-policy-desk/backtest-report.jsonLucisZhang/credit-policy-desk · bbad7e0dbf99
sha256:615db22426cb120e40b2eaf8cd4f7ffef61eac59ae46e408ed39e316f0ffffb2
public/case-studies/credit-policy-desk/methods-evidence.jsonLucisZhang/credit-policy-desk · bbad7e0dbf99
sha256:aed9bf83bfd71a914fd0da70a3425180ac468e437d0e50ff6a778aa9e7c54f0b
public/case-studies/credit-policy-desk/credit-backtest-compact.jsonLucisZhang/portfolio-site · acf05ae78859
sha256:77637de537299d3e19dbfb8fecc8ac644aaa13f253a46f71f63418ca6843d3b6
public/case-studies/credit-policy-desk/policy-frontier-report.jsonLucisZhang/portfolio-site · acf05ae78859
sha256:bba03c36b0b2923bc9c6250593eeb695a7e25eaae2c2c093005eb779b4c230cb
public/case-studies/credit-policy-desk/scored-backtest.parquetLucisZhang/credit-policy-desk · bbad7e0dbf99
sha256:2bbc97350d28123a1b056e4d475cdc90000954df1e6226d54d4fa35f2e7e0b95
View verification SQL
SELECT * FROM read_parquet('scored-backtest.parquet')

This query re-reads the exact committed Parquet bytes in your browser and checks their SHA-256. The logistic/XGBoost training, isotonic calibration, and policy-frontier derivation behind exhibits 01–03 run offline in Python (reproduce commands below), not live in this SQL.

scored-backtest.parquet, backtest-report.json, and methods-evidence.json are produced by the pipeline below from a licensed Lending Club source lock; credit-backtest-compact.json and policy-frontier-report.json are derived from that same committed artifact by two npm scripts. docs/evidence/digits-credit.md pins every number on this page to one of these files.

python3 -m venv .venv && .venv/bin/pip install -r pipelines/credit-backtest/requirements.txt.venv/bin/python pipelines/credit-backtest/build.py --download.venv/bin/python pipelines/credit-backtest/build.py --verify-onlynpm run generate:credit-previewnpm run generate:credit-policy-frontier

public/case-studies/credit-policy-desk/methods-evidence.jsonLucisZhang/credit-policy-desk · bbad7e0dbf99

Pipeline-backed analysis

Open source record

Methods / Results / Real-data analysis.

Dataset Lending Club loan dataset for granting models · CC BY 4.0 · retrieved 2026-07-17
Source → derived 1,347,681 → 120,000 · 2007-06 → 2018-12
Artifact SHA-256 2bbc97350d28123a1b056e4d475cdc90000954df1e6226d54d4fa35f2e7e0b95
120,000Scored applications
0.1593 / 0.1593Baseline / challenger Brier
21.82%Observed backtest defaults

Acquire and clean

  1. Pinned the UCM-curated Zenodo archive and verified its 167,468,415 bytes plus published MD5 and locked SHA-256.
  2. Parsed month vintages, removed 0 duplicate IDs, retained explicit no-information employment encodings, and excluded free text/ZIP from modeling.
  3. Audited 16,657 missing titles and 1,228,582 missing descriptions; neither text field enters the model.
  4. Selected 120,000 applications by deterministic ID hash within disjoint chronological windows so the browser artifact stays reviewable; the resulting IDs are pseudonymous and linkable within the artifact, not anonymous.

Train and estimate

  1. Baseline: L2-regularized logistic regression over scaled numeric and one-hot categorical application-time features.
  2. Nine source features enter both models: log reported income, capped DTI, log requested amount, FICO, entity-experience flag, employment months, purpose, home ownership, and state.
  3. Challenger: 240-tree depth-4 XGBoost; both scores are independently isotonic-calibrated on the calibration window.
  4. Score columns retain float64 precision, and AUC, Brier, log-loss, and default rate are computed after rereading the final Parquet.
  5. Top-three reason codes are the largest absolute per-application XGBoost SHAP contributions and encode whether each contribution increases or decreases PD. EAD is requested amount; LGD is a disclosed 45% assumption.
  6. The legacy display fields utilization, late payments, and bureau age are respectively mapped to requested-amount/income and explicit unavailable sentinels; they are not model inputs.

Split and prevent leakage

  1. Train 72,000 / calibration 24,000 / backtest 24,000; time cutoffs end 2015-12 and 2016-10.
  2. Preprocessing and DTI caps fit on train only; isotonic maps fit on calibration only; all reported metrics use later backtest outcomes.

Outcome / anomaly labels

  1. Observed outcome is the archive's final resolution: charged-off/default = 1 and fully paid = 0.
  2. No transitory loan state is relabeled, and no protected-class or fairness label is inferred.

Quality controls

  1. The pipeline fails closed on the source size, MD5, SHA-256, application grain, split coverage, score bounds, outcome types, and exactly three reason codes; verification recomputes AUC, Brier, log-loss, and default rate from the exact Parquet columns.
  2. The Parquet embeds DOI, authorship, license, hashes, time cutoffs, algorithm versions, sampling rule, metric-recomputation rule, and LGD/display-field assumptions.
What changed with real data

Real is the default requested source. Once scored-backtest.parquet verifies, real mode compares calibrated scores on observed later outcomes: baseline Brier 0.1593, challenger Brier 0.1593, 21.82% observed defaults. If the artifact is missing or invalid, the real path fails closed and the real-artifact state remains explicitly pending or blocked; separately, the active data falls back to the governed synthetic walkthrough.

Reproduce
python3 -m venv .venv && .venv/bin/pip install -r pipelines/credit-backtest/requirements.txt.venv/bin/python pipelines/credit-backtest/build.py --download.venv/bin/python pipelines/credit-backtest/build.py --verify-only

Architecture

I built the time-disciplined training and backtest pipeline, the score-to-policy contracts, the expected-loss and queue engine, and the browser UI that holds it together.

  1. Backtest

    Train, calibrate, and score disjoint chronological Lending Club windows offline.

  2. Economics

    Compute expected loss from PD, LGD, and EAD.

  3. Policy

    Apply approve, review, and decline thresholds.

  4. Review

    Enforce analyst-capacity constraints.

  5. Monitor

    Backtest vintages, PSI, slices, and audit changes.

Results & negatives

The default path verifies committed offline scores, then recomputes approve/review/decline bands, swap sets, queue overflow, expected loss, calibration, vintage drift, descriptive slices, and a policy audit record.

  1. I expected the 240-tree XGBoost challenger to pull away from the calibrated logistic baseline on the 24,000 later-backtest rows. It didn't: Brier 0.159253 against 0.159280, log loss 0.494320 against 0.493403, ROC AUC 0.675262 against 0.674615. Better on two of the three, worse on one, every margin far too small to act on.

  2. The workbench opens on a policy that cannot be published, and that is deliberate. At approve ≤12% / review ≤28% with a review capacity of 180, the fixture's final vintage sends 345 applications to manual review — 165 over capacity — so the capacity gate blocks publication. I did not slide the threshold until the queue fit.

Limitations

  1. Synthetic Brier, PSI, loss, and slice values are fixture results, not real performance or fairness evidence.

  2. The lab does not process PII or represent regulatory compliance, deployed accuracy, or a real applicant decision.

  3. LGD is a disclosed 45% assumption; legacy browser fields absent upstream use explicit unavailable sentinels, and home-ownership slices are descriptive only.

  4. This granted-loan-only archive does not represent rejected applicants or identify acceptance-population policy effects; it is an offline historical backtest, not causal impact, live or production decisioning, regulatory validation, or real-world fairness evidence.