The first question most ML engineers ask when they hear about synthetic training data is: what's the accuracy cost? It's the right question. There's no point in privacy-safe training if the model that comes out isn't good enough to use.
The honest answer is that the accuracy gap depends on how faithfully the synthetic data preserves the joint distribution of the real data. That's not a dodge, it's a precise technical claim. The TSTR (train on synthetic, test on real) accuracy delta is correlated with the fidelity of the generation step, and fidelity is measurable before you commit to training.
We ran a set of internal benchmarks on five tabular classification tasks to understand how this plays out in practice. The methodology and numbers below are from that work. We're sharing them because vague claims about accuracy preservation are common in this space and concrete numbers are harder to find.
Benchmark Methodology
All five tasks used publicly available or internally constructed tabular classification datasets spanning healthcare administration, financial transactions, and customer behavior prediction. For each dataset, we ran the following procedure:
First, a real-data baseline: train a gradient boosting classifier (XGBoost with default hyperparameters) on 80% of real data, evaluate on the held-out 20%. This establishes the performance ceiling for each task.
Second, synthetic data generation: fit a Gaussian copula model on the training partition of real data, generate a synthetic dataset of the same size, with no real records included. Train an identical XGBoost model on the synthetic data only, evaluate on the same real-data held-out 20%. This is the TSTR setup.
Third, fidelity measurement: before training, measure the synthetic data quality using Jensen-Shannon divergence per column (marginal fidelity) and Spearman correlation matrix deviation (bivariate structure fidelity). We use a composite fidelity score where 1.0 is perfect reproduction and 0.0 is random noise.
Test sets are real data only. No synthetic records appear in any test set.
What We Found
Across the five tasks, TSTR AUC ranged from 1.1 to 4.7 percentage points below the real-data baseline. The accuracy gap was tightly correlated with fidelity score: tasks where the synthetic data achieved fidelity above 0.87 had gaps of 1.1 to 2.3 points. Tasks where fidelity was below 0.82 had gaps of 3.1 to 4.7 points.
The task with the smallest gap was a binary classification problem on customer account features with 15 numeric and 4 categorical columns. This dataset had fairly smooth marginal distributions and moderate inter-column correlations, making it a good fit for Gaussian copula generation. AUC delta: 1.1 points.
The task with the largest gap was a multi-class problem on transaction records with 28 features including several high-cardinality categorical columns (merchant category codes, geographic region). High-cardinality categoricals are harder for copula models to represent faithfully, and the joint dependencies between these categoricals and numeric features were meaningful for the classification task. AUC delta: 4.7 points.
We also ran a mixed approach for the high-cardinality task: use synthetic data for numeric features and mode-frequency-sampled categorical features. That reduced the gap to 2.9 points, suggesting the cardinality problem is addressable with better generation strategies, not just an inherent limit of the approach.
What These Numbers Mean for Real Use Cases
A 2-point AUC delta is often negligible in practice. For development-phase training, model architecture search, and feature engineering experiments, a model trained on synthetic data that scores 0.82 AUC when the real-data ceiling is 0.84 tells you almost everything you need to know. You can iterate on features and architecture freely, and the relative performance of different model configurations will mostly hold when you eventually train on real data.
A 4-5 point AUC delta is more significant and depends on the use case. If you're building a fraud detection model where false negatives are costly, a 4-point drop in AUC can translate to a meaningfully higher false negative rate. If you're building a churn prediction model where you're taking broad action on the top-20% risk segment anyway, the difference between 0.76 and 0.80 AUC may be acceptable given the compliance benefit.
We're not saying a 4-point gap is always acceptable. We're saying the decision depends on what accuracy you actually need, not on a universal threshold. The right question is: what AUC is good enough for my use case, and does synthetic-only training get me there?
Fidelity Score as a Pre-Training Signal
The most useful practical takeaway from this work is that fidelity score measured before training is a reliable predictor of downstream accuracy gap. You don't have to wait for TSTR results to know whether your synthetic data is likely to produce an acceptable model. If fidelity is above 0.87, you can expect the accuracy gap to be small. If it's below 0.82, you know upfront that you have a generation quality problem to address before committing to full training.
This is meaningful because fidelity measurement is fast: it runs in seconds on a laptop, before any model training. If fidelity is below your threshold, you have options. You can increase the number of synthetic samples (more synthetic records often improve fidelity by better coverage of the distribution tails). You can switch generation methods (CTGAN handles high-cardinality categoricals better than pure Gaussian copula in some cases). You can inspect which specific columns have poor marginal fidelity and address them.
The loop becomes: generate synthetic data, measure fidelity, adjust generation parameters, repeat until fidelity meets your threshold, then train. This is faster than the alternative loop of train on synthetic, evaluate on real, discover a 5-point accuracy gap, try to figure out why.
Hybrid Approaches and Their Tradeoffs
Some teams use a hybrid: train on synthetic data during development, then fine-tune on a small amount of real data before production deployment. This can close the accuracy gap almost entirely while preserving most of the compliance benefit, since the real data exposure is limited to a small fine-tuning set rather than full training.
The compliance benefit is partial, not complete. The model has seen real records during fine-tuning, and depending on your regulatory environment, that may still require access controls and audit trails even if the exposure is smaller. Whether the hybrid approach is sufficient depends on what specific controls your compliance framework requires.
A fully synthetic training pipeline has a cleaner compliance story: the training environment never touches real records. That story is worth something, especially in health data or financial data contexts where proving data minimization is part of your audit response. Hybrid approaches complicate that story in exchange for better model accuracy. It's a legitimate tradeoff, and the right choice depends on your specific accuracy requirements and compliance environment.
What We'd Do Differently Next Time
These benchmarks used fixed hyperparameters. It's likely that hyperparameters optimized for synthetic data training would close part of the gap, because the synthetic data distribution has slightly different properties than real data and the optimal regularization settings may differ. We didn't tune per-dataset, which means the measured gaps probably overstate what a careful practitioner would see.
The five tasks are also not a representative sample of all tabular classification problems. Highly structured financial transaction data with many categorical features is probably harder for current generation methods than health outcome data with mostly continuous features. The 1.1 to 4.7 range we observed reflects that variation but doesn't cover it comprehensively.
The core finding holds: synthetic data training accuracy is predictable from fidelity score, the gap is often small enough to be acceptable for many use cases, and the combination of pre-training fidelity measurement plus iterative generation adjustment gives you the tools to know in advance whether you're in the acceptable range.