When ML teams first try synthetic data, they usually make the same mistake: they generate a dataset, load it straight into their training script, and then spend days debugging why their model is behaving strangely. The synthetic data looked fine in a quick scroll. The column types matched. But something in the distributions or the feature relationships was off, and the model picked it up immediately even if the human reviewer did not.
Quality evaluation before training is not optional. It is the step that turns synthetic data from a risky substitute into a reliable tool. This post walks through a practical checklist we have converged on after building this into our own tooling: what to check, in what order, and what failure modes each check catches.
Step 1: Per-Column Distribution Checks
Start with the most visible layer: do the individual column distributions in the synthetic data match the real data? For continuous columns, you want mean, standard deviation, and percentile distribution (minimum, P5, P25, P50, P75, P95, maximum). For categorical columns, you want class frequency ratios. For boolean columns, the true/false ratio.
The Kolmogorov-Smirnov two-sample test gives you a p-value and a statistic for continuous columns. A KS statistic above 0.1 is a flag worth investigating. The Wasserstein distance (L1 earth mover's distance) is more interpretable: it tells you, in the original units of the column, how much the two distributions differ. A column representing transaction amount in dollars with a Wasserstein distance of $120 is clearly wrong; a column representing a score between 0 and 1 with a Wasserstein distance of 0.03 is probably fine.
For categorical columns, Jensen-Shannon divergence between the real and synthetic class frequency vectors captures both the direction and magnitude of the difference. A JSD above 0.05 on a column with more than 20 categories usually indicates a synthesis artifact: rare categories being over-represented or common categories being under-sampled.
These per-column checks are fast to run and catch the most obvious synthesis failures. They do not tell you whether the joint distribution is correct, but they are a necessary first gate. A synthetic dataset that fails per-column checks is not worth proceeding with.
Step 2: Pairwise Correlation Matrix Comparison
Once per-column distributions look reasonable, check whether the relationships between columns are preserved. Compute the Pearson correlation matrix for all continuous columns and the Spearman correlation matrix if you suspect non-linear relationships. Do the same for the synthetic dataset. Then compute the mean absolute difference across all off-diagonal entries.
A mean absolute correlation delta above 0.08 suggests the synthesis model did not capture the joint distribution well. But the mean can obscure important specifics: a single feature pair with a correlation delta of 0.4 is a serious problem even if 95% of pairs are fine, because that pair may be precisely the relationship your model needs to learn.
Look specifically at the top-10 highest-correlation pairs in the real data, the pairs where the real correlation magnitude is highest. These are the structural relationships that most models rely on. If any of them show a large delta in the synthetic data, that is the most impactful signal to investigate, not the average delta across all pairs.
For mixed-type columns, mutual information provides a unified measure of statistical dependence that works across continuous-continuous, continuous-categorical, and categorical-categorical pairs. Computing mutual information matrices for both real and synthetic data and comparing them column-pair by column-pair is computationally heavier but gives a more complete picture of dependency preservation.
Step 3: Schema and Constraint Validation
If your dataset has relational structure, this step comes before TSTR and can save you a lot of wasted compute. Load the synthetic dataset into a database with the original schema constraints applied and see what happens. Foreign key violations, primary key duplicates, and check constraint failures all surface immediately.
If you do not have an easy path to load with constraints enforced, run these checks programmatically. For each foreign key relationship, check that all child values appear in the parent column. For primary keys, check for duplicate values. For columns with known business rules (amounts must be positive, end dates must be after start dates, status codes must be in a specific allowed set), write assertions and run them against the synthetic data.
A synthetic dataset with referential integrity violations is not suitable for training tasks that involve joins across tables. Fix this at the synthesis configuration level, not at the training preprocessing level.
Step 4: Train-on-Synthetic, Test-on-Real (TSTR)
The TSTR benchmark is the operational gold standard for synthetic data quality in ML contexts. The procedure: split your real dataset into train and test sets using your standard split strategy. Generate a synthetic dataset of the same size as the training split. Train the same model architecture on the synthetic training data and on the real training data. Evaluate both models on the same real test set. The difference in evaluation metric (accuracy, AUC, F1, RMSE, depending on your task) is the TSTR delta.
For a well-fitted synthesis model on a moderately complex tabular classification task, a TSTR AUC delta below 0.05 is achievable. Anything above 0.10 usually indicates a systematic fidelity gap: a feature interaction or distributional property the synthesis model failed to capture. When the TSTR delta is large, go back to Step 2 and look for which column pairs have the highest correlation delta. Those pairs are usually the culprit.
We want to be clear about what TSTR does and does not measure. It measures whether the synthetic data carries enough joint statistical structure to train a model that generalizes to real data. It does not measure privacy: a TSTR delta of zero could theoretically be achieved by a synthesis model that memorized the training data. Privacy evaluation requires a separate membership inference audit. TSTR and privacy are orthogonal concerns, and you need to evaluate both.
Step 5: Membership Inference Audit
A membership inference attack asks: given a synthetic record, can an adversary determine whether a specific real record was used to fit the synthesis model? A successful attack suggests the synthesis model memorized some training examples, which means those examples are partially recoverable from the synthetic data.
The standard approach is the shadow model attack: train multiple "shadow" synthesis models on different subsets of the real data, then train a binary classifier to distinguish between synthetic records that came from models trained on a given real record versus models trained without it. If this classifier achieves better than random performance, the synthesis is leaking information about individual records.
For teams in regulated industries or working with sensitive data, a membership inference audit is not optional. A synthetic dataset that looks statistically excellent but has a high membership inference advantage may still carry legal exposure as personal data or PHI, because it preserves enough information to partially identify individual records from the training set.
At Synthpylon, we run a lightweight membership inference test as part of the default quality report. We flag synthesis runs where the inferred attack advantage exceeds a threshold that would concern a privacy lawyer. The threshold is conservative, not because we are paranoid, but because the cost of a false negative (releasing data with privacy leakage) is much higher than the cost of a false positive (generating a new synthesis run with stronger privacy parameters).
Putting the Checklist Together
In order: per-column distributions, pairwise correlations, schema constraint validation, TSTR benchmark, membership inference audit. Each step catches a different failure mode. Skipping Step 2 because Step 1 passed misses the most common cause of TSTR delta. Skipping Step 5 because everything else looks good is a privacy risk that quality metrics cannot catch.
This is not a complicated checklist to automate. Each of these steps can be expressed as a function that takes a real dataframe and a synthetic dataframe and returns a pass/fail signal with supporting numbers. Building this evaluation into your synthesis pipeline, so it runs automatically every time you generate a new dataset, takes a few hours and saves many debug cycles later.
The teams that get the most value from synthetic data are not the ones who trust it blindly. They are the ones who have a fast, repeatable evaluation loop that tells them quickly when a synthesis run is good enough to proceed and when it needs another iteration.