"High fidelity" is one of the most abused phrases in synthetic data marketing. Every vendor claims it. Virtually none define it operationally. This post is our attempt to give that phrase a concrete meaning: what properties does a synthetic dataset need to preserve in order to be a valid substitute for real data in ML training, and how do you measure whether those properties are present?
The short answer is that fidelity has three distinct layers, and a synthetic dataset can pass two of them while failing the third. Passing the third one is what actually matters for model quality.
Layer 1: Marginal Distribution Fidelity
The first and easiest check: do the individual columns of the synthetic dataset look like the corresponding columns in the real dataset? For a numerical column like customer age, is the mean close? Is the standard deviation similar? Does the histogram shape roughly match? For a categorical column like product category, are the class frequencies preserved?
This is what most synthetic data quality tooling reports by default, and it is the least informative layer on its own. A naive synthesis approach can do well here simply by sampling from each column's marginal distribution independently. If the real age column has mean 42, std 14, and a roughly normal shape, you can generate synthetic ages that match those parameters almost perfectly, with zero information about any other column in the dataset.
The Kolmogorov-Smirnov statistic and the Wasserstein distance (also called the earth mover's distance) are the standard tools for measuring marginal fidelity between real and synthetic distributions. A KS statistic below 0.05 and a Wasserstein distance normalized to the column's range below 0.02 are reasonable thresholds for "this column's marginal is well-reproduced." At Synthpylon, we report these per-column as part of the fidelity report generated after every synthesis job.
The reason marginal fidelity alone is insufficient: machine learning models do not learn from columns in isolation. They learn from the joint distribution across columns. A model that learns "older customers with high tenure and low recent activity are at high churn risk" is learning a multivariate pattern that is invisible in any single column's marginal distribution.
Layer 2: Pairwise Correlation Fidelity
The second layer asks: does the synthetic dataset preserve the statistical relationships between pairs of columns? If in the real data customer age and account tenure have a Pearson correlation of 0.61, does the synthetic data have a similar correlation? If product category and return rate have a Spearman correlation structure where three specific categories have elevated return rates, does the synthetic data reflect that structure?
This is where Gaussian copula models earn their value. A Gaussian copula transforms each marginal distribution to a standard normal using its empirical CDF, learns the correlation matrix across all transformed columns, and samples new records from the joint normal with that correlation structure before inverse-transforming back to the original marginals. It is not a perfect model of the joint distribution, but it explicitly preserves pairwise linear dependencies while respecting each column's marginal shape.
Pairwise correlation fidelity is measured using a correlation matrix delta: compute the Pearson or Spearman correlation matrix for both real and synthetic datasets, then take the mean absolute error across all off-diagonal entries. A MAE below 0.05 across the correlation matrix suggests the pairwise structure is well-preserved. For columns with non-linear relationships (which is most real-world data), conditional dependency measures like mutual information give a richer picture.
A synthetic dataset that passes Layer 1 and Layer 2 is useful for many purposes: exploratory analysis, building dashboards, testing data pipeline logic. But it may still fail to train a good model, because pairwise correlations capture at most the second-order structure of the joint distribution. The real data may have three-way interactions, conditional distributions that shift dramatically across subgroups, and tail behaviors that only become visible in combinations of columns that a pairwise analysis misses.
Layer 3: Downstream Model Accuracy (TSTR)
The third layer is the operationally correct one for ML training use cases: train a model on the synthetic data, evaluate it on held-out real data, and measure the accuracy delta relative to a model trained on real data and evaluated on held-out real data. This is the Train-on-Synthetic, Test-on-Real (TSTR) benchmark.
TSTR is the right measure because it directly answers the question you actually care about: will my model trained on synthetic data perform adequately on real-world inputs? If the TSTR accuracy is within a few percentage points of the real-data baseline, the synthetic data is a valid training substitute for that task. If it is substantially worse, the synthesis is missing signal that the model needs.
For tabular binary classification on well-structured datasets, we typically see TSTR accuracy within 2-5 percentage points of the real-data baseline when the synthesis model is well-fitted. For tasks with strong non-linear feature interactions or high-cardinality categoricals, the gap widens. For tasks with very rare positive classes (under 1%), preserving the tail behavior of the minority class is the limiting factor and requires specific handling during synthesis.
We report TSTR as part of our fidelity output. We fit a lightweight gradient-boosted classifier on the synthetic dataset, evaluate it on a held-out real test split, and compare to the same classifier trained on real training data. This is not a substitute for your actual model evaluation, but it is a reliable early signal that the synthetic data contains the joint statistical structure your model needs.
What Fidelity Scores Cannot Tell You
Even a synthetic dataset that scores well on all three layers has one category of limitation that fidelity metrics cannot capture: novel subpopulation behavior. If the synthesis model was fitted on a dataset that overrepresents one demographic group, the synthetic data will overrepresent it too. Fidelity metrics compare synthetic to real, so they will report high fidelity, because the synthetic data accurately reflects the real data's distribution, including its biases.
We are not saying this is a problem specific to synthetic data. Models trained on real data inherit the same sampling biases. But it is worth being explicit: high fidelity means the synthetic data faithfully reproduces the statistical properties of the real training set. If those properties include meaningful sampling biases, the synthetic data inherits them. Debiasing, if needed, happens at the level of your data collection strategy or your class weighting during training, not at the synthesis stage.
How We Use These Three Layers in Practice
When we generate a synthetic dataset for a team, we run all three checks and surface them in the fidelity report: per-column KS statistics for Layer 1, correlation matrix MAE and top-10 correlation delta pairs for Layer 2, and a TSTR result across multiple classifiers for Layer 3. We flag any column or column pair where the fidelity falls below threshold and, in most cases, we can trace the gap to a specific modeling choice: insufficient training rows to fit the conditional distribution, a high-cardinality categorical that needed a different encoding strategy, or a column with structural zeros that required separate handling.
The point is that fidelity is not a single number and should not be reported as one. A synthetic dataset can look excellent on marginals and acceptable on correlations while quietly losing the specific three-way interaction that your churn model depends on. Knowing which layer failed, and why, is how you decide whether to refine the synthesis or accept the current result for the task at hand.