Asymptotic Variance of the MMD Estimator

Oct 13, 2025 by pinkandtrash.com 144 views

Introduction

In the field of statistics and machine learning, two popular metrics used to measure the similarity between two probability distributions are the Maximum Mean Discrepancy (MMD) and the Earth Mover's Distance (EMD). The MMD is a kernel-based measure that calculates the maximum difference between the means of two distributions, while the EMD is a measure of the minimum cost of transforming one distribution into another. Both metrics have been widely used in various applications, including classification, regression, and clustering. However, a crucial aspect of these metrics is their asymptotic behavior, particularly their variance. In this article, we will delve into the asymptotic variance of the MMD and EMD estimators, providing a comprehensive overview of the underlying theory and its implications.

Background

Given two probability distributions $P_X$ and $P_Y$ over $\mathbb{R}^m$ , we can sample $n$ data points from each distribution, denoted as $x_1,\dots,x_n$ and $y_1,\dots,y_n$ , respectively. The MMD and EMD estimators are then calculated using these samples. The MMD estimator is defined as:

\hat{MMD} = \frac{1}{n^2} \sum_{i=1}^n \sum_{j=1}^n k(x_i, x_j) - \frac{2}{n^2} \sum_{i=1}^n \sum_{j=1}^n k(x_i, y_j) + \frac{1}{n^2} \sum_{i=1}^n \sum_{j=1}^n k(y_i, y_j)

where $k(x, y)$ is a kernel function that measures the similarity between two points $x$ and $y$ . The EMD estimator, on the other hand, is defined as:

\hat{EMD} = \frac{1}{n} \sum_{i=1}^n \min_{j=1,\dots,n} \|x_i - y_j\|

where $\| \cdot \|$ denotes the Euclidean norm.

Asymptotic Variance of the MMD Estimator

The asymptotic variance of the MMD estimator has been extensively studied in the literature. A key result is that the MMD estimator is asymptotically normal with a variance that depends on the kernel function and the underlying distributions. Specifically, it has been shown that:

\sqrt{n} (\hat{MMD} - MMD) \xrightarrow{d} \mathcal{N}(0, \sigma^2_{MMD})

where $MMD$ is the true MMD value, and $\sigma^2_{MMD}$ is the asymptotic variance of the MMD estimator. The expression for $\sigma^2_{MMD}$ involves the kernel function and the covariance matrices of the two distributions.

Asymptotic Variance of the EMD Estimator

The asymptotic variance of the EMD estimator has also been studied, although less extensively than the MMD estimator. A key result is that the EMD estimator is asymptotically normal with a variance that depends on the underlying distributions. Specifically, it has been shown that:

\sqrt{n} (\hat{EMD} - EMD) \xrightarrow{d} \mathcal{N}(0, \sigma^2_{EMD})

where $EMD$ is the true EMD value, and $\sigma^2_{EMD}$ is the asymptotic variance of the EMD estimator. The expression for $\sigma^2_{EMD}$ involves the covariance matrices of the two distributions.

Implications and Future Directions

The asymptotic variance of the MMD and EMD estimators has important implications for their use in practice. For instance, the variance of the estimators can affect the choice of kernel function and the sample size required to achieve a certain level of accuracy. Furthermore, the asymptotic variance can be used to construct confidence intervals and hypothesis tests for the true MMD and EMD values.

In conclusion, the asymptotic variance of the MMD and EMD estimators is a crucial aspect of these metrics, and understanding its behavior is essential for their use in practice. Future research directions include the development of more efficient estimators and the study of their asymptotic properties.

References

[1] Gretton, A., Borgwardt, K. M., Rasch, M., Schölkopf, B., & Smola, A. J. (2007). A kernel two-sample test. Journal of Machine Learning Research, 8, 2341-2357.
[2] Cuturi, M. (2013). Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems (pp. 2292-2300).
[3] Borgwardt, K. M., & Gretton, A. (2007). Kernel methods for measuring independence. Journal of Machine Learning Research, 8, 2005-2018.

Appendix

The following is a brief overview of the mathematical derivations involved in the asymptotic variance of the MMD and EMD estimators.

MMD Estimator

The MMD estimator can be written as:

\hat{MMD} = \frac{1}{n^2} \sum_{i=1}^n \sum_{j=1}^n k(x_i, x_j) - \frac{2}{n^2} \sum_{i=1}^n \sum_{j=1}^n k(x_i, y_j) + \frac{1}{n^2} \sum_{i=1}^n \sum_{j=1}^n k(y_i, y_j)

Using the law of large numbers, we can show that:

\hat{MMD} \xrightarrow{p} MMD

where $MMD$ is the true MMD value. The asymptotic variance of the MMD estimator can then be derived using the delta method.

EMD Estimator

The EMD estimator can be written as:

\hat{EMD} = \frac{1}{n} \sum_{i=1}^n \min_{j=1,\dots,n} \|x_i - y_j\|

Using the law of large numbers, we can show that:

\hat{EMD} \xrightarrow{p} EMD

where $EMD$ is the true EMD value. The asymptotic variance of the EMD estimator can then be derived using the delta method.

Introduction

In our previous article, we discussed the asymptotic variance of the Maximum Mean Discrepancy (MMD) and Earth Mover's Distance (EMD) estimators. These metrics are widely used in machine learning and statistics to measure the similarity between two probability distributions. However, understanding the asymptotic behavior of these estimators is crucial for their use in practice. In this article, we will address some common questions and concerns related to the asymptotic variance of the MMD and EMD estimators.

Q: What is the difference between the MMD and EMD estimators?

A: The MMD estimator is a kernel-based measure that calculates the maximum difference between the means of two distributions, while the EMD estimator is a measure of the minimum cost of transforming one distribution into another. The MMD estimator is typically used for classification and regression tasks, while the EMD estimator is used for clustering and dimensionality reduction tasks.

Q: How do I choose the kernel function for the MMD estimator?

A: The choice of kernel function depends on the underlying distribution and the specific problem you are trying to solve. Common kernel functions include the Gaussian kernel, polynomial kernel, and Laplace kernel. You can also use a combination of kernel functions to achieve better results.

Q: What is the effect of the sample size on the asymptotic variance of the MMD and EMD estimators?

A: The sample size has a significant impact on the asymptotic variance of the MMD and EMD estimators. As the sample size increases, the asymptotic variance decreases, and the estimators become more accurate. However, the rate at which the asymptotic variance decreases depends on the kernel function and the underlying distribution.

Q: Can I use the MMD and EMD estimators for high-dimensional data?

A: Yes, you can use the MMD and EMD estimators for high-dimensional data. However, the choice of kernel function and the sample size may need to be adjusted to achieve good results. Additionally, you may need to use dimensionality reduction techniques, such as PCA or t-SNE, to reduce the dimensionality of the data.

Q: How do I interpret the results of the MMD and EMD estimators?

A: The results of the MMD and EMD estimators can be interpreted in different ways depending on the specific problem you are trying to solve. For classification tasks, the MMD estimator can be used to measure the similarity between the means of two distributions. For clustering tasks, the EMD estimator can be used to measure the minimum cost of transforming one distribution into another.

Q: Can I use the MMD and EMD estimators for non-parametric distributions?

A: Yes, you can use the MMD and EMD estimators for non-parametric distributions. However, the choice of kernel function and the sample size may need to be adjusted to achieve good results. Additionally, you may need to use non-parametric density estimation techniques, such as kernel density estimation, to estimate the underlying distribution.

Q: How do I handle outliers in the data when using the MMD and EMD estimators?

A: Outliers can have a significant impact on the results of the MMD and EMD estimators. To handle outliers, you can use robust kernel functions, such as the Huber kernel, or use data preprocessing techniques, such as winsorization or trimming, to remove outliers from the data.

Q: Can I use the MMD and EMD estimators for time-series data?

A: Yes, you can use the MMD and EMD estimators for time-series data. However, the choice of kernel function and the sample size may need to be adjusted to achieve good results. Additionally, you may need to use time-series analysis techniques, such as spectral analysis or wavelet analysis, to analyze the time-series data.

Conclusion

In conclusion, the asymptotic variance of the MMD and EMD estimators is a crucial aspect of these metrics, and understanding its behavior is essential for their use in practice. By addressing common questions and concerns related to the asymptotic variance of the MMD and EMD estimators, we hope to provide a better understanding of these metrics and their applications in machine learning and statistics.

References

[1] Gretton, A., Borgwardt, K. M., Rasch, M., Schölkopf, B., & Smola, A. J. (2007). A kernel two-sample test. Journal of Machine Learning Research, 8, 2341-2357.
[2] Cuturi, M. (2013). Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems (pp. 2292-2300).
[3] Borgwardt, K. M., & Gretton, A. (2007). Kernel methods for measuring independence. Journal of Machine Learning Research, 8, 2005-2018.

Appendix

The following is a brief overview of the mathematical derivations involved in the asymptotic variance of the MMD and EMD estimators.

MMD Estimator

The MMD estimator can be written as:

\hat{MMD} = \frac{1}{n^2} \sum_{i=1}^n \sum_{j=1}^n k(x_i, x_j) - \frac{2}{n^2} \sum_{i=1}^n \sum_{j=1}^n k(x_i, y_j) + \frac{1}{n^2} \sum_{i=1}^n \sum_{j=1}^n k(y_i, y_j)

Using the law of large numbers, we can show that:

\hat{MMD} \xrightarrow{p} MMD

where $MMD$ is the true MMD value. The asymptotic variance of the MMD estimator can then be derived using the delta method.

EMD Estimator

The EMD estimator can be written as:

\hat{EMD} = \frac{1}{n} \sum_{i=1}^n \min_{j=1,\dots,n} \|x_i - y_j\|

Using the law of large numbers, we can show that:

\hat{EMD} \xrightarrow{p} EMD

where $EMD$ is the true EMD value. The asymptotic variance of the EMD estimator can then be derived using the delta method.

Note that the derivations above are brief and are intended to provide a high-level overview of the mathematical concepts involved. For a more detailed treatment, the reader is referred to the original research papers.