Are these datasets the same?
Efficient, fair two-sample testing
with learned kernels

[ICLR-17]

Hsiao-Yu (Fish) Tung

Heiko Strathmann

Soumyajit De

Aaditya Ramdas

Alex Smola

Arthur Gretton

[ICML-20]

Feng Liu

Wenkai Xu

Jie Lu

Guangquan Zhang

Arthur Gretton

[NeurIPS-21]

Feng Liu

Wenkai Xu

Jie Lu

new!

Namrata Deka

Toronto Womxn in Data Science - April 27, 2022

(Swipe or arrow keys to move through slides; for a menu to jump; to show more.)

Data drift

• Textbook machine learning:
  • Train on i.i.d. samples from some distribution,
  • If it works on , probably works on new samples from
• Really:
  • Train on
  • Pretend there's a that is an i.i.d. sample from
  • If it works on , maybe it sorta works on
  • Deploy on something that's maybe a distribution
    • which might be sort of like
    • but probably changes over time…

This talk

Based on samples and :

• How is different from ?
• Is close enough to for our model?
• Is ?

Two-sample testing

• Given samples from two unknown distributions
• Question: is ?
• Hypothesis testing approach:
• Reject null hypothesis if test statistic

What's a hypothesis test again?

Permutation testing to find

Need

: th quantile of

Classifier two-sample tests

• We need a that's large if , small if

• Can choose as the accuracy of on the test set
  • If , classification impossible:
• Usually better:
  • If , normal (permuting on test set is better)

A more general framework

• C2ST-L:
  • is a classifier's logit:
    log probability is from rather than , plus const
• Basically same:
• What if we use more general features of the data?

Difference between mean embeddings

Only use data through : can kernelize!

	1.0	0.2	0.6
	0.2	1.0	0.5
	0.6	0.5	1.0

	1.0	0.8	0.7
	0.8	1.0	0.6
	0.7	0.6	1.0

	0.3	0.1	0.2
	0.2	0.3	0.3
	0.2	0.1	0.4

What's a kernel again?

• Linear classifiers: ,
• Use a “richer” :
• Can avoid explicit ; instead
• “Kernelized” algorithms access data only through
• gives kernel notion of smoothness

Reproducing Kernel Hilbert Space (RKHS)

• Ex: Gaussian RBF / exponentiated quadratic / squared exponential / …
• Some functions with small :

MMD-based tests

• If is characteristic, iff
• Efficient permutation testing for
  • : converges in distribution
  • : asymptotically normal
• Any characteristic kernel gives consistent test…eventually
• Need enormous if kernel is bad for problem

Deep learning and deep kernels

• C2ST-L is basically MMD with
  • is a (learned) deep net – a learned kernel
• We can generalize some more to deep kernels:
  • is a deep net, maps data points to
  • is a simple kernel on
  • gives MMD as

Optimizing power of MMD tests

• Asymptotics of give us immediately that , , are constants: first term usually dominates
• Pick to maximize an estimate of
• Use from before, get from U-statistic theory
• Can show uniform convergence of estimator

Blobs dataset

Blobs kernels

Blobs results

Investigating a GAN on MNIST

CIFAR-10 vs CIFAR-10.1

Ablation vs classifier-based tests

But…

• What if you don't have much data for your testing problem?
• Need enough data to pick a good kernel
• Also need enough test data to actually detect the difference
• Best split depends on best kernel's quality / how hard to find
  • Don't know that ahead of time; can't try more than one

Meta-testing

• One idea: what if we have related problems?
• Similar setup to meta-learning:(from Wei+ 2018)

Meta-testing for CIFAR-10 vs CIFAR-10.1

• CIFAR-10 has 60,000 images, but CIFAR-10.1 only has 2,031
• Where do we get related data from?
• One option: set up tasks to distinguish classes of CIFAR-10 (airplane vs automobile, airplane vs bird, ...)

One approach (MAML-like)

is, e.g., 5 steps of gradient descent

we learn the initialization, maybe step size, etc

This works, but not as well as we'd hoped…
Initialization might work okay on everything, not really adapt

Another approach: Meta-MKL

Inspired by classic multiple kernel learning

Only need to learn linear combination on test task:
much easier

Theoretical analysis for Meta-MKL

• Same big-O dependence on test task size 😐
• But multiplier is much better:
  based on number of meta-training tasks, not on network size
• Coarse analysis: assumes one meta-tasks is “related” enough
  • We compete with picking the single best related kernel
  • Haven't analyzed meaningfully combining related kernels (yet!)

Results on CIFAR-10.1

But...

• Sometimes we know ahead of time that there are differences that we don't care about
  • In the MNIST GAN criticism, initial attempt just picked out that the GAN outputs numbers that aren't one of the 256 values MNIST has
• Can we find a kernel that can distinguish from ,
  but can't distinguish from ?
• Also useful for fair representation learning
  • e.g. can distinguish “creditworthy” vs not,
    can't distinguish by race

High on one power, low on another

Choose with

• First idea:
  • No good: doesn't balance power appropriately
• Second idea:
  • Can estimate inside the optimization
  • Better, but tends to “stall out” in minimizing

Block estimator [Zaremba+ NeurIPS-13]

• Use previous on blocks, each of size
• Final estimator: average of each block's estimate
  • Each block has previous asymptotics
  • Central limit theorem across blocks
• Power is

MMD-B-Fair

• Choose as
  • is the power of a test with blocks of size
  • We don't actually use a block estimator computationally
  • , have nothing to do with minibatch size
• Representation learning:
  • Deep kernel is
  • could be deep itself, with adversarial optimization
  • For now, just Gaussians with different lengthscales

Adult

Shapes3D

:	:
:	:

Multiple targets / sensitive attributes

Remaining challenges

• MMD-B-Fair fails when and are very correlated
• Meta-testing: more powerful approaches, better analysis
• When , can we tell how they're different?
  • Methods so far: low-, and/or points w/ large critic value
• Avoid the need for data splitting (selective inference)
  • Kübler+ NeurIPS-20 gave one method, but very limited
• Online detection of data shifts

A good takeaway

Combining a deep architecture with a kernel machine that takes the higher-level learned representation as input can be quite powerful.

— Y. Bengio & Y. LeCun (2007), “Scaling Learning Algorithms towards AI”