algorithms.clustering.bgmm¶
Module: algorithms.clustering.bgmm¶
Inheritance diagram for nipy.algorithms.clustering.bgmm:
Bayesian Gaussian Mixture Model Classes: contains the basic fields and methods of Bayesian GMMs the high level functions are/should be binded in C
The base class BGMM relies on an implementation that perfoms Gibbs sampling
A derived class VBGMM uses Variational Bayes inference instead
A third class is introduces to take advnatge of the old C-bindings, but it is limited to diagonal covariance models
Author : Bertrand Thirion, 2008-2011
Classes¶
BGMM¶
-
class
nipy.algorithms.clustering.bgmm.BGMM(k=1, dim=1, means=None, precisions=None, weights=None, shrinkage=None, dof=None)[source]¶ Bases:
nipy.algorithms.clustering.gmm.GMMThis class implements Bayesian GMMs
this class contains the follwing fields k: int,
the number of components in the mixture
- dim: int,
the dimension of the data
- means: array of shape (k, dim)
all the means of the components
- precisions: array of shape (k, dim, dim)
the precisions of the componenets
- weights: array of shape (k):
weights of the mixture
- shrinkage: array of shape (k):
scaling factor of the posterior precisions on the mean
- dof: array of shape (k)
the degrees of freedom of the components
- prior_means: array of shape (k, dim):
the prior on the components means
- prior_scale: array of shape (k, dim):
the prior on the components precisions
- prior_dof: array of shape (k):
the prior on the dof (should be at least equal to dim)
- prior_shrinkage: array of shape (k):
scaling factor of the prior precisions on the mean
- prior_weights: array of shape (k)
the prior on the components weights
- shrinkage: array of shape (k):
scaling factor of the posterior precisions on the mean
dof : array of shape (k): the posterior dofs
-
__init__(k=1, dim=1, means=None, precisions=None, weights=None, shrinkage=None, dof=None)[source]¶ Initialize the structure with the dimensions of the problem Eventually provide different terms
-
set_priors(prior_means, prior_weights, prior_scale, prior_dof, prior_shrinkage)[source]¶ Set the prior of the BGMM
- Parameters
prior_means: array of shape (self.k,self.dim)
prior_weights: array of shape (self.k)
prior_scale: array of shape (self.k,self.dim,self.dim)
prior_dof: array of shape (self.k)
prior_shrinkage: array of shape (self.k)
-
guess_priors(x, nocheck=0)[source]¶ Set the priors in order of having them weakly uninformative this is from Fraley and raftery; Journal of Classification 24:155-181 (2007)
- Parameters
x, array of shape (nb_samples,self.dim)
the data used in the estimation process
nocheck: boolean, optional,
if nocheck==True, check is skipped
-
initialize(x)[source]¶ initialize z using a k-means algorithm, then upate the parameters
- Parameters
x: array of shape (nb_samples,self.dim)
the data used in the estimation process
-
pop(z)[source]¶ compute the population, i.e. the statistics of allocation
- Parameters
z array of shape (nb_samples), type = np.int
the allocation variable
- Returns
hist : array shape (self.k) count variable
-
update_weights(z)[source]¶ Given the allocation vector z, resample the weights parameter
- Parameters
z array of shape (nb_samples), type = np.int
the allocation variable
-
update_means(x, z)[source]¶ Given the allocation vector z, and the corresponding data x, resample the mean
- Parameters
x: array of shape (nb_samples,self.dim)
the data used in the estimation process
z: array of shape (nb_samples), type = np.int
the corresponding classification
-
update_precisions(x, z)[source]¶ Given the allocation vector z, and the corresponding data x, resample the precisions
- Parameters
x array of shape (nb_samples,self.dim)
the data used in the estimation process
z array of shape (nb_samples), type = np.int
the corresponding classification
-
update(x, z)[source]¶ update function (draw a sample of the GMM parameters)
- Parameters
x array of shape (nb_samples,self.dim)
the data used in the estimation process
z array of shape (nb_samples), type = np.int
the corresponding classification
-
sample_indicator(like)[source]¶ sample the indicator from the likelihood
- Parameters
like: array of shape (nb_samples,self.k)
component-wise likelihood
- Returns
z: array of shape(nb_samples): a draw of the membership variable
-
sample(x, niter=1, mem=0, verbose=0)[source]¶ sample the indicator and parameters
- Parameters
x array of shape (nb_samples,self.dim)
the data used in the estimation process
niter=1 : the number of iterations to perform
mem=0: if mem, the best values of the parameters are computed
verbose=0: verbosity mode
- Returns
best_weights: array of shape (self.k)
best_means: array of shape (self.k, self.dim)
best_precisions: array of shape (self.k, self.dim, self.dim)
possibleZ: array of shape (nb_samples, niter)
the z that give the highest posterior to the data is returned first
-
sample_and_average(x, niter=1, verbose=0)[source]¶ sample the indicator and parameters the average values for weights,means, precisions are returned
- Parameters
x = array of shape (nb_samples,dim)
the data from which bic is computed
niter=1: number of iterations
- Returns
weights: array of shape (self.k)
means: array of shape (self.k,self.dim)
precisions: array of shape (self.k,self.dim,self.dim)
or (self.k, self.dim) these are the average parameters across samplings
Notes
All this makes sense only if no label switching as occurred so this is wrong in general (asymptotically).
fix: implement a permutation procedure for components identification
-
probability_under_prior()[source]¶ Compute the probability of the current parameters of self given the priors
-
conditional_posterior_proba(x, z, perm=None)[source]¶ Compute the probability of the current parameters of self given x and z
- Parameters
x: array of shape (nb_samples, dim),
the data from which bic is computed
z: array of shape (nb_samples), type = np.int,
the corresponding classification
perm: array ok shape(nperm, self.k),typ=np.int, optional
all permutation of z under which things will be recomputed By default, no permutation is performed
-
bayes_factor(x, z, nperm=0, verbose=0)[source]¶ Evaluate the Bayes Factor of the current model using Chib’s method
- Parameters
x: array of shape (nb_samples,dim)
the data from which bic is computed
z: array of shape (nb_samples), type = np.int
the corresponding classification
nperm=0: int
the number of permutations to sample to model the label switching issue in the computation of the Bayes Factor By default, exhaustive permutations are used
verbose=0: verbosity mode
- Returns
bf (float) the computed evidence (Bayes factor)
Notes
See: Marginal Likelihood from the Gibbs Output Journal article by Siddhartha Chib; Journal of the American Statistical Association, Vol. 90, 1995
-
average_log_like(x, tiny=1e-15)¶ returns the averaged log-likelihood of the mode for the dataset x
- Parameters
x: array of shape (n_samples,self.dim)
the data used in the estimation process
tiny = 1.e-15: a small constant to avoid numerical singularities
-
bic(like, tiny=1e-15)¶ Computation of bic approximation of evidence
- Parameters
like, array of shape (n_samples, self.k)
component-wise likelihood
tiny=1.e-15, a small constant to avoid numerical singularities
- Returns
the bic value, float
-
check_x(x)¶ essentially check that x.shape[1]==self.dim
x is returned with possibly reshaping
-
estimate(x, niter=100, delta=0.0001, verbose=0)¶ Estimation of the model given a dataset x
- Parameters
x array of shape (n_samples,dim)
the data from which the model is estimated
niter=100: maximal number of iterations in the estimation process
delta = 1.e-4: increment of data likelihood at which
convergence is declared
verbose=0: verbosity mode
- Returns
bic : an asymptotic approximation of model evidence
-
guess_regularizing(x, bcheck=1)¶ Set the regularizing priors as weakly informative according to Fraley and raftery; Journal of Classification 24:155-181 (2007)
- Parameters
x array of shape (n_samples,dim)
the data used in the estimation process
-
initialize_and_estimate(x, z=None, niter=100, delta=0.0001, ninit=1, verbose=0)¶ Estimation of self given x
- Parameters
x array of shape (n_samples,dim)
the data from which the model is estimated
z = None: array of shape (n_samples)
a prior labelling of the data to initialize the computation
niter=100: maximal number of iterations in the estimation process
delta = 1.e-4: increment of data likelihood at which
convergence is declared
ninit=1: number of initialization performed
to reach a good solution
verbose=0: verbosity mode
- Returns
the best model is returned
-
likelihood(x)¶ return the likelihood of the model for the data x the values are weighted by the components weights
- Parameters
x array of shape (n_samples,self.dim)
the data used in the estimation process
- Returns
like, array of shape(n_samples,self.k)
component-wise likelihood
-
map_label(x, like=None)¶ return the MAP labelling of x
- Parameters
x array of shape (n_samples,dim)
the data under study
like=None array of shape(n_samples,self.k)
component-wise likelihood if like==None, it is recomputed
- Returns
z: array of shape(n_samples): the resulting MAP labelling
of the rows of x
-
mixture_likelihood(x)¶ Returns the likelihood of the mixture for x
- Parameters
x: array of shape (n_samples,self.dim)
the data used in the estimation process
-
plugin(means, precisions, weights)¶ Set manually the weights, means and precision of the model
- Parameters
means: array of shape (self.k,self.dim)
precisions: array of shape (self.k,self.dim,self.dim)
or (self.k, self.dim)
weights: array of shape (self.k)
-
show(x, gd, density=None, axes=None)¶ Function to plot a GMM, still in progress Currently, works only in 1D and 2D
- Parameters
x: array of shape(n_samples, dim)
the data under study
gd: GridDescriptor instance
density: array os shape(prod(gd.n_bins))
density of the model one the discrete grid implied by gd by default, this is recomputed
-
show_components(x, gd, density=None, mpaxes=None)¶ Function to plot a GMM – Currently, works only in 1D
- Parameters
x: array of shape(n_samples, dim)
the data under study
gd: GridDescriptor instance
density: array os shape(prod(gd.n_bins))
density of the model one the discrete grid implied by gd by default, this is recomputed
mpaxes: axes handle to make the figure, optional,
if None, a new figure is created
-
test(x, tiny=1e-15)¶ Returns the log-likelihood of the mixture for x
- Parameters
x array of shape (n_samples,self.dim)
the data used in the estimation process
- Returns
ll: array of shape(n_samples)
the log-likelihood of the rows of x
-
train(x, z=None, niter=100, delta=0.0001, ninit=1, verbose=0)¶ Idem initialize_and_estimate
-
unweighted_likelihood(x)¶ return the likelihood of each data for each component the values are not weighted by the component weights
- Parameters
x: array of shape (n_samples,self.dim)
the data used in the estimation process
- Returns
like, array of shape(n_samples,self.k)
unweighted component-wise likelihood
Notes
Hopefully faster
-
unweighted_likelihood_(x)¶ return the likelihood of each data for each component the values are not weighted by the component weights
- Parameters
x: array of shape (n_samples,self.dim)
the data used in the estimation process
- Returns
like, array of shape(n_samples,self.k)
unweighted component-wise likelihood
VBGMM¶
-
class
nipy.algorithms.clustering.bgmm.VBGMM(k=1, dim=1, means=None, precisions=None, weights=None, shrinkage=None, dof=None)[source]¶ Bases:
nipy.algorithms.clustering.bgmm.BGMMSubclass of Bayesian GMMs (BGMM) that implements Variational Bayes estimation of the parameters
-
__init__(k=1, dim=1, means=None, precisions=None, weights=None, shrinkage=None, dof=None)[source]¶ Initialize the structure with the dimensions of the problem Eventually provide different terms
-
evidence(x, like=None, verbose=0)[source]¶ computation of evidence bound aka free energy
- Parameters
x array of shape (nb_samples,dim)
the data from which evidence is computed
like=None: array of shape (nb_samples, self.k), optional
component-wise likelihood If None, it is recomputed
verbose=0: verbosity model
- Returns
ev (float) the computed evidence
-
initialize(x)[source]¶ initialize z using a k-means algorithm, then upate the parameters
- Parameters
x: array of shape (nb_samples,self.dim)
the data used in the estimation process
-
map_label(x, like=None)[source]¶ return the MAP labelling of x
- Parameters
x array of shape (nb_samples,dim)
the data under study
like=None array of shape(nb_samples,self.k)
component-wise likelihood if like==None, it is recomputed
- Returns
z: array of shape(nb_samples): the resulting MAP labelling
of the rows of x
-
estimate(x, niter=100, delta=0.0001, verbose=0)[source]¶ estimation of self given x
- Parameters
x array of shape (nb_samples,dim)
the data from which the model is estimated
z = None: array of shape (nb_samples)
a prior labelling of the data to initialize the computation
niter=100: maximal number of iterations in the estimation process
delta = 1.e-4: increment of data likelihood at which
convergence is declared
verbose=0:
verbosity mode
-
average_log_like(x, tiny=1e-15)¶ returns the averaged log-likelihood of the mode for the dataset x
- Parameters
x: array of shape (n_samples,self.dim)
the data used in the estimation process
tiny = 1.e-15: a small constant to avoid numerical singularities
-
bayes_factor(x, z, nperm=0, verbose=0)¶ Evaluate the Bayes Factor of the current model using Chib’s method
- Parameters
x: array of shape (nb_samples,dim)
the data from which bic is computed
z: array of shape (nb_samples), type = np.int
the corresponding classification
nperm=0: int
the number of permutations to sample to model the label switching issue in the computation of the Bayes Factor By default, exhaustive permutations are used
verbose=0: verbosity mode
- Returns
bf (float) the computed evidence (Bayes factor)
Notes
See: Marginal Likelihood from the Gibbs Output Journal article by Siddhartha Chib; Journal of the American Statistical Association, Vol. 90, 1995
-
bic(like, tiny=1e-15)¶ Computation of bic approximation of evidence
- Parameters
like, array of shape (n_samples, self.k)
component-wise likelihood
tiny=1.e-15, a small constant to avoid numerical singularities
- Returns
the bic value, float
-
check()¶ Checking the shape of sifferent matrices involved in the model
-
check_x(x)¶ essentially check that x.shape[1]==self.dim
x is returned with possibly reshaping
-
conditional_posterior_proba(x, z, perm=None)¶ Compute the probability of the current parameters of self given x and z
- Parameters
x: array of shape (nb_samples, dim),
the data from which bic is computed
z: array of shape (nb_samples), type = np.int,
the corresponding classification
perm: array ok shape(nperm, self.k),typ=np.int, optional
all permutation of z under which things will be recomputed By default, no permutation is performed
-
guess_priors(x, nocheck=0)¶ Set the priors in order of having them weakly uninformative this is from Fraley and raftery; Journal of Classification 24:155-181 (2007)
- Parameters
x, array of shape (nb_samples,self.dim)
the data used in the estimation process
nocheck: boolean, optional,
if nocheck==True, check is skipped
-
guess_regularizing(x, bcheck=1)¶ Set the regularizing priors as weakly informative according to Fraley and raftery; Journal of Classification 24:155-181 (2007)
- Parameters
x array of shape (n_samples,dim)
the data used in the estimation process
-
initialize_and_estimate(x, z=None, niter=100, delta=0.0001, ninit=1, verbose=0)¶ Estimation of self given x
- Parameters
x array of shape (n_samples,dim)
the data from which the model is estimated
z = None: array of shape (n_samples)
a prior labelling of the data to initialize the computation
niter=100: maximal number of iterations in the estimation process
delta = 1.e-4: increment of data likelihood at which
convergence is declared
ninit=1: number of initialization performed
to reach a good solution
verbose=0: verbosity mode
- Returns
the best model is returned
-
likelihood(x)[source]¶ return the likelihood of the model for the data x the values are weighted by the components weights
- Parameters
x: array of shape (nb_samples, self.dim)
the data used in the estimation process
- Returns
like: array of shape(nb_samples, self.k)
component-wise likelihood
-
mixture_likelihood(x)¶ Returns the likelihood of the mixture for x
- Parameters
x: array of shape (n_samples,self.dim)
the data used in the estimation process
-
plugin(means, precisions, weights)¶ Set manually the weights, means and precision of the model
- Parameters
means: array of shape (self.k,self.dim)
precisions: array of shape (self.k,self.dim,self.dim)
or (self.k, self.dim)
weights: array of shape (self.k)
-
probability_under_prior()¶ Compute the probability of the current parameters of self given the priors
-
sample(x, niter=1, mem=0, verbose=0)¶ sample the indicator and parameters
- Parameters
x array of shape (nb_samples,self.dim)
the data used in the estimation process
niter=1 : the number of iterations to perform
mem=0: if mem, the best values of the parameters are computed
verbose=0: verbosity mode
- Returns
best_weights: array of shape (self.k)
best_means: array of shape (self.k, self.dim)
best_precisions: array of shape (self.k, self.dim, self.dim)
possibleZ: array of shape (nb_samples, niter)
the z that give the highest posterior to the data is returned first
-
sample_and_average(x, niter=1, verbose=0)¶ sample the indicator and parameters the average values for weights,means, precisions are returned
- Parameters
x = array of shape (nb_samples,dim)
the data from which bic is computed
niter=1: number of iterations
- Returns
weights: array of shape (self.k)
means: array of shape (self.k,self.dim)
precisions: array of shape (self.k,self.dim,self.dim)
or (self.k, self.dim) these are the average parameters across samplings
Notes
All this makes sense only if no label switching as occurred so this is wrong in general (asymptotically).
fix: implement a permutation procedure for components identification
-
sample_indicator(like)¶ sample the indicator from the likelihood
- Parameters
like: array of shape (nb_samples,self.k)
component-wise likelihood
- Returns
z: array of shape(nb_samples): a draw of the membership variable
-
set_priors(prior_means, prior_weights, prior_scale, prior_dof, prior_shrinkage)¶ Set the prior of the BGMM
- Parameters
prior_means: array of shape (self.k,self.dim)
prior_weights: array of shape (self.k)
prior_scale: array of shape (self.k,self.dim,self.dim)
prior_dof: array of shape (self.k)
prior_shrinkage: array of shape (self.k)
-
show(x, gd, density=None, axes=None)¶ Function to plot a GMM, still in progress Currently, works only in 1D and 2D
- Parameters
x: array of shape(n_samples, dim)
the data under study
gd: GridDescriptor instance
density: array os shape(prod(gd.n_bins))
density of the model one the discrete grid implied by gd by default, this is recomputed
-
show_components(x, gd, density=None, mpaxes=None)¶ Function to plot a GMM – Currently, works only in 1D
- Parameters
x: array of shape(n_samples, dim)
the data under study
gd: GridDescriptor instance
density: array os shape(prod(gd.n_bins))
density of the model one the discrete grid implied by gd by default, this is recomputed
mpaxes: axes handle to make the figure, optional,
if None, a new figure is created
-
test(x, tiny=1e-15)¶ Returns the log-likelihood of the mixture for x
- Parameters
x array of shape (n_samples,self.dim)
the data used in the estimation process
- Returns
ll: array of shape(n_samples)
the log-likelihood of the rows of x
-
train(x, z=None, niter=100, delta=0.0001, ninit=1, verbose=0)¶ Idem initialize_and_estimate
-
unweighted_likelihood(x)¶ return the likelihood of each data for each component the values are not weighted by the component weights
- Parameters
x: array of shape (n_samples,self.dim)
the data used in the estimation process
- Returns
like, array of shape(n_samples,self.k)
unweighted component-wise likelihood
Notes
Hopefully faster
-
unweighted_likelihood_(x)¶ return the likelihood of each data for each component the values are not weighted by the component weights
- Parameters
x: array of shape (n_samples,self.dim)
the data used in the estimation process
- Returns
like, array of shape(n_samples,self.k)
unweighted component-wise likelihood
-
update(x, z)¶ update function (draw a sample of the GMM parameters)
- Parameters
x array of shape (nb_samples,self.dim)
the data used in the estimation process
z array of shape (nb_samples), type = np.int
the corresponding classification
-
update_means(x, z)¶ Given the allocation vector z, and the corresponding data x, resample the mean
- Parameters
x: array of shape (nb_samples,self.dim)
the data used in the estimation process
z: array of shape (nb_samples), type = np.int
the corresponding classification
-
update_precisions(x, z)¶ Given the allocation vector z, and the corresponding data x, resample the precisions
- Parameters
x array of shape (nb_samples,self.dim)
the data used in the estimation process
z array of shape (nb_samples), type = np.int
the corresponding classification
-
update_weights(z)¶ Given the allocation vector z, resample the weights parameter
- Parameters
z array of shape (nb_samples), type = np.int
the allocation variable
-
Functions¶
-
nipy.algorithms.clustering.bgmm.detsh(H)[source]¶ Routine for the computation of determinants of symmetric positive matrices
- Parameters
H array of shape(n,n)
the input matrix, assumed symmmetric and positive
- Returns
dh: float, the determinant
-
nipy.algorithms.clustering.bgmm.dirichlet_eval(w, alpha)[source]¶ Evaluate the probability of a certain discrete draw w from the Dirichlet density with parameters alpha
- Parameters
w: array of shape (n)
alpha: array of shape (n)
-
nipy.algorithms.clustering.bgmm.dkl_dirichlet(w1, w2)[source]¶ Returns the KL divergence between two dirichlet distribution
- Parameters
w1: array of shape(n),
the parameters of the first dirichlet density
w2: array of shape(n),
the parameters of the second dirichlet density
-
nipy.algorithms.clustering.bgmm.dkl_gaussian(m1, P1, m2, P2)[source]¶ Returns the KL divergence between gausians densities
- Parameters
m1: array of shape (n),
the mean parameter of the first density
P1: array of shape(n,n),
the precision parameters of the first density
m2: array of shape (n),
the mean parameter of the second density
P2: array of shape(n,n),
the precision parameters of the second density
-
nipy.algorithms.clustering.bgmm.dkl_wishart(a1, B1, a2, B2)[source]¶ returns the KL divergence bteween two Wishart distribution of parameters (a1,B1) and (a2,B2),
- Parameters
a1: Float,
degrees of freedom of the first density
B1: array of shape(n,n),
scale matrix of the first density
a2: Float,
degrees of freedom of the second density
B2: array of shape(n,n),
scale matrix of the second density
- Returns
dkl: float, the Kullback-Leibler divergence
-
nipy.algorithms.clustering.bgmm.generate_Wishart(n, V)[source]¶ Generate a sample from Wishart density
- Parameters
n: float,
the number of degrees of freedom of the Wishart density
V: array of shape (n,n)
the scale matrix of the Wishart density
- Returns
W: array of shape (n,n)
the draw from Wishart density
-
nipy.algorithms.clustering.bgmm.generate_normals(m, P)[source]¶ Generate a Gaussian sample with mean m and precision P
- Parameters
m array of shape n: the mean vector
P array of shape (n,n): the precision matrix
- Returns
ng : array of shape(n): a draw from the gaussian density
-
nipy.algorithms.clustering.bgmm.generate_perm(k, nperm=100)[source]¶ returns an array of shape(nbperm, k) representing the permutations of k elements
- Parameters
k, int the number of elements to be permuted
nperm=100 the maximal number of permutations
if gamma(k+1)>nperm: only nperm random draws are generated
- Returns
p: array of shape(nperm,k): each row is permutation of k
-
nipy.algorithms.clustering.bgmm.multinomial(probabilities)[source]¶ Generate samples form a miltivariate distribution
- Parameters
probabilities: array of shape (nelements, nclasses):
likelihood of each element belongin to each class each row is assumedt to sum to 1 One sample is draw from each row, resulting in
- Returns
z array of shape (nelements): the draws,
that take values in [0..nclasses-1]
-
nipy.algorithms.clustering.bgmm.normal_eval(mu, P, x, dP=None)[source]¶ Probability of x under normal(mu, inv(P))
- Parameters
mu: array of shape (n),
the mean parameter
P: array of shape (n, n),
the precision matrix
x: array of shape (n),
the data to be evaluated
- Returns
(float) the density
-
nipy.algorithms.clustering.bgmm.wishart_eval(n, V, W, dV=None, dW=None, piV=None)[source]¶ Evaluation of the probability of W under Wishart(n,V)
- Parameters
n: float,
the number of degrees of freedom (dofs)
V: array of shape (n,n)
the scale matrix of the Wishart density
W: array of shape (n,n)
the sample to be evaluated
dV: float, optional,
determinant of V
dW: float, optional,
determinant of W
piV: array of shape (n,n), optional
inverse of V
- Returns
(float) the density