Question about the noise

[SiGe]

Hi,

What exactly is the definition of the noise in Spearmint? Is it the same as the nugget parameter in Scikit-learn's Gaussian Process (1.7.1.2)? I.e., [Std./Mean]^2? If so, why can't we change this parameter? Any insights why it is set to 0.001 all around the code?

Thanks,
Omid

[JasperSnoek]

In my opinion, 'nugget' is a terrible name for measurement error. However, I won't go into a rant here :-) No it is not in the same units as in Scikit. The noise term is Gaussian observation noise, after normalizing the data (so the units are not in the same scale as in the original problem). The code here either leaves this term at 1e-3 for computational stability if the 'noiseless' flag is passed in, or it marginalizes it out given a spike-and-slab kind of prior (horseshoe). Essentially, all computation is averaged over a variety of likely values of the noise term given the data (i.e. Monte Carlo averaging via sampling over the GP marginal likelihood). A scale term is also sampled in the same way, so the whole procedure can be seen as computing a Gaussian scale mixture (a Student's t process).

[JasperSnoek]

There is some more detail about this in the supplementary materials of this paper: http://www.dmi.usherb.ca/~larocheh/publications/gpopt_nips_appendix.pdf