NMST434 : Kernel Density Estimation

2021/2022

Kernel density estimation: Bias and higher order kernels

The kernel density estimators are always biased, because 
E (Subscript[f, n])(x) = E (1/h)K(((x-X)/h)) = ∫(1/h)K(((x-y)/h)) f(y) dy = ∫L(x-y) f(y) dy  
where L(x) is the density of h.Z for h>0 the bandwidth, and Z an independent random variable with density K.
Therefore, the expectation of (Subscript[f, n])(x) is in fact the density of a convolution, i.e. density of the random variable h.Z + X.
Based on an asymptotic expansion, the bias is typically of order O((h^2)). But, if K is taken so that also 
∫K(t)(t^j) d t =0 for j=1,...,p-1, 
then it can be shown that the bias is only of order O((h^p)). However, this condition means that K must be negative at parts of the domain - it can no longer be a density, and can give negative density estimators. Illustrations for X Gaussian, and X Exponential.

Gaussian density

f[x_]=PDF[NormalDistribution[],x]; (* True density *)
K[t_]=PDF[NormalDistribution[],t]; (* Basic kernel *)
Kb[t_]=(1/2)(3-(t^2))(1/(√(2π)))Exp[-(t^2)/2]; (* Higher order kernel *)
fh[x_,h_]=(1/h)Integrate[K[((x-y)/h)]f[y],{y,-∞,∞}]; 
fhb[x_,h_]=(1/h)Integrate[Kb[((x-y)/h)]f[y],{y,-∞,∞}]; 
Manipulate[Plot[{f[x],fh[x,h],fhb[x,h]},{x,-5,5},PlotLegends->{"Density","Second order kernel", "Fourth order kernel"},PlotLabel->"Kernel density estimation"],{h,0.1,1}]
Manipulate[Plot[{f[x],fh[x,h],fhb[x,h]},{x,3,5},PlotLegends->{"Density","Second order kernel", "Fourth order kernel"},PlotLabel->"Kernel density estimation (tail)"],{h,0.1,1}]

Exponential density

fb[x_]=PDF[GammaDistribution[1,1],x];(* True density *)
K[t_]=PDF[NormalDistribution[],t]; (* Basic kernel *)
fhc[x_,h_]=(1/h)Integrate[K[((x-y)/h)]fb[y],{y,-∞,∞}]; 
Manipulate[Plot[{fb[x],fhc[x,h]},{x,-5,5},PlotLegends->{"Density","Second order kernel"},PlotLabel->"Kernel density estimation",PlotRange->All],{h,0.1,1}]