Automatic Differentiation

[[n00b@localhost]]

I am writing a small example program using automatic differentiation via operator overloading. The program requires the 1st and 2nd order derivatives of the log of the probability density functions of the uniform, normal, lognormal and gamma distributions. I have unit tests that compare my log PDF function with the log of the PDF from another library I am assuming to be correct. I then take the source for the log PDF function and run it through an online derivative calculator to give me the formula for the 1st and 2nd order derivatives of the log PDFs. This gives me the basis for unit testing my log PDF function using my automatic derivative type.

All tests comparing my log PDF with the log of the PDF from the standard library pass. Using the automatic derivative type the tests for the uniform, normal and gamma distribution pass but the 2nd derivative of the lognormal distribution fails.

The log PDF of the normal distribution is:

logpdf(x, mu, sigma) = -0.5 * log(2 * M_PI) - log(sigma) - (x - mu)*(x - mu)/(2 * sigma * sigma)

while the log PDF for the lognormal distribution is:

logpdf(x, mu, sigma) = -0.5 * log(2 * M_PI) - log(sigma) - log(x) - (log(x) - mu)*(log(x) - mu)/(2 * sigma * sigma)

The only additional operator needed to evaluate the derivatives of the lognormal PDF as well as the normal PDF is an overloaded log() operator. I calculated the 1st and 2nd order derivatives shown below but I'm not sure they're correct. Could someone with a better understanding of calculus please check them?

h(x) = log(g(x))
(use log rule and chain rule)
h'(x) = g'(x) / g(x)
(use quotient rule)
h''(x) = (g''(x)g(x) - g'(x)g'(x)) / (g(x)g(x))

I also have a feeling my 2nd derivative divide operator may not be correct. I attempted to derive the 2nd derivative of the quotient rule myself but this wasn't giving correct answers for the unit tests. I looked up the 2nd derivative of the quotient rule on Wikipedia and it gives the answers I expect but doesn't show the derivation. Could someone show me how this is derived?

Thanks!

[Akkara]

[[n00b@localhost]]

Is that the 2nd order derivative of the quotient rule? Why is it different to the one on Wikipedia?

Also, what about the 2nd order log rule? Is that correct?

[Akkara]

Oops! That's the regular 1st order. Didn't notice you wanted 2nd. To get it, run it thru again, similar procedure, longer and more complicated expression. It'll be messy. But you'll get your answer.

The 2nd order log rule, same thing. Write two "D" operators and peel the onion: D[ D[ ln(x) ] ] == D[ 1/x ] == -1/x^2

If you meant 2nd order of ln(f(x)), then it's chain rule and log rule. Twice. And one application of the quotient rule since the 1st derivative has a 1/f(x) in it.