View previous topic :: View next topic 
Author 
Message 
[n00b@localhost] Apprentice
Joined: 30 Aug 2004 Posts: 266 Location: London, UK

Posted: Sun Aug 25, 2013 11:46 pm Post subject: Automatic Differentiation 


I am writing a small example program using automatic differentiation via operator overloading. The program requires the 1st and 2nd order derivatives of the log of the probability density functions of the uniform, normal, lognormal and gamma distributions. I have unit tests that compare my log PDF function with the log of the PDF from another library I am assuming to be correct. I then take the source for the log PDF function and run it through an online derivative calculator to give me the formula for the 1st and 2nd order derivatives of the log PDFs. This gives me the basis for unit testing my log PDF function using my automatic derivative type.
All tests comparing my log PDF with the log of the PDF from the standard library pass. Using the automatic derivative type the tests for the uniform, normal and gamma distribution pass but the 2nd derivative of the lognormal distribution fails.
The log PDF of the normal distribution is:
logpdf(x, mu, sigma) = 0.5 * log(2 * M_PI)  log(sigma)  (x  mu)*(x  mu)/(2 * sigma * sigma)
while the log PDF for the lognormal distribution is:
logpdf(x, mu, sigma) = 0.5 * log(2 * M_PI)  log(sigma)  log(x)  (log(x)  mu)*(log(x)  mu)/(2 * sigma * sigma)
The only additional operator needed to evaluate the derivatives of the lognormal PDF as well as the normal PDF is an overloaded log() operator. I calculated the 1st and 2nd order derivatives shown below but I'm not sure they're correct. Could someone with a better understanding of calculus please check them?
h(x) = log(g(x))
(use log rule and chain rule)
h'(x) = g'(x) / g(x)
(use quotient rule)
h''(x) = (g''(x)g(x)  g'(x)g'(x)) / (g(x)g(x))
I also have a feeling my 2nd derivative divide operator may not be correct. I attempted to derive the 2nd derivative of the quotient rule myself but this wasn't giving correct answers for the unit tests. I looked up the 2nd derivative of the quotient rule on Wikipedia and it gives the answers I expect but doesn't show the derivation. Could someone show me how this is derived?
Thanks! 

Back to top 


Akkara Administrator
Joined: 28 Mar 2006 Posts: 6270 Location: &akkara

Posted: Sun Aug 25, 2013 11:59 pm Post subject: 


Quote:  I attempted to derive the 2nd derivative of the quotient rule myself [...but couldn't...] Could someone show me how this is derived? 
Write as product: D[n(x)/d(x)] == D[n(x) * (1/d(x))]
Use product rule: == n(x) * D[1/d(x)] + (1/d(x)) * D[n(x)]
Use power rule and chain rule for D[1/d(x)]: == 1/((d(x))^2) * D[d(x)]
Substitute for D[1/d(x)]: D[n(x)/d(x)] == n(x) * (1/((d(x))^2) * D[d(x)]) + (1/d(x)) * D[n(x)]
Use ' notation: n(x) * (1/((d(x))^2) * d'(x) + (1/d(x)) * n'(x)
Factor out a 1/d(x)^2 == (n(x) * (1) * d'(x) + d(x) * n'(x)) / (d(x))^2
Rewrite: (d(x) * n'(x)  n(x) * d'(x)) / (d(x))^2 

Back to top 


[n00b@localhost] Apprentice
Joined: 30 Aug 2004 Posts: 266 Location: London, UK

Posted: Mon Aug 26, 2013 9:56 am Post subject: 


Is that the 2nd order derivative of the quotient rule? Why is it different to the one on Wikipedia?
Also, what about the 2nd order log rule? Is that correct? 

Back to top 


Akkara Administrator
Joined: 28 Mar 2006 Posts: 6270 Location: &akkara

Posted: Mon Aug 26, 2013 12:14 pm Post subject: 


Oops! That's the regular 1st order. Didn't notice you wanted 2nd. To get it, run it thru again, similar procedure, longer and more complicated expression. It'll be messy. But you'll get your answer.
The 2nd order log rule, same thing. Write two "D" operators and peel the onion: D[ D[ ln(x) ] ] == D[ 1/x ] == 1/x^2
If you meant 2nd order of ln(f(x)), then it's chain rule and log rule. Twice. And one application of the quotient rule since the 1st derivative has a 1/f(x) in it. 

Back to top 


