Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Automatic Differentiation
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
[n00b@localhost]
Apprentice
Apprentice


Joined: 30 Aug 2004
Posts: 211
Location: London, UK

PostPosted: Sun Aug 25, 2013 11:46 pm    Post subject: Automatic Differentiation Reply with quote

I am writing a small example program using automatic differentiation via operator overloading. The program requires the 1st and 2nd order derivatives of the log of the probability density functions of the uniform, normal, lognormal and gamma distributions. I have unit tests that compare my log PDF function with the log of the PDF from another library I am assuming to be correct. I then take the source for the log PDF function and run it through an online derivative calculator to give me the formula for the 1st and 2nd order derivatives of the log PDFs. This gives me the basis for unit testing my log PDF function using my automatic derivative type.

All tests comparing my log PDF with the log of the PDF from the standard library pass. Using the automatic derivative type the tests for the uniform, normal and gamma distribution pass but the 2nd derivative of the lognormal distribution fails.

The log PDF of the normal distribution is:

logpdf(x, mu, sigma) = -0.5 * log(2 * M_PI) - log(sigma) - (x - mu)*(x - mu)/(2 * sigma * sigma)

while the log PDF for the lognormal distribution is:

logpdf(x, mu, sigma) = -0.5 * log(2 * M_PI) - log(sigma) - log(x) - (log(x) - mu)*(log(x) - mu)/(2 * sigma * sigma)

The only additional operator needed to evaluate the derivatives of the lognormal PDF as well as the normal PDF is an overloaded log() operator. I calculated the 1st and 2nd order derivatives shown below but I'm not sure they're correct. Could someone with a better understanding of calculus please check them?

h(x) = log(g(x))
(use log rule and chain rule)
h'(x) = g'(x) / g(x)
(use quotient rule)
h''(x) = (g''(x)g(x) - g'(x)g'(x)) / (g(x)g(x))

I also have a feeling my 2nd derivative divide operator may not be correct. I attempted to derive the 2nd derivative of the quotient rule myself but this wasn't giving correct answers for the unit tests. I looked up the 2nd derivative of the quotient rule on Wikipedia and it gives the answers I expect but doesn't show the derivation. Could someone show me how this is derived?

Thanks!
Back to top
View user's profile Send private message
Akkara
Administrator
Administrator


Joined: 28 Mar 2006
Posts: 4950
Location: &akkara

PostPosted: Sun Aug 25, 2013 11:59 pm    Post subject: Reply with quote

Quote:
I attempted to derive the 2nd derivative of the quotient rule myself [...but couldn't...] Could someone show me how this is derived?

Write as product: D[n(x)/d(x)] == D[n(x) * (1/d(x))]

Use product rule: == n(x) * D[1/d(x)] + (1/d(x)) * D[n(x)]

Use power rule and chain rule for D[1/d(x)]: == -1/((d(x))^2) * D[d(x)]

Substitute for D[1/d(x)]: D[n(x)/d(x)] == n(x) * (-1/((d(x))^2) * D[d(x)]) + (1/d(x)) * D[n(x)]

Use ' notation: n(x) * (-1/((d(x))^2) * d'(x) + (1/d(x)) * n'(x)

Factor out a 1/d(x)^2 == (n(x) * (-1) * d'(x) + d(x) * n'(x)) / (d(x))^2

Rewrite: (d(x) * n'(x) - n(x) * d'(x)) / (d(x))^2
Back to top
View user's profile Send private message
[n00b@localhost]
Apprentice
Apprentice


Joined: 30 Aug 2004
Posts: 211
Location: London, UK

PostPosted: Mon Aug 26, 2013 9:56 am    Post subject: Reply with quote

Is that the 2nd order derivative of the quotient rule? Why is it different to the one on Wikipedia?

Also, what about the 2nd order log rule? Is that correct?
Back to top
View user's profile Send private message
Akkara
Administrator
Administrator


Joined: 28 Mar 2006
Posts: 4950
Location: &akkara

PostPosted: Mon Aug 26, 2013 12:14 pm    Post subject: Reply with quote

Oops! That's the regular 1st order. Didn't notice you wanted 2nd. To get it, run it thru again, similar procedure, longer and more complicated expression. It'll be messy. But you'll get your answer.

The 2nd order log rule, same thing. Write two "D" operators and peel the onion: D[ D[ ln(x) ] ] == D[ 1/x ] == -1/x^2

If you meant 2nd order of ln(f(x)), then it's chain rule and log rule. Twice. And one application of the quotient rule since the 1st derivative has a 1/f(x) in it.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum