Java memory consumption.

ebrostig · Posted: Thu Jun 12, 2003 1:11 am Post subject: Java memory consumption.

I have a small issue that I need some input or ideas on.

We are currently testing a new internal tool that is using a java based htnl server as basis for it's operation. The reason for this is that this needs to be portable and possible to install and run without having to perform a complex setup (type field support or consultants and even customers).

The tool was developed under Solaris using Sun's JDK 1.4.1. I installed it under Gentoo using Sun's JDK 1.4.1 emerged from Portage. It works fine, no issues here whatsoever.

The problem is the amount of memory it consumes under Gentoo compared to under Solaris.

Solaris: 100MB
Gentoo: 237MB
Why?

My other issue is that when performing some specific operations, the memory usage increases up to around 900M under Gentoo, but remains stable at around 100M on Solaris. What can cause this huge difference?

Any ideas are more than welcome. I have tried to strace the whole process but it creates something like a total of ~150 tracefiles when using the -ff option

Kind of hard to wade through all of this.

Erik
_________________
'Yes, Firefox is indeed greater than women. Can women block pops up for you? No. Can Firefox show you naked women? Yes.'

stustill · Posted: Thu Jun 12, 2003 3:31 am Post subject:

I am very surprised by this! I read in the past that Sun developers admitted that the Solaris implentation of the Java VM was by far the worst and tended to hog much more memory than either the Windows or linux implementations. Sorry I don't have any answers, but I am sure this is contray to how the VM's should be performing!

Stu

Donut · n00b Joined: 27 Apr 2003 Posts: 40

This is probably due to most Linuxes lacking a proper implementation of 'native threads' (aka 'Light Weight Processes').

On most JVM implementations on Linux, the JVM just uses 'cloned processes', whenever a Java application creates a thread. This consumes a large amount of memory because the parent O.S. process is completely duplicated. In Solaris/Windows, native cloned processes are not spawned, and most application resources can be shared between the parent native process and the child native threads. You can see this by running something like 'ps aux | grep java' on Linux when the Java app is running. On Solaris/Windows you will see one O.S. process for your app. On Linux you will see as many O.S. processes as there are Threads in your Java App. When this app is a Web Server / App Server, this is likely to be in the hundreds!!!

Implementations of native threading (eg. POSIX Threads) are uncommon in Linux, and the JVMs subsequently needs to utilise these. Unlike most types of applications, the sad truth is that without this, Java applications on Linux have not been as performant as Solaris (and maybe even Windows!!).

To add to this, Solaris (but not Windows) uses a 'many-to-few' threading model instead of a 'one-to-one' threading model. In Linux, each Thread in a Java application will be mapped to a process in the O.S. which can subsequently be scheduled by the O.S., etc. However, on Solaris, all the Java app threads will be mapped to a smaller pool of Light Weight Processes in the O.S.. Again this reduces memory consumption. The Solaris O.S. can schedule/pre-empt the Light Weight Processes, but additionally, inside the Java app itself, the Solaris JVM can also do some scheduling on the internal Java Threads, mapping them to one of the pooled O.S. light weight processes as it sees fit.

Hope this helps

Paul

zhenlin · Veteran Joined: 09 Nov 2002 Posts: 1361

What happens when using:

USE="ntpl" emerge sun-j2sdk and without?

This could have something to do with the garbage collection cycles... See whether they are the same.

Donut · n00b Joined: 27 Apr 2003 Posts: 40

Great stuff - I assume 'nptl' (Native POSIX Thread Library) is Red Hat's recent version which they've added to their distro? This is great news!

This should avoid the cloned process issue. Does anyone know if, in addition, the nptl JVM will use an m:n or 1:1 mapping between Java threads and Native threads?

Cheers

Paul

ebrostig · Posted: Thu Jun 12, 2003 5:13 pm Post subject:

Paul and Zhenlin!

Thank you for the great info! I really appreciate it.

I'm gonna go ahead and re-emerge java with nptl and test to see how it works

I'll let you guys know the result.

Again, thanks!

Erik
_________________
'Yes, Firefox is indeed greater than women. Can women block pops up for you? No. Can Firefox show you naked women? Yes.'

cdmstro · Tux's lil' helper Joined: 26 Apr 2003 Posts: 93

I was just wondering how you determined that the JVM was using that much memory. I am developing java software and was confused when first running java on linux, after running it on Solaris, and the way linux reports it in top, etc lead me to believe it was taking up more memory than it actually was. As said above, it was due to the way Linux handles threads, vice the way Solaris handles threads.

ebrostig · Posted: Fri Jun 13, 2003 7:40 pm Post subject:

Basically what I looked at was the 3 columns:
Virt Res Shr
On Linux they was at start:
237 50 53
Then they grew to:
889 657 XXX (Don't rememebr the last from the top of my head)

On Solaris it was something like:
101 XXX XXX (Don't rememeber exactly here either, can get the numbers though)

And on Solaris this remained the same throughout the whole test.

I'm trying to get glibc compile with nptl to see if it makes a difference but I've hit some errors that I need to sort out first.

Erik
_________________
'Yes, Firefox is indeed greater than women. Can women block pops up for you? No. Can Firefox show you naked women? Yes.'

Donut · n00b Joined: 27 Apr 2003 Posts: 40

I've tried the Native Posix Thread Library VM option and I don't seem to be getting the results I expect.

In addition to the existing blackdown 1.4.1 jdk I had installed, I installed the sun-j2sdk 1.4.1 source version with the 'nptl' option included - I noticed during the installation that some patches were applied to the source as I would expect it to do to include the nptl stuff.

I then used the java-config tool to change my system-wide jdk to the new Sun one with nptl patched in. My java-config --list-available-vms now shows....

zhenlin · Veteran Joined: 09 Nov 2002 Posts: 1361

I forgot to add, you must have a glibc with USE="nptl", i.e. >=glibc-2.3.2

Donut · n00b Joined: 27 Apr 2003 Posts: 40

Ah ok, this is my current glibc config....

ebrostig · Posted: Mon Jun 16, 2003 10:40 pm Post subject:

Donut, nptl works fine with gcc 3.3 and glibc 2.3.2-r2.

I emerged up to gcc 3.3 and glibc 3.2.3-r2 in order to utilize the nptl libraries. I then emerge sun-j2sdk and with great success.

I still have some memory issues though.

The initial memory usage when I start the application is the same, about 230MB. However, after re-running the testcase that previously increased memory usage to around 890M, it now tops out at about 376M. A nice reduction, but still 276MB more than on Solaris. Any ideas how to debug the memory usage?

Erik
_________________
'Yes, Firefox is indeed greater than women. Can women block pops up for you? No. Can Firefox show you naked women? Yes.'

Donut · n00b Joined: 27 Apr 2003 Posts: 40

That's encouraging - maybe I'll take the plunge into the world of masked ebuilds

On the issue of inital memory consumption, this could be more to do with default settings for the JVM on the particular O.S. rather than specific performance capabilities/deficiencies in a particular JVM?

For example, I am not sure what the default initial heap sizes are for the JVM on Solaris compared with the Linux JVM - they may be different. On both Solaris and Linux try setting the minumum and maximum command line heap parameters to fixed values (eg. -Xms256m -Xmx512m).

Also, the Solaris JVM is probably set-up to run in 'server' mode by default which may have different inital memory consumption characteristics. On both Solaris and Linux try setting the '-server' command line paramter. Its also good practice to always pass the -showversion parameter whatever you are doing so that you can always be sure what JVM you are using in what mode (eg. client / server). One thing to note though is that on none-Solaris JVMs, the 'server' mode has tended to be less stable than client so if the server mode works for you, you may want to just keep an eye on it. The latest Linux JVMs may be ok with this?

If you still don't have sucess, you could try tuning some of the garbage collection settings like Zhenlin suggested (eg. nursury sizes for newly created objects, etc). Here's two links which I've found useful in the past...

http://java.sun.com/docs/hotspot/gc/
http://developer.java.sun.com/developer/technicalArticles/Programming/JVMPerf/

java -h and java -X will show 'some' of the JVM parameters which can be tuned.

Also, it may be true that the Solaris JVM will be using less native threads than the linux JVM, because it uses a many-to-few (m:n) mapping of java threads to native-threads whereas I would guess the new nptl Linux JVM uses a one-to-one (1:1) mapping like most other none-Solaris JVMs??

If you want to further investigate you could turn profiling on. You don't necessarily need something like JProbe, OptimiseIt or HPjmeter, you can just set the -Xrunhprof command line parameter with the right settings to get a dump of all the objects in the heap, thread usages, etc. This should give you a good way of comparing the differences between the Solaris and Linux JVMs. The following document has more help on these settings...

http://developer.java.sun.com/developer/onlineTraining/Programming/JDCBook/perf3.html

Let me know how you get on - the results will be interesting.

Cheers

Paul

ebrostig · Posted: Tue Jun 17, 2003 5:24 pm Post subject:

Paul!

Thank you for a bunch of great links! Really appreciated!

I'm going to read through the lot and see what I can do. I 've already tried the -server without any change in memory. I also tried -Xm(xs) without any better result.

I will keep you updated on what I find.

Thanks again!

Erik
_________________
'Yes, Firefox is indeed greater than women. Can women block pops up for you? No. Can Firefox show you naked women? Yes.'

ebrostig · Posted: Wed Jun 18, 2003 9:24 pm Post subject:

I have read through all of the links that were listed above and I have tried a lot of different suggestions, but to no avail. I can't seem to get the memory usage below 537M which is not good

I'm going to look into how sun-j2sdk is compiled too, maybe it is possible to tweak it that way, else this is a total failuer and we can't use the tool under Linux due to excessive memory usage...

Erik
_________________
'Yes, Firefox is indeed greater than women. Can women block pops up for you? No. Can Firefox show you naked women? Yes.'