Java 1.8 Migration - Performance and Garbage Collection


Java 7 to Java 8 - that is easy!


I have been working on migrating our web application from Java 1.7 to Java 1.8. Migrating our web app is a lot of challenge. What makes it more challenging is that our web application has a really unique process footprint (well that can be said for all web application). You have to know your application like the back of your hand especially if you want to tune garbage collection for it. When I accepted the challenge of changing our web application from Java 1.7 to Java 1.8. I thought that it was going to just a breeze considering that from 1.7 to 1.8 was not that far of a version. It turned out that I was totally wrong. Here are some of the major challenges that I encountered:


1. Permanent Generation turned into MetaSpace

    Before Java, 1.8 class metadata is located in the permanent generation of the java heap. This can be set using the -XX:PermSize option. This was removed in Java 1.8 (Remove Permanent Generation). The reason why it was removed is because of the convergence effort between the Oracle JRockit JVM and the HotSpot JVM. The JRockit JVM does not have a permanent generation. So the specification proposed the MetaSpace. The MetaSpace is not part of the heap. It will be allocated as part of native memory. So you need to account for the memory size that you use for the MetaSpace. By default, the MetaSpace is unlimited in how much it can grow in native memory. To limit the MetaSpace use the  -XX:MaxMetaspaceSize option. The option -XX:MetaspaceSize is the size when you want to trigger GC on this memory space. I am not sure why you might want to have the values for the previous options to be different. So if you are migrating from Java 1.7 to Java 1.8. You can just practically grab the -XX:PermSize setting and put it in the -XX:MaxMetaspaceSize setting but just take note that the MetaSpace is an additional memory you need on top of whatever your heap size setting is.

2. TieredCompilation is on by default in Java 1.8

   This was one of the challenging issues that I faced.  In Java 1.7 our application has -XX:ReservedCodeCacheSize set to 64 MB (Default size of code cache in 64-bit Java 1.7 JVM without TieredCompliation is 48 MB). So like the MetaSpace I copied this setting. The default size of the code cache for Java 1.8 with TieredCompilation is 240 MB. My smaller code cache setting did not become apparent when I was tuning garbage collection using G1GC from CMS and I was looking at the CPU usage. The CPU Usage is about 50% to 60% higher than usual. The higher CPU usage is caused by the fact that the JVM is not able to optimize more code thru compilation. So what is basically doing is doing more Just-In-Time compilation like an interpreted language. It took me a while to hunt this down but once I set my -XX:ReservedCodeCacheSize to about 256MB that CPU spike went away.

3. Garbage Collection Tuning - the new G1GC 

   Coming from Java 1.7 we are using Concurrent Mark Sweep GC with Parallel Remark and compaction.  I have to start from scratch in order to determine the settings that we will need for our web app with G1GC. After all the testing that we did we were able to settle with these set of parameters:

-XX:+UseG1GC -XX:G1HeapRegionSize=32m -XX:+ParallelRefProcEnabled -XX:MaxTenuringThreshold=6 -XX:InitiatingHeapOccupancyPercent=45 -XX:G1HeapWastePercent=5 -XX:G1RSetUpdatingPauseTimePercent=5

The biggest issue that I have encountered here is the -XX:G1HeapRegionSize - which the default value is determined dynamically based on the heap size.  The region size is how G1 will try and manage the memory blocks in terms of allocation and compaction. When I was looking at our GC logs there were many humongous allocations in our logs which will trigger a lot of GC activity to find space for our big objects (our biggest allocation is about ~20 MB). Once I set this region size to 32 MB that humongous allocation went down and GC activity went down as well.

Conclusion:

Migrating from Java 1.7 to 1.8 will be unique for each application. Just make sure that you separate issues with Garbage collection (if you are tuning garbage collection) and issues with performance (just in time compilers).  I suggest grouping the change in JVM options together if it is for performance (might want to change them first then do testing) and grouping the options for garbage collection (change them once you nailed the performance options already).

Tuning garbage collection was a real challenge too and here is what I have learned:
The most important factor affecting garbage collection it totals available memory. So if you have a chance not to tune garbage collection then don't. It will be cheaper to buy memory and add it to your server than spending the amount of time tuning garbage collection. But if your application really needs a fine tune garbage collection, available memory is still the most important factor affecting GC.
You need to increase your memory as you increase the number of processor on the machine you are running your JVM. Another thing that I have learned is that in Java 1.8 an Out of Memory error can be thrown by GC activity.


Comments

Post a Comment

Popular posts from this blog

OAuth 1.0a Request Signing and Verification - HMAC-SHA1 - HMAC-SHA256

Image
This is a security project that I did this Fall semester. I was interested in why the OAuth1.0a specification did not require you to sign a request using RSA-SHA1 or any RSA algorithm. The first issue that I found is where would you store your private keys in a javascript client? - as a javascript encrypted file ? from the browser ?  - that will be a security hole if you allow javascript to get private keys. These are some of the questions that got me asking and which I cannot find answers to for now. OAuth 1.0a Signature Signing and Verification using JavaScript Implementation Background The OAuth specification tries to define an API authentication method that allows consumer websites or applications to access protected API Resources from a Service provider. The advantage that OAuth provides is that the user of the website or application is not required to disclose their credentials to their service provider. Another advantage of using OAuth to authenticate is that it does not
Read more

Spark DataFrame - Array[ByteBuffer] - IllegalAurmentException

IllegalArgumentException - ByteBuffer - Spark DataFrame I was processing a several million documents (~ 20 million) in which we need to extract the NLP features using NLP4J, OpenNLP, and WordNet. The combination of the three NL features blows up each record to 11 times its original size. We are using all three because we do not know yet what feature sets will be helpful to us. The original dataset is in parquet files in HDFS (16 partitions). I thought that was convenient just use withColumn and pass a UDF (User Defined Function) on the column where it needs those features. withColumn adds the calculated column back to the DataFrame. So I created the spark job (I am on Spark 1.5.2-cdh5.5.2)for the above, and things started to get nasty. I am blowing up the ByteBuffer array on the in-memory columnar storage. This is the exception that I am getting. There seems to be no reference in my code in this stack trace. java.lang.IllegalArgumentException at java.nio.ByteB
Read more

Gensim Doc2Vec on Spark - a quest to get the right Vector

Ever since I joined the R & D group we have been doing a lot of cool things, like trying IBM Watson (see previous blog entry). Now we are doing a lot of Natural language processing.  We wanted to compare the similarity of two documents. There is this excellent project Gensim ( doc2vec ) that easily allow to you to translate large blocks of text to a fixed length feature vector to make comparisons. Here is the link to the  original paper  from some people from Google that explains the approach. In essence, they wanted to find a representation that will overcome the weaknesses of the bag of words model. The doc2vec approach proves to be a reliable approach to comparing similarities of documents because it takes into consideration the semantics and the order of the word in context. So with that, we wanted to use it for a corpus of 26 million documents. Calculating the doc2vec for 26 million documents is not a small task, so we need to process it in Spark. The problem is that there
Read more