Spring Data JPA Meta JpaMetamodelMappingContext Memory Consumption

My Spring Data JPA/Hibernate Application consumes over 2GB of memory at start without a single user hitting it. I am using Hazelcast as the second level cache but I had the same issue when I used ehCache as well so that is probably not the cause of the issue.

I ran a profile with a Heap Dump in Visual VM and I see where the bulk of the memory is being consumed by JpaMetamodelMappingContext and secondary a ton of Map objects. I just need help in deciphering what I am seeing and if this is actually a problem. I do have a hundred classes in the model so this may be normal but I have no point of reference. It just seems a bit excessive.

Once I get a load of 100 concurrent users, my memory consumption increases to 6-7 GB. That is quite normal for the amount of data I push around and cache, but I feel like if I could reduce the initial memory, I'd have a lot more room for growth.

1 Answers

Answers 1

I don't think you have a problem here. Instead, I think you are misinterpreting the data you are looking at.

Note that the heap space diagram displays two numbers: Heap size and Used heap

Heap size (orange) is the amount of memory available to the JVM for the heap. This means it is the amount that the JVM requested at some point from the OS.

Used heap is the part of the Heap size that is actually used. Ignoring the startup phase, it grows linear and then drops repeatedly over time. This is typical behavior of an idling application. Some part of the application generates a moderate amount of garbage (rising part of the curve) which from time to time gets collected.

The low points of that curve are the amount of memory you are actually really using. It seems to be about 250MB which doesn't sound very much to me, especially when you say that the total consumption of 6-7GB when actually working sounds reasonable to you.

Some other observations:

Both CPU load and heap grows fast/fluctuates a lot at start time. This is to be expected because the analysis of repositories and entities happen at that time.

JpaMetamodelMappingContext s retained size is about 23MB. Again, a good chunk of memory, but not that huge. This includes the stuff it references, which is almost exclusively metadata from the JPA implementation as you can easily see when you take a look at its source.

Read More

Red exclamation mark next to object in JNDI Tree

I have successfully deployed a Hazelcast Resource Adapter to Weblogic 12c using a weblogic hazelcast template, which results in a Hazelcast Connection Factory (Main#com.company.HazelcastCF) and the resource itself (Main#com.company.MyHazelcastResource) showing up in the JNDI tree when I go through the server into View JNDI Tree. Everything is functional and ears that depend on this resource adapter deploy successfully, and can use my Hazelcast node without issue.

However, there is a peculiar red exclamation mark next to the two entries connected to my resource adapter in the JNDI tree, like this one:

How can I find out what causes this red exclamation mark to show up - and, assuming something is wrong, fix the underlying issue?

The following is the weblogic-ra.xml in META-INF for the resource adapter.

<?xml version="1.0"?> <weblogic-connector  xmlns="http://xmlns.oracle.com/weblogic/weblogic-connector"     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"     xsi:schemaLocation="http://xmlns.oracle.com/weblogic/weblogic-connector              http://xmlns.oracle.com/weblogic/weblogic-connector/1.0/weblogic-connector.xsd">      <jndi-name>Main#com.company.MyHazelcastResource</jndi-name>     <enable-access-outside-app>true</enable-access-outside-app>     <enable-global-access-to-classes>true</enable-global-access-to-classes>     <outbound-resource-adapter>         <connection-definition-group>             <connection-factory-interface>javax.resource.cci.ConnectionFactory</connection-factory-interface>             <connection-instance>                 <jndi-name>Main#com.company.HazelcastCF</jndi-name>                 <connection-properties>                     <pool-params>                         <initial-capacity>1</initial-capacity>                         <max-capacity>50</max-capacity>                         <capacity-increment>1</capacity-increment>                     </pool-params>                 </connection-properties>             </connection-instance>         </connection-definition-group>     </outbound-resource-adapter> </weblogic-connector> 

And this is a screenshot of the whole JNDI Tree after only the resource adapter being deployed (note that whatever happens onMouseOver for the exclamation mark results in a JavaScript error in the console).

0 Answers

Read More

Wierd Hazelcat IMap#put() behaviour

My Hazelcast-based program can work in two modes: submitter and worker.

Submitter puts some POJO to the distributed map by some key, e.g.: hazelcastInstance.getMap(MAP_NAME).put(key, value);

Worker has an infinite loop (with Thread.sleep(1000L); inside for timeout) which must process entities from map. For now I'm just printing the map size in this loop.

Now here's the problem. I start worker app. Then I start four submitters simultaneously (each adds an entry to the map and terminates it's work). But after all submitter apps are done, the worker app prints arbitrary size: sometimes it detects that only one entry was added, sometimes two, sometimes three (actually it never has seen all four entries).

What is the problem with this simple flow? I've read in Hazelcast docs that put() method is synchronous, so it guarantees that after it returns, entry is placed to distributed map and is replicated. But it doesn't seem so in my experiment.

UPD (code)

Submitter:

public void submit(String key) {     Object mySerializableObject = ...     IMap<String, Object> map = hazelcastInstance.getMap(MAP_NAME);     map.putIfAbsent(key, mySerializableObject, TASK_TTL_IN_HOURS, TimeUnit.HOURS); } 

Worker:

public void process() {     while (true) {         IMap<String, Object> map = hazelcastInstance.getMap(MAP_NAME);         System.out.println(map.size());          // Optional<Map.Entry<String, Object>> objectToProcess = getObjectToProcess();         // objectToProcess.ifPresent(objectToProcess-> processObject(id, objectToProcess));         try {             Thread.sleep(PAUSE);         } catch (InterruptedException e) {             LOGGER.error(e.getMessage(), e);         }     } } 

I commented out "processing" part itself, because now I'm just trying to get consistent state of the map. The code above prints different results each time, e.g.: "4, 3, 1, 1, 1, 1, 1..." (so it can even see 4 submitted tasks for a moment, but then they... disappear).

UPD (log)

Worker:

... tasksMap.size() = 0 tasksMap.size() = 0 tasksMap.size() = 0 tasksMap.size() = 0 tasksMap.size() = 1 tasksMap.size() = 2 tasksMap.size() = 2 tasksMap.size() = 2 tasksMap.size() = 2 tasksMap.size() = 2 ... 

Submitter 1:

Before: tasksMap.size() = 0 After: tasksMap.size() = 1 

Submitter 2:

Before: tasksMap.size() = 1 After: tasksMap.size() = 4 

Submitter 3:

Before: tasksMap.size() = 1 After: tasksMap.size() = 2 

Submitter 4:

Before: tasksMap.size() = 3 After: tasksMap.size() = 4 

2 Answers

Answers 1

Well, I guess, I've figured out the problem. As far as I understand, distributed IMap returned by hazelcastInstance.getMap doesn't guarantee that data is replicated over all existing nodes in the cluster: some portions of data may be replicated to some nodes, another portion - to another nodes. That's why in my example some of submitted tasks were replicated not to worker node (which works perpetually), but to some other submitters, which terminate their execution after submission. So such entries were lost on submitters exit.

I solved this issue by replacing hazelcastInstance.getMap to hazelcastInstance.getReplicatedMap. This method returns ReplicatedMap, which, AFAIK, guarantees that entries placed into it will be replicated to all nodes of the cluster. So now everything works fine in my system.

Answers 2

public interface IMap extends ConcurrentMap, BaseMap Concurrent, distributed, observable and queryable map. This class is not a general-purpose ConcurrentMap implementation! While this class implements the Map interface, it intentionally violates Map's general contract, which mandates the use of the equals method when comparing objects. Instead of the equals method, this implementation compares the serialized byte version of the objects.

Gotchas:

Methods, including but not limited to get, containsKey, containsValue, evict, remove, put, putIfAbsent, replace, lock, unlock, do not use hashCode and equals implementations of keys. Instead, they use hashCode and equals of binary (serialized) forms of the objects. The get method returns a clone of original values, so modifying the returned value does not change the actual value in the map. You should put the modified value back to make changes visible to all nodes. For additional info, see get(Object). Methods, including but not limited to keySet, values, entrySet, return a collection clone of the values. The collection is NOT backed by the map, so changes to the map are NOT reflected in the collection, and vice-versa. This class does not allow null to be used as a key or value.

Read More