Quantcast

out of memory error: GC overhead limit

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

out of memory error: GC overhead limit

Anne-Marie Albrecht

Dear all,

I am using Repast Simphony to program a model to simulate daily household water demands. For this purpose each household agent has connections to its neighbours and to other random household agents.

My complete study area has 340,000 households (+100,000 shall be added over 40 years), but as we are interested in spatial differences, we have subdivided this into 44 submodels.

When I programmed the model I used a study area with only around 350 agents to increase the runtime during programming. I have tried it regularly with another submodel containing 8,000 households. Now that I am done with all the rules, I tried to run it with the biggest submodel as well.

My problem:

With 22,000 household agents I get an “OutofMemory: GC overhead limit exceeded “ error while running this piece of code (first code, then error). I have increased the settings of eclipse to the following, which resolved the error I got first (Java heap limit reached), but led me to the error described above:

-startup

plugins/org.eclipse.equinox.launcher_1.3.0.v20140415-2008.jar

--launcher.library

plugins/org.eclipse.equinox.launcher.win32.win32.x86_64_1.1.200.v20150204-1316

-product

org.eclipse.epp.package.standard.product

--launcher.defaultAction

openFile

--launcher.XXMaxPermSize

256M

-showsplash

org.eclipse.platform

--launcher.XXMaxPermSize

256m

--launcher.defaultAction

openFile

--launcher.appendVmargs

-vmargs

-Dosgi.requiredJavaVersion=1.7

-Xms512m

-Xmx6G

 

When I try to open the run window I get an additional heap space error (Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: Java heap space) again.

I know that I have other pieces of code that presumably are much more inefficient than the initialization of the connections. Could you give me advice how I could solve this error (so that I hopefully don’t have to change my whole code)?

 

I also have a second question regarding the coupling of the submodels. My initial idea was to give a certain available water volume as input to the first submodel, subtract the demand in that subarea and pass this number on to the next submodel and so on until the last submodel. That means each submodel needs to wait until it gets this water volume number from the last submodel for each time step. I have found these commands which seem to (maybe?) be what I need, yet I don’t know how to use them and would be thankful to get an example for that (or whether I need to use something completely different):

RunEnvironment.getInstance().pauseRun() and

resume with RunEnvironment.getInstance().resumeRun()

(I am thinking: Could I tell it to watch a certain line in a txt-file and only if that is filled resume the run? So that each model would have to result in such a filling of a txt-file line?)

 

I have basically subdivided the model because the percentages of households with different size, age, gardens, social grade, etc. are different in each submodel and they are located closer or further away from the reservoir that supplies them with water.

(My second idea is therefore to initialize all the submodels, read their characteristics out into a common excel-file and read that in again, so that all households would be in one context (then of course we lose the hierarchy of water distribution, but the heterogeneity of characteristics would be preserved).

Yet, due to the issue above I wonder whether the model can even work with 340,000 agents at once…)

 

I would be very thankful for advice on these issues (as I am a hydrologist and my programming skills are therefore limited)!

 

Anne-Marie

 

CODE:

//// INITIALIZATION ////////////////////////////////////////////////////////////////////////////////////////////////////////////////

       @ScheduledMethod(start = 0, priority = 50)

       public void getConnections() {

 

              // establishes the connection network of each Hh agent, by connecting it

              // to its neighbors and to a number of other random Hh connections

 

              // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

              // Create connection (/edges) between the Hh agent and its neighbors

              // get location of this Hh agent

              GridPoint pt = grid.getLocation(this);

              // get neighbors of that Hh agent (nghCreator)

              GridCellNgh<Hh> nghCreator = new GridCellNgh<Hh>(grid, pt, Hh.class, 1,    1);

              // put these neighbors in a list

              List<GridCell<Hh>> gridCells = nghCreator.getNeighborhood(false);

 

              // iterate through neighbor cells and only add to connections if they are occupied grid cells

              // for that if cell of Hh-object is the same as the gridCell of the neighbor, then...

              for (GridCell<Hh> cell : gridCells) {

                     // read out X and Y from the gridCell of the neighbor

                     GridPoint ngPt = cell.getPoint();

                     // get the neighbor object if it is a Hh, those can not be empty,

                     // returned as Hh object

                     Object allObj = grid.getObjectAt(ngPt.getX(), ngPt.getY());

 

                     // Avoid nullpointerException and only create an edge if there is another Hh agent

                     if (allObj instanceof Hh) {

                           // get only the object belonging to class Hh and return as a single object, otherwise the code runs into errors using the connections as

                            // it has stored iterable object lists as objects

                           Object obj = (Hh) grid.getObjectAt(ngPt.getX(), ngPt.getY());

                           // add edge/connection between the two Hhs

                           network.addEdge(this, obj);

                           // }

                     }

              }

 

              // /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

              // add a number (numCon) of random connections random connections (numCon specified as parameter)

 

              // helper variable for while-loop

              int i = 0;

              // get all Hh objects into an indexed list

              IndexedIterable Hhs = RunState.getInstance().getMasterContext().getObjects(Hh.class);

 

              while (i < numCon) {

                     // use RandomHelper to create a random index "number" of same range as indexed list

                     int number = RandomHelper.nextIntFromTo(0, Hhs.size() - 1);

                     // get a random object (with random index from above) as new connection

                     Hh obj = (Hh) Hhs.get(number);

                     // form a new edge if the edge does not already exist and the hh agent is not itself

                     if (network.getEdge(this, obj) == null & this.id != obj.getId()) {

                           // increase i by one for while loop only if edge does not exist already --> everybody HAS numCon connections in the end

                           i++;

                           // add edge/connection between the two Hhs

                           network.addEdge(this, obj);

                           // // Checking the outdegree

                           // System.out.println("OutDegree:" +

                           // network.getOutDegree(this));

                           // System.out.println(network.getSuccessors(this));

                           // System.out.println("InDegree:" + network.getInDegree(this));

                           // System.out.println("Degree " + network.getDegree(this));

                     }

              }

 

              // putting all connected edges into an iterable list (as the network is not directed, predecessor and successor - edges are the same)

              Iterable<Object> connectionsIter = network.getSuccessors(this);

              // converting type iterable to type List

              connections = IteratorUtils.toList(connectionsIter.iterator());

              if (this.id == 1) {

                     this.exampleNetwork();

              }

              System.out.println("get con done.");

              long freeMemory = Runtime.getRuntime().freeMemory() / MegaBytes;

            long totalMemory = Runtime.getRuntime().totalMemory() / MegaBytes;

            long maxMemory = Runtime.getRuntime().maxMemory() / MegaBytes;

 

            System.out.println("Used Memory in JVM: " + (maxMemory - freeMemory));

            System.out.println("freeMemory in JVM: " + freeMemory);

            System.out.println("totalMemory in JVM shows current size of java heap : "

                                       + totalMemory);

            System.out.println("maxMemory in JVM: " + maxMemory);

       }

 

 

OUTPUT:

Used Memory in JVM: 32

freeMemory in JVM: 4

totalMemory in JVM shows current size of java heap : 36

maxMemory in JVM: 36

Exception in thread "Thread-5" java.lang.OutOfMemoryError: GC overhead limit exceeded

       at java.util.Arrays.copyOfRange(Unknown Source)

       at java.lang.StringBuffer.toString(Unknown Source)

       at java.io.WinNTFileSystem.normalize(Unknown Source)

       at java.io.WinNTFileSystem.normalize(Unknown Source)

       at java.io.File.<init>(Unknown Source)

       at sun.net.www.protocol.file.Handler.openConnection(Unknown Source)

       at sun.net.www.protocol.file.Handler.openConnection(Unknown Source)

       at java.net.URL.openConnection(Unknown Source)

       at sun.net.www.protocol.jar.JarURLConnection.<init>(Unknown Source)

       at sun.net.www.protocol.jar.Handler.openConnection(Unknown Source)

       at java.net.URL.openConnection(Unknown Source)

       at sun.misc.URLClassPath$Loader.getResource(Unknown Source)

       at sun.misc.URLClassPath.getResource(Unknown Source)

       at java.net.URLClassLoader$1.run(Unknown Source)

       at java.net.URLClassLoader$1.run(Unknown Source)

       at java.security.AccessController.doPrivileged(Native Method)

       at java.net.URLClassLoader.findClass(Unknown Source)

       at org.java.plugin.standard.StandardPluginClassLoader.loadClass(Unknown Source)

       at org.java.plugin.standard.StandardPluginClassLoader.loadClass(Unknown Source)

       at java.lang.ClassLoader.loadClass(Unknown Source)

       at repast.simphony.ui.widget.ErrorLog.addError(ErrorLog.java:52)

       at repast.simphony.ui.RSApplication$1.messageReceived(RSApplication.java:123)

       at simphony.util.messages.MessageCenter.fireMessageEvent(MessageCenter.java:141)

       at simphony.util.messages.MessageCenter.fatal(MessageCenter.java:130)

       at repast.simphony.ui.GUIScheduleRunner$ScheduleLoopRunnable.run(GUIScheduleRunner.java:66)

       at java.lang.Thread.run(Unknown Source)

 

 

 

Anne-Marie Albrecht

Visiting Researcher, Water Management

 

E: [hidden email]

T: +44 (0)1491 822265

 


HR Wallingford Logo

HR Wallingford
Howbery Park, Wallingford, Oxfordshire OX10 8BA, United Kingdom
T: +44 (0)1491 835381     F: +44 (0)1491 832233
www.hrwallingford.com

HR Wallingford Secondary Logo


 



HR Wallingford and its subsidiaries uses faxes and emails for confidential and legally privileged business communications. They do not of themselves create legal commitments. Disclosure to parties other than addressees requires our specific consent. We are not liable for unauthorised disclosures nor reliance upon them.
If you have received this message in error please advise us immediately and destroy all copies of it.

HR Wallingford Limited
Howbery Park, Wallingford, Oxfordshire, OX10 8BA, United Kingdom
Registered in England No. 02562099



------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity planning
reports. http://pubads.g.doubleclick.net/gampad/clk?id=1444514421&iu=/41014381
_______________________________________________
Repast-interest mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/repast-interest
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: out of memory error: GC overhead limit

srcnick
If I understand you correctly, the model is throwing the out of memory error in which case you need to increase the memory available to the model rather than eclipse. In the launcher configuration for the model  (Run -> Run Configurations -> “your model launcher”, arguments tab) you’ll see VM arguments like:

-Xss10M -Xmx400M 

Increase the -Xmx400M to increase the available memory, e.g. -Xmx1g. With enough memory, the 340,000 agents shouldn’t be a problem. You will probably have trouble visualizing them though. That said, I’m not sure why max memory below is 36, that’s very low.

Re. your second issue, can you say more about the submodels and why the need to wait? How are they operating independenly of each other? If they are subcontexts they will not execute code independently without some explicit (and complicated) programming.

Nick


On Jun 15, 2016, at 4:36 AM, Anne-Marie Albrecht <[hidden email]> wrote:

Dear all,
I am using Repast Simphony to program a model to simulate daily household water demands. For this purpose each household agent has connections to its neighbours and to other random household agents. 
My complete study area has 340,000 households (+100,000 shall be added over 40 years), but as we are interested in spatial differences, we have subdivided this into 44 submodels. 
When I programmed the model I used a study area with only around 350 agents to increase the runtime during programming. I have tried it regularly with another submodel containing 8,000 households. Now that I am done with all the rules, I tried to run it with the biggest submodel as well.
My problem:
With 22,000 household agents I get an “OutofMemory: GC overhead limit exceeded “ error while running this piece of code (first code, then error). I have increased the settings of eclipse to the following, which resolved the error I got first (Java heap limit reached), but led me to the error described above:
-startup
plugins/org.eclipse.equinox.launcher_1.3.0.v20140415-2008.jar
--launcher.library
plugins/org.eclipse.equinox.launcher.win32.win32.x86_64_1.1.200.v20150204-1316
-product
org.eclipse.epp.package.standard.product
--launcher.defaultAction
openFile
--launcher.XXMaxPermSize
256M
-showsplash
org.eclipse.platform
--launcher.XXMaxPermSize
256m
--launcher.defaultAction
openFile
--launcher.appendVmargs
-vmargs
-Dosgi.requiredJavaVersion=1.7
-Xms512m
-Xmx6G
 
When I try to open the run window I get an additional heap space error (Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: Java heap space) again.
I know that I have other pieces of code that presumably are much more inefficient than the initialization of the connections. Could you give me advice how I could solve this error (so that I hopefully don’t have to change my whole code)?
 
I also have a second question regarding the coupling of the submodels. My initial idea was to give a certain available water volume as input to the first submodel, subtract the demand in that subarea and pass this number on to the next submodel and so on until the last submodel. That means each submodel needs to wait until it gets this water volume number from the last submodel for each time step. I have found these commands which seem to (maybe?) be what I need, yet I don’t know how to use them and would be thankful to get an example for that (or whether I need to use something completely different): 
RunEnvironment.getInstance().pauseRun() and
resume with RunEnvironment.getInstance().resumeRun()
(I am thinking: Could I tell it to watch a certain line in a txt-file and only if that is filled resume the run? So that each model would have to result in such a filling of a txt-file line?) 
 
I have basically subdivided the model because the percentages of households with different size, age, gardens, social grade, etc. are different in each submodel and they are located closer or further away from the reservoir that supplies them with water.
(My second idea is therefore to initialize all the submodels, read their characteristics out into a common excel-file and read that in again, so that all households would be in one context (then of course we lose the hierarchy of water distribution, but the heterogeneity of characteristics would be preserved).
Yet, due to the issue above I wonder whether the model can even work with 340,000 agents at once…)
 
I would be very thankful for advice on these issues (as I am a hydrologist and my programming skills are therefore limited)!
 
Anne-Marie
 
CODE:
//// INITIALIZATION ////////////////////////////////////////////////////////////////////////////////////////////////////////////////
       @ScheduledMethod(start = 0, priority = 50)
       public void getConnections() {
 
              // establishes the connection network of each Hh agent, by connecting it
              // to its neighbors and to a number of other random Hh connections
 
              // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
              // Create connection (/edges) between the Hh agent and its neighbors
              // get location of this Hh agent
              GridPoint pt = grid.getLocation(this);
              // get neighbors of that Hh agent (nghCreator)
              GridCellNgh<Hh> nghCreator = new GridCellNgh<Hh>(grid, pt, Hh.class, 1,    1);
              // put these neighbors in a list
              List<GridCell<Hh>> gridCells = nghCreator.getNeighborhood(false);
 
              // iterate through neighbor cells and only add to connections if they are occupied grid cells
              // for that if cell of Hh-object is the same as the gridCell of the neighbor, then...
              for (GridCell<Hh> cell : gridCells) {
                     // read out X and Y from the gridCell of the neighbor
                     GridPoint ngPt = cell.getPoint();
                     // get the neighbor object if it is a Hh, those can not be empty,
                     // returned as Hh object
                     Object allObj = grid.getObjectAt(ngPt.getX(), ngPt.getY());
 
                     // Avoid nullpointerException and only create an edge if there is another Hh agent
                     if (allObj instanceof Hh) {
                           // get only the object belonging to class Hh and return as a single object, otherwise the code runs into errors using the connections as
                            // it has stored iterable object lists as objects
                           Object obj = (Hh) grid.getObjectAt(ngPt.getX(), ngPt.getY());
                           // add edge/connection between the two Hhs
                           network.addEdge(this, obj);
                           // }
                     }
              }
 
              // /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
              // add a number (numCon) of random connections random connections (numCon specified as parameter)
 
              // helper variable for while-loop
              int i = 0;
              // get all Hh objects into an indexed list
              IndexedIterable Hhs = RunState.getInstance().getMasterContext().getObjects(Hh.class);
 
              while (i < numCon) {
                     // use RandomHelper to create a random index "number" of same range as indexed list
                     int number = RandomHelper.nextIntFromTo(0, Hhs.size() - 1);
                     // get a random object (with random index from above) as new connection
                     Hh obj = (Hh) Hhs.get(number);
                     // form a new edge if the edge does not already exist and the hh agent is not itself
                     if (network.getEdge(this, obj) == null & this.id != obj.getId()) {
                           // increase i by one for while loop only if edge does not exist already --> everybody HAS numCon connections in the end
                           i++;
                           // add edge/connection between the two Hhs
                           network.addEdge(this, obj);
                           // // Checking the outdegree
                           // System.out.println("OutDegree:" +
                           // network.getOutDegree(this));
                           // System.out.println(network.getSuccessors(this));
                           // System.out.println("InDegree:" + network.getInDegree(this));
                           // System.out.println("Degree " + network.getDegree(this));
                     }
              }
 
              // putting all connected edges into an iterable list (as the network is not directed, predecessor and successor - edges are the same)
              Iterable<Object> connectionsIter = network.getSuccessors(this);
              // converting type iterable to type List
              connections = IteratorUtils.toList(connectionsIter.iterator());
              if (this.id == 1) {
                     this.exampleNetwork();
              }
              System.out.println("get con done.");
              long freeMemory = Runtime.getRuntime().freeMemory() / MegaBytes;
            long totalMemory = Runtime.getRuntime().totalMemory() / MegaBytes;
            long maxMemory = Runtime.getRuntime().maxMemory() / MegaBytes;
 
            System.out.println("Used Memory in JVM: " + (maxMemory - freeMemory));
            System.out.println("freeMemory in JVM: " + freeMemory);
            System.out.println("totalMemory in JVM shows current size of java heap : "
                                       + totalMemory);
            System.out.println("maxMemory in JVM: " + maxMemory);
       }
 
 
OUTPUT:
Used Memory in JVM: 32
freeMemory in JVM: 4
totalMemory in JVM shows current size of java heap : 36
maxMemory in JVM: 36
Exception in thread "Thread-5" java.lang.OutOfMemoryError: GC overhead limit exceeded
       at java.util.Arrays.copyOfRange(Unknown Source)
       at java.lang.StringBuffer.toString(Unknown Source)
       at java.io.WinNTFileSystem.normalize(Unknown Source)
       at java.io.WinNTFileSystem.normalize(Unknown Source)
       at java.io.File.<init>(Unknown Source)
       at sun.net.www.protocol.file.Handler.openConnection(Unknown Source)
       at sun.net.www.protocol.file.Handler.openConnection(Unknown Source)
       at java.net.URL.openConnection(Unknown Source)
       at sun.net.www.protocol.jar.JarURLConnection.<init>(Unknown Source)
       at sun.net.www.protocol.jar.Handler.openConnection(Unknown Source)
       at java.net.URL.openConnection(Unknown Source)
       at sun.misc.URLClassPath$Loader.getResource(Unknown Source)
       at sun.misc.URLClassPath.getResource(Unknown Source)
       at java.net.URLClassLoader$1.run(Unknown Source)
       at java.net.URLClassLoader$1.run(Unknown Source)
       at java.security.AccessController.doPrivileged(Native Method)
       at java.net.URLClassLoader.findClass(Unknown Source)
       at org.java.plugin.standard.StandardPluginClassLoader.loadClass(Unknown Source)
       at org.java.plugin.standard.StandardPluginClassLoader.loadClass(Unknown Source)
       at java.lang.ClassLoader.loadClass(Unknown Source)
       at repast.simphony.ui.widget.ErrorLog.addError(ErrorLog.java:52)
       at repast.simphony.ui.RSApplication$1.messageReceived(RSApplication.java:123)
       at simphony.util.messages.MessageCenter.fireMessageEvent(MessageCenter.java:141)
       at simphony.util.messages.MessageCenter.fatal(MessageCenter.java:130)
       at repast.simphony.ui.GUIScheduleRunner$ScheduleLoopRunnable.run(GUIScheduleRunner.java:66)
       at java.lang.Thread.run(Unknown Source)
 
 
 
Anne-Marie Albrecht
Visiting Researcher, Water Management
 
T: +44 (0)1491 822265
 

HR Wallingford Logo
HR Wallingford
Howbery Park, Wallingford, Oxfordshire OX10 8BA, United Kingdom
T: +44 (0)1491 835381     F: +44 (0)1491 832233
<a href="x-msg://2/www.hrwallingford.com" style="color: purple; text-decoration: underline;" class="">www.hrwallingford.com
HR Wallingford Secondary Logo

 


HR Wallingford and its subsidiaries uses faxes and emails for confidential and legally privileged business communications. They do not of themselves create legal commitments. Disclosure to parties other than addressees requires our specific consent. We are not liable for unauthorised disclosures nor reliance upon them. 
If you have received this message in error please advise us immediately and destroy all copies of it. 

HR Wallingford Limited
Howbery Park, Wallingford, Oxfordshire, OX10 8BA, United Kingdom
Registered in England No. 02562099




------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity planning
reports. http://pubads.g.doubleclick.net/gampad/clk?id=1444514421&iu=/41014381_______________________________________________
Repast-interest mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/repast-interest


------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity planning
reports. http://pubads.g.doubleclick.net/gampad/clk?id=1444514421&iu=/41014381
_______________________________________________
Repast-interest mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/repast-interest
Loading...