How to handle large output file and logging

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

How to handle large output file and logging

Arnon Tonmukayakul
Hi all,

I have been developing a model where the output of each component is  
written into a simple text file through DataRecorder.  Now as the  
model gets larger, it is impossible to browse the output data from  
these text files.  The file size is in hundreds of MB and sometimes  
in GB.  Logging the status/error and current progress of the  
simulation is also a problem.

I am looking into log4j for logging and HDF5 for the output file.  I  
don't know much about these two.  Am I on the right track to get my  
output more organized and easy to retrieve?  Other candidates,  
tradeoffs? And.. will these make my simulation faster? My model is  
running on Solaris machine.

Thanks,
Arnon


-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.  Get Certified Today
Register for a JBoss Training Course.  Free Certification Exam
for All Training Attendees Through End of 2005. For more info visit:
http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click
_______________________________________________
Repast-interest mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/repast-interest
Reply | Threaded
Open this post in threaded view
|

Re: How to handle large output file and logging

Steve Korecki
You might be better off sending your output to a database instead of a flat text file.
A text file over 1GB is pretty hard to manage.  Using the built-in data logging for
Java 1.4 you can write a custom formatter that sends the logging output to database
tables.  You could then use SQL, MS Access, Crystal Reports, or some other database
reporting tool to query the content you need.

Here are some links to get you started:

      http://java.sun.com/j2se/1.4.2/docs/guide/util/logging/
      http://javaboutique.internet.com/tutorials/logging_API/

Steve




                                                                                                                                                                                                       
             Arnon Tonmukayakul <[hidden email]>                                                                                                                                                        
             Sent by: [hidden email]                                                                                                                                      
                                                                                                                                                                                                     To
                                                                                         [hidden email]                                                                          
             11/18/2005 03:27 PM                                                                                                                                                                     cc
                                                                                         [hidden email]                                                                                                
                                                                                                                                                                                                Subject
                                                                                         [Repast-interest] How to handle large output file and logging                                                  
                                                                                                                                                                                                       
                                                                                                                                                                                                       
                                                                                                                                                                                                       
                                                                                                                                                                                                       
                                                                                                                                                                                                       
                                                                                                                                                                                                       




Hi all,

I have been developing a model where the output of each component is
written into a simple text file through DataRecorder.  Now as the
model gets larger, it is impossible to browse the output data from
these text files.  The file size is in hundreds of MB and sometimes
in GB.  Logging the status/error and current progress of the
simulation is also a problem.

I am looking into log4j for logging and HDF5 for the output file.  I
don't know much about these two.  Am I on the right track to get my
output more organized and easy to retrieve?  Other candidates,
tradeoffs? And.. will these make my simulation faster? My model is
running on Solaris machine.

Thanks,
Arnon


-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.  Get Certified Today
Register for a JBoss Training Course.  Free Certification Exam
for All Training Attendees Through End of 2005. For more info visit:
http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click
_______________________________________________
Repast-interest mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/repast-interest




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
Repast-interest mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/repast-interest