copying VERY large arrays with JAVA  | | | 
March 14th, 2003, 03:58 AM
|
#1 (permalink)
| | Senior Member
Join Date: Oct 2001
Posts: 881
| copying VERY large arrays with JAVA
I'll try and simplify this so that you understand why I'm asking the question...
I am working with a VERY VERY large data set (345600 lines w/ x,y,z coordinates on each line). I have 2 arrays of this size that are created by reading in a file containing them.
I have to separate the data in the main method (every other line goes to either array) and then I create two different objects (from different classes) one for each array (position and velocity). I instantiate each object by using the arrays that I just made as an argument.
here's the question though. .... once i create the 2 objects (1 position, 1 velocity) I don't need the static arrays that are taking up space in the main method. can I just delete them to free up the space
now.. if that isn't clear enough.. try this...
[dataset]
x1,y1,z1
vx1,vy1,vz1
x2,y2,z2
vx2,vy2,vz2
.
.
.
xn,yn,zn
vxn,vyn,vzn
I create two static arrays in my main method
[position array]
x1,y1,z1
x2,y2,z2
........
[velocity array]
vx1,vy1,vz1
vx2,vy2,vz2
......
I create to objects (in the main method) using [position array],[velocity array].
POSobj, VELobj.
Now I no longer need the original two static arrays [pos_ary],[vel_ary] once the objects are created. Furthermore, since they are huge arrays, I'd rather just free up the memory that they are using.
so.... suggestions??
thanks
-Z | 
|  | |
March 14th, 2003, 04:20 AM
|
#2 (permalink)
| | Member
Join Date: Mar 2003 Location: Kanata
Posts: 105
|
__________________
To think there might be giants.
| |
| | |
March 14th, 2003, 04:26 AM
|
#3 (permalink)
| | Senior Member
Join Date: Oct 2001
Posts: 881
|
ok, I gave that a look. however, I have sort of a unique situation. I already know exactly what the size of the array is goinng to be (it is constructed from specific input). I'm not wasting any time resizing the array, so I'm not worried about that.
regardless, that will be a helpful link, so thanks a ton!
please keep the suggestions rolling in!
-Z
Last edited by zskillz : March 14th, 2003 at 04:29 AM.
| |
| | |
March 16th, 2003, 10:02 PM
|
#4 (permalink)
| | Banned
Join Date: Oct 2001
Posts: 447
|
just curious, but how do you know this is an issue? would be helpful... Quote: |
I'll try and simplify this so that you understand why I'm asking the question...
|
There is a system call to gc(), but results will be spotty.
Java garbage collects by 'scope'. Therefore you need to populate your array(s) from file, create your objects, do whatever, then have array(s), go out of scope.
If you populate array(s) in main(), they will be garbage collected when main(), goes out of scope. That appears to be not happening.
Some things to try:
Put the read file, populate array in static(vs instance) function, call from main().
Put the read file, populate array, create objects, in static(vs instance) function, call from main().
get the array creation/use out of main(), so 'scope' not same as main(). get whatever you use for creation/use to got out of scope, let garbage collection handle... | |
| | |
March 18th, 2003, 04:24 AM
|
#5 (permalink)
| | Senior Member
Join Date: Oct 2001
Posts: 881
|
ahh... thank you that is very helpful. I'll have to think about this a bit...
now.. I don't actually populate them in main... main makes a static method call to the method that populates them.
however, i'm not sure I really understand scope though... I mean... it seems that main is always in scope... ??
thanks q,
-Z | |
| | |
March 18th, 2003, 11:54 AM
|
#6 (permalink)
| | Member
Join Date: Feb 2003 Location: France
Posts: 55
| Re: copying VERY large arrays with JAVA
Quote: Originally posted by zskillz
I am working with a VERY VERY large data set (345600 lines w/ x,y,z coordinates on each line). I have 2 arrays of this size that are created by reading in a file containing them.
here's the question though. .... once i create the 2 objects (1 position, 1 velocity) I don't need the static arrays that are taking up space in the main method. can I just delete them to free up the space | Hey, I'm new to posting here which explains why this
is so late but i just wanted to clear something up.
In Java you can't delete objects that have been created
or directly free the resources that they're using.
Object's are eligible for garbage collection when there's
no more references to that object. You can have multiple
references to the same object, so all references to an
object must be dropped before Garbage Collection could
occur.
You can however guide the JVM into Garbage Collecting
objects that are no longer needed.
An example....
int anArray[] = new int[345600]; //a big object
int anArray = null; &nb sp; //a big object with no references
System.gc(); &n bsp; &nbs p; //a suggestion to the JVM to garbage collect
The call to System.gc is only a suggestion and the JVM
may well ignore your request, but if you've created, used
and set to null all references for a big resource hogging
object then it may well be Garbage Collected.
DaveMark
-------------------------------------
Imagination is more important than knowledge.
-- Albert Einstein | |
| | |
March 18th, 2003, 05:26 PM
|
#7 (permalink)
| | Senior Member
Join Date: Oct 2001
Posts: 881
|
see Dave, that is what I was originally thinking, but that might not be considered good practice... and as you said, it's only a suggestion, so that might not actually be freed!
-Z | |
| | |
March 18th, 2003, 05:55 PM
|
#8 (permalink)
| | Member
Join Date: Feb 2003 Location: France
Posts: 55
|
yeah Z, but that's the point. The JVM knows how to recover system memory if it needs to. So if you've created redundant objects in memory, suggested garbage collection and it doesn't happen, then it means there's plenty more resources available and the JVM doesn't want to slow things down by running when it doesn' t need to.
As was mentioned by 'qball' if you're arrays were part of a method, when that method had finished, any variables and objects are destroyed memory recovered, but there is no definite time when this happens either. The JVM can make better decisions than us programmers.
DaveMark
-------------------------------------
Imagination is more important than knowledge.
-- Albert Einstein | |
| | |
March 18th, 2003, 06:56 PM
|
#9 (permalink)
| | Senior Member
Join Date: Oct 2001
Posts: 881
|
good call on that... i hadn't htought about it that way...
can you give me any more insight on the "scope" of a method?
thanks
-Z | |
| | |
March 18th, 2003, 09:47 PM
|
#10 (permalink)
| | Banned
Join Date: Oct 2001
Posts: 447
| Quote:
Object's are eligible for garbage collection when there's
no more references to that object.
| That is correct and I believe the java definition of when resources are eligible for gc-ing. As stated gc is really left to JVM, but any resources that go out of scope (vs. =null are generally gc-ed efficiently.
Quote: |
now.. I don't actually populate them in main... main makes a static method call to the method that populates them.
|
Ooh, ouch! So you have one class file with all STATIC functionality? That makes scope tougher. Here's why:
With static (class) methods in java, there is no real reference to the class object itself, your basically running a C program (loose reference). When you run 'java classname', you run the main method, which is static. This effectively 'loads' all static qualified members of classname. I used 'loads' carefully, because it doesn't exactly load all static to memory (if this is an issue, I can explain), it 'loads'... Thus main is in SCOPE (and static members**) until main finished, then everything out of SCOPE. **= not exactly true, but good enough for now.
What does this mean? if you call a static method from main, even if method finished and returned to main, SCOPE of that method unknown for sure. For many reasons...
Here is how to quarantee that the Object you use for the 'file reading, array filling, object creating' mayhem goes out of SCOPE and gets gc-ed effectively from the main method.
You can actually use your existing class to do this, but for ease of use, create new java class, call it qball or whatever. This class will read file and create huge arrays. The tricky part, will be the object creation, but more later.
public class qball
{
public void someMethod()
{
int anArray[] = new int[345600];
int anArray2[] = new int[345600];
//read file
//fill arrays
}
}
==============
compile qball where your main method has access. Now in your main method do this:
...
qball myqball=new qball();
myqball.someMethod();
myqball = null;
//now those arrays, way outsa SCOPE!
...
But I need to instantiate some objects based upon each array val! Yes you do. But they need to be in SCOPE of your main method (I would think), not class qball. Yes they do, and here is a trick that should work.
Create a static method (createObj) on your class (the one that run main) that creates the objects. Then call that from the qball class...
public class qball
{
public void someMethod()
{
int anArray[] = new int[345600];
int anArray2[] = new int[345600];
//read file
//fill arrays
//create objects
yourclass.createObj();
}
}
This when you call:
...
myqball.someMethod();
...
myqball object will create objects through your STATIC class method, for your static main method.
=======================
I'm guessing, none of this makes any sense? | |
| | | Thread Tools | Search this Thread | | | | |
Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | | | | Most Active Discussions  | | | | |   Recent Discussions  | | | | |
 |
hardware
prices 
