copying VERY large arrays with JAVA

[DJDaveMark]

OK, I'm gonna make a suggestion as a shot in the dark. I may be way off...but hey.

I'm gonna assume that zskillz code looks something like this

class Zskillz
{
    public static void main(String [] args)
    {
        foo();
    }
    
    static void foo()
    {
        int anArray1[] = new int[345600];
        int anArray2[] = new int[345600];
        //read file
        //fill arrays
    }
}

So I'm gonna suggest the following amendments

class Zskillz
{
    public static void main(String [] args)
    {
        Zskillz z = new Zskillz();   // 1
        z.foo();                     // 2
        z = null;                    // 3
    }
    
    void foo()        // 4
    {
        int anArray1[] = new int[345600];
        int anArray2[] = new int[345600];
        //read file
        //fill arrays
    }
}

In English...
1. In the main method create an object of the class name your already in (Zskillz).
2. Use this objects variable name (z) to call the method that populates the arrays (foo).
3. Declare the object null
4. Remove the 'static' modifier from the foo method declaration

That should cover both angles being suggested and save you creating extra classes.

Hope this helps,

DaveMark

-------------------------------------
Imagination is more important than knowledge.
-- Albert Einstein

[qball]

what does 4 do?

oops, scoping...

[DJDaveMark]

Quote:

what does 4 do?

Just to clarify.....

Part 4 is the key to modifying zskillz code to take advantage of the 'scope gc-ing' without using any more classes.

A Java class is really just a blue-print for objects you'd like to create. If you only use this blue-print by calling static methods from the main method (itself static) you can think of it as the JVM creating one big explicit area in memory for your program to work in. It gets assigned as much room as it needs coz the JVM knows that's all the room it could possibly need.

Think about it like this. Say a JVM assigns 50 pidgeon holes to a class. This class needs 10 of these for it's member variables and assigns it's 4 static methods 10 pidgeon holes each for their own variables. But if those 4 methods were non-static the JVM could assign the same class only 20 pidgeon holes.....10 for the class member variables and it would re-use the 10 left over assigning it to each method in turn. Thus 'scope gc-ing' any variables

Now this simple example makes the assumption that each method call would not call any of the other methods though. The point being, if you create instances of classes (objects) the JVM dynamically assigns resources it can reclaim when it knows you don't need the resources any more.

Hope that makes sense.
(it would to pidgeons at least!)

DaveMark

Edit: changed my spelling mistakes!

[qball]

Not a pidgeon, just a 'silly wabbit'...

"what does 4 do?" Doh, should have read post more thoroughly.

Or here is what I was thinking:

Create NON-STATIC simple class to read file, pop arrays, create objects(maybe)... Instantiate this class to do the dirty work, that's it. Dereference(?) class asap. Out of scope, gc-ed efficently. Plus, little chance to cross ref original class...

Sure you can do all in one class and maybe thats all one needs. The concept of instantiating self in static method, is/may be a bit funky. Creating seperate class and using in another class's static capabilities easier to understand? maybe?

Oh well, pidgeons rule!

[DJDaveMark]

Nice!

There's never only the one way & variety is the spice of life. You just need to choose your flavour of the day!

[zskillz]

thanks guys!!!... those are both very good approaches, and I have relly enjoyed you 'debate' (of sorts) on the topic.

Do you know if there is a way for the JVM to give 'statistics' about how much memory it is/has used for a certain instance of a class or anything like that??

-Z

[qball]

First, this may be of interest...

http://java.sun.com/docs/books/tutor...m/garbage.html

Quote:

Do you know if there is a way for the JVM to give 'statistics' about how much memory it is/has used for a certain instance of a class or anything like that??

I don't thinks so, or haven't found. The reason is simple, the jvm is written for a specific platform, they all different. The jvm works within the given system, so the system will know. I don't know a jvm to system API that exists, though if it does, please share.

What you can do, and rather easily, is have a given system tell you about the jvm's that are currently in use. winders, task mangler. Did jvm optimization in an AS400 environment and used the existing 'process manager' (whatever it is called), to find leaks and issues.

[zskillz]

ah yes.. a very nice link... making my way through it now!
-Z

[zskillz]

So I've been poking around a bit more with this array/scope/static business and have come across some very strange things to me. I can't really make sense of what's going on. i'm going to go ahead and post 2 of the classes because it'll just be easier that way.

I'm very confused about a couple of things, but i'll start out w/ this. Ok here's the deal....

I tried making an object of my main class and then populating the arrays that way, but then I started thinking about it, and it turns out, that I'm not even creating a brand new array when I execute the line:

position = new moldynPos(z.pos_array);

it turns out that I can from the static main method change a value in the pos_array, and that change is reflected both in the moldynPos.position object as well as in pos_array. How can this be??? It seems like in my main class, pos_array is a static thing, but in my moldynPos class, it should be something that cannot be changed in any way from the main method.

Now i know that clearly I just have 2 references to the same array, but I still don't understand how one seems static, and the other seems dynamic (part of an object). Furthermore, I shouldn't be able to change anything in the moldynPos object once I instantiate it. That is how I want it to be (and thought that is considered the proper way to treat OOP stuff).

anyway, there a plenty of comments to help you understand stuff, but most of it has nothing to do with the issue at hand. lots of lines in the main method have been '//' commented out b/c this program takes a long time to run if given a large data set, so I just work on one part at a time. If you want the data sets, then I can probably work on uploading them somewhere, although I really think that I'm having a fundamental (but probably fairly subtle/simple) misunderstanding here that one of you can probably explain.

Thanks for anything you got!

-Z

/*
		ZskillZ	03/13/03
		
			This program is written to fulfill Statistical Mechanics Final Project assignment.
		All	of the important literature and related files are:
		
			traj.c
			traj.f
			traj.in
			traj.new
			traj.old
			projectB-444.pdf
		
			The two required command line arguments are "traj.in" and "traj.new".  "traj.new" is
		a large text file listing all of the x,y,z components of both position and velocity.
		This program will compute all of the requirements listed in "projectB-444.pdf" (pg.1)
		using methods contained in the "moldynPos.java" and "moldynVel.java" classes
		
*/

import java.io.*;
import java.util.*; //needed for StringTokenizer,StreamTokenizer

public class moldyn //moldyn = Molecular Dynamics
{	
	//Boltzman's constant = 1.3807E-23 J/K
	public static final double kb = 1.3807E-23;
	
	//parameters read in from traj.in
	public static int n, sx, sy, sz, ntime, nsnap, tterm, cf;
	public static double T_0;
	
	//arrays of position and velocity data
	private double[][] pos_array;
	private double[][] vel_array;
	
	
	public static void main (String args[]) throws IOException
	{
		//position and velocity class objects
		moldynPos position;
		moldynVel velocity;
		
		//buffered reader class to read in file
		BufferedReader in_traj_in, in_traj_new;
			
		//a try,catch statement is implemented to detect a proper command line argument
		try 
		{
			//instantiating the BufferedReader object based on the command line argument
			in_traj_in = new BufferedReader(new InputStreamReader(new FileInputStream(args[0])));
			in_traj_new  = new BufferedReader(new InputStreamReader(new FileInputStream(args[1])));		
		} catch (Exception e)
			{
				System.out.println("\nIncorrect command line argument specified.  Please try again\n");
				System.out.println(e);
				return;
			}//end try,catch
		
		//read traj.in file
		getParams(in_traj_in);

		moldyn z = new moldyn();
		//create position and velocity arrays (This takes noticible time depending on array size)
		z.createPosVel(in_traj_new);
		
		//create moldynPos and moldynArray objects to perform calculations on
		//at some point I might want to pass the getParams vector to these constructors
		position = new moldynPos(z.pos_array);
		velocity = new moldynVel(z.vel_array);
		
//		velocity.getTemp(); // (2a)
		//A time value of -1 computes over all time configurations
//		velocity.getVelProbDist(-1,0,7, 0.1); //(2b)
		//position is found for 75 different timesteps for a cleaner graph
		//Since this is equilibrium data, any 75 time configs. work, so 0-74 are chosen arbitrarily
//		position.getPairDistFxn(75,0,9,0.01); //(3)
//		position.getMeanSqrDisp();//(4)
		
//		velocity.getVelAutoCorr();//(5 + 6)

									
	}//end main
	
	
	
	public static void getParams(BufferedReader in_traj_in) throws IOException
	{
		/*	this method will get all of the parameters as specified in the 
				traj.in file.  Since it has a specific format, the exact position
				of 'n' (# of particles) and 'nsnap' (# of time iterations) is known
				and used to select those specific values from the vector.
		*/
		
		StreamTokenizer smt = new StreamTokenizer(in_traj_in);
		Vector traj_in_params = new Vector();
		
		//get the first token
		smt.nextToken();
		
		//insert all traj.in parameters (that are numbers) into a vector
		while (smt.lineno()<= 9) // no parameters contained after line #9 in traj.in
		{
			if (smt.ttype == StreamTokenizer.TT_NUMBER)
				traj_in_params.add(new Double(smt.nval));
			
			smt.nextToken();
		}//end while
		
		//set parameters based on their fixed position in the traj.in file
		n 		= ((Double)traj_in_params.elementAt(2)).intValue();
		sx 		= ((Double)traj_in_params.elementAt(3)).intValue();
		sy 		= ((Double)traj_in_params.elementAt(4)).intValue();
		sz 		= ((Double)traj_in_params.elementAt(5)).intValue();
		ntime	= ((Double)traj_in_params.elementAt(6)).intValue();		
		nsnap	= ((Double)traj_in_params.elementAt(7)).intValue();
		tterm	= ((Double)traj_in_params.elementAt(9)).intValue();
		cf		= ((Double)traj_in_params.elementAt(10)).intValue();
		T_0		= ((Double)traj_in_params.elementAt(16)).doubleValue();
						
	}//end method getParams
	
	
	
	public void createPosVel(BufferedReader in_traj_new) throws IOException
	{
		/*	This method breaks the traj.new file into two separate arrays containing
				position and velocity data.  The format of the traj.new file dictates
				that odd line is position values and every even line is velocity values.
		*/
		
		StringTokenizer sgt;
		pos_array = new double[n*nsnap][3];//2D array to hold all values from traj_new
		vel_array = new double[n*nsnap][3];
		
		//this is the for loop that separates the data
		//the tokens must be converted from strings to double values
		for (int i=0; in_traj_new.ready(); i++)
		{
			//position data
			sgt = new StringTokenizer(in_traj_new.readLine());
			pos_array[i][0] = (new Double(sgt.nextToken())).doubleValue();
			pos_array[i][1] = (new Double(sgt.nextToken())).doubleValue();
			pos_array[i][2] = (new Double(sgt.nextToken())).doubleValue();
				
			//velocity data	
			sgt = new StringTokenizer(in_traj_new.readLine());
			vel_array[i][0] = (new Double(sgt.nextToken())).doubleValue();
			vel_array[i][1] = (new Double(sgt.nextToken())).doubleValue();
			vel_array[i][2] = (new Double(sgt.nextToken())).doubleValue();		
		}
//these may be the needed lines to disable referencing from the static context		
//		pos_array = null;
//		vel_array = null;
				
	}//end method createPosVel
	
	
	
	public static void createFile(double[][] array, int time, String file_name) throws IOException
	{
		//simple method to create a file with probability distrobution for a given time configuration
		//or all time if time=-1
		
		StringBuffer sb = new StringBuffer();
		
		int end;
		String filename = file_name;
		if (time == -1)
			end = moldyn.nsnap;
		else
			end = 2;
		
		for (int i=0; i<array.length; i++)
		{
			for (int j=0; j<end; j++)
				sb.append(array[i][j] + ",");
			sb.append("\n");
		}//end for			
		
		//this is a neat little trick to tell if a file exists!.
		if (!new File("D:\\"+filename+".csv").exists())
		{
			//create file in directory
			PrintWriter pw = new PrintWriter(new BufferedWriter(new FileWriter("D:\\"+filename+".csv",true)));
			pw.write(sb.toString());
			
			//this is necessary
			pw.close();	
		} else 
				System.out.println("FILE EXISTS!!! NEW FILE NOT CREATED!!!");
			
	}//end method createFile
	
	
	
	public static void createFile(StringBuffer file, String name) throws IOException
	//this is method overloading of the createFile method
	{
		//this is a neat little trick to tell if a file exists!.
		if (!new File("D:\\"+name+".csv").exists())
		{
			//create file in directory
			PrintWriter pw = new PrintWriter(new BufferedWriter(new FileWriter("D:\\"+name+".csv",true)));
			pw.write(file.toString());
			
			//this is necessary
			pw.close();	
		} else 
				System.out.println("FILE EXISTS!!!");
			
	}//end method createFile


}//end class moldyn

object class

/*
		ZskillZ	03/13/03
		
			This class is accessed by the main method of class moldyn.  
		It will contain all of the position coordinates contained in the
		traj.in file. All of the methods included are used to calculate
		what is asked for in the assignment		
*/

import java.io.*;

public class moldynPos //moldynPos = Molecular Dynamics Positions
{
	//Posocity array to perform calculations on
	private double[][] coord_array;// = new double[moldyn.n*moldyn.nsnap][2];
		
	
	public moldynPos(double[][] incmng_coord_array)
	{
		coord_array = incmng_coord_array;
	
	}//end Constructor moldynArrayd
		
	
	public void getPairDistFxn
		(int time, int min_size, int max_size, double bucket_size) throws IOException
	//get Pair Distrobution Function
	//IOException needed for moldyn.createFile() call at end of method
	{
		/*	In a homogeneous and isotropic system, the Pair Correlation fxn depends only on
				|r0-rx|=r so that g(r) can be calculated quite simply by using the distance 
				formula:
					d=SQRT((x0-x1)^2+(y0-y1)^2+(z0-z1)^2)
				Then bin sorting and multiplying by the volume element for a spherical shell.  
				Finally, one must normalize the system.
								
				This method then sends an array to the moldyn.createFile method that contains g(r)
				as a function of distance.  The reduced unit g(r*): r* = r/SIGMA;				
		*/
		
		//this is a bith sketchy as there might be some precision loss here
		//After consideration and testing, I think it's fine, but I'm not positive
		int size = new Double(Math.ceil((max_size-min_size)/bucket_size)).intValue();
		double[][] bucket_array = new double[size][2];
		int start_pos;
		double distance;
		
		//to make the bucket_array contain the bucket values
		for (int i=0; i<bucket_array.length; i++)
			bucket_array[i][0]= (i+1)*bucket_size;
		
		//bucket_sort the distance over total time frame
		for (int t=0; t<time; t++)
		{
			start_pos = t*moldyn.n;
			for (int i=start_pos; i<(start_pos+moldyn.n); i++)
			{
				for (int j=i+1; j<(start_pos+moldyn.n); j++)
				{
					distance = Math.sqrt(
						Math.pow((coord_array[i][0]-coord_array[j][0]),2.0)
						+Math.pow((coord_array[i][1]-coord_array[j][1]),2.0)
						+Math.pow((coord_array[i][2]-coord_array[j][2]),2.0)
							);
					
					int bin = new Double(Math.ceil(distance/bucket_size)).intValue();
					//this little correction may truncate some extremely far outlying data
					//points into the last bin
					if (bin > (size-1))
						bin=size-1;
					
					bucket_array[bin][1] += 1;
				}//end 'j' for loop
			}//end 'i' for loop
		}//end 't' for loop
		
		normalizePairDistFxn(bucket_array, time, bucket_size);

	}//end method getPairDistFxn()
	
	private void normalizePairDistFxn(double[][] bucket_array,int time, double bucket_size) throws IOException
	{

		for (int i=0; i<bucket_array.length; i++)
		{
			double g_r = (moldyn.sx*moldyn.sy*moldyn.sz)/(moldyn.n)*bucket_array[i][1]
				/((moldyn.n/2)*time*4*Math.PI*bucket_array[i][0]*bucket_array[i][0]*bucket_size);
//********************************************************
//I think that the problem in the above calculation may be with the "time" value being incorrect
			
			//overwrites the values in the un-normalized bucket array w/ the normalized values
			bucket_array[i][1] = g_r;
		}//end for
		
		moldyn.createFile(bucket_array, 83, "(3) Pair Distribution");
	}
	
	
	public void getMeanSqrDisp() throws IOException
	{
		/*	This method is aptly named.  It computes the mean square displacement using
				the formula:
					<(r-r0)^2> = 1/N * SUM[i=0->N] (ri(t)-ri(0))^2
				The mean sqr disp. is then calculated for each time configuration in the system.		
		
		*/
		StringBuffer m_s_d = new StringBuffer("");
		double[] mean_sqr_disp_array = new double[moldyn.nsnap];

		for (int i=1; i<moldyn.nsnap; i++)
		{
			double distance = 0.0;

			for (int j=0; j<moldyn.n; j++)
			{
				distance += 
					Math.pow((coord_array[j+(moldyn.n*i)][0]-coord_array[j][0]),2.0)
					+Math.pow((coord_array[j+(moldyn.n*i)][1]-coord_array[j][1]),2.0)
					+Math.pow((coord_array[j+(moldyn.n*i)][2]-coord_array[j][2]),2.0);
			}//end 'j' for loop
			
			m_s_d.append(distance/moldyn.n + "\n");
				
		}//end 'i' for loop
		
		moldyn.createFile(m_s_d, "(4) MeanSqrDisp");
		
	}//end method getMeanSqrDisp

}//end class modlyPos

[qball]

Quote:

it turns out that I can from the static main method change a value in the pos_array, and that change is reflected both in the moldynPos.position object as well as in pos_array. How can this be??? It seems like in my main class, pos_array is a static thing, but in my moldynPos class, it should be something that cannot be changed in any way from the main method.

Now i know that clearly I just have 2 references to the same array, but I still don't understand how one seems static, and the other seems dynamic (part of an object). Furthermore, I shouldn't be able to change anything in the moldynPos object once I instantiate it. That is how I want it to be (and thought that is considered the proper way to treat OOP stuff).

How can it be? Well java passes objects by reference. Therefore, if you use any reference to object to change object, all references will see that change. This is fundamental to how java works, so learn well. Don't bother trying to pass another reference to same object, same thing will occur. You can get around this by cloning the original object and passing that to method/object, but then there is a resource/memory issue with that, but, will preserve state of original object to be passed. This, actually is a good thing. If this helps, regardless of 'type' of object (STATIC, not), at runtime, really just an object. STATIC refers to how object is accessed, not how it is handled.

Furthermore, how you design your objects will determine how those objects methods and members are handled. If you want to create an object and make it unchangeable, you can. Now, if you pass an object reference to moldynPos constructor, the original reference can be manipulated, so bets are off. You need to keep all of moldynPos 'stuff', private to moldynPos. If no way around passing an existing object to given class, yet needs to be 'safe', you can do the cloning thing on passed reference and use that.

Hope this helps???