help deleting lines

[jon112981]

I need some help processing a rather large file I have. The file contains some data in a nice matrix form (40x13 all numbers) and some ugly headerlines with a random assortment of strings and numbers. It follows the pattern 6 lines of header, 40 lines of data about 2100 times. What I am interested in doing is finding some program, in anything, to delete the headerlines and leave the 40 lines of data. It can be a unix script, a dos script or in any language at all. The end result will go into MATLAB. I know next to nothing about UNIX but was looking around at some of the commands, that might be the way to go. If anyone could help start me out on a solution I'd greatly appreciate it.

[originel]

hmm...well with something like this i don't know of anything that would do it for you. you could write your own shell script in 'nix to do it (since it can do some c stuff). but personally i would recommend just writing an actual program to do that. it would be pretty easy if you're familiar with programming.

[Emc2]

mmmmm.....Matlab.....I love Matlab for lunch....it just requires late late nights of preparation...

Look around in Matlab...I know there's a way to input data from a non-Matlab file into Matlab, but I don't remember how.

Then you could just use for loops to manipulate in a various amount of ways:

head=0;
for n=[0:2100] % 2100=# of instances this pattern occurs
for i=[1:40]; %row i of occurance n of the pattern
for j=[1:13]; %column
newMatrix[i+40*n,j] = oldMatrixFromFile[i+40*n+head*n,j];
end
end
head = head+6; %6 is the number of header lines b/t data
end

I wrote this mini code, just to give an initial idea. I don't think it will directly work. This also assumes that you can get the entire data file into Matlab into matrix form.

Hope this gives you some direction...I'll soon be doing some research in which I'll have to manipulate large amounts of data as well

[jon112981]

I'm not incredibly familiar with programing syntax for any acutal language. I've had theory (lots of it) but we only used MATLAB (I still have to take the basic compsci classes before they let me graduate). MATLAB's main limitaitions comes in that the data file must all be formatted the same for it to read it. So the varying number of columns doens't work at all. Would something like C/C++ or Java be capable of doing this?

[jon112981]

Yeah EMC that's origionally what I wanted to do but loading the matrix is a pain in the @$$ in matlab. Okay the first line has 13 colums, the next line has 9 columns, the third has 3, the 4th has 5 and so on. The 6th has 2 strings in it, then I hit the nice 40 lines and start over again. I know textread won't load this as all rows must have the same number of columns. I totally don't get fscanf, someone here said that should work but I don't understand how or what to do with it.

[originel]

yeah, c/c++ and java can do this no problem.

in c++ it would look something kinda like this:

int c, line = 0;
ifstream infile;
ofstream outfile;
string mystring;
string mydata[<number of lines of data>];
infile.open("<nameoffile>");
while(infile)
{
//this takes care of the 6 lines of header
     for(c = 0;c < 6;c++)
           infile.getline(mystring);
//this takes care of the 40 lines of data
     for(c = 0; c < 40;c++)
     {
           infile.getline(mystring);
           mydata[line] = mystring;
           line++;
     }
}
infile.close();
outfile.open("<output file name>");
for(c = 0; c < number of lines of data; c++)
     outfile.write(mydata[line]);
outfile.close();

note: if there are blank lines separating headers or lines of code and the like you will need to add an extra getline for each. also i used the fstream library (part of the standard library), but you can use whatever you want to use.

EDIT: also i don't think this will work right off, it's been a while since i've used the standard library and i didn't compile this. and i think that getline and write take another parameter, just can't remember what it is

[jon112981]

I got it! thanks all you guys for your help. If anyone is looking for a way to do this in matlab there is a command similar to getline(in c++) called fgetl. It reads one line at a time as a string.