sendToSingle() doesn't send until script finishes

[ericnd01]

I have a script the reads a text file of file paths and processes them in a loop. There could be a thousand or more items in the text file. It appears that the sendToSingle() command doesn't "release" the jobs until the script finishes. This is an issue because there could be hundreds of gigabytes of files that are suddenly copied to the Switch server for processing, then the system drive fills up and crashes. Is there a way to have the script send the jobs along WHILE the script is running? I know I could make a single text file for each file to process, but like I said before, there could be a thousand or thousands of files... Thanks!

Below is the script:

Code: Select all

function jobArrived( s : Switch, job : Job ) {

	var debugLevel = s.getPropertyValue( "debugLevel" );

	// create the file handler
	var fileList = new File(job.getPath());

	// open the file for reading
	fileList.open(File.ReadOnly);

	// read each line into an array
	var lines = fileList.readLines();
	
	var theSourcePath;
	
	for(i = 0; i < lines.length; i++) {
		
		sourcePath = lines[i];
		
		// don't process blank lines
		if (sourcePath == '') {
			continue;
		}
		
		// create a file handle for the file to be moved
		sourceFile = new File(sourcePath);
		
		if (!File.exists(sourcePath)) {
			s.log(3, sourcePath + " does not exist");
			continue;
		}
			
		
		// get the name of the file (without the path information)
		fileName = sourceFile.name;
		
		// create a new job using the name of the file to copy
		newJob = s.createNewJob(fileName);
		jobPath = newJob.createPathWithName(fileName, true);
		
		// copy the file to the new job.
		s.copy( sourcePath, jobPath );
		if(debugLevel == 1){
			s.log(2, "Copy " + sourcePath + " to job");
		}
		
		// trim hierarchy segments
		var jobPathArray = sourcePath.split("/");
		jobPathArray.splice(0, s.getPropertyValue("Trim_Hierarchy"));
		jobPathArray.shift();
		jobPathArray.pop();
		
		// set hierarchy information
		newJob.setHierarchyPath(jobPathArray);
		
		// send the file along to a single connection
		newJob.sendToSingle( jobPath );
		if(debugLevel == 1){
			s.log(2, "Send to single: " + jobPath);
		}		
	}
	if(debugLevel == 1){
		s.log(2, "Done processing");
	}
	// cleanup
	job.sendToNull( job.getPath() );
	
	fileList.close();
}

[jan_suhr]

That's the way Switch works, it only uses the outgoing connection when the script is done with all processing. Otherwise it wouldn't be sure that everything is done in the process of the job.

You have to make another script that runs before this one that splits the large file into one file per line of text, then that text file is processed and outputs your file.

[ericnd01]

I'm splitting the file now via commandline (25 line chunks), but I'll look into breaking it up via Switch.

Thanks!

[cstevens]

I believe you can also use s.copy() or s.move() to move files in the middle of the script, but that takes away a lot of Switch's built-in file handling which can be problematic.