File list completeness check

[ArielRauch]

I hope this title was confusing enough to read this post:)

I am looking for a solution based on Switch WITHOUT the scripting module:

A hot folder receives a number of pdf files. The file names are structured in a way that one can know how many files belong to this series and which number the specific file has - something like xxx_6_of_50.pdf. (Alternatively, an XML file can be pre-delivered which contains summary information).
The flow now shall wait a predefined time until all files of this series have arrived. If - after the timeout - files are missing the flow shall respond with a list of missing files. When those files are re-delivered the flow shall recognize the completeness of the series and goon with other stuff.

Any ideas?

Important to note:
The number of files per series may vary.
There may arrive several series in parallel
The necessity of responding with only the missing files is crucial as the "network" to the flow is very problematic and a full resend will probably fail again.

[gabrielp]

You can assemble based on the total number of expected files and you can have that assemble fail out if the total number is not found within a certain amount of time, but I don't think you can report on which ones are missing within the flow without a script unless you build in a segment to check for that number, which wouldn't work for you. So I think you can get 80% there.

[ArielRauch]

Thank you, Gabriel - that is what I thought.
What do you mean by segment?

[gabrielp]

Thank you, Gabriel - that is what I thought.
What do you mean by segment?

Well... you could check to see if a file is missing by having anything failing from the assemble to assemble again into whatever files you have. Then, you could take that assembled job and send it into an Inject. That Inject would pull out N number of "tokens" (blank files or whatever) for how many the job is supposed to have (let's say, 50). You could then send those tokens into another Inject that would attempt to Inject the actual assembled job. If an Inject fails, then it is missing from the files assembled and you can attempt a redelivery. If it Injects successfully, it exists and there isn't a problem.

Seems like a huge pain in the butt and Injecting against a job in Switch seems dangerous. I suppose you could archive that job, then attempt the Inject. In any case, seems like it's not worth your time but I think that would be possible.

[ArielRauch]

I love your thinking!

I think I might be able to write a small shell script (Python,php,Perl) which checks for the missing files.

Assemble Job is able to wait for a number of files while this number is provided by the first file?

[Terkelsen]

If your jobs are named as mentioned in your first post "xxx_6_of_50.pdf", you should be able to assemble the jobs using a Custom scheme where
Job identifier = [Job.NameProper:Before="_"]
Number of files = [Job.NameProper:After="_"]

[gabrielp]

I love your thinking!

I think I might be able to write a small shell script (Python,php,Perl) which checks for the missing files.

Assemble Job is able to wait for a number of files while this number is provided by the first file?

They'd all need the number of files expected. You would tell the Assemble configurator to wait for N number of files that have an identifier of X. If you don't know the number of files, then you can use a timeout to just use what you have after Y minutes. I think the shell/python script sounds like a good idea. You could assemble what you have and make a script that loops through with a counter and tells you if it's missing anything!

[ArielRauch]

Thanks, guys.
I have made some progress but stuck in one point (so far):
1. I am sending the files to the second flow and in parallel to a not-auto-managed temp directory.
2. The second flow uses the assemble job to find out whether all jobs arrived.
3. After a timeout - in case not all files were received, all the received ones (error connection) are joined with a dummy xml.
4. Export Metadata is now separating the metadata (which is only the original filename with a small dummy cml content) and the metadata files with the original filename are sent back to the first flow, while the original files are redirected to the assemble job waiting for the missing files.
5. With the help of the xml files I remove all the already transferred files from the temp-directory (see 1).

6. THE PROBLEM: I now would like to resend the remaining files from the temp directory again to the second flow. Unfortunately I have a problem to inject a group of files

If someone understood me I would be happy for get some ideas

Ariel

[Terkelsen]

If you make sure that the temp directory is a specific folder for each job, you can use Inject Job to inject an entire folder and then dismantle this folder before passing the files on into the other flow. The name of the folder to be injected can be defined by a variable.

[ArielRauch]

Hi Terkelsen,

just found it out:)

I have now two flows: one sending files and verifying that all files arrived the other side, and the second receiving the files and sending ack back.
All without scripting.

Thank you all for your help!!!

Ariel