Hi , I have some outputs that are intermediary steps in a process and aren’t sent to a collection, how do I download those for multiple samples, without having to go to the page and downloading them per sample?
Hi Meghana,
Let me see if I understand you correctly. It sounds like you ran a workflow and some of the steps produced intermediate results which were not propagated to the final output, but going to each intermediate collection to download the file is tedious. Is this right?
The easiest way to do this would be to modify the workflow so that those files are interested in are propagated to the final output, and then re-run the workflow. If you submit the workflow with exactly the same input as before, it should re-use the results from the previous execution and complete the workflow without having to wait for anything to be reprocessed.
A more complex solution would be to write a Python script to go through the workflow steps, find the intermediate output collections, and do something with them.
Finally, if you need to send the files somewhere else, you could modify the workflow to add a step which uploads your files to external storage, e.g. an S3 bucket.
Thanks,
Peter