Inconsistent workflow runs between arvados-cwl-runner and cwltool


I have run into a specific problem regarding the execution of CWL workflows on Arvados. I am currently using arvbox for local testing and trying to run workflows with basecommand being either java or python. I have followed the guide to set up a docker image based on the arvados/jobs image, extended this image by installing openjdk-11 and commited it. Then I go to my workflow and add the hint at the end to pull the image and use it.

What is weird is that cwltool would find java (always with the docker requirement enabled in the hints), but arvados-cwl-runner does not. I stripped down my CWL’s to only run java -version, in which case I still get the message that java was not found in the $PATH (same even if I use /usr/bin/java as base command). Note that of course I can see java through the docker shell.

Then I took it one step further and added some simple files to the docker instances filesystem at /usr/bin. I created a workflow running ls -la /usr/bin and as output (in a file correctly uploaded to keep) I got every file in that directory, except for the files I added (oh, and java!). I tried making them executable, giving them 777 permissions, changing the owner from crunch to root (only necessary when I would use docker cp), but nothing changed.

Finally, I have installed tree through apt, which was too installed to /usr/bin and noticed that I can strangely use tree through arvados-cwl-runner and I can also see it using the aforementioned ls method.

Is this expected behavior? Am I missing something concerning how arvados handles these specific images ? Thanks in advance.

It sounds like you have the correct image locally but it wasn’t uploaded to Arvados. How did you tag the image? If you need to upload a new one manually, try something like arv-keepdocker myimage:latest.

This is the most likely thing to have happened (and such a stupid mistake!). I suppose I was uploading whatever I needed to the running container, but it wasn’t commited to the image. I was misguided by the sha:xxxxx string returned by the arvados-cwl-runner, stating that it has connected to the image, because this number matched what I was seeing in docker images.
Thanks for the help!