Invalid field `acrContainerImage` error when running a workflow

How-to
#1

I’m trying to run a workflow, but get the following error in the process logs:

2020-10-29T19:50:45.224135600Z e[1;30mINFOe[0m /usr/bin/arvados-cwl-runner 2.1.0.dev20200814195416, arvados-python-client 2.1.0.dev20200814195416, cwltool 3.0.20200807132242
2020-10-29T19:50:45.230296484Z e[1;30mINFOe[0m Resolved '/var/lib/cwl/workflow.json#main' to 'file:///var/lib/cwl/workflow.json#main'
2020-10-29T19:50:46.113388394Z e[1;30mERRORe[0m e[31mTool definition failed validation:
2020-10-29T19:50:46.113388394Z ../../lib/cwl/workflow.json:1:111: invalid field `acrContainerImage`, expected one of: 'class',
2020-10-29T19:50:46.113388394Z                                    'coresMin', 'keep_cache', 'ramMin'e[0m
2020-10-29T19:50:46.845399277Z Container exited with code: 1

Screenshot:

acrContainer_error
acrContainer_error2576×1769 218 KB

Here is the workflow definition:

#!/usr/bin/env cwl-runner
# Copyright (C) The Arvados Authors. All rights reserved.
#
# SPDX-License-Identifier: CC-BY-SA-3.0

cwlVersion: v1.0
class: CommandLineTool

hints:
  DockerRequirement:
    dockerPull: quay.io/biocontainers/bwa:0.7.17--ha92aebf_3

baseCommand: [bwa, mem]

arguments:
  - {prefix: "-t", valueFrom: $(runtime.cores)}
  - {prefix: "-R", valueFrom: '@RG\\\tID:$(inputs.group_id)\\\tPL:$(inputs.PL)\\\tSM:$(inputs.sample_id)'}

inputs:
  reference:
    type: File
    inputBinding:
      position: 1
      valueFrom: $(self.dirname)/$(self.nameroot)
    secondaryFiles:
      - ^.ann
      - ^.amb
      - ^.pac
      - ^.sa
    doc: The index files produced by `bwa index`
  read_p1:
    type: File
    inputBinding:
      position: 2
    doc: The reads, in fastq format.
  read_p2:
    type: File?
    inputBinding:
      position: 3
    doc:  For mate paired reads, the second file (optional).
  group_id: string
  sample_id: string
  PL: string

stdout: $(inputs.read_p1.nameroot).sam

outputs:
  aligned_sam:
    type: stdout

Here is the command:

[
  "arvados-cwl-runner",
  "--local",
  "--api=containers",
  "--project-uuid=3rzp3-tpzed-p1y40tr27brcm2r",
  "--collection-cache-size=256",
  "/var/lib/cwl/workflow.json#main",
  "/var/lib/cwl/cwl.input.json"
]

And here is the cwl.input.json:

{
  "PL": "illumina",
  "group_id": "illumina",
  "read_p1": {
    "class": "File",
    "http://arvados.org/cwl#collectionUUID": "Copy of HWI-ST1027_129_D0THKACXX for CWL tutorial",
    "location": "keep:ae480c5099b81e17267b7445e35b4bc7+180/HWI-ST1027_129_D0THKACXX.1_1.fastq"
  },
  "read_p2": {
    "class": "File",
    "http://arvados.org/cwl#collectionUUID": "Copy of HWI-ST1027_129_D0THKACXX for CWL tutorial",
    "location": "keep:ae480c5099b81e17267b7445e35b4bc7+180/HWI-ST1027_129_D0THKACXX.1_2.fastq"
  },
  "reference": {
    "class": "File",
    "http://arvados.org/cwl#collectionUUID": "Copy of chromosome 19 ref",
    "location": "keep:29eb6121bd9fcba8ecfd05ce176718bd+510/19.fasta.bwt"
  },
  "sample_id": "HWI-ST1027_129_D0THKACXX"
}

Here are the versions of Arvados I’m using (I had to downgrade to 2.0.1 since the newer versions were not in the APT repository).

arvados:
  versions:
    distribution:
      arvadosApiServer: 2.0.1-1
      arvadosController: 2.0.1-1
      arvadosGitHttpd: 2.0.1-1
      arvadosHealth: 2.0.1-1
      arvadosWorkbench: 2.0.1-1
      arvadosWorkbench2: 2.0.1-1
      arvadosWs: 2.0.1-1
      arvadosSSOServer: 1.1.4-1
      crunchDispatchSlurm: 2.0.1-1
      crunchRun: 2.0.1-1
      crunchRunner: 2.0.1-1
      keepBalance: 2.0.1-1
      keepStore: 2.0.1-1
      keepWeb: 2.0.1-1
      keepProxy: 2.1.0.dev20200422013447-1
      libPamArvados: 2.0.1-1
      pythonArvadosFuse: 2.0.1-1
      pythonArvadosPythonClient: 2.0.1-1
    gem:
      arvados: 2.0.1
      arvadosCLI: 2.0.1
      arvadosLoginSync: 2.0.1

The process error was while using the arvados/jobs:latest (5ad1019f39) image, but it also fails when using tag 2.0.1. This issue is what lead me to believe it might be the jobs image: https://dev.arvados.org/issues/16602.

2020-10-29T21:57:04.974715586Z e[1;30mINFOe[0m /usr/bin/arvados-cwl-runner 2.0.1, arvados-python-client 2.0.1, cwltool 1.0.20190831161204
2020-10-29T21:57:04.981440474Z e[1;30mINFOe[0m Resolved '/var/lib/cwl/workflow.json#main' to 'file:///var/lib/cwl/workflow.json#main'
2020-10-29T21:57:06.324211971Z e[1;30mERRORe[0m e[31mTool definition failed validation:
2020-10-29T21:57:06.324211971Z ../../lib/cwl/workflow.json:1:111: invalid field `acrContainerImage`, expected one of: 'class', 'coresMin', 'keep_cache', 'ramMin'e[0m
2020-10-29T21:57:02.135009111Z Starting Docker container id '97e8bafeff859ab54f06f2f239d4612e473b5e77a88b63364a37b5280cc9d99f'
2020-10-29T21:57:03.014404172Z Waiting for container to finish
2020-10-29T21:57:06.974117621Z Container exited with code: 1

I also tried running the process using both Workbench v1 and v2, but neither worked. By best guess is some sort of version mismatch, but I’m not sure where.

#2

Hi @mluds, the problem is likely caused by a mismatch in the version of arvados-cwl-runner (looks like you used 2.1) and Arvados itself (version 2.0.1).

The quick fix would be to use arvados-cwl-runner 2.0.1 to submit the workflow, that should fix the problem.

I have a question - you mentioned that you couldn’t use 2.1 because the packages are not available in the apt repo. I just checked, our repos seem OK (with 2.1 available). Which distribution and which version of it are you using? It is true that the k8s helm charts have not been updated yet to 2.1, I hope to do that soon.

#3

Thanks for the quick response. Using arvados-cwl-runner 2.0.1 to submit the workflow did indeed work.

For the versions, I tried using the default helm chart value of 2.0.2, but it failed with the error below (for the workbench pod). I downgraded to 2.0.1 (the latest version in the apt cache) to fix this. I originally thought this was caused by the version not existing in the remote apt repo, but I think it’s something else in the bootstrap.sh script.

E: Version '2.0.2-1' for 'arvados-workbench' was not found
E: Version '2.0.2-1' for 'arvados-workbench2' was not found
Get:1 http://apt.arvados.org bionic InRelease [3,242 B]
Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease
Get:3 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:5 http://apt.arvados.org bionic/main amd64 Packages [117 kB]
Ign:6 https://oss-binaries.phusionpassenger.com/apt/passenger bionic InRelease
Get:7 https://oss-binaries.phusionpassenger.com/apt/passenger bionic Release [8,201 B]
Get:8 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:9 https://oss-binaries.phusionpassenger.com/apt/passenger bionic Release.gpg [833 B]
Get:10 http://security.ubuntu.com/ubuntu bionic-security/universe Sources [334 kB]
Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Sources [11.6 kB]
Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/main Sources [622 kB]
Get:13 http://security.ubuntu.com/ubuntu bionic-security/restricted Sources [13.0 kB]
Get:14 http://security.ubuntu.com/ubuntu bionic-security/multiverse Sources [4,354 B]
Get:15 http://security.ubuntu.com/ubuntu bionic-security/main Sources [298 kB]
Get:16 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,353 kB]
Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Sources [18.4 kB]
Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/universe Sources [551 kB]
Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [239 kB]
Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [45.9 kB]
Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2,118 kB]
Get:22 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [213 kB]
Get:23 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [15.4 kB]
Get:24 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,748 kB]
Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [2,165 kB]
Get:26 http://archive.ubuntu.com/ubuntu bionic-backports/main Sources [5,823 B]
Get:27 http://archive.ubuntu.com/ubuntu bionic-backports/universe Sources [5,759 B]
Get:28 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [11.4 kB]
Get:29 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [11.3 kB]
Reading package lists...
E: Repository 'https://oss-binaries.phusionpassenger.com/apt/passenger bionic Release' changed its 'Origin' value from 'packagecloud.io/phusion/passenger' to '. bionic'
E: Repository 'https://oss-binaries.phusionpassenger.com/apt/passenger bionic Release' changed its 'Label' value from 'passenger' to '. bionic'
E: Unable to correct problems, you have held broken packages.
 * Stopping nginx nginx
   ...done.
*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
*** Running /etc/my_init.d/10_syslog-ng.init...
Nov  2 15:56:13 arvados-workbench-fcd76d79c-lgq7p syslog-ng[381]: syslog-ng starting up; version='3.13.2'
*** Running /etc/my_init.d/30_presetup_nginx.sh...
*** Booting runit daemon...
*** Runit started as PID 391
Nov  2 15:56:14 arvados-workbench-fcd76d79c-lgq7p cron[397]: (CRON) INFO (pidfile fd = 3)
ok: run: /etc/service/nginx-log-forwarder: (pid 396) 0s
Nov  2 15:56:14 arvados-workbench-fcd76d79c-lgq7p cron[397]: (CRON) INFO (Running @reboot jobs)
[ N 2020-04-11 15:44:32.2055 14405/To Ser/Server.h:558 ]: [ApiServer] Shutdown finished
[ N 2020-04-11 15:44:32.8257 14405/T1 age/Cor/CoreMain.cpp:1325 ]: Passenger core shutdown finished
[ N 2020-04-11 15:44:33.2702 26347/T1 age/Wat/WatchdogMain.cpp:1373 ]: Starting Passenger watchdog...
[ N 2020-04-11 15:44:33.2845 26350/T1 age/Cor/CoreMain.cpp:1340 ]: Starting Passenger core...
[ N 2020-04-11 15:44:33.2847 26350/T1 age/Cor/CoreMain.cpp:256 ]: Passenger core running in multi-application mode.
[ N 2020-04-11 15:44:33.3236 26350/T1 age/Cor/CoreMain.cpp:1015 ]: Passenger core online, PID 26350
[ N 2020-11-02 15:56:14.5248 400/T1 age/Wat/WatchdogMain.cpp:1373 ]: Starting Passenger watchdog...
[ N 2020-11-02 15:56:14.5610 403/T1 age/Cor/CoreMain.cpp:1340 ]: Starting Passenger core...
[ N 2020-11-02 15:56:14.5613 403/T1 age/Cor/CoreMain.cpp:256 ]: Passenger core running in multi-application mode.
[ N 2020-11-02 15:56:14.5769 403/T1 age/Cor/CoreMain.cpp:1015 ]: Passenger core online, PID 403
[ N 2020-11-02 15:56:17.0735 403/T5 age/Cor/SecurityUpdateChecker.h:519 ]: Security update check: no update found (next check in 24 hours)
2020/11/02 16:01:18 [error] 421#421: *5 rewrite or internal redirection cycle while internally redirecting to "/index.html", client: 10.233.79.0, server: public-workbench2, request: "GET / HTTP/1.1", host: "10.8.47.220"
2020/11/02 16:01:18 [error] 421#421: *7 rewrite or internal redirection cycle while internally redirecting to "/index.html", client: 10.8.35.35, server: public-workbench2, request: "GET /favicon.ico HTTP/1.1", host: "10.8.47.220", referrer: "https://10.8.47.220/"
2020/11/02 16:01:33 [error] 421#421: *9 rewrite or internal redirection cycle while internally redirecting to "/index.html", client: 10.8.35.35, server: public-workbench2, request: "GET / HTTP/1.1", host: "10.8.47.220"
2020/11/02 16:01:33 [error] 421#421: *11 rewrite or internal redirection cycle while internally redirecting to "/index.html", client: 10.8.35.35, server: public-workbench2, request: "GET /favicon.ico HTTP/1.1", host: "10.8.47.220", referrer: "https://10.8.47.220/"

Looking at the bootstrap.sh script, it should try installing again after the apt update, but it doesn’t look like it gets installed:

root@arvados-workbench-fcd76d79c-lgq7p:/# apt list --installed | grep workbench

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

root@arvados-workbench-fcd76d79c-lgq7p:/# apt list -a arvados-workbench
Listing... Done
arvados-workbench/bionic 2.1.0-1 amd64 [residual-config]
arvados-workbench/bionic 2.0.4-1 amd64 [residual-config]
arvados-workbench/bionic 2.0.3-1 amd64 [residual-config]
arvados-workbench/bionic 2.0.2-1 amd64 [residual-config]
arvados-workbench/bionic,now 2.0.1-1 amd64 [residual-config]
arvados-workbench/bionic 2.0.0-1 amd64 [residual-config]
arvados-workbench/bionic 1.4.3-1 amd64 [residual-config]
arvados-workbench/bionic 1.4.2-1 amd64 [residual-config]
arvados-workbench/bionic 1.4.1-1 amd64 [residual-config]
arvados-workbench/bionic 1.4.0-1 amd64 [residual-config]
arvados-workbench/bionic 1.3.3-1 amd64 [residual-config]
arvados-workbench/bionic 1.3.2-1 amd64 [residual-config]
arvados-workbench/bionic 1.3.1-1 amd64 [residual-config]
arvados-workbench/bionic 1.3.0-1 amd64 [residual-config]
arvados-workbench/bionic 1.2.1-1 amd64 [residual-config]
arvados-workbench/bionic 1.2.0-1 amd64 [residual-config]

It does however work for the api-server pod, which is strange since I think they use the same bootstrap script.

root@arvados-api-server-d9b6ff6bb-7vfkr:/# apt list --installed | grep arvados

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

arvados-api-server/bionic,now 2.0.2-1 amd64 [installed,upgradable to: 2.1.0-1]
arvados-server/bionic,now 2.0.2-1 amd64 [installed,upgradable to: 2.1.0-1]
root@arvados-api-server-d9b6ff6bb-7vfkr:/# apt list -a arvados-server
Listing... Done
arvados-server/bionic 2.1.0-1 amd64 [upgradable from: 2.0.2-1]
arvados-server/bionic 2.0.4-1 amd64
arvados-server/bionic 2.0.3-1 amd64
arvados-server/bionic,now 2.0.2-1 amd64 [installed,upgradable to: 2.1.0-1]
arvados-server/bionic 2.0.1-1 amd64
arvados-server/bionic 2.0.0-1 amd64
arvados-server/bionic 1.4.3-1 amd64
arvados-server/bionic 1.4.2-1 amd64
arvados-server/bionic 1.4.1-1 amd64
arvados-server/bionic 1.4.0-1 amd64
arvados-server/bionic 1.3.3-1 amd64
arvados-server/bionic 1.3.2-1 amd64
arvados-server/bionic 1.3.1-1 amd64
arvados-server/bionic 1.3.0-1 amd64
arvados-server/bionic 1.2.1-1 amd64
arvados-server/bionic 1.2.0-1 amd64

E: Version '2.0.2-1' for 'arvados-server' was not found
E: Version '2.0.2-1' for 'arvados-api-server' was not found
Get:1 http://apt.arvados.org bionic InRelease [3,242 B]
Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease
Get:3 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:5 http://apt.arvados.org bionic/main amd64 Packages [117 kB]
Ign:6 https://oss-binaries.phusionpassenger.com/apt/passenger bionic InRelease
Get:7 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:8 https://oss-binaries.phusionpassenger.com/apt/passenger bionic Release [8,201 B]
Get:9 https://oss-binaries.phusionpassenger.com/apt/passenger bionic Release.gpg [833 B]
Get:10 http://security.ubuntu.com/ubuntu bionic-security/main Sources [298 kB]
Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/universe Sources [551 kB]
Get:12 http://security.ubuntu.com/ubuntu bionic-security/restricted Sources [13.0 kB]
Get:13 http://security.ubuntu.com/ubuntu bionic-security/multiverse Sources [4,354 B]
Get:14 http://security.ubuntu.com/ubuntu bionic-security/universe Sources [334 kB]
Get:15 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [213 kB]
Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Sources [11.6 kB]
Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/main Sources [622 kB]
Get:18 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,748 kB]
Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Sources [18.4 kB]
Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [239 kB]
Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [45.9 kB]
Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2,118 kB]
Get:23 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [15.4 kB]
Get:24 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,353 kB]
Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [2,165 kB]
Get:26 http://archive.ubuntu.com/ubuntu bionic-backports/universe Sources [5,759 B]
Get:27 http://archive.ubuntu.com/ubuntu bionic-backports/main Sources [5,823 B]
Get:28 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [11.3 kB]
Get:29 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [11.4 kB]
Reading package lists...
E: Repository 'https://oss-binaries.phusionpassenger.com/apt/passenger bionic Release' changed its 'Origin' value from 'packagecloud.io/phusion/passenger' to '. bionic'
E: Repository 'https://oss-binaries.phusionpassenger.com/apt/passenger bionic Release' changed its 'Label' value from 'passenger' to '. bionic'
dpkg-preconfigure: unable to re-open stdin: 
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 20877 files and directories currently installed.)
Preparing to unpack .../arvados-server_2.0.2-1_amd64.deb ...
Unpacking arvados-server (2.0.2-1) over (2.0.1-1) ...
Selecting previously unselected package arvados-api-server.
Preparing to unpack .../arvados-api-server_2.0.2-1_amd64.deb ...
Unpacking arvados-api-server (2.0.2-1) ...
Setting up arvados-server (2.0.2-1) ...
Setting up arvados-api-server (2.0.2-1) ...

Assumption: nginx is configured to serve Rails from
            /var/www/arvados-api/current
Assumption: nginx and passenger run as www-data

Creating symlinks to configuration in /etc/arvados/api ...... done.
Running bundle install... done.
Ensuring directory and file permissions ...... done.
Setting up database...Defaulting to memory cache, because /var/www/arvados-api/current/tmp/cache does not exist
 set_config 
------------
 
(1 row)

 done.
Checking configuration for completeness...Defaulting to memory cache, because /var/www/arvados-api/current/tmp/cache does not exist
 done.
Precompiling assets... done.
Creating git_internal_dir '/var/lib/arvados/internal.git'... done.
Initializing git_internal_dir '/var/lib/arvados/internal.git'... done.
Making sure '/var/lib/arvados/internal.git' has the right permission... done.
 * Restarting nginx nginx
   ...done.
 * Stopping nginx nginx
   ...done.
*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
*** Running /etc/my_init.d/10_syslog-ng.init...
Nov  2 15:58:08 arvados-api-server-d9b6ff6bb-7vfkr syslog-ng[11618]: syslog-ng starting up; version='3.13.2'
*** Running /etc/my_init.d/30_presetup_nginx.sh...
*** Running /etc/my_init.d/90-init-db.sh...
Defaulting to memory cache, because /var/www/arvados-api/current/tmp/cache owner (uid=33) is not me (uid=0)
Defaulting to memory cache, because /var/www/arvados-api/current/tmp/cache owner (uid=33) is not me (uid=0)
super
Defaulting to memory cache, because /var/www/arvados-api/current/tmp/cache owner (uid=33) is not me (uid=0)
anon
Defaulting to memory cache, because /var/www/arvados-api/current/tmp/cache owner (uid=33) is not me (uid=0)
*** Booting runit daemon...
*** Runit started as PID 12678
[ N 2020-11-02 15:58:07.6262 11561/T8 Ser/Server.h:902 ]: [ServerThr.1] Freed 0 spare client objects
[ N 2020-11-02 15:58:07.6262 11561/T8 Ser/Server.h:558 ]: [ServerThr.1] Shutdown finished
[ N 2020-11-02 15:58:07.6262 11561/Td Ser/Server.h:902 ]: [ServerThr.3] Freed 0 spare client objects
[ N 2020-11-02 15:58:07.6262 11561/Td Ser/Server.h:558 ]: [ServerThr.3] Shutdown finished
[ N 2020-11-02 15:58:07.6263 11561/Tf Ser/Server.h:902 ]: [ServerThr.4] Freed 0 spare client objects
[ N 2020-11-02 15:58:07.6263 11561/Tf Ser/Server.h:558 ]: [ServerThr.4] Shutdown finished
[ N 2020-11-02 15:58:07.6264 11561/Th Ser/Server.h:902 ]: [ApiServer] Freed 0 spare client objects
[ N 2020-11-02 15:58:07.6264 11561/Th Ser/Server.h:558 ]: [ApiServer] Shutdown finished
[ N 2020-11-02 15:58:08.0377 11561/T1 age/Cor/TelemetryCollector.h:531 ]: Message from Phusion: End time can not be before or equal to begin time
[ N 2020-11-02 15:58:08.0897 11561/T1 age/Cor/CoreMain.cpp:1325 ]: Passenger core shutdown finished
ok: run: /etc/service/nginx-log-forwarder: (pid 12684) 0s
Nov  2 15:58:35 arvados-api-server-d9b6ff6bb-7vfkr cron[12683]: (CRON) INFO (pidfile fd = 3)
Nov  2 15:58:35 arvados-api-server-d9b6ff6bb-7vfkr cron[12683]: (CRON) INFO (Running @reboot jobs)
#4

Okay, I think I see what’s happening here. I had not pushed new base images after the 2.0.1 release, and only bumped the default values.yaml to use 2.0.2 packages instead.

As you observed, this should have worked - the bootstrap.sh script runs apt-get update if the package version isn’t in the local apt cache. But, that call failed with this error:

E: Version '2.0.2-1' for 'arvados-api-server' was not found
Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:2 http://apt.arvados.org bionic InRelease [3,242 B]
Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease
Get:4 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Ign:5 https://oss-binaries.phusionpassenger.com/apt/passenger bionic InRelease
Get:6 https://oss-binaries.phusionpassenger.com/apt/passenger bionic Release [8,201 B]
Get:7 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:8 https://oss-binaries.phusionpassenger.com/apt/passenger bionic Release.gpg [833 B]
Get:9 http://security.ubuntu.com/ubuntu bionic-security/restricted Sources [13.0 kB]
Get:10 http://security.ubuntu.com/ubuntu bionic-security/main Sources [300 kB]
Get:11 http://security.ubuntu.com/ubuntu bionic-security/universe Sources [334 kB]
Get:12 http://security.ubuntu.com/ubuntu bionic-security/multiverse Sources [4,354 B]
Get:13 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,750 kB]
Get:14 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [213 kB]
Get:15 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,353 kB]
Get:16 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [15.4 kB]
Get:17 http://apt.arvados.org bionic/main amd64 Packages [117 kB]
Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/universe Sources [551 kB]
Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Sources [11.6 kB]
Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Sources [18.4 kB]
Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/main Sources [623 kB]
Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [239 kB]
Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2,118 kB]
Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [2,167 kB]
Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [45.9 kB]
Get:26 http://archive.ubuntu.com/ubuntu bionic-backports/main Sources [5,823 B]
Get:27 http://archive.ubuntu.com/ubuntu bionic-backports/universe Sources [5,759 B]
Get:28 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [11.4 kB]
Get:29 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [11.3 kB]
Reading package lists...
E: Repository 'https://oss-binaries.phusionpassenger.com/apt/passenger bionic Release' changed its 'Origin' value from 'packagecloud.io/phusion/passenger' to '. bionic'
E: Repository 'https://oss-binaries.phusionpassenger.com/apt/passenger bionic Release' changed its 'Label' value from 'passenger' to '. bionic'
E: Unable to correct problems, you have held broken packages.
 * Stopping nginx nginx
   ...done.
sh: 1: cd: can't cd to /var/www/arvados-api/current

It looks like the Passenger apt repositories changed in a way that made apt-get update very unhappy. Indeed, see https://github.com/phusion/passenger/issues/2275 for more background.

The fix is simple; I’ve rebuilt and pushed updated the base images with an updated apt cache.

I’ve also pushed a commit to the git repository that fixes a few other problems.

Does this help?

#5

A quick update: I’ve pushed updated helm charts that support Arvados 2.1. You’ll need the latest git commit for Arvados 2.1.

#6

Thanks, I was able to get 2.1.0 running. Maybe at some point in the future we can transition to versioned docker images.