Issue with Keep Service

Hello,

I am running a single host Arvados install on Ubuntu 18.04, after some minor issues I did get a full installation up with the workbench and workbench2 working. Furthermore, the cli works as expected once you get your api token.

However, when accessing a collection using workbench2 I get the following error in the browse:

“error getting file list”

The developer console shows:

https://download.arvdv.i.fungalgenomics.ca/c%3Darvdv-4zz18-yphfkr8gtwldsh1 404”

journalctl for keep-web shows:

– Logs begin at Tue 2021-03-09 12:08:57 EST, end at Mon 2021-05-03 09:37:43 EDT. –
May 03 09:35:46 arvados.i.fungalgenomics.ca keep-web[36798]: {“RequestID”:“req-8so3ducdnragudik4kia”,“level”:“info”,“msg”:“response”,“remoteAddr”:“10.10.11.57:48218”,“reqBytes”:0,“reqForwardedFor”:“10.211.128.40”,“reqHost”:“download.arvdv.i.fungalgenomics.ca”,“reqMethod”:“PROPFIND”,“reqPath”:“c=arvdv-4zz18-yphfkr8gtwldsh1”,“reqQuery”:"",“respBody”:“404 Not found\r\n\r\nThe requested path was not found, or you do not have permission to access it.\r\n”,“respBytes”:96,“respStatus”:“Not Found”,“respStatusCode”:404,“time”:“2021-05-03T09:35:46.550200539-04:00”,“timeToStatus”:0.008063,“timeTotal”:0.008074,“timeWriteBody”:0.000012}
root@arvados:~# journalctl -ru keep-web -n 10 --no-pager
– Logs begin at Tue 2021-03-09 12:08:57 EST, end at Mon 2021-05-03 09:37:49 EDT. –
May 03 09:34:34 arvados.i.fungalgenomics.ca keep-web[36798]: {“RequestID”:“req-1cj3nywoxxqrkmws98l9”,“level”:“info”,“msg”:“request”,“remoteAddr”:“10.10.11.57:47542”,“reqBytes”:0,“reqForwardedFor”:“10.211.128.40”,“reqHost”:“download.arvdv.i.fungalgenomics.ca”,“reqMethod”:“GET”,“reqPath”:“c=arvdv-4zz18-yphfkr8gtwldsh1”,“reqQuery”:"",“time”:“2021-05-03T09:34:34.946600715-04:00”}
May 03 09:34:09 arvados.i.fungalgenomics.ca keep-web[36798]: {“RequestID”:“req-10rqobvhog8i4egf4mfg”,“level”:“info”,“msg”:“response”,“remoteAddr”:“10.10.11.57:47312”,“reqBytes”:0,“reqForwardedFor”:“10.211.128.40”,“reqHost”:“download.arvdv.i.fungalgenomics.ca”,“reqMethod”:“PROPFIND”,“reqPath”:“c=arvdv-4zz18-yphfkr8gtwldsh1”,“reqQuery”:"",“respBody”:“404 Not found\r\n\r\nThe requested path was not found, or you do not have permission to access it.\r\n”,“respBytes”:96,“respStatus”:“Not Found”,“respStatusCode”:404,“time”:“2021-05-03T09:34:09.980882132-04:00”,“timeToStatus”:0.008091,“timeTotal”:0.008101,“timeWriteBody”:0.000009}
May 03 09:34:09 arvados.i.fungalgenomics.ca keep-web[36798]: {“RequestID”:“req-10rqobvhog8i4egf4mfg”,“level”:“info”,“msg”:“request”,“remoteAddr”:“10.10.11.57:47312”,“reqBytes”:0,“reqForwardedFor”:“10.211.128.40”,“reqHost”:“download.arvdv.i.fungalgenomics.ca”,“reqMethod”:“PROPFIND”,“reqPath”:“c=arvdv-4zz18-yphfkr8gtwldsh1”,“reqQuery”:"",“time”:“2021-05-03T09:34:09.972776910-04:00”}
May 03 09:32:53 arvados.i.fungalgenomics.ca keep-web[36798]: {“RequestID”:“req-1h0po7zy2wlescbnbosd”,“level”:“info”,“msg”:“response”,“remoteAddr”:“10.10.11.57:46608”,“reqBytes”:0,“reqForwardedFor”:“10.211.128.40”,“reqHost”:“download.arvdv.i.fungalgenomics.ca”,“reqMethod”:“PROPFIND”,“reqPath”:“c=arvdv-4zz18-yphfkr8gtwldsh1”,“reqQuery”:"",“respBody”:“404 Not found\r\n\r\nThe requested path was not found, or you do not have permission to access it.\r\n”,“respBytes”:96,“respStatus”:“Not Found”,“respStatusCode”:404,“time”:“2021-05-03T09:32:53.445144690-04:00”,“timeToStatus”:0.008123,“timeTotal”:0.008133,“timeWriteBody”:0.000010}
May 03 09:32:53 arvados.i.fungalgenomics.ca keep-web[36798]: {“RequestID”:“req-1h0po7zy2wlescbnbosd”,“level”:“info”,“msg”:“request”,“remoteAddr”:“10.10.11.57:46608”,“reqBytes”:0,“reqForwardedFor”:“10.211.128.40”,“reqHost”:“download.arvdv.i.fungalgenomics.ca”,“reqMethod”:“PROPFIND”,“reqPath”:“c=arvdv-4zz18-yphfkr8gtwldsh1”,“reqQuery”:"",“time”:“2021-05-03T09:32:53.437004768-04:00”}
May 03 09:32:53 arvados.i.fungalgenomics.ca keep-web[36798]: {“RequestID”:“req-1p6ahd0aepkhf159g2ny”,“level”:“info”,“msg”:“response”,“remoteAddr”:“10.10.11.57:46606”,“reqBytes”:0,“reqForwardedFor”:“10.211.128.40”,“reqHost”:“download.arvdv.i.fungalgenomics.ca”,“reqMethod”:“OPTIONS”,“reqPath”:“c=arvdv-4zz18-yphfkr8gtwldsh1”,“reqQuery”:"",“respBytes”:0,“respStatus”:“OK”,“respStatusCode”:200,“time”:“2021-05-03T09:32:53.435121639-04:00”,“timeToStatus”:0.000094,“timeTotal”:0.000094,“timeWriteBody”:0.000000}
May 03 09:32:53 arvados.i.fungalgenomics.ca keep-web[36798]: {“RequestID”:“req-1p6ahd0aepkhf159g2ny”,“level”:“info”,“msg”:“request”,“remoteAddr”:“10.10.11.57:46606”,“reqBytes”:0,“reqForwardedFor”:“10.211.128.40”,“reqHost”:“download.arvdv.i.fungalgenomics.ca”,“reqMethod”:“OPTIONS”,“reqPath”:“c=arvdv-4zz18-yphfkr8gtwldsh1”,“reqQuery”:"",“time”:“2021-05-03T09:32:53.435022138-04:00”}
May 03 09:19:28 arvados.i.fungalgenomics.ca keep-web[36798]: {“RequestID”:“req-13sd5md6783fwm2prx2c”,“level”:“info”,“msg”:“response”,“remoteAddr”:“10.10.11.57:39502”,“reqBytes”:0,“reqForwardedFor”:“23.148.145.29”,“reqHost”:“X.X.X.X”,“reqMethod”:“GET”,“reqPath”:“vicidial/admin.php”,“reqQuery”:“ADD=140000000000”,“respBody”:“404 Not found\r\n\r\nThe requested path was not found, or you do not have permission to access it.\r\n”,“respBytes”:96,“respStatus”:“Not Found”,“respStatusCode”:404,“time”:“2021-05-03T09:19:28.129151853-04:00”,“timeToStatus”:0.000092,“timeTotal”:0.000109,“timeWriteBody”:0.000017}
May 03 09:19:28 arvados.i.fungalgenomics.ca keep-web[36798]: {“RequestID”:“req-13sd5md6783fwm2prx2c”,“level”:“info”,“msg”:“request”,“remoteAddr”:“10.10.11.57:39502”,“reqBytes”:0,“reqForwardedFor”:“23.148.145.29”,“reqHost”:“X.X.X.X”,“reqMethod”:“GET”,“reqPath”:“vicidial/admin.php”,“reqQuery”:“ADD=140000000000”,“time”:“2021-05-03T09:19:28.129035951-04:00”}
May 03 07:33:37 arvados.i.fungalgenomics.ca keep-web[36798]: {“RequestID”:“req-tvg58rqwfd89cnyfod9t”,“level”:“info”,“msg”:“response”,“remoteAddr”:“10.10.11.57:40310”,“reqBytes”:0,“reqForwardedFor”:“193.106.29.210”,“reqHost”:“X.X.X.X”,“reqMethod”:“GET”,“reqPath”:"",“reqQuery”:"",“respBody”:“401 Unauthorized\r\n\r\nA valid Arvados token must be provided to access this resource.\r\n”,“respBytes”:85,“respStatus”:“Unauthorized”,“respStatusCode”:401,“time”:“2021-05-03T07:33:37.488039301-04:00”,“timeToStatus”:0.000098,“timeTotal”:0.000106,“timeWriteBody”:0.000008}

This seems to be the only place that is showing this error - when using workbench (and not workbench2) we actually have no issues. However, there are a number of things our bioinformaticians are getting errors for that may be related to the issue above (namely 404 errors) I can get more details about those if necessary.

I suspect I have done something wrong during the install, but I’m not sure what that could be. The only change I have made is to move the keep storage from /tmp to /mnt/arv, but the problem predates that changes (although I acknowledge there can be more than one cause for any problem)

Thank you so much for your help, whoever you are!

Andrei

Hi @Andrei_Wasylyk,

Just to get the whole picture, did you perform a ‘single host / multiple hostnames’ installation, right?

If that’s the case, your situation seems to be caused by a bug we discovered (and fixed) recently , where keep-web does not handle correctly the explicit specification of port 443 in the URLs for the Services:WebDAV and Services:WebDAVDownload sections.

A quick fix would be that you:

  1. edit the /etc/arvados/config.yml
  2. remove the :443 part in the ExternalURL: entries for those two services
  3. save the file.
  4. restart the service with sudo systemctl restart keep-web

Please, let me know if this helps you.

Regards, Javier