Issue with Keep Service

Installation & Configuration
#1

Hello,

I am running a single host Arvados install on Ubuntu 18.04, after some minor issues I did get a full installation up with the workbench and workbench2 working. Furthermore, the cli works as expected once you get your api token.

However, when accessing a collection using workbench2 I get the following error in the browse:

“error getting file list”

The developer console shows:

“https://download.arvdv.i.fungalgenomics.ca/c%3Darvdv-4zz18-yphfkr8gtwldsh1 404”

journalctl for keep-web shows:

– Logs begin at Tue 2021-03-09 12:08:57 EST, end at Mon 2021-05-03 09:37:43 EDT. –
May 03 09:35:46 arvados.i.fungalgenomics.ca keep-web[36798]: {“RequestID”:“req-8so3ducdnragudik4kia”,“level”:“info”,“msg”:“response”,“remoteAddr”:“10.10.11.57:48218”,“reqBytes”:0,“reqForwardedFor”:“10.211.128.40”,“reqHost”:“download.arvdv.i.fungalgenomics.ca”,“reqMethod”:“PROPFIND”,“reqPath”:“c=arvdv-4zz18-yphfkr8gtwldsh1”,“reqQuery”:"",“respBody”:“404 Not found\r\n\r\nThe requested path was not found, or you do not have permission to access it.\r\n”,“respBytes”:96,“respStatus”:“Not Found”,“respStatusCode”:404,“time”:“2021-05-03T09:35:46.550200539-04:00”,“timeToStatus”:0.008063,“timeTotal”:0.008074,“timeWriteBody”:0.000012}
root@arvados:~# journalctl -ru keep-web -n 10 --no-pager
– Logs begin at Tue 2021-03-09 12:08:57 EST, end at Mon 2021-05-03 09:37:49 EDT. –
May 03 09:34:34 arvados.i.fungalgenomics.ca keep-web[36798]: {“RequestID”:“req-1cj3nywoxxqrkmws98l9”,“level”:“info”,“msg”:“request”,“remoteAddr”:“10.10.11.57:47542”,“reqBytes”:0,“reqForwardedFor”:“10.211.128.40”,“reqHost”:“download.arvdv.i.fungalgenomics.ca”,“reqMethod”:“GET”,“reqPath”:“c=arvdv-4zz18-yphfkr8gtwldsh1”,“reqQuery”:"",“time”:“2021-05-03T09:34:34.946600715-04:00”}
May 03 09:34:09 arvados.i.fungalgenomics.ca keep-web[36798]: {“RequestID”:“req-10rqobvhog8i4egf4mfg”,“level”:“info”,“msg”:“response”,“remoteAddr”:“10.10.11.57:47312”,“reqBytes”:0,“reqForwardedFor”:“10.211.128.40”,“reqHost”:“download.arvdv.i.fungalgenomics.ca”,“reqMethod”:“PROPFIND”,“reqPath”:“c=arvdv-4zz18-yphfkr8gtwldsh1”,“reqQuery”:"",“respBody”:“404 Not found\r\n\r\nThe requested path was not found, or you do not have permission to access it.\r\n”,“respBytes”:96,“respStatus”:“Not Found”,“respStatusCode”:404,“time”:“2021-05-03T09:34:09.980882132-04:00”,“timeToStatus”:0.008091,“timeTotal”:0.008101,“timeWriteBody”:0.000009}
May 03 09:34:09 arvados.i.fungalgenomics.ca keep-web[36798]: {“RequestID”:“req-10rqobvhog8i4egf4mfg”,“level”:“info”,“msg”:“request”,“remoteAddr”:“10.10.11.57:47312”,“reqBytes”:0,“reqForwardedFor”:“10.211.128.40”,“reqHost”:“download.arvdv.i.fungalgenomics.ca”,“reqMethod”:“PROPFIND”,“reqPath”:“c=arvdv-4zz18-yphfkr8gtwldsh1”,“reqQuery”:"",“time”:“2021-05-03T09:34:09.972776910-04:00”}
May 03 09:32:53 arvados.i.fungalgenomics.ca keep-web[36798]: {“RequestID”:“req-1h0po7zy2wlescbnbosd”,“level”:“info”,“msg”:“response”,“remoteAddr”:“10.10.11.57:46608”,“reqBytes”:0,“reqForwardedFor”:“10.211.128.40”,“reqHost”:“download.arvdv.i.fungalgenomics.ca”,“reqMethod”:“PROPFIND”,“reqPath”:“c=arvdv-4zz18-yphfkr8gtwldsh1”,“reqQuery”:"",“respBody”:“404 Not found\r\n\r\nThe requested path was not found, or you do not have permission to access it.\r\n”,“respBytes”:96,“respStatus”:“Not Found”,“respStatusCode”:404,“time”:“2021-05-03T09:32:53.445144690-04:00”,“timeToStatus”:0.008123,“timeTotal”:0.008133,“timeWriteBody”:0.000010}
May 03 09:32:53 arvados.i.fungalgenomics.ca keep-web[36798]: {“RequestID”:“req-1h0po7zy2wlescbnbosd”,“level”:“info”,“msg”:“request”,“remoteAddr”:“10.10.11.57:46608”,“reqBytes”:0,“reqForwardedFor”:“10.211.128.40”,“reqHost”:“download.arvdv.i.fungalgenomics.ca”,“reqMethod”:“PROPFIND”,“reqPath”:“c=arvdv-4zz18-yphfkr8gtwldsh1”,“reqQuery”:"",“time”:“2021-05-03T09:32:53.437004768-04:00”}
May 03 09:32:53 arvados.i.fungalgenomics.ca keep-web[36798]: {“RequestID”:“req-1p6ahd0aepkhf159g2ny”,“level”:“info”,“msg”:“response”,“remoteAddr”:“10.10.11.57:46606”,“reqBytes”:0,“reqForwardedFor”:“10.211.128.40”,“reqHost”:“download.arvdv.i.fungalgenomics.ca”,“reqMethod”:“OPTIONS”,“reqPath”:“c=arvdv-4zz18-yphfkr8gtwldsh1”,“reqQuery”:"",“respBytes”:0,“respStatus”:“OK”,“respStatusCode”:200,“time”:“2021-05-03T09:32:53.435121639-04:00”,“timeToStatus”:0.000094,“timeTotal”:0.000094,“timeWriteBody”:0.000000}
May 03 09:32:53 arvados.i.fungalgenomics.ca keep-web[36798]: {“RequestID”:“req-1p6ahd0aepkhf159g2ny”,“level”:“info”,“msg”:“request”,“remoteAddr”:“10.10.11.57:46606”,“reqBytes”:0,“reqForwardedFor”:“10.211.128.40”,“reqHost”:“download.arvdv.i.fungalgenomics.ca”,“reqMethod”:“OPTIONS”,“reqPath”:“c=arvdv-4zz18-yphfkr8gtwldsh1”,“reqQuery”:"",“time”:“2021-05-03T09:32:53.435022138-04:00”}
May 03 09:19:28 arvados.i.fungalgenomics.ca keep-web[36798]: {“RequestID”:“req-13sd5md6783fwm2prx2c”,“level”:“info”,“msg”:“response”,“remoteAddr”:“10.10.11.57:39502”,“reqBytes”:0,“reqForwardedFor”:“23.148.145.29”,“reqHost”:“X.X.X.X”,“reqMethod”:“GET”,“reqPath”:“vicidial/admin.php”,“reqQuery”:“ADD=140000000000”,“respBody”:“404 Not found\r\n\r\nThe requested path was not found, or you do not have permission to access it.\r\n”,“respBytes”:96,“respStatus”:“Not Found”,“respStatusCode”:404,“time”:“2021-05-03T09:19:28.129151853-04:00”,“timeToStatus”:0.000092,“timeTotal”:0.000109,“timeWriteBody”:0.000017}
May 03 09:19:28 arvados.i.fungalgenomics.ca keep-web[36798]: {“RequestID”:“req-13sd5md6783fwm2prx2c”,“level”:“info”,“msg”:“request”,“remoteAddr”:“10.10.11.57:39502”,“reqBytes”:0,“reqForwardedFor”:“23.148.145.29”,“reqHost”:“X.X.X.X”,“reqMethod”:“GET”,“reqPath”:“vicidial/admin.php”,“reqQuery”:“ADD=140000000000”,“time”:“2021-05-03T09:19:28.129035951-04:00”}
May 03 07:33:37 arvados.i.fungalgenomics.ca keep-web[36798]: {“RequestID”:“req-tvg58rqwfd89cnyfod9t”,“level”:“info”,“msg”:“response”,“remoteAddr”:“10.10.11.57:40310”,“reqBytes”:0,“reqForwardedFor”:“193.106.29.210”,“reqHost”:“X.X.X.X”,“reqMethod”:“GET”,“reqPath”:"",“reqQuery”:"",“respBody”:“401 Unauthorized\r\n\r\nA valid Arvados token must be provided to access this resource.\r\n”,“respBytes”:85,“respStatus”:“Unauthorized”,“respStatusCode”:401,“time”:“2021-05-03T07:33:37.488039301-04:00”,“timeToStatus”:0.000098,“timeTotal”:0.000106,“timeWriteBody”:0.000008}

This seems to be the only place that is showing this error - when using workbench (and not workbench2) we actually have no issues. However, there are a number of things our bioinformaticians are getting errors for that may be related to the issue above (namely 404 errors) I can get more details about those if necessary.

I suspect I have done something wrong during the install, but I’m not sure what that could be. The only change I have made is to move the keep storage from /tmp to /mnt/arv, but the problem predates that changes (although I acknowledge there can be more than one cause for any problem)

Thank you so much for your help, whoever you are!

Andrei

#2

Hi @Andrei_Wasylyk,

Just to get the whole picture, did you perform a ‘single host / multiple hostnames’ installation, right?

If that’s the case, your situation seems to be caused by a bug we discovered (and fixed) recently , where keep-web does not handle correctly the explicit specification of port 443 in the URLs for the Services:WebDAV and Services:WebDAVDownload sections.

A quick fix would be that you:

  1. edit the /etc/arvados/config.yml
  2. remove the :443 part in the ExternalURL: entries for those two services
  3. save the file.
  4. restart the service with sudo systemctl restart keep-web

Please, let me know if this helps you.

Regards, Javier

#3

Yep, single host deploy with with the multiple hostnames pointing to the private IP.

This is exactly the bug you linked and your workaround fixed it. I only handle the infrastructure so my experience is mostly with the GUI but our bioinformaticians are reporting their issues at the terminal are also resolved (presumably whenever they were doing something that was hitting keep-web).

Thank you so much for your reply, the deployment has been quite painless and this was the only issue I noticed at install so that’s promising.

1 Like