We are migrating our servers to Oracle Cloud. However Oracle Cloud only supports S3 storage with PathStyle.
We wonder if Arvados can support usePathStyle for S3 storage? Any possibility to add configuration flag to allow user to specify usePathStyle for S3 storage?
Thank you.
I believe that when you provide a custom storage endpoint it will attempt to use hostname style and fall back to path style if that doesn’t work. So you may not need to do anything. However I have not checked the code to be sure. It is possible path style requires a flag to be passed to the AWS sdk. If that is the case that should be a straightforward change and we’d happily accept a PR.
Thank you for the reply.
We tried using PathStyle but failed with below error:
Sep 11 02:12:50 keep0 keepstore[971]: {“ClusterID”:“cborg”,“PID”:971,“RequestID”:“req-dkpsz75ng85d3pgre68n”,“error”:“exceeded maximum number of attempts, 3, request send failed, Put “https://arvados-bucket1.axpl2yfqp7q2.compat.objectstorage.me-jeddah-1.oraclecloud.com/d41d8cd98f00b204e9800998ecf8427e”: tls: failed to verify certificate: x509: certificate is valid for swiftobjectstorage.me-jeddah-1.oraclecloud.com, not arvados-bucket1.axpl2yfqp7q2.compat.objectstorage.me-jeddah-1.oraclecloud.com”,“level”:“error”,“msg”:“s3-bucket:“arvados-bucket1”: Put(d41d8cd98f00b204e9800998ecf8427e) failed”,“remoteAddr”:“10.1.1.2:41092”,“reqBytes”:0,“reqForwardedFor”:"",“reqHost”:“keep0.cborg.projectnelly.com:25107”,“reqMethod”:“PUT”,“reqPath”:“d41d8cd98f00b204e9800998ecf8427e”,“reqQuery”:"",“time”:“2024-09-11T02:12:50.758511772Z”}
This is how we define in the config file:
Clusters:
clustername:
Volumes:
volumename:
AccessViaHosts:
“http://keep0.domain.com:25107”: {}
Driver: S3
DriverParameters:
Bucket: arvados-bucket1
#ORACLE ENDPOINT:
Endpoint: "https://axpl2yfqp7q2.compat.objectstorage.me-jeddah-1.oraclecloud.com"
LocationConstraint: false
StorageClasses: null
So it seems we may need a flag to arvados to use Path Style?
This seems to be a certificate error?
Put “https://arvados-bucket1.axpl2yfqp7q2.compat.objectstorage.me-jeddah-1.oraclecloud.com/d41d8cd98f00b204e9800998ecf8427e”: tls: failed to verify certificate: x509: certificate is valid for swiftobjectstorage.me-jeddah-1.oraclecloud.com, not arvados-bucket1.axpl2yfqp7q2.compat.objectstorage.me-jeddah-1.oraclecloud.com”
The hostname “swiftobjectstorage.me-jeddah-1.oraclecloud.com” is different from “axpl2yfqp7q2.compat.objectstorage.me-jeddah-1.oraclecloud.com”.
Is there public documentation about Oracle cloud object storage?
Based on Oracle documentation it only supports path style of S3 storage.
https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm
Arvados is trying to access Oracle storage using virtual hosted style: arvados-bucket1.axpl2yfqp7q2.compat.objectstorage.me-jeddah-1.oraclecloud.com.
But Oracle does not support virtual hosted style for S3, have to use path style (in this case: axpl2yfqp7q2.compat.objectstorage.me-jeddah-1.oraclecloud.com/arvados-bucket1).
The error says that the certificate is valid but since arvados-bucket1.axpl2yfqp7q2.compat.objectstorage.me-jeddah-1.oraclecloud.com does not exist, Oracle fallbacks to the default API for storage access which is swift.
So, it exists as an internal flag used in testing:
But what you want to do is move it to S3VolumeDriverParameters which is part of the config file.
We’ll be happy to accept a pull request with this change.
We try to add option in config for S3 UsePathStyle. And need some advice if we should set the default value of this flag to false in below file https://github.com/arvados/arvados/blob/main/lib/config/config.default.yml. ?
Yes, the default should be set to false in config.default.yml.
Thank you for the advice.
Have created PR #265 for this change
Appreciate your review and inclusion into next release.
