[Bug]: CLI: connection to Iceberg REST API fails unexpectedly

[adutra]

What happened

Attempting to connect to Iceberg REST API results in a weird error:

Nessie> connect to http://x.y.w.z:19120/iceberg/v1
Connecting to http://x.y.w.z:19120/iceberg/v1 ...
No Iceberg REST endpoint at http://x.y.w.z:19120/iceberg/v1 ...
Encountered an error executing the following statement
    connect to http://x.y.w.z:19120/iceberg/v1

NessieBadResponseException: OK (HTTP/200): Expected the server to return a JSON compatible response, but the server replied with Content-Type 'application/json;charset=UTF-8' from 'http://x.y.w.z:19120/iceberg/v1/config' and a response body beginning with: "{
  "defaults" : {
    "io-impl" : "org.apache.iceberg.io.ResolvingFileIO",
    "prefix" : "main",
    "warehouse" : "s3://warehouse",
    "rest-page-size" : "200",
    "rest-metrics-reporting-enabled" : "false"
  },
  "overrides" : {
    "nessie.default-branch.name" : "main",
    "nessie.is-nessie-catalog" : "true",
    "nessie.prefix-pattern" : "{ref}|{warehouse}",
    "nessie.core-base-uri" : "http://x.y.w.z:19120/api/",
    "nessie.iceberg-base-uri" : "http://x.y.w.z:19120/iceberg/",
    "nessie.catalog-base-uri" : "http://x.y.w.z:19120/catalog/v1/",
    "s3.remote-signing-enabled" : "true",
    "uri" : "http://x.y.w.z:19120/iceberg/"
  }
}". Check the Nessie REST API base URI. Nessie REST API base URI usually ends in '/api/v1' or '/api/v2', but your service provider may have a different URL pattern.
    caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "defaults" (class org.projectnessie.model.ImmutableNessieConfiguration$Json), not marked as ignorable (9 known properties: "noAncestorHash", "defaultBranch", "maxSupportedApiVersion", "minSupportedApiVersion", "actualApiVersion", "specVersion", "oldestPossibleCommitTimestamp", "additionalProperties", "repositoryCreationTimestamp"])
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 2, column: 17] (through reference chain: org.projectnessie.model.ImmutableNessieConfiguration$Json["defaults"])

How to reproduce it

See above

Nessie server type (docker/uber-jar/built from source) and version

0.94.3

Client type (Ex: UI/Spark/pynessie ...) and version

No response

Additional information

Nessie CLI 0.94.3

[adutra]

Debugging a bit, this happens because the Nessie server requires OAUTH2 authentication with complex grant types, but RESTCatalog only accepts client_credentials.

The error happens when fetching the config endpoint.

I think we should remove RESTCatalog here and replace it with Nessie's own HTTP client.

[adutra]

org.apache.iceberg.exceptions.NotAuthorizedException: Not authorized: 
	at org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.accept(ErrorHandlers.java:210)
	at org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.accept(ErrorHandlers.java:188)
	at org.apache.iceberg.rest.HTTPClient.throwFailure(HTTPClient.java:211)
	at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:323)
	at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:262)
	at org.apache.iceberg.rest.HTTPClient.get(HTTPClient.java:358)
	at org.apache.iceberg.rest.RESTSessionCatalog.fetchConfig(RESTSessionCatalog.java:980)
	at org.apache.iceberg.rest.RESTSessionCatalog.initialize(RESTSessionCatalog.java:223)
	at org.apache.iceberg.rest.RESTCatalog.initialize(RESTCatalog.java:78)

[adutra]

It's non trivial to determine why the iceberg detection failed, and the fact that both APIs try to hit a /config endpoint first makes it even harder.

@snazy wdyt of stopping guessing and adding this to the grammar:

CONNECT TO ((NESSIE|ICEBERG) API AT)?  http://...

[snazy]

Damn - if that's the only viable option, we have to bite the bullet. Looking for /api/ or /iceberg/ in the URI path might be misleading as well.

[dimas-b]

Looking for /api/ or /iceberg/ in the URI path might be misleading as well.

I guess we cannot reliably derive the Nessie URI from the Iceberg URI and vice versa.

How about: CONNECT TO [NESSIE API AT ...] [ICEBERG API AT ...])?

I think we'll need similar changes to the CLI options. How about -n / --nessie for the Nessie URI, and -i / --iceberg-rest for the Iceberg URI (both settable independently)?