Bucket listing API should include non-default bucket types

[michellep-basho]

DESCRIPTION
Per Randy:
As of 2.0 build pre7 the list buckets operation only includes buckets of the 'default' bucket type. If bucket types are added (say through usage of Yokozuna), those buckets are no longer listed. There is a riak-admin command which uses internal Erlang APIs to list bucket types; but this is only useable to operational staff using Riak. This change also breaks developer / web admin tools (such as rekon), used for data modeling.

IMPACT:
We have 4 known customers potentially impacted by this. Additionally, it impact the ProServ developers. Please contact me or Randy S for further more details.

[seancribbs]

We have tests that demonstrate that listing buckets within known bucket-types works. Whether bucket-types themselves should be able to be listed from a client API is a separate issue, I believe (caveats around security/disclosure, etc).

[Vagabond]

We can certainly add a permission to allow you to list bucket types, along the lines of the permission that allows bucket listing.

[jrwest]

We have discussed a list bucket types API previously as part of the larger bucket type API discussion. We are aware that several features are missing from that API and do not plan to expose any of it this release (the only way to work w/ bucket types should be the command line w/ Riak 2.0). I believe we plan to revist and fully implement the API in the future (caveats re: security).

Additionally, we need to make a distinction between "listing bucket types" and "listing buckets under a type". The current list bucket operation maps to "listing buckets under the default type". I believe we have extended that to including "listing buckets under any specific type" however, it sounds like there is a feature request for "listing buckets across multiple types at once"?

Sent from my iPhone

On Jan 3, 2014, at 11:40 AM, Andrew Thompson notifications@github.com wrote:

We can certainly add a permission to allow you to list bucket types, along the lines of the permission that allows bucket listing.

—
Reply to this email directly or view it on GitHub.

[michellep-basho]

...however, it sounds like there is a feature request for "listing buckets across multiple types at once"?
yes

[jrwest]

ok. just wanted to clarify. considering that would be an entirely new feature I was confused by the report that customers are currently affected...Even with the current state of non-existence of the API one could continue to operate web admin tools that use the "listing buckets under/within a type" as @seancribbs mentioned. Riak 2.0 operates just as it used to under the default bucket type and web-admin tools and other applications should not be affected just because they exist.

In addition, bucket types, from a data modeling perspective, at least at this time, should not be created dynamically (something that would obviously be difficult anyways since there is no write operations via API currently). This should make it easy to enumerate them all even w/o an API to do so. Tools may continue to operate but a w/ degraded user experience but they are not broken. Once the API is available they can of course be improved, and that is why we would like to provide the API in a future release.

I would be curious to know more about the customers that may potentially have issues w/ this to see if we can find a workaround, in the meantime, as well as to ensure we build the API they want, in the future. Would be happy to discuss that elsewhere.

Regardless, I think it would be should to use this issue to discuss the future work on the bucket types API, so we have a central place. It has been scattered across several issues which makes the discussion harder to follow.

For completeness, here is the current state of bucket types as has been shown to users (the only mention of an API is there isn't one): https://gist.github.com/jrwest/3652ecd1db609e25cdaa

[1] https://github.com/basho/riak_test/blob/master/tests/bucket_types.erl#L125

[michellep-basho]

I will ask Randy to weigh in more on the existing customer use cases and security considerations, so we have more context. I understand that there are other things we need to consider like security, but I can also see how from a customer perspective it might seem like basic functionality is missing in this initial implementation. thanks

[jrwest]

Sounds good. I agree there is functionality missing. Unfortunately, it was a casualty of the release cycle and the lack of time necessary to really map out how things played w/ security on/off, etc. Customers also probably won't love that we are using JSON on the command line either, but we were unable to fix that in time as well...

[michellep-basho]

Randy is available for next Tue's core/kv mumble. We can discuss more.

[jrwest]

Unfortunately I probably won't be able to make that mumble -- not that I am required for the conversation but fyi. I will be out of town. I may be able to set something up for the evening (pacific) while I'm away otherwise I will be back the following Tuesday.

[michellep-basho]

Let's plan on discussing this the following week when Jordan is back. thanks

[jaredmorrow]

Any status on this since the last planned meeting? (3 months ago)

[jrwest]

The tl;dr is there is no user-facing API for bucket types (code exists for some of it, but it is not documented or supported in Riak 2.0). We should use this issue to track work for 2.1 to add it.