Runs a REST API to offer management of Kafka topics, users, and ACLs, schema registry Avro schema mappings to topic
get /octhc
get /v1/kafka/clusters
get /v1/kafka/topics/:topic
get /v1/kafka/topics
get /v1/kafka/cluster/:cluster/credentials/:user
get /v1/kafka/cluster/:cluster/acls
get /v1/kafka/cluster/:cluster/schemas
get /v1/kafka/cluster/:cluster/schemas/:schema/versions
get /v1/kafka/cluster/:cluster/schemas/:schema
get /v1/kafka/cluster/:cluster/topics/:topic/topic-schema-mappings
post /v1/kafka/cluster/:cluster/topic
post /v1/kafka/cluster/:cluster/user
post /v1/kafka/cluster/:cluster/acls
post /v1/kafka/cluster/:cluster/topic-schema-mapping
post /v1/kafka/cluster/:cluster/topic-key-mapping
delete /v1/kafka/acls/:id
delete /v1/kafka/cluster/:cluster/topics/:topic
SANDBOX_KAFKA_PORT
SANDBOX_DEFAULT_PARTITIONS
SANDBOX_DEFAULT_REPLICAS
SANDBOX_DEFAULT_RETENTION
SANDBOX_KAFKA_ADMIN_USERNAME
SANDBOX_KAFKA_ADMIN_PASSWORD
SANDBOX_KAFKA_LOCATION
SANDBOX_KAFKA_HOSTNAME
SANDBOX_ZK
SANDBOX_KAFKA_AVRO_REGISTRY_LOCATION
SANDBOX_KAFKA_AVRO_REGISTRY_HOSTNAME
SANDBOX_KAFKA_AVRO_REGISTRY_PORT
DEV_KAFKA_PORT
DEV_DEFAULT_PARTITIONS
DEV_DEFAULT_REPLICAS
DEV_DEFAULT_RETENTION
DEV_KAFKA_ADMIN_USERNAME
DEV_KAFKA_ADMIN_PASSWORD
DEV_KAFKA_LOCATION
DEV_KAFKA_HOSTNAME
DEV_ZK
DEV_KAFKA_AVRO_REGISTRY_LOCATION
DEV_KAFKA_AVRO_REGISTRY_HOSTNAME
DEV_KAFKA_AVRO_REGISTRY_PORT
PRODUCTION_KAFKA_POST
PRODUCTION_DEFAULT_PARTITIONS
PRODUCTION_DEFAULT_REPLICAS
PRODUCTION_DEFAULT_RETENTION
PRODUCTION_KAFKA_ADMIN_USERNAME
PRODUCTION_KAFKA_ADMIN_PASSWORD
PRODUCTION_KAFKA_LOCATION
PRODUCTION_KAFKA_HOSTNAME
PRODUCTION_ZK
PRODUCTION_KAFKA_AVRO_REGISTRY_LOCATION
PRODUCTION_KAFKA_AVRO_REGISTRY_HOSTNAME
PRODUCTION_KAFKA_AVRO_REGISTRY_PORT
BROKERDB
BROKERDBUSER
BROKERDBPASS
PORT
source local_env.sh
sbt run
##DB setup
Insert all the the user names and passwords for all required kafka cluster(ex: dev, sandbox, nonprod, prod) to acl_source database table, with claimed being false.
Insert all cluster name and description to CLUSTER table
##Endpoints Description
None
[
{
"name": "nonprod",
"description": "Non Prod Kafka Cluster with 3 Brokers"
},
{
"name": "prod",
"description": "Production Kafka Cluster"
}
]Get All Clusters
200: Ok with response Json
404: Cannot find topic :topic
500: Internal Server Error
None
{
"topic":{
"name":"test.some.topic",
"description":"Test topic creation",
"organization":"testOrg",
"config":{
"name":"ledger",
"cleanup.policy":"delete",
"partitions":1,
"retention.ms":888888,
"replicas":1
},
"keyMapping": {
"keyType": "string"
},
"schemas": [
"BMacTest1-key"
],
"cluster": "maru"
}
}Get Topic info on a specific topic
200: Ok with response Json
404: Cannot find topic :topic
500: Internal Server Error
None
{
"topics":[
{
"name":"test.some.topic",
"description":"Test topic creation",
"organization":"testOrg",
"config":{
"name":"ledger",
"cleanup.policy":"delete",
"partitions":1,
"retention.ms":888888,
"replicas":1
},
"keyMapping": {
"keyType": "string"
},
"cluster": "maru"
}
]
}Gets topics across all clusters known to the app. Gets this information from topic table
200: Ok with above response
500: Internal Service Error
None
{
"KAFKA_LOCATION":"localhost:6001",
"KAFKA_USERNAME":"testusername",
"KAFKA_CONSUMER_TOPICS":"test.some.topic",
"KAFKA_PORT":"6001",
"DEV_SOME_TOPIC_TOPIC_KEY_TYPE":"NONE",
"DEV_SOME_TOPIC_2_TOPIC_SCHEMAS":"",
"KAFKA_HOSTNAME":"localhost",
"KAFKA_AVRO_REGISTRY_LOCATION":"testLocation",
"KAFKA_AVRO_REGISTRY_PORT":"",
"KAFKA_PRODUCER_TOPICS":"test.some.topic.2",
"DEV_SOME_TOPIC_TOPIC_NAME":"test.some.topic",
"DEV_SOME_TOPIC_TOPIC_SCHEMAS":"testschema",
"DEV_SOME_TOPIC_2_TOPIC_KEY_TYPE":"NONE",
"KAFKA_PASSWORD":"testpassword",
"DEV_SOME_TOPIC_2_TOPIC_NAME":"test.some.topic.2",
"KAFKA_AVRO_REGISTRY_HOSTNAME":""
}Endpoint to get credentials / environment variables for a claimed user.
200: Ok with above response.
400: User does not exist or is unclaimed.
{
"errors":[
{
"title":"Invalid User",
"detail":"Either user :user does not exist in cluster test or the user is not claimed."
}
]
}topic: Topic name. Required string.
{
"acls":[
{
"id":"7ab2d06a-937d-4da4-8bc7-323e219c34f5",
"user":"testusername",
"topic":"test.some.topic",
"cluster":"test",
"role":"consumer"
},
{
"id":"76f06077-c50c-4242-8e08-0444e579a60e",
"user":"testusername",
"topic":"test.some.topic",
"cluster":"test",
"role":"producer"
}
]
}Gets all Acls for the given topic.
200: Ok with above response
400: Invalid ACL role value in the database.
None
{
"schemas":[
"schema1",
"schema2"
]
}Lists all Schema subjects for the given cluster. Get this result from schema registry.
200: With above response
500: Internal Service Error
None
{
"versions":[
1,
2,
3
]
}Makes a rest call to cluster's schema registry to get all versions for the given schema subject.
200: With above response
404: Schema with subject $schema not found
500: External service exception
None
{
"subject":"test.schema1",
"version":2,
"schema":"{\"type\": \"string\"}"
}Get Schema Details and schema for the cluster, schema subject and version. Call Schema registry to get the same.
200: Ok with above response
404: Schema not found for $schema version $version
500: Internal Service Error
None
{
"mappings":[
{
"topic":"test.some.topic",
"schema":{
"name":"testSchema",
"version":1
}
}
]
}Get all the schema mappings for the topic from database.
200: Ok with above response
500: Internal Service Error
{
"topic":{
"name":"test.some.topic",
"description":"Test topic creation",
"organization":"testOrg",
"config":{
"name":"compact",
"cleanup.policy":"compact", //optional
"partitions":1, //optional
"retention.ms":-1, //optional
"replicas":1 //optional
}
}
}{
"topic":{
"name":"test.some.topic",
"description":"Test topic creation",
"organization":"testOrg",
"config":{
"name": "compact"
"cleanup.policy":"compact",
"partitions":1,
"retention.ms":-1,
"replicas":1
}
}
}Create a topic with delete or compact topic type.
200: With above response.
500: Internal Service Error
None
{
"aclCredentials":{
"username":"testusername",
"password":"testpassword"
}
}Endpoint to Claim the user.
200: with above response
500: Internal Service Error
{
"topic":"test.some.topic",
"user":"testusername",
"role":"Producer"
}{
"id":"901125d6-0e47-43c7-92d5-a9d8229c7f0f",
}
Create Acl for a topic, user and role
200: Ok with above response
500: Internal Server Error
{
"topic":"test.some.topic.1",
"schema":{
"name":"testSchema"
}
}{
"topic":"test.some.topic.1",
"schema":{
"name":"testSchema"
}
}Create Acl for a topic, user and role
200: Ok with above response
400: Invalid Schema Name or Version
500: Internal Server Error
{
"topic":"test.some.topic.3",
"keyType":"AVRO", //"None", "String" Or "Avro"(Schema is required for Avro)
"schema":{
"name":"testSchema"
}
}Create topic key mapping
200: Ok with empty response
400: Invalid Request. Invalid Schema or schema not provided for Avro key type
500: Internal Server Error
None
Delete the Acl using Acl id.
200
None
Deletes the topic. Also Deletes Acls, Topic Schema Mappings, Topic Key Mappings.
200