cm_shell: feature enhancement to show configs #54

kmizumar · 2017-01-07T16:18:56Z

Added a new feature to show configuration parameters in Cloudera Manager Shell.

Welcome to the Cloudera Manager Console
Select a cluster using 'show clusters' and 'use'
cloudera> use Cluster 1
Connected to Cluster 1
Cluster 1> show configs HDFS-1
+---------+----------------------------------------------+----------------------------------------------------+
| SERVICE | NAME                                         | VALUE                                              |
+---------+----------------------------------------------+----------------------------------------------------+
| HDFS-1  | dfs_data_transfer_protection                 | privacy                                            |
| HDFS-1  | hdfs_hadoop_ssl_enabled                      | true                                               |
| HDFS-1  | hadoop_security_authorization                | true                                               |
| HDFS-1  | zookeeper_service                            | ZOOKEEPER-1                                        |
| HDFS-1  | ssl_server_keystore_password                 | cloudera                                           |
| HDFS-1  | hdfs_sentry_sync_enable                      | true                                               |
| HDFS-1  | hadoop_security_authentication               | kerberos                                           |
| HDFS-1  | kms_service                                  | KMS-1                                              |
| HDFS-1  | dfs_datanode_read_shortcircuit               | true                                               |
| HDFS-1  | dfs_namenode_acls_enabled                    | true                                               |
| HDFS-1  | dfs_datanode_hdfs_blocks_metadata_enabled    | true                                               |
| HDFS-1  | ssl_server_keystore_keypassword              | cloudera                                           |
| HDFS-1  | hdfs_blocks_with_corrupt_replicas_thresholds | {"warning":"2.5","critical":"3.0"}                 |
| HDFS-1  | dfs_encrypt_data_transfer_algorithm          | AES/CTR/NoPadding                                  |
| HDFS-1  | ssl_server_keystore_location                 | /etc/cdep-ssl-conf/CA_STANDARD/hadoop-keystore.jks |
| HDFS-1  | ssl_client_truststore_password               | cloudera                                           |
| HDFS-1  | dfs_encrypt_data_transfer                    | true                                               |
| HDFS-1  | hadoop_rpc_protection                        | privacy                                            |
| HDFS-1  | dfs_ha_fencing_methods                       | shell(true)                                        |
| HDFS-1  | dfs_replication                              | 3                                                  |
| HDFS-1  | ssl_client_truststore_location               | /etc/cdep-ssl-conf/CA_STANDARD/truststore.jks      |
+---------+----------------------------------------------+----------------------------------------------------+
Cluster 1>

(cherry picked from commit d6c8398) (cherry picked from commit 4a47d5491e9ad682a8ee2769f262f014ebcd1cb5)

Added list_dfs_services to endpoints/cluster.py's ApiCluster, to match API addition in OPSAPS-31495. Testing: - new unit test - verified with manual test against CM via a simple test script: --- from cm_api.api_client import ApiResource api=ApiResource("nightly-1.vpc.cloudera.com", username="foo", password="bar") cluster=api.get_cluster("Cluster 1") apiList=cluster.list_dfs_services(view="export") print ("%s" %(apiList)) --- (cherry picked from commit adef14d) (cherry picked from commit e40c6259f119cc9067db9d03b092ad0e72ac5f14)

in the exported template Added cardinality attribute to host template in cluster template (cherry picked from commit 0c8595d) (cherry picked from commit 5fd588f77b0300c2bc25e6ebf508f66e338d1f22)

(cherry picked from commit e6bc828) (cherry picked from commit 44c479034092ff51c6f206f4101a156b086f7757)

- Fixed the api (cherry picked from commit 2151df0) (cherry picked from commit 47531a5d9c4670aac1c8089d4c76774ec5c5e73a)

Nothing special. It's been MIA for a while. >>> from cm_api.api_client import ApiResource >>> api=ApiResource("nightly-1.gce.cloudera.com", username="admin", password="admin") >>> cluster=api.get_cluster("Cluster 1") >>> cmd = cluster.pools_refresh() (cherry picked from commit c385539) (cherry picked from commit 3b6cb12bb0ff730d14bf847d3b4c2f96a4c39d17)

Since it's actually been around for a while. (cherry picked from commit ff09be8) (cherry picked from commit 8846594073fe12823afb3ef17757dd8cc823debd)

- enable ClouderaShell.__init__ method - docstring should be enclosed with triple quotes

List non-default configuration parameters: > show configs <service name> List all configuration parameters including default values: > show full_configs <service name>

kmizumar · 2017-01-10T04:01:08Z

Modified to show role type configurations:

Cluster 1> show configs HDFS-1
+---------+--------------------+---------------------------------------------------------------------------+--------------------------------------------------------------------------+
| SERVICE | ROLETYPE           | NAME                                                                      | VALUE                                                                    |
+---------+--------------------+---------------------------------------------------------------------------+--------------------------------------------------------------------------+
| HDFS-1  |                    | dfs_data_transfer_protection                                              | privacy                                                                  |
| HDFS-1  |                    | hdfs_hadoop_ssl_enabled                                                   | true                                                                     |
| HDFS-1  |                    | hadoop_security_authorization                                             | true                                                                     |
| HDFS-1  |                    | zookeeper_service                                                         | ZOOKEEPER-1                                                              |
| HDFS-1  |                    | ssl_server_keystore_password                                              | cloudera                                                                 |
| HDFS-1  |                    | hdfs_sentry_sync_enable                                                   | true                                                                     |
| HDFS-1  |                    | hadoop_security_authentication                                            | kerberos                                                                 |
| HDFS-1  |                    | kms_service                                                               | KMS-1                                                                    |
| HDFS-1  |                    | dfs_datanode_read_shortcircuit                                            | true                                                                     |
| HDFS-1  |                    | dfs_namenode_acls_enabled                                                 | true                                                                     |
| HDFS-1  |                    | dfs_datanode_hdfs_blocks_metadata_enabled                                 | true                                                                     |
| HDFS-1  |                    | ssl_server_keystore_keypassword                                           | cloudera                                                                 |
| HDFS-1  |                    | hdfs_blocks_with_corrupt_replicas_thresholds                              | {"warning":"2.5","critical":"3.0"}                                       |
| HDFS-1  |                    | dfs_encrypt_data_transfer_algorithm                                       | AES/CTR/NoPadding                                                        |
| HDFS-1  |                    | ssl_server_keystore_location                                              | /etc/cdep-ssl-conf/CA_STANDARD/hadoop-keystore.jks                       |
| HDFS-1  |                    | ssl_client_truststore_password                                            | cloudera                                                                 |
| HDFS-1  |                    | dfs_encrypt_data_transfer                                                 | true                                                                     |
| HDFS-1  |                    | hadoop_rpc_protection                                                     | privacy                                                                  |
| HDFS-1  |                    | dfs_ha_fencing_methods                                                    | shell(true)                                                              |
| HDFS-1  |                    | dfs_replication                                                           | 3                                                                        |
| HDFS-1  |                    | ssl_client_truststore_location                                            | /etc/cdep-ssl-conf/CA_STANDARD/truststore.jks                            |
| HDFS-1  | FAILOVERCONTROLLER | log_directory_free_space_percentage_thresholds                            | {"critical":"10.0","warning":"20.0"}                                     |
| HDFS-1  | FAILOVERCONTROLLER | heap_dump_directory_free_space_percentage_thresholds                      | {"critical":"10.0","warning":"20.0"}                                     |
| HDFS-1  | FAILOVERCONTROLLER | heap_dump_directory_free_space_absolute_thresholds                        | {"critical":"never","warning":"never"}                                   |
| HDFS-1  | FAILOVERCONTROLLER | log_directory_free_space_absolute_thresholds                              | {"critical":"never","warning":"never"}                                   |
| HDFS-1  | JOURNALNODE        | log_directory_free_space_percentage_thresholds                            | {"critical":"10.0","warning":"20.0"}                                     |
| HDFS-1  | JOURNALNODE        | journalnode_edits_directory_free_space_percentage_thresholds              | {"critical":"10.0","warning":"20.0"}                                     |
| HDFS-1  | JOURNALNODE        | log_directory_free_space_absolute_thresholds                              | {"critical":"never","warning":"never"}                                   |
| HDFS-1  | JOURNALNODE        | heap_dump_directory_free_space_percentage_thresholds                      | {"critical":"10.0","warning":"20.0"}                                     |
| HDFS-1  | JOURNALNODE        | dfs_journalnode_edits_dir                                                 | /dfs/jn                                                                  |
| HDFS-1  | JOURNALNODE        | heap_dump_directory_free_space_absolute_thresholds                        | {"critical":"never","warning":"never"}                                   |
| HDFS-1  | JOURNALNODE        | journalnode_edits_directory_free_space_absolute_thresholds                | {"critical":"never","warning":"never"}                                   |
| HDFS-1  | HTTPFS             | log_directory_free_space_percentage_thresholds                            | {"critical":"10.0","warning":"20.0"}                                     |
| HDFS-1  | HTTPFS             | httpfs_https_keystore_file                                                | /etc/cdep-ssl-conf/CA_STANDARD/httpfs-keystore.jks                       |
| HDFS-1  | HTTPFS             | log_directory_free_space_absolute_thresholds                              | {"critical":"never","warning":"never"}                                   |
| HDFS-1  | HTTPFS             | httpfs_https_keystore_password                                            | cloudera                                                                 |
| HDFS-1  | HTTPFS             | httpfs_https_truststore_password                                          | cloudera                                                                 |
| HDFS-1  | HTTPFS             | httpfs_https_truststore_file                                              | /etc/cdep-ssl-conf/CA_STANDARD/truststore.jks                            |
| HDFS-1  | HTTPFS             | heap_dump_directory_free_space_percentage_thresholds                      | {"critical":"10.0","warning":"20.0"}                                     |
| HDFS-1  | HTTPFS             | heap_dump_directory_free_space_absolute_thresholds                        | {"critical":"never","warning":"never"}                                   |
| HDFS-1  | HTTPFS             | httpfs_use_ssl                                                            | true                                                                     |
| HDFS-1  | SECONDARYNAMENODE  | log_directory_free_space_percentage_thresholds                            | {"critical":"10.0","warning":"20.0"}                                     |
| HDFS-1  | SECONDARYNAMENODE  | secondarynamenode_checkpoint_directories_free_space_absolute_thresholds   | {"critical":"never","warning":"never"}                                   |
| HDFS-1  | SECONDARYNAMENODE  | log_directory_free_space_absolute_thresholds                              | {"critical":"never","warning":"never"}                                   |
| HDFS-1  | SECONDARYNAMENODE  | fs_checkpoint_dir_list                                                    | /dfs/snn                                                                 |
| HDFS-1  | SECONDARYNAMENODE  | heap_dump_directory_free_space_percentage_thresholds                      | {"critical":"10.0","warning":"20.0"}                                     |
| HDFS-1  | SECONDARYNAMENODE  | secondarynamenode_checkpoint_directories_free_space_percentage_thresholds | {"critical":"10.0","warning":"20.0"}                                     |
| HDFS-1  | SECONDARYNAMENODE  | dfs_secondary_http_port                                                   | 20201                                                                    |
| HDFS-1  | SECONDARYNAMENODE  | heap_dump_directory_free_space_absolute_thresholds                        | {"critical":"never","warning":"never"}                                   |
| HDFS-1  | SECONDARYNAMENODE  | secondary_namenode_java_heapsize                                          | 606076928                                                                |
| HDFS-1  | SECONDARYNAMENODE  | dfs_secondary_https_port                                                  | 20202                                                                    |
| HDFS-1  | BALANCER           | balancer_java_heapsize                                                    | 606076928                                                                |
| HDFS-1  | DATANODE           | log_directory_free_space_percentage_thresholds                            | {"critical":"10.0","warning":"20.0"}                                     |
| HDFS-1  | DATANODE           | dfs_datanode_https_port                                                   | 20004                                                                    |
| HDFS-1  | DATANODE           | datanode_data_directories_free_space_absolute_thresholds                  | {"critical":"never","warning":"never"}                                   |
| HDFS-1  | DATANODE           | heap_dump_directory_free_space_absolute_thresholds                        | {"critical":"never","warning":"never"}                                   |
| HDFS-1  | DATANODE           | dfs_datanode_data_dir_perm                                                | 700                                                                      |
| HDFS-1  | DATANODE           | dfs_datanode_du_reserved                                                  | 10736126771                                                              |
| HDFS-1  | DATANODE           | dfs_datanode_http_port                                                    | 1006                                                                     |
| HDFS-1  | DATANODE           | log_directory_free_space_absolute_thresholds                              | {"critical":"never","warning":"never"}                                   |
| HDFS-1  | DATANODE           | datanode_java_heapsize                                                    | 1073741824                                                               |
| HDFS-1  | DATANODE           | heap_dump_directory_free_space_percentage_thresholds                      | {"critical":"10.0","warning":"20.0"}                                     |
| HDFS-1  | DATANODE           | datanode_config_safety_valve                                              |                                                                          |
|         |                    |                                                                           |         <property>                                                       |
|         |                    |                                                                           |           <name>dfs.thrift.datanode.address</name>                       |
|         |                    |                                                                           |           <value>0.0.0.0:20005</value>                                   |
|         |                    |                                                                           |           <description>                                                  |
|         |                    |                                                                           |             The address where the Thrift datanode server will listen to. |
|         |                    |                                                                           |             If the port is 0 then the server will start on a free port.  |
|         |                    |                                                                           |           </description>                                                 |
|         |                    |                                                                           |         </property>                                                      |
| HDFS-1  | DATANODE           | dfs_datanode_max_locked_memory                                            | 1404043264                                                               |
| HDFS-1  | DATANODE           | dfs_data_dir_list                                                         | /dfs/dn                                                                  |
| HDFS-1  | DATANODE           | dfs_datanode_failed_volumes_tolerated                                     | 0                                                                        |
| HDFS-1  | DATANODE           | dfs_datanode_ipc_port                                                     | 20001                                                                    |
| HDFS-1  | DATANODE           | datanode_data_directories_free_space_percentage_thresholds                | {"critical":"10.0","warning":"20.0"}                                     |
| HDFS-1  | DATANODE           | dfs_datanode_port                                                         | 20002                                                                    |
| HDFS-1  | NFSGATEWAY         | log_directory_free_space_percentage_thresholds                            | {"critical":"10.0","warning":"20.0"}                                     |
| HDFS-1  | NFSGATEWAY         | log_directory_free_space_absolute_thresholds                              | {"critical":"never","warning":"never"}                                   |
| HDFS-1  | NFSGATEWAY         | nfsgateway_dump_directory_free_space_percentage_thresholds                | {"critical":"10.0","warning":"20.0"}                                     |
| HDFS-1  | NFSGATEWAY         | heap_dump_directory_free_space_percentage_thresholds                      | {"critical":"10.0","warning":"20.0"}                                     |
| HDFS-1  | NFSGATEWAY         | nfsgateway_dump_directory_free_space_absolute_thresholds                  | {"critical":"never","warning":"never"}                                   |
| HDFS-1  | NFSGATEWAY         | heap_dump_directory_free_space_absolute_thresholds                        | {"critical":"never","warning":"never"}                                   |
| HDFS-1  | NAMENODE           | log_directory_free_space_percentage_thresholds                            | {"critical":"10.0","warning":"20.0"}                                     |
| HDFS-1  | NAMENODE           | namenode_data_directories_free_space_percentage_thresholds                | {"critical":"10.0","warning":"20.0"}                                     |
| HDFS-1  | NAMENODE           | dfs_name_dir_list                                                         | /dfs/nn                                                                  |
| HDFS-1  | NAMENODE           | log_directory_free_space_absolute_thresholds                              | {"critical":"never","warning":"never"}                                   |
| HDFS-1  | NAMENODE           | dfs_namenode_servicerpc_address                                           | 8022                                                                     |
| HDFS-1  | NAMENODE           | heap_dump_directory_free_space_percentage_thresholds                      | {"critical":"10.0","warning":"20.0"}                                     |
| HDFS-1  | NAMENODE           | dfs_namenode_handler_count                                                | 30                                                                       |
| HDFS-1  | NAMENODE           | dfs_http_port                                                             | 20101                                                                    |
| HDFS-1  | NAMENODE           | dfs_namenode_service_handler_count                                        | 30                                                                       |
| HDFS-1  | NAMENODE           | heap_dump_directory_free_space_absolute_thresholds                        | {"critical":"never","warning":"never"}                                   |
| HDFS-1  | NAMENODE           | namenode_data_directories_free_space_absolute_thresholds                  | {"critical":"never","warning":"never"}                                   |
| HDFS-1  | NAMENODE           | namenode_java_heapsize                                                    | 606076928                                                                |
| HDFS-1  | NAMENODE           | dfs_https_port                                                            | 20102                                                                    |
| HDFS-1  | GATEWAY            | hdfs_client_config_safety_valve                                           |                                                                          |
| HDFS-1  | GATEWAY            | dfs_client_use_trash                                                      | true                                                                     |
+---------+--------------------+---------------------------------------------------------------------------+--------------------------------------------------------------------------+
Cluster 1>

cloudera-hudson and others added 16 commits February 12, 2016 17:07

[CLOUDERA-BUILD] Bump up version number to 5.7.0

cad7434

[cm_api] OPSAPS-22366 Python API changes for cluster template import

2ea63bf

(cherry picked from commit d6c8398) (cherry picked from commit 4a47d5491e9ad682a8ee2769f262f014ebcd1cb5)

[scm] OPSAPS-31656 ClusterTemplates: Cardinality needs to be described

b5b56c6

in the exported template Added cardinality attribute to host template in cluster template (cherry picked from commit 0c8595d) (cherry picked from commit 5fd588f77b0300c2bc25e6ebf508f66e338d1f22)

[yamon] OPSAPS-32606 Add the new fields for YARN apps

70c1716

(cherry picked from commit e6bc828) (cherry picked from commit 44c479034092ff51c6f206f4101a156b086f7757)

[scm] OPSAPS-32644 addRepositories argument is not present in python API

1ff5863

- Fixed the api (cherry picked from commit 2151df0) (cherry picked from commit 47531a5d9c4670aac1c8089d4c76774ec5c5e73a)

[api] OPSAPS-32658. Support pools refresh starting v6

13dc996

Since it's actually been around for a while. (cherry picked from commit ff09be8) (cherry picked from commit 8846594073fe12823afb3ef17757dd8cc823debd)

[java] Updated with v12 java source

1591b82

[java] Updated with v13 java source

79dd937

[py] replace ht(09) with appropriate number of sp(20)

785f122

[py] fix typo, long option --seperator is changed to --separator

7a2e899

[py] minor tweaks

12a031d

- enable ClouderaShell.__init__ method - docstring should be enclosed with triple quotes

[py] check service existence in status command

5e13f92

[py] add `show configs' command to cm_shell

577b8ff

List non-default configuration parameters: > show configs <service name> List all configuration parameters including default values: > show full_configs <service name>

[py] modified `show configs' to show role type configurations

f171478

mthak force-pushed the master branch from 6fea167 to 16b4bc5 Compare April 17, 2017 20:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cm_shell: feature enhancement to show configs #54

cm_shell: feature enhancement to show configs #54

kmizumar commented Jan 7, 2017

kmizumar commented Jan 10, 2017

cm_shell: feature enhancement to show configs #54

Are you sure you want to change the base?

cm_shell: feature enhancement to show configs #54

Conversation

kmizumar commented Jan 7, 2017

kmizumar commented Jan 10, 2017