Exception Server has invalid Kerberos principal when try to connect using reverseDNS hostname by IP resolving

[smatytsin]

When I try to create HDFS FileSystem object using Kerberos auth, there is an exception
java.lang.IllegalArgumentException: Server has invalid Kerberos principal: nn/ex1.example.com@EXAMPLE.COM, expecting: nn/ex1.example.com.@EXAMPLE.COM
at org.apache.hadoop.security.SaslRpcClient.getServerPrincipal(SaslRpcClient.java:337)
at org.apache.hadoop.security.SaslRpcClient.createSaslClient(SaslRpcClient.java:234)
at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:160)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:627)
at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:421)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:814)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:810)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:810)
... 73 common frames omitted

The reason of exception:
In org.apache.hadoop.security.SaslRpcClient#getServerPrincipal there is verifying process of NameNode server principal and principal which gets from config. In config principal keeps in format nn/_HOST@EXAMPLE.COM where _HOST is a placeholder for real NameNode host
Then there is replacing of _HOST placeholder with real host name of name node. And real host name gets as result of InetAddress.getCanonicalHostName() method (look /Users/a16689075/.m2/repository/org/apache/hadoop/hadoop-common/3.1.3/hadoop-common-3.1.3-sources.jar!/org/apache/hadoop/security/SecurityUtil.java:211)

But if there is some reverse DNS in infrastructure where I run this code, reverse DNS resolves host by IP and return host with dot at the end. So InetAddress.getCanonicalHostName() returns real host of NameNode but with dot at the end: nn/ex1.example.com. (the same is actual when you execute for example nslookup ip-addr in command line)
So principal from config after placeholder replacement looks like nn/ex1.example.com.@EXAMPLE.COM (where hostname keeps dot at the end)

Then there is checking if nameNode server pricipal is equals with principal from config. And of course they are different because of dot exists at the end of hostname in config principal

At the same time we can look at constructor sun.security.krb5.PrincipalName#PrincipalName(java.lang.String, int, java.lang.String where we can see similar logic for processing InetAddress.getCanonicalHostName();
If InetAddress.getCanonicalHostName() received host with dot at the end, there it cuts this dot and keeps only hostname without dot