switch all worker to private dns

[pragnesh]

This PR makes the following changes:

switch all worker to private dns

I tested this PR by...
launching cluster where i am seeing slow startup(flintrock needed bend-aid to start cluster).
After this change cluster start in first try.

Fixes #157.
Fixes #129.

When instance start on aws, aws update public dns host from public ip -> private ip,
This create issue with cluster communication. If you switch all worker to private dns name,
Cluster does not have any issue in starting which fix band-aid required for starting hdfs cluster and spark cluster.

[nchammas]

Hey @pragnesh! Somehow I missed this PR and am only just seeing it now. I'm sorry about that.

This PR seems to add private VPC support by removing public VPC support. Or are you saying this PR patches Flintrock's behavior even when it's launching into a public VPC?

There is another PR attempting to add private VPC support: #296. If that's what you're interested in, I suggest we close this PR in favor of that one and consolidate our efforts there.

[pragnesh]

Hi @nchammas, This is meant for stability, i am using this change since 1 year and this is fix for issue describe in #157 , #129.

[pragnesh]

This PR has nothing do with private VPC support.

[nchammas]

OK, thanks for clarifying. I will test this out and see if it works for me.

To be clear, do you occasionally run into the issues reported in #129 and #157 when running on a recent version of Flintrock (like 0.9.0 or 1.0.0)? If you haven't tested out a recent version of Flintrock, could you do that and confirm the issue is still there?

[pragnesh]

Hi @nchammas , i was actully facing issue #198, while investing and fixing i found that issue #157 , #129 also same fundamental problem.

I am using this change rebased on flintrock master. I rebased my change regulary after flintrock version change.

[pragnesh]

This PR only change cluster internal communication address. Rest of thing remain as it is.

[pragnesh]

Hi @nchammas, by the way, i have also added alluxio support to flintrock and waiting to raise PR, but unfortunately alluxio related change depend on current PR.

I think alluxio is very useful for spark user.

[nchammas]

Tested this out and it seems to work well. End-to-end tests all pass, too. Nice work @pragnesh.

[pragnesh]

thanks @nchammas

[pragnesh]

@nchammas does flintrock project intrested in Alluxio service ? I am using it sometime with flintrock and if there is interest, I can work on it, here is my Alluxio change

[nchammas]

Thank you for sharing this change. I'd like to keep the services bundled with Flintrock to a bare minimum to keep the maintenance burden low, so I wouldn't want to add Alluxio at this time. I wish I could offer a neat plugin system that lets users contribute new services like Alluxio without having to bake them directly into Flintrock.

Perhaps the one new service I'd consider adding directly into Flintrock is Jupyter (#23) because of how widely used it is, but otherwise I think Flintrock will support only Spark and Hadoop for the foreseeable future.

[pragnesh]

I understand @nchammas, no issue.