Unhelpful error when EMR cluster doesn't start

[zroadhouse-rmn]

This is for a start_datastore action on emr_engine.

Error message was:

'NoneType' object has no attribute 'state'


Traceback (most recent call last):
  File "emr.py", line 71, in run
    handler(self, datastore, action)
  File "/src/python/dart/engine/emr/actions/start_datastore.py", line 59, in start_datastore
    raise Exception('cluster_id=%s not in WAITING state, but in state: %s' % (cluster_id, cluster.state))
AttributeError: 'NoneType' object has no attribute 'state'

Looking at the AWS console, the actual error was

Subnet is required : The specified instance type m4.large can only be used in a VPC.

[zroadhouse-rmn]

Ran into this in another case where the actual error was

On the master instance (i-099c21d245d50508e), bootstrap action 3 returned a non-zero return code

[maybeiambatman]

@dannymcpherson we are considering spinning up the emr clusters in a vpc going forward. Did you have any particular reason for not spinning them up in a vpc before?

[dannymcpherson]

Dart is assumed to be deployed in a VPC, and we had never launched an EMR cluster outside of the VPC afaik. How were you able to deploy an EMR cluster in EC2 classic?

[maybeiambatman]

Looks like we don't spin up emrs in vpc in dart. https://github.com/RetailMeNotSandbox/dart/blob/master/src/python/dart/engine/emr/actions/start_datastore.py#L69

We will bake this change in under this issue.

[dannymcpherson]

Great catch. That was definitely an oversight!