Circular dependency between resources

[tuliomourazup]

Error: The CloudFormation template is invalid: Circular dependency between resources: [ApiGatewayMethodCategoriesCategoryidVarOptions, ApiGatewayMethodHospitalsHospitalidVarMedicinesGet, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarEquipmentsEquipmentidVarOptions, ApiGatewayMethodHospitalsHospitalidVarCleanOptions, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarStaffsGet, ApiGatewayMethodUsersListallGet, ApiGatewayResourceHospitalsHospitalidVarEpi, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarEquipmentsOptions, ApiGatewayMethodHospitalsPost, ApiGatewayResourceMedicinesMedicineidVarVolumesMedicinevolumeidVar, ApiGatewayMethodHospitalsHospitalidVarMedicinesMedicineidVarPost, ApiGatewayMethodUsersHospitalidVarPut, ApiGatewayMethodMedicinesMedicineidVarVolumesMedicinevolumeidVarOptions, ApiGatewayMethodFederationDashstatesFederationstateidVarCitiesGet, ApiGatewayMethodTermStatusGet, ApiGatewayMethodHospitalsHospitalidVarPendenciesAreasMovementsOptions, ApiGatewayMethodMedicinesMedicineidVarVolumesGet, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarPut, ApiGatewayResourceHospitalsHospitalidVar, ApiGatewayMethodUsersHospitalidVarUserGet, ApiGatewayResourceHospitalsHospitalidVarMedicinesMedicineidVarVolumes, ApiGatewayMethodUsersByemailOptions, ApiGatewayMethodHospitalsHospitalidVarEpiEpiidVarPost, ApiGatewayAuthorizer, ApiGatewayMethodCategoriesCategoryidVarGet, ApiGatewayMethodApidocsGet, ApiGatewayResourceHospitalsHospitalidVarAreasAreaidVarMovements, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarMovementsDelete, ApiGatewayMethodHospitalsHospitalidVarOptions, ApiGatewayResourceHospitalsHospitalidVarEquipments, ApiGatewayMethodTableauOptions, ApiGatewayResourceHospitalsHospitalidVarMedicinesMedicineidVarInactivate, ApiGatewayMethodHospitalsProfileOptions, ApiGatewayResourceMedicinesMedicineidVarVolumes, ApiGatewayMethodHospitalsHospitalidVarEquipmentsMovementsGet, ApiGatewayMethodUsersHospitalidVarGet, ApiGatewayMethodUserDashmanagementDisassociateDashuserHospitalidVarPut, ApiGatewayResourceHospitalsHospitalidVarAreasAreaidVar, ApiGatewayMethodHospitalsHospitalidVarInactivatePut, ApiGatewayMethodUsersGet, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarGet, ApiGatewayMethodMedicinesHospitalsGet, ApiGatewayResourceHospitalsHospitalidVarAreasAreaidVarStaffs, ApiGatewayMethodHospitalsHospitalidVarInactivateDashuserOptions, ApiGatewayMethodHospitalsHospitalidVarEquipmentsEquipmentidVarMovementsPost, ApiGatewayResourceHospitalsHospitalidVarAreasAreaidVarEquipmentsEquipmentidVar, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarEquipmentsGet, ApiGatewayMethodProfilesOptions, ApiGatewayResourceFederationDashstatesFederationstateidVarCities, ApiGatewayResourceHospitalsProfile, ApiGatewayResourceUserDashmanagementResetpass, ApiGatewayMethodFederationDashstatesOptions, ApiGatewayMethodHospitalsHospitalidVarMedicinesMedicineidVarVolumesOptions, ApiGatewayMethodUsersLoggeduserinfoOptions, ApiGatewayResourceHospitals, ApiGatewayMethodHospitalsHospitalidVarMedicinesMedicineidVarOptions, ApiGatewayResourceHospitalsHospitalidVarAreasAreaidVarShiftsShiftidVar, ApiGatewayResourceCategories, ApiGatewayResourceMedicinesHospitals, ApiGatewayResourceProfiles, ApiGatewayMethodHospitalsHospitalidVarPendenciesMedicinesMovementsGet, ApiGatewayMethodUsersPut, ApiGatewayMethodHospitalsHospitalidVarPendenciesEquipmentsMovementsOptions, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarEquipmentsEquipmentidVarGet, ApiGatewayResourceHospitalsHospitalidVarEpiEpiidVar, ApiGatewayResourceHospitalsHospitalidVarMedicinesMedicineidVar, ApiGatewayResourceHospitalsListall, ApiGatewayMethodMedicinesHospitalsOptions, ApiGatewayMethodHospitalsHospitalidVarEpiGet, ApiGatewayResourceHospitalsHospitalidVarClean, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarMovementsOptions, ApiGatewayMethodHospitalsProfileGet, ApiGatewayResourceUsersHospitalidVarUser, ApiGatewayResourceHospitalsHospitalidVarMedicines, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarDelete, ApiGatewayMethodHospitalsHospitalidVarPendenciesEpiMovementsOptions, APINestedStack, ApiGatewayMethodUsersLoggeduserinfoGet, ApiGatewayMethodHospitalsHospitalidVarEquipmentsGet, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarEquipmentsEquipmentidVarPost, ApiGatewayMethodCategoriesGet, ApiGatewayMethodUserDashmanagementInactiveOptions, ApiGatewayMethodProfilesTypesGet, ApiGatewayMethodHospitalsListallGet, ApiGatewayResourceFederationDashstates, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarOptions, ApiGatewayMethodUsersPost, ApiGatewayResourceHospitalsHospitalidVarAreasAreaidVarShifts, ApiGatewayResourceUsersHospitalidVar, ApiGatewayResourceUsersListall, ApiGatewayMethodHospitalsHospitalidVarPendenciesStaffsMovementsGet, ApiGatewayResourceProfilesTypes, ApiGatewayMethodUserDashmanagementAssociateDashuserHospitalidVarOptions, ApiGatewayResourceUserDashmanagement, ApiGatewayResourceMedicinesMedicineidVar, ApiGatewayMethodHospitalsHospitalidVarPendenciesMedicinesMovementsOptions, ApiGatewayResourceUserDashmanagementAssociateDashuserHospitalidVar, ApiGatewayResourceHospitalsHospitalidVarInactivateDashuser, ApiGatewayMethodFederationDashstatesFederationstateidVarCitiesOptions, ApiGatewayMethodUserDashmanagementOptions, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarStaffsOptions, ApiGatewayMethodProfilesTypesOptions, ApiGatewayMethodHospitalsHospitalidVarEquipmentsEquipmentidVarOptions, ApiGatewayResourceHospitalsHospitalidVarAreas, ApiGatewayMethodMedicinesMedicineidVarVolumesOptions, ApiGatewayMethodTableauGet, ApiGatewayMethodHospitalsHospitalidVarEquipmentsEquipmentidVarMovementsOptions, ApiGatewayMethodApidocsOptions, ApiGatewayMethodHospitalsHospitalidVarEquipmentsMovementsOptions, ApiGatewayMethodHospitalsHospitalidVarAreasOptions, ApiGatewayMethodHospitalsHospitalidVarMedicinesMedicineidVarInactivatePut, ApiGatewayResourceUserDashmanagementInactive, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarShiftsShiftidVarOptions, ApiLambdaPermissionApiGateway, ApiGatewayMethodSystemStatusTpsOptions, ApiGatewayMethodHospitalsHospitalidVarAreasPost, ApiGatewayMethodUsersByemailGet, ApiGatewayResourceUsersByemail, ApiGatewayMethodHospitalsHospitalidVarCleanDelete, ApiGatewayResourceHospitalsHospitalidVarAreasAreaidVarEquipments, ApiGatewayMethodFederationDashstatesGet, ApiNestedStack, ApiGatewayMethodHospitalsOptions, ApiGatewayMethodHospitalsHospitalidVarMedicinesOptions, ApiGatewayMethodUsersHospitalidVarUserOptions, ApiGatewayMethodHospitalsHospitalidVarPendenciesEpiMovementsGet, ApiGatewayMethodUsersListallOptions, ApiGatewayMethodHospitalsHospitalidVarMedicinesMedicineidVarVolumesPost, ApiGatewayMethodHospitalsHospitalidVarAreasGet, ApiGatewayMethodProfilesGet, ApiGatewayResourceUsers, ApiGatewayMethodTermStatusOptions, ApiGatewayMethodHospitalsHospitalidVarPendenciesEquipmentsMovementsGet, ApiGatewayResourceFederationDashstatesFederationstateidVar, ApiGatewayMethodHospitalsHospitalidVarPendenciesStaffsMovementsOptions, ApiGatewayResourceUserDashmanagementAssociateDashuser, ApiGatewayMethodTermGet, ApiGatewayResourceSystemStatus, ApiGatewayResourceTerm, ApiGatewayMethodUsersHospitalidVarOptions, ApiGatewayMethodHospitalsHospitalidVarGet, ApiGatewayResourceTableau, ApiGatewayMethodCategoriesOptions, ApiGatewayMethodHospitalsGet, ApiGatewayMethodUserDashmanagementDisassociateDashuserHospitalidVarOptions, ApiGatewayMethodTermPost, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarShiftsShiftidVarPost, ApiGatewayMethodMedicinesMedicineidVarVolumesMedicinevolumeidVarGet, ApiGatewayResourceApidocs, ApiGatewayMethodUserDashmanagementInactiveDelete, ApiGatewayMethodHospitalsHospitalidVarInactivateOptions, ApiGatewayRestApi, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarMovementsPut, ApiGatewayMethodUserDashmanagementResetpassOptions, ApiGatewayMethodMedicinesMedicineidVarGet, ApiGatewayMethodUserDashmanagementAssociateDashuserHospitalidVarPost, ApiGatewayMethodHospitalsHospitalidVarEpiOptions, ApiGatewayMethodHospitalsHospitalidVarEpisCleanOptions, ApiGatewayMethodSystemStatusTpsGet, ApiGatewayResourceSystemStatusTps, ApiGatewayMethodHospitalsHospitalidVarEquipmentsEquipmentidVarPut, ApiGatewayResourceCategoriesCategoryidVar, ApiGatewayMethodHospitalsHospitalidVarEpisCleanDelete, ApiGatewayResourceUserDashmanagementDisassociateDashuserHospitalidVar, ApiGatewayResourceHospitalsHospitalidVarInactivate, ApiGatewayResourceTermStatus, ApiGatewayMethodUserDashmanagementResetpassPut, ApiGatewayMethodUserDashmanagementPost, ApiGatewayResourceHospitalsHospitalidVarEquipmentsEquipmentidVar, ApiGatewayMethodMedicinesMedicineidVarOptions, ApiGatewayMethodHospitalsHospitalidVarEpiEpiidVarOptions, ApiGatewayMethodHospitalsHospitalidVarPut, ApiGatewayMethodHospitalsListallOptions, ApiGatewayMethodHospitalsHospitalidVarMedicinesMedicineidVarInactivateOptions, ApiGatewayResourceUsersLoggeduserinfo, ApiGatewayResourceMedicines, ApiGatewayMethodUsersOptions, ApiGatewayMethodHospitalsHospitalidVarPendenciesAreasMovementsGet, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarMovementsGet, ApiGatewayResourceSystem, ApiGatewayResourceUserDashmanagementDisassociateDashuser, ApiGatewayMethodHospitalsHospitalidVarEquipmentsOptions, ApiGatewayMethodTermOptions, ApiGatewayMethodHospitalsHospitalidVarEpiEpiidVarGet, ApiGatewayMethodHospitalsHospitalidVarInactivateDashuserPut]

[lepirlouit]

I had this error when a resource has the same name/prefix as a function.
I fixed this by changing the name of the resource

[jweyrich]

We had the same problem.
We added a new lambda function and it caused the circular dependency issue.
Renaming the new lambda function fixed the problem.

[gautampambhar]

I am getting a similar issue. The plugin worked fine on the first try. but after getting the following warning -

Serverless: Recoverable error occurred (TooManyRequestsException: Rate exceeded

I added resourceConcurrency(set to 10) to my splitStacks configuration to avoid the above warning. but upon redeploy, I am getting the following error

ServerlessError: The CloudFormation template is invalid: ValidationError: Circular dependency between resources:[GetAllUsersLambdaFunction,....

Below is the configuration I am using with this plugin. is there any workaround available to this?

custom:
  splitStacks:
      nestedStackCount: 2 
      perFunction: false
      perType: false
      perGroupFunction: true
      resourceConcurrency: 10

[sweepy84]

@gautampambhar - I just wasted a day on this. For some reason the resourceConcurrency features adds references to dependsOn field as soon as you move resources to a nested stack, which is what perGroupFunction does. This is a clear bug to me, about to raise an issue for it.

[sweepy84]

@gautampambhar #188

[turakvlad]

We have the same issue as described here and #188.

If we have resourceConcurrency in our config, it does not allow us to deploy the stack for the first time (completely new deployment from scratch).

On the other hand, stackConcurrency works for the first deployment but does not work for subsequent ones.

To better show these issues, please take a look at the cases below.

The config below is not working if we want to deploy the stack for the first time:

  splitStacks:
    perFunction: true
    perType: false
    perGroupFunction: false
    stackConcurrency: 5
    resourceConcurrency: 10

We have this error:

Circular dependency between resources: [DisconnectIdpIamRoleLambdaExecution, DisconnectIdpLambdaFunction, ApiGatewayMethodUsersDisconnectDashidpOptions, ApiGatewayResourceUsersDisconnectDashidp, ApiGatewayMethodUsersDisconnectDashidpPost, DisconnectIdpLambdaPermissionApiGateway..., ].

Removing resourceConcurrency from the config above makes deploying the stack for the first time possible.

  splitStacks:
    perFunction: true
    perType: false
    perGroupFunction: false
    stackConcurrency: 5

However, the above config is not working for subsequent deployments. We have the same error as we have when trying to deploy the stack for the first time with resourceConcurrency set.

The CloudFormation template is invalid: Circular dependency between resources: [UpdateEmailTemplateNestedStack, CustomMessageNestedStack, SetDefaultEmailTemplateNestedStack, UpdateGroupNestedStack, ...].

It looks like resourceConcurrency creates some circular dependencies between resources and stackConcurrency does the same but between nested stacks (if that makes sense?).

Is there any way to handle this issue? Currently, we removed both options and left only:

  splitStacks:
    perFunction: true
    perType: false
    perGroupFunction: false

However, I am afraid that we will hit API rate limit errors in the nearest feature and the problem will arise again.

@gautampambhar, @sweepy84, @vicary, maybe, you found some solution to this problem? Thank you for any tips in advance! 🙏

[vicary]

@turakvlad tl;dr craft your own stack-mapping.js in logical layers to prevent them from stacking on each others.

There are limitations of a fully managed SAM template composition. There is no easy fix and you are required to be concise about all the resources being created, and how are they allocated into sub-stacks.

The most probable cause from our experience is when the following resources for the same lambda being sliced into different nested stacks:

1. Lambda Function
2. Lambda Version
3. Lambda Alias
4. Lambda Log Group
5. Lambda Event sources (cron / sqs / http / event bridge)

It is compounded for the 2nd deployment because existing resources are not allowed to move between nested stacks by default, although you may force it to move it's still not scalable.

Try to allocate your resources by type and dependency into it's own "stack chain", for example we have 4 layers of stack chains:

1. API Gateway goes into stack chain called "API", such that you have "APINestedStack1", "APINestedStack2"...etc.
2. Lambda functions goes into stack chain called "Functions", such that you have "FunctionsNestedStack1", "FunctionsNestedStack2"... etc.
3. Lambda Versions, Alias goes to stack chain called "Alias", such that you have "AliasNestedStack1", "AliasNestedStack2"...etc.
4. Log groups goes into it's own stack chain.

The ideas is that resources in a layer will never try to depends on it's own layer or lower layers.

Bare with the abstract descriptions because there is really no way to exhaustively explain CloudFormation, it's a pain and you kinda have to dig into those generated SAM templates in JSON in the directory .serverless/ and really understand your own stack.

EDIT

We never use resourceConcurrency because CloudFormation tries to create resources as a top-down dependency graph, resources will be created once it's dependency is ready. It is already the quickest way possible.

CloudFormation has built-in rate limit backoff and retry mechanism. If you encounter rate limits, it's already a result after multiple retries. If you see them, you should contact your account manager or create a support ticket.

Hint: It's not the first time we see an internal bug where CloudFormation is out of sync with the underlying resources' creation status. So don't be afraid to ask them for help.

[turakvlad]

@vicary, I have no idea how I missed your answer!

Thank you very much for such a detailed response. We will definitely try this approach and craft our own stack-mapping.js.