ModuleNotFoundError: No module named 'argcomplete'

[TravelingLex]

I'm attempting to get airflow running on Kubernetes using the official airflow helm chart but I've been running into this error on webserver startup:

*** running webserver...
Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 23, in <module>
    import argcomplete
ModuleNotFoundError: No module named 'argcomplete'

The only references to this problem is not setting 'AIRFLOW__KUBERNETES__RUN_AS_USER: "50000"' which is something that I've already done. Does anyone have any suggestions for this?

[potiuk]

Which image version do you use? Can you post the helm configuration/image used?

[TravelingLex]

Hello,

I'm pulling the apache/airflow:1.10.12-python3.6 image and baking my dags into this image.

Here is my Dockerfile:

FROM apache/airflow:1.10.12-python3.6

ARG PROJECT

#Copy and install python requirements
COPY --chown=airflow ${PROJECT}/requirements.txt /opt/airflow/dags/

#Copy DAGS
COPY --chown=airflow ${PROJECT}/dags /opt/airflow/dags/

#Copy include folder
COPY --chown=airflow ${PROJECT}/include /opt/airflow/include/

I'm using the 7.14.0 helm chart from airflow (no real reason behind version at this point), I've tried 7.15.0 and 7.16.0 as well. Below are the custom values I'm passing in.

###################################
# Airflow - Common Configs
###################################
airflow:
  image:
    repository: {private_repo}
    tag: {{ requiredEnv "TAG" }}
    pullSecret: {secret}

  executor: KubernetesExecutor
  fernetKey: {{ requiredEnv "AIRFLOW__FERNET_KEY" }}

  config:
    #AIRFLOW__CORE__PARALLELISM: 500
    #AIRFLOW__CORE__DAG_CONCURRENCY: 256
    #AIRFLOW__CORE__DAG_RUN_CONF_OVERRIDES_PARAMS: true
    AIRFLOW__CORE__LOAD_EXAMPLES: "False"
    AIRFLOW__KUBERNETES__DAGS_IN_IMAGE: 'True'
    AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY: {private_repo}
    AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG: {{ requiredEnv "TAG" }}
    AIRFLOW__KUBERNETES__WORKER_CONTAINER_IMAGE_PULL_POLICY: Always
    AIRFLOW__KUBERNETES__RUN_AS_USER: "50000"
    AIRFLOW__WEBSERVER__BASE_URL: {base_url}

  extraEnv: []

###################################
# Airflow - Scheduler Configs
###################################
scheduler:
  resources:
    limits:
      cpu: "1000m"
      memory: "1Gi"
    requests:
      cpu: "500m"
      memory: "512Mi"

  connections: []

  variables: |
    {}

  pools: |
    {}

  nodeSelector: {
    role: linux
  }

###################################
# Airflow - WebUI Configs
###################################
web:
  replicas: 2
  resources:
    limits:
      cpu: "2"
      memory: "3Gi"
    requests:
      cpu: "300m"
      memory: "1Gi"

  nodeSelector: {
    role: linux
  }
  service:
    annotations: {}
    sessionAffinity: "None"
    sessionAffinityConfig: {}
    type: ClusterIP
    externalPort: 443
    loadBalancerIP: ""
    loadBalancerSourceRanges: []
    nodePort:
      http: ""
  baseUrl: {base_url}

###################################
# Airflow - Worker Configs
###################################
workers:
  enabled: false

  nodeSelector: {
    role: linux
  }

###################################
# Airflow - Flower Configs
###################################
flower:
  enabled: false

###################################
# Airflow - Logs Configs
###################################
logs:
  path: /opt/airflow/logs
  persistence:
    enabled: false

###################################
# Airflow - DAGs Configs
###################################
dags:
  path: /opt/airflow/dags
  doNotPickle: true
  installRequirements: true

  persistence:
    enabled: false

  git:
    enabled: false

  initContainer:
    enabled: false

###################################
# Kubernetes - Ingress Configs
###################################
ingress:
  enabled: true
  web:
    annotations: {
      kubernetes.io/ingress.class: "nginx",
      kubernetes.io/tls-acme: "true",
      cert-manager.io/cluster-issuer: "letsencrypt-prod"
    }
  host: {host_name}
  path: "/airflow/"

###################################
# Database - PostgreSQL Chart
# - https://github.com/helm/charts/tree/master/stable/postgresql
###################################
postgresql:
  enabled: false

###################################
# Database - External Database
# - these configs are only used when `postgresql.enabled` is false
###################################
externalDatabase:
  type: postgres
  host: {db}
  port: 5432
  database: airflow
  user: {user}
  passwordSecret: {secret_value}
  passwordSecretKey: {secret_key}

###################################
# Database - Redis Chart
# - https://github.com/helm/charts/tree/master/stable/redis
###################################
redis:
  enabled: false

[hauthorn]

Did you find a solution for this? I'm trying a docker compose setup with the apache/airflow:1.10.15 image, and I'm getting the same error.

[potiuk]

You likely run it with wrong user - that have no airflow dependencies installed. Make sure to enter the image the same way you use is in your deployment (entrypoint, users, group) and see if everything works fine there

[potiuk]

You likely run it with wrong user - that have no airflow dependencies installed. Make sure to enter the image the same way you use is in your deployment (entrypoint, users, group) and see if everything works fine there

[hauthorn]

@potiuk This was my first time using it.
It's not for deployment, it's basically the standard dev setup, but trying out airflow 1, because that's what we have in production at the moment.

Switching to airflow 2 worked, so I'm assuming there's something in the airflow 1.10.15* builds that doesn't work (with this particular compose setup).

[potiuk]

Welll. 1.10.15 is already past end of life for 1.5 year and it has not been maintained since and "standard dev setup" you mention is Airflow 2.2+ only, So yeah it is very likely things are not workking.

BTW. You should move to Airflow 2. With 1.10, not only you will not get support from maintainers here but you are one of the very few users left who did not migrate - see survey from 2021 where < 15% users (in 2021!) were using 1.10. Most likely low-single-percentage-digit now.

As you noticed, you are basically in this situation:

[hauthorn]

You are absolutely right @potiuk! I've not been responsible for the data warehouse infrastructure and coming on board there, so there's probably more obvious improvements to make.

Thanks for taking the time to reply.