Issue with serverless deploy, requirements.txt not found

[manav95]

I have the following serverless.yml:

# Welcome to Serverless!
#
# This file is the main config file for your service.
# It's very minimal at this point and uses default values.
# You can always add more config options for more control.
# We've included some commented out config examples here.
# Just uncomment any of them to get that config option.
#
# For full config options, check the docs:
#    docs.serverless.com
#
# Happy Coding!

service: awsTest

# You can pin your service to only deploy with a specific Serverless version
# Check out our docs for more details
# frameworkVersion: "=X.X.X"
plugins:
  - serverless-python-requirements
custom:
    pythonRequirements:
      invalidateCaches: true
      dockerizePip: true  
      dockerImage: lambda-python3.6-with-mysql-build-deps        


provider:
  name: aws
  runtime: python3.6
  role: arn:aws:iam::443746630310:role/EMR_DefaultRole

# you can overwrite defaults here
#  stage: dev
#  region: us-east-1

# you can add statements to the Lambda function's IAM Role here
#  iamRoleStatements:
#    - Effect: "Allow"
#      Action:
#        - "s3:ListBucket"
#      Resource: { "Fn::Join" : ["", ["arn:aws:s3:::", { "Ref" : "ServerlessDeploymentBucket" } ] ]  }
#    - Effect: "Allow"
#      Action:
#        - "s3:PutObject"
#      Resource:
#        Fn::Join:
#          - ""
#          - - "arn:aws:s3:::"
#            - "Ref" : "ServerlessDeploymentBucket"
#            - "/*"

# you can define service wide environment variables here
#  environment:
#    variable1: value1

# you can add packaging information here
#package:
#  include:
#    - include-me.py
#    - include-me-dir/**
#  exclude:
#    - exclude-me.py
#    - exclude-me-dir/**

functions:
  emotion-analysis:
    handler: handler.emotionAnalysis
    events:
       - http:
            path: emotionAnalysis
            method: post

  audio-analysis:
    handler: handler.audioAnalysis
    events:
       - http:
            path: vokaturiAnalysis
            method: post


#    The following are a few example events you can configure
#    NOTE: Please make sure to change your handler code to work with those events
#    Check the event documentation for details
#    events:
#      - http:
#          path: users/create
#          method: get
#      - s3: ${env:BUCKET}
#      - schedule: rate(10 minutes)
#      - sns: greeter-topic
#      - stream: arn:aws:dynamodb:region:XXXXXX:table/foo/stream/1970-01-01T00:00:00.000
#      - alexaSkill
#      - iot:
#          sql: "SELECT * FROM 'some_topic'"
#      - cloudwatchEvent:
#          event:
#            source:
#              - "aws.ec2"
#            detail-type:
#              - "EC2 Instance State-change Notification"
#            detail:
#              state:
#                - pending
#      - cloudwatchLog: '/aws/lambda/hello'
#      - cognitoUserPool:
#          pool: MyUserPool
#          trigger: PreSignUp

#    Define function environment variables here
#    environment:
#      variable2: value2

# you can add CloudFormation resource templates here
#resources:
#  Resources:
#    NewResource:
#      Type: AWS::S3::Bucket
#      Properties:
#        BucketName: my-new-bucket
#  Outputs:
#     NewOutput:
#       Description: "Description for the output"
#       Value: "Some output value"

and the requirements.txt:

cycler==0.10.0
decorator==4.1.2
imutils==0.4.3
Keras==2.1.1
matplotlib==2.1.0
networkx==2.0
numpy==1.13.3
olefile==0.44
opencv-python==3.3.0.10
pandas==0.21.0
Pillow==4.3.0
pyparsing==2.2.0
python-dateutil==2.6.1
pytz==2017.3
PyWavelets==0.5.2
PyYAML==3.12
scikit-image==0.13.1
scikit-learn==0.19.1
scipy==1.0.0
six==1.11.0
sklearn==0.0
dlib==19.7.0

I am using this Dockerfile to compile dlib and boost:

FROM amazonlinux:latest

RUN touch /var/lib/rpm/* 
RUN yum install -y yum-plugin-ovl && cd /usr/src
#RUN  yum check-update
#RUN rpm --rebuilddb
RUN yum history sync 
RUN yum install -y wget 
RUN yum install -y sudo
RUN yum install -y sudo && sudo yum install -y yum-utils && sudo yum groupinstall -y development
RUN sudo yum install -y https://centos6.iuscommunity.org/ius-release.rpm && sudo yum install -y python36u && yum install -y python36u-pip && yum install -y python36u-devel
#RUN yum install -y grub2


RUN ln -s /usr/include/python3.6m /usr/include/python3.6

RUN wget --no-check-certificate -P /tmp http://flydata-rpm.s3-website-us-east-1.amazonaws.com/patchelf-0.8.tar.gz 
RUN    tar xvf /tmp/patchelf-0.8.tar.gz -C /tmp 
RUN    cd /tmp/patchelf-0.8 && ./configure && make && sudo make install 

RUN  yum install -y blas-devel boost-devel lapack-devel gcc-c++ cmake git 
RUN     git clone https://github.com/davisking/dlib.git 
RUN     cd dlib/python_examples/ 
RUN     mkdir build && cd build 
RUN     cmake -DPYTHON_INCLUDE_DIR=$(python3.6 -c "from distutils.sysconfig import get_python_inc; print(get_python_inc())") -DPYTHON_LIBRARY=$(python3.6 -c "import distutils.sysconfig as sysconfig; print(sysconfig.get_config_var('LIBDIR'))") -DUSE_SSE4_INSTRUCTIONS:BOOL=ON dlib/tools/python
RUN     sed -i 's/\/\/all/all/' Makefile && sed -i 's/\/\/preinstall/preinstall/' Makefile
RUN     cmake --build . --config Release --target install   
RUN     cd .. 

RUN    mkdir ~/dlib  
RUN    cp dlib.so ~/dlib/__init__.so 
RUN    cp /usr/lib64/libboost_python-mt.so.1.53.0 ~/dlib/  
RUN    touch ~/dlib/__init__.py 
RUN    patchelf --set-rpath '$ORIGIN' ~/dlib/__init__.so 

When I run serverless deploy, I get the following error:

Error --------------------------------------------------
 
Error: Could not open requirements file: [Errno 2] No such file or directory: '.serverless/requirements.txt'

    at ServerlessPythonRequirements.installRequirements (/Users/manavdutta1/Downloads/awsTest/node_modules/serverless-python-requirements/lib/pip.js:80:11)
From previous event:
    at PluginManager.invoke (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:366:22)
    at PluginManager.spawn (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:384:17)
    at Deploy.BbPromise.bind.then.then (/usr/local/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:120:50)
From previous event:
    at Object.before:deploy:deploy [as hook] (/usr/local/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:110:10)
    at BbPromise.reduce (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:366:55)
From previous event:
    at PluginManager.invoke (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:366:22)
    at PluginManager.run (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:397:17)
    at variables.populateService.then (/usr/local/lib/node_modules/serverless/lib/Serverless.js:104:33)
    at runCallback (timers.js:785:20)
    at tryOnImmediate (timers.js:747:5)
    at processImmediate [as _immediateCallback] (timers.js:718:5)
From previous event:
    at Serverless.run (/usr/local/lib/node_modules/serverless/lib/Serverless.js:91:74)
    at serverless.init.then (/usr/local/lib/node_modules/serverless/bin/serverless:42:50)
    at <anonymous>
 

I have no idea why this is happening. I have the requirements.txt under .serverless in my local directory and it looks fine. Does anyone know why this is happening?

[dschep]

You should keep your requirements.txt in the root of your service, the plugin creates the file at .serverless/requirements.txt.

[heri16]

Are you running on windows @manav95? I noticed you enabled dockerizePip. See Issue #105

[manav95]

im using mac os x

[heri16]

Check your shared drives in the docker for Mac settings.

[JFox]

I'm having the exact same issue as @manav95. I'm using default docker image on debian jeesie

[dschep]

Just thought of something it might be.. Add this to your Dockerfile:

RUN mkdir /var/task
WORKDIR /var/task

[thesmith]

I'm hitting this too. Runs perfectly locally (OSX) but when using Codeship and the following Dockerfile:

FROM docker:dind

RUN apk add --update \
    nodejs \
    python3 \
    py-pip \
    build-base \
  && pip install virtualenv \
  && rm -rf /var/cache/apk/*

RUN mkdir /var/task
WORKDIR /var/task

COPY . /var/task

RUN npm install -g serverless
RUN sls plugin install -n serverless-python-requirements

Then when I run serverless deploy I get the same error.

[Sitin]

I have the same problem.

Also when I am trying to pull image inside the dicker:dindimage:

I got:

(image: codeship_smr-chatbot) (service: smr-chatbot) Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

(image: codeship_smr-chatbot) (service: smr-chatbot) Removing intermediate container 0f5038935f88
(step: deploy to dev) error ✗
(step: deploy to dev) error loading services during run step: failure to build Image{ name: "smr-chatbot", branch: "dev", dockerfile: "/Users/sitin/Documents/Workspace/Chimplie/smr-chatbot/Dockerfile", cache: true }: The command '/bin/sh -c docker pull lambci/lambda:build-python3.6' returned a non-zero code: 1

I am running via jet and have add_docker: true.

[avli]

I get this error too when I try using the plugin on CircleCI to automate deployment. I don't use any custom Docker images but circleci/python:3.6.4. The plugin configuration I use is as follows:

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    dockerizePip: true

And yes, everything runs perfectly on my local machine which runs macOS.

[thesmith]

I've been playing with this a bit more and there's definitely something about running the pip install through docker, from within another docker.

I guess one way to get around this would be to run the pip install command without docker, given we're already within a docker container - as long as the host docker is the right kind to build the package for lambda.

If there was an extended version of https://github.com/lambci/docker-lambda/tree/master/python3.6 that we could use to run serverless deploy from then we could set dockerizePip: false.

[avli]

@thesmith Yes, this is the current workaround I use. Thank you for posting it – can be useful for other users who are hitting this issue.

[thesmith]

So this Dockerfile seems to be working, obviously dockerizePip has to be false:

FROM lambci/lambda:build-nodejs6.10

ENV AWS_DEFAULT_REGION eu-west-1 \
    PYTHONPATH=/var/runtime \
    PKG_CONFIG_PATH=/var/lang/lib/pkgconfig:/usr/lib64/pkgconfig:/usr/share/pkgconfig

RUN curl https://lambci.s3.amazonaws.com/fs/python3.6.tgz | tar -xz -C / && \
    sed -i '/^prefix=/c\prefix=/var/lang' /var/lang/lib/pkgconfig/python-3.6.pc && \
    curl https://www.python.org/ftp/python/3.6.1/Python-3.6.1.tar.xz | tar -xJ && \
    cd Python-3.6.1 && \
    LIBS="$LIBS -lutil -lrt" ./configure --prefix=/var/lang && \
    make -j$(getconf _NPROCESSORS_ONLN) libinstall inclinstall && \
    cd .. && \
    rm -rf Python-3.6.1 && \
    pip3 install awscli virtualenv --no-cache-dir

RUN npm install -g serverless

COPY . .

RUN npm install

Annoyingly this means you have to flip dockerizePip between deploying via CI and locally.

[dschep]

Ah. yeah I'll ahve to check docker-in-docker out at some point.

Re this @thesmith:

Annoyingly this means you have to flip dockerizePip between deploying via CI and locally.

You an do something like:

custom:
  pythonRequirements:
    dockerizePip: ${self:custom.isCI.${env:CI}, self:custom.isCI.false}
  isCI:
    true: true
    false: non-linux

(this assumes you have a CI env var set to true in CI (CircleCI does this automatically, not sure how standard it is, but it'd be easy to add the var or adapt this technique to your CI provider)