Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

I have created a deployment package for AWS Lambda with my python file and the dependencies including sqlalchemy and psycopg2. The code works perfectly in accessing the DB locally. But when I imported this zip file, I am getting the following error.

No module named 'psycopg2._psycopg': ModuleNotFoundError
The stack trace of the error is, 
  "errorMessage": "No module named 'psycopg2._psycopg'",
  "errorType": "ModuleNotFoundError",
  "stackTrace": [
      "/var/task/DBAccessLamdaHandler.py",
      "lambda_handler",
      "engine = create_engine(rds_host)"
      "/var/task/sqlalchemy/engine/__init__.py",
      "create_engine",
      "return strategy.create(*args, **kwargs)"
      "/var/task/sqlalchemy/engine/strategies.py",
      "create",
      "dbapi = dialect_cls.dbapi(**dbapi_args)"
      "/var/task/sqlalchemy/dialects/postgresql/psycopg2.py",
      "dbapi",
      "import psycopg2"
      "/var/task/psycopg2/__init__.py",
      "<module>",
      "from psycopg2._psycopg import (                     # noqa"
Any help is appreciable
                You don't need to build it from scratch. The repo includes a pre-built package in the psycopg2 folder that you can simply include in the ZIP file that you're going to upload to Lambda (allegedly, I haven't actually used it).
– jarmod
                Jun 30, 2017 at 22:51
The AWS Lambda runtime environment doesn't include the PostgreSQL libraries so you need to include them within your AWS Lambda upload.
One way to do this is to get them from the jkehler/awslambda-psycopg2 repo at GitHub. Note that you don't need to build this project from scratch as the repo includes a pre-built package in the psycopg2 folder that you can simply include in your Lambda upload. 
                How would this solution work with the serverless framework where packages are installed from the requirmets.txt file?
– johnny
                May 18, 2018 at 12:43
                I've included this library like so however it is still not working. Something I'm going wrong?
– Jack
                Mar 27, 2019 at 22:36
                For anyone working with CloudFormation + CodeBuild the following two lines in buildspec.yml worked like a charm:   - git clone https://github.com/a-j/awslambda-psycopg2.git && cd awslambda-psycopg2 && git checkout python-3.8-libraries and - cp -r awslambda-psycopg2/psycopg2-3.8 ./src && mv ./src/psycopg2-3.8 ./src/psycopg2
– Stefano Messina
                Jul 31, 2020 at 7:52
The psycopg2 build library from jkehler/awslambda-psycopg2 was built for python 3.6 and make sure that while uploading your code to AWS lambda, select Python Runtime environment as 3.6, and it should work. I banged my head on this for a full day and then when i changed to 3.6, the import error just vanished.
If you are going to attempt to build it yourself, remember that you must build on a machine or VM with the same architecture as your target at AWS.
                Thanks Nikhil, but I tried this and it didn't work on p3.6, could you please help in detail?
– GD- Ganesh Deshmukh
                Oct 11, 2019 at 20:08
                @ganeshdeshmukh It work with me on python 3.7. I simply copied the whole folder for python 3.7, rename it to psycopg2, and zip it with my lambda function file. After uploading the zipped file to aws lambda, it worked as expected. I suppose the procedure for python 3.6 should be similar.
– Fanchen Bao
                Oct 21, 2019 at 20:17
                It work with me on python 3.6. I have followed the same procedure as mentioned by @FanchenBao
– Mahan
                Feb 24, 2020 at 5:26
Like other answers, psycopg2-binary worked fine for python3.9 (it looks like other package awslambda-psycopg2 was only available for python3.6).
But, if you run on MacOs before sending to aws lambda, you must specify the platform to your pip install like this:
pip3.9 install --platform=manylinux1_x86_64 --only-binary=:all: psycopg2-binary
Latest as of 26/MAR/2020
I was skeptical about depending on a third-party library for my production code. On research the following works,
The issue happens only when the packages are built from MAC OS.
I can confirm today that the issue is fixed when I build the package from Centos 7 ( AWS AMI )
The following is my approach
requirement.txt
psycopg2-binary==2.8.4
Build process
pip install -r requirements.txt --target .
Lambda code is in the root directory
+-- lambda_function.py
+-- psycopg2
    +-- psycopg2 files
Zip the directory and test the code in lambda works.
The only additional step is to build the package in Linux env instead of macOS using the Docker container. An example can be found here: Deploy AWS Amplify Python Lambda from macOS with Docker
                How is this related to the AWS Lambda runtime? Packages are installed by pip on the aws/codebuild/standard:2.0 image
– Stefano Messina
                Jul 31, 2020 at 7:07
                Python runtime checks for the local library first. Give this a shot - works for me in production till date
– viru
                Jul 31, 2020 at 19:14
For Windows (Looks like in any OS which is not build on Amazon AMI or Centos), the easiest  fix is to use
psycopg2 Python Library for AWS Lambda
which you can find https://github.com/jkehler/awslambda-psycopg2
I have faced a similar issue. I tried to create the layers using AWS Cloud shell it worked with Python3.8. But failed in Python3.9 though.
Please change the region name as per your AWS Region
sudo amazon-linux-extras install python3.8
curl -O https://bootstrap.pypa.io/get-pip.py
python3.8 get-pip.py --user
mkdir python
python3.8 -m pip install --platform=manylinux1_x86_64 --only-binary=:all: pandas numpy pymysql psycopg2-binary SQLAlchemy -t python/
zip -r layer.zip python
aws lambda publish-layer-version \
 --layer-name mics-layer \
 --description "Pandas Numpy psycopg2-binary SQLAlchemy pymysql" \
 --zip-file fileb://layer.zip \
 --compatible-runtimes python3.8 python3.9 \
 --region eu-west-1
I've had issues getting SQLAlchemy to run on AWS Lambda despite the fact that I have tried multiple psycopg2 versions, but what eventually solved it for me was to use an older Python version. I went from Python 3.9 to 3.7 and it was finally able to run (Using psycopg2-binary 2.8.4, but I didn't try other versions or the none binary version with 3.7)
The psycopg2 python package requires Postgres libraries which is missing in AMI used by the AWS Lambda function.
The easiest solution is to use a pre-compiled package aws-psycopg2 which has the required libraries. The package is present in the PyPi repository.
pip install aws-psycopg2
After this, just do the import of psycopg2 as usual in your program. Sample import is given below.
import psycopg2
With this package, my Lambda function started working perfectly.
instead of using psycopg2, try using pg8000
in command prompt go the directory that you are currently working(example F:\example)
pip install pg8000 -t. (-t. is for installing pg8000 in the directory that you are working)
// handler.py
import pg8000
database=''
host=''
port=''
user=''
password=''
conn = pg8000.connect(database=database, host=host, port=port, user=user, 
password=password)
def lambda_function(event, context):
after writing your code zip the code and upload in your aws lambda.
It worked for me!!!
If you are using CDK you can use SQLAlchemy on AWS Lambda by bundling with Docker and providing the right pip install command:
Python 3.8.
    sqlachemy_layer = lambda_.LayerVersion(
        scope=self,
        id=LAYERS_SQLALCHEMY_LAYER,
        layer_version_name=LAYERS_SQLALCHEMY_LAYER,
        code=lambda_.Code.from_asset(
            str(pathlib.Path(__file__).parent.joinpath("dependencies").resolve()),
            bundling=BundlingOptions(
                image=lambda_.Runtime.PYTHON_3_8.bundling_image,
                command=[
                    "bash", "-c",
                    "pip install -r requirements-sqlalchemy.txt -t /asset-output/python && cp -au . /asset-output/python",
        compatible_runtimes=[
            lambda_.Runtime.PYTHON_3_8,
Python 3.9
    sqlachemy_layer = lambda_.LayerVersion(
        scope=self,
        id=LAYERS_SQLALCHEMY_LAYER,
        layer_version_name=LAYERS_SQLALCHEMY_LAYER,
        code=lambda_.Code.from_asset(
            str(pathlib.Path(__file__).parent.joinpath("dependencies").resolve()),
            bundling=BundlingOptions(
                image=lambda_.Runtime.PYTHON_3_9.bundling_image,
                command=[
                    "bash", "-c",
                    "pip3.9 install --platform=manylinux1_x86_64 --only-binary=:all: -r requirements-sqlalchemy.txt -t /asset-output/python && cp -au . /asset-output/python",
        compatible_runtimes=[
            lambda_.Runtime.PYTHON_3_9,
The requirements-sqlalchemy.txt:
psycopg2-binary
sqlalchemy
And the file structure:
dependencies/
requirements-sqlalchemy.txt
my_stack.py
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.