如何使用SQLAlchemy Postgres ORM的声明性基础从字典中动态创建带有列名和约束的表?

0 人关注

设置。Postgres13, Python 3.7, SQLAlchemy 1.4

我的问题是关于动态创建类而不是依赖 models.py 的内容。我有一个 schema.json文件,其中有许多表的元数据。列的数量、列的名称、列的约束条件因表而异,事先并不知道。

JSON被解析,其结果被映射到ORM的Postgres方言中(例如:{'column_name1': 'bigint'}变成'column_name1 = Column(BigInt)')。这将创建一个字典,其中包含 表名、列名和列约束。由于所有的表都是通过增强的基数,它们自动地 收到一个PK id字段。

然后我把这个字典传递给一个 create_class 的函数,这个函数使用这些数据来动态地创建表 并将这些新表提交给数据库。

问题是,当我运行代码时,表确实被创建了,但只有一列--PK id。 这是它自动收到的。所有其他的列都被忽略了。

我怀疑我产生这个错误的方式是 我在调用Session或Base的时候,或者在传递列约束的时候,产生了这个错误。我不知道如何向ORM表明我正在传递列和约束对象。

我试着改变了一些东西,比如。

  • 创建类的方式--传入一个列对象而不是一个列字符串 例如。替换代码2VS 【替换代码3

  • 改变收集列约束的方式

  • 以不同方式调用 Base create 。我试图在下面的 create_class 的注释行中显示这些变化。

    我不知道是哪一部分的互动产生了这个错误。非常感谢任何帮助

    Here is the code:

    schema.json的例子

      "groupings": {
        "imaging": {
          "owner": { "type": "uuid", "required": true, "index": true },
          "tags": { "type": "text", "index": true }
          "filename": { "type": "text" },
        "user": {
          "email": { "type": "text", "required": true, "unique": true },
          "name": { "type": "text" },
          "role": {
            "type": "text",
            "required": true,
            "values": [
              "admin",
              "customer",
            "index": true
          "date_last_logged": { "type": "timestamptz" }
      "auths": {
        "boilerplate": {
          "owner": ["read", "update", "delete"],
          "org_account": [],
          "customer": ["create", "read", "update", "delete"]
          "loggers": {
          "owner": [],
          "customer": []
    

    base.py

    from sqlalchemy import Column, create_engine, Integer, MetaData
    from sqlalchemy.orm import declared_attr, declarative_base, scoped_session, sessionmaker
    engine = create_engine('postgresql://user:pass@localhost:5432/dev', echo=True)
    db_session = scoped_session(
        sessionmaker(
            bind=engine,
            autocommit=False,
            autoflush=False
    # Augment the base class by using the cls argument of the declarative_base() function so all classes derived
    # from Base will have a table name derived from the class name and an id primary key column.
    class Base:
        @declared_attr
        def __tablename__(cls):
            return cls.__name__.lower()
        id = Column(Integer, primary_key=True)
    metadata_obj = MetaData(schema='collect')
    Base = declarative_base(cls=Base, metadata=metadata_obj)
    

    models.py

    from base import Base
    from sqlalchemy import Column, DateTime, Integer, Text
    from sqlalchemy.dialects.postgresql import UUID
    import uuid
    class NumLimit(Base):
        org = Column(UUID(as_uuid=True), default=uuid.uuid4, unique=True)
        limits = Column(Integer)
        limits_rate = Column(Integer)
        rate_use = Column(Integer)
        def __init__(self, org, limits, allowance_rate, usage, last_usage):
            super().__init__()
            self.org = org
            self.limits = limits
            self.limits_rate = limits_rate
            self.rate_use = rate_use
    

    create_tables.py(我知道这个很乱!只是想展示所有尝试的变化......)

    def convert_snake_to_camel(name):
        return ''.join(x.capitalize() or '_' for x in name.split('_'))
    def create_class(table_data):
        constraint_dict = {'__tablename__': 'TableClass'}
        table_class_name = ''
        column_dict = {}
        for k, v in table_data.items():
            # Retrieve table, alter the case, store it for later use
            if 'table' in k:
                constraint_dict['__tablename__'] = v
                table_class_name += convert_snake_to_camel(v)
            # Retrieve the rest of the values which are the column names and constraints, ex: 'org = Column(UUID(as_uuid=True), default=uuid.uuid4, unique=True)'
            else:
                constraint_dict[k] = f'= Column({v})'
                column_dict[k] = v
        # When type is called with 3 arguments it produces a new class object, so we use it here to create the table Class
        table_cls = type(table_class_name, (Base,), constraint_dict)
        # Call ORM's 'Table' on the Class
        # table_class = Table(table_cls) # Error "TypeError: Table() takes at least two positional-only arguments 'name' and 'metadata'"
        # db_session.add(table_cls) # Error "sqlalchemy.orm.exc.UnmappedInstanceError: Class 'sqlalchemy.orm.decl_api.DeclarativeMeta'
                                    # is not mapped; was a class (__main__.Metadata) supplied where an instance was required?"
        # table_class = Table(
        #    table_class_name,
        #    Base.metadata,
        #    constraint_dict) # Error "sqlalchemy.orm.exc.UnmappedInstanceError: Class 'sqlalchemy.orm.decl_api.DeclarativeMeta'
                              # is not mapped; was a class (__main__.Metadata) supplied where an instance was required?"
        # table_class = Table(
        #    table_class_name,
        #    Base.metadata,
        #    column_dict)  
        #    table_class.create(bind=engine, checkfirst=True) # sqlalchemy.exc.ArgumentError: 'SchemaItem' object, such as a 'Column' or a 'Constraint' expected, got {'limits': 'Integer'}
        # table_class = Table(
        #    table_class_name,
        #    Base.metadata,
        #    **column_dict) # TypeError: Additional arguments should be named <dialectname>_<argument>, got 'limits'
        # Base.metadata.create_all(bind=engine, checkfirst=True)
        # table_class.create(bind=engine, checkfirst=True)
        new_row_vals = table_cls(**column_dict)
        db_session.add(new_row_vals)  # sqlalchemy.exc.ArgumentError: 'SchemaItem' object, such as a 'Column' or a 'Constraint' expected, got {'limits': 'Integer'}
        db_session.commit()
        db_session.close()
        
  • 3 个评论
    为了能够回答这个问题,我缺少的是你的输入数据是什么样子的。你能添加一个schema.json的例子吗?
    @JesseBakker 我刚刚在问题的顶部添加了一个JSON的例子,谢谢!
    在GitHub上也有回答 here
    python
    postgresql
    sqlalchemy
    orm
    dynamic-tables
    alphaomega
    alphaomega
    发布于 2021-11-04
    1 个回答
    Jesse Bakker
    Jesse Bakker
    发布于 2021-11-05
    已采纳
    0 人赞同

    我已经为你创建了一个独立的例子。这应该会给你提供基本的构建模块,让你自己建立这个。它包括typemap,将类型字符串映射到sqlchemy类型,以及argumentmap,将非sqlchemy参数映射到其sqlchemy对应的参数( required: True 在sqlchemy中是 nullable: False )。 这种方法使用 元数据 来定义表格,然后将这些转换为声明性的映射,如图所示。 使用 __table__ 的混合方法 with the python type() 功能。这些生成的类然后被导出到模块范围的 globals()

    不是所有你提供的 schema.json 都支持,但这应该给你一个很好的起点。

    from sqlalchemy import Column, Integer, Table, Text
    from sqlalchemy.dialects.postgresql import UUID
    from sqlalchemy.orm import declarative_base
    def convert_snake_to_camel(name):
        return "".join(part.capitalize() for part in name.split("_"))
    data = {
        "groupings": {
            "imaging": {
                "id": {"type": "integer", "primary_key": True},
                "owner": {"type": "uuid", "required": True, "index": True},
                "tags": {"type": "text", "index": True},
                "filename": {"type": "text"},
            "user": {
                "id": {"type": "integer", "primary_key": True},
                "email": {"type": "text", "required": True, "unique": True},
                "name": {"type": "text"},
                "role": {
                    "type": "text",
                    "required": True,
                    "index": True,
    Base = declarative_base()
    typemap = {
        "uuid": UUID,
        "text": Text,
        "integer": Integer,
    argumentmap = {
        "required": lambda value: ("nullable", not value),
    for tablename, columns in data["groupings"].items():
        column_definitions = []
        for colname, parameters in columns.items():
            type_ = typemap[parameters.pop("type")]
            params = {}
            for name, value in parameters.items():
                    name, value = argumentmap[name](value)
                except KeyError:
                finally:
                    params[name] = value
            column_definitions.append(Column(colname, type_(), **params))
        # Create table in metadata
        table = Table(tablename, Base.metadata, *column_definitions)
        classname = convert_snake_to_camel(tablename)
        # Dynamically create a python class with definition
        # class classname:
        #     __table__ = table