Accelerating Data Insertion in Django with bulk_create()
In Django,
bulk_create
is a method provided by the Django ORM to efficiently insert multiple objects into the database in a single query. This is particularly useful when you need to create a large number of records and want to minimize database hits.
Here’s an example of how to use
bulk_create
:
Example 1: Basic Usage
from myapp.models import MyModel
# Create a list of objects to be inserted
objects = [
MyModel(field1="Value 1", field2="Value A"),
MyModel(field1="Value 2", field2="Value B"),
MyModel(field1="Value 3", field2="Value C"),
]
# Use bulk_create to insert all objects at once
MyModel.objects.bulk_create(objects)
Example 2: Handling Large Data
If you are dealing with a very large dataset, you can split the data into smaller chunks to avoid memory issues:
from myapp.models import MyModel
from django.db import transaction
# Create a large list of objects
objects = [MyModel(field1=f"Value {i}", field2=f"Value {chr(65 + (i % 26))}") for i in range(100000)]
# Insert in chunks
chunk_size = 1000
with transaction.atomic():
for i in range(0, len(objects), chunk_size):
MyModel.objects.bulk_create(objects[i:i + chunk_size])
Example 3: Using
ignore_conflicts
If you want to ignore duplicate entries (e.g., based on unique constraints), you can use the
ignore_conflicts
parameter (available from Django 2.2+):
from myapp.models import MyModel
objects = [
MyModel(field1="Value 1", field2="Value A"),
MyModel(field1="Value 2", field2="Value B"), # Assume this might conflict
]
# Insert and ignore conflicts
MyModel.objects.bulk_create(objects, ignore_conflicts=True)
Limitations
class Book(models.Model):
title = models.CharField(max_length=100)
author = models.CharField(max_length=100)
created_at = models.DateTimeField(auto_now_add=True)
1.
auto_now_add
and
auto_now
are not applied
Fields like
created_at = models.DateTimeField(auto_now_add=True)
will not be automatically filled.
Example:
Book.objects.bulk_create([
Book(title="Test Book", author="A")
])
# created_at will be NULL unless manually set
Fix:
from django.utils import timezone
now = timezone.now()
Book.objects.bulk_create([
Book(title="Book 1", author="A", created_at=now),
Book(title="Book 2", author="B", created_at=now),
])
2. Model signals (
pre_save
,
post_save
) are not triggered
If your model relies on signals for side effects (like sending emails or logging), those signals will not be triggered.
Example:
@receiver(post_save, sender=Book)
def notify_admin(sender, instance, **kwargs):
print(f"{instance.title} saved")
This will not run when using
bulk_create()
.
3. Model validation is skipped
Model field validation and
clean()
methods are not executed.
Example:
def clean(self):
if not self.title:
raise ValidationError("Title is required")
This check is skipped during
bulk_create()
.
Fix:
Manually run validation before calling
bulk_create()
:
for book in books:
book.full_clean()
4. Primary keys are not automatically available (before Django 4.0)
In Django versions earlier than 4.0, auto-generated primary keys (
id
) are not populated in the objects after
bulk_create()
.
Example:
books = Book.objects.bulk_create([...])
print(books[0].id) # Will be None in Django < 4.0
Fix for Django 4.0+:
Book.objects.bulk_create([...], return_defaults=True)
This will populate
id
,
auto_now_add
, and default fields.
5. Unique constraints can cause errors or silent skipping
If a model field has
unique=True
, and you insert duplicates, Django will raise an error or skip silently if
ignore_conflicts=True
is used.
Example:
class Book(models.Model):
title = models.CharField(max_length=100, unique=True)
books = [Book(title="Duplicate") for _ in range(3)]
Book.objects.bulk_create(books, ignore_conflicts=True)
Only one will be inserted. No error will be shown.
6. Related models must be saved first
You cannot insert related foreign key objects within the same
bulk_create()
call.
Example:
class Author(models.Model):
name = models.CharField(max_length=100)
class Book(models.Model):
title = models.CharField(max_length=100)
author = models.ForeignKey(Author, on_delete=models.CASCADE)
You must save the
Author
objects first:
authors = Author.objects.bulk_create([...])
books = [Book(title="Book A", author=random.choice(authors)) for _ in range(10)]
Book.objects.bulk_create(books)
7. Cannot use
update_fields
like in
bulk_update()
bulk_create()
cannot be used to update existing records. It is for inserting new rows only.
If you want to update many rows, use
bulk_update()
:
Book.objects.bulk_update(book_list, ["title"])
Comparison:
bulk_create()
vs
.save()
Feature
.save()
in loop
bulk_create()
Speed Slow Fast Signals triggered Yes No Auto fields like
created_at
Yes No (unless manually set) Validation Yes No Primary key available Yes No (unless
return_defaults=True
) Handles unique conflict Raises error Skips (if
ignore_conflicts=True
)
When to Use
bulk_create()
Use
bulk_create()
when:
- You need to insert many records efficiently
- You do not need signals or validations
- Your data is already clean
-
You do not need to access
idfields immediately (unless using Django 4.0+)
Avoid using
bulk_create()
when:
- You rely on model signals
-
You need
auto_now_add, validators, orclean()methods - You need to update existing records
