Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Learn more about Collectives

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

Learn more about Teams

django.db.utils.IntegrityError: could not create unique index - DETAIL: Key (player)=(Lonergan) is duplicated. - without unique constraint in model

Ask Question

After a few commits offline including updating the model a couple of times and deployment to production the error message is: (It is working for development but failing in production)

django.db.utils.IntegrityError: could not create unique index - DETAIL:  Key (player)=(Lonergan) is duplicated.

It is over 100 players with a duplicate name in the table and the field player was never set to be unique. Why is this happening?

class Player(models.Model):
    player = models.CharField(max_length=50)
    team = models.ForeignKey(Team, related_name='players', on_delete=models.PROTECT)
    position = models.CharField(max_length=5)
    cost = models.FloatField()
    selection = models.FloatField()
    form = models.FloatField()
    points = models.IntegerField()
    lastgwscrape = models.DateTimeField(null=True)
    lastapiupdate = models.DateTimeField(null=True)
    currentgwdt = models.DateTimeField(null=True)
    apiid = models.IntegerField(null=True)

The apiid field was previously defined as unique and it was removed as a test to make this migration work.

class APIPlayerGW(models.Model):
    player = models.ForeignKey(Player, related_name='apigws', on_delete=models.CASCADE)
    gwid = models.IntegerField()
    points = models.IntegerField()
    minutesplayed = models.IntegerField()
    goalsscored = models.IntegerField()
    assists = models.IntegerField()
    cleansheets = models.IntegerField()
    goalsconceded = models.IntegerField()
    owngoals = models.IntegerField()
    penaltiessaved = models.IntegerField()
    penaltiesmissed = models.IntegerField()
    yellowcards = models.IntegerField()
    redcards = models.IntegerField()
    saves = models.IntegerField()
    bonuspoints = models.IntegerField()
    bonuspointsystem = models.IntegerField()
    influence = models.FloatField()
    creativity = models.FloatField()
    threat = models.FloatField()
    ictindex = models.FloatField()
    datetime = models.DateTimeField(default=timezone.now)
    season = models.CharField(max_length=10)
    # class Meta:
    #     unique_together = ('player','gwid','season') # Double gameweeks are lumped together

The above table had the unique_together combination that was commented out as a test to make this migration work. What can I do to make this migration work?

What do you mean by "this migration"? What changes did you make in this migration? Please add the contents of the migration file for the migration that is not working to your question. – Abdul Aziz Barkat May 17, 2021 at 11:23 @AbdulAzizBarkat I had a look in the django_migrations table and found that it is multiple not yet executed migrations.... None of them with changes to the Player model as far as I could see though. – embe May 17, 2021 at 13:36 In your app there would be a directory migrations in which there would be files like 0001, 0002, .... Show the contents of the failing migration. – Abdul Aziz Barkat May 17, 2021 at 13:40

The migration files had come out of sync with the migrations and Django was trying to use an old migration file.

I assume something like follows happened (or perhaps there may have been other scenarios too, like some migration files were manually edited, etc.):

  • You had your project in some version control.
  • Some people generated migrations locally and pushed to the version control (their names started with the same prefix e.g. 0003), which unfortunately wasn't detected, and perhaps more migrations were later generated on top of these ones.
  • I would suggest the following 3 ways to solve such issues:

    1. Merge migrations using --merge

    Generally the first thing one should try to fix such situations is to try using the --merge flag [Django docs] to merge the conflict. If the conflict is not very complex, in most cases Django will be able to fix it for you with this command.

    2. Roll the migrations back and regenerate them

    The next solution one should try (instead of making the migrations on the production server directly) is to fix the migrations locally by rolling back to some previous migration that is consistent with the production server's state. Suppose production is migrated properly till 0005, we will do something like follows (on the development server):

  • Migrate back to 0005:

     python manage.py migrate <app_label> 0005
    
  • Delete all migrations after 0005 OR selectively delete some migrations OR rename some migrations so that they are in order.

  • Generate migrations again:

     python manage.py makemigrations
    
  • Push these migrations to production and migrate.

    3. Edit the migrations manually

    If these steps are not feasible then one can try actually editing the migration files themselves to fix the problem by referring the documentation on Writing database migrations.

    Note: Although your solution (fixing the migrations on production) works well, many people aren't allowed to do this, also this can lead to problems later on if we have multiple production servers since the migrations between them might not be consistent in such case.

    I did a complete rebuild of the git-repository and deployed the app to a new folder in the VPS. In the new repository (built locally) the migrations files did not match the migration history in production. I tried to copy the old production migration files but it was no longer possible to apply them so I did a "hard reset". I learned that it is better to generate migration files locally only and push to production. The above is then a good recipe for solving conflicts! – embe May 19, 2021 at 9:36

    The migration files had come out of sync with the migrations and Django was trying to use an old migration file. @Abdul Aziz Barkat's comment led me in the right direction. It was possible to solve this by resetting the Django migrations with the following steps:

  • Comment out all changes in Models.py offline to reflect the production database's state
  • Push to production
  • python manage.py migrate --fake app_name zero
  • Navigate to the migrations folder in your app in production and delete the old migration files with find . -path "*.py" -not -name "__init__.py" -delete and find . -path "*.pyc" -delete
  • Create the new inital migration python manage.py makemigrations
  • Since the database tables already exists run python manage.py migrate --fake-initial
  • I got an error message running with the --fake-initial flag, however replacing it with the --fake flag was working: python manage.py migrate --fake

    This kind of reset is like using a sledge hammer to fix a nail and it doesn't teach you anything about how you got into the situation, which means it's very likely to happen again. Migrations are an integral part of a Django application and it's imperative to understand how they work, what the errors can be and how to diagnose them correctly. – user1600649 May 19, 2021 at 7:16 @Melvyn I see your point. I got in the situation because migration files was generated both locally and in production. If migration files had been generated only locally and pushed to production the situation would have never occured. I didn't know of the --merge flag suggested by Abdul Aziz Barkat and I will further on push migration files from local to production. – embe May 19, 2021 at 9:22 So this means, that the same situation can happen, when somebody else pushes to the main line and your feature branch also has migrations in the same app. This is quite common and can be solved by merge or by re-aligning the dependencies. I'm usually re-aligning dependencies, because merge on the long run, makes other migration problems really hard to diagnose (especially circular dependencies). I can highly recommend this tutorial. – user1600649 May 19, 2021 at 17:22 Thanks for the reading advice. Github is a middle storage in this case so the plan will be to merge migration files to Github before migration in production. – embe May 19, 2021 at 19:05

    Thanks for contributing an answer to Stack Overflow!

    • Please be sure to answer the question. Provide details and share your research!

    But avoid

    • Asking for help, clarification, or responding to other answers.
    • Making statements based on opinion; back them up with references or personal experience.

    To learn more, see our tips on writing great answers.

  •