Forums

Hi everybody! I get the following error when running a batched django data migration:

  File "/home/michaelgolth/.virtualenvs/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.DiskFull: could not resize shared memory segment "/PostgreSQL.230765310" to 8388608 bytes: No space left on device

I'm migrating one field of a table whose total size is 1.4 GB, so it's unlikely that it's a memory issue (the migration is batched and only the one field is pulled from the db). I also have 17 GB of disk space and 40 GB or so db space left.

Please let me know what's wrong.

The "shared memory" issue is not related to how much disk space you have on the account or for the database. Could you try reducing the batch size and reruning the migration?

I've already tried running it with a batch size of 5 (out of 4 million instances), turned off all other tasks (3 GB memory limit is shared?), and reran countless times. I'm running the migration with atomic = False committing every batch to the db. It worked up to a point (900k instances) but now every single attempt fails.

Oh and another question, is this entirely related to memory?

Looks like that.

So do I only have 8 MB available? At least that's what the error message indicates (8388608 bytes). I've also tried around a lot and I don't see how what I'm doing consumes 3 GB. Simple object serialization of 5 instances with only 2 char fields already throws this error...

The "disk" that it is referring to is shared memory that is not related to the 3G limit. You have significantly less that 3G of shared memory on your postgres instance. Some postgres queries create temporary tables or files and those tables or files are stored in shared memory. If the queries you are running create temporary tables ro files that are too big to be help in shared memory, then you get that error.

Some poking around online suggests that hash tables - that may be created when joining tables could be the culprit - you can check that using EXPLAIN on a query and see if there are references to Hash in the resulting output. If there are, then you can add indexes on the keys that are joined on the tables.

Thanks for clarifying this! I've managed to "solve" my current issue by just brute forcing it, running the migration instance by instance without any batching.

But that seems fairly crude. Especially given that my query was very simple. Is there a way to increase the shared memory?

Not at the moment. We may consider it in the future.