Forums

Hi! I am persisting data of my app in a plain json file (I know, very bare bones). I am having the issue that this file is not updated reliably. I'd say no content is written in 10% of trials. The method looks like this:

with open(self.data_file, "w+") as file:
   file.write(self.data)
   file.flush()
   os.fsync(file.fileno())
   file.close()

The flush() and fsync() I added after I first discovered the issue, however this did not help things.

The file is opened once the server starts up and the data kept in memory after that:

def __load_invoices(self):
   if(os.path.exists(self.data_file)):
          with open(self.data_file) as file:
                 self.invoices = json.load(
                    file, object_hook=InvoiceManager.date_hook)
                 file.close()

Any hint to what else I could do to have this file written properly?

Thanks!

Hmm. I don't think this is the cause of the problem, but you don't need to close files if you're using the with pattern -- the code

with open(file, "w") as f:
    f.write("stuff")

...is equivalent to

f = open(file, "w")
try:
    f.write("stuff")
finally:
    f.close()

So right now, you're closing the file twice. Like I say, I don't think that's the cause of the problem, but could you try removing the extra close and see if that helps?

I'll check, thanks!

I just had the same problem when trying to insert a record into a tinydb database, which essentially is also just a json file. I feel that flush() and fsync() have no effect on the server here …

Hmm -- I can imagine flush() and fsync() doing unexpected things, but either way, when you close the file it should definitely have the content.

How are you reading it to check that stuff has been written? Are you (for example) checking immediately? Or are you checking sometime after? I ask because although as soon as the file is closed, the contents should be there, I can imagine a setup where if you were to read the file from one server (say, where your website is running) a millisecond or so after it was written from another (saw, in a scheduled task) then you might see an empty file.

But if you're seeing the kind of problem where you write to the file, then check a few seconds later, and the content isn't there, then it's clearly not that.

One other possibility -- are you sure you're checking the same file as you're writing to? If the filename you're writing to is a relative path (say, "mydata.json") then the specific file written to will depend on the current working directory that the executing program is using -- it won't default to the directory containing the Python script that contains the code.

The file is read entirely independently from writing and the file is the same. I'll have to do some more logging to make entirely sure that for some reason no old content is written to the file. I'll let you know what I'll find out!

Thanks again for the fast and elaborate help! Happy customer, Thomas

OK -- looking forward to hearing what you find out!