Forums

Hi, i have a disk quota exceeded error even though I have the required storage space for my files to be created.

I am able to run this code on my own but I i only received this error when I am running it on a scheduled task. This is the full error from the log of my scheduled task.

sys:1: DtypeWarning: Columns (0) have mixed types.Specify dtype option
on import or set low_memory=False. converting data Traceback (most
recent call last): [path of script]    File
"/usr/local/lib/python3.9/site-packages/pandas/core/generic.py", line
3170, in to_csv
    formatter.save()   File "/usr/local/lib/python3.9/site-packages/pandas/io/formats/csvs.py",
line 227, in save
    f.close() OSError: [Errno 122] Disk quota exceeded

2022-08-07 06:02:53 -- Completed task, took 157.04 seconds, return
code was 1.

Can you add some more logging to establish what exactly the task is doing and what is the state of your environment when you get this error?

@pafk

The script reads a csv file using pandas. Once the csv file is converted to a pandas dataframe, I am filtering through the dataframe to extract a certain element given the condition. After filtering, the desired element is appended to a list. The list is then saved as a csv file.

I am not so sure what do you mean by the state of the environment. This is a scheduled task. However, before this task is ran, there were 3 other tasks that had ran successfully. I have also checked and ensured that we have enough computation time so we did not exceed the quota. This scheduled task has been running successfully until a month ago.

Also, I am not running on any virtual environment. I am running this script directly using the python 3.9 interpreter.

How big is the input csv file?

@pafk

The input CSV file is about 50mb.

Is it behaving constantly like that or only sometimes? Is it always the same file (or similar) of 50mb or is it possible that sometimes it's bigger? Are you able to reproduce this when you run the script in the Bash console?

@pafk

It is behaving consistently starting 1 month ago. It is always a similar file of around 50mb.

And I am unable to reproduce this error when i run it via bash or through the python3 interpreter. It runs successfully in both instances.

So you've been getting the "disk quota exceeded" error every time your scheduled task was running for last month, but you never get such error when you run the same code in a Bash console, is that correct?

@pafk

Yes, that is correct. The error is consistently being produced in my scheduled task. While I did not face this error at all when I run it via my bash console.

That suggests 2 possibilities to me:

1. When you're running from the console, you're trying to write to a different file system location than when you're running as a task. That could be because you are using a relative path for the file and the working directory is different in the task than in the console.
2. The code follows a different path in the console and the task for some reason. Add some logging to the code so that you can compare which code is being run in each.

@glenn @pafk

Thank you so much for the help. I have just checked the log and for some reason, my scheduled task started running miraculously again even though I did nothing to the script nor did I make any changes.

That's strange! Anyway, glad to hear it's working again.