Forums

A few days ago, a script for a scheduled task that worked with no problems for months, stopped working. I tried to understand what's going on and added comments to my script. So, now it goes like this

print("Start")

import time
from IPython.display import display, Markdown
from datetime import datetime, timedelta, date
from pathlib import Path
import argparse
import yaml

print("Import 1")

import shutil
import sys
import os

print("Import 2")

# Change the current directory to where the script is located.
script_dir = os.path.dirname(os.path.abspath(__file__))
# Change the current working directory
os.chdir(script_dir)

print(f"Current working directory: {os.getcwd()}")

# Add the parent directory of the current directory to Python path
parent_dir = os.path.dirname(os.getcwd())
if parent_dir not in sys.path:
    sys.path.append(parent_dir)

from utils.ai_wrapper import AIWrapper
from utils.news_scraper import NewsReader, NewsConfig, NewsListFilter, ImageDataExtractor, DateTimeHandler, NavigationManager

etc.

But then in the log I am getting:

WARNING:utils.ai_wrapper:Attempt 1 failed: Error code: 400 - {'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'messages.0: all messages must have non-empty content except for the optional final assistant message'}}
WARNING:utils.ai_wrapper:Attempt 2 failed: Error code: 400 - {'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'messages.0: all messages must have non-empty content except for the optional final assistant message'}}
ERROR:utils.ai_wrapper:Final attempt failed after 3 retries. Error: Error code: 400 - {'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'messages.0: all messages must have non-empty content except for the optional final assistant message'}}
Start
Import 1
Import 2
Current working directory: /home/id/folder

So

1. I am getting errors prior to the very first line of code (print)
2. Errors and warnings are for utils.ai_wrapper, which is not yet imported
3. The code ends up failing at the end.

When I am running the same command from the console, it works fine.

Can someone please help me to understand what is going on? I spent 3 days with not much progress. One thing I learned though is that the scheduler runs my script as

python /bin/run_scheduled_task.py 'python3.10 myscript.py'

But that command also works alright. I am puzzled. I would appreciate any help!

Could you add flush=True to the print statements? What is utils.ai_wrapper trying to do? Is it your code or is it an installed dependency? Is it trying to connect to some external service?

Thank you for the response! I will add flush=True to the print statements, this is a good idea.

What is utils.ai_wrapper trying to do?

This is my own abstraction layer that allows to make calls to different AI engines (OpenAI, Anthropic etc.) in the same way. In my code I can then use yaml to configure different types of calls - without making changes to the code. Here is the method that is failing:

def get_completion(self, sysprompt, prompt, temperature=0.65, quality="cheap", engine="openai", maxtokens = 1500):

Essentially, it defines which model to use, sets parameters and then passes sysprompt and prompt to the right API "as is".

> Is it trying to connect to some external service?

Yes, that is correct. It has some logging, which shows that, a) this class gets initialized (before the "import" statement?!), b) the "prompt" parameter is empty, so, OpenAI API returns

"Error code: 400 - {'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'messages.0: all messages must have non-empty content except for the optional final assistant message'}"

My method captures this exception and displays the error "as is". The class also has a wrapper that, in case of a failure, makes 2 more attempts to call the API again, therefore we see 3 error messages.

I have a Task that runs the same script every 24 hours. It worked up until January 30th, then something broke. I am wondering whether PythonAnywhere had some deployment around that date. In any case, the scheduled tasks keep failing. I am re-running the same task manually, and it works alright, although, I saw a similar error once, but the next run was OK, and I didn't see it ever again, so I can't repro it.

Thanks again!

We haven't made any system updates yet this year, so I don't think that could be the cause of any of this.

I think that the first thing to try is to add the flush kwarg to your prints -- I think that will sort out the out-of-order logging that you're seeing. So, for example, you'd change this:

print("Start")

...to this:

print("Start", flush=True)

If that means that things start coming out in an order that makes sense, the next thing would be to add some similar prints inside the code that is currently generating those errors to find out what it's sending to the API when it's getting those responses.

Thank you very much! I made the change, will see what is going to happen next time the task runs. One question related to updates and a potential root cause(s). A part of my code uses Chromium, below is the init method, just for reference. My understanding is that Chromium is a part of the environment, and I am using it "as is". Do you know if there was any change/update to the default Chromium version close to 31-Jan 2025 08:54:13 (that is, server time)? I want to rule out Chromium timeouts as a potential cause of the failures.

def init_driver(self) -> None:
    """Initialize the Chrome WebDriver with specified options."""
    chrome_options = Options()
    chrome_options.add_argument('--no-sandbox')
    chrome_options.add_argument("--headless")
    chrome_options.add_argument('--disable-dev-shm-usage')
    chrome_options.add_argument('--disable-gpu')
    chrome_options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36")
    self.driver = webdriver.Chrome(options=chrome_options)

Thanks again!

There weren't any updates on our side.