Forums

Hi! I have a python script that fetches a file from google drive using the API. When I run it from the console, it works OK, but when executing it from an automated task (using the interpreter from the virtual env - /home/pfisnqn/.virtualenvs/venv/bin/python /home/pfisnqn/webprofe/extract.py) it doesn't go as planned and I can't make the links from the task logs...

[?1049l [?1l>[?1049h(B[?7h[?1h=(B(BGetting https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=(B Looking up proxy.server:3128(B Making HTTP connection to proxy.server:3128(B Sending HTTP request.(B HTTP request sent; waiting for response.(B Verified connection to accounts.google.com (cert=accounts.google.com)(B Certificate issued by: /C=US/O=Google Trust Services/CN=GTS CA 1O1(B Secure 128-bit TLS1.2 (ECDHE_ECDSA_AES_128_GCM_SHA256) HTTP connection(Bnding HTTP request.(B HTTP request sent; waiting for response.(B/1.0 302 Moved Temporarily(B accounts.google.com cookie: __Ho=1:T-OXvDXZG9GnBw16- Allow? (Y/N/Always/neVer)(B[?1049h(B[?7h[?1h=(B(BGetting https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=(B Looking up proxy.server:3128(B Making HTTP connection to proxy.server:3128(B Sending HTTP request.(B/bin/bash: line 1: 1236589 Killed python /bin/run_scheduled_task.py '/home/pfisnqn/.virtualenvs/venv/bin/python /home/pfisnqn/webprofe/extract.py'

HTTP request sent; waiting for response.(B
Verified connection to accounts.google.com (cert=accounts.google.com)(B
Certificate issued by: /C=US/O=Google Trust Services/CN=GTS CA 1O1(B
Secure 128-bit TLS1.2 (ECDHE_ECDSA_AES_128_GCM_SHA256) HTTP connection(Bnding HTTP request.(B
HTTP request sent; waiting for response.(B/1.0 302 Moved Temporarily(B
accounts.google.com cookie: __Ho=1:hbmrMKdSzLAum2gQM Allow? (Y/N/Always/neVer)(B
2020-10-14 15:08:10 -- Completed task, took 10445.11 seconds, return code was 137.

I'd appreciate any help in making this work as if I ran it manually from the console... Cheers

I think your task is being killed because it has been running for too long -- with a free account, your scheduled tasks can only run for two hours (roughly -- the system that checks for tasks that have been running for too long only runs every half-hour or so, so you may get longer than that). Paid accounts can have tasks that run for up to 12 hours.

I believe it runs for that long because is awaits for user input at some point (eg: (Y/N/Always/neVer) probably to authenticate using a mobile device, but I'm just guessing)... I have properly authenticated from the virtual environment using that same script (that's why it runs ok when manually executed)... It shouldn't take more than a couple of minutes to run if everything is fine. Also, sorry for doubling the output on the first post..

EDIT: Could it be (from what I've just googled) that lynx pops up on the console running the script and asks how to handle that cookie and it stalls waiting for that input??

That does look like it's trying to get you to authenticate in a browser in the task. Check the documentation of the library you're using to see how you can re-use the authentication credentials that you've created in an interactive run of the same code where you could authenticate in the browser or see if there's a different form of authentication you can use that does not involve use input.

I think it should be using the same credentials and tokens existing in the same directory... is there any difference between running the script manually from a bash console and running it from the task using the venv interpreter?

It might be a problem with the working directory -- if you're specifying a file containing the credentials as something like "credentials.txt" as opposed to "/home/pfisnqn/something/credentials.txt" then that could certainly lead to problems like this. This help page has some suggestions about how to avoid problems like that, in particular in the section "Tip: file for cross-platform scripts".

Wow, that makes more sense. I'm gonna review my script and check how I'm accessing the credentials file. Thanks a lot, if you don't mind, I might use this same thread if anything else comes up.

UPDATE: That did the trick! thanks a lot! I was using os.path.join(os.path.abspath(os.getcwd(), which in retrospective could have picked up any dir, even while using the venv's interpreter.

Excellent, glad to hear you got it working! Yes, the code using os.getcwd wouldn't help, as it's explicitly using the (potentially unpredictable) current working directory. Much better to use the magic __file__ variable.