Hi, I'm trying to run a daily task to scrape some data from the web. When I run my script manually it works but when I try to schedule it I get file not found error or no output. Example of the script is below.
First I tried to schedule the script like this:
but got 'no such file or directory error:
bash: /home/rainermesi/.virtualenvs/web_scrape_venv/bin/pyhton: No such file or directory 2020-07-31 07:54:06 -- Completed task, took 2.53 seconds, return code was 127.
Then I searched the forum and tried adding a hashbang in front of the command to run the script in a venv where I have the required modules:
#! /home/rainermesi/.virtualenvs/web_scrape_venv/bin/pyhton3.7 home/rainermesi/Documents/f_min_scrape/scrape_min.py
This makes the script run but the script produces no output, not even the print() command at the end of the script:
2020-07-31 07:38:08 -- Completed task, took 3.00 seconds, return code was 0.
The script itself is something like this. I tried using absolute paths, but that did not help either:
import requests from lxml import html import pandas as pd import os #chdir os.chdir(os.path.abspath(os.path.dirname(__file__))) #functions def scrape_data() def write_data(): dataframe.to_csv(str(os.path.abspath(os.path.dirname(__file__)))+'/filename.csv') scrape_data() write_data() print('Script ran successfully')
I feel like I am missing something obvious but can't figure out what I am doing wrong. Can anyone suggest a solution?