Forums

Hi, I'm trying to run a daily task to scrape some data from the web. When I run my script manually it works but when I try to schedule it I get file not found error or no output. Example of the script is below.

First I tried to schedule the script like this:

/home/rainermesi/.virtualenvs/web_scrape_venv/bin/pyhton
/home/rainermesi/Documents/f_min_scrape/scrape_min.py

but got 'no such file or directory error:

bash: /home/rainermesi/.virtualenvs/web_scrape_venv/bin/pyhton: No such file or directory 
2020-07-31 07:54:06 -- Completed task, took 2.53 seconds, return code was 127.

Then I searched the forum and tried adding a hashbang in front of the command to run the script in a venv where I have the required modules:

1

#! /home/rainermesi/.virtualenvs/web_scrape_venv/bin/pyhton3.7 home/rainermesi/Documents/f_min_scrape/scrape_min.py

This makes the script run but the script produces no output, not even the print() command at the end of the script:

2020-07-31 07:38:08 -- Completed task, took 3.00 seconds, return code was 0.

The script itself is something like this. I tried using absolute paths, but that did not help either:

import requests
from lxml import html
import pandas as pd
import os

#chdir
os.chdir(os.path.abspath(os.path.dirname(__file__)))
#functions
def scrape_data()
def write_data():
    dataframe.to_csv(str(os.path.abspath(os.path.dirname(__file__)))+'/filename.csv')

scrape_data()
write_data()
print('Script ran successfully')

I feel like I am missing something obvious but can't figure out what I am doing wrong. Can anyone suggest a solution?

1. you mis-spelt python.
2. home/rainermesi/Documents/f_min_scrape/scrape_min.py should be /home/rainermesi/Documents/f_min_scrape/scrape_min.py
3. that is not how the #! works. check what you copied from again

Thanks for your help! Yep, I had mistyped 'pyhton'. Also looked it up and had completley misnuderstood '#!'