Forums

On my computer I have a webscraping script that uses requests and Beautiful Soup to save information to a csv file. At the moment I have to manually run it on my own machine, or schedule it to do so. I want to run it on a cloud so I don't have to keep my machine on, or remember to turn it on and run the script.

I'm not sure what framework I used. I wrote it in Python IDLE. I have a script saved as a file with a .py extension that spits out a CSV when I run it.

I'm not sure what is meant by 'web app'. Sorry. Ditto 'framework'.

The site I'm scraping isn't on the white list. I realise my free account wouldn't be able to run it.

Long story short, how do I write a python script and run it? My first step is to use requests to get something from a whitelisted site, and Beautiful Soup to parse it, where do I put my code on the site and how do I run it?

Thanks so much for your help, I hope my questions aren't too simple, or my post too rambling!

That should be pretty easy to set up :-) Web apps are only needed if you're planning to host a website (for example, at https://maxiewawa.pythonanywhere.com), and you'd normally use a framework like Django or Flask if you were writing one. For a simple script that doesn't have its own website -- that instead accesses other existing websites -- then you don't need to worry about that stuff.

So, for what you need to do, just upload the file, then run it from a bash console to make sure that it works as expected -- for example, if you upload it to your home directory as myscript.py and it uses Python 3.5, then you would run it using the command

python3.5 myscript.py

Once you've done that, and it's run without errors (if it produces errors, just copy/paste them into a post on this forum thread and we'll help), then you can go to the "Schedule" tab. There you can set it up to run once a day -- free accounts can schedule one script to run automatically once a day, paid accounts have more options. Just enter the same command, python3.5 myscript.py into the input field on the "Schedule" tab, choose a time to run it, and you should be all set.

Am I doing something wrong? My test code gives me a csv with the headline of a whitelisted site on my computer, but a csv with "Access Denied" when running through pythonanywhere.com.

import requests
from bs4 import BeautifulSoup
import csv
res = requests.get('http://www.stroeder.com/')
soup = BeautifulSoup(res.text, 'html.parser')
h1 = soup.h1.text
def write_csv(filename, data):
    with open(filename, 'w') as f:
        writer = csv.writer(f)
        writer.writerows(data)
        f.close()

write_csv('test.csv',[[h1]])

EDIT: The same script with 'http://www.boddie.org.uk' results in a message about the site not being on the whitelist... it is though! I must be doing <i>something</i> right... EDIT2: http://svcs.ebay.com is working! Strange....

www.stroeder.com is not on the whitelist. svcs.ebay.com is.

I am trying to run a simple telethon script that adds members from members of groups i scraped in my local machine as a csv file into my group chat but i encountering the following error.

21:02 ~/Villians TV/Villians5/TeleGram-Scraper-master $ python3 villians5.py

Traceback (most recent call last): File "villians5.py", line 17, in <module> client.connect() File "/home/somti13/.local/lib/python3.8/site-packages/telethon/sync.py", line 39, in syncified return loop.run_until_complete(coro) File "/usr/lib/python3.8/asyncio/base_events.py", line 608, in run_until_complete return future.result() File "/home/somti13/.local/lib/python3.8/site-packages/telethon/client/telegrambaseclient.py", line 472, in connect if not await self._sender.connect(self._connection( File "/home/somti13/.local/lib/python3.8/site-packages/telethon/network/mtprotosender.py", line 125, in connect await self._connect() File "/home/somti13/.local/lib/python3.8/site-packages/telethon/network/mtprotosender.py", line 250, in _connect raise ConnectionError('Connection to Telegram failed {} time(s)'.format(self._retries)) ConnectionError: Connection to Telegram failed 5 time(s)

Free accounts will not be able to use an mtproto connection to telegram. Check the docs for telethon to see how you can make it use an http connection and that should work in a free account.

import PyPDF2

pdfFileObject = open(r"D:\pdf.pdf", 'rb')

pdfReader = PyPDF2.PdfFileReader(pdfFileObject)

print(" No. Of Pages :", pdfReader.numPages)

pageObject = pdfReader.getPage(0)

print(pageObject.extractText())

pdfFileObject.close()

how to run this?

You can put it in a file with a ".py" extension and run it directly from the editor, or you can enter the lines into a Python console.

Hello, i am trying to run an instagram bot made with instapy. But i keep getting this error: Traceback (most recent call last): File "/home/nat/i_chatbot.py", line 1, in <module> from instapy import InstaPy ModuleNotFoundError: No module named 'instapy

You need to install the module you're trying to use into the version of Python/virtualenv that you're using to run the code: https://help.pythonanywhere.com/pages/InstallingNewModules/

I deleted the 'flask_app.py' because I already have my python script on another folder, so I have been trying to connect it to be able to see my own python script but i keep getting this message.

Error running WSGI application 2021-01-04 22:34:59,846: ImportError: cannot import name 'Flask' 2021-01-04 22:34:59,846: File "/var/www/www_thepurge_ng_wsgi.py", line 16, in <module> 2021-01-04 22:34:59,847: from flask_app import app as application # noqa 2021-01-04 22:34:59,847: 2021-01-04 22:34:59,847: File "/home/thepurge/mysite/flask_app.py", line 4, in <module> 2021-01-04 22:34:59,847: from flask import Flask 2021-01-04 22:34:59,847: ********* 2021-01-04 22:34:59,847: If you're seeing an import error and don't know why, 2021-01-04 22:34:59,847: we have a dedicated help page to help you debug: 2021-01-04 22:34:59,847: https://help.pythonanywhere.com/pages/DebuggingImportError/

Please, how do I get this resolved, Its urgent.

Thanks!

Have you tried https://help.pythonanywhere.com/pages/DebuggingImportError/?

I don't really understood what I am reading at https://help.pythonanywhere.com/pages/DebuggingImportError/ please can you just help me out so I can get this work done, Its urgent. Thanks!

Your error message is this:

 cannot import name 'Flask'

I see that you are using a virtualenv. Have you installed Flask into the virtualenv? You can start a Bash console inside the env from the "Web" page, and then use "pip install Flask" to install it.

Looking in links: /usr/share/pip-wheels Requirement already satisfied: Flask in /usr/local/lib/python2.7/dist-packages (1.1.1) Requirement already satisfied: itsdangerous>=0.24 in /usr/local/lib/python2.7/dist-packages (from Flask) (1.1.0) Requirement already satisfied: Jinja2>=2.10.1 in /usr/local/lib/python2.7/dist-packages (from Flask) (2.10.3) Requirement already satisfied: click>=5.1 in /usr/local/lib/python2.7/dist-packages (from Flask) (7.0) Requirement already satisfied: Werkzeug>=0.15 in /usr/local/lib/python2.7/dist-packages (from Flask) (0.16.0) Requirement already satisfied: MarkupSafe>=0.23 in /usr/lib/python2.7/dist-packages (from Jinja2>=2.10.1->Flask) (0.23)

this was what i got back, It shows that I already have it installed. The fact is that I already have my own .py running my python codes and that is exactly what I want to use to run my site, is called "blackmirror.py". My problem is that it is not connecting to it, rather it is connecting to flask_app.py which i have already deleted. I really need to get this done as quick as possible, please help me out

Is your web app running in the same venv where Flask is installed?

Should I keep your website tab open to run my code or i can close it ?

you can close PythonAnywhere tabs-- any code will continue to run