Forums

how i can use Beautifulsoup for external urls for scraping the content,,,i am not getting the correct output,

need it urgent

Could you give an example of what you're getting? Perhaps a copy/paste of the code and the errors you're getting?

from django.shortcuts import render
from bs4 import BeautifulSoup
import urllib

def home(request):
    message = request.GET.get('txtweb-message')
    str2 ="mys"
    str = 'http://www.cleartrip.com/trains/stations/BDVT'
    str1 = str + str2
    url = urllib.urlopen(str)
    soup = BeautifulSoup(url)

    #tag=soup.body.string
    #soup.body.string
    tag=soup.get_text()
    #st = soup.td.string
    #st1 = soup.td.strin
    #ta = soup.find('table')
    #tag=soup.findAll('title')
    #tag_a=tag[0].find('title')
    #t = soup.find("div",{"class":"stFixBody"})
    table = soup.find( "table", {"class":"results"} )
    tag1 ='hi'
    tag2 =soup.find_all('tr')
    for r in tag2:
      cols=r.find_all('td')
      for row in range(0,3):
         tag1 +='  '
         tag1 += cols[row].text
    #tag4 = tag2[0].text

    return render(request, "home.html", {'hello':tag1,'hi':message ,'tag':'have a nice day','text':tag})

i am getting output as follows:

hi None have a nice day Access Denied Access Denied Access to arbitrary websites is only available to Premium users. You can sign up for a premium account at http://www.pythonanywhere.com/account/ Alternatively, if you want to suggest something to add to our whitelist (http://www.pythonanywhere.com/whitelist) drop us a line at support@pythonanywhere.com

Hello: your output tells you the problem.

Only premium users (like me!) get unfettered access to all Internet sites from PythonAnywhere. The reason, I believe, is that some users had been using the service for bad stuff (TM), and doing denial of service attacks and the like.

So, if you want to access sites that aren't on the whitelist, you'll need to pay up Thanks

Thank u sir,,But i can nt afford it being as a poor fellow. I need that for my college project..can u tel me someother free servers for deploying my django those are easy like pythonanywhere.com to operate

Hmmm... I'm just a user.

If I were you, I'd sent a nice email to the developers asking very nicely.

i hav done that,,

It looks like cleartrip.com has an API: http://www.cleartrip.com/api/,

We're happy to add their API domains to our site. You'll have to re-write your code to use the API instead of scraping the front page, but I think the cleartrip.com people will appreciate that.

Adding api.cleartrip.com and api.staging.cleartrip.com to the whitelist now...

That's done now.

Thanks a lot sir,,I ill never forget ur help sir..... thank u so much sir....

i just want to scrap the frontpage,,, so i just want to open that url using Beautifulsoup.I need that only sir,, I dont know how can i do that??

What is it you're trying to do? Is there something on the front page that you can't get via the API?