Forums

Selenium error

Hi, I'm using selenium to take a screenshot. It was working last night but today I'm getting an error...help?

My code:

display = Display(visible=0, size=(800, 600))
display.start()

browser = webdriver.Firefox()
browser.get('http://www.work-points.com/report_chart')
browser.save_screenshot('/home/kyspy/esa-cm5/static/img/report.jpg')

browser.quit()
display.stop()

The error:

2014-03-06 16:51:22,657 :Exception on /report_as_pdf/3 [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1687, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1360, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1358, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1344, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python2.7/dist-packages/flask_login.py", line 515, in decorated_view
    return fn(*args, **kwargs)
  File "/home/kyspy/esa-cm5/views.py", line 366, in report_as_pdf
    browser = webdriver.Firefox()
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.py", line 61, in __init__
    self.binary, timeout),
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/extension_connection.py", line 47, in __init__
    self.binary.launch_browser(self.profile)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.py", line 51, in launch_browser
    self._wait_until_connectable()
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.py", line 90, in _wait_until_connectable
    self._get_firefox_output())
WebDriverException: Message: "The browser appears to have exited before we could connect. The output was:     \n(process:7397): GLib-CRITICAL **: g_slice_set_config: assertion 'sys_page_size == 0' failed\nError: cannot open display:     :1358\n"
deleted-user-129847 | 8 posts | March 6, 2014, 4:55 p.m. | permalink

That sounds odd. Perhaps it's worth waiting a bit and retrying, just in case it's taking a while for the display to start? eg.

import time

display = Display(visible=0, size=(800, 600))
display.start()

browser = None
for retry in range(5):
    try:
        browser = webdriver.Firefox()
    except:
        time.sleep(5)

if browser is not None:
    browser.get('http://www.work-points.com/report_chart')
    browser.save_screenshot('/home/kyspy/esa-cm5/static/img/report.jpg')
    browser.quit()
else:
    # do some error handling

display.stop()
Staff giles | 12074 posts | PythonAnywhere staff | March 6, 2014, 6:13 p.m. | permalink

Good idea, but still not working...I left out the "if browser is not None:" statement

2014-03-06 18:54:28,259 :Exception on /report_as_pdf/3 [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1687, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1360, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1358, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1344, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python2.7/dist-packages/flask_login.py", line 515, in decorated_view
    return fn(*args, **kwargs)
  File "/home/kyspy/esa-cm5/views.py", line 375, in report_as_pdf
    browser.get('http://www.work-points.com/report_chart')
AttributeError: 'NoneType' object has no attribute 'get'
deleted-user-129847 | 8 posts | March 6, 2014, 6:55 p.m. | permalink

OK. Where are you running this? That is, is it in a console, or a web app, or a scheduled task?

Staff giles | 12074 posts | PythonAnywhere staff | March 6, 2014, 7:55 p.m. | permalink

It's in my web app, in views.py. I'm using Flask.

deleted-user-130102 | 1 post | March 6, 2014, 7:56 p.m. | permalink

I guess you have two accounts, kyspy and OlegMoshkovich?

I can see that there are a bunch of Xvfbs and Firefoxes running on the web server your app is on as kyspy, one started on 3 March and a bunch from today. It's possible that you've run out of space because sometimes they're not being cleaned up -- that would explain why it worked yesterday and stopped today.

I suggest I stop your app temporarily, kill all of the extra processes owned by kyspy, then restart it. Is that OK?

Staff giles | 12074 posts | PythonAnywhere staff | March 6, 2014, 8:05 p.m. | permalink

Sorry...I'm kyspy...I just was on Oleg's computer for a minute there :)

Yes, please kill the processes. How do I prevent this in the future?

deleted-user-129847 | 8 posts | March 6, 2014, 8:08 p.m. | permalink

Ah, that would explain it :-)

I've killed the extra processes, could you try again now?

The best way to prevent this kind of problem is to make sure that you always quit() all of your browsers and stop() all of your displays, even if there's an error -- just make sure there's appropriate try...finally code everywhere.

One other thing I recommend for anyone using selenium is to keep an eye on temporary files -- it has a bad habit of filling up /tmp with stuff that will eventually suck up all of your quota.

Staff giles | 12074 posts | PythonAnywhere staff | March 6, 2014, 8:20 p.m. | permalink

Great, it's working again!

Just to make sure I understand -- I'll use the code you've provided above and that should help with preventing the extra processes, correct?

Also, I'm using selenium to take a screenshot of a javascript chart so I can output it on a pdf...I've read that phantomjs would be another way to do this, however, from a cursory review of the forums it looks like phantomjs is not supported on pythonanywhere. Is that correct?

Thanks!

deleted-user-129847 | 8 posts | March 6, 2014, 8:31 p.m. | permalink

Excellent, I'm glad it's working now :-)

The code I gave earlier wouldn't shut stuff down properly (probably should have put that in, I don't want to get people into bad habits!). This should do the job:

display = Display(visible=0, size=(800, 600))
try:
    display.start()

    browser = webdriver.Firefox()
    try:
        browser.get('http://www.work-points.com/report_chart')
        browser.save_screenshot('/home/kyspy/esa-cm5/static/img/report.jpg')
    finally:
        browser.quit()
finally:
    display.stop()

You're right, phantomjs doesn't work on PythonAnywhere right now.

Staff giles | 12074 posts | PythonAnywhere staff | March 6, 2014, 8:41 p.m. | permalink

Great, super helpful!

deleted-user-129847 | 8 posts | March 6, 2014, 8:46 p.m. | permalink