selenium webdriver : Forums : PythonAnywhere

selenium webdriver

Hey there,

I was trying to get the selenium webdriver to run on my web server (running on python 2.7). My code looks like this

display = Display(visible=0, size=(800, 600))
display.start()
browser = webdriver.Firefox()
return "got here"

I checked the forum for similar problems, and I have also clear the temp folder as instructed. However, I'm still running on this problem

check_installed
    raise EasyProcessCheckInstalledError(self)
EasyProcessCheckInstalledError: cmd=['Xvfb', '-help']
OSError=[Errno 11] Resource temporarily unavailable
Program install error! 
You can install it in terminal:
sudo apt-get install xvfb

Any help would be greatly appreciated

deleted-user-574652 | 12 posts | Jan. 4, 2015, 1:25 a.m. | permalink

Your code doesn't close the web driver when it's finished, so you've got processes hanging around. They may have been cleaned up in the meantime by our automated processes.

glenn | 9718 posts | PythonAnywhere staff | Jan. 4, 2015, 5:31 p.m. | permalink

Thanks glenn for the help. I tried the your recommendation in the following code. But whenever I ran the code

browser = None
display = Display(visible=0, size=(800, 600))

try:
    display.start()
    browser = webdriver.Firefox()
finally:
    if (browser != None):
        sourceCode = browser.get(url).page_source
        browser.quit()
        display.stop()
        return "got here"
    else:
        sourceCode = requests.get(url).text
        display.stop()
        return "got there"

it always return "got there". If i removed the 'try' and 'finally' block. Even on closing of the display and browser, I still face the same problem. Any suggestions?

deleted-user-574652 | 12 posts | Jan. 5, 2015, 3:34 a.m. | permalink

Hi there, just stepping in while Glenn's away. Is there anything in your webapp's error logs?

harry | 2710 posts | PythonAnywhere staff | Jan. 5, 2015, 12:05 p.m. | permalink

Hi minhthao,

I see that you still have a process running that uses xvfb/pyvirtualdisplay. Maybe you could try killing that and then rerunning it? (ie. in the consoles tab, fetch all the running processes)

Conrad

conrad | 4232 posts | PythonAnywhere staff | Jan. 5, 2015, 12:25 p.m. | permalink

Hi guys, I closed all the processes and still running into the same problem. Because of try-finally block, I did not get any error log. However, if I don't use the try/finally block, I got the same error as the first post - which I will copy here for the references.

check_installed
    raise EasyProcessCheckInstalledError(self)
EasyProcessCheckInstalledError: cmd=['Xvfb', '-help']
OSError=[Errno 11] Resource temporarily unavailable
Program install error! 
You can install it in terminal:
sudo apt-get install xvfb

Thanks

deleted-user-574652 | 12 posts | Jan. 5, 2015, 3:08 p.m. | permalink

OSError=[Errno 11] Resource temporarily unavailable is what we normally see when a user has exceeded the max allowed number of processes on a particular machine.

Try amending the try/finally block to print a traceback:

import traceback
browser = None
display = Display(visible=0, size=(800, 600))

try:
    display.start()
    browser = webdriver.Firefox()
    print(browser.get(url).page_source)

except Exception as e:
    print('caught exception:')
    print(e)
    traceback.print_exc()

finally:
    if browser is not None:
        browser.quit()
    display.stop()

harry | 2710 posts | PythonAnywhere staff | Jan. 5, 2015, 3:32 p.m. | permalink

Thanks Harry, this is the most recent message that I got from the traceback

2015-01-05 15:39:00,852 :Traceback (most recent call last):
2015-01-05 15:39:00,853 :  File "/home/minhthao/easyswap/property.py", line 320, in getPropertyInfo
2015-01-05 15:39:00,855 :    browser = webdriver.Firefox()
2015-01-05 15:39:00,855 :  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.py", line 61, in __init__
2015-01-05 15:39:00,856 :    self.binary, timeout),
2015-01-05 15:39:00,856 :  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/extension_connection.py", line 47, in __init__
2015-01-05 15:39:00,856 :    self.binary.launch_browser(self.profile)
2015-01-05 15:39:00,856 :  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.py", line 51, in launch_browser
2015-01-05 15:39:00,857 :    self._wait_until_connectable()
2015-01-05 15:39:00,857 :  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.py", line 90, in _wait_until_connectable
2015-01-05 15:39:00,857 :    self._get_firefox_output())
2015-01-05 15:39:00,857 :WebDriverException: Message: 'The browser appears to have exited before we could connect. The output was: \n(process:12004): GLib-CRITICAL **: g_slice_set_config: assertion \'sys_page_size == 0\' failed\nXlib:  extension "RANDR" missing on display ":1167".\n*** LOG addons.xpi: startup\n*** LOG addons.xpi: checkForChanges\n*** LOG addons.xpi-utils: Opening database\n*** LOG addons.xpi: No changes found\n'

deleted-user-574652 | 12 posts | Jan. 5, 2015, 3:40 p.m. | permalink

Interesting. That extension "RANDR" missing on display message looks like it might be relevant. Googling for it might help, here's the first thing I found: https://stackoverflow.com/questions/17944234/xlib-extension-randr-missing-on-display-21-trying-to-run-headless-googl

I'm also not sure whether that visible=0 line is necessary or harmless...

harry | 2710 posts | PythonAnywhere staff | Jan. 5, 2015, 3:44 p.m. | permalink

Incidentally, I just started up a display and a webdriver.Firefox() myself, and it worked fine. But I didn't use that visible=0 thing. So I would try removing that.

harry | 2710 posts | PythonAnywhere staff | Jan. 5, 2015, 3:48 p.m. | permalink

It still produce the same traceback after the removing the visible=0 item. Still extension "RANDR" missing on display appeared.

2015-01-05 16:01:33,291 :Traceback (most recent call last):
2015-01-05 16:01:33,292 :  File "/home/minhthao/easyswap/property.py", line 320, in getPropertyInfo
2015-01-05 16:01:33,294 :    browser = webdriver.Firefox()
2015-01-05 16:01:33,294 :  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.py", line 61, in __init__
2015-01-05 16:01:33,294 :    self.binary, timeout),
2015-01-05 16:01:33,294 :  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/extension_connection.py", line 47, in __init__
2015-01-05 16:01:33,295 :    self.binary.launch_browser(self.profile)
2015-01-05 16:01:33,295 :  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.py", line 51, in launch_browser
2015-01-05 16:01:33,295 :    self._wait_until_connectable()
2015-01-05 16:01:33,295 :  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.py", line 90, in _wait_until_connectable
2015-01-05 16:01:33,295 :    self._get_firefox_output())
2015-01-05 16:01:33,295 :WebDriverException: Message: 'The browser appears to have exited before we could connect. The output was: \n(process:12953): GLib-CRITICAL **: g_slice_set_config: assertion \'sys_page_size == 0\' failed\nXlib:  extension "RANDR" missing on display ":1218".\n*** LOG addons.xpi: startup\n*** LOG addons.xpi: checkForChanges\n*** LOG addons.xpi-utils: Opening database\n*** LOG addons.xpi: No changes found\n'

deleted-user-574652 | 12 posts | Jan. 5, 2015, 4:05 p.m. | permalink

I don't know if this would helps. But my server is running python 2.7. However, before that, my server was implemented using python 3.4. However, after I found out that 'python 3.4' did not come with selenium, I decided to delete that webserver, and override in the same folder python 2.7, instead of creating a virtualenvir. Could that some how be the case?

Best

deleted-user-574652 | 12 posts | Jan. 5, 2015, 4:09 p.m. | permalink

OK let's strip it back down to a minimal repro. Create a new Python file, with the following code:

from pyvirtualdisplay import Display
from selenium import webdriver
import traceback
browser = None

try:
    display = Display().start()
    browser = webdriver.Firefox()
    browser.get('http://www.google.com')
    print browser.title

except Exception as e:
    traceback.print_exc()

finally:
    if browser is not None:
        browser.quit()
    display.stop()

And hit 'Save & Run'

If it works, then it must be something that's different about your specific code, and then we can try and track that down. If it doesn't work, then it must be something to do with your user account (because it works fine in mine), and we can try and track that down...

harry | 2710 posts | PythonAnywhere staff | Jan. 5, 2015, 4:13 p.m. | permalink

Create a new file with the previous code works fine with me. I guess it must be something with the code then. For the code. I create a webserver with a RESTful-API, so when the server receive a request with the particular url, it goes ahead and try to scrape that url. When i use the function

requests.get(url)

I got everything in the response send out correctly. However, as some of the information is populated using javascript, i have to use selenium to grab that information. And i ran into problem when i tried to do that.

here is the code for my main function

@app.route('***/***/***/<string:pid>', methods=['GET'])
def getProperty(pid):
    property = getPropertyInfo(pid)
    if (property != None):
        return property.getJson()
    else:
        abort(400)

and here is the partial code for the scraping function

 def getPropertyInfo(pid):
    url = urlpath + pid
    browser = None

    try:
        display = Display().start()
        browser = webdriver.Firefox()

    except Exception as e:
        print('caught exception:')
        print(e)
        traceback.print_exc()

    finally:
        if browser is not None:
            browser.quit()
            display.stop()
            #get and return the object from the page source, return simple JSON in test.
        else:
            display.stop()

deleted-user-574652 | 12 posts | Jan. 5, 2015, 5:06 p.m. | permalink