Forums

Have a web app utilizing Selenium and have installed geckodriver and included in the executable path. Have a virtualenv set up and everything, however when it gets to the code where I actually try to open up a firefox browser, I get the following error messages from my log:

2017-09-26 17:53:25,585: [2017-09-26 17:53:25,571] ERROR in app: Exception on /results [POST]
2017-09-26 17:53:25,600: Traceback (most recent call last):
2017-09-26 17:53:25,600:   File "/home/ttataryn/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
2017-09-26 17:53:25,600:     response = self.full_dispatch_request()
2017-09-26 17:53:25,600:   File "/home/ttataryn/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
2017-09-26 17:53:25,600:     rv = self.handle_user_exception(e)
2017-09-26 17:53:25,600:   File "/home/ttataryn/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
2017-09-26 17:53:25,600:     reraise(exc_type, exc_value, tb)
2017-09-26 17:53:25,600:   File "/home/ttataryn/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
2017-09-26 17:53:25,601:     raise value
2017-09-26 17:53:25,601:   File "/home/ttataryn/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
2017-09-26 17:53:25,601:     rv = self.dispatch_request()
2017-09-26 17:53:25,601:   File "/home/ttataryn/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
2017-09-26 17:53:25,601:     return self.view_functions[rule.endpoint](**req.view_args)
2017-09-26 17:53:25,601:   File "/home/ttataryn/mysite/flask_app.py", line 29, in results
2017-09-26 17:53:25,601:     driver = webdriver.Firefox(firefox_binary = binary)
2017-09-26 17:53:25,601:   File "/home/ttataryn/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/selenium/webdriver/firefox/webdriver.py", line 103, in __init__
2017-09-26 17:53:25,601:     self.binary, timeout)
2017-09-26 17:53:25,602:   File "/home/ttataryn/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/selenium/webdriver/firefox/extension_connection.py", line 51, in __init__
2017-09-26 17:53:25,602:     self.binary.launch_browser(self.profile, timeout=timeout)
2017-09-26 17:53:25,602:   File "/home/ttataryn/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/selenium/webdriver/firefox/firefox_binary.py", line 67, in launch_browser
2017-09-26 17:53:25,602:     self._start_from_profile_path(self.profile.path)
2017-09-26 17:53:25,602:   File "/home/ttataryn/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/selenium/webdriver/firefox/firefox_binary.py", line 90, in _start_from_profile_path
2017-09-26 17:53:25,602:     env=self._firefox_env)
2017-09-26 17:53:25,602:   File "/usr/lib/python3.6/subprocess.py", line 707, in __init__
2017-09-26 17:53:25,602:     restore_signals, start_new_session)
2017-09-26 17:53:25,602:   File "/usr/lib/python3.6/subprocess.py", line 1326, in _execute_child
2017-09-26 17:53:25,603:     raise child_exception_type(errno_num, err_msg)
2017-09-26 17:53:25,603: PermissionError: [Errno 13] Permission denied

Anyone?

Unfortunately the only version of Firefox that will work on PythonAnywhere is the version that we currently have installed -- this help page explains how to use it.

Thanks for the reply. So I already have Selenium version 2.53.2, so I should be good right? Do i have to do anything else with Firefox? I also have geckodriver in my path. Should I remove that?

Update: I installed xvfbwrapper and was finally able to get a different error:

raise WebDriverException("The browser appears to have exited " 2017-09-27 15:34:06,369: selenium.common.exceptions.WebDriverException: Message: The browser appears to have exited before we could connect.

I realize I must use "sudo apt-get install xvfb" in order to avoid this error. However I have realized we can not used the commands "sudo apt-get". Is there a solution to this?

Thanks

Update: I installed xvfbwrapper and was finally able to get a different error:

raise WebDriverException("The browser appears to have exited " 2017-09-27 15:34:06,369: selenium.common.exceptions.WebDriverException: Message: The browser appears to have exited before we could connect.

I realize I must use "sudo apt-get install xvfb" in order to avoid this error. However I have realized we can not used the commands "sudo apt-get". Is there a solution to this?

Thanks

You don't need to install anything, everything should be there already. Are you working in a virtualenv?

Yeah I should be working in a virtual env. I included the path to it. Anything else I need to do?

OK. Could you install selenium== 2.53.6 into it, and then try the code on this help page just so that we can make sure everything is working? Also, in answer to your earlier question, yes, you should remove geckodriver from your path.

I tried installing selenium, and did use that help page. However, I am now getting this error: selenium.common.exceptions.WebDriverException: Message: The browser appears to have exited before we could connect. If you specified a log_file in the FirefoxBinary constructor, check it for details.

Thanks for the help

what is the code that you are running, and are you running this from within a webapp or from a console or from a scheduled task?

I am running it from within a web app. Would you like to see the actual code?

sure- if you give us the file location and explicit permission we can go take a look at it.

Thank you. https://www.pythonanywhere.com/user/ttataryn/shares/5fcfcdfad2be49929ecd3a2d6e0478c1/

You're not cleaning up your displays and browsers. See the code that conrad linked to to see how to do that properly. Also by creating an extra display inside the with Display(): block you're probably causing all sorts of trouble with the displays - only create the one display the the with statement creates.

Thanks for the help. I did have just that before but it gave me some weird error. However now I went back and now i am getting this error: selenium.common.exceptions.WebDriverException: Message: The browser appears to have exited before we could connect. If you specified a log_file in the FirefoxBinary constructor, check it for details.

Is it something having to do with Firefox being installed? I really do not know how to update firefox via bash command line here so I could use some help. Thanks

It may be related to having firefox installed, if it's on your path. If you have a firefox binary on that you've put there on your path, remove it and see what happens.

I've tried a bunch of different options, including trying splinter's display and browser() function, but it ended up asking about geckodriver, and whenever I try to specify an executable path it essentially brings me back to this error. I then tried removing the firefox binary and not specifying a path for the geckodriver and I am now getting:

BlockingIOError: [Errno 11] Resource temporarily unavailable

Is there something wrong i'm doing? I feel like there should be a simple way to do this.

Thanks

are you using selenium 2? selenium 3 won't work with our version of firefox: http://help.pythonanywhere.com/pages/selenium/

Harry: Yes, I have tried uninstalling and using selenium 2.53.6. I currently am on Selenium 2.53.6 and am back to getting this error: selenium.common.exceptions.WebDriverException: Message: The browser appears to have exited before we could connect. If you specified a log_file in the FirefoxBinary constructor, check it for details.

It looks like your code is leaving Xfvb instances running after it exits, which might explain at least some of the problems. If you have lots of processes running, then starting a new one can fail.

I've cleared down the rogue instances that I found on the web server where your site runs; if you try again now, does it work? Or do you at least get a different error message?

I appreciate the help. Unfortunately, I am still getting this error: selenium.common.exceptions.WebDriverException: Message: The browser appears to have exited before we could connect. If you specified a log_file in the FirefoxBinary constructor, check it for details.

Here again is a link to my code: https://www.pythonanywhere.com/user/ttataryn/shares/5fcfcdfad2be49929ecd3a2d6e0478c1/

Is there an easy way to run this headless? Or is that my only option?

Yes, there is. There are currently 2 links in this thread to the simple instructions (with code example) for running selenium headless.

Are you talking about the pyvirtualdisplay? I have tried using that to no avail. Getting frustrated because I feel like it's a simple mistake I am making but can not find it. Here again is a link to my code: https://www.pythonanywhere.com/user/ttataryn/shares/5fcfcdfad2be49929ecd3a2d6e0478c1/

You're getting frustrated because you're not following our advice. This is now the third time this link is being posted in this thread - http://help.pythonanywhere.com/pages/selenium/. It has an example in it that works. Your code is still different to it in important ways.

I appreciate the help, but I am frustrated because I keep being told to check this page, but I literally have already tried that code numerous times. Check my code now if you don't believe me. I am still getting the same error saying that the browser exited before we could connect.

I start the browser with the "With Display():, and initiate the driver with the same exact code, then quit once I'm out of it. How is it different?

Again, thanks for the help.

I feel like the issues lies with the firefox version I am using as what everything points to when I google the issue, however I am not able to update or change it as I can't use sudo apt-get.

Ok. Now we're sort of getting somewhere. Check whether you still have geckodriver or firefox on the path - you should only have firefox and it should be at /usr/local/bin/firefox

Just confirmed that in /usr/local/bin/firefox I only have firefox, no geckodriver. However, I did upload the gecko driver executable to another folder... it's in something like /home/myusername/projectname/bin. Should I remove that or will it not make a difference?

Also, if it helps, I did do a firefox -v. And it returned 17.0. I understand that's a very old version. Do I need to somehow upgrade?

It's possible that the geckodriver that you have is being used and that is interfering. Remove it.

No, don't try to upgrade Firefox. We have that version for a reason (explained further up in this thread).

Are you sure that you're running selenium 2.53.6?

I just tried reloading my web app after removing the geckodriver, however am getting the same error. I have tried running selenium -v and it does return 2.53.6. To be sure I also recently un-installed and re-installed this version of selenium. Don't know where to go from here.

I just tried reloading my web app after removing the geckodriver, however am getting the same error. I have tried running selenium -v and it does return 2.53.6. To be sure I also recently un-installed and re-installed this version of selenium. Don't know where to go from here.

I just noticed that you're importing Browser from something called splinter. If that's another browser automation thing, its presence may be interfering with the selenium that you're using. Try destroying your virtualenv and then reinstalling everything except splinter.

I appreciate the help.

I just destroyed the virtualenv, created a new one and reinstalled all of my packages. However, my original webpage is now not showing up and I'm getting a weird import error that I can not decipher: Error running WSGI application ModuleNotFoundError: No module named 'pandas.compat'

When you reinstalled the packages into the virtualenv, did you specify the versions? That sounds like a difference between the version of pandas that you have installed and the version some of your code (or the code of one of your dependencies) is expecting.

If there's nothing obvious to do with versions that you can see, could you paste the full stacktrace so that we can see which code is generating the error?

The only thing I thought you had to specify for was selenium? The first time I did not specify a version for pandas.

Here's the link to my error log: https://www.pythonanywhere.com/user/ttataryn/files/var/log/ttataryn.pythonanywhere.com.error.log

OK, that's really strange -- the error is coming from inside pandas, so it looks like it wasn't fully installed. What command did you use to install it? Was it just "pip install pandas"? Did you get any errors?

Well, that was short-lived. I un-installed and re-installed pandas, then restarted my app, and now I'm back to the great error of: selenium.common.exceptions.WebDriverException: Message: The browser appears to have exited before we could connect. If you specified a log_file in the FirefoxBinary constructor, check it for details.

Is there something i'm missing here? I feel like it should be easier.

it's definitely frustrating debugging errors like these.

I would suggest starting out very simple first:

eg: can you get the code here to run from a file/console script? then try to move that to a webapp- if it stops working try to figure out why before proceeding. then if that works try to add in some of your own code etc.

also if you are leaving lots of processes behind hanging around, be sure to run the code more than a couple times to make sure you aren't doing that before proceeding to the next step.

I tried doing the code from your help page in a python script, and it finally worked -- printed the browser title. However, as soon as I started to add some of my own code it began to behave weirdly.

I essentially added the website im looking to visit (linkedin.com/jobs) then trying to find an input element.

When I run it again with these new additions, it says it couldn't find the element. However, with some more investigating (I added a save_screenshot to see where it was stopping), the firefox window was giving me an error stating "proxy server is refusing connections". Is this something I can handle? I haven't seen anything like this online before.

.

elem = browser.find_element_by_class_name("keyword")
  File "/home/ttataryn/.virtualenvs/venv/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 413, in find_e
lement_by_class_name
    return self.find_element(by=By.CLASS_NAME, value=name)
  File "/home/ttataryn/.virtualenvs/venv/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 752, in find_e
lement
    'value': value})['value']
  File "/home/ttataryn/.virtualenvs/venv/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 236, in execut
e
    self.error_handler.check_response(response)
  File "/home/ttataryn/.virtualenvs/venv/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 192, in che
ck_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: {"method":"class name","selector":"keyword"
}
Stacktrace:
    at FirefoxDriver.findElementInternal_ (file:///tmp/tmpw7igbpvp/extensions/fxdriver@googlecode.com/components/driver-component
.js:10770)
    at FirefoxDriver.findElement (file:///tmp/tmpw7igbpvp/extensions/fxdriver@googlecode.com/components/driver-component.js:10779
)
    at DelayedCommand.executeInternal_/h (file:///tmp/tmpw7igbpvp/extensions/fxdriver@googlecode.com/components/command-processor
.js:12661)
    at DelayedCommand.executeInternal_ (file:///tmp/tmpw7igbpvp/extensions/fxdriver@googlecode.com/components/command-processor.j
s:12666)
    at DelayedCommand.execute/< (file:///tmp/tmpw7igbpvp/extensions/fxdriver@googlecode.com/components/command-processor.js:12608
)

[edit by admin: formatting]

Well, some progress! What's happening now is probably that you're accessing a site that isn't on our whitelist for free accounts. Free accounts can only access specific sites, paid ones get unrestricted Internet access.

Ah. Probably should have looked at that beforehand, sorry about that.

You list the LinkedIn API on the whitelist, do you know if that can be used for what I'm trying to do (simply look at job postings)? Also, does the hacker account siffice for unrestricted internet access?

Hmm, not sure about whether you can use the API -- job postings would be a sensible thing for them to include in it, but they might have commercial reasons to not do that. Their API docs are probably the place for a definitive answer.

But yes, a Hacker account definitely will suffice for unrestricted Internet access!

When I try to install selenium== 2.53.6 using pip3 uninstallation of selenium==3.0.2 fails.

Exception:
Traceback (most recent call last):
  File "/usr/lib/python3.6/shutil.py", line 538, in move
    os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: '/usr/local/lib/python3.6/dist-packages/selenium-3.0.2.dist-info/DESCRIPTION.rst' -> '/tmp/pip-x3imfzt4-uninstall/usr/local/lib/python3.6/dist-packages/selenium-3.0.2.dist-info/DESCRIPTION.rst'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/local/lib/python3.6/dist-packages/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/usr/local/lib/python3.6/dist-packages/pip/req/req_set.py", line 778, in install
    requirement.uninstall(auto_confirm=True)
  File "/usr/local/lib/python3.6/dist-packages/pip/req/req_install.py", line 754, in uninstall
    paths_to_remove.remove(auto_confirm)
  File "/usr/local/lib/python3.6/dist-packages/pip/req/req_uninstall.py", line 115, in remove
    renames(path, new_path)
  File "/usr/local/lib/python3.6/dist-packages/pip/utils/__init__.py", line 267, in renames
    shutil.move(old, new)
  File "/usr/lib/python3.6/shutil.py", line 553, in move
    os.unlink(src)
PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.6/dist-packages/selenium-3.0.2.dist-info/DESCRIPTION.rst'

What command are you using to install it? It should be this:

pip3.6 install --user selenium== 2.53.6

Don't forget the "--user" flag.