Forums

Selenium Error: "Can't load the profile"

I keep seeming to get this error randomly with my scheduled runs:

Traceback (most recent call last):
  File "/home/aaleks/.virtualenvs/myvirtualenv/Add_Tracking.py", line 341, in <module>
    run()
  File "/home/aaleks/.virtualenvs/myvirtualenv/Add_Tracking.py", line 193, in run
    driver = webdriver.Firefox()
  File "/home/aaleks/.virtualenvs/myvirtualenv/local/lib/python2.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 77, in __init__
    self.binary, timeout),
  File "/home/aaleks/.virtualenvs/myvirtualenv/local/lib/python2.7/site-packages/selenium/webdriver/firefox/extension_connection.py", line 49, in __init__
    self.binary.launch_browser(self.profile)
  File "/home/aaleks/.virtualenvs/myvirtualenv/local/lib/python2.7/site-packages/selenium/webdriver/firefox/firefox_binary.py", line 68, in launch_browser
    self._wait_until_connectable()
  File "/home/aaleks/.virtualenvs/myvirtualenv/local/lib/python2.7/site-packages/selenium/webdriver/firefox/firefox_binary.py", line 103, in _wait_until_connectable
    raise WebDriverException("Can't load the profile. Profile "
selenium.common.exceptions.WebDriverException: Message: Can't load the profile. Profile Dir: %s If you specified a log_file in the FirefoxBinary constructor, check it for details.

I'm running this in a virtualenv if that helps.

Hm. Selenium/Firefox can be a bit flakey... I think we built a retry loop into our code for when we try to start browsers -- may be worth a try?

I am constantly getting this error now whenever I run using schedule. It works fine otherwise.

Check you haven't run out of disk space? Selenium tends to fill up /tmp with junk...

Nope, every time I run it I make sure I still have room left.

Have you tried adding a retry loop?

try:
    driver = webdriver.Firefox()
except WebDriverException:
    for i in range 3:
        print('retrying webdriver start, attempt {}'.format(i))
        try: 
            driver=webdriver.Firefox()
            break
        except WebDriverException:
            if i == 2: raise

hideously ugly, i know, but maybe worth trying?

I tried putting in a loop but it still was giving that error every time.

OK, having tried to repro in a scheduled task of my own, I can confirm mine is also giving me problems. Will have to investigate this further...

Hi, Also looking for a solution to this problem. Only been getting it recently when run as a scheduled task. Started about a week ago.

When same code is run manually in the console, no such error appears.

This is the error that I'm getting:

selenium.common.exceptions.WebDriverException: Message: 'Can\'t load the profile. Profile Dir: /tmp/tmpy_Mubx Firefox output: \n(process:5151): GLib-CRITICAL **: g_slice_set_config: assertion \'sys_page_size == 0\' failed\nXlib:  extension "RANDR" missing on display ":1242".\n*** LOG addons.xpi: startup\n*** LOG addons.xpi: checkForChanges\n*** LOG addons.xpi: No changes found\n*** LOG addons.xpi: Add-ons list is invalid, rebuilding\n*** LOG addons.xpi-utils: Opening database\n'

[edited by admin: formatting]

Looks like we had a resource issue on the task server that your jobs were running on. We've sorted that out and we're writing some code to make it less likely in future. Give it a go now.

I'm still still getting a profile error only this time is has a slightly different error message.

    Traceback (most recent call last):
      File "/home/aaleks/.virtualenvs/myvirtualenv/Add_Tracking.py", line 382, in <module>
        run()
      File "/home/aaleks/.virtualenvs/myvirtualenv/Add_Tracking.py", line 202, in run
        driver = webdriver.Firefox()
      File "/home/aaleks/.virtualenvs/myvirtualenv/local/lib/python2.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 77, in __init__
        self.binary, timeout),
      File "/home/aaleks/.virtualenvs/myvirtualenv/local/lib/python2.7/site-packages/selenium/webdriver/firefox/extension_connection.py", line 49, in __init__
        self.binary.launch_browser(self.profile)
      File "/home/aaleks/.virtualenvs/myvirtualenv/local/lib/python2.7/site-packages/selenium/webdriver/firefox/firefox_binary.py", line 69, in launch_browser
        self._wait_until_connectable()
      File "/home/aaleks/.virtualenvs/myvirtualenv/local/lib/python2.7/site-packages/selenium/webdriver/firefox/firefox_binary.py", line 104, in _wait_until_connectable
        raise ProfileException("Can't load the profile.")
    selenium.common.exceptions.WebDriverException: Message: Can't load the profile.

Are there lots of processes running for your account on the scheduled tasks tab? (Check out the table near the bottom of the tab.)

No, there are none running when it starts and while it's running there are only 3-4. Also my storage is not maxed out.

I also still get an error. Only 2 process running, and manual invocation of the same function through console still works.

selenium.common.exceptions.WebDriverException: Message: 'Can\'t load the profile. Profile Dir: /tmp/tmpuTNfFd Firefox output: \n(process:24192): GLib-CRITICAL : g_slice_set_config: assertion \'sys_page_size == 0\' failed\nXlib: extension "RANDR" missing on display ":1116".\n LOG addons.xpi: startup\n LOG addons.xpi: checkForChanges\n LOG addons.xpi: No changes found\n LOG addons.xpi: Add-ons list is invalid, rebuilding\n* LOG addons.xpi-utils: Opening database\n'

OK, I think I'm going to have to discuss this with Glenn and Harry on Monday when we're all back in the office; I know they were working on a fix for this, but don't know the details. We'll get back to you ASAP.

It looks like we've been having problems with some of our scheduled task server infrastructure. It should be working OK again now, but we're implementing some changes that will stop it from getting clogged up again like this.

I am constantly getting the same error again now.

Hi @aaleks, what's the most recent task run that had the problem? Can I take a look at the logs?

Strange, it seems to be working again now, but yesterday none of my tasks with it were working for half the day. I'll let you know if it happens again.

We've been putting some tools in place to try and make the task scheduler more reliable -- we're not there yet, but do keep letting us know whenever you spot something's not right, and we'll keep at it...

My runs for almost the past 24 hours have all been getting the "can't load the profile error".

Hm. There did seem to be a few zombie xvfb processes on the server which I cleaned up, but not so many that it would ordinarily cause problems.

Still, I tidied them, and then I confirmed things were working by running my own test that starts and stops 6 firefox sessions. it seemed to work.

When's your next run due?

I've had two more tasks run so far and they both still got the error.

Ok. There's clearly something going on here that we don't understand. We'll continue investigating.

raise WebDriverException("Can't load the profile. Profile " I get the similar error at my scheduled runs

It should be working now. Let us know if you are still experiencing problems. We are working on a long term fix for these issues.

For anyone else running into selenium issues, we have a help page on how to use selenium on pythonanywhere, which has some important info re supported versions etc

The same problem. I made a retry loop as an example above

This helps to avoid fail the whole script.

That sounds like a good plan -- thanks for posting!

I made some tests and it seems that I was wrong with my answer above. If you use retry loop like harry said than you will get:

Traceback (most recent call last):
  File "/home/Aleksey/facebook_scraper/main/facebook_scraper.py", line 67, in find_posts
    browser = webdriver.Firefox()
  File "/usr/lib/python3.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 80, in __init__
    self.binary, timeout)
  File "/usr/lib/python3.7/site-packages/selenium/webdriver/firefox/extension_connection.py", line 52, in __init__
    self.binary.launch_browser(self.profile, timeout=timeout)
  File "/usr/lib/python3.7/site-packages/selenium/webdriver/firefox/firefox_binary.py", line 68, in launch_browser
    self._wait_until_connectable(timeout=timeout)
  File "/usr/lib/python3.7/site-packages/selenium/webdriver/firefox/firefox_binary.py", line 108, in _wait_until_connectable
    % (self.profile.path))
selenium.common.exceptions.WebDriverException: Message: Can't load the profile. Profile Dir: /tmp/tmp1cump89v If you specified a log_file in the FirefoxBinary constructor, check it for details.


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/Aleksey/facebook_scraper/main/facebook_scraper.py", line 68, in find_posts
    except selenium.common.exceptions.WebDriverException:
AttributeError: 'bool' object has no attribute 'common'

So this is it not just ugly way to mask problem, this will elso crash you script. So now I made temporary fix until I found a good solution.

                    try:
                        try:
                            browser = webdriver.Firefox()
                        except selenium.common.exceptions.WebDriverException:
                            # just second try to load Selenium
                            try:
                                browser = webdriver.Firefox()
                            except selenium.common.exceptions.WebDriverException:
                                logging.exception("Failed to load Selenium")
                                return
                        except AttributeError:
                            fb_scraper_logger.exception("Failed to load Selenium")
                            return

I put browser = webdriver.Firefox() into try/except AttributeError to avoid script fail.

Also I want to say that all this loading eats a lot of resources and time. I hope that Selenium update will be soon on Pythonanywhere.

Thanks for posting the further details. We definitely want to upgrade Selenium; the problem is that it requires a change to the way our virtualization system works, which is quite a big job.