Forums

Headless Chrome Selenium

Hello, i see that is possible to ask for enabling the use of headless chrome. Is it possible to have it enabled? many thanks

Sure! I've enabled the new feature; the next step is for you to upgrade Selenium for your account -- for example, if you're using Python 3.7, run this in Bash:

pip3.7 install --user --upgrade selenium

...and then you can run Selenium with Chrome using code like this:

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--disable-gpu")
browser = webdriver.Chrome(options=chrome_options)

try:
    browser.get("https://www.google.com")
    return "Page title was '{}'".format(browser.title)

finally:
    browser.quit()

many thanks! i am running it in a pipenv virtual enviroment how am i supposed to update selenium?

sorry, upgrade

I don't use pipenv myself, but for a normal virtualenv you would just activate it then run pip install --upgrade selenium

Hello, Could you please enable headless chrome for my account as well. Thank you

No problem. I have enabled it for your account.

Amazing, Thank you.

No problem :-)

Hi! Could you please enable headless Chrome for my account too. Thanks!

You have already new virtualization system enabled for your account. To use headless Chrome follow the steps mentioned above.

Can you please tell me what Chrome version is installed on my account? I don't know what chromedriver version I need to use.

The chrome and chromedriver that are already installed work together. You do not need to get a different chromedriver version.

I tried to run the code above but I got this two errors:

[Errno 2] No such file or directory: 'chromedriver': 'chromedriver'

selenium.common.exceptions.WebDriverException: Message: 'chromedriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home

Right, it appears that your system image has to be upgraded. One word of warning first -- because changing the system image upgrades a lot of the pre-installed Python packages, any code that you have that relies on those packages might break if it's not compatible with the new versions. Also, because the new image has newer versions of Python, if you have any virtualenvs, you may need to rebuild them. If you're happy to go ahead despite that, just let us know and we'll switch you over.

Yes, switch it on please.

OK, it's done. You're on the most recent system image now.

It works now, thank you very much!

Excellent, thanks for confirming!

Could I also this get enabled on my account? Running the code provided above (after including "from selenium import webdriver") produces the following exception:

selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally

No problem. It's done for you.

Hi, I'd also like headless chrome activated for my account. Is there any procedure I can follow here?

@mjgibbons3 Just confirm that you are aware of what I posted before in this thread.

@Pafk I'm aware. Can you please upgrade the system image?

No problem -- that's done now. You'll get the new system image and the new virtualization system in any new consoles you start; websites and scheduled/always-on task will pick them both up the next time they are restarted. For the next step for using headless Chrome, see my earlier post.

Thank you @giles - much appreciated!

Hi I followed the instructions but i have the following error: selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally

Ok. I have enabled the feature for your account.

Thank you @glenn

Hi, can I have selenium activated for me as well? I'm going to use Chrome driver. Thanks

Hi, can I have headless chrome enable for my account?

@wladzioo I enabled new virtualization for you.

@hamad290sca Yes, but we need to upgrade your system image first. One word of warning first -- because changing the system image upgrades a lot of the pre-installed Python packages, any code that you have that relies on those packages might break if it's not compatible with the new versions. Also, because the new image has newer versions of Python, if you have any virtualenvs, you may need to rebuild them. If you're happy to go ahead despite that, just let us know and we'll switch you over.

@pafk yes please go ahead

OK, I upgraded your system image to the most recent one and enabled beta virtualization feature for you.

Hi, could you please enable headless chrome for my account too? many thanks!

Yes, it's done for you.

Hi pafk, thank you for enabling headless chrome for me. However when I run the sample script supplied at the begining of this topic I got the following error:

browser = webdriver.Chrome(options=chrome_options)
TypeError: __init__() got an unexpected keyword argument 'options'

Could you please help me with this issue?

@ioanf did you upgrade Selenium first as it was advised?

@pafk yes I do.

Did you do that for the same Python version you are using to run the script? The error suggests that Selenium is not upgraded.

@pafk Huh, my bad. Obviously I upgraded Selenium to Python 3.7 not 3.8 I am currently using. Now everything works like charm :) Thank you once again!

Good to hear that you sorted it!

please enable for me

No problem. I have enabled it for your account.

Can you please enable for my account as well? Thank you!

No problem. I have enabled it for your account.

Hello, would it be possible to enable headless Chrome for my account please? Thanks!

ok, I have switched that on for you (ie. you will be able to run chrome). However, note that you are currently on a system image that may not have the most updated chrome binaries installed. If you want we can switch you over to the newest system image, but note that that will update python libraries and break your virtualenvs etc (you would have to reinstall them), as the python versions would have changed etc. Alternatively you can build and install the binaries yourself.

Hi Conrad, thanks - please go ahead and switch me over to the newest system image :) I don't mind reinstalling my virtualenvs

Ok. I have switched your system image for you.

Can you please enable Chrome headless for my account as well? Thank you! The code above gives me an error: selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally

@mouhabsnyeei no problem. It's done for you.

Can you please enable Chrome headless for my account as well? Thank you!

Sure, that's done now -- see my previous post for the next steps.

I would like to have Chrome headless enabled as well, thanks!

@TreeofPython. No problem, it's done for you

Awesome, this fixed my selenium issues. Thanks!

hey can anyone help i have used the above code but i am receiving the following exception.

WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally (unknown error: DevToolsActivePort file doesn't exist) (The process started from chrome location /usr/bin/chromium-browser is no longer running, so ChromeDriver is assuming that Chrome has crashed.) (Driver info: chromedriver=2.42.591071 (0b695ff80972cc1a65a5cd643186d2ae582cd4ac),platform=Linux 5.4.0-1020-aws x86_64)

There's a beta feature that we have to activate for your account to make it work; I've done that, so if you try again in a newly-started console it should work now.

thank you. headless chrome is now working but selenium webdriver wait is not working properly i am getting timeoutException when trying to wait for an element to load. can you look into it ThankYou

That suggests just that the element that you are expecting to be present is not there; you can use the get_screenshot_as_file method on your WebDriver object to get a screenshot of the page and find out what it is displaying.

One thing to remember is that free accounts have restricted Internet access and can only access sites on our whitelist, so if you're trying to access a site that is not there, you might be getting an "Access Denied" page rather than the page you expect.

i am getting this error on my code was working fine before

WebDriverException: Message: unknown error: failed to write automation extension zip (Driver info: chromedriver=2.42.591071 (0b695ff80972cc1a65a5cd643186d2ae582cd4ac),platform=Linux 5.4.0-1020-aws x86_64)

You're over your disk quota, so there is no space for web driver to write the files that it needs to work.

Hi there, I am trying to run the above code but I am getting the following error:

"selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally"

I think my account needs to be enabled too? TIA, Clem.

hey there. if i upgrade my account will i be able to use the webdriver wait function

hi. can you enable the headless chrome on my account too i am currently getting the following error

WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally (unknown error: DevToolsActivePort file doesn't exist) (The process started from chrome location /usr/bin/chromium-browser is no longer running, so ChromeDriver is assuming that Chrome has crashed.) (Driver info: chromedriver=2.42.591071 (0b695ff80972cc1a65a5cd643186d2ae582cd4ac),platform=Linux 5.4.0-1020-aws x86_64)

–°an you please activate chrome headless for my account? Thank you!

@clemlanoe and @MrGorsky -- no problem -- that's done now. You'll get the new system image and the new virtualization system in any new consoles you start; websites and scheduled/always-on task will pick them both up the next time they are restarted. For the next step for using headless Chrome, see my earlier post.

@tweets12 -- likewise, that's done for you, but one thing to remember is that free accounts have restricted Internet access and can only access sites on our whitelist.

@testpro1 -- you can use the webdriver wait function in a free account as well as in a paid account.

i am using selenium to take screenshot of tweet but twitter just loads the body of the site and tweet portion shows the try again button. tweet is not loading. anyone can help

Can you please upgrade my system image? i've the same problem with chrome

@tweets12 that sounds like you are running into the limitations of a free account. Free accounts have restricted Internet access and can only access sites on our whitelist.

@Abv1206 -- no problem -- that's done now. You'll get the new system image and the new virtualization system in any new consoles you start; websites and scheduled/always-on task will pick them both up the next time they are restarted. For the next step for using headless Chrome, see my earlier post.

twitter is on white list still not opening :/

@giles thank you! i'll try it :)

@tweets12 their API is, but their website as a whole is not. It is likely that the page that you are trying to scrape is using a hostname that is not part of their API.

@Abv1206 great!

if i upgrade my account will i be able to access twiiter like its working on localhost server?

@tweets12 yes

Can you please upgrade my system image? i've the same problem with chrome

No problem. I have updated your account.

thanks! :)

Hey, could you please activate selenium for my account

No problem, it's done for you.

Hello, would it be possible to enable headless Chrome for my account please? Thanks!

No problem -- that's done now. You'll get the new virtualization system in any new consoles you start; websites and scheduled/always-on tasks will pick it up the next time they are restarted. For the next step for using headless Chrome, see my earlier post.

Hello, could you please activate headless Chrome for my account? Thanks!

No problem, that's enabled for you @Monega.

Hello, could you please activate headless Chrome for my account? Thanks!

Sure -- that's done now.

Hi, even-though i have my chrome webdriver in the path , am getting below error No such file or directory: '/home/pythontrad/xxx.exe'

Could you show your code that you use to start Chrome?

Hello, could you please activate headless Chrome for my account? Thanks!

No problem. It's done for you.

Hello, would it be possible to have headless Chrome enabled on my account? Thank you

Sure! However, we would need to upgrade your system image as well as enabling that; one word of warning before we do that -- because of the changes to the point releases of Python, any virtualenvs you have might break -- and if you're not using virtualenvs, the pre-installed Python modules will be upgraded so that might break any code you have that relies on the old installed versions.

If you're happy for us to switch you over despite that, then let us know.

Thank you, Giles. Acknowledged - That's fine by me.

No problem -- that's done now. You'll get the new system image and the new virtualization system in any new consoles you start; websites and scheduled/always-on task will pick them both up the next time they are restarted. For the next step for using headless Chrome, see my earlier post.

Hello, could you please activate headless Chrome for my account? Thanks!

Hi @yonataliasu13 --- it's already enabled for you. Maybe you need to update selenium as Giles advised here: https://www.pythonanywhere.com/forums/topic/27202/#id_post_78144 ?

Hello, could you please activate headless Chrome for my account? Thanks! And how should I update selenium? I am new to this and I do not know where to run the following command: "pip3.7 install --user --upgrade selenium"

@Bernas99 -- it's done for you. You should run the command in the Bash console running on your PA account.

How should I call the webdriver? I am doing it like this:

from selenium import webdriver; driver = webdriver.Chrome()

because I can not upload de chromedriver.exe to my pa files. Thanks in advance, and sorry for being a noob

Do you get any errors?

Yes. Message: unknown error: Chrome failed to start: exited abnormally

I can not run my .py file. Please help me. I think that the headless chrome does not work for my scrape. I would like to use the regular chrome webdriver. However I can not do that

You have a free account, so most of the scraping won't work. See https://www.pythonanywhere.com/whitelist/

Also, the correct headless Chrome configuration is described here: https://www.pythonanywhere.com/forums/topic/27202/#id_post_78144