Forums

Getting only first element from selenium

from selenium import webdriver from flask import Flask, render_template import random from pyvirtualdisplay import Display

app = Flask(__name__)
with Display():
     options = webdriver.ChromeOptions()
     options.add_argument('--headless')
     driver = webdriver.Chrome(chrome_options=options)
     driver.implicitly_wait(10)
     t_file = '/home/parshuram/mysite/static/links'
     target = open(t_file).read().splitlines()
try:
    @app.route('/')
    def get_jokes():
        driver.get(random.choice(target))
        jokes = driver.find_elements_by_tag_name('p')
        mylist = []
        for joke in jokes:
            mylist.append(joke.text)
            return render_template('joke.html', s=mylist)

    if __name__ == "__main__":
        app.run(debug=True)
except Exception as e:
    print(e)

The same code works fine on my PC. However, here I only get the 1st element 'joke' displayed in 's' Here is the template

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Sick Jokes</title>
    <style>
    body {
    background-color:yellow;
    }

div {
  display: block;
  overflow: auto;
  border: 5px solid green;
  padding: 45px;
  margin: 15px;
  text-align: center;
  vertical-align: middle;
}
h2 {
font-size: 50px;
font-family: sans-serif;
color: black;
text-align: center
}
li {
font-family: Copperplate Gothic Light;
font-size: 22px;
color: black;
}
button {
position: relative;
color: black !important;
text-transform: uppercase;
font-family: Copperplate Gothic Light;
font-size: 18px;
background: #49a6cc;
padding: 20px;
border-radius: 50px;
display: inline-block;
border: none;
transition: all 0.4s ease 0s;

}
</style>
</head>

<body>
   <header>
       <h2>Sickest Jokes on the Internet!</h2></header>
   <div>
   <a href="{{ url_for('get_jokes') }}"><button>Press here for new Jokes if the page has no Jokes or you have read them already!</button></a>
   </div>

         {%for i in range(0, 21)%}
    <div>
          <li>{{s[i]}}</li>
    </div>
         {%endfor%}


</body>
</html>

Any ideas?

This has probably got to do with the with Display() function, I am not sure how. The code on my PC has no With()..it is running fine, I have also removed the Exception part...still getting only 1st element. So, the scraping is fine, the website is fine.It has to do with the code here.

can someone check this out please? Tried everything and failed.

Check whether the list in the view actually has other elements and, if it does not, check the HTML that you are getting back to see why there is only one element extracted.

How should I do that? I added a print after the append. However, when I run in in the console it keeps saying Chrome has crashed?

Also, I can't check it on my end because the same code works fine on my end. :(

selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: crashed (unknown error: DevToolsActivePort file doesn't exist) (The process started from chrome location /usr/bin/chromium-browser is no longer running, so ChromeDriver is assuming that Chrome has crashed.) (Driver info: chromedriver=2.42.591071 (0b695ff80972cc1a65a5cd643186d2ae582cd4ac),platform=Linux 5.4.0-1020-aws x86_64)

Here is the error

It looks like you're opening your browser in your module setup code, and then trying to use it in a view function. That won't work in a production environment, because you have multiple processes running your website.

You should put all of the code to create the browser and so on inside the view function.

My site is running. This was the message when I tried to run it as a single file to check the print(mylist) to see if there is a single element or many. Can you tell me a way to check that?

Damn!...How did you miss it?...I stupidly put the return statement INSIDE the for loop. Sorry to have wasted your time.

Happy you figured it out!