The Python Oracle

python bs4 get element in popup without selenium

--------------------------------------------------
Hire the world's top talent on demand or became one of them at Toptal: https://topt.al/25cXVn
and get $2,000 discount on your first invoice
--------------------------------------------------

Music by Eric Matyas
https://www.soundimage.org
Track title: Sunrise at the Stream

--

Chapters
00:00 Python Bs4 Get Element In Popup Without Selenium
00:46 Accepted Answer Score 3
01:31 Thank you

--

Full question
https://stackoverflow.com/questions/5030...

--

Content licensed under CC BY-SA
https://meta.stackexchange.com/help/lice...

--

Tags
#python #beautifulsoup

#avk47


ACCEPTED ANSWER

Score 3


If you are looking for the popup from subjects I used

res = soup.findAll("div", {"class": "subjects-skills__item"})

and the return was is:

<div class="subjects-skills__item">
<h5 class="subjects-skills__label">Subjects</h5>
<ul>
<li>Science</li>
</ul>
</div>,
 <div class="subjects-skills__item">
<h5 class="subjects-skills__label">Skills</h5>
<ul>
<li>Creativity</li>
<li>Critical Thinking</li>
</ul>
</div>

I got it by clicking the popup.. Highlighting the text, then right-click and go to inspect to locate the class.

from bs4 import BeautifulSoup as bs4
import requests

def get_data():

    url = 'https://www.commonsense.org/education/game/garrys-mod'
    r = requests.get(url, headers={"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.103 Safari/537.36"})
    html_bytes = r.text
    soup = bs4(html_bytes, 'lxml')

    res = soup.findAll("div", {"class": "subjects-skills__item"})

    return res

test1 = get_data()

If you just want the text..

# For just the Text
for i in test1:
    print(i.text)

returns

Subjects
Science

Skills
Creativity
Critical Thinking