I want to scrape news headlines from this link: https://www.newsnow.co.uk/h/Business+&+Finance?type=ln
I want to expand news by clicking (using selenium) on the button 'view more headlines' to collect the max number of news headlines possible
I created this code but failed to make the click to expand news :
import timefrom selenium import webdriveru = 'https://www.newsnow.co.uk/h/Business+&+Finance?type=ln'driver = webdriver.Chrome(executable_path=r"C:\chromedriver.exe")driver.get(u)driver.execute_script("window.scrollTo(0, document.body.scrollHeight)") driver.implicitly_wait(60) # secondselem = driver.find_element_by_css_selector('span:contains("view more headlines")')for i in range(10): elem.click() time.sleep(5) print(f'click {i} done')returns: selenium.common.exceptions.InvalidSelectorException: Message: invalid selector: An invalid or illegal selector was specified
I tried using xpath selector:
elem = driver.find_element_by_xpath('//[@id="nn_container"]/div[2]/main/div[2]/div/div/div[3]/div/a')returns: selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element <a class="rs-button-more js-button-more btn--primary btn--primary--no-spacing" href="#">...</a> is not clickable at point (353, 551). Other element would receive the click: <div class="alerts-scroller">...</div>
