Can't Select Element With Selenium No Matter What I Do

I'm trying to develop a scraper using selenium but I can't select elements in the following page: http://comprasnet.gov.br/acesso.asp?url=/ConsultaLicitacoes/ConsLicitacao_texto.as

Solution 1:

The element you are trying to fetch is inside a frame. Frames are their own document and treated as a separate search context by Selenium:

>>> from selenium import webdriver
>>> driver = webdriver.Chrome()
>>> driver.get("http://comprasnet.gov.br/acesso.asp?url=/ConsultaLicitacoes/ConsLicitacao_texto.asp")
>>> element = driver.find_element_by_id("txtTermo")
Traceback (most recent call last):
  File "<pyshell#3>", line 1, in <module>
    element = driver.find_element_by_id("txtTermo")
  File "#PATH_REDACTED#\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 360, in find_element_by_id
    return self.find_element(by=By.ID, value=id_)
  File "#PATH_REDACTED#\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 978, in find_element
    'value': value})['value']
  File "#PATH_REDACTED#\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "#PATH_REDACTED#\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"id","selector":"txtTermo"}
  (Session info: chrome=71.0.3578.98)
  (Driver info: chromedriver=2.45.615291 (ec3682e3c9061c10f26ea9e5cdcf3c53f3f74387),platform=Windows NT 10.0.15063 x86_64)

>>> driver.switch_to.frame('main2')
>>> element = driver.find_element_by_id("txtTermo")
>>> element
<selenium.webdriver.remote.webelement.WebElement (session="d5a30c9f45d49a23152767da0b811f58", element="0.6642244007973428-1")>

Think of a hierarchy of frames that you can navigate through, each containing their own complete DOM hierarchy. When you are done working inside the frame, if you want to return to the top level document and do some more work, switch to the default content:

driver.switch_to.default_content()

Frames are highly discouraged these days and you aren't that likely to run into them on newer applications, but in the past I've found that (as silly as it sounds) examining the value of driver.page_source can help you get a handle on what part of the overall document you are currently acting on. This will show you only the source for the current search context, i.e. the frame you are currently switched to.