神刀安全网

Crawling a page that loads content dynamically with Selenium in Python

Selenium is a browser automation tool that is used primarily for testing web applications. You can simulate real user actions and interactions with your web applications. Selenium supports all the major browser platforms  and operating systems . There are bindings for all the popular programming languages . The power of Selenium is not just restricted to testing your web apps, one other use can be of crawling or scraping websites, in particular, the ones which don’t provide an API and load content lazily using Javascript.

Today, we will be crawling an online merchant website www.jabong.com with Selenium using its python bindings. Jabong loads more products as you scroll down a web page. We will use Selenium to simulate this user action of scrolling down a web page and then retrieving all the product titles and the corresponding links to the product detail pages.

Reference:

Platform: Ubuntu 12.04

Tools Required

sudo pip install selenium  sudo pip install xvfb  sudo pip install pyvirtualdisplay

We use pyvirtualdisplay which is a wrapper around xvfb and enables you to run Firefix headlessly.

Page to Crawl:

Crawling a page that loads content dynamically with Selenium in Python

A quick "Inspect Element" on a shoe above shows that each of the product is wrapped by a "div" element with class "hover-box" and the title and links are embedded in an "a" element within those "div" elements.

Crawling a page that loads content dynamically with Selenium in Python

Code:

from selenium import webdriver from selenium.webdriver.common.keys import Keys from pyvirtualdisplay import Display  def correct_url(url):   if not url.startswith("http://") and not url.startswith("https://"):   url = "http://" + url  return url  def scrollDown(browser, numberOfScrollDowns):  body = browser.find_element_by_tag_name("body")  while numberOfScrollDowns >=0:   body.send_keys(Keys.PAGE_DOWN)   numberOfScrollDowns -= 1  return browser  def crawl_url(url, run_headless=True):  if run_headless:   display = Display(visible=0, size=(1024, 768))   display.start()   url = correct_url(url)  browser = webdriver.Firefox()  browser.get(url)  browser = scrollDown(browser, 10)   all_hover_elements = browser.find_elements_by_class_name("hover-box")     for hover_element in all_hover_elements:   a_element = hover_element.find_element_by_tag_name("a")   product_title = a_element.get_attribute("title")   product_link = a_element.get_attribute("href")   print product_title, product_link   browser.quit()  if __name__=='__main__':  url = "http://www.jabong.com/men/shoes/new-products/"  crawl_url(url)

Output

javascript:sendSizeFormWithSize('http://www.jabong.com/Nike-Liteforce-Ii-Mid-White-Sneakers-503958.html','6') Nike Ballista Iv Msl Grey Running Shoes http://www.jabong.com/Nike-Ballista-Iv-Msl-Grey-Running-Shoes-503907.html U.S. Polo Assn. Navy Blue Sneakers http://www.jabong.com/Us_Polo_Assn-Navy-Blue-Sneakers-518870.html Nike Dewired Navy Blue Sneakers http://www.jabong.com/Nike-Dewired-Navy-Blue-Sneakers-503897.html  javascript:sendSizeFormWithSize('http://www.jabong.com/Us_Polo_Assn-Black-Dress-Shoes-518868.html','6')  javascript:sendSizeFormWithSize('http://www.jabong.com/Us_Polo_Assn-Destiny-Navy-Blue-Sneakers-518878.html','6') United Colors of Benetton Black Boat Shoes http://www.jabong.com/United-Colors-of-Benetton-Black-Boat-Shoes-512546.html Phosphorus Brown Loafers http://www.jabong.com/phosphorus-Brown-Loafers-506097.html  javascript:sendSizeFormWithSize('http://www.jabong.com/United-Colors-of-Benetton-Blue-Sneakers-512548.html','6') U.S. Polo Assn. Delta Beige Sneakers http://www.jabong.com/Us_Polo_Assn-Delta-Beige-Sneakers-518873.html Asics Kayano 20 Black Running Shoes http://www.jabong.com/Asics-Kayano-20-Black-Running-Shoes-507673.html Nike Air Max 2014 Blue Running Shoes http://www.jabong.com/Nike-Air-Max-2014-Blue-Running-Shoes-503964.html Asics Excel 33 3 Navy Blue Running Shoes http://www.jabong.com/Asics-Excel-33-3-Navy-Blue-Running-Shoes-507674.html Nike Air Relentless 3 Msl Blue Running Shoes http://www.jabong.com/Nike-Air-Relentless-3-Msl-Blue-Running-Shoes-503954.html Asics Kayano 20 Navy Blue Running Shoes http://www.jabong.com/Asics-Kayano-20-Navy-Blue-Running-Shoes-507672.html Nike Lunarinternationalist Grey Running Shoes http://www.jabong.com/Nike-Lunarinternationalist-Grey-Running-Shoes-503975.html Nike Free 5.0+ Blue Running Shoes http://www.jabong.com/Nike-Free-50-Blue-Running-Shoes-503919.html Nike Fs Lite Run Black Running Shoes http://www.jabong.com/Nike-Fs-Lite-Run-Black-Running-Shoes-503955.html Nike Lunarinternationalist Blue Running Shoes http://www.jabong.com/Nike-Lunarinternationalist-Blue-Running-Shoes-503976.html Nike Free Trainer 5.0 Grey Running Shoes http://www.jabong.com/Nike-Free-Trainer-50-Grey-Running-Shoes-503886.html  javascript:sendSizeFormWithSize('http://www.jabong.com/Nike-Zoom-Terra-Kiger-Blue-Running-Shoes-503930.html','6') Andrew Hill Brown Dress Shoes http://www.jabong.com/andrew-hill-Brown-Dress-Shoes-506110.html Nike Lunar Forever 3 Msl Grey Running Shoes http://www.jabong.com/Nike-Lunar-Forever-3-Msl-Grey-Running-Shoes-503970.html Asics Kayano 20 Red Running Shoes http://www.jabong.com/Asics-Kayano-20-Red-Running-Shoes-507671.html Nike Free Trainer 5.0 Black Running Shoes http://www.jabong.com/Nike-Free-Trainer-50-Black-Running-Shoes-503915.html Nike Chroma Thong Iii Green Slippers http://www.jabong.com/Nike-Chroma-Thong-Iii-Green-Slippers-503977.html Nike Aquahype Blue Flip Flops http://www.jabong.com/Nike-Aquahype-Blue-Flip-Flops-503890.html Z Collection Green Loafers http://www.jabong.com/z-collection-Green-Loafers-517513.html Nike Flex 2013 Rn Black Running Shoes http://www.jabong.com/Nike-Flex-2013-Rn-Black-Running-Shoes-503916.html U.S. Polo Assn. Brown Sneakers http://www.jabong.com/Us_Polo_Assn-Brown-Sneakers-518869.html Phosphorus Black Loafers http://www.jabong.com/phosphorus-Black-Loafers-506093.html Phosphorus Black Loafers http://www.jabong.com/phosphorus-Black-Loafers-506102.html Nike Eliminate Ii Leather Grey Sneakers http://www.jabong.com/Nike-Eliminate-Ii-Leather-Grey-Sneakers-503909.html Nike Fs Lite Trainer Blue Running Shoes http://www.jabong.com/Nike-Fs-Lite-Trainer-Blue-Running-Shoes-503952.html Nike Suketo 2 Leather Red Sneakers http://www.jabong.com/Nike-Suketo-2-Leather-Red-Sneakers-503972.html Nike Free Flyknit+ Red Running Shoes http://www.jabong.com/Nike-Free-Flyknit-Red-Running-Shoes-469765.html  javascript:sendSizeFormWithSize('http://www.jabong.com/Nike-Zoom-Wildhorse-Blue-Running-Shoes-503931.html','6') Nike Air Pegasus+ 30 Grey Running Shoes http://www.jabong.com/Nike-Air-Pegasus-30-Grey-Running-Shoes-503934.html Nike Flex Supreme Tr 2 Grey Running Shoes http://www.jabong.com/Nike-Flex-Supreme-Tr-2-Grey-Running-Shoes-503942.html Phosphorus Black Loafers http://www.jabong.com/phosphorus-Black-Loafers-506088.html Phosphorus Black Loafers http://www.jabong.com/phosphorus-Black-Loafers-506087.html Phosphorus Black Loafers http://www.jabong.com/phosphorus-Black-Loafers-506091.html Nike Zoom Structure+ 17 Black Running Shoes http://www.jabong.com/Nike-Zoom-Structure-17-Black-Running-Shoes-503949.html Nike Flyknit Lunar2 Black Running Shoes http://www.jabong.com/Nike-Flyknit-Lunar2-Black-Running-Shoes-503959.html Phosphorus Brown Loafers http://www.jabong.com/phosphorus-Brown-Loafers-506103.html Andrew Hill Black Dress Shoes http://www.jabong.com/andrew-hill-Black-Dress-Shoes-506109.html Phosphorus Tan Loafers http://www.jabong.com/phosphorus-Tan-Loafers-506095.html Phosphorus Black Loafers http://www.jabong.com/phosphorus-Black-Loafers-506090.html U.S. Polo Assn. Delta Navy Blue Sneakers http://www.jabong.com/Us_Polo_Assn-Delta-Navy-Blue-Sneakers-518872.html Phosphorus Brown Loafers http://www.jabong.com/phosphorus-Brown-Loafers-506094.html Nike Lunar Forever 3 Msl White Running Shoes http://www.jabong.com/Nike-Lunar-Forever-3-Msl-White-Running-Shoes-503971.html Asics Kayano 20 White Running Shoes http://www.jabong.com/Asics-Kayano-20-White-Running-Shoes-507670.html

转载本站任何文章请注明:转载至神刀安全网,谢谢神刀安全网 » Crawling a page that loads content dynamically with Selenium in Python

分享到:更多 ()

评论 抢沙发

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址
分享按钮