Actuarial Outpost

Actuarial Outpost (http://www.actuarialoutpost.com/actuarial_discussion_forum/index.php)
-   Software & Technology (http://www.actuarialoutpost.com/actuarial_discussion_forum/forumdisplay.php?f=17)
-   -   Fun with python (http://www.actuarialoutpost.com/actuarial_discussion_forum/showthread.php?t=348650)

binyamin 06-26-2020 10:58 AM

Fun with python
 
Has anyone done web scraping for fun using python? Like simple legal stuff, going to a website give your credentials, search, and download. Which package did you use? Has anyone come across good example codes using "Beautiful Soup"? I googled, but I want to know any of the readers here have used it.

dmbymdt 08-12-2020 08:47 PM

Quote:

Originally Posted by binyamin (Post 9887101)
Has anyone done web scraping for fun using python? Like simple legal stuff, going to a website give your credentials, search, and download. Which package did you use? Has anyone come across good example codes using "Beautiful Soup"? I googled, but I want to know any of the readers here have used it.

I used beautifulsoup for web scraping james allen website. Since doing this james allen actually changed their website design and didn't have all of the same information. It's a good scraper for python. https://pypi.org/project/beautifulsoup4/

A few others I used

import pandas as pd
import urllib3
import certifi
import os
import time
from bs4 import BeautifulSoup
from collections import namedtuple
from json import loads
import requests
import json

Many sites have api's and it is suggested to use this structured where possible.


All times are GMT -4. The time now is 08:07 AM.

Powered by vBulletin®
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.

Page generated in 0.08590 seconds with 9 queries