Ive been getting into web scraping lately and i was wondering what like… the rules are? like can I scrape any website i want or are there limitations and stuff
you have to check the robot.txt of the website to see whats allowed and what isnt
2 Likes
Just have to make sure you check what a website allows and doesnt. Dont overload your scraping, keep it simple if youre just starting off. start small.
2 Likes
Hey @jbowley2208 , glad to hear youre interesting in web scraping! Just like @BeautifulSoupDrinker said, its best to check out the robot.txt of a website to see what is allowed and what isnt. This is a set of instructions for your scraping bot that will tell it which pieces of information it is allowed to scrape. If you need some more information, please do not hesitate to let us know