Forums

Can we request to add a new website to the proxy whitelist?

I am scraping images from Reddit and I am getting 403 error while using requests module. I guess this is because of scraping from preview.redd.it

Can you add it to the whitelist?

deleted-user-4803918 | 9 posts | Jan. 2, 2019, 2:56 p.m. | permalink

reddit is already in the whitelist. If you're getting a 403 it may be coming from the service itself because you have not authenticated with it.

Staff glenn | 9718 posts | PythonAnywhere staff | Jan. 2, 2019, 3:14 p.m. | permalink

reddit.com is in the whitelist but not redd.it

deleted-user-4803918 | 9 posts | Jan. 3, 2019, 5:42 a.m. | permalink

Someone must have provided us with API documentation that said that it was an official public API at some point -- could you give us a link to anything saying that redd.it hosts an API?

Staff giles | 12082 posts | PythonAnywhere staff | Jan. 3, 2019, 3:57 p.m. | permalink

Actually reddit hosts its images on preview.redd.it.

So when I use the official Reddit API, the JSON it returns contains image links hosted on redd.it

If I want to download them, or do something else, I must have access to redd.it

deleted-user-4803918 | 9 posts | Jan. 4, 2019, 12:27 p.m. | permalink

In order to whitelist the site for free users, please provide a link to the API documentation by reddit that shows redd.it as one of the public api endpoints

Staff conrad | 4232 posts | PythonAnywhere staff | Jan. 4, 2019, 6:56 p.m. | permalink

I couldn't find anything about it in the official documentation but if you are going to scrape images from any subreddit, you will be getting the media hosted at something like "https://external-preview.redd.it".

For example, if you see https://www.reddit.com/r/funny.json, and Ctrl+F "redd.it', you will find many instances for it.

deleted-user-4803918 | 9 posts | Jan. 13, 2019, 6:40 a.m. | permalink

Ok. That looks legitimate. I have added external-preview.redd.it to the whitelist.

Staff glenn | 9718 posts | PythonAnywhere staff | Jan. 14, 2019, 12:43 p.m. | permalink

If you could just whitelist all the subdomains of redd.it, that would be great, because they keep changing it. Although it's your call.

deleted-user-4803918 | 9 posts | Jan. 16, 2019, 6:26 a.m. | permalink

We try to avoid whitelisting large numbers of subdomains (for example, what if they start using something like dontscrapethis.redd.it in the future). But if their API starts providing content on a new subdomain in the future, just post back here.

Staff giles | 12082 posts | PythonAnywhere staff | Jan. 16, 2019, 1:04 p.m. | permalink

Can you add preview.redd.it ?

deleted-user-4803918 | 9 posts | March 26, 2019, 2:06 p.m. | permalink

Send us a link to the API documentation that shows the URL for the API and we'll consider it for the whitelist.

Staff conrad | 4232 posts | PythonAnywhere staff | March 26, 2019, 3:01 p.m. | permalink

The same thing that I mentioned earlier in this thread. https://www.pythonanywhere.com/forums/topic/13699/#id_post_54550

Copying and pasting: I couldn't find anything about it in the official documentation but if you are going to scrape images from any subreddit, you will be getting the media hosted at something like "https://preview.redd.it".

For example, if you see https://www.reddit.com/r/funny.json, and Ctrl+F "redd.it', you will find many instances for it.

deleted-user-4803918 | 9 posts | March 26, 2019, 4:28 p.m. | permalink

OK, thanks -- that's whitelisted now.

Staff giles | 12082 posts | PythonAnywhere staff | March 28, 2019, 1:09 p.m. | permalink

Thanks a lot :)

deleted-user-4803918 | 9 posts | March 29, 2019, 1:43 p.m. | permalink