Scrapy Fakeuseragenterror: Error Occurred During Getting Browser
I use Scrapy FakeUserAgent and keep getting this error on my Linux Server. Traceback (most recent call last): File '/usr/local/lib64/python2.7/site-packages/twisted/internet/defe
Solution 1:
I am not sure about RAM and why the error only happens on the Linux Server with a minimum spec. I solved it by using fake-useragent
fallback feature. Sadly, scrapy-fake-useragent
doesn't give any feature to set it conveniently, so I have to override the middleware feature on middlewares.py
like this:
from fake_useragent import UserAgent
from scrapy_fake_useragent.middleware import RandomUserAgentMiddleware
classFakeUserAgentMiddleware(RandomUserAgentMiddleware):
def__init__(self, crawler):
super(FakeUserAgentMiddleware, self).__init__(crawler)
# If failed to get random user agent, use the most common one
self.ua = UserAgent(fallback='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36')
self.per_proxy = crawler.settings.get('RANDOM_UA_PER_PROXY', False)
self.ua_type = crawler.settings.get('RANDOM_UA_TYPE', 'random')
self.proxy2ua = {}
Then I activate the middleware on settings.py
like this:
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
# 'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400, # disable the original middleware'myproject.middlewares.FakeUserAgentMiddleware': 400,
# omitted
}
UPDATE
Try updating fake-useragent to version 0.1.5. I was using 0.1.4 and after upgrading, the problem is gone from the root, not by using fallback.
Solution 2:
Using fake_useragent 0.1.7 here, having same issue.
However I have fixed it for my server. Here is the issue ticket with my suggestion to bypass the error.
https://github.com/hellysmile/fake-useragent/issues/59
Hope that helps.
Post a Comment for "Scrapy Fakeuseragenterror: Error Occurred During Getting Browser"