UserAgent in parameters #170
Conversation
There was a problem hiding this comment.
I'm not sure it's a great idea to have the default UA as the name of the app. It seems too direct/easy to shut down/prevent default settings from working by just setting a ban on it, forcing users to change UA.
I believe a Firefox/Chrome version would be best as default and could be changed if user wants/need to.
Does this make any sense?
There was a problem hiding this comment.
I'm not sure it's a great idea to have the default UA as the name of the app. It seems too direct/easy to shut down/prevent default settings from working by just setting a ban on it, forcing users to change UA.
I believe a Firefox/Chrome version would be best as default and could be changed if user wants/need to.
|
Sounds logical. Changed the default to useragent Firefox 80 on Windows 10 |
bin/wayback_machine_downloader
Outdated
| @@ -58,6 +58,10 @@ option_parser = OptionParser.new do |opts| | |||
| options[:list] = true | |||
| end | |||
|
|
|||
| opts.on("-u", "--user-agent STRING", String, "UserAgent for connection (Default is WayBack Machine Downloader)") do |t| | |||
README.md
Outdated
|
|
||
| Example: | ||
|
|
||
| wayback_machine_downloader http://example.com --user-agent "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:77.0) Gecko/20190101 Firefox/77.0" |
There was a problem hiding this comment.
maybe use another UA than FF for the example, some smart tv UA's have long lifespans
|
Maybe, another suggestion... How about adding 'DNT: 1' headers by default? Not sure if it's something that IA_ARCHIVER cares about thi |
A good suggestion, but I think it's better to do it in a separate MR, where you can make some more adjustments to privacy, or to bypass locks. |
|
would it make sense to add a commandline flag to set a user agent along with defaulting to something like firefox or chrome? |
|
i believe it does |
Possible solution to access restriction problems (502-504, 403 HTTP codes) related to blocking most UserAgents by default (curl, pythonlib, Ruby). With this parameter, you can "disguise" as a browser and eventually bypass the restriction. In this way, 350 thousand pages of one of the sites were previously downloaded (full history from 2008)