Module: Ronin::Web::CLI::SpiderOptions Private

Included in:
Commands::Spider, Commands::Vulns, Commands::Wordlist
Defined in:
lib/ronin/web/cli/spider_options.rb

Overview

This module is part of a private API. You should avoid using this module if possible, as it may be removed or be changed in the future.

Adds options for spidering a website.

Since:

  • 2.0.0

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Instance Attribute Details

#agent_kwargs ⇒ Hash{Symbol => Object} (readonly)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Keyword arguments to initialize a new Spidr::Agent.

Returns:

  • (Hash{Symbol => Object})

Since:

  • 2.0.0



385
386
387
 
# File 'lib/ronin/web/cli/spider_options.rb', line 385

def agent_kwargs
  @agent_kwargs
end

Class Method Details

.included(command) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Adds options for configuring a web spider and spidering a website.

Parameters:

Since:

  • 2.0.0



39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
 
# File 'lib/ronin/web/cli/spider_options.rb', line 39

def self.included(command)
  command.usage '[options] {--host HOST | --domain DOMAIN | --site URL}'

  command.option :host, value: {
                          type:  String,
                          usage: 'HOST'
                        },
                        desc: 'Spiders the specific HOST'

  command.option :domain, value: {
                            type:  String,
                            usage: 'DOMAIN'
                          },
                          desc: 'Spiders the whole domain'

  command.option :site, value: {
                          type:  String,
                          usage: 'URL'
                        },
                        desc: 'Spiders the website, starting at the URL'

  command.option :open_timeout, value: {
                                  type: Integer,
                                  usage: 'SECS',
                                  default: Spidr.open_timeout
                                },
                                desc: 'Sets the connection open timeout' do |timeout|
                                  self.open_timeout = timeout
                                end

  command.option :read_timeout, value: {
                                  type: Integer,
                                  usage: 'SECS',
                                  default: Spidr.read_timeout
                                },
                                desc: 'Sets the read timeout' do |timeout|
                                  self.read_timeout = timeout
                                end

  command.option :ssl_timeout, value: {
                                 type: Integer,
                                 usage: 'SECS',
                                 default: Spidr.ssl_timeout
                               },
                               desc: 'Sets the SSL connection timeout' do |timeout|
                                 self.ssl_timeout = timeout
                               end

  command.option :continue_timeout, value: {
                                      type:    Integer,
                                      usage:   'SECS',
                                      default: Spidr.continue_timeout
                                    },
                                    desc: 'Sets the continue timeout' do |timeout|
                                      self.continue_timeout = timeout
                                    end

  command.option :keep_alive_timeout, value: {
                                        type:    Integer,
                                        usage:   'SECS',
                                        default: Spidr.keep_alive_timeout
                                      },
                                      desc: 'Sets the connection keep alive timeout' do |timeout|
                                        self.keep_alive_timeout = timeout
                                      end

  command.option :proxy, short: '-P',
                         value: {
                           type:  String,
                           usage: 'PROXY'
                         },
                         desc: 'Sets the proxy to use' do |proxy|
                           self.proxy = proxy
                         end

  command.option :header, short: '-H',
                          value: {
                            type:  /\A[^\s:]+:.*\z/,
                            usage: 'NAME: VALUE'
                          },
                          desc: 'Sets a default header' do |header|
                            name, value = header.split(/:\s*/,2)

                            self.default_headers[name] = value
                          end

  command.option :host_header, value: {
                                 type: /\A[^\s=]+=[^\s=]+\z/,
                                 usage: 'NAME=VALUE'
                               },
                               desc: 'Sets a default header' do |name_value|
                                 name, value = name_value.split('=',2)

                                 self.host_headers[name] = value
                               end

  command.option :user_agent_string, short: '-U',
                                     value: {
                                       type:  String,
                                       usage: 'STRING'
                                     },
                                     desc: 'The User-Agent string to use' do |ua|
                                       self.user_agent = ua
                                     end

  command.option :user_agent, short: '-u',
                              value: {
                                type: Support::Network::HTTP::UserAgents::ALIASES.transform_keys { |key|
                                  key.to_s.tr('_','-')
                                }
                              },
                              desc: 'The User-Agent to use' do |name|
                                self.user_agent = name
                              end

  command.option :referer, short: '-R',
                           value: {
                             type:  String,
                             usage: 'URL'
                           },
                           desc: 'Sets the Referer URL' do |referer|
                             self.referer = referer
                           end

  command.option :delay, short: '-d',
                         value: {
                           type:  Numeric,
                           usage: 'SECS'
                         },
                         desc: 'Sets the delay in seconds between each request' do |delay|
                           self.delay = delay
                         end

  command.option :limit, short: '-l',
                         value: {
                           type:  Integer,
                           usage: 'COUNT'
                         },
                         desc: 'Only spiders up to COUNT pages' do |limit|
                           self.limit = limit
                         end

  command.option :max_depth, short: '-d',
                             value: {
                               type:  Integer,
                               usage: 'DEPTH'
                             },
                             desc: 'Only spiders up to max depth' do |depth|
                               self.max_depth = depth
                             end

  command.option :enqueue, value: {
                             type:  String,
                             usage: 'URL'
                           },
                           desc: 'Adds the URL to the queue' do |url|
                             self.queue << url
                           end

  command.option :visited, value: {
                             type:  String,
                             usage: 'URL'
                           },
                           desc: 'Marks the URL as previously visited' do |url|
                             self.history << url
                           end

  command.option :strip_fragments, desc: 'Enables/disables stripping the fragment component of every URL' do
    self.strip_fragments = true
  end

  command.option :strip_query, desc: 'Enables/disables stripping the query component of every URL' do
    self.strip_query = true
  end

  command.option :visit_scheme, value: {
                                  type:  String,
                                  usage: 'SCHEME'
                                },
                                desc: 'Visit URLs with the URI scheme' do |scheme|
                                  self.visit_schemes << scheme
                                end

  command.option :visit_schemes_like, value: {
                                        type:  Regexp,
                                        usage: '/REGEX/'
                                      },
                                      desc: 'Visit URLs with URI schemes that match the REGEX' do |regex|
                                        self.visit_schemes << regex
                                      end

  command.option :ignore_scheme, value: {
                                   type:  String,
                                   usage: 'SCHEME'
                                 },
                                 desc: 'Ignore the URLs with the URI scheme' do |scheme|
                                   self.ignore_schemes << scheme
                                 end

  command.option :ignore_schemes_like, value: {
                                         type:  Regexp,
                                         usage: '/REGEX/'
                                       },
                                       desc: 'Ignore the URLs with URI schemes matching the REGEX' do |regex|
                                         self.ignore_schemes << regex
                                       end

  command.option :visit_host, value: {
                                type:  String,
                                usage: 'HOST'
                              },
                              desc: 'Visit URLs with the matching host name' do |host|
                                self.visit_hosts << host
                              end

  command.option :visit_hosts_like, value: {
                                      type:  Regexp,
                                      usage: '/REGEX/'
                                    },
                                    desc: 'Visit URLs with hostnames that match the REGEX' do |regex|
                                      self.visit_hosts << regex
                                    end

  command.option :ignore_host, value: {
                                 type:  String,
                                 usage: 'HOST'
                               },
                               desc: 'Ignore the host name' do |host|
                                 self.ignore_hosts << host
                               end

  command.option :ignore_hosts_like, value: {
                                       type:  Regexp,
                                       usage: '/REGEX/'
                                     },
                                     desc: 'Ignore the host names matching the REGEX' do |regex|
                                       self.ignore_hosts << regex
                                     end

  command.option :visit_port, value: {
                                type:  Integer,
                                usage: 'PORT'
                              },
                              desc: 'Visit URLs with the matching port number' do |port|
                                self.visit_ports << port
                              end

  command.option :visit_ports_like, value: {
                                      type:  Regexp,
                                      usage: '/REGEX/'
                                    },
                                    desc: 'Visit URLs with port numbers that match the REGEX' do |regex|
                                      self.visit_ports << regex
                                    end

  command.option :ignore_port, value: {
                                 type:  Integer,
                                 usage: 'PORT'
                               },
                               desc: 'Ignore the port number' do |port|
                                 self.ignore_ports << port
                               end

  command.option :ignore_ports_like, value: {
                                       type:  Regexp,
                                       usage: '/REGEX/'
                                     },
                                     desc: 'Ignore the port numbers matching the REGEXP' do |regex|
                                       self.ignore_ports << regex
                                     end

  command.option :visit_link, value: {
                                type:  String,
                                usage: 'URL'
                              },
                              desc: 'Visit the URL' do |link|
                                self.visit_links << link
                              end

  command.option :visit_links_like, value: {
                                      type:  Regexp,
                                      usage: '/REGEX/'
                                    },
                                    desc: 'Visit URLs that match the REGEX' do |regex|
                                      self.visit_links << regex
                                    end

  command.option :ignore_link, value: {
                                 type:  String,
                                 usage: 'URL'
                               },
                               desc: 'Ignore the URL' do |link|
                                 self.ignore_links << link
                               end

  command.option :ignore_links_like, value: {
                                       type:  Regexp,
                                       usage: '/REGEX/'
                                     },
                                     desc: 'Ignore URLs matching the REGEX' do |regex|
                                       self.ignore_links << regex
                                     end

  command.option :visit_ext, value: {
                               type:  String,
                               usage: 'FILE_EXT'
                             },
                             desc: 'Visit URLs with the matching file ext' do |ext|
                               self.visit_exts << ext
                             end

  command.option :visit_exts_like, value: {
                                     type:  Regexp,
                                     usage: '/REGEX/'
                                   },
                                   desc: 'Visit URLs with file exts that match the REGEX' do |regex|
                                     self.visit_exts << regex
                                   end

  command.option :ignore_ext, value: {
                                type:  String,
                                usage: 'FILE_EXT'
                              },
                              desc: 'Ignore the URLs with the file ext' do |ext|
                                self.ignore_exts << ext
                              end

  command.option :ignore_exts_like, value: {
                                      type:  Regexp,
                                      usage: '/REGEX/'
                                    },
                                    desc: 'Ignore URLs with file exts matching the REGEX' do |regex|
                                      self.ignore_exts << regex
                                    end

  command.option :robots, short: '-r',
                          desc:  'Specifies whether to honor robots.txt' do
                            self.robots = true
                          end
end

Instance Method Details

#continue_timeout ⇒ Integer?

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The continue timeout.

Returns:

  • (Integer, nil)

Since:

  • 2.0.0



505
506
507
 
# File 'lib/ronin/web/cli/spider_options.rb', line 505

def continue_timeout
  @agent_kwargs[:continue_timeout]
end

#continue_timeout=(new_timeout) ⇒ Integer

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Sets the continue timeout.

Parameters:

  • new_timeout (Integer)

Returns:

  • (Integer)

Since:

  • 2.0.0



518
519
520
 
# File 'lib/ronin/web/cli/spider_options.rb', line 518

def continue_timeout=(new_timeout)
  @agent_kwargs[:continue_timeout] = new_timeout
end

#default_headers ⇒ Hash{String => String}

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The default headers to send with every request.

Returns:

  • (Hash{String => String})

Since:

  • 2.0.0



578
579
580
 
# File 'lib/ronin/web/cli/spider_options.rb', line 578

def default_headers
  @agent_kwargs[:default_headers] ||= {}
end

#delay ⇒ Integer, ...

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The amount of seconds to pause between each request.

Returns:

  • (Integer, Float, nil)

Since:

  • 2.0.0



648
649
650
 
# File 'lib/ronin/web/cli/spider_options.rb', line 648

def delay
  @agent_kwargs[:delay]
end

#delay=(new_delay) ⇒ Integer, Float

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Sets the amount of seconds to pause between each request.

Parameters:

  • new_delay (Integer, Float)

Returns:

  • (Integer, Float)

Since:

  • 2.0.0



661
662
663
 
# File 'lib/ronin/web/cli/spider_options.rb', line 661

def delay=(new_delay)
  @agent_kwargs[:delay] = new_delay
end

#history ⇒ Array<String>

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The pre-existing history of URLs that have already been spidered.

Returns:

  • (Array<String>)

Since:

  • 2.0.0



731
732
733
 
# File 'lib/ronin/web/cli/spider_options.rb', line 731

def history
  @agent_kwargs[:history] ||= []
end

#host_headers ⇒ Hash{String => String}

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The default Host headers to send with every request.

Returns:

  • (Hash{String => String})

Since:

  • 2.0.0



589
590
591
 
# File 'lib/ronin/web/cli/spider_options.rb', line 589

def host_headers
  @agent_kwargs[:host_headers] ||= {}
end

#ignore_exts ⇒ Array<String>

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The list of URI file extensions to ignore while spidering.

Returns:

  • (Array<String>)

Since:

  • 2.0.0



889
890
891
 
# File 'lib/ronin/web/cli/spider_options.rb', line 889

def ignore_exts
  @agent_kwargs[:ignore_exts] ||= []
end

#ignore_hosts ⇒ Array<String>

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The list of URI hosts to ignore while spidering.

Returns:

  • (Array<String>)

Since:

  • 2.0.0



856
857
858
 
# File 'lib/ronin/web/cli/spider_options.rb', line 856

def ignore_hosts
  @agent_kwargs[:ignore_hosts] ||= []
end

#ignore_links ⇒ Array<String>

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The list of URI links to ignore while spidering.

Returns:

  • (Array<String>)

Since:

  • 2.0.0



878
879
880
 
# File 'lib/ronin/web/cli/spider_options.rb', line 878

def ignore_links
  @agent_kwargs[:ignore_links] ||= []
end

#ignore_ports ⇒ Array<Integer>

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The list of URI ports to ignore while spidering.

Returns:

  • (Array<Integer>)

Since:

  • 2.0.0



867
868
869
 
# File 'lib/ronin/web/cli/spider_options.rb', line 867

def ignore_ports
  @agent_kwargs[:ignore_ports] ||= []
end

#ignore_schemes ⇒ Array<String>

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The list of URI schemes to ignore while spidering.

Returns:

  • (Array<String>)

Since:

  • 2.0.0



845
846
847
 
# File 'lib/ronin/web/cli/spider_options.rb', line 845

def ignore_schemes
  @agent_kwargs[:ignore_schemes] ||= []
end

#initialize(**kwargs) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Initializes the command.

Parameters:

  • kwargs (Hash{Symbol => Object}) —

    Additional keyword arguments.

Since:

  • 2.0.0



393
394
395
396
397
 
# File 'lib/ronin/web/cli/spider_options.rb', line 393

def initialize(**kwargs)
  super(**kwargs)

  @agent_kwargs = {}
end

#keep_alive_timeout ⇒ Integer?

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The Keep-Alive timeout.

Returns:

  • (Integer, nil)

Since:

  • 2.0.0



529
530
531
 
# File 'lib/ronin/web/cli/spider_options.rb', line 529

def keep_alive_timeout
  @agent_kwargs[:keep_alive_timeout]
end

#keep_alive_timeout=(new_timeout) ⇒ Integer

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Sets the Keep-Alive timeout.

Parameters:

  • new_timeout (Integer)

Returns:

  • (Integer)

Since:

  • 2.0.0



542
543
544
 
# File 'lib/ronin/web/cli/spider_options.rb', line 542

def keep_alive_timeout=(new_timeout)
  @agent_kwargs[:keep_alive_timeout] = new_timeout
end

#limit ⇒ Integer?

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The limit to how many URLs to visit.

Returns:

  • (Integer, nil)

Since:

  • 2.0.0



672
673
674
 
# File 'lib/ronin/web/cli/spider_options.rb', line 672

def limit
  @agent_kwargs[:limit]
end

#limit=(new_limit) ⇒ Integer

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Sets the limit of how many URLs to visit.

Parameters:

  • new_limit (Integer)

Returns:

  • (Integer)

Since:

  • 2.0.0



685
686
687
 
# File 'lib/ronin/web/cli/spider_options.rb', line 685

def limit=(new_limit)
  @agent_kwargs[:limit] = new_limit
end

#max_depth ⇒ Integer?

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The maximum depth to spider.

Returns:

  • (Integer, nil)

Since:

  • 2.0.0



696
697
698
 
# File 'lib/ronin/web/cli/spider_options.rb', line 696

def max_depth
  @agent_kwargs[:max_depth]
end

#max_depth=(new_max_depth) ⇒ Integer

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Sets the maximum depth to spider.

Parameters:

  • new_max_depth (Integer)

Returns:

  • (Integer)

Since:

  • 2.0.0



709
710
711
 
# File 'lib/ronin/web/cli/spider_options.rb', line 709

def max_depth=(new_max_depth)
  @agent_kwargs[:max_depth] = new_max_depth
end

#new_agent {|agent| ... } ⇒ Ronin::Web::Spider::Agent

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Creates a new web spider agent.

Yields:

  • (agent) —

    The given block will be given the newly created and configured web spider agent.

Yield Parameters:

  • agent (Ronin::Web::Spider::Agent) —

    The newly created web spider agent.

Returns:

  • (Ronin::Web::Spider::Agent) —

    The newly created web spider agent, after the agent has completed it's spidering.

Since:

  • 2.0.0



413
414
415
416
417
418
419
420
421
422
423
424
 
# File 'lib/ronin/web/cli/spider_options.rb', line 413

def new_agent(&block)
  if options[:host]
    Web::Spider.host(options[:host],**agent_kwargs,&block)
  elsif options[:domain]
    Web::Spider.domain(options[:domain],**agent_kwargs,&block)
  elsif options[:site]
    Web::Spider.site(options[:site],**agent_kwargs,&block)
  else
    print_error "must specify --host, --domain, or --site"
    exit(-1)
  end
end

#open_timeout ⇒ Integer?

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The open connection timeout.

Returns:

  • (Integer, nil)

Since:

  • 2.0.0



433
434
435
 
# File 'lib/ronin/web/cli/spider_options.rb', line 433

def open_timeout
  @agent_kwargs[:open_timeout]
end

#open_timeout=(new_timeout) ⇒ Integer

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Sets the open connection timeout.

Parameters:

  • new_timeout (Integer)

Returns:

  • (Integer)

Since:

  • 2.0.0



446
447
448
 
# File 'lib/ronin/web/cli/spider_options.rb', line 446

def open_timeout=(new_timeout)
  @agent_kwargs[:open_timeout] = new_timeout
end

#proxy ⇒ String?

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The proxy to use for spidering.

Returns:

  • (String, nil)

Since:

  • 0.2.0



553
554
555
 
# File 'lib/ronin/web/cli/spider_options.rb', line 553

def proxy
  @agent_kwargs[:proxy]
end

#proxy=(new_proxy) ⇒ String

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Sets the proxy to use for spidering.

Parameters:

  • new_proxy (String) —

    The new proxy URI.

Returns:

  • (String)

Since:

  • 2.0.0



567
568
569
 
# File 'lib/ronin/web/cli/spider_options.rb', line 567

def proxy=(new_proxy)
  @agent_kwargs[:proxy] = new_proxy
end

#queue ⇒ Array<String>

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The pre-existing queue of URLs to start spidering.

Returns:

  • (Array<String>)

Since:

  • 2.0.0



720
721
722
 
# File 'lib/ronin/web/cli/spider_options.rb', line 720

def queue
  @agent_kwargs[:queue] ||= []
end

#read_timeout ⇒ Integer?

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The read timeout.

Returns:

  • (Integer, nil)

Since:

  • 2.0.0



457
458
459
 
# File 'lib/ronin/web/cli/spider_options.rb', line 457

def read_timeout
  @agent_kwargs[:read_timeout]
end

#read_timeout=(new_timeout) ⇒ Integer

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Sets the read timeout.

Parameters:

  • new_timeout (Integer)

Returns:

  • (Integer)

Since:

  • 2.0.0



470
471
472
 
# File 'lib/ronin/web/cli/spider_options.rb', line 470

def read_timeout=(new_timeout)
  @agent_kwargs[:read_timeout] = new_timeout
end

#referer ⇒ String?

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The Referer header to use for spidering.

Returns:

  • (String, nil)

Since:

  • 2.0.0



624
625
626
 
# File 'lib/ronin/web/cli/spider_options.rb', line 624

def referer
  @agent_kwargs[:referer]
end

#referer=(new_referer) ⇒ String?

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Sets the Referer header to use for spidering.

Parameters:

  • new_referer (String)

Returns:

  • (String, nil)

Since:

  • 2.0.0



637
638
639
 
# File 'lib/ronin/web/cli/spider_options.rb', line 637

def referer=(new_referer)
  @agent_kwargs[:referer] = new_referer
end

#robots ⇒ Boolean

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Whether to honor the robots.txt file while spidering.

Returns:

  • (Boolean)

Since:

  • 2.0.0



900
901
902
 
# File 'lib/ronin/web/cli/spider_options.rb', line 900

def robots
  @agent_kwargs[:robots]
end

#robots=(new_value) ⇒ Boolean

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Sets whether to honor the robots.txt file while spidering.

Parameters:

  • new_value (Boolean)

Returns:

  • (Boolean)

Since:

  • 2.0.0



913
914
915
 
# File 'lib/ronin/web/cli/spider_options.rb', line 913

def robots=(new_value)
  @agent_kwargs[:robots] = new_value
end

#ssl_timeout ⇒ Integer?

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The SSL timeout.

Returns:

  • (Integer, nil)

Since:

  • 2.0.0



481
482
483
 
# File 'lib/ronin/web/cli/spider_options.rb', line 481

def ssl_timeout
  @agent_kwargs[:ssl_timeout]
end

#ssl_timeout=(new_timeout) ⇒ Integer

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Sets the SSL timeout.

Parameters:

  • new_timeout (Integer)

Returns:

  • (Integer)

Since:

  • 2.0.0



494
495
496
 
# File 'lib/ronin/web/cli/spider_options.rb', line 494

def ssl_timeout=(new_timeout)
  @agent_kwargs[:ssl_timeout] = new_timeout
end

#strip_fragments ⇒ Boolean

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Whether to strip the #fragment components of links.

Returns:

  • (Boolean)

Since:

  • 2.0.0



742
743
744
 
# File 'lib/ronin/web/cli/spider_options.rb', line 742

def strip_fragments
  @agent_kwargs[:strip_fragments]
end

#strip_fragments=(new_value) ⇒ Boolean

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Sets whether to strip the #fragment components of links.

Parameters:

  • new_value (Boolean)

Returns:

  • (Boolean)

Since:

  • 2.0.0



755
756
757
 
# File 'lib/ronin/web/cli/spider_options.rb', line 755

def strip_fragments=(new_value)
  @agent_kwargs[:strip_fragments] = new_value
end

#strip_query ⇒ Boolean

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Whether to strip the ?query components of links.

Returns:

  • (Boolean)

Since:

  • 2.0.0



766
767
768
 
# File 'lib/ronin/web/cli/spider_options.rb', line 766

def strip_query
  @agent_kwargs[:strip_query]
end

#strip_query=(new_value) ⇒ Boolean

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Sets whether to strip the ?query components of links.

Parameters:

  • new_value (Boolean)

Returns:

  • (Boolean)

Since:

  • 2.0.0



779
780
781
 
# File 'lib/ronin/web/cli/spider_options.rb', line 779

def strip_query=(new_value)
  @agent_kwargs[:strip_query] = new_value
end

#user_agent ⇒ String?

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Sets the new User-Agent header to use for spidering.

Returns:

  • (String, nil)

Since:

  • 2.0.0



600
601
602
 
# File 'lib/ronin/web/cli/spider_options.rb', line 600

def user_agent
  @agent_kwargs[:user_agent]
end

#user_agent=(new_user_agent) ⇒ String

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Sets the new User-Agent header to use for spidering.

Parameters:

  • new_user_agent (String)

Returns:

  • (String)

Since:

  • 2.0.0



613
614
615
 
# File 'lib/ronin/web/cli/spider_options.rb', line 613

def user_agent=(new_user_agent)
  @agent_kwargs[:user_agent] = new_user_agent
end

#visit_exts ⇒ Array<String>

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The list of URI file extensions to allow spidering.

Returns:

  • (Array<String>)

Since:

  • 2.0.0



834
835
836
 
# File 'lib/ronin/web/cli/spider_options.rb', line 834

def visit_exts
  @agent_kwargs[:exts] ||= []
end

#visit_hosts ⇒ Array<String>

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The list of URI hosts to allow spidering.

Returns:

  • (Array<String>)

Since:

  • 2.0.0



801
802
803
 
# File 'lib/ronin/web/cli/spider_options.rb', line 801

def visit_hosts
  @agent_kwargs[:hosts] ||= []
end

#visit_links ⇒ Array<String>

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The list of URI links to allow spidering.

Returns:

  • (Array<String>)

Since:

  • 2.0.0



823
824
825
 
# File 'lib/ronin/web/cli/spider_options.rb', line 823

def visit_links
  @agent_kwargs[:links] ||= []
end

#visit_ports ⇒ Array<Integer>

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The list of URI ports to allow spidering.

Returns:

  • (Array<Integer>)

Since:

  • 2.0.0



812
813
814
 
# File 'lib/ronin/web/cli/spider_options.rb', line 812

def visit_ports
  @agent_kwargs[:ports] ||= []
end

#visit_schemes ⇒ Array<String>

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The list of URI schemes to allow spidering.

Returns:

  • (Array<String>)

Since:

  • 2.0.0



790
791
792
 
# File 'lib/ronin/web/cli/spider_options.rb', line 790

def visit_schemes
  @agent_kwargs[:schemes] ||= []
end