Module: Ronin::Support::Web::Agent::Mixin

Included in:: Mixin

Defined in:: lib/ronin/support/web/agent/mixin.rb

Overview

Provides helper methods for performing high-level web requests.

Features

Automatically follows redirects.
Provides high-level methods for requesting and parsing HTML, XML, or JSON.
Maintains a persistent connection pool.

Anti-Features

Does not cache files or write to the disk.
Does not evaluate JavaScript.

Instance Method Summary collapse

#web_agent ⇒ Agent
The web agent object.
#web_get(url, **kwargs) {|response| ... } ⇒ Net::HTTPResponse (also: #get)
Gets a URL and returns the response.
#web_get_html(url, **kwargs) ⇒ Nokogiri::HTML::Document (also: #get_html)
Gets the URL and returns the parsed HTML.
#web_get_json(url, **kwargs) ⇒ Hash{String => Object}, Array (also: #get_json)
Gets the URL and returns the parsed JSON.
#web_get_xml(url, **kwargs) ⇒ Nokogiri::XML::Document (also: #get_xml)
Gets the URL and returns the parsed XML.
#web_post(url, **kwargs) {|response| ... } ⇒ Net::HTTPResponse (also: #post)
Performs an HTTP POST to the URL.
#web_post_html(url, **kwargs) ⇒ Nokogiri::HTML::Document (also: #post_html)
Performs an HTTP POST to the URL and parses the HTML response.
#web_post_json(url, **kwargs) ⇒ Hash{String => Object}, Array (also: #post_json)
Performs an HTTP POST to the URL and parses the JSON response.
#web_post_xml(url, **kwargs) ⇒ Nokogiri::XML::Document (also: #post_xml)
Performs an HTTP POST to the URL and parses the XML response.

Instance Method Details

#web_agent ⇒ `Agent`

The web agent object.

Returns:

(Agent)



48
49
50

# File 'lib/ronin/support/web/agent/mixin.rb', line 48

def web_agent
  @web_agent ||= Agent.new
end

#web_get(url, **kwargs) {|response| ... } ⇒ `Net::HTTPResponse` Also known as: get

Note:

This method will follow redirects by default.

Gets a URL and returns the response.

Examples:

response = web_get('https://example.com/')
# => #<Net::HTTPResponse:...>

Parameters:

url (URI::HTTP, Addressable::URI, String) —
The URL to create the HTTP GET request for.
kwargs (Hash) —
a customizable set of options

Options Hash (**kwargs):

:query (String, nil) —
The query-string to append to the request path.
:query_params (Hash, nil) —
The query-params to append to the request path.
:user (String, nil) —
The user to authenticate as.
:password (String, nil) —
The password to authenticate with.
:headers (Hash{Symbol,String => String}, nil) —
Additional HTTP headers to use for the request.
:content_type (String, :text, :xml, :html, :json, nil) —
The Content-Type header value for the request. If a Symbol is given it will be resolved to a common MIME type:
- :text - text/plain
- :xml - text/xml
- :html - text/html
- :json - application/json
:accept (String, :text, :xml, :html, :json, nil) —
The Accept header value for the request. If a Symbol is given it will be resolved to a common MIME type:
- :text - text/plain
- :xml - text/xml
- :html - text/html
- :json - application/json
:cookie (String, Hash{String => String}, Ronin::Support::Network::HTTP::Cookie, nil) —
Additional Cookie header.
- If a Hash is given, it will be converted to a String using Ronin::Support::Network::HTTP::Cookie.
- If the cookie value is empty, the Cookie header will not be set.
:body (String, nil) —
The body of the request.
:form_data (Hash, String, nil) —
The form data that may be sent in the body of the request.
:json (#to_json, nil) —
The JSON data that will be sent in the body of the request. Will also default the Content-Type header to application/json, unless already set.

Yields:

(response) —
If a block is given it will be passed the received HTTP response.

Yield Parameters:

response (Net::HTTPResponse) —
The received HTTP response object.

Returns:

(Net::HTTPResponse) —
The HTTP response object.

Raises:

(TooManyRedirects) —
Maximum number of redirects reached.



130
131
132

# File 'lib/ronin/support/web/agent/mixin.rb', line 130

def web_get(url,**kwargs,&block)
  web_agent.get(url,**kwargs,&block)
end

#web_get_html(url, **kwargs) ⇒ `Nokogiri::HTML::Document` Also known as: get_html

Note:

This method will follow redirects by default.

Gets the URL and returns the parsed HTML.

Examples:

doc = web_get_html('https://example.com/page.html')
# => #<Nokogiri::HTML::Document:...>

Parameters:

url (URI::HTTP, Addressable::URI, String) —
The URL to create the HTTP GET request for.
kwargs (Hash) —
a customizable set of options

Options Hash (**kwargs):

:query (String, nil) —
The query-string to append to the request path.
:query_params (Hash, nil) —
The query-params to append to the request path.
:user (String, nil) —
The user to authenticate as.
:password (String, nil) —
The password to authenticate with.
:headers (Hash{Symbol,String => String}, nil) —
Additional HTTP headers to use for the request.
:content_type (String, :text, :xml, :html, :json, nil) —
The Content-Type header value for the request. If a Symbol is given it will be resolved to a common MIME type:
- :text - text/plain
- :xml - text/xml
- :html - text/html
- :json - application/json
:accept (String, :text, :xml, :html, :json, nil) —
The Accept header value for the request. If a Symbol is given it will be resolved to a common MIME type:
- :text - text/plain
- :xml - text/xml
- :html - text/html
- :json - application/json
:cookie (String, Hash{String => String}, Ronin::Support::Network::HTTP::Cookie, nil) —
Additional Cookie header.
- If a Hash is given, it will be converted to a String using Ronin::Support::Network::HTTP::Cookie.
- If the cookie value is empty, the Cookie header will not be set.
:body (String, nil) —
The body of the request.
:form_data (Hash, String, nil) —
The form data that may be sent in the body of the request.
:json (#to_json, nil) —
The JSON data that will be sent in the body of the request. Will also default the Content-Type header to application/json, unless already set.

Returns:

(Nokogiri::HTML::Document) —
The parsed HTML response.

Raises:

(ContentTypeError) —
Did not receive a response with a Content-Type of text/html.
(TooManyRedirects) —
Maximum number of redirects reached.



159
160
161

# File 'lib/ronin/support/web/agent/mixin.rb', line 159

def web_get_html(url,**kwargs)
  web_agent.get_html(url,**kwargs)
end

#web_get_json(url, **kwargs) ⇒ `Hash{String => Object}`, `Array` Also known as: get_json

Note:

This method will follow redirects by default.

Gets the URL and returns the parsed JSON.

Examples:

json = web_get_json('https://example.com/data.json')
# => {...}

Parameters:

url (URI::HTTP, Addressable::URI, String) —
The URL to create the HTTP GET request for.
kwargs (Hash) —
a customizable set of options

Options Hash (**kwargs):

:query (String, nil) —
The query-string to append to the request path.
:query_params (Hash, nil) —
The query-params to append to the request path.
:user (String, nil) —
The user to authenticate as.
:password (String, nil) —
The password to authenticate with.
:headers (Hash{Symbol,String => String}, nil) —
Additional HTTP headers to use for the request.
:content_type (String, :text, :xml, :html, :json, nil) —
The Content-Type header value for the request. If a Symbol is given it will be resolved to a common MIME type:
- :text - text/plain
- :xml - text/xml
- :html - text/html
- :json - application/json
:accept (String, :text, :xml, :html, :json, nil) —
The Accept header value for the request. If a Symbol is given it will be resolved to a common MIME type:
- :text - text/plain
- :xml - text/xml
- :html - text/html
- :json - application/json
:cookie (String, Hash{String => String}, Ronin::Support::Network::HTTP::Cookie, nil) —
Additional Cookie header.
- If a Hash is given, it will be converted to a String using Ronin::Support::Network::HTTP::Cookie.
- If the cookie value is empty, the Cookie header will not be set.
:body (String, nil) —
The body of the request.
:form_data (Hash, String, nil) —
The form data that may be sent in the body of the request.
:json (#to_json, nil) —
The JSON data that will be sent in the body of the request. Will also default the Content-Type header to application/json, unless already set.

Returns:

(Hash{String => Object}, Array) —
The parsed JSON.

Raises:

(ContentTypeError) —
Did not receive a response with a Content-Type of application/json.
(TooManyRedirects) —
Maximum number of redirects reached.



218
219
220

# File 'lib/ronin/support/web/agent/mixin.rb', line 218

def web_get_json(url,**kwargs)
  web_agent.get_json(url,**kwargs)
end

#web_get_xml(url, **kwargs) ⇒ `Nokogiri::XML::Document` Also known as: get_xml

Note:

This method will follow redirects by default.

Gets the URL and returns the parsed XML.

Examples:

doc = web_get_xml('https://example.com/data.xml')
# => #<Nokogiri::XML::Document:...>

Parameters:

url (URI::HTTP, Addressable::URI, String) —
The URL to create the HTTP GET request for.
kwargs (Hash) —
a customizable set of options

Options Hash (**kwargs):

:query (String, nil) —
The query-string to append to the request path.
:query_params (Hash, nil) —
The query-params to append to the request path.
:user (String, nil) —
The user to authenticate as.
:password (String, nil) —
The password to authenticate with.
:headers (Hash{Symbol,String => String}, nil) —
Additional HTTP headers to use for the request.
:content_type (String, :text, :xml, :html, :json, nil) —
The Content-Type header value for the request. If a Symbol is given it will be resolved to a common MIME type:
- :text - text/plain
- :xml - text/xml
- :html - text/html
- :json - application/json
:accept (String, :text, :xml, :html, :json, nil) —
The Accept header value for the request. If a Symbol is given it will be resolved to a common MIME type:
- :text - text/plain
- :xml - text/xml
- :html - text/html
- :json - application/json
:cookie (String, Hash{String => String}, Ronin::Support::Network::HTTP::Cookie, nil) —
Additional Cookie header.
- If a Hash is given, it will be converted to a String using Ronin::Support::Network::HTTP::Cookie.
- If the cookie value is empty, the Cookie header will not be set.
:body (String, nil) —
The body of the request.
:form_data (Hash, String, nil) —
The form data that may be sent in the body of the request.
:json (#to_json, nil) —
The JSON data that will be sent in the body of the request. Will also default the Content-Type header to application/json, unless already set.

Returns:

(Nokogiri::XML::Document) —
The parsed XML response.

Raises:

(ContentTypeError) —
Did not receive a response with a Content-Type of text/xml.
(TooManyRedirects) —
Maximum number of redirects reached.



188
189
190

# File 'lib/ronin/support/web/agent/mixin.rb', line 188

def web_get_xml(url,**kwargs)
  web_agent.get_xml(url,**kwargs)
end

#web_post(url, **kwargs) {|response| ... } ⇒ `Net::HTTPResponse` Also known as: post

Note:

If the response is an HTTP redirect, then #get will be called to follow any redirects.

Performs an HTTP POST to the URL.

Examples:

response = web_post('https://example.com/form', form_data: {'foo' => 'bar'})
# => #<Net::HTTPResponse:...>

Parameters:

url (URI::HTTP, Addressable::URI, String) —
The URL to create the HTTP GET request for.
kwargs (Hash) —
a customizable set of options

Options Hash (**kwargs):

:query (String, nil) —
The query-string to append to the request path.
:query_params (Hash, nil) —
The query-params to append to the request path.
:user (String, nil) —
The user to authenticate as.
:password (String, nil) —
The password to authenticate with.
:headers (Hash{Symbol,String => String}, nil) —
Additional HTTP headers to use for the request.
:content_type (String, :text, :xml, :html, :json, nil) —
The Content-Type header value for the request. If a Symbol is given it will be resolved to a common MIME type:
- :text - text/plain
- :xml - text/xml
- :html - text/html
- :json - application/json
:accept (String, :text, :xml, :html, :json, nil) —
The Accept header value for the request. If a Symbol is given it will be resolved to a common MIME type:
- :text - text/plain
- :xml - text/xml
- :html - text/html
- :json - application/json
:cookie (String, Hash{String => String}, Ronin::Support::Network::HTTP::Cookie, nil) —
Additional Cookie header.
- If a Hash is given, it will be converted to a String using Ronin::Support::Network::HTTP::Cookie.
- If the cookie value is empty, the Cookie header will not be set.
:body (String, nil) —
The body of the request.
:form_data (Hash, String, nil) —
The form data that may be sent in the body of the request.
:json (#to_json, nil) —
The JSON data that will be sent in the body of the request. Will also default the Content-Type header to application/json, unless already set.

Yields:

(response) —
If a block is given it will be passed the received HTTP response.

Yield Parameters:

response (Net::HTTPResponse) —
The received HTTP response object.

Returns:

(Net::HTTPResponse) —
The HTTP response object.

Raises:

(TooManyRedirects) —
Maximum number of redirects reached.



252
253
254

# File 'lib/ronin/support/web/agent/mixin.rb', line 252

def web_post(url,**kwargs,&block)
  web_agent.post(url,**kwargs,&block)
end

#web_post_html(url, **kwargs) ⇒ `Nokogiri::HTML::Document` Also known as: post_html

Note:

If the response is an HTTP redirect, then #get will be called to follow any redirects.

Performs an HTTP POST to the URL and parses the HTML response.

Examples:

Send a POST request and parses the HTML response:

doc = web_post_html 'https://example.com/form', form_data: {foo: 'bar'})
# => #<Nokogiri::HTML::Document:...>

Parameters:

url (URI::HTTP, Addressable::URI, String) —
The URL to create the HTTP POST request for.
kwargs (Hash) —
a customizable set of options

Options Hash (**kwargs):

:query (String, nil) —
The query-string to append to the request path.
:query_params (Hash, nil) —
The query-params to append to the request path.
:user (String, nil) —
The user to authenticate as.
:password (String, nil) —
The password to authenticate with.
:headers (Hash{Symbol,String => String}, nil) —
Additional HTTP headers to use for the request.
:content_type (String, :text, :xml, :html, :json, nil) —
The Content-Type header value for the request. If a Symbol is given it will be resolved to a common MIME type:
- :text - text/plain
- :xml - text/xml
- :html - text/html
- :json - application/json
:accept (String, :text, :xml, :html, :json, nil) —
The Accept header value for the request. If a Symbol is given it will be resolved to a common MIME type:
- :text - text/plain
- :xml - text/xml
- :html - text/html
- :json - application/json
:cookie (String, Hash{String => String}, Ronin::Support::Network::HTTP::Cookie, nil) —
Additional Cookie header.
- If a Hash is given, it will be converted to a String using Ronin::Support::Network::HTTP::Cookie.
- If the cookie value is empty, the Cookie header will not be set.
:body (String, nil) —
The body of the request.
:form_data (Hash, String, nil) —
The form data that may be sent in the body of the request.
:json (#to_json, nil) —
The JSON data that will be sent in the body of the request. Will also default the Content-Type header to application/json, unless already set.

Returns:

(Nokogiri::HTML::Document) —
The parsed HTML response.

Raises:

(TooManyRedirects) —
Maximum number of redirects reached.
(ContentTypeError) —
Did not receive a response with a Content-Type of text/html.



284
285
286

# File 'lib/ronin/support/web/agent/mixin.rb', line 284

def web_post_html(url,**kwargs)
  web_agent.post_html(url,**kwargs)
end

#web_post_json(url, **kwargs) ⇒ `Hash{String => Object}`, `Array` Also known as: post_json

Note:

If the response is an HTTP redirect, then #get will be called to follow any redirects.

Performs an HTTP POST to the URL and parses the JSON response.

Examples:

Send a POST request to the form and parse the JSON response:

json = web_post_json 'https://example.com/form', form_data: {foo: 'bar'}
# => {...}

Send a POST request containing JSON and parse the JSON response:

json = web_post_json 'https://example.com/api/end-point', json: {foo: 'bar'}
# => {...}

Parameters:

url (URI::HTTP, Addressable::URI, String) —
The URL to create the HTTP POST request for.
kwargs (Hash) —
a customizable set of options

Options Hash (**kwargs):

:query (String, nil) —
The query-string to append to the request path.
:query_params (Hash, nil) —
The query-params to append to the request path.
:user (String, nil) —
The user to authenticate as.
:password (String, nil) —
The password to authenticate with.
:headers (Hash{Symbol,String => String}, nil) —
Additional HTTP headers to use for the request.
:content_type (String, :text, :xml, :html, :json, nil) —
The Content-Type header value for the request. If a Symbol is given it will be resolved to a common MIME type:
- :text - text/plain
- :xml - text/xml
- :html - text/html
- :json - application/json
:accept (String, :text, :xml, :html, :json, nil) —
The Accept header value for the request. If a Symbol is given it will be resolved to a common MIME type:
- :text - text/plain
- :xml - text/xml
- :html - text/html
- :json - application/json
:cookie (String, Hash{String => String}, Ronin::Support::Network::HTTP::Cookie, nil) —
Additional Cookie header.
- If a Hash is given, it will be converted to a String using Ronin::Support::Network::HTTP::Cookie.
- If the cookie value is empty, the Cookie header will not be set.
:body (String, nil) —
The body of the request.
:form_data (Hash, String, nil) —
The form data that may be sent in the body of the request.
:json (#to_json, nil) —
The JSON data that will be sent in the body of the request. Will also default the Content-Type header to application/json, unless already set.

Returns:

(Hash{String => Object}, Array) —
The parses JSON response.

Raises:

(TooManyRedirects) —
Maximum number of redirects reached.
(ContentTypeError) —
Did not receive a response with a Content-Type of application/json.



352
353
354

# File 'lib/ronin/support/web/agent/mixin.rb', line 352

def web_post_json(url,**kwargs)
  web_agent.post_json(url,**kwargs)
end

#web_post_xml(url, **kwargs) ⇒ `Nokogiri::XML::Document` Also known as: post_xml

Note:

If the response is an HTTP redirect, then #get will be called to follow any redirects.

Performs an HTTP POST to the URL and parses the XML response.

Examples:

Send a POST request to the form and parses the XML response:

doc = web_post_xml 'https://example.com/form', form_data: {foo: 'bar'}
# => #<Nokogiri::XML::Document:...>

Parameters:

url (URI::HTTP, Addressable::URI, String) —
The URL to create the HTTP POST request for.
kwargs (Hash) —
a customizable set of options

Options Hash (**kwargs):

:query (String, nil) —
The query-string to append to the request path.
:query_params (Hash, nil) —
The query-params to append to the request path.
:user (String, nil) —
The user to authenticate as.
:password (String, nil) —
The password to authenticate with.
:headers (Hash{Symbol,String => String}, nil) —
Additional HTTP headers to use for the request.
:content_type (String, :text, :xml, :html, :json, nil) —
The Content-Type header value for the request. If a Symbol is given it will be resolved to a common MIME type:
- :text - text/plain
- :xml - text/xml
- :html - text/html
- :json - application/json
:accept (String, :text, :xml, :html, :json, nil) —
The Accept header value for the request. If a Symbol is given it will be resolved to a common MIME type:
- :text - text/plain
- :xml - text/xml
- :html - text/html
- :json - application/json
:cookie (String, Hash{String => String}, Ronin::Support::Network::HTTP::Cookie, nil) —
Additional Cookie header.
- If a Hash is given, it will be converted to a String using Ronin::Support::Network::HTTP::Cookie.
- If the cookie value is empty, the Cookie header will not be set.
:body (String, nil) —
The body of the request.
:form_data (Hash, String, nil) —
The form data that may be sent in the body of the request.
:json (#to_json, nil) —
The JSON data that will be sent in the body of the request. Will also default the Content-Type header to application/json, unless already set.

Returns:

(Nokogiri::XML::Document) —
The parsed XML response.

Raises:

(TooManyRedirects) —
Maximum number of redirects reached.
(ContentTypeError) —
Did not receive a response with a Content-Type of text/xml.



316
317
318

# File 'lib/ronin/support/web/agent/mixin.rb', line 316

def web_post_xml(url,**kwargs)
  web_agent.post_xml(url,**kwargs)
end

Module: Ronin::Support::Web::Agent::Mixin

Overview

Features

Anti-Features

Instance Method Summary collapse

Instance Method Details

#web_agent ⇒ Agent

#web_get(url, **kwargs) {|response| ... } ⇒ Net::HTTPResponse Also known as: get

#web_get_html(url, **kwargs) ⇒ Nokogiri::HTML::Document Also known as: get_html

#web_get_json(url, **kwargs) ⇒ Hash{String => Object}, Array Also known as: get_json

#web_get_xml(url, **kwargs) ⇒ Nokogiri::XML::Document Also known as: get_xml

#web_post(url, **kwargs) {|response| ... } ⇒ Net::HTTPResponse Also known as: post

#web_post_html(url, **kwargs) ⇒ Nokogiri::HTML::Document Also known as: post_html

#web_post_json(url, **kwargs) ⇒ Hash{String => Object}, Array Also known as: post_json

#web_post_xml(url, **kwargs) ⇒ Nokogiri::XML::Document Also known as: post_xml

#web_agent ⇒ `Agent`

#web_get(url, **kwargs) {|response| ... } ⇒ `Net::HTTPResponse` Also known as: get

#web_get_html(url, **kwargs) ⇒ `Nokogiri::HTML::Document` Also known as: get_html

#web_get_json(url, **kwargs) ⇒ `Hash{String => Object}`, `Array` Also known as: get_json

#web_get_xml(url, **kwargs) ⇒ `Nokogiri::XML::Document` Also known as: get_xml

#web_post(url, **kwargs) {|response| ... } ⇒ `Net::HTTPResponse` Also known as: post

#web_post_html(url, **kwargs) ⇒ `Nokogiri::HTML::Document` Also known as: post_html

#web_post_json(url, **kwargs) ⇒ `Hash{String => Object}`, `Array` Also known as: post_json

#web_post_xml(url, **kwargs) ⇒ `Nokogiri::XML::Document` Also known as: post_xml