Module: Ronin::Web

Includes:
OpenNamespace
Defined in:
lib/ronin/web.rb,
lib/ronin/web/cli.rb,
lib/ronin/web/xml.rb,
lib/ronin/web/html.rb,
lib/ronin/web/root.rb,
lib/ronin/web/version.rb,
lib/ronin/web/mechanize.rb,
lib/ronin/web/cli/command.rb,
lib/ronin/web/cli/ruby_shell.rb,
lib/ronin/web/cli/commands/irb.rb,
lib/ronin/web/cli/commands/new.rb,
lib/ronin/web/cli/commands/diff.rb,
lib/ronin/web/cli/commands/html.rb,
lib/ronin/web/cli/commands/server.rb,
lib/ronin/web/cli/commands/spider.rb,
lib/ronin/web/cli/commands/new/server.rb,
lib/ronin/web/cli/commands/new/spider.rb,
lib/ronin/web/cli/commands/new/webapp.rb,
lib/ronin/web/cli/commands/new/nokogiri.rb,
lib/ronin/web/cli/commands/reverse_proxy.rb

Overview

Top-level web helper methods.

Defined Under Namespace

Modules: HTML, XML Classes: CLI, Mechanize

Constant Summary collapse

ROOT =

This constant is part of a private API. You should avoid using this constant if possible, as it may be removed or be changed in the future.

Path to ronin-web root directory.

File.expand_path(File.join(__dir__,'..','..','..'))
VERSION =

ronin-web Version

'1.0.2'

Class Method Summary collapse

Class Method Details

.agentMechanize

A persistent Mechanize Agent.

Returns:

  • (Mechanize)

    The persistent Mechanize Agent.

See Also:



235
236
237
# File 'lib/ronin/web.rb', line 235

def self.agent
  @agent ||= Mechanize.new
end

.build_html { ... } ⇒ Nokogiri::HTML::Builder

Creates a new Nokogiri::HTML::Builder.

Examples:

Web.build_html do
  html {
    body {
      div(style: 'display:none;') {
        object(classid: 'blabla')
      }
    }
  }
end

Yields:

  • [] The block that will be used to construct the HTML document.

Returns:

  • (Nokogiri::HTML::Builder)

    The new HTML builder object.

See Also:



92
93
94
# File 'lib/ronin/web.rb', line 92

def self.build_html(&block)
  HTML.build(&block)
end

.build_xml { ... } ⇒ Nokogiri::XML::Builder

Creates a new Nokogiri::XML::Builder.

Examples:

Web.build_xml do
  post(id: 2) {
    title { text('some example') }
    body { text('this is one contrived example.') }
  }
end

Yields:

  • [] The block that will be used to construct the XML document.

Returns:

  • (Nokogiri::XML::Builder)

    The new XML builder object.

See Also:



141
142
143
# File 'lib/ronin/web.rb', line 141

def self.build_xml(&block)
  XML.build(&block)
end

.get(url, parameters = {}, headers = {}) {|page| ... } ⇒ Mechanize::Page

Creates a Mechanize Page for the contents at a given URL.

Examples:

Web.get('http://www.rubyinside.com')
# => Mechanize::Page
Web.get('http://www.rubyinside.com') do |page|
  page.search('div.post/h2/a').each do |title|
    puts title.inner_text
  end
end

Parameters:

  • url (URI::Generic, String)

    The URL to request.

  • parameters (Array, Hash) (defaults to: {})

    Additional parameters for the GET request.

  • headers (Hash) (defaults to: {})

    Additional headers for the GET request.

Yields:

  • (page)

    If a block is given, it will be passed the page for the requested URL.

Yield Parameters:

  • page (Mechanize::Page)

    The requested page.

Returns:

  • (Mechanize::Page)

    The requested page.

See Also:



276
277
278
# File 'lib/ronin/web.rb', line 276

def self.get(url,parameters={},headers={},&block)
  agent.get(url,parameters,nil,headers,&block)
end

.get_body(url, parameters = {}, headers = {}) {|body| ... } ⇒ String

Requests the body of the Mechanize Page created from the response of the given URL.

Examples:

Web.get_body('http://www.rubyinside.com') # => String
Web.get_body('http://www.rubyinside.com') do |body|
  puts body
end

Parameters:

  • url (URI::Generic, String)

    The URL to request.

  • parameters (Array, Hash) (defaults to: {})

    Additional parameters for the GET request.

  • headers (Hash) (defaults to: {})

    Additional headers for the GET request.

Yields:

  • (body)

    If a block is given, it will be passed the body of the page.

Yield Parameters:

  • body (String)

    The requested body of the page.

Returns:

  • (String)

    The requested body of the page.

See Also:



314
315
316
317
318
319
# File 'lib/ronin/web.rb', line 314

def self.get_body(url,parameters={},headers={})
  body = get(url,parameters,headers).body

  yield body if block_given?
  return body
end

.html(body) {|doc| ... } ⇒ Nokogiri::HTML::Document

Parses the body of a document into a HTML document object.

Parameters:

  • body (String, IO)

    The body of the document to parse.

Yields:

  • (doc)

    If a block is given, it will be passed the newly created document object.

Yield Parameters:

  • doc (Nokogiri::HTML::Document)

    The new HTML document object.

Returns:

  • (Nokogiri::HTML::Document)

    The new HTML document object.

See Also:



64
65
66
# File 'lib/ronin/web.rb', line 64

def self.html(body,&block)
  HTML.parse(body,&block)
end

.open(url, proxy: Support::Network::HTTP.proxy, user_agent: Support::Network::HTTP.user_agent, referer: nil, cookie: nil, **kwargs) ⇒ File

Opens a URL as a temporary file.

The User-Agent string to use.

Examples:

Open a given URL.

Web.open('https://www.example.com/')

Open a given URL, using a built-in User-Agent:

Web.open('https://www.example.com/', user_agent: :linux)

Open a given URL, using a custom User-Agent string:

Web.open('https://www.example.com/', user_agent: '...')

Open a given URL, using a custom User-Agent string.

Web.open('https://www.example.com/', user_agent: 'the future')

Parameters:

  • user_agent (String, :random, :chrome, :chrome_linux, :chrome_macos, :chrome_windows, :chrome_iphone, :chrome_ipad, :chrome_android, :firefox, :firefox_linux, :firefox_macos, :firefox_windows, :firefox_iphone, :firefox_ipad, :firefox_android, :safari, :safari_macos, :safari_iphone, :safari_ipad, :edge, :linux, :macos, :windows, :iphone, :ipad, :android, nil) (defaults to: Support::Network::HTTP.user_agent)
  • proxy (String, URI::HTTP, nil) (defaults to: Support::Network::HTTP.proxy)

    The proxy URI to use.

  • referer (String, URI::HTTP, nil) (defaults to: nil)

    The optional Referer header to send.

  • cookie (String, Ronin::Support::Network::HTTP::Cookie, nil) (defaults to: nil)

    The optional Cookie header to send.

  • kwargs (Hash{Symbol => Object})

    Additional keyword arguments.

Options Hash (**kwargs):

  • :user (String)

    The HTTP Basic Authentication user name.

  • :password (String)

    The HTTP Basic Authentication password.

  • :content_length_proc (Proc)

    A callback which will be passed the content-length of the HTTP response.

  • :progress_proc (Proc)

    A callback which will be passed the size of each fragment, once received from the server.

Returns:

  • (File)

    The contents of the URL.

See Also:



202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
# File 'lib/ronin/web.rb', line 202

def self.open(url, proxy:      Support::Network::HTTP.proxy,
                   user_agent: Support::Network::HTTP.user_agent,
                   referer:    nil,
                   cookie:     nil,
                   **kwargs)
  uri     = URI(url)
  options = {proxy: proxy, **kwargs}

  if user_agent
    options['User-Agent'] = case user_agent
                            when Symbol
                              Support::Network::HTTP::UserAgents[user_agent]
                            else
                              user_agent
                            end
  end

  options['Referer'] = referer if referer
  options['Cookie']  = cookie  if cookie

  return uri.open(options)
end

.post(url, query = {}, headers = {}) {|page| ... } ⇒ Mechanize::Page

Posts to a given URL and creates a Mechanize Page from the response.

Examples:

Web.post('http://www.rubyinside.com')
# => Mechanize::Page

Parameters:

  • url (URI::Generic, String)

    The URL to request.

  • query (Hash) (defaults to: {})

    Additional query parameters for the POST request.

  • headers (Hash) (defaults to: {})

    Additional headers for the POST request.

Yields:

  • (page)

    If a block is given, it will be passed the page for the requested URL.

Yield Parameters:

  • page (Mechanize::Page)

    The requested page.

Returns:

  • (Mechanize::Page)

    The requested page.

See Also:



351
352
353
# File 'lib/ronin/web.rb', line 351

def self.post(url,query={},headers={},&block)
  agent.post(url,query,headers,&block)
end

.post_body(url, query = {}, headers = {}) {|body| ... } ⇒ Mechanize::Page

Posts to a given URL and returns the body of the Mechanize Page created from the response.

Examples:

Web.post_body('http://www.rubyinside.com')
# => String
Web.post_body('http://www.rubyinside.com') do |body|
  puts body
end

Parameters:

  • url (URI::Generic, String)

    The URL to request.

  • query (Hash) (defaults to: {})

    Additional query parameters for the POST request.

  • headers (Hash) (defaults to: {})

    Additional headers for the POST request.

Yields:

  • (body)

    If a block is given, it will be passed the body of the page.

Yield Parameters:

  • page (Mechanize::Page)

    The body of the requested page.

Returns:

  • (Mechanize::Page)

    The body of the requested page.

See Also:



390
391
392
393
394
395
# File 'lib/ronin/web.rb', line 390

def self.post_body(url,query={},headers={})
  body = post(url,query,headers).body

  yield body if block_given?
  return body
end

.xml(body) {|doc| ... } ⇒ Nokogiri::XML::Document

Parses the body of a document into a XML document object.

Parameters:

  • body (String, IO)

    The body of the document to parse.

Yields:

  • (doc)

    If a block is given, it will be passed the newly created document object.

Yield Parameters:

  • doc (Nokogiri::XML::Document)

    The new XML document object.

Returns:

  • (Nokogiri::XML::Document)

    The new XML document object.

See Also:



116
117
118
# File 'lib/ronin/web.rb', line 116

def self.xml(body,&block)
  XML.parse(body,&block)
end