Class: Ronin::Support::Network::PublicSuffix::List

Inherits:
SuffixSet
  • Object
show all
Includes:
Enumerable
Defined in:
lib/ronin/support/network/public_suffix/list.rb

Overview

Represents the public suffix list.

Since:

  • 1.0.0

Constant Summary collapse

FILE_NAME =

File name of the public suffix list.

Since:

  • 1.0.0

'public_suffix_list.dat'
URL =

The https://publicsuffix.org/list/public_suffix_list.dat URL.

Since:

  • 1.0.0

"https://publicsuffix.org/list/#{FILE_NAME}"
PATH =

The path to ~/.cache/ronin/ronin-support/public_suffix_list.dat list file.

Since:

  • 1.0.0

File.join(Home::CACHE_DIR,'ronin','ronin-support',FILE_NAME)
ONE_DAY =

One day in seconds.

Since:

  • 1.0.0

60 * 60 * 24

Instance Attribute Summary collapse

Attributes inherited from SuffixSet

#suffixes

Class Method Summary collapse

Instance Method Summary collapse

Methods included from Enumerable

#map_hash

Methods inherited from SuffixSet

#each, #icann, #length, #non_wildcards, #private, #to_a, #type, #wildcards

Constructor Details

#initialize(path = PATH) ⇒ List

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Initializes the public suffix list.

Parameters:

  • path (String) (defaults to: PATH) —

    The path to the list file.

Since:

  • 1.0.0



71
72
73
74
75
76
 
# File 'lib/ronin/support/network/public_suffix/list.rb', line 71

def initialize(path=PATH)
  super()

  @path = path
  @tree = {}
end

Instance Attribute Details

#path ⇒ String (readonly)

The path to the list file.

Returns:

Since:

  • 1.0.0



56
57
58
 
# File 'lib/ronin/support/network/public_suffix/list.rb', line 56

def path
  @path
end

#tree ⇒ Hash{String => Hash} (readonly)

The tree of all public suffix TLDs.

Returns:

Since:

  • 1.0.0



61
62
63
 
# File 'lib/ronin/support/network/public_suffix/list.rb', line 61

def tree
  @tree
end

Class Method Details

.download(url: URL, path: PATH) ⇒ Object

Downloads the list file.

Parameters:

  • url (String) (defaults to: URL) —

    An optional alternate URL to download the ip2asn-combined.tsv.gz file.

  • path (String) (defaults to: PATH) —

    An optional alternate path to the list file.

Since:

  • 1.0.0



115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
 
# File 'lib/ronin/support/network/public_suffix/list.rb', line 115

def self.download(url: URL, path: PATH)
  uri = URI(url)

  Net::HTTP.start(uri.host,uri.port, use_ssl: true) do |http|
    request  = Net::HTTP::Get.new(uri.path)

    http.request(request) do |response|
      FileUtils.mkdir_p(File.dirname(path))

      File.open("#{path}.part",'wb') do |file|
        response.read_body do |chunk|
          file.write(chunk)
        end
      end

      FileUtils.mv("#{path}.part",path)
    end
  end
end

.downloaded?(path = PATH) ⇒ Boolean

Determines whether the list file has been previously downloaded.

Parameters:

  • path (String) (defaults to: PATH) —

    An optional alternate path to the list file.

Returns:

  • (Boolean)

Since:

  • 1.0.0



86
87
88
 
# File 'lib/ronin/support/network/public_suffix/list.rb', line 86

def self.downloaded?(path=PATH)
  File.file?(path)
end

.load_file(path = PATH) ⇒ List

Loads the public suffix list from the given file.

Parameters:

  • path (String) (defaults to: PATH) —

    The path to the public suffix list file.

Returns:

  • (List) —

    The parsed public suffix list file.

Since:

  • 1.0.0



200
201
202
203
204
205
206
207
208
 
# File 'lib/ronin/support/network/public_suffix/list.rb', line 200

def self.load_file(path=PATH)
  list = new(path)

  parse(path) do |suffix|
    list << suffix
  end

  return list
end

.parse(path = PATH) {|suffix| ... } ⇒ Enumerator

Parses the contents of the list file.

Parameters:

  • path (String) (defaults to: PATH) —

    An optional alternate path to the list file.

Yields:

  • (suffix) —

    If a block is given, it will be passed each parsed suffix from the list file.

Yield Parameters:

  • suffix (Suffix) —

    A parsed suffix in the list file.

Returns:

  • (Enumerator) —

    If no block is given, an Enumerator object will be returned.

Since:

  • 1.0.0



173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
 
# File 'lib/ronin/support/network/public_suffix/list.rb', line 173

def self.parse(path=PATH)
  return enum_for(__method__,path) unless block_given?

  type = nil

  File.open(path) do |file|
    file.each_line(chomp: true) do |line|
      if line == '// ===BEGIN ICANN DOMAINS==='
        type = :icann
      elsif line == '// ===BEGIN PRIVATE DOMAINS==='
        type = :private
      elsif !(line.empty? || line.start_with?('//'))
        yield Suffix.new(line, type: type)
      end
    end
  end
end

.stale?(path = PATH) ⇒ Boolean

Determines if the downloaded list file is older than one day.

Parameters:

  • path (String) (defaults to: PATH) —

    An optional alternate path to the list file.

Returns:

  • (Boolean)

Since:

  • 1.0.0



101
102
103
 
# File 'lib/ronin/support/network/public_suffix/list.rb', line 101

def self.stale?(path=PATH)
  !File.file?(path) || File.stat(path).mtime < (Time.now - ONE_DAY)
end

.update(url: URL, path: PATH) ⇒ Object

Optionally update the cached list file if it is older than one day.

Parameters:

  • url (String) (defaults to: URL) —

    An optional alternate URL to download the ip2asn-combined.tsv.gz file.

  • path (String) (defaults to: PATH) —

    An optional alternate path to the list file.

Since:

  • 1.0.0



145
146
147
148
149
150
151
152
153
154
155
 
# File 'lib/ronin/support/network/public_suffix/list.rb', line 145

def self.update(url: URL, path: PATH)
  if !downloaded?(path)
    download(url: url, path: path)
  elsif stale?(path)
    begin
      download(url: url, path: path)
    rescue
      # ignore any network failures
    end
  end
end

Instance Method Details

#<<(suffix) ⇒ self

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Adds a public suffix to the list.

Parameters:

  • suffix (Suffix) —

    The suffix String to add.

Returns:

  • (self)

Since:

  • 1.0.0



220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
 
# File 'lib/ronin/support/network/public_suffix/list.rb', line 220

def <<(suffix)
  super(suffix)

  if suffix.name.include?('.')
    tree = @tree

    suffix.name.split('.').reverse_each.each_cons(2) do |parent,child|
      subtree = tree[parent] ||= {}

      subtree[child] ||= nil

      tree = subtree
    end
  else
    @tree[suffix.name] ||= nil
  end

  return self
end

#inspect ⇒ String

Inspects the public suffix list.

Returns:

  • (String) —

    The inspected list object.

Since:

  • 1.0.0



296
297
298
 
# File 'lib/ronin/support/network/public_suffix/list.rb', line 296

def inspect
  "#<#{self.class}: #{@path}>"
end

#split(host_name) ⇒ (String, String)

Splits a hostname into it's name and public suffix components.

Parameters:

  • host_name (String) —

    The host name to split.

Returns:

  • ((String, String)) —

    The host name's name and public suffix components.

Raises:

  • (InvalidHostname) —

    The given hostname does not end with a valid suffix.

Since:

  • 1.0.0



252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
 
# File 'lib/ronin/support/network/public_suffix/list.rb', line 252

def split(host_name)
  components = host_name.split('.')
  suffixes   = []

  tree = @tree

  while tree
    component = components.last

    tld, subtree = tree.find do |tld,subtree|
      tld == '*' || component == tld
    end

    suffixes.prepend(components.pop) if tld
    tree = subtree
  end

  if suffixes.empty?
    raise(InvalidHostname,"hostname does not have a valid suffix: #{host_name.inspect}")
  end

  return components.join('.'), suffixes.join('.')
end

#to_regexp ⇒ Regexp

Creates a regular expression that can match every domain suffix in the list.

Returns:

Since:

  • 1.0.0



282
283
284
285
286
287
288
 
# File 'lib/ronin/support/network/public_suffix/list.rb', line 282

def to_regexp
  regexp = Regexp.union(@tree.map { |tld,subtree|
    tld_regexp(tld,subtree)
  })

  return /(?<=[^a-zA-Z0-9_-]|^)#{regexp}(?=[^\.a-z0-9-]|$)/
end