hensei-api/app/helpers/summon_parser.rb
Justin Edmund 0ccbb7ecfe
Implement wiki parsers (#121)
* Remove ap call

* Fix remix render method

* Downcase username on db end

There was a bug where users with capital letters in their name could not access their profiles after we tried to make things case insensitive.

* Remove ap call and unused code

* Add granblue.team to cors

This works now!

* Implement all-entity search to support tagging objects (#117)

* Add table for multisearch

* Add new route for searching all entities

* Make models multisearchable

We're going to start with Character, Summon, Weapon and Jobs

* Add method to Search controller

This will search with trigram first, and then if there aren't enough results, search with prefixed text search

* Add support for Japanese all-entity search

* Update grid_summons_controller.rb

Set the proper uncap level for transcended summons

* Search is broken in Japanese!

* Grid model object updates

* Adds has_one association to canonical objects
* GridWeapon is_mainhand refactored
* GridCharacter add_awakening refactored

* (WIP) Add support for inclusion/exclusion + refactor

This commit adds basic support for including/excluding objects from collection filters. There is also a refactor of the filter logic as a whole. It is only implemented in `teams` for now and is a work in progress.

* Revert "Grid model object updates"

This reverts commit 70e820b781.

* Revert "(WIP) Add support for inclusion/exclusion + refactor"

This reverts commit b5f9889c00.

* Add new dependencies

* Update database with new columns

* Create WikiError

This is modeled after the errors we might receive from the wiki

* Update GranblueWiki

This is an adaptation and cleanup of the original GranblueWiki class. We extracted the object-related code into a parser, and this class is now only responsible for requests and fetching common property maps.

* Create CharacterParser

The CharacterParser is responsible for taking the response from a wiki object and turning it into something usable. Most of the logic from the original GranblueWiki object ended up here.

To use, you can instantiate a CharacterParser with a Granblue ID. It will create the wiki object for you using the provided character and fetch data when you call `fetch` on the CharacterParser.

You can also fetch data for all characters with the static method `fetch_all`

Currently a specific subset of fields are persisted, but in the future more can and probably should be saved

* Add data tables for weapons and summons

This lets us store wiki data and release dates for weapons and summons

* Add mapping for bullets

* Properly pass props

* Update debug string

* Update character parser to use correct vars

* Add weapon parser

The weapon parser parses through weapon pages.

It has the ability to parse skills but that is not complete yet.

It can go through every weapon on the wiki with the aid of the new `wiki_en` column and parse basic data successfully

* Update code for FLB

If a weapon has ULB, it has FLB too

* Add summon parser

The summon parser parses through summon pages. Aura/Call parsing has not been added.

* Add XLB date column for summons

* Add fetch_from_list static method

This backports the `fetch_from_list` static method to CharacterParser and WeaponParser. This will let us fetch data for a specific set of items instead of having to fetch one-by-one or fetch the entire dataset again.
2023-08-19 23:05:32 -07:00

251 lines
7.1 KiB
Ruby

# frozen_string_literal: true
require 'pry'
# SummonParser parses summon data from gbf.wiki
class SummonParser
attr_reader :granblue_id
def initialize(granblue_id: String, debug: false)
@summon = Summon.find_by(granblue_id: granblue_id)
@wiki = GranblueWiki.new(debug: debug)
@debug = debug || false
end
# Fetches using @wiki and then processes the response
# Returns true if successful, false if not
# Raises an exception if something went wrong
def fetch(name = nil, save: false)
response = fetch_wiki_info(name)
return false if response.nil?
if response.starts_with?('#REDIRECT')
# Fetch the string inside of [[]]
redirect = response[/\[\[(.*?)\]\]/m, 1]
fetch(redirect, save: save)
else
# return response if response[:error]
handle_fetch_success(response, save)
end
end
private
# Handle the response from the wiki if the response is successful
# If the save flag is set, it will persist the data to the database
def handle_fetch_success(response, save)
ap "#{@summon.granblue_id}: Successfully fetched info for #{@summon.wiki_en}" if @debug
extracted = parse_string(response)
unless extracted[:template].nil?
template = @wiki.fetch("Template:#{extracted[:template]}")
extracted.merge!(parse_string(template))
end
info, skills = parse(extracted)
# ap info
# ap skills
persist(info[:info]) if save
true
end
# Fetches the wiki info from the wiki
# Returns the response body
# Raises an exception if something went wrong
def fetch_wiki_info(name = nil)
@wiki.fetch(name || @summon.wiki_en)
rescue WikiError => e
ap e
# ap "There was an error fetching #{e.page}: #{e.message}" if @debug
{
error: {
name: @summon.wiki_en,
granblue_id: @summon.granblue_id
}
}
end
# Iterates over all summons in the database and fetches their data
# If the save flag is set, data is saved to the database
# If the overwrite flag is set, data is fetched even if it already exists
# If the debug flag is set, additional information is printed to the console
def self.fetch_all(save: false, overwrite: false, debug: false, start: nil)
errors = []
summons = Summon.all.order(:granblue_id)
start_index = start.nil? ? 0 : summons.index { |w| w.granblue_id == start }
count = summons.drop(start_index).count
# ap "Start index: #{start_index}"
summons.drop(start_index).each_with_index do |w, i|
percentage = ((i + 1) / count.to_f * 100).round(2)
ap "#{percentage}%: Fetching #{w.wiki_en}... (#{i + 1}/#{count})" if debug
next unless w.release_date.nil? || overwrite
begin
SummonParser.new(granblue_id: w.granblue_id,
debug: debug).fetch(save: save)
rescue WikiError => e
errors.push(e.page)
end
end
ap 'The following pages were unable to be fetched:'
ap errors
end
def self.fetch_list(list: [], save: false, overwrite: false, debug: false, start: nil)
errors = []
start_index = start.nil? ? 0 : list.index { |id| id == start }
count = list.drop(start_index).count
# ap "Start index: #{start_index}"
list.drop(start_index).each_with_index do |id, i|
summon = Summon.find_by(granblue_id: id)
percentage = ((i + 1) / count.to_f * 100).round(2)
ap "#{percentage}%: Fetching #{summon.wiki_en}... (#{i + 1}/#{count})" if debug
next unless summon.release_date.nil? || overwrite
begin
SummonParser.new(granblue_id: summon.granblue_id,
debug: debug).fetch(save: save)
rescue WikiError => e
errors.push(e.page)
end
end
ap 'The following pages were unable to be fetched:'
ap errors
end
# Parses the response string into a hash
def parse_string(string)
data = {}
lines = string.split("\n")
stop_loop = false
lines.each do |line|
next if stop_loop
if line.include?('Gameplay Notes')
stop_loop = true
next
end
if line.starts_with?('{{')
substr = line[2..].strip! || line[2..]
# All template tags start with {{ so we can skip the first two characters
disallowed = %w[#vardefine #lsth About]
next if substr.start_with?(*disallowed)
if substr.start_with?('Summon')
ap "--> Found template: #{substr}" if @debug
substr = substr.split('|').first
data[:template] = substr if substr != 'Summon'
next
end
end
next unless line[0] == '|' && line.size > 2
key, value = line[1..].split('=', 2).map(&:strip)
regex = /\A\{\{\{.*\|\}\}\}\z/
next if value =~ regex
data[key] = value if value
end
data
end
# Parses the hash into a format that can be saved to the database
def parse(hash)
info = {}
skills = {}
info[:name] = { en: hash['name'], ja: hash['jpname'] }
info[:flavor] = { en: hash['flavor'], ja: hash['jpflavor'] }
info[:id] = hash['id']
info[:flb] = hash['evo_max'].to_i >= 4
info[:ulb] = hash['evo_max'].to_i >= 5
info[:xlb] = hash['evo_max'].to_i == 6
info[:rarity] = rarity_from_hash(hash['rarity'])
info[:series] = hash['series']
info[:obtain] = hash['obtain']
info[:hp] = {
min_hp: hash['hp1'].to_i,
max_hp: hash['hp2'].to_i,
max_hp_flb: hash['hp3'].to_i,
max_hp_ulb: hash['hp4'].to_i.zero? ? nil : hash['hp4'].to_i,
max_hp_xlb: hash['hp5'].to_i.zero? ? nil : hash['hp5'].to_i
}
info[:atk] = {
min_atk: hash['atk1'].to_i,
max_atk: hash['atk2'].to_i,
max_atk_flb: hash['atk3'].to_i,
max_atk_ulb: hash['atk4'].to_i.zero? ? nil : hash['atk4'].to_i,
max_atk_xlb: hash['atk5'].to_i.zero? ? nil : hash['atk5'].to_i
}
info[:dates] = {
release_date: parse_date(hash['release_date']),
flb_date: parse_date(hash['4star_date']),
ulb_date: parse_date(hash['5star_date']),
xlb_date: parse_date(hash['6star_date'])
}
info[:links] = {
wiki: { en: hash['name'], ja: hash['link_jpwiki'] },
gamewith: hash['link_gamewith'],
kamigame: hash['link_kamigame']
}
{
info: info.compact
# skills: skills.compact
}
end
# Saves select fields to the database
def persist(hash)
@summon.release_date = hash[:dates][:release_date]
@summon.flb_date = hash[:dates][:flb_date] if hash[:dates].key?(:flb_date)
@summon.ulb_date = hash[:dates][:ulb_date] if hash[:dates].key?(:ulb_date)
@summon.wiki_ja = hash[:links][:wiki][:ja] if hash[:links].key?(:wiki) && hash[:links][:wiki].key?(:ja)
@summon.gamewith = hash[:links][:gamewith] if hash[:links].key?(:gamewith)
@summon.kamigame = hash[:links][:kamigame] if hash[:links].key?(:kamigame)
if @summon.save
ap "#{@summon.granblue_id}: Successfully saved info for #{@summon.wiki_en}" if @debug
puts
true
end
false
end
# Converts rarities from a string to a hash
def rarity_from_hash(string)
string ? GranblueWiki.rarities[string.upcase] : nil
end
# Parses a date string into a Date object
def parse_date(date_str)
Date.parse(date_str) unless date_str.blank?
end
end