This refactor focuses on implementing parallelization. This allows us to pass in a number of threads and download concurrently. This makes downloading lots of images a lot faster.
* Weapons can now download the “raw”image size, which is the weapon art without a background
* Characters now download the “detail” image size, which is a horizontal crop of the character’s art
* Summons now download the “detail” image size, which is a horizontal crop of the summon’s art
* Summons also download “ls” and “m” instead of “party_main” and “party_sub”, as they match the aspect ratio of weapon sizes better, which should make our lives a lot easier.
* Update weapon series migration
This update fixes MigrateWeaponSeries from 20250218 such that it can be run on an empty database without throwing errors.
* Update .gitignore
Hide backups and logs directories, since we’ll be storing these in the project folder. Also hide mise’s .local directory.
* Change NewRelic log directory
Moved from log/ to logs/
* Add rake task for backing up/restoring prod db
* Rubocop fixes
* Fix error where :preview_state didn’t have an attribute
* Add supervisord ini
This uses my local paths, so we should try to abstract that away later.
* Ignore mise.toml
* Update weapon series migration
This update fixes MigrateWeaponSeries from 20250218 such that it can be run on an empty database without throwing errors.
* Add items from late 2025-02
* Yuel (Grand)
* Tsukuyomi
* Sennen Goji
* Nightgaze Gale
* Bane of Avidya
* Klesha-Cleaning Dharmachakra
* Add default value for limit column
* Update weapon series migration
This update fixes MigrateWeaponSeries from 20250218 such that it can be run on an empty database without throwing errors.
* Add items from late 2025-02
* Yuel (Grand)
* Tsukuyomi
* Sennen Goji
* Nightgaze Gale
* Bane of Avidya
* Klesha-Cleaning Dharmachakra
* Add new items
Items from:
- Unite and Fight (January 2025)
- Story Event (January 2025)
- Negima Collab (February 2025)
- January Flash Gala
- January Legend Festival
- February Flash Gala
- February character uncap
Data fixes for
- Chat Noir FLB (rarity was set to a bad value)
- End of year weapons (FLB date not set)
* Fix CSVs
* Fix data update rake task
* Add CSV require in BaseImporter
* Update test csvs
* Fix count filters and refactor apply_filters
* Update party_querying_concern.rb
* +tests/-debug logs
* Make party association optional in Job
* Updates for weapon series
- Change to new series numbers
- Add static method for querying whether the weapon's element is changeable
- Add a new method to return a text slug for the weapon's series
* Add and update test data
- Updates canonical.rb for loading multiple types of data with multiple types of associations
- Adds test data for Guidebooks, Job Accessories, Job Skills, and Jobs
- Updates test data for Weapons and Summons
* Migrations
- Adds series of migrations for changing the weapon's series to the values used by Cygames
- Shuffled around some foreign keys
* Implement BaseProcessor
Processors are in charge of processing deck data straight from Granblue.
* Implement CharacterProcessor
Process character data from deck
* Implement WeaponProcessor
Process weapon data from deck
* Implement JobProcessor
Process job, job skill, and job accessory data from deck
* Implement SummonProcessor
Process summon data from deck
* Update SummonProcessor to work like the others
* ImportController should use processors
* Process element for changeable weapons
* Moved queries into PartyQueryBuilder service
PartyQueryBuilder supersedes PartyQueryingConcern as it is also used for UsersController (and is our fix for profiles being broken)
* Implement PartyQueryBuilder in controllers
* Update summon_transformer.rb
This should fix the transformer so that we properly capture summons and subaura summons
* Update parties_controller_spec.rb
* Add NewRelic license key
* Add Sentry
Why not?
* Install Rspec
* Create .aidigestignore
* Update rails_helper
- Added sections and comments
- Add support for loading via canonical.rb
- Add FactoryBot syntax methods
- Disable SQL logging in test environment
* Move gems around
* Add canonical.rb and test env CSVs
We load these CSVs via canonical.rb when we run tests as a data source for canonical objects.
* Remove RBS for now
This is too much and we need to find the right solution
* Refactor GridSummonsController and add tests
* Create GridSummon factory
* Refactor GridSummon and add documentation and tests
* Create have_error_on.rb
* Update .aidigestignore
* Fix warnings
* Add GridWeapons and Parties factories
* Refactor GridWeapon and add documentation and tests
* Create .rubocop.yml
* Create no_weapon_provided_error.rb
* Refactor GridWeaponsController
- Refactors controller
- Adds YARD documentation
- Adds Rspec tests
* Refactor GridSummonsController
- Refactors controller
- Adds YARD documentation
- Adds Rspec tests
* Enable shoulda/matchers
* Update User factory
* Update party.rb
We moved updating the party's element and extra flag to inside the party. We use an after_commit hook to minimize the amount of queries we're running to do this.
* Update party.rb
We change setting the edit key to use the conditional assignment operator so that it doesn't get overridden when we're running tests. This shouldn't have an effect in production.
* Update api_controller.rb
Change render_unprocessable_entity_response to render the errors hash instead of the exception so that we get more helpful errors.
* Add new errors
Added NoCharacterProvidedError and NoSummonProvidedError
* Add tests and docs to GridCharacter
We added a factory, spec and documentation to the GridCharacter model
* Ensure numericality
* Move enums into GranblueEnums
We don't use these yet, but it gives us a structured place to pull them from.
* Refactor GridCharactersController
- Refactors controller
- Adds YARD documentation
- Adds Rspec tests
* Add debug hook and other small changes
* Update grid_characters_controller.rb
Removes logs
* Update .gitignore
* Update .aidigestignore
* Refactored PartiesController
- Split PartiesController into three concerns
- Implemented testing for PartiesController and two concerns
- Implemented fixes across other files to ensure PartiesController tests pass
- Added Favorites factory
* Implement SimpleCov
* Refactor Party model
- Refactors Party model
- Adds tests
- Adds documentation
* Update granblue_enums.rb
Remove included block
* Update gitignore
There is a mystery postgres folder and we are going to ignore it
* Add migrations
* Update preview state default to pending
* Adds indexes
* Adds PgHero and PgStatements
* Update .gitignore
* Update Gemfile
Production:
- `newrelic_rpm`
Development:
- `pg_query`
- `prosopite`
* Configure Sidekiq
Create job for cleaning up party previews
* Configure Prosopite and remove CacheFreeLogger
* Enable query logging
* Update api_controller.rb
Add N+1 detectioin via Prosopite in development/test environments
* Refactor canonical object blueprints
* Refactor grid object blueprints
* Remove N+1 from grid object models
Reimplementing `character` `summon` and `weapon` was making N+1s which made queries really slow
* Add counter caches to party
* Add preview generation helpers
The Party model can respond to queries about its preview state with the following models:
- `schedule_preview_generation`
- `preview_content_changed?`
- `preview_expired?`
- `should_generate_preview?`
- `ready_for_preview?`
- `needs_preview_generation?`
- `preview_relevant_attributes`
Removes the following methods:
- `schedule_preview_regeneration`
- `preview_relevant_changes?`
* Add cache to is_favorited(user)
* Refactored PartyBlueprint to minimize N+1s
* Remove preview expiry constants
These are defined in the Coordinator instead
* Add method comments
* Create parties_controller.rbs
* Update logic and logs
* Updates excluded methods and calculate_count
* Use `includes` instead of `joins`
* Use a less-insane way of counting
* Adds a helper method for party privacy
* Update filter condition helpers
Just minor refactoring
* Fix old view name in PartyBlueprint
* Refactor parties#create
* Remove redundant return
* Update parties_controller.rbs
* Update parties#index
* Update parties_controller.rb
Updates apply_includes and apply_excludes, along with modifying id_to_table and build_query
* Update parties_controller.rb
Adds the rest of the changes, too tired to write them all out. Some preview generation, some filtering
* Refactor parties#index and parties#favorites
These are mostly the same methods, so we remove common code into build_parties_query and render_paginated_parties
* Alias table name to object to maintain API consistency
* Maintain API consistency with raid blueprint
* Optimize party loading by adding eager loading to `set_from_slug`
- Refactored `set_from_slug` to use `includes` for eager loading associated models:
- `user`, `job`, `raid` (with `group`)
- `characters` (with `character` and `awakening`)
- `weapons` (with `weapon`, `awakenings`, `weapon_key1`, `weapon_key2`, `weapon_key3`)
- `summons` (with `summon`)
- `guidebooks` (`guidebook1`, `guidebook2`, `guidebook3`)
- `source_party`, `remixes`, `skills`, and `accessory`
- This change improves query efficiency by reducing N+1 queries and ensures all relevant associations are preloaded.
- Removed redundant favorite check as it was not necessary in this context.
* Refactor grid blueprints
- **GridCharacterBlueprint:**
- Removed `:minimal` view restriction on `party` association.
- Improved nil checks for `ring1`, `ring2`, and `earring` to prevent errors.
- Converted string values in `awakening_level`, `over_mastery`, and `aetherial_mastery` fields to integers for consistency.
- Ensured `over_mastery` and `aetherial_mastery` only include valid entries, filtering out blank or zero-modifier values.
- **GridWeaponBlueprint:**
- Removed `:minimal` view restriction on `party` association.
- Ensured `weapon` association exists before accessing `ax`, `series`, or `awakening`.
- Improved conditional checks for `weapon_keys` to prevent errors when `weapon` or `series` is nil.
- Converted `awakening_level` field to integer for consistency.
- **GridCharacterBlueprint:**
- Removed `:minimal` view restriction on `party` association.
* Update raid blueprints
- Show flat representation of raid group in RaidBlueprint's nested view
- Show nested representation of raid in RaidGroupBlueprint's full view
* Move n+1 detection to around_action hook
* Improve handling mastery bonuses
- Improved handling of nested attributes:
- Replaced old mastery structure with new `rings` and `awakening` assignments.
- Added `new_rings` and `new_awakening` virtual attributes for easier updates.
- Updated `assign_attributes` to exclude `rings` and `awakening` to prevent conflicts.
- Enhanced parameter transformation:
- Introduced `transform_character_params` to process `rings`, `awakening`, and `earring` more reliably.
- Ensured proper type conversion (`to_i`) for numeric values in `uncap_level`, `transcendence_step`, and `awakening_level`.
- Improved error handling for missing values by setting defaults where needed.
- Optimized database queries:
- Added `.includes(:awakening)` to `set` to prevent N+1 query issues.
- Updated strong parameters:
- Changed `rings` from individual keys (`ring1`, `ring2`, etc.) to a structured array format.
- Refactored permitted attributes to align with the new nested structure.
* Eager-load jobs when querying job skills
* Eager load raids/groups when querying
* Update users_controller.rb
More efficient way of denoting favorited parties.
* Update awakening.rb
- Removes explicitly defined associations and adds ActiveRecord associations instead
* Update party.rb
- Removes favorited accessor
- Renames derivative_parties to remixes and adds in-built sort
* Update weapon_awakening.rb
- Removes redefined explicit associations
* Update grid_character.rb
- Adds code transforming incoming ring and awakening values into something the db understands
* Update character.rb
Add explicit Awakenings enum
* Update coordinator.rb
Adds 'queued' as a state for generation
* Add default preview images
* Update application.rb
* Adds app assets path in API mode
* Cleans up file
* Create assets.rb
An initializer for font assets (for image generation)
* Updates to Canvas and Coordinator
* Update parties_controller.rb
* Adds retry header if generation is still in progress
* Streams S3 content instead of redirecting to prevent 302
* Update coordinator.rbs
* Create previews.rake
A rake task for generating images offline
* Add commands to build phase
* Only re-index search when records are updated
* Add logs and fixes
* Add preview_s3_key to Parties
* Add some extra packages to Nixfile
* Add logging around custom font use
* Add mini_magick and rufus-scheduler
* Expose attributes and add sigs to AwsService
* Get Party ready for preview state
* Added new fields for preview state and generated_at timestamp
* Add preview state enum to model
* Add preview_relevant_changes? after_commit hook
* Add jobs for generating and cleaning up party previews
* Add new endpoints to PartiesController
* `preview` shows the preview and queues it up for generation if it doesn't exist yet
* `regenerate_preview` allows the party owner to force regeneration of previews
* Schedule jobs
* Stalled jobs are checked every 5 minutes
* Failed jobs are retried every hour
* Old preview jobs are cleaned up daily
* Add the preview service
This is where the bulk of the work is. This service renders out the preview images bit by bit. Currently we render the party name, creator, job icon, and weapon grid.
This includes signatures and some fonts.
* Move app/helpers/granblue_wiki to lib/parsers/wiki
This clears up the namespace beginning with "Granblue"
* Removed some top-level Granblue libs
DataImporter and DownloadManager exist inside of the PostDeployment namespace now so these files are redundant
* Fix Downloaders namespace
Our namespace was singular Downloader, now it is plural Downloaders to match the folder name
* Fix import paths
* DownloadManager was moved to downloaders/
* import_data task now uses the PostDeployment version of DataImporter
* Update application.rb
Eager-Load/Autoload the lib/ folder
* Update cors.rb
Add Granblue website and Extension ID to CORS
* Add transformers
Transformers take raw data from Granblue Fantasy and transforms them into hensei-compatible JSON. Transformers heavily borrow from vazkii/hensei-transfer.
* Add ImportController and route
This adds the controller that handles creating a full party from transformed Granblue Fantasy data
* Adds more new items to the database
* Updates to older migrations
These were manually updated in the database
* Adds more new items to the database
* Fixes misclassification of Grand Weapons
* Add the final new items
* Delete 20250115-summons-003.csv
* Updates weapons with granblue_id reference to Characters
* Remove extra empty lines
* Adds Orologia summon
* Modify deployment manager
Use - to make it easier to copy paste into Github PR
* Fresh credentials.yml.enc
* Update .ruby-gemset
* Made PostDeploymentManager modular
We broke PostDeploymentManager out into several files to make it easier to maintain.
We also added a "force" mode that forces the script to consider all CSV files. This is useful for testing the post-deploy script itself. This should only be used in test mode or you will dirty your database.
We also fine tuned some of the logging to make sure that both verbose and non-verbose modes are helpful.
