hensei-api/docs/importers.md

Data Importers Documentation

The importer system provides a framework for importing game data from CSV files into the database. It supports test mode for validation, tracks new and updated records, and provides detailed error reporting.

Architecture

Base Importer

All importers inherit from BaseImporter which provides:

• CSV parsing and validation
• Test mode for dry runs
• New and updated record tracking
• Error handling and reporting
• Verbose logging support
• Batch processing with ActiveRecord transactions

Available Importers

CharacterImporter

Imports character data from CSV files.

Required CSV Fields:

• granblue_id - Unique character ID
• name_en - English name
• name_jp - Japanese name
• rarity - Character rarity
• element - Element type
• flb - Has 5★ uncap (true/false)
• ulb - Has 6★ uncap (true/false)
• wiki_en - Wiki page name

Example CSV:

granblue_id,name_en,name_jp,rarity,element,flb,ulb,wiki_en
3040001000,Katalina,カタリナ,4,3,true,false,Katalina

WeaponImporter

Imports weapon data from CSV files.

Required CSV Fields:

• granblue_id - Unique weapon ID
• name_en - English name
• name_jp - Japanese name
• rarity - Weapon rarity
• element - Element type
• weapon_type - Type of weapon
• wiki_en - Wiki page name

Example CSV:

granblue_id,name_en,name_jp,rarity,element,weapon_type,wiki_en
1040001000,Murgleis,ミュルグレス,5,3,1,Murgleis

SummonImporter

Imports summon data from CSV files.

Required CSV Fields:

• granblue_id - Unique summon ID
• name_en - English name
• name_jp - Japanese name
• rarity - Summon rarity
• element - Element type
• max_level - Maximum level
• wiki_en - Wiki page name

Example CSV:

granblue_id,name_en,name_jp,rarity,element,max_level,wiki_en
2040001000,Bahamut,バハムート,5,0,150,Bahamut

Usage

Ruby API

# Import character data
importer = Granblue::Importers::CharacterImporter.new(
  'db/seed/updates/characters.csv',
  test_mode: false,
  verbose: true
)
result = importer.import

# Check results
result[:new_records]     # => Array of newly created records
result[:updated_records] # => Array of updated records
result[:errors]          # => Array of error messages

# Test mode - validate without importing
test_importer = Granblue::Importers::WeaponImporter.new(
  'db/seed/updates/weapons.csv',
  test_mode: true,
  verbose: true
)
test_result = test_importer.import

Rake Task

The main import task processes all CSV files in db/seed/updates/:

# Import all CSV files
rake granblue:import_data

# Test mode - validate without importing
rake granblue:import_data TEST=true

# Verbose output
rake granblue:import_data VERBOSE=true

# Both test and verbose
rake granblue:import_data TEST=true VERBOSE=true

CSV File Format

File Location

Place CSV files in: db/seed/updates/

Naming Convention

• characters_YYYYMMDD.csv - Character data
• weapons_YYYYMMDD.csv - Weapon data
• summons_YYYYMMDD.csv - Summon data

Encoding

• UTF-8 encoding required
• Unix line endings (LF) preferred

Headers

• First row must contain field names
• Field names are case-sensitive
• Order doesn't matter

Data Types

• Strings: Plain text, quotes optional unless contains commas
• Numbers: Integer or decimal values
• Booleans: true or false (lowercase)
• Dates: ISO 8601 format (YYYY-MM-DD)
• Empty values: Leave blank or use empty string

Field Mappings

Element Values

0 = Null/None
1 = Wind
2 = Fire
3 = Water
4 = Earth
5 = Dark
6 = Light

Weapon Types

1 = Sword
2 = Dagger
3 = Spear
4 = Axe
5 = Staff
6 = Gun
7 = Melee
8 = Bow
9 = Harp
10 = Katana

Rarity Values

1 = R (Rare)
2 = SR (Super Rare)
3 = SSR (Super Super Rare)
4 = SSR+
5 = Grand/Limited

Test Mode

Test mode validates data without making database changes:

rake granblue:import_data TEST=true

Test mode will:

1. Parse CSV files
2. Validate all data
3. Check for duplicates
4. Report what would be created/updated
5. Show any validation errors
6. NOT save to database

Output example:

[TEST MODE] Would create Character: Katalina (3040001000)
[TEST MODE] Would update Weapon: Murgleis (1040001000)
[TEST MODE] Validation error for Summon: Invalid element value

Error Handling

Import Errors

The importer tracks various error types:

{
  errors: [
    {
      row: 5,
      field: 'element',
      message: 'Invalid element value: 99',
      record: { granblue_id: '3040001000', name_en: 'Katalina' }
    }
  ]
}

Validation Errors

Records are validated before save:

• Required fields must be present
• Granblue ID must be unique
• Element must be valid (0-6)
• Rarity must be valid (1-5)

Duplicate Handling

When a record with the same granblue_id exists:

1. Existing record is updated with new values
2. Update is tracked in updated_records
3. Original values are preserved in update log

Batch Processing

The import system uses transactions for efficiency:

ActiveRecord::Base.transaction do
  records.each do |record|
    # Process record
  end
end

Benefits:

• All-or-nothing imports
• Better performance
• Automatic rollback on errors

Best Practices

1. Always Test First

# Test mode first
rake granblue:import_data TEST=true VERBOSE=true

# Review output, then import
rake granblue:import_data VERBOSE=true

2. Use Dated Filenames

db/seed/updates/
├── characters_20240101.csv
├── weapons_20240115.csv
└── summons_20240201.csv

3. Validate Data Format

Before importing:

• Check CSV encoding (UTF-8)
• Verify headers match expected fields
• Validate element and rarity values
• Ensure granblue_id uniqueness

4. Backup Before Large Imports

# Backup database
pg_dump hensei_development > backup_$(date +%Y%m%d).sql

# Run import
rake granblue:import_data

# If issues, restore
psql hensei_development < backup_20240315.sql

5. Monitor Import Results

# In Rails console
import_log = ImportLog.last
import_log.new_records_count
import_log.updated_records_count
import_log.errors

Custom Importer Implementation

To create a custom importer:

module Granblue
  module Importers
    class CustomImporter < BaseImporter
      private

      # Required: specify model class
      def model_class
        CustomModel
      end

      # Required: build attributes from CSV row
      def build_attributes(row)
        {
          granblue_id: parse_value(row['granblue_id']),
          name_en: parse_value(row['name_en']),
          name_jp: parse_value(row['name_jp']),
          custom_field: parse_custom_value(row['custom'])
        }
      end

      # Optional: custom parsing logic
      def parse_custom_value(value)
        # Custom parsing
        value.to_s.upcase
      end

      # Optional: additional validation
      def validate_record(attributes)
        errors = []
        if attributes[:custom_field].blank?
          errors << "Custom field is required"
        end
        errors
      end
    end
  end
end

Data Pipeline

Complete data import pipeline:

1. Export from source → CSV files
2. Place in updates folder → db/seed/updates/
3. Test import → rake granblue:import_data TEST=true
4. Review results → Check logs for errors
5. Execute import → rake granblue:import_data
6. Download images → rake granblue:download_all_images[type]
7. Fetch wiki data → rake granblue:fetch_wiki_data

Troubleshooting

Import Not Finding Files

1. Check files are in db/seed/updates/
2. Verify file extensions are .csv
3. Ensure file permissions allow reading

Validation Errors

1. Check CSV headers match expected fields
2. Verify data types (strings, numbers, booleans)
3. Validate element and rarity values
4. Ensure granblue_id is unique

Encoding Issues

1. Save CSV as UTF-8
2. Remove BOM if present
3. Use Unix line endings (LF)
4. Check for special characters

Performance Issues

For large imports:

1. Use batch processing
2. Disable callbacks if safe
3. Consider direct SQL for bulk operations
4. Import in smaller chunks

Debugging

Enable verbose mode:

rake granblue:import_data VERBOSE=true

Check Rails console:

# Recent imports
ImportLog.recent

# Failed imports
ImportLog.where(status: 'failed')

# Check specific record
Character.find_by(granblue_id: '3040001000')