hensei-api/docs/downloaders.md

Image Downloaders Documentation

The downloader system provides a flexible framework for downloading game asset images from Granblue Fantasy servers. It supports multiple image sizes, variants, and storage backends (local filesystem and AWS S3).

Architecture

Base Downloader

All downloaders inherit from BaseDownloader which provides:

• Storage management (local, S3, or both)
• Download retry logic
• 404 error handling
• Verbose logging support
• Test mode for dry runs

Available Downloaders

CharacterDownloader

Downloads character portrait images in multiple variants.

Image Sizes:

• main - Full character art (f/)
• grid - Medium portrait (m/)
• square - Small square icon (s/)
• detail - Detailed view (detail/)

Variants:

• _01 - Base art
• _02 - First uncap art
• _03 - FLB (5★) art (if available)
• _04 - ULB (6★) art (if available)

Example:

downloader = Granblue::Downloaders::CharacterDownloader.new(
  "3040001000",
  storage: :both,
  verbose: true
)
downloader.download        # Downloads all sizes and variants
downloader.download("grid") # Downloads only grid size

WeaponDownloader

Downloads weapon images with elemental variations.

Image Sizes:

• main - Full weapon art (ls/)
• grid - Grid view (m/)
• square - Small icon (s/)

Example:

downloader = Granblue::Downloaders::WeaponDownloader.new(
  "1040001000",
  storage: :s3,
  verbose: true
)
downloader.download

SummonDownloader

Downloads summon images including uncap variants.

Image Sizes:

• main - Full summon art (b/)
• grid - Grid view (m/)
• square - Small icon (s/)

Variants:

• Base art and uncap variations based on summon's max_level

Example:

downloader = Granblue::Downloaders::SummonDownloader.new(
  "2040001000",
  storage: :local,
  verbose: true
)
downloader.download

JobDownloader

Downloads job class images with gender variants.

Image Sizes:

• wide - Wide format portrait
• zoom - Close-up portrait (1138x1138)

Variants:

• _a - Male variant
• _b - Female variant

URLs:

• Wide: https://prd-game-a3-granbluefantasy.akamaized.net/assets_en/img/sp/assets/leader/m/{id}_01.jpg
• Zoom: https://media.skycompass.io/assets/customizes/jobs/1138x1138/{id}_{0|1}.png

Example:

downloader = Granblue::Downloaders::JobDownloader.new(
  "100401",
  storage: :both,
  verbose: true
)
downloader.download        # Downloads both wide and zoom with gender variants
downloader.download("zoom") # Downloads only zoom images

Rake Tasks

Download Images for Specific Items

# Character images
rake granblue:export:character_images[3040001000]
rake granblue:export:character_images[3040001000,false,true,s3,grid]

# Weapon images
rake granblue:export:weapon_images[1040001000]
rake granblue:export:weapon_images[1040001000,false,true,both,main]

# Summon images
rake granblue:export:summon_images[2040001000]
rake granblue:export:summon_images[2040001000,false,true,local]

# Job images
rake granblue:export:job_images[100401]
rake granblue:export:job_images[100401,false,true,both,zoom]

Bulk Download All Images

# Download all images for a type with parallel processing
rake granblue:download_all_images[character,4]      # 4 threads
rake granblue:download_all_images[weapon,8,grid]    # 8 threads, grid only
rake granblue:download_all_images[summon,4,square]  # 4 threads, square only
rake granblue:download_all_images[job,2]            # 2 threads, all sizes

# Download all with specific parameters
rake granblue:export:character_images[,false,true,s3]  # All characters to S3
rake granblue:export:weapon_images[,true]               # Test mode
rake granblue:export:summon_images[,false,false,local]  # Local only, quiet
rake granblue:export:job_images[,false,true,both]       # Both storages

Storage Options

Local Storage

Files are saved to Rails.root/download/:

download/
├── character-main/
├── character-grid/
├── character-square/
├── character-detail/
├── weapon-main/
├── weapon-grid/
├── weapon-square/
├── summon-main/
├── summon-grid/
├── summon-square/
├── job-wide/
└── job-zoom/

S3 Storage

Files are uploaded with the following key structure:

{object_type}-{size}/{filename}

Examples:
character-grid/3040001000_01.jpg
weapon-main/1040001000.jpg
summon-square/2040001000.jpg
job-zoom/100401_a.png
job-zoom/100401_b.png

Storage Mode Selection

storage: :local  # Save to filesystem only
storage: :s3     # Upload to S3 only
storage: :both   # Save locally AND upload to S3 (default)

Parameters Reference

Common Parameters

All downloaders accept these parameters:

Parameter	Type	Default	Description
id	String	required	Granblue ID of the item
test_mode	Boolean	false	Simulate downloads without downloading
verbose	Boolean	false	Enable detailed logging
storage	Symbol	:both	Storage mode (:local, :s3, :both)
logger	Logger	Rails.logger	Logger instance to use

Rake Task Parameters

For rake tasks, parameters are passed in order:

rake task_name[id,test_mode,verbose,storage,size]

Examples:

# All parameters
rake granblue:export:character_images[3040001000,false,true,both,grid]

# Skip parameters with commas
rake granblue:export:weapon_images[,true,true]  # All weapons, test mode, verbose

# Partial parameters
rake granblue:export:summon_images[2040001000,false,true,s3]

Test Mode

Test mode simulates downloads without actually downloading files:

# Ruby
downloader = Granblue::Downloaders::CharacterDownloader.new(
  "3040001000",
  test_mode: true,
  verbose: true
)
downloader.download

# Rake task
rake granblue:export:character_images[3040001000,true,true]

Output in test mode:

-> 3040001000
  (Test mode - would download images)

Error Handling

404 Errors

When an image doesn't exist on the server, the downloader logs it and continues:

├ grid: https://example.com/image.jpg...
  404 returned  https://example.com/image.jpg

Network Errors

Network errors are caught and logged, allowing the process to continue:

begin
  download_to_local(url, path)
rescue OpenURI::HTTPError => e
  log_info "404 returned\t#{url}"
rescue StandardError => e
  log_info "Error downloading #{url}: #{e.message}"
end

Best Practices

1. Use Parallel Processing for Bulk Downloads

# Good - uses multiple threads
rake granblue:download_all_images[character,8]

# Less efficient - sequential
rake granblue:export:character_images

2. Check Test Mode First

Before running bulk operations:

# Test first
rake granblue:export:character_images[,true,true]

# Then run for real
rake granblue:export:character_images[,false,true]

3. Use Specific Sizes When Needed

If you only need certain image sizes:

# Download only grid images (faster)
rake granblue:download_all_images[weapon,4,grid]

# Instead of all sizes
rake granblue:download_all_images[weapon,4]

4. Monitor S3 Usage

When using S3 storage:

• Check AWS costs regularly
• Use lifecycle policies for old images
• Consider CDN caching for frequently accessed images

Custom Downloader Implementation

To create a custom downloader:

module Granblue
  module Downloaders
    class CustomDownloader < BaseDownloader
      # Define available sizes
      SIZES = %w[large small].freeze

      # Required: specify object type
      def object_type
        'custom'
      end

      # Required: base URL for assets
      def base_url
        'https://example.com/assets'
      end

      # Required: map size to directory
      def directory_for_size(size)
        case size
        when 'large' then 'lg'
        when 'small' then 'sm'
        end
      end

      # Optional: custom download logic
      def download(selected_size = nil)
        # Custom implementation
        super
      end
    end
  end
end

Troubleshooting

Images Not Downloading

1. Check network connectivity
2. Verify the Granblue ID exists in the database
3. Ensure the asset exists on the game server
4. Check storage permissions (filesystem or S3)

S3 Upload Failures

1. Verify AWS credentials are configured
2. Check S3 bucket permissions
3. Ensure bucket exists and is accessible
4. Check for S3 rate limiting

Slow Downloads

1. Use parallel processing with more threads
2. Download specific sizes instead of all
3. Check network bandwidth
4. Consider using S3 only mode to avoid local I/O

Debugging

Enable verbose mode for detailed output:

rake granblue:export:character_images[3040001000,false,true]

Check Rails logs:

tail -f log/development.log