takescake

AI Vision Card Scanner - Multimodal MTG Card Recognition

2025-10-06 | 7 min read

AI Vision Card Scanner: Next-Generation MTG Card Recognition

We've completely reimagined our card scanner with a vision-first AI approach using Ollama's multimodal API that actually "sees" and understands your Magic: The Gathering cards, not just reading text.

The Problem with OCR-Only Scanning

Traditional card scanners rely solely on Optical Character Recognition (OCR) to read card names:

  • Unreliable on glossy cards - Reflections and glare confuse text recognition
  • Small text is hard to read - Card names can be difficult for OCR to capture accurately
  • Angle sensitive - Card must be perfectly aligned for OCR to work
  • No visual context - Ignores the most distinctive feature: the artwork

Our Solution: Vision-First Recognition

Instead of forcing OCR to read tiny text on glossy, angled cards, we use multimodal AI vision to understand the card as a whole image:

How It Works

  1. Visual Description Generation

    • Ollama's multimodal API passes your card image to the AI model (Gemma 3 4B or LLaMA 3.2 Vision)
    • AI "looks" at the image and generates rich description: artwork scene, colors, frame style, symbols
    • Example: "A red legendary creature card featuring a fierce dragon with scales and fire, black card frame, mountain landscape background..."

  2. Semantic Embedding Creation

    • Description is converted to embedding vector using embeddinggemma
    • Creates mathematical representation of visual features
    • Enables similarity matching against database

  3. Database Matching

    • Your card's vision embedding is compared to pre-built database
    • Uses cosine similarity to find closest visual matches
    • Returns ranked results with confidence scores

  4. OCR Fallback

    • If vision matching fails or database unavailable
    • Falls back to traditional OCR + fuzzy text matching
    • Ensures scanning always works

Setup Requirements

Ollama Models

You need two models installed:

# Option 1: Use Gemma 3 4B (you likely already have this)
ollama pull gemma3:4b

# Option 2: Or use dedicated vision model LLaMA 3.2 Vision
ollama pull llama3.2-vision

# Embedding model for semantic matching (required)
ollama pull embeddinggemma

Note: Ollama's API allows passing images to any model via base64. You can use Gemma 3 4B with images!

Build Vision Database

One-time setup: Generate vision embeddings for your card collection:

npm run cards:vision-embeddings

This process:

  • Scans all local card images in public/cards/
  • Generates AI vision descriptions for each card
  • Creates embeddings from descriptions
  • Saves to content/cards/card-vision-embeddings.json

Time estimate: ~10-20 seconds per card (50,000 cards = ~140 hours)

Tip: Start with a subset by temporarily moving most images out of public/cards/, build embeddings, then do full build later.

Configuration

Environment Variables

# Vision model (default: gemma3:4b, or use llama3.2-vision)
OLLAMA_VISION_MODEL=gemma3:4b

# Embeddings model (default: embeddinggemma)  
OLLAMA_EMBEDDINGS_MODEL=embeddinggemma

# Ollama host (default: http://127.0.0.1:11434)
OLLAMA_HOST=http://localhost:11434

Tip: You can use gemma3:4b (which you already have) or switch to llama3.2-vision for dedicated vision capabilities.### Incremental Builds

The vision embeddings builder is smart:

  • Skips cards that already have embeddings
  • Processes new cards only
  • Safe to re-run after adding more images

How to Use

1. Upload Image Method

  1. Go to /scan page
  2. Click "Upload Image"
  3. Select card photo from your device
  4. AI vision automatically analyzes the image
  5. Confirm or correct the best guess, then Log the right print

2. Camera Method

  1. Click "Open Full-Screen Camera"
  2. Position card in frame guide
  3. Capture photo
  4. Auto-processed with vision AI
  5. Compact match rows appear — Log the best hit

3. Debug Mode

Click "Debug" (when available) to see:

  • AI vision description of your card
  • Similarity scores for each match
  • Which model was used (vision, vision-embedding, or OCR fallback)
  • Detailed matching process

What changed for users (results UX)

Scan results now optimize for one job: identify the right card and log it.

  • Feedback first: “Was this scan right?” stays above the list with clear Correct / Wrong / Scan again paths. Wrong opens a short correction form for training.
  • Compact rows: Matches render as dense list rows (thumb + name + set + score + price), not equal-weight action cards — especially easier on phones.
  • One primary action: each row shows a single Log button. View card, Buy on TCGplayer, Add to batch, and score/price breakdown live under More.
  • Details on demand: foil/non-foil pricing and text/art score breakdown expand only when you ask.
  • Matching unchanged: vision → vision-embedding → OCR fallback still runs the same way; only the results chrome got quieter.

Accuracy Path (2026 refresh)

The live scanner now treats vision as a full stack, not a single regex hit:

  1. GPU vision JSON identity – fine-tuned Gemma via the local gateway is prompted for one line of JSON: {"name":"…","set":"XLN","collector":"383"}. Legacy free-text (Card Name: **…**, This is …) still parses when the current adapter has not been retrained yet.
  2. Name resolve – exact match, Secret Lair-style aliases (e.g. Brachiosaurus → Colossal Dreadmaw), then high-confidence fuzzy match.
  3. Print preference – when set / collector are present, the matching print is ranked first among oracle siblings.
  4. Vision embedding fallback – if the name cannot be resolved, the model text (or JSON description) is embedded with embeddinggemma and compared to card-vision-embeddings.json (the { metadata, cards: [...] } format the builders write). The client accepts method: "vision-embedding" and skips OCR when that path succeeds.
  5. All prints – a successful vision/embedding hit returns matching prints for the same oracle_id (Secret Lair + retail side-by-side).
  6. Faster uploads – capture/upload images are JPEG-compressed (max width ~896) before POST so mobile scans pay less payload tax.
  7. OCR last – Tesseract + fuzzy name matching only runs when vision and embedding both miss.

Retrain (JSON identity refresh):

# Build JSONL from embeddings (+ scanner-feedback hard negatives, upweighted ×2)
SAMPLE_SIZE=400 bun run cards:prepare-json-finetune

# Resume LoRA for 1 epoch on the new JSON targets (requires CUDA + Unsloth)
python train_mtg_vision.py --epochs 1
# or: bun run cards:train-json-refresh

Images are resolved from public/cards or cards-store. Feedback photos live under training-data/scanner-feedback/images/. Training resumes adapters in mtg-card-model-100-3epochs.

Ops knobs: OLLAMA_HOST (default http://127.0.0.1:11434), OLLAMA_VISION_MODEL (default gemma-3-4b-mtg-finetuned), OLLAMA_EMBEDDINGS_MODEL (default embeddinggemma).

Advantages Over OCR

Vision AI Approach

✅ Sees the whole card - Art, frame, colors, layout
✅ Works at angles - No need for perfect alignment
✅ Handles reflections - Understands card despite glare
✅ Multi-print aware - Matches specific artwork version
✅ High confidence - 85%+ similarity = very likely correct

Traditional OCR

❌ Reads only text
❌ Requires perfect alignment
❌ Fails on glossy/reflective surfaces
❌ Can't distinguish between printings
❌ Low confidence on similar names

Technical Details

Vision Description Example

When you scan Emerald Dragon, the vision model sees:

"This is a Magic: The Gathering creature card featuring a green dragon with emerald scales and glowing eyes. The artwork shows the dragon in flight against a forest background with magical energy radiating from its body. The card has a green creature card frame, indicating a green mana cost visible in the top right corner. The card name 'Emerald Dragon' appears at the top in the characteristic MTG font style."

This description is then converted to a 768-dimensional embedding vector that encodes these visual features mathematically.

Matching Algorithm

// Simplified matching logic
1. Upload card image
2. Generate vision description using llama3.2-vision
3. Create embedding from description using embeddinggemma
4. Calculate cosine similarity with all database embeddings
5. Return top matches sorted by similarity score
6. Filter by confidence threshold (>70% = show result)

Performance

  • Vision matching: ~2-3 seconds per card
  • Database size: ~500KB per 1,000 cards
  • Accuracy: 90-95% correct in top 3 results
  • Memory usage: Loads full embedding database (~50MB for 50K cards)

Building the Database

Full Collection

# Ensure all card images downloaded
npm run cards:download-images

# Build vision embeddings (long process)
npm run cards:vision-embeddings

Subset for Testing

# Move most images temporarily
mkdir temp_cards
mv public/cards/*.jpg temp_cards/

# Keep ~100 cards in public/cards for testing
# (e.g., your favorite commander's deck)

# Build vision embeddings (fast)
npm run cards:vision-embeddings

# Restore full collection when ready
mv temp_cards/*.jpg public/cards/
rm -r temp_cards

# Build remaining embeddings
npm run cards:vision-embeddings

Troubleshooting

"Vision database not found"

Problem: No card-vision-embeddings.json file exists.

Solution: Run npm run cards:vision-embeddings to build it.

"Failed to load vision model"

Problem: llama3.2-vision not installed in Ollama.

Solution: ollama pull llama3.2-vision

Scanner falls back to OCR

Problem: Vision matching failed or not configured.

Solution:

  1. Check Ollama is running: ollama list
  2. Verify vision model installed
  3. Check card-vision-embeddings.json exists
  4. Review server logs for errors

Slow scanning

Problem: Vision processing takes time.

Solution:

  • This is normal - AI vision analysis requires ~2-3 seconds
  • OCR fallback is faster but less accurate
  • Consider pre-building embeddings for your collection

Future Enhancements

  • Real-time camera scanning - Continuous vision analysis while positioning card
  • Batch mode - Upload multiple card images at once
  • Condition assessment - AI describes card condition from image
  • Foil detection - Identify foil vs non-foil from visual patterns
  • Set symbol recognition - Extract set information visually
  • Rotation correction - Auto-rotate angled cards before analysis
  • Multi-language support - Vision works on any language card
  • Counterfeit detection - Compare visual features against known authentic cards

Why This Matters

This vision-first approach represents a fundamental shift in how we think about card recognition:

  1. Human-like understanding - AI "sees" cards like a player would
  2. Robust to real-world conditions - Works with phone photos, not just scanner-quality images
  3. Future-proof - As vision models improve, accuracy increases automatically
  4. Multimodal AI - Combines visual + textual understanding
  5. Scalable - Build once, match forever (until new sets release)

Comparison: Before vs After

Before (OCR-Only)

User scans: Emerald Dragon (glossy, slight angle)
OCR reads: "Emerald Dragon ann"
Fuzzy match: "Emerald Dragon" (65% confidence)
But also: "Dragon Turtle" (60% confidence)
Result: Uncertain match, user must verify

After (Vision-First)

User scans: Emerald Dragon (glossy, slight angle)
AI sees: "Green dragon with emerald scales, forest background..."
Vision match: "Emerald Dragon" (92% confidence)
Result: Confident match, correct artwork variant

Performance Monitoring

Check scanner performance in debug mode:

  • Method: vision-ai or ocr-fallback
  • Top match similarity: >85% = high confidence
  • Processing time: Vision ~2-3s, OCR ~5-10s
  • Description quality: Detailed = good model output

Contributing

Help improve vision-based scanning:

  1. Report mismatches - Which cards get identified incorrectly?
  2. Share edge cases - Unusual lighting, damaged cards, proxies?
  3. Suggest features - What else should AI "see" in cards?
  4. Test different models - Try other vision models (LLaVA, Gemini, Claude)

Conclusion

Vision-first scanning is a game-changer for MTG card recognition. By letting AI truly "see" and understand cards visually, we've eliminated the brittleness of OCR-only systems.

Setup once, scan forever.

Powered by Ollama llama3.2-vision + embeddinggemma