Cost Optimization for Vision-Based Browser Agents: Image Compression and Caching

Understanding GPT Vision Token Costs

GPT-4V's image processing costs are directly tied to the number of 512x512 pixel tiles an image is divided into. A 1280x720 screenshot at detail: "high" is split into approximately 6 tiles, each costing 170 tokens, plus a base cost of 85 tokens — roughly 1,105 tokens total per image. At GPT-4o pricing, sending 1,000 screenshots costs around $2.75 in input tokens alone.

For a browser automation agent making 10-50 vision calls per task, these costs add up quickly. The optimization strategies below can reduce per-image costs by 60-80% without meaningfully degrading analysis quality.

Strategy 1: Image Resizing

The simplest optimization. Resize screenshots before sending them to the API.

flowchart LR
    GOAL(["High level goal"])
    PLAN["Planner LLM"]
    SCREEN["Screen capture<br/>every step"]
    VLM["Vision LLM<br/>reads UI state"]
    ACT{"Action type"}
    CLICK["Click coordinate"]
    TYPE["Type text"]
    KEY["Keyboard shortcut"]
    GUARD["Safety filter<br/>allow lists"]
    OS[("OS sandbox<br/>ephemeral VM")]
    DONE(["Goal verified"])
    GOAL --> PLAN --> SCREEN --> VLM --> ACT
    ACT --> CLICK --> GUARD
    ACT --> TYPE --> GUARD
    ACT --> KEY --> GUARD
    GUARD --> OS --> SCREEN
    OS --> DONE
    style PLAN fill:#4f46e5,stroke:#4338ca,color:#fff
    style GUARD fill:#f59e0b,stroke:#d97706,color:#1f2937
    style OS fill:#ede9fe,stroke:#7c3aed,color:#1e1b4b
    style DONE fill:#059669,stroke:#047857,color:#fff

from PIL import Image
import io
import base64

def resize_screenshot(
    screenshot_bytes: bytes,
    max_width: int = 1024,
    max_height: int = 768,
) -> bytes:
    """Resize a screenshot to reduce token cost."""
    img = Image.open(io.BytesIO(screenshot_bytes))
    original_size = img.size

    # Calculate new size preserving aspect ratio
    ratio = min(max_width / img.width, max_height / img.height)
    if ratio >= 1.0:
        return screenshot_bytes  # already small enough

    new_size = (int(img.width * ratio), int(img.height * ratio))
    img = img.resize(new_size, Image.LANCZOS)

    buffer = io.BytesIO()
    img.save(buffer, format="PNG", optimize=True)
    return buffer.getvalue()

def estimate_token_cost(width: int, height: int) -> int:
    """Estimate the token cost for an image at given dimensions."""
    # GPT-4V tiles are 512x512
    tiles_wide = (width + 511) // 512
    tiles_high = (height + 511) // 512
    total_tiles = tiles_wide * tiles_high
    return 85 + (170 * total_tiles)  # base + per-tile

# Cost comparison
print(f"1280x720: {estimate_token_cost(1280, 720)} tokens")  # ~1105
print(f"1024x576: {estimate_token_cost(1024, 576)} tokens")  # ~765
print(f"768x432:  {estimate_token_cost(768, 432)} tokens")   # ~425

Resizing from 1280x720 to 768x432 cuts token cost by approximately 60% and is perfectly adequate for layout analysis and element detection.

Strategy 2: Region Cropping

Often you only need to analyze part of the page. Crop the screenshot to the relevant region before sending it.

def crop_region(
    screenshot_bytes: bytes,
    x: int, y: int, width: int, height: int,
) -> bytes:
    """Crop a specific region from a screenshot."""
    img = Image.open(io.BytesIO(screenshot_bytes))
    cropped = img.crop((x, y, x + width, y + height))

    buffer = io.BytesIO()
    cropped.save(buffer, format="PNG")
    return buffer.getvalue()

class SmartCapture:
    """Capture only the relevant region of a page."""

    REGIONS = {
        "header": (0, 0, 1280, 80),
        "navigation": (0, 0, 250, 720),
        "main_content": (250, 80, 780, 640),
        "form_area": (200, 100, 880, 520),
        "footer": (0, 620, 1280, 100),
    }

    @staticmethod
    async def capture_region(
        page, region_name: str
    ) -> str:
        """Capture a specific named region."""
        screenshot = await page.screenshot(type="png")
        x, y, w, h = SmartCapture.REGIONS[region_name]
        cropped = crop_region(screenshot, x, y, w, h)
        return base64.b64encode(cropped).decode()

Strategy 3: Detail Level Selection

Use detail: "low" when high resolution is not needed. Low detail uses a fixed 85 tokens regardless of image size.

class DetailSelector:
    """Choose the right detail level for each task."""

    # Tasks that work fine with low detail
    LOW_DETAIL_TASKS = {
        "page_type_classification",
        "blocker_detection",
        "general_layout",
        "navigation_check",
    }

    # Tasks that need high detail
    HIGH_DETAIL_TASKS = {
        "text_extraction",
        "form_field_detection",
        "small_element_location",
        "contrast_checking",
    }

    @staticmethod
    def get_detail(task_type: str) -> str:
        if task_type in DetailSelector.LOW_DETAIL_TASKS:
            return "low"   # 85 tokens fixed
        return "high"      # 170 tokens per tile

    @staticmethod
    def build_image_payload(
        b64: str, task_type: str
    ) -> dict:
        detail = DetailSelector.get_detail(task_type)
        return {
            "type": "image_url",
            "image_url": {
                "url": f"data:image/png;base64,{b64}",
                "detail": detail,
            },
        }

Strategy 4: Screenshot Caching

Avoid sending identical or near-identical screenshots to the API. Cache results and reuse them when the page has not changed.

import hashlib
from functools import lru_cache
from dataclasses import dataclass
from time import time

@dataclass
class CachedResult:
    result: dict
    timestamp: float
    image_hash: str

class VisionCache:
    def __init__(self, ttl_seconds: int = 30):
        self.cache: dict[str, CachedResult] = {}
        self.ttl = ttl_seconds
        self.hits = 0
        self.misses = 0

    def _hash_image(self, image_b64: str) -> str:
        """Create a hash of the image for cache lookup."""
        return hashlib.md5(image_b64.encode()).hexdigest()

    def get(self, image_b64: str, task: str) -> dict | None:
        """Check cache for a previous result."""
        key = f"{self._hash_image(image_b64)}:{task}"
        cached = self.cache.get(key)

        if cached and (time() - cached.timestamp) < self.ttl:
            self.hits += 1
            return cached.result

        self.misses += 1
        return None

    def set(self, image_b64: str, task: str, result: dict):
        """Cache a vision API result."""
        key = f"{self._hash_image(image_b64)}:{task}"
        self.cache[key] = CachedResult(
            result=result,
            timestamp=time(),
            image_hash=self._hash_image(image_b64),
        )

    def stats(self) -> dict:
        total = self.hits + self.misses
        hit_rate = self.hits / total if total > 0 else 0
        return {
            "hits": self.hits,
            "misses": self.misses,
            "hit_rate": f"{hit_rate:.1%}",
            "cached_entries": len(self.cache),
        }

Strategy 5: Perceptual Hashing for Similar Screenshots

Sometimes consecutive screenshots are nearly identical (e.g., a cursor moved but nothing else changed). Use perceptual hashing to detect similarity and skip redundant API calls.

def perceptual_hash(image_bytes: bytes, hash_size: int = 8) -> int:
    """Compute a perceptual hash for an image."""
    img = Image.open(io.BytesIO(image_bytes))
    img = img.convert("L")  # grayscale
    img = img.resize((hash_size + 1, hash_size), Image.LANCZOS)

    pixels = list(img.getdata())
    hash_value = 0
    for i in range(hash_size):
        for j in range(hash_size):
            idx = i * (hash_size + 1) + j
            if pixels[idx] < pixels[idx + 1]:
                hash_value |= 1 << (i * hash_size + j)

    return hash_value

def hamming_distance(hash1: int, hash2: int) -> int:
    """Count the number of differing bits between two hashes."""
    return bin(hash1 ^ hash2).count("1")

def images_are_similar(
    img1_bytes: bytes, img2_bytes: bytes, threshold: int = 5
) -> bool:
    """Check if two images are perceptually similar."""
    h1 = perceptual_hash(img1_bytes)
    h2 = perceptual_hash(img2_bytes)
    return hamming_distance(h1, h2) <= threshold

Putting It All Together: A Cost-Aware Vision Client

class CostAwareVisionClient:
    def __init__(self):
        self.client = OpenAI()
        self.cache = VisionCache(ttl_seconds=30)
        self.total_tokens = 0
        self.calls_saved = 0

    async def analyze(
        self,
        screenshot_bytes: bytes,
        task: str,
        task_type: str = "general_layout",
        region: str | None = None,
    ) -> str:
        """Cost-optimized vision analysis."""
        # Step 1: Crop region if specified
        if region and region in SmartCapture.REGIONS:
            x, y, w, h = SmartCapture.REGIONS[region]
            screenshot_bytes = crop_region(
                screenshot_bytes, x, y, w, h
            )

        # Step 2: Resize
        screenshot_bytes = resize_screenshot(
            screenshot_bytes, max_width=768, max_height=432
        )

        b64 = base64.b64encode(screenshot_bytes).decode()

        # Step 3: Check cache
        cached = self.cache.get(b64, task)
        if cached:
            self.calls_saved += 1
            return cached

        # Step 4: Select detail level
        detail = DetailSelector.get_detail(task_type)

        # Step 5: Make API call
        response = self.client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {
                    "role": "user",
                    "content": [
                        {"type": "text", "text": task},
                        {
                            "type": "image_url",
                            "image_url": {
                                "url": f"data:image/png;base64,{b64}",
                                "detail": detail,
                            },
                        },
                    ],
                },
            ],
            max_tokens=500,
        )

        result = response.choices[0].message.content
        self.total_tokens += response.usage.total_tokens

        # Step 6: Cache result
        self.cache.set(b64, task, result)

        return result

    def cost_report(self) -> dict:
        return {
            "total_tokens": self.total_tokens,
            "estimated_cost_usd": self.total_tokens * 0.0000025,
            "api_calls_saved_by_cache": self.calls_saved,
            "cache_stats": self.cache.stats(),
        }

FAQ

What is the optimal image size for GPT-4V browser automation?

For most browser automation tasks, 768x432 pixels provides the best cost-to-accuracy ratio. This reduces token usage by roughly 60% compared to 1280x720 while preserving enough detail for element detection and text reading. Drop to 512x288 for pure layout classification tasks where you only need to identify the page type.

Does JPEG compression help reduce costs compared to PNG?

GPT-4V token cost is based on the decoded image dimensions, not the file size. A JPEG and PNG of the same dimensions cost the same number of tokens. However, JPEG reduces the base64 payload size, which speeds up API requests. Use JPEG at quality 85 for faster uploads without any token cost difference.

How much can caching realistically save in a typical automation session?

In multi-step workflows, 20-40% of screenshots are identical or nearly identical to a previous one — loading states, confirmation pages, or repeated checks of the same page. Perceptual hash deduplication combined with result caching typically saves 25-35% of API calls across a session, translating directly to the same percentage cost reduction.

---

#CostOptimization #GPTVision #ImageCompression #TokenReduction #Caching #BrowserAutomation #APIOptimization #AgenticAI