Browsing the Web from a VPS Agent: Roadblocks & the Unblocking Stack

The setup

I run an AI agent (Hermes) on a VPS. One of its jobs is research — search the web, read pages, summarize. The pitch sounds clean: a reliable cloud research assistant that runs 24/7.

The reality: the web fights back hard. Here's what actually broke and the stack that actually works, built through real trial and error over a few weeks.

Roadblock 1: Web search gets blocked

The first thing that died was search itself. Standard search APIs and scraping from a datacenter IP get rate-limited or flat-out blocked. You type a query, you get nothing useful back.

Fix: SearXNG. Self-hosted meta-search engine. It aggregates results from multiple search engines — no API keys, no rate limits from a shared IP, and it runs on your own infrastructure. This replaced the blocked web search entirely and handles the discovery layer.

Roadblock 2: Page extraction returns garbage

Next problem: even when I had a URL, extracting the readable content was painful. Raw HTML scraping pulls in navigation, ads, scripts — noise everywhere. The agent wastes tokens parsing junk.

Fix: Jina AI. Instead of raw scraping, Jina extracts the page as clean markdown. Handles the "give me this page as readable text" job. Pair it with SearXNG for search, and most of the web becomes readable.

Roadblock 3: Sites that refuse non-browser clients

Some sites don't just check your IP — they check if you look like a real browser. A simple HTTP fetch gets rejected outright. You need an actual browser engine that renders JavaScript and behaves like a human visitor.

Fix: CamoFox. A stealth browser that handles the "site refuses non-browser clients" job. Bonus: VNC attached, so the agent can actually see rendered pages and inspect them visually. This covers the JavaScript-rendered, bot-detecting sites that SearXNG + Jina can't crack.

Roadblock 4: The datacenter IP problem (the real wall)

Here's where it gets honest. SearXNG, Jina, and CamoFox handle most cases. But sites like Reddit and similar platforms hard-block datacenter IP ranges entirely. It doesn't matter how stealthy your browser is — the IP itself is the problem. I tried to summarize a Reddit post and the whole stack struggled.

No amount of tooling fixes this at the agent level. The IP is the signal.

Fix: Residential proxy as last resort. Route requests through a home IP. This is the only thing that reliably gets through hard datacenter blocks. I set this up strictly for reading websites when everything else fails — not the first tool to reach for, but the final fallback.

The stack, in fallback order

After all that, here's what a reliable VPS research agent actually looks like — layered, not silver-bullet:

1. SearXNG — search/discovery (handles the blocked-search problem)
2. Jina AI — clean page extraction (handles the noisy-HTML problem)
3. CamoFox — stealth browser with VNC (handles JS-rendered and bot-detecting sites)
4. Residential proxy — last resort (handles hard datacenter IP blocks)

Each layer covers what the previous one can't. You don't reach for the browser when a markdown extractor would do. You don't burn proxy bandwidth when SearXNG already found the answer.

The honest takeaway

The point isn't to build a system that works everywhere. It's to have one that handles most cases reliably — and fails gracefully on the rest.

SearXNG + Jina handles the majority. CamoFox handles the stubborn middle. Residential proxy handles the rest. And knowing when a site just isn't worth the fight is part of the stack too.

The "reliable cloud research agent" exists. It's just not one piece — it's four, deployed in the right order.

There's more

These four roadblocks aren't the whole picture. Some sites don't block you — they want you to log in. Auth-gated content is a different problem entirely, and it needs a different approach. More on that later.