Self Hosting

Searchbase can run with different search backends depending on how much infrastructure and control you want.

Deployment Options

Searchbase includes Docker Compose and Kubernetes examples for evaluating the services. Adapt the deployment to your environment before production use.

Docker Compose Examples

The repository includes user-facing Compose examples under examples/compose/ for quickly trying each backend. Kubernetes examples live under examples/k8s/. To build from a checkout or run benchmarks, see Development.

Native DuckDuckGo backend:

1

docker compose -f examples/compose/docker-compose.native.yml up

DDGS backend:

1

docker compose -f examples/compose/docker-compose.ddgs.yml up

SearXNG backend:

1

docker compose -f examples/compose/docker-compose.searxng.yml up

Experimental Lightpanda fetch worker with the native DuckDuckGo search backend:

1

docker compose -f examples/compose/docker-compose.lightpanda.yml up --build

This example builds the Go /extract wrapper on Lightpanda’s official 1.0.0 release image, pinned by digest for amd64 and arm64. Only the gateway publishes a host port; the worker stays on the Compose network. For production, pin the published lightpanda-worker image by version or digest. Lightpanda is AGPL-3.0 licensed; review its terms before redistribution or deployment. The Go worker uses Lightpanda’s native Markdown dump with its clutter stripping heuristic, which may differ in cleaning and content from Crawl4AI. The worker passes --strip-mode clutter; 1.0.0 includes upstream fix #3663 so anchors without href no longer cause content to be discarded as link-heavy navigation or produce empty Markdown links. Navigation and footers can still be removed intentionally. Lightpanda executes JavaScript for every fetch; both js_render: false and js_render: true have the same behavior. This is an alternative worker, not a new search provider and not an automatic replacement for existing deployments. See Crawl4AI and Lightpanda for saved load-test results and their rate-ceiling limits. The previously reported Markdown omission has been resolved; further fidelity and token-efficiency checks are internal regression work.

The worker accepts POST /extract with {"url":"https://example.org","js_render":false} and returns {"markdown":"...","success":true,"error":""}. It also has GET /healthz. Extraction failures return HTTP 200 with success:false and a fixed error category, preserving the worker response shape without exposing the fetched URL. Categories include not_found, forbidden, robots_denied, rate_limited, timeout, unreachable, upstream_error, and extraction_failed. The wrapper parses the browser’s --json output internally; it never forwards raw browser JSON, URLs, headers, or errors. See the API reference for gateway status mappings. Invalid targets return HTTP 400 before the browser starts: the worker accepts absolute HTTP(S) URLs without credentials, whitespace, control characters, backslashes, or malformed ports. It normalizes the host and removes URL fragments before passing the URL to Lightpanda. This check is separate from the browser SSRF guard: the worker passes --block-private-networks by default, which blocks private/internal destination IPs after DNS resolution. Private websites, localhost, and internal services will therefore fail extraction. For trusted local crawling only, set LIGHTPANDA_WORKER_ALLOW_PRIVATE_NETWORKS=true to disable this guard. Keep network restrictions as defense in depth, especially for untrusted callers. The isolated load-test stack explicitly disables the guard to reach its local fixtures.

Every fetch also passes --obey-robots, so pages denied by the site’s robots.txt fail extraction. Set LIGHTPANDA_WORKER_USER_AGENT on the worker to identify your deployment. Its default is Searchbase (+https://github.com/coolapso/searchbase). For example:

1
2

environment:
  LIGHTPANDA_WORKER_USER_AGENT: "SearchbaseCloud (+https://searchbase.md)"

Personal deployments can use their own bot name and public information URL. This identifies the operator to sites; it does not make the browser less detectable. Empty settings use the default. Whitespace-only identities, control characters, values over 1024 bytes, and values containing Mozilla are rejected at startup. These settings belong to the deployment, not individual fetch requests. Both Compose stacks accept the variables from the host environment; in Kubernetes, add them to the worker Deployment’s env list. Set LIGHTPANDA_WORKER_CONCURRENCY (default 4), LIGHTPANDA_WORKER_TIMEOUT_SECONDS (default 30), and LIGHTPANDA_WORKER_WAIT_MS (default 5000) to tune it. This is the maximum page-loading wait in milliseconds, not a delay after navigation. Lightpanda normally returns earlier when the page load completes; if this budget expires, it dumps the page as it stands, which can be incomplete or empty. A value such as 500 can therefore make a first fetch fail while a faster repeat succeeds. The worker rejects empty Markdown. Increase this budget for slower pages; the worker’s overall request timeout still bounds extraction, and the gateway REST fetch endpoint has its own 30-second timeout. Existing deployments that explicitly set LIGHTPANDA_WORKER_WAIT_MS=500 must update that setting as well as rebuilding or updating the worker image. Set LIGHTPANDA_WORKER_LOG_LEVEL (error by default) for generic, URL-free request metadata logs. Its container disables Lightpanda’s upstream telemetry and core dumps by default; retain those settings if rebuilding. A fetch starts a Lightpanda process, so measure its process-start overhead and Markdown quality against your real pages.

Lightpanda worker telemetry

The Go wrapper can export OpenTelemetry traces and metrics to an OTLP/HTTP collector. It is off by default and does not expose a Prometheus /metrics endpoint. Set both variables on the worker (the endpoint must be reachable from its container):

1
2

LIGHTPANDA_WORKER_OTEL_ENABLED=true
LIGHTPANDA_WORKER_OTEL_ENDPOINT=http://otel-collector:4318

The endpoint is an HTTP(S) base URL; the worker appends /v1/traces and /v1/metrics. The Compose example passes these variables through from the host but does not create a collector. Configure gateway tracing separately with SEARCHBASE_TRACING_ENABLED=true and its collector endpoint to see a single trace across gateway and worker. The worker accepts W3C traceparent and creates a server span for /extract plus a child span around the browser process. It exports completed request counts by fixed outcome (success, invalid_request, busy_or_timeout, extraction_error), active requests, request-duration histograms, and browser-duration histograms. The metric export interval is 15 seconds, with a final flush on graceful shutdown.

The spans and metric attributes omit requested URLs, content, and raw browser errors. This instruments the Go wrapper, not the Lightpanda executable. It does not measure browser child-process memory; use cgroup/container monitoring for total worker memory, including children. The image’s LIGHTPANDA_DISABLE_TELEMETRY=true controls Lightpanda’s separate upstream telemetry and remains set even when wrapper OpenTelemetry is enabled.

To build or test this worker from source, see Contributing.

Kubernetes

The manifests under examples/k8s/ are demonstrations, not production deployment templates. They create one gateway and your choice of Crawl4AI or Lightpanda worker, each with an internal ClusterIP Service. From the repository root, apply the Crawl4AI example to the default namespace:

1
2

kubectl apply -f examples/k8s/crawl-worker.yaml \
  -f examples/k8s/search-gateway.yaml

For the experimental Lightpanda alternative, apply this pair in the default namespace without applying the Crawl4AI worker manifest:

1
2

kubectl apply -f examples/k8s/lightpanda-worker.yaml \
  -f examples/k8s/search-gateway.yaml

With either worker, reach the internal gateway Service locally:

1

kubectl port-forward svc/search-gateway 8080:8080

Both worker manifests provide the crawl-worker Service used by the gateway’s crawl-worker.default.svc.cluster.local address. Apply only one worker manifest: applying the other later changes the Service selector but leaves the previous worker Deployment running until you remove it. The Lightpanda example uses the published lightpanda-worker image, disables upstream telemetry, and probes its /healthz endpoint. It accepts js_render but always executes JavaScript, as described above. The resource values in both examples are illustrative, not measured sizing recommendations.

The gateway uses the default native search provider. examples/k8s/ddgs.yaml is fully commented and creates no resources. Port forwarding provides local test access because the gateway Service is not externally exposed. For production self-hosting, configure your namespace and service addresses, pinned image versions, external access, secrets, resource limits, and network policy for your environment.

Configuration

The search-gateway can be configured using the following environment variables:

| SEARCHBASE_MCP_HEARTBEAT_ENABLED | false | Send periodic MCP heartbeat pings on the SSE and Streamable HTTP transports. Off unless explicitly set to true, so existing deployments keep the current behavior. Enable it when MCP clients drop idle connections, for example the reference MCP SSE client, which closes the stream after its default 300s read timeout. | | SEARCHBASE_MCP_HEARTBEAT_INTERVAL | 60 | Heartbeat interval in seconds. Only used when SEARCHBASE_MCP_HEARTBEAT_ENABLED=true. Minimum accepted value is 15; a lower value fails startup instead of hammering the gateway and its host with excessively frequent pings. | | SEARCHBASE_TRACING_ENABLED | false | Enable OpenTelemetry distributed tracing. (Experimental) | | SEARCHBASE_TRACING_ENDPOINT | (empty) | Endpoint for the tracing backend. (Experimental) |

Search Backend Options

Searchbase initializes one search provider at startup through SEARCHBASE_SEARCH_PROVIDER.

Native DuckDuckGo Backend

searchbase_ddg is the default and easiest backend.

1

SEARCHBASE_SEARCH_PROVIDER=searchbase_ddg

This backend is implemented directly in the Go gateway. It posts to DuckDuckGo’s HTML endpoint, parses organic results, and returns compact search metadata.

Use this when you want the simplest possible self-hosted setup with no additional backend service.

DDGS Backend

ddgs is a lightweight metasearch backend based on deedy5/ddgs.

1
2

SEARCHBASE_SEARCH_PROVIDER=ddgs
SEARCHBASE_DDGS_PROVIDER_ADDRESS=http://ddgs:8001

DDGS supports multiple search engines through a small Python service. It is a good middle ground when you want metasearch behavior without operating a full SearXNG instance.

The engine request field applies when using this backend. Examples include auto, google, duckduckgo, bing, brave, and other DDGS-supported backends.

SearXNG Backend

searxng connects Searchbase to an external SearXNG instance.

1
2

SEARCHBASE_SEARCH_PROVIDER=searxng
SEARCHBASE_SEARXNG_PROVIDER_ADDRESS=http://searxng:8080

SearXNG is the most complete option if you want a full private metasearch engine with its own engine configuration, rate limits, proxy setup, and privacy controls.

Refer to the official SearXNG documentation for details on how to host, configure, and operate a SearXNG instance.

The engine request field applies when using this backend and maps to SearXNG’s engines query parameter.

Brave Search API Backend

brave connects Searchbase directly to the official Brave Search API.

1
2

SEARCHBASE_SEARCH_PROVIDER=brave
SEARCHBASE_BRAVE_API_TOKEN=your_brave_search_api_token

This backend returns Brave web results as compact Searchbase results using each result’s title, url, and description. Searchbase requests only web results from Brave and disables text decorations so snippets are easier for agents to consume.

The limit request field maps to Brave’s count parameter and is capped at Brave’s maximum of 20. Omit limit or set it to 0 to use Brave’s default result count.

The timelimit request field maps to Brave freshness filters: d to pd, w to pw, m to pm, and y to py.

The region request field can provide Brave country and language hints. Searchbase maps supported country-language style values such as us-en to Brave country=US and search_lang=en; unsupported country or language parts are omitted. Brave also supports ALL as a country value.

Mojeek Search API Backend

mojeek connects Searchbase directly to the official Mojeek Search API.

1
2

SEARCHBASE_SEARCH_PROVIDER=mojeek
SEARCHBASE_MOJEEK_API_KEY=your_mojeek_search_api_key

This backend returns Mojeek web results as compact Searchbase results using each result’s title, url, and desc.

The limit request field maps to Mojeek’s t parameter. Omit limit or set it to 0 to use Mojeek’s default result count.

Choosing a Backend

Start with searchbase_ddg if you only need a quick, low-maintenance deployment.

Move to ddgs if you want lightweight metasearch without managing a full private search engine.

Use searxng if you already run SearXNG or want full control over metasearch engines, privacy settings, and routing.

Use brave if you want a cloud-safe official search API backed by Brave’s independent index.

Use mojeek if you want a cloud-safe official search API backed by Mojeek’s independent index.

Production Notes

Before exposing Searchbase publicly, plan for:

• TLS and ingress configuration.
• API authentication once available.
• network isolation between public gateway and internal workers.
• secrets management for any future paid provider credentials.
• resource limits and worker scaling.
• structured logs and optional tracing.
• provider rate limits and acceptable use.

Gateway logging

Set SEARCHBASE_LOG_LEVEL on the gateway to error (the default), warn, info, or debug. The logger applies this threshold at startup. Ordinary requests, including health checks, are logged at info; HTTP 5xx requests are logged at error. With warn or error, ordinary request logs are suppressed and errors remain visible. Invalid levels fail startup.

Output is JSON. SEARCHBASE_LOG_FORMAT is not currently supported. Versions through 1.3.3 hard-coded the logger to info despite parsing the configured level; rebuild or update the gateway to apply this fix.