What is Firecrawl Monitoring?
Firecrawl monitoring gives you real-time visibility into your web scraping and crawling workloads by collecting traces and metrics using OpenTelemetry. Firecrawl does not emit OpenTelemetry itself, so this guide uses firecrawl-otel, a client-side instrumentation package that wraps the Firecrawl Python SDK. It traces every scrape, crawl, search, map, extract, and batch_scrape call without any changes to your call sites, and records metrics for call counts, latency, credits consumed, and results returned.
With full Firecrawl observability in SigNoz, you can trace every scraping call end to end, see which operations are slow or failing, track credit consumption against your Firecrawl plan, set alerts on latency and error rates, and continuously improve the reliability of your scraping pipelines.
Prerequisites
- A SigNoz Cloud account with an active ingestion key or Self Hosted SigNoz instance
- Python 3.9 or later
- A Firecrawl API key. Sign up at firecrawl.dev if you don't have one
- An application that calls the Firecrawl Python SDK (
firecrawl-py2.0.0 or later)
Monitor Firecrawl with OpenTelemetry
The firecrawl-otel package registers an OpenTelemetry instrumentor entry point, so the OpenTelemetry CLI can enable it with no code changes at all. For details on the package, see the firecrawl-otel PyPI page.
Step 1: Install the packages
pip install "firecrawl-otel[instruments]" opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap --action=install
The [instruments] extra pulls in the Firecrawl SDK (firecrawl-py). Drop it and install plain firecrawl-otel if you manage firecrawl-py yourself.
Step 2: Configure the export to SigNoz
Set the standard OpenTelemetry environment variables:
export OTEL_SERVICE_NAME=<your-service-name>
export OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.<region>.signoz.cloud:443
export OTEL_EXPORTER_OTLP_HEADERS="signoz-ingestion-key=<your-ingestion-key>"
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
Verify these values:
<your-service-name>: A name for your application, for examplefirecrawl-prod. This is how the service appears in SigNoz.<region>: Your SigNoz Cloud region.<your-ingestion-key>: Your SigNoz ingestion key.
Step 3: Run your application
Start your app with opentelemetry-instrument, which enables the Firecrawl instrumentor automatically:
opentelemetry-instrument python your_app.py
Your Firecrawl calls are now traced. Each call produces a span carrying the operation type, target URL, credits used, and results returned, alongside metrics for call volume, latency, credits, and results.
View Firecrawl Traces in SigNoz
Once configured, your application automatically emits a trace for every Firecrawl call.
Traces are available in SigNoz under the Traces tab:
When you click on a trace in SigNoz, you'll see a detailed view of the trace, including all associated spans, along with their events and attributes. Each Firecrawl span carries firecrawl.* attributes describing the operation, the target URL, the credits it consumed, and how many results it returned.
View Firecrawl Metrics in SigNoz
Alongside traces, the instrumentation records metrics for call volume, latency, credits consumed, and results returned, each tagged by operation. They are available under the Metrics tab:
Click any metric to inspect its type, unit, and attributes, or open it in the Explorer to chart it:
Firecrawl Observability Dashboard
You can also check out our custom Firecrawl dashboard which provides specialized visualizations for monitoring your Firecrawl usage. The dashboard includes pre-built charts specifically tailored for scraping operations, along with import instructions to get started quickly.
Troubleshooting Firecrawl Observability
No traces in SigNoz
- Confirm your application actually made a Firecrawl call. An idle application emits nothing.
- Verify the region in
OTEL_EXPORTER_OTLP_ENDPOINTmatches your SigNoz account. - OpenTelemetry batches data before sending, so wait 10-30 seconds after a call.
Verify the instrumentor is active
If you instrument in code rather than through opentelemetry-instrument, make sure FirecrawlInstrumentor().instrument() runs before you construct the Firecrawl client. Clients created beforehand keep their original, uninstrumented methods.
The instrumentation targets the Firecrawl Python SDK version 2.0.0 or later. On older SDK layouts it instruments nothing rather than crashing, so an unexpectedly empty Traces tab can mean the installed firecrawl-py is too old.
Auth errors (401 / 403)
Re-check the ingestion key in OTEL_EXPORTER_OTLP_HEADERS="signoz-ingestion-key=<your-ingestion-key>". It must be the exact key from your SigNoz Ingestion Settings, with no extra spaces or quotes.
Endpoint issues
Keep the https:// scheme and :443 port in OTEL_EXPORTER_OTLP_ENDPOINT. The exporter appends the signal path for you, so don't add it yourself.
Setup OpenTelemetry Collector (Optional)
What is the OpenTelemetry Collector?
Think of the OTel Collector as a middleman between your app and SigNoz. Instead of your application sending data directly to SigNoz, it sends everything to the Collector first, which then forwards it along.
Why use it?
- Cleaning up data - Filter out noisy traces you don't care about, or remove sensitive info before it leaves your servers.
- Keeping your app lightweight - Let the Collector handle batching, retries, and compression instead of your application code.
- Adding context automatically - The Collector can tag your data with useful info like which Kubernetes pod or cloud region it came from.
- Future flexibility - Want to send data to multiple backends later? The Collector makes that easy without changing your app.
See Switch from direct export to Collector for step-by-step instructions to convert your setup.
For more details, see Why use the OpenTelemetry Collector? and the Collector configuration guide.
Additional resources:
- Set up alerts for high latency or error rates
- Learn more about querying traces
- Explore log correlation
