How do I investigate an incident across application and cloud logs?

Q: How do I investigate an incident across application and cloud logs?

Bring application and cloud logs into SigNoz Cloud, compare the records for a failed request, and check gaps in IDs and timestamps.

Last Updated: October 23, 20263 min read

You can investigate an incident in SigNoz Cloud by collecting application and cloud logs, then comparing them in Logs Explorer. Start with the failure in your application and look for a provider request ID in its log. If the cloud record contains the same ID, it can help you find the corresponding call. A trace ID can help connect instrumented application logs, but it is separate from the provider's request ID.

Follow one failed upload

Suppose a checkout service accepts a payment and then saves the receipt in Amazon S3. If the upload fails, the service may return an HTTP 500 even though the payment succeeded. To illustrate the investigation, we created sample application logs and a CloudTrail-format record for this scenario.

Our sample records use a checkout request ID to identify the customer request and an AWS request ID to identify the S3 call. We searched for both IDs in Logs Explorer to bring the related records together. The results show the payment being accepted, an AccessDenied response in the CloudTrail-format record, the upload error and the HTTP 500 response.

SigNoz Cloud shows payment accepted, a CloudTrail-format AccessDenied record, the application error and HTTP 500. — *Four collected example records, from accepted payment to HTTP 500. Times shown are UTC+05:30.*

The example points to the failed upload, but AccessDenied alone does not identify the permission or policy responsible. In your own investigation, compare the available account, region, bucket and object details before treating records as related. If an ID is missing, those details and the time can help you find candidates; a close timestamp alone does not establish a connection.

If you cannot find a record, check whether your filters include both sources. A service or severity filter may exclude a cloud record, and account or environment fields may be missing or use different names or values. Check the timestamps too. CloudTrail's eventTime records when AWS completed the request; the log can arrive later. Widen the time range if stored timestamps or clocks differ.

A record can also be missing if it was not collected or has expired. See the application log collection methods to send application logs to SigNoz Cloud. For an S3 upload like this one, CloudTrail must capture the relevant S3 data events, which AWS does not enable by default. You can then send those events to CloudWatch Logs and follow the CloudWatch collection guide.

Other cloud providers

For Azure Blob Storage, the SigNoz integration guide explains how to collect diagnostic logs. A client request ID can help compare records when it is present in both; see Azure's log fields.

For Google Cloud, follow the Cloud Logging collection guide to send selected logs to SigNoz Cloud. If both records carry the same trace ID, it can help connect them. The Google Cloud guide to correlating logs explains when trace information is available.