Skip to main content

​
AI Analytics Log Ingestion

Edgee’s AI Analytics helps you understand AI traffic on your website through our log ingestion service. Send your web server access logs to our endpoint and gain insights into how AI crawlers interact with your content.

​
Prerequisites

Before setting up log ingestion, you’ll need:
  • An active Edgee account
  • A project configured for AI Analytics
  • Access to your web server’s access logs or logging system

​
Quick Start

​
1. Get Your Project Configuration

First, retrieve your project ID from the Edgee console:
  1. Navigate to your project dashboard
  2. Go to the AI Analytics section
  3. Copy your unique Project ID

​
2. Configure Your Logging Endpoint

Send logs to the following endpoint: 
POST https://logs.edgee.app/{PROJECT_ID}
Replace {PROJECT_ID} with your actual project ID.

​
3. Set Content Type

All log submissions must use: 
Content-Type: application/x-ndjson

​
Log Format Specification

​
Required Fields

Each log entry must be a JSON object containing these fields:
FieldTypeDescriptionExample
timestampstringISO 8601 format (YYYY-MM-DDTHH:MM:SSZ)"2024-01-01T00:00:00Z"
ipstringClient IP address"192.168.1.1"
uastringUser-Agent header"Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
methodstringHTTP method"POST", "GET", "PUT"
pathstringRequest path"/api/chat", "/api/completion"
statusstringHTTP status code"200", "404", "500"
content_typestringResponse content type"application/json"
hoststringHost header"api.mycompany.com"

​
Optional Fields

Note: While these fields are optional, they are highly recommended to help classify and measure traffic provenance, enabling better AI analysis and insights.
FieldTypeDescriptionExample
refererstringHTTP Referer header"https://example.com/page"
signaturestringHTTP Signature header (for request authentication)"sig1=:K2qGT...:"
signature_inputstringHTTP Signature-Input header"sig1=..."
signature_agentstringCustom signature agent identifier"https://chatgpt.com"
ja3stringJA3 fingerprint6f7889b9fb1a62a9577e685c1fcfa919
ja4stringJA4 fingerprintt13d1717h2_5b57614c22b0_5ac152f973c6

​
Data Format

  • Format: Newline-delimited JSON (NDJSON)
  • Encoding: UTF-8
  • Timestamp: No milliseconds (remove .000 from ISO string)

​
Implementation Examples

curl.sh
curl -X POST https://logs.edgee.app/YOUR_PROJECT_ID \
  -H "Content-Type: application/x-ndjson" \
  -d '{"timestamp": "2024-01-01T00:00:00Z", "ip": "192.168.1.1", "ua": "GPTBot/1.0 (+https://openai.com/gptbot)", "method": "GET", "path": "/blog/ai-trends-2024", "status": "200", "content_type": "text/html", "host": "www.mycompany.com", "referer": "https://example.com/page"}
{"timestamp": "2024-01-01T00:01:00Z", "ip": "192.168.1.2", "ua": "Claude-Web/1.0", "method": "GET", "path": "/docs/api-reference", "status": "200", "content_type": "text/html", "host": "www.mycompany.com"}'

​
Best Practices

​
Performance Optimization

  1. Asynchronous Logging: Always send logs asynchronously to avoid blocking your main application
  2. Batch Processing: Group multiple log entries in a single request when possible
  3. Error Handling: Use fire-and-forget pattern for logging to prevent application failures

​
Data Quality

  1. Consistent Timestamps: Always use UTC timezone in ISO 8601 format
  2. IP Address: Use the real client IP, not proxy IPs when possible
  3. User Agent: Include complete User-Agent strings for better analytics

​
Security

  1. No Sensitive Data: Never include passwords, API keys, or personal data in logs
  2. Data Retention: Understand that logs are stored for analytics purposes
  3. Compliance: Ensure logging practices comply with your privacy policies

​
Troubleshooting

​
Common Issues

​
400 Bad Request

  • Cause: Invalid project ID format in URL path
  • Solution: Ensure project ID is a valid UUID format
  • Example: Use /12345678-1234-1234-1234-123456789abc not /invalid-id

​
400 Bad Request (JSON)

  • Cause: Invalid JSON format or malformed log entries
  • Solution: Validate each line is valid JSON with all required fields
  • Example: Missing quotes around field names or values

​
404 Not Found

  • Cause: Project ID not found or not configured for AI Analytics
  • Solution: Verify project ID exists and AI Analytics is enabled for your project

​
405 Method Not Allowed

  • Cause: Using HTTP method other than POST
  • Solution: Only use POST requests to the log ingestion endpoint

​
Invalid IP Address

  • Cause: IP field contains invalid IP address format
  • Solution: Ensure IP field contains valid IPv4 or IPv6 address
  • Example: Use "192.168.1.1" not "invalid-ip"

​
Validation Checklist

  • Content-Type header is set to application/x-ndjson
  • Project ID is a valid UUID format
  • Using POST method (not GET, PUT, etc.)
  • Each line contains valid JSON
  • All required fields are present
  • IP address is valid IPv4 or IPv6 format
  • Timestamp format is correct (no milliseconds)

​
Testing Your Integration

Use this test script to verify your setup: 
# Test with a single log entry simulating AI crawler traffic
curl -X POST https://logs.edgee.app/YOUR_PROJECT_ID \
  -H "Content-Type: application/x-ndjson" \
  -d '{"timestamp": "2024-01-01T00:00:00Z", "ip": "127.0.0.1", "ua": "GPTBot/1.0 (+https://openai.com/gptbot)", "method": "GET", "path": "/test", "status": "200", "content_type": "text/html", "host": "test.example.com"}' \
  -w "Status: %{http_code}\n"

​
Support

For technical support or questions about log ingestion:
  1. Check this documentation for common solutions
  2. Review the troubleshooting section above
  3. Contact support through the Edgee console

​
API Reference

For detailed API specifications, see our Log Ingestion API Reference.