Troubleshooting the Ghost: Decoding the “400 Bad Request” from the Prometheus API
If you request 6 months of data at 1-second resolution, you are asking Prometheus to scan billions of samples. The server may reject the query immediately with a 400 because it recognizes the resource cost is impossible.
curl -G http://localhost:9090/api/v1/query \ --data-urlencode 'query=upjob="node"' (This is the safest way to use curl with Prometheus.) Sometimes, it genuinely isn't your fault. A bug in a third-party exporter or a service discovery crash can inject NaN or Inf values into the label set. If your query tries to filter on a label that contains a newline character ( \n ) or a control character, the JSON marshaller fails.
debugging-prometheus-400-bad-request