Start with one known stream.
An empty production dashboard can come from a selector that excludes the streams already arriving. Start with the data source, tenant and a fixed UTC interval. Keep those three values unchanged while comparing selectors.
In Explore, choose a stable label and value that you have observed for the service. Remove the environment matcher and any later line filters for this first read. Keep the interval small enough to inspect within your permitted access.
{service_name="demo-api"}Example selector: replace service_name and demo-api with a label and value from your own source.
Record whether the request returned data, an empty result, a permission error or a query error. A 401 belongs in the permission column; it cannot tell you how many log entries arrived.
Change one matcher at a time.
If the known stream returns data, inspect its labels before adding the dashboard’s environment matcher. The following reads divide the example stream into records with a nonempty environment label and records where that label is empty or absent.
{service_name="demo-api", environment!=""}
{service_name="demo-api", environment=""}Loki follows Prometheus label-selection rules: an empty-value match includes absent labels. The second selector therefore does not distinguish a missing label from an explicitly empty value. It also does not identify a stream as production: that needs a separate service or deployment identity.
| Observation | Next read |
|---|---|
| Known stream has data; environment-filtered view is empty | Inspect actual label names and values, then compare the deployed collector. |
| Known stream is also empty | Check source identity, tenant, interval and collection path. Ingestion is still unresolved. |
| Permission or query error | Resolve that error before interpreting the result as an observation. |
Inspect the configuration that is deployed.
Find the collector instance for the stream and record its serving revision or configuration identity. Follow where the expected environment value is introduced and whether it becomes a Loki stream label. A label present in a repository candidate may be absent from the running collector.
Compare the deployed configuration with the exact repository revision containing the intended label rule. Record the difference and which deployment owns it. An application release may leave a separately deployed collector unchanged.
In the historical investigation, development contained label promotion while the production collector used an earlier configuration. That difference explained the empty view. The investigation did not deploy a collector change.
Keep diagnosis and deployment separate.
Leave a short handoff containing the fixed interval, original selector, observed labels, collector identity and configuration difference. The worksheet has space for each, including unavailable observations.
If a collector change is proposed, name its owner, candidate revision, rollback target and the destination readback that would establish the expected label. Route that change through its own review.