A partial probe cannot produce a complete new inventory. That sounds obvious until an incremental recon job replaces its state file with only the hosts it happened to observe during the latest run.

My pipeline separates discovery, HTTP probing, and downstream scanning. The HTTP job rotates through chunks of the resolved host list. This makes each run manageable, but it also means the state update is a merge with a defined scope.

Distinguish input, observation, and retained state

For one probe run, there are three relevant sets:

known inventory
hosts selected for this run
successful observations from this run

Hosts outside the selected chunk have not been retested. Their absence from the result says nothing about their current availability.

A selected host without a successful observation is different. The job attempted to inspect it and did not obtain the expected result. Retaining its old positive observation forever would make the inventory progressively less trustworthy.

The code therefore needs to update all hosts in the attempted chunk, including those without successful output, while retaining state for hosts outside that chunk. That is the core rule behind the state parser in this pipeline.

Queue changes instead of coupling jobs

The HTTP job accumulates new and changed observations in a downstream queue. It does not directly start a full vulnerability scan for every probe result.

This separates observation frequency from scan frequency. It also gives the scanning job a recoverable input boundary: a failed scan does not require repeating domain discovery just to reconstruct its work.

Queue deduplication and state deduplication serve different purposes. One avoids repeated work items; the other avoids conflicting descriptions of a host. Treating both files as interchangeable lists hides those differences.

Serialize the writer

The Jenkins job disables concurrent builds. That is particularly useful when a rotating pointer and shared state files are part of the design.

Without a clear writer boundary, two successful processes can both read the same pointer, process the same chunk, and overwrite each other's updates. A scheduler interval alone does not prevent that overlap when a run takes longer than expected.

For a future multi-worker design, I would replace the shared pointer with explicitly claimed work items. Adding workers before changing ownership would make the system faster at producing uncertain state.

Test transitions, including silence

The most useful regression cases are small inventories: one host changes, one becomes unavailable, one remains outside the chunk, and one is excluded from downstream scanning.

The expected result should specify both retained state and queued work. A test that checks only successful HTTP output misses the failure mode that matters most: incorrectly interpreting missing observations.

The same distinction applies beyond recon. Whenever a job reads part of a dataset, it must carry the scope of that read into the state update.