Tracking events swamp everything else

The situation

Every shipment generates a stream of milestone events from carriers, and they never stop. They are small, they are constant, and they are the reason the database is sized the way it is — so the shipments, the bookings and the invoices all pay for the tracking volume. Separating them out has been deferred for a year because it means a storage migration and a change to every client that reads them.

These are illustrative scenarios showing how the products are used, not customer case studies.

What gets modeled

The structure as it would actually be defined — sections that repeat are marked.

How it gets built

In the order somebody would actually do it. Each step names the capability it rests on, so it can be checked against the product pages.

1. 1

   Model the shipment as it is routed

   A shipment has legs, and a leg has milestones, so both repeat and the milestones nest inside the leg they belong to. The structure is the routing rather than a flattened event log with a shipment reference on every row.

   What it uses: Sections that nest and repeat to any depth

2. 2

   Give the noisy dataset its own database

   Tracking events are pinned to their own provisioned database — from a dedicated logical database up to a dedicated cluster — while shipments stay where they are. Each schema’s tables live in their own namespace, and how the APIs are called does not change, so no client is rewritten.

   What it uses: A schema pinned to its own provisioned database

3. 3

   Make the failure mode explicit

   If that database cannot be reached the request fails rather than quietly falling back to the main one. This is a placement decision you make and can reason about, not automatic sharding that moves data when you are not looking.

   What it uses: Deliberate placement, with no silent fallback

4. 4

   Stop the volume starving the urgent work

   Low-priority tracking volume runs in its own work pool, so a burst of milestone events cannot delay a booking confirmation. Limits on concurrent runs, dispatch rate and run duration are configured rather than discovered.

   What it uses: Priority work pools and configured concurrency limits

5. 5

   Take the carrier feeds as steps

   Each carrier is registered by importing its API description, or by hand where there is none, with its own authentication, retry policy and throttling. Adding a carrier is a registration and a mapping rather than a project.

   What it uses: Any API becomes a step, with per-connection authentication and throttling

6. 6

   Tolerate the same event arriving twice

   Carriers resend. Repeat deliveries of the same event are recognized and ignored, and steps are claimed so that running several workers never fires the same step twice — delivery is at least once, so the steps are written to tolerate it.

   What it uses: Repeat-event recognition, and claimed steps across workers

7. 7

   Let customers subscribe instead of polling

   Milestone events are delivered to a customer’s endpoint, queue or stream, signed, with retries, a delivery log with per-attempt detail and dead-lettering that can be redriven in bulk. A failure rate crossing a threshold raises an alert rather than being noticed by the customer.

   What it uses: Signed event delivery with retries, dead-lettering and alerting

Numbers used above, and where they come from

Every figure on this site is counted from the product itself. Nothing here is a performance or customer claim, because there is no measurement to cite for one.