Skip to main content

​
Overview

StatusAggregator is a .NET 4.7.2 console application (run via NuGet.Jobs.JobRunner) that continuously polls an internal incident API, parses and aggregates incidents into a three-level hierarchy (Incident → IncidentGroup → Event), and serializes the resulting service health state to Azure Blob Storage as status.json. It also supports operator-driven manual overrides — adding, editing, or deleting events and messages — stored in Azure Table Storage. The job runs on a schedule and each execution:
  1. Processes manual status change entries from Azure Table Storage (primary and secondary accounts).
  2. Fetches new incidents from the incident API since the last cursor position.
  3. Updates all active events and their associated component states.
  4. Exports the full service component tree plus recent events to status.json.

Incident Ingestion

Pulls raw incidents from the internal NuGet incident API, authenticated via an X.509 certificate stored in Azure Key Vault.

Status Page Feed

Writes status.json to Azure Blob Storage, which is the data source for the public NuGet status page at status.nuget.org.

Manual Override Support

Operators can write ManualStatusChangeEntity records to Azure Table Storage to inject, edit, or remove events and messages without a code deploy.

Dual-Region Resilience

Reads and writes to both a primary and a secondary Azure Storage account, ensuring status data is kept consistent across regions.

​
Key Files and Classes

File PathClassPurpose
Program.csProgramEntry point; delegates to JobRunner.Run(new Job(), args)
Job.csJobWires up Autofac/DI container, configures storage, parsers, factories, updaters, and exporters
StatusAggregator.csStatusAggregatorOrchestrates a single run: initializes storage, calls IStatusUpdater.Update, then IStatusExporter.Export
StatusAggregatorConfiguration.csStatusAggregatorConfigurationStrongly-typed configuration: storage endpoints, table/container names, delay timings, severity/environment filters
Collector/EntityCollector.csEntityCollectorWraps a cursor + processor; fetches new entities since the last recorded cursor position
Collector/IncidentEntityCollectorProcessor.csIncidentEntityCollectorProcessorFetches incidents from the incident API since a cursor date and stores them as IncidentEntity table rows
Collector/ManualStatusChangeCollectorProcessor.csManualStatusChangeCollectorProcessorReads ManualStatusChangeEntity rows from a storage account and dispatches them to IManualStatusChangeHandler
Update/StatusUpdater.csStatusUpdaterDrives the update cycle: runs manual collectors, runs the incident collector, then calls ActiveEventEntityUpdater
Update/ActiveEventEntityUpdater.csActiveEventEntityUpdaterQueries all active EventEntity rows and calls IComponentAffectingEntityUpdater<EventEntity> on each
Update/AggregationEntityUpdater.csAggregationEntityUpdater<TChild, TAgg>Generic updater that deactivates an aggregation entity when all its children are mitigated
Factory/NuGetServiceComponentFactory.csNuGetServiceComponentFactoryBuilds the static NuGet component tree: Gallery, Restore (V2/V3), Search (Global/China), Package Publishing
Factory/AggregationStrategy.csAggregationStrategy<TChild, TAgg>Determines whether a new entity can be linked to an existing aggregation by checking active state and child presence
Factory/IncidentFactory.csIncidentFactoryCreates IncidentEntity rows from a ParsedIncident
Factory/IncidentGroupFactory.csIncidentGroupFactoryCreates IncidentGroupEntity rows grouping related incidents
Factory/EventFactory.csEventFactoryCreates EventEntity rows grouping related incident groups
Parse/AggregateIncidentParser.csAggregateIncidentParserFans out to all registered IIncidentParsers and collects every ParsedIncident result
Parse/IncidentRegexParsingHandler.csIncidentRegexParsingHandlerAbstract base for regex-based parsers; subclasses extract component path and status from named capture groups
Parse/SeverityRegexParsingFilter.csSeverityRegexParsingFilterDrops incidents whose severity exceeds MaximumSeverity in configuration
Parse/EnvironmentRegexParsingFilter.csEnvironmentRegexParsingFilterDrops incidents not matching the configured Environments list
Export/StatusExporter.csStatusExporterBuilds the ServiceStatus from IComponentExporter + IEventsExporter and hands it to IStatusSerializer
Export/StatusSerializer.csStatusSerializerSerializes ServiceStatus to JSON and uploads it as status.json to the primary blob container
Manual/ManualStatusChangeHandler.csManualStatusChangeHandlerDispatches ManualStatusChangeEntity records to typed handlers (add/edit/delete for events and messages)
Table/TableWrapper.csTableWrapperThin wrapper over Azure SDK TableClient; enforces partition key discipline via TablePartitionKeys
Container/ContainerWrapper.csContainerWrapperWraps BlobContainerClient for blob create/save operations

​
Dependencies

​
Internal Project References

ProjectPurpose
NuGet.Jobs.CommonJsonConfigurationJob, JobRunner, StorageAccountHelper, StorageMsiConfiguration base infrastructure
NuGet.Services.IncidentsIIncidentApiClient, Incident, IncidentApiConfiguration — the incident API client contract
NuGet.Services.Status.TableIncidentEntity, IncidentGroupEntity, EventEntity, ComponentAffectingEntity, manual change table entities
Validation.PackageSigning.CoreProvides X509Certificate2 extension ComputeSHA256Thumbprint used when loading the API certificate

​
Key NuGet / Framework Packages (resolved transitively)

PackagePurpose
AutofacIoC container used alongside Microsoft DI for named/keyed storage registrations
Azure.Data.TablesAzure Table Storage SDK v12 (TableClient, TableServiceClient)
Azure.Storage.BlobsAzure Blob Storage SDK v12 (BlobServiceClient, BlobContainerClient)
Microsoft.Extensions.OptionsIOptionsSnapshot<T> for scoped configuration binding
Newtonsoft.JsonJSON serialization of ServiceStatus with UTC dates and string enums

​
Notable Patterns and Implementation Details

The job registers two named storage accounts (“Primary” and “Secondary”) in Autofac using Named<T> registrations. The primary account is registered last so it becomes the default unnamed binding. Both accounts get their own ManualStatusChangeCollectorProcessor, meaning manual overrides written to either account are processed.
Incidents are aggregated through a three-level hierarchy: raw IncidentEntity records are grouped into IncidentGroupEntity records (same component path), which are further grouped into EventEntity records. AggregationStrategy<TChild, TAgg> enforces that a child can only be linked to an aggregation that has existing children (preventing reuse of manually-created or broken aggregations) and that is still active if the child is active.
The cursor pattern (ICursor backed by Azure Table Storage) gives each collector a named timestamp watermark. The incident collector advances the cursor only on success. The StatusUpdater wraps incident ingestion in a try/catch so a transient API failure does not block the export step — the last successfully exported status remains live.
Certificate loading in Job.GetCertificateFromConfiguration supports two legacy formats: a JSON object with Base64 Data + Password (older Key Vault certificates) and a plain Base64 string (newer Key Vault certificates). If neither parses successfully the job will throw at startup. Ensure Key Vault secrets are in one of these two forms.
The target framework is net472. All Azure SDK calls use async/await but the entry point calls .GetAwaiter().GetResult() — this is intentional for the JobRunner pattern used across NuGet background jobs. Do not migrate to net6+ without verifying JobRunner compatibility.
Configuration delays (EventStartMessageDelayMinutes, EventEndDelayMinutes) default to 15 minutes each. EventVisibilityPeriodDays defaults to 10 days. These can be tuned in the StatusAggregator config section without a code change, which is useful during incidents when faster message cadence is desired.

​
Data Flow Summary

Incident API
    │
    ▼
IncidentEntityCollectorProcessor  ──►  IncidentEntity (Table)
    │                                        │
    │                                        ▼
    │                               AggregationStrategy
    │                               IncidentGroupFactory  ──►  IncidentGroupEntity (Table)
    │                                        │
    │                                        ▼
    │                               AggregationStrategy
    │                               EventFactory          ──►  EventEntity (Table)
    │
ManualStatusChangeCollectorProcessor ──►  Manual overrides applied to Table entities
    │
    ▼
ActiveEventEntityUpdater  ──►  Updates EventEntity / IncidentGroupEntity active states
    │
    ▼
StatusExporter
    ├── ComponentExporter  ──►  NuGetServiceComponentFactory (static tree + live statuses)
    ├── EventsExporter     ──►  Recent EventEntity rows
    └── StatusSerializer   ──►  status.json  ──►  Azure Blob Storage (Primary + Secondary)