Skip to main content

​
Overview

Ng is the background-job executable (ng.exe) for the NuGet Gallery infrastructure. It is a single .NET 4.7.2 console application that hosts a named-job dispatcher: at startup the caller passes a job name as the first argument, and NgJobFactory instantiates and runs the matching NgJob subclass. The project has no REST API surface. Its purpose is to run long-lived, looping or one-shot background workers that keep derived NuGet infrastructure (catalog, flat container, registration hives, monitoring queues, icon CDN) in sync with the gallery database.
ng.exe is the legacy job host. Newer NuGet jobs use NuGet.Jobs-based infrastructure with JSON configuration. Ng predates that pattern and uses its own key-value CLI argument convention (-argName value).
Gallery DB ──► db2catalog ──► Catalog (Azure Blob)
                                   │
                    ┌──────────────┼──────────────┐
                    ▼              ▼               ▼
             catalog2dnx    catalog2icon   catalog2monitoring
            (flat container) (icon CDN)   (validation queue)
                                               │
                                        monitoringprocessor
                                        monitoring2monitoring

Catalog Feeds

db2catalog reads the Gallery SQL database and writes package create / edit / delete events into the V3 catalog JSON feed stored in Azure Blob Storage.

Flat Container (DNX)

catalog2dnx reads the catalog and populates the flat-container blob hierarchy used by NuGet restore.

Icon Pipeline

catalog2icon extracts embedded package icons and external icon URLs from catalog leaves and copies them into a CDN-backed icon storage container.

Monitoring

catalog2monitoring, monitoring2monitoring, and monitoringprocessor queue packages for endpoint validation and process the results.

​
Key Files and Classes

FileClass / TypePurpose
Program.csProgramEntry point; configures Application Insights, Serilog, Autofac DI, KeyVault secret injection, then delegates to NgJobFactory.
NgJobFactory.csNgJobFactoryStatic string→Type map; resolves and instantiates the requested NgJob subclass via reflection.
Jobs/NgJob.csNgJob (abstract)Base class for all jobs. Owns logger, telemetry client, SecretInjector, HTTP connection-limit tuning (DefaultConnectionLimit = 64). Defines Init + RunInternalAsync template.
Jobs/LoopingNgJob.csLoopingNgJob (abstract)Extends NgJob with an infinite polling loop. Supports configurable interval and ReinitializeIntervalSec. Swallows TransientException without stopping the loop.
Jobs/Db2CatalogJob.csDb2CatalogJobReads package creates, edits, and deletes from the Gallery SQL DB and appends them to the V3 catalog using AppendOnlyCatalogWriter. Detects deletes from auditing blob storage.
Jobs/Catalog2DnxJob.csCatalog2DnxJobDrives DnxCatalogCollector to mirror catalog entries into the flat container. Uses DurableCursorWithUpdates. Max parallelism 256.
Jobs/Catalog2IconJob.csCatalog2IconJobDrives IconsCollector with IconProcessor + ExternalIconContentProvider. Caches failure results via IconCopyResultCache.
Jobs/LightningJob.csLightningJobTwo-phase bulk registration reindex tool. Prepare traverses the entire catalog index and emits batched cursor files. Strike processes one batch, writing registration hive blobs.
Jobs/Catalog2MonitoringJob.csCatalog2MonitoringJobEnqueues PackageValidatorContext messages for packages that appear in the catalog.
Jobs/MonitoringProcessorJob.csMonitoringProcessorJobDequeues and executes validations against registration, flat-container, and search endpoints.
Jobs/Db2MonitoringJob.csDb2MonitoringJobSeeds the monitoring queue directly from the Gallery DB rather than from the catalog.
Jobs/Monitoring2MonitoringJob.csMonitoring2MonitoringJobRequeues packages that previously failed validation.
CommandHelpers.csCommandHelpers (static)Parses CLI arguments, wires up KeyVault secret injection, creates StorageFactory instances, builds HTTP handler pipelines with telemetry.
Arguments.csArguments (static)Centralised string constants for every recognized CLI argument name.

​
Dependencies

​
NuGet Package References

PackagePurpose
Serilog.Sinks.FileRolling file log sink; writes Log.txt with 3-file retention, 1 MB size limit.

​
Internal Project References

ProjectRole
NuGet.Jobs.Catalog2RegistrationProvides IRegistrationUpdater, Catalog2RegistrationConfiguration, and Autofac modules used by LightningJob to write registration hive blobs.
NuGet.Services.Metadata.Catalog.MonitoringProvides ValidationCollector, PackageValidatorContextEnqueuer, monitoring endpoint types.
NuGet.Services.StorageProvides IStorageQueue<T>, StorageQueue<T>, AzureStorageQueue, and JsonMessageSerializer<T>.

​
Notable Patterns and Implementation Details

Cursor-based resumability. Every looping job tracks its position in the catalog via a DurableCursor written to blob storage (cursor.json). On restart the job resumes from the last committed position rather than replaying the entire catalog.
Reinitialization interval. LoopingNgJob separates the loop interval (-interval, default 3s) from the reinitialization interval (-ReinitializeIntervalSec, default 3600s). Expensive operations like creating storage clients and resolving Key Vault secrets only happen once per hour.
Lightning job is Azure-only. LightningJob.Init hard-codes arguments[Arguments.StorageType] = "azure". Attempting to point it at file storage will fail at runtime. It also requires three separate storage containers (legacy, gzipped, SemVer 2.0.0) to be configured.
Db2CatalogJob deduplicates deletes per commit. When multiple audit entries for the same package ID+version appear within one batch, SegmentPackageDeletes splits them into separate catalog commits. A Thread.Sleep(1s) is injected between segments to guarantee distinct commit timestamps.
KeyVault is optional. If -vaultName is not supplied, CommandHelpers substitutes an EmptySecretReader. This makes local development against file storage straightforward without Azure KeyVault credentials.
Managed Identity vs. certificate auth. CommandHelpers.GetSecretInjector selects between ManagedIdentityCredential (when -useManagedIdentity true) and certificate-based KeyVaultReader automatically, allowing the same job binary to run in both MSI-enabled pods and environments authenticated via storage account keys.