Skip to main content

​
Overview

NuGet.Jobs.Db2AzureSearch is a run-once bootstrap tool that creates and populates the Azure AI Search infrastructure required by the NuGet.org search service. It provisions two Azure Search indexes (the “search” index used by end-user queries and the “hijack” index used by the NuGet client), creates the Azure Blob Storage container for search auxiliary files, and bulk-loads every available package registration from the Gallery SQL database into those indexes. Once complete, it writes a catalog cursor blob so that NuGet.Jobs.Catalog2AzureSearch and NuGet.Jobs.Auxiliary2AzureSearch can take over incremental updates. The job reads package metadata, ownership, download counts, excluded package lists, verified package flags, and popularity transfer data all in a single pass. These are written both into the Azure Search document store (as indexed documents) and into Azure Blob Storage (as canonical JSON auxiliary files). Download counts are taken from the external downloads.v1.json statistics file rather than from the Gallery DB, because that is the authoritative source used by the live search service. The core data pipeline uses a producer/consumer pattern: a single producer task fetches package registrations from the database in key-range batches (default 10,000 packages per batch), and a configurable number of consumer tasks (default 4, MaxConcurrentBatches) concurrently build index action documents and push them to Azure Search. A back-pressure mechanism pauses the producer when unprocessed packages in memory exceed twice the DatabaseBatchSize, preventing out-of-memory conditions during large runs. For local development, data can alternatively be sourced from NuGet.Insights Kusto tables instead of the Gallery database.

​
Role in System

After this job completes, NuGet.Jobs.Catalog2AzureSearch picks up from the written cursor and applies incremental catalog changes, while NuGet.Jobs.Auxiliary2AzureSearch keeps the auxiliary files and download counts current on an ongoing basis.

Full Index Bootstrap

Creates both Azure Search indexes from scratch and populates every available package registration. This is the only job that performs a complete rebuild; all other search jobs are incremental.

Dual Data Source Support

Supports two producer backends: NewPackageRegistrationFromDbProducer for production (Gallery SQL DB) and NewPackageRegistrationFromKustoProducer for development (NuGet.Insights Kusto tables). The selection is automatic based on whether KustoConnectionString is configured.

Auxiliary File Initialization

Writes the initial versions of all four search auxiliary files to Blob Storage with an IfNotExists access condition, ensuring they are only written once and not overwritten if they already exist.

Catalog Cursor Seeding

Captures the NuGet catalog’s current commit timestamp before reading the database (ensuring the cursor is no newer than the data read), then writes it as the starting cursor for Catalog2AzureSearch to avoid missing updates.

​
Key Files and Classes

​
Dependencies

​
Internal Project References

​
NuGet Package References

These are declared on NuGet.Services.AzureSearch, which the job depends on transitively:

​
Notable Patterns and Implementation Details

This job is destructive by design. When Development:ReplaceContainersAndIndexes is true, it deletes the existing Azure Search indexes and Blob Storage container before recreating them. This flag must be false in production environments. Running the job against an already-populated production index without this guard would first delete all existing documents.
The catalog cursor is captured before any database reads begin, not after. This is intentional: the database is always more up-to-date than the catalog, so capturing the catalog timestamp first guarantees that any data read from the database is at least as fresh as that timestamp. When Catalog2AzureSearch starts, it may re-process some catalog entries already reflected in the index, but duplicate upserts are safe.
Download counts are intentionally not read from the Gallery database. The job fetches downloads.v1.json from the configured URL (the statistics pipeline), which is the same source the live search service uses. This ensures the index is seeded with the same download counts that will be maintained by Auxiliary2AzureSearch going forward.
The back-pressure mechanism in both producer implementations prevents unbounded memory growth. The producer pauses when the number of package entities in the ConcurrentBag exceeds 2 * DatabaseBatchSize. This allows consumer tasks to drain the queue before more data is loaded from the database or Kusto.
The Kusto producer (NewPackageRegistrationFromKustoProducer) handles oversized result sets by halving the page size whenever Kusto returns an E_QUERY_RESULT_SET_TOO_LARGE error. It starts at 20,000 records per page (well below the 64 MB Kusto response limit) and will retry with progressively smaller pages down to a minimum of 100 records.
Auxiliary files are written with AccessConditionWrapper.GenerateIfNotExistsCondition(). This means a second run of the job (without ReplaceContainersAndIndexes) will skip rewriting the auxiliary files if they were already created by the first run. The index documents, however, are re-uploaded unconditionally.
PackageEntityIndexActionBuilder validates consistency between the database NormalizedVersion field and the parsed NuGetVersion. If they do not match, it throws InvalidOperationException and aborts the job. This guard catches data quality issues in the Gallery database before they corrupt the index.