Skip to main content

​
Overview

NuGet.Protocol.Catalog is a reusable class library for reading and processing the NuGet V3 Catalog API. The catalog is an append-only event log published at https://api.nuget.org/v3/index.json that records every package publish, metadata edit, and delete event on NuGet.org. This library provides the typed JSON models, an HTTP deserialization client, and a high-level processor that walks the catalog hierarchically (index → pages → leaves) in strict chronological order. The central design is the cursor pattern: an ICursor abstraction records the last successfully processed commit timestamp so that any consumer can resume exactly where it left off after a restart or transient failure. The built-in FileCursor stores this value as a JSON file on disk, while callers can implement their own ICursor backed by a database or Azure Blob Storage. The CatalogProcessor writes to the cursor at commit boundaries — not after every leaf — which means cursor advancement is always aligned with atomic catalog commits. The library deliberately contains no application-level logic. All domain behavior lives behind the ICatalogLeafProcessor interface, which callers implement to decide what to do with each PackageDetailsCatalogLeaf or PackageDeleteCatalogLeaf as it is delivered. This keeps the library focused on reliable catalog traversal and leaves processing concerns entirely to the consumer.

​
Role in System

Internal consumers of this library are NuGet.Services.Metadata.Catalog (the metadata pipeline) and Monitoring.PackageLag (the package lag monitor).

Cursor-Driven Resumability

The ICursor abstraction records the last committed catalog timestamp. The CatalogProcessor advances the cursor at commit boundaries so processing always resumes from a consistent point after a crash or restart.

Redundant Leaf Deduplication

When ExcludeRedundantLeaves is enabled (the default), multiple entries for the same package ID and version within a single page are collapsed to the latest one, reducing unnecessary downstream work.

Hierarchical Traversal

The processor walks the three-level catalog hierarchy (index → pages → leaves) and filters pages and leaves by commit timestamp bounds before fetching them, minimising HTTP requests to only the range of interest.

Two-Pass Leaf Deserialization

CatalogClient.GetLeafAsync downloads leaf JSON once as a byte array, peeks at the @type field to determine the concrete leaf type, then deserializes again into the correct strongly-typed model without a second HTTP round-trip.

​
Key Files and Classes

​
Dependencies

​
NuGet Package References

​
Internal Project References

​
Notable Patterns and Implementation Details

Commit-boundary cursor advancement. The CatalogProcessor only writes to the cursor when transitioning from one commit timestamp to the next, not after every individual leaf. This means if processing fails mid-commit, the entire commit is retried on the next run. The ICatalogLeafProcessor contract explicitly states the same package/version pair may be delivered more than once and implementations must be idempotent.
Two-level timestamp filtering. GetPagesInBounds filters the index to pages whose commit timestamp is strictly greater than the cursor value. GetLeavesInBounds then applies the same bounds at the leaf level within each page. A page is included even if only one of its leaves falls in range because a page’s commit timestamp represents the maximum timestamp of all its leaves.
ExcludeRedundantLeaves reduces work for burst edits. When a package is edited multiple times within a single catalog page (e.g. metadata corrections in rapid succession), only the latest leaf for that package ID + version is delivered to the processor. This is on by default in CatalogProcessorSettings and is the recommended setting for consumers that only care about current state.
GetLeafAsync downloads the JSON body twice from memory. When the leaf type is unknown in advance, CatalogClient.GetLeafAsync downloads the full JSON as a byte array, deserializes it once to read @type, then deserializes it a second time into the concrete type. This avoids a second HTTP request but does hold the entire leaf document in memory simultaneously. Prefer GetPackageDetailsLeafAsync or GetPackageDeleteLeafAsync when the leaf type is already known from the page summary item.
CatalogLeafTypeConverter handles JSON-LD array @type. The @type field in full leaf documents may arrive as either a plain string or a JSON array of strings (as used by JSON-LD). The CatalogLeafTypeConverter handles both forms by iterating the array and returning the first recognized value. The page summary CatalogLeafItemTypeConverter does not handle arrays and expects a single nuget:-prefixed string.
ModelExtensions.IsListed handles legacy unlisted encoding. Some very old NuGet.org catalog entries do not include a listed property. The extension method falls back to checking whether Published.Year == 1900, which is the legacy server-side convention for marking a package as unlisted. This is explicitly called out in the code with a catalog example URL.