Large Dataverse imports should balance throughput with service protection limits and the complexity of each operation. A high request rate can trigger throttling and make the overall import slower and less reliable.
Overview
Plan Dataverse data imports with controlled concurrency, batch sizing, retries and service-protection awareness for reliable bulk integration.
Implementation steps
- Estimate record volume, relationships, plug-in workload and required completion window.
- Start with small batches and modest concurrency, then measure latency, throughput and throttling responses.
- Use $batch or supported bulk messages where they fit the operation and transaction requirements.
- Handle partial failures and preserve a per-record outcome so failed records can be retried safely.
- Use bounded concurrency and honour Retry-After when service protection limits are reached.
Example
// Simple bounded concurrency pattern in C#
var options = new ParallelOptions { MaxDegreeOfParallelism = 4 };
await Parallel.ForEachAsync(records, options, async (record, ct) =>
{
await UpsertRecordAsync(record, ct);
});
// Tune concurrency from observed latency and throttling, not guesswork.
Common pitfalls
- Do not launch an unbounded Task for every row.
- Do not retry a failed batch without understanding which operations committed.
- Avoid increasing concurrency when the service is already throttling.
Validation checklist
Run a representative load test, capture throughput and 429 rates, verify per-record outcomes and confirm retries do not duplicate data.
Conclusion
Adopt a measured rollout, monitor the real workload and make recovery part of the design. Reliable production systems depend on tested operational practices as much as on working code.