How to Write an AI Infrastructure RFP: GPU Cloud, Clusters and Private AI

An AI infrastructure RFP asks vendors to propose the compute, networking, storage and platform your AI workloads will run on: rented GPU capacity, a cluster you own, a private AI platform on your existing stack, or a managed inference service. It follows the structure in how to write an RFP, but most vendor claims can only be proven by running your own jobs.

Start with the workload, not the hardware

Vendors can quote hardware all day; they can’t tell you what you need. List the workloads first, because each one changes what you buy.

• Training a model: many accelerators on one job for days or weeks. The network between nodes and checkpoint storage matter as much as the chips.
• Fine-tuning: short, repeated bursts. Capacity you can get quickly and hand back.
• Inference: a steady load that scales with users. Latency and cost per request matter; utilization decides whether owning pays.
• Retrieval-augmented generation: inference plus a search index, document pipelines and control over what a model may read.
• Agents: many model calls per task, tool access, and identities of their own.

For each, record model sizes, the largest job, monthly volume, where the data may live and the latency users will accept. If you can’t write them down yet, issue a short RFI first.

Leave the sourcing decision open

Four ways to get AI capacity: rent it from a GPU cloud, buy a cluster, run a private AI platform on your existing virtualization or container stack, or buy model calls from a managed inference service. A mix is common: rent the training bursts, own or commit to the inference baseline. Enterprise Techie’s buy-or-rent break-even analysis and its self-hosting versus paying per call analysis show how to run that comparison.

Write the RFP so any bidder can answer it. State workloads and outcomes, not node counts, and require every bidder to complete the same pricing form, in units you define (accelerator-hour, request or token served), totaled over the term at your expected utilization. Then a rental, a cluster and a managed service compare directly. RFPhub’s free GPU Cloud and AI Infrastructure RFP Template is written for the rented case; other bids can answer the same lines on a different pricing form.

Requirement areas, with lines you can paste

Mark each line Must-have or Nice-to-have, ask for a response code on every one (standard, configuration, custom, third party, roadmap or not available), and for capacity, whether it is installed today or still to come.

Compute

• [Number] accelerators of [generation and memory each] in [regions] from [date] for [term], dedicated or from a shared pool (state which).
• On-demand capacity above our reservation, with lead times and allocation rules under contention.
• Notice periods and charges for returning capacity mid-term.
• A path to the next accelerator generation mid-term, on stated terms.

Interconnect and networking

• Specifications per node type: accelerators, memory, CPU, local storage, intra-node links.
• A dedicated low-latency fabric with remote direct memory access for multi-node jobs: bandwidth per accelerator, topology and oversubscription at [number] accelerators.
• All nodes of a reserved cluster in one site on one fabric, and the largest cluster offered that way.
• Private connectivity to our data centers and clouds; bulk data transfer at onboarding.

Storage and data pipelines

• Shared file storage for training data and checkpoints, with read and checkpoint write throughput stated and verified in the proof of concept.
• Object storage for datasets and weights with a widely used API, and its throughput to the nodes.
• Storage classes per region, and charges for moving data between them.
• Support for our own data loading, caching and pipeline tools.

Data residency, sovereignty and tenancy isolation

• A contractual commitment that our data, weights, prompts, logs and backups stay in [jurisdictions], with sub-processors listed.
• Documented tenant isolation wherever compute, fabric or storage is shared; a single-tenant option for [sensitive workloads].
• Wiping of local storage and accelerator memory before a node we used is reassigned.
• Provider staff access only with our approval, logged and exportable.

Model and runtime support

• Open-weight and our own models on the serving frameworks we use [list], without the provider’s tooling.
• A model catalog with versions and approval status, recording which version served each request.
• Validated images with current drivers, version pinning, maintenance scheduled around long jobs.
• Managed inference endpoints with autoscaling, including to zero, and staged rollouts with rollback.

Security and identity

• Single sign-on and role-based access through our identity provider, on open standards, with group provisioning.
• Customer-managed encryption keys for all storage, and a secrets store for keys and tokens, or integration with ours.
• Identities for agents, services and workloads separate from human accounts, with short-lived credentials, a named human owner and an audit trail.
• Independent evidence of controls, such as a SOC 2 report or ISO/IEC 27001 certificate covering the proposed services and locations.

Observability and cost controls

• Utilization, memory, power and error metrics per accelerator, node, job and team, exportable on open standards.
• Quotas and priorities per team or project, with alerts for idle capacity.
• Budgets with hard limits, and cost by team, project and model for chargeback.
• Billing data exportable by API or file.

Operations and support

• Service levels we define for availability, node replacement and support response, with credits and how each is measured.
• Health checks that catch failing accelerators and fabric faults, pull the node from scheduling and alert us.
• Burn-in and acceptance testing before handover; billing starts after acceptance.
• Maintenance notice, root-cause reports, and roadmap briefings at a frequency we set.

Portability and exit

• No proprietary formats for data, weights, containers or pipelines; state anything that would not run unchanged elsewhere.
• Export of data, checkpoints, weights, logs and configuration at exit, with method, time and charges stated.
• Notice periods for ending or reducing a commitment, and transition assistance.
• Certified deletion of our data after exit.

Commercial model

• Price per unit for each accelerator and capacity type (committed, on demand, interruptible), showing list, discount and net.
• Every other charge itemized (storage, networking, data transfer out, services, support, model usage), marked metered or fixed.
• Billing increments and minimums, and what happens to commitments we don’t use.
• A cap on renewal increases, and remedies if committed capacity arrives late.

Vendor questions that separate bidders

• Will you run our reference job on the exact hardware and fabric proposed, at our batch sizes, and share the results and configuration?
• In the proof of concept, show the utilization our workload reaches and the reports we would use to see it in production.
• At our quota, are jobs queued or rejected, does work overflow to on-demand capacity, at what price, and how fast?
• If a new accelerator generation ships mid-term, what are our options for the committed capacity, and on what terms?

Why the proof of concept matters more here

The RFP process guide (step 8) says to run a proof of concept when a demo can’t prove a requirement that matters, such as performance at your volumes. For AI infrastructure, performance at your volumes is the purchase. Throughput depends on your model, batch size, data loader, fabric and storage together, so vendor benchmarks don’t transfer. Plan a proof of concept with your two or three finalists as a normal step, and say so in the RFP.

Measure, with your own people timing it: throughput of the reference job on one node and as nodes are added; checkpoint write and restore times; data loading from the proposed storage; inference latency under a load shaped like yours; what happens when a node fails mid-job; and whether utilization reports match reality. Agree scope, duration, success criteria, data handling and who pays before it starts, using the proof of concept template.

Scoring the proposals

Score the way the RFP evaluation criteria guide describes: a 1-to-5 scale with defined scores, weights set before proposals arrive, and a weighted score of score × weight ÷ 100. Gate on residency and isolation Must-haves, and hold performance scores until the proof of concept is done. As an example only:

An example timeline

The GPU cloud template’s example schedule runs about 10 weeks from kickoff to award; the RFP process guide’s general example runs about 17 weeks. A cluster purchase adds delivery lead time.

Common mistakes

• Buying for peak training when the load is inference. Size the baseline from inference; rent the peaks.
• Skipping storage. Accelerators waiting on data cost the same as busy ones.
• Forgetting egress and exit. Data transfer out and exit charges decide whether you can ever change providers.
• Naming the hardware instead of the workload. A specific part closes out bids that would meet the need differently.
• Treating planned capacity as installed. Ask, and write delivery remedies into the contract.

Start from the GPU Cloud and AI Infrastructure RFP Template. For neighboring purchases, see the LLM gateway and AI platform template, the non-human identity and AI agent template and the AI governance and compliance platform template.

Further reading on Enterprise Techie

• The buy-or-rent break-even analysis
• What an on-premises AI cluster costs
• Self-hosted models versus paying per call
• Private AI platforms compared
• Governing AI agent identities

Frequently asked questions

What should an AI infrastructure RFP include?

Each workload with sizes and volumes; requirements for compute, networking, storage, residency, models, security, observability, operations, exit and price, marked Must-have or Nice-to-have; one pricing form for every bidder; the scoring weights; and a proof of concept before award.

How do you write GPU requirements for an RFP?

Describe the job, not the part: model sizes, the largest single job, memory per accelerator, regions, start date and term. Ask whether the capacity is installed today or still to come, how it scales, and the path to the next generation. Then verify it in a proof of concept with your own job.

Should we buy or rent GPUs?

The RFP should let vendors propose both. Steady, well-utilized load tends to favor owning or committing, bursty load favors renting, and the crossover depends on your utilization, facilities and staff. Ask every bidder to price your workloads on the same form and compare the totals.

What is a private AI RFP?

A private AI RFP asks vendors to propose a platform for running models on infrastructure you control, usually a virtualization or container stack in your own data center, so prompts, data and weights never leave your environment. It covers the same areas as any AI infrastructure RFP, weighted toward residency, isolation and the skills your team has.