Managed LLM Services: The No-Hype Guide

You want to add a language model to your product. You do not want to rent eight GPUs, babysit an inference server, or explain to finance why there's a datacenter on the credit card. That gap is exactly what managed LLM services fill, and it's why most teams shipping today never touch the model weights at all.

The three big options are Amazon Bedrock, Azure OpenAI (now folded into Azure AI Foundry), and Google Vertex AI. They rhyme more than they differ. This is what they actually give you, where they diverge, and how to choose without over-thinking it.

What a Managed LLM Service Actually Is

Strip away the branding and a managed LLM service is one thing: a serverless API in front of somebody else's foundation models. You send text (and sometimes images or audio), you get tokens back, and you pay per token. No GPUs to provision, no model to download, no autoscaling to tune. The provider handles hosting, capacity, and uptime, and wires the whole thing into their existing security and billing.

That last part is the real selling point. Because the model lives inside your cloud account, the same identity, logging, networking, and compliance controls you already use for storage and databases apply to your AI calls too. For an enterprise, "it's governed like everything else" is worth more than a benchmark point.

The Big Three, Briefly

They overlap heavily, so the useful question is what each one is really built around:

• Amazon Bedrock. A model marketplace. One API in front of many model families (Anthropic, Meta's Llama, Mistral, Amazon's own, and more), so you can swap providers without rewriting your app. Deeply wired into AWS identity and networking, which is the whole appeal if you already live in AWS.
• Azure OpenAI / Azure AI Foundry. The enterprise front door to OpenAI's models, plus a growing catalog, wrapped in Microsoft's compliance and networking story. The natural pick if your org is a Microsoft shop and legal has already blessed Azure.
• Google Vertex AI. Google's Gemini models plus open options, sitting inside a mature end-to-end ML platform. Strong if your data already lives in Google Cloud and your team cares about the wider MLOps toolchain, not just chat completions.

Notice the pattern: teams rarely choose on model quality alone. They choose the platform their data, their identity system, and their procurement paperwork already point to.

How Pricing Really Works

Almost everything is priced per million tokens, billed separately for input and output, with output the more expensive of the two. On top of that base model, two knobs matter:

• On-demand vs reserved. By default you pay per token with no floor, which is perfect for spiky or early traffic. Once volume is steady and predictable, every provider offers a reserved-capacity mode (Azure calls it Provisioned Throughput) that trades a fixed hourly commitment for lower effective cost and guaranteed headroom.
• Discounts you should actually use. Batch endpoints for work that isn't time-sensitive, and prompt caching for repeated context, can each cut the bill meaningfully. If you send the same long system prompt on every call, caching alone can pay for itself.

Calling a model is refreshingly boring once it's set up. Here's a complete Bedrock request with the AWS SDK for Python:

1import boto32
3client = boto3.client("bedrock-runtime", region_name="us-east-1")4
5response = client.converse(6    modelId="anthropic.claude-3-5-sonnet-20240620-v1:0",7    messages=[8        {"role": "user", "content": [{"text": "Explain a vector embedding in one sentence."}]}9    ],10    inferenceConfig={"maxTokens": 200, "temperature": 0.2},11)12
13print(response["output"]["message"]["content"][0]["text"])

Swap the modelId and you're talking to a different model family through the same call. That portability is the point of a marketplace-style service.

How to Choose (a Quick Heuristic)

You can spend a month on a comparison matrix, or you can answer four questions:

1. Where does your data and identity already live? Start with that cloud. Staying in-network saves you egress fees, latency, and a security review.
2. Is the model you want available in your region? Model and region availability lags announcements. Check before you commit.
3. How does it do on your tasks? Build a small private evaluation set and run the finalists against it. Real workload beats leaderboard.
4. What happens at scale? Estimate the token bill on real traffic, then check whether reserved capacity or batching changes the math.

My Take

For most teams the honest answer is "use the one you're already on." The models are close enough that the deciding factors are governance, region availability, and how little new plumbing you have to build. Reach for a specialist provider or self-hosting only when you have a concrete reason: a niche open model, a hard data-residency rule, or volume big enough that owning the GPUs finally beats renting them.

Start on-demand, instrument your token usage before you optimize, and keep a thin abstraction over the API so switching later is a config change, not a rewrite.

The Bottom Line

• A managed LLM service is a serverless, per-token API to foundation models, governed like the rest of your cloud.
• Bedrock, Azure, and Vertex overlap heavily. Pick the one your data and identity already sit in.
• Pricing is per-million-tokens, input and output split; move to reserved capacity only once traffic is steady.
• Meter usage early, watch data residency, and abstract the API to avoid lock-in.

What to Read Next

• How the transformer architecture works - what you're actually renting when you call one of these models.

Further Reading

• Amazon Bedrock documentation
• Azure OpenAI documentation
• Google Vertex AI documentation