Runpod Python SDK

Use the runpod Python package to define GPU functions, deploy apps, call endpoints, build Serverless workers, and manage Runpod resources from one SDK.

The runpod Python package is the single SDK for building on Runpod. It includes the decorator-based app SDK that was previously shipped as Flash, the rp command-line interface, the client for calling existing Serverless endpoints, the Serverless worker SDK, and the API wrapper for managing Pods and other resources.

import runpod

app = runpod.App("hello")

@app.queue(gpu="H100", dependencies=["torch"])
def hello():
    import torch
    return {"gpu": torch.cuda.get_device_name(0)}

@runpod.local_entrypoint
def main():
    print(hello.remote())

Run the file with rp flash dev main.py. The hello function runs on a Runpod GPU, and main runs on your machine.

Flash is now part of the Runpod SDK. If you have existing code that uses the runpod-flash package, it continues to work. To move an existing project, see Migrating from Flash.

Get started

Quickstart

Install the SDK and run a function on a GPU.

Migrating from Flash

Move an existing Flash project to the Runpod SDK.

CLI reference

Use the rp CLI to log in, develop, and deploy.

Install

The SDK requires Python 3.10 to 3.14, and the features on this page require runpod 2.0 or later.

pip install runpod

# If using uv:
uv add runpod

Until 2.0.0 is released, install the release candidate with pip install runpod==2.0.0rc1.

Installing the package also installs the rp CLI. The same CLI is available under the longer name runpod.

Authenticate

Log in once to store your credentials:

rp login

This opens your browser to approve access. Credentials are saved to ~/.runpod/config.toml and are shared by the SDK and the CLI. See rp login for other options, including named profiles.

You can also set the RUNPOD_API_KEY environment variable. This is the only environment variable the SDK reads for authentication.

What’s in the SDK

The runpod package covers four areas. Each one is independent, so you can use any of them on its own.

Area	Use it to	Main entry points
Apps	Define functions and classes in Python that run on Runpod GPUs or CPUs, then deploy them as endpoints.	runpod.App, @app.queue, @app.task, @app.api, rp flash
Endpoint client	Call a Serverless endpoint that already exists, by ID.	runpod.Endpoint
Serverless worker	Write the handler that runs inside a custom Serverless worker container.	runpod.serverless.start
API wrapper	Create and manage Pods, endpoints, templates, and other resources.	runpod.create_pod, runpod.get_pods, and related functions

Apps

An app is a collection of Python functions and classes that run on Runpod. Create one with runpod.App, then attach resources to it with decorators. Each decorator defines a different kind of remote compute.

Resource types

Decorator	What it creates	Use it for
@app.queue	An autoscaling, queue-based Serverless endpoint.	Inference, batch jobs, and other request/response work.
@app.task	One ephemeral Pod per call. The Pod provisions, runs to completion, and terminates.	Training, fine-tuning, and other long-running jobs.
@app.api	A load-balanced Serverless endpoint that serves a Python class over HTTP.	Web services and APIs with multiple routes.

The following example defines one of each:

import runpod
from runpod import App, Model, NetworkVolume, Secret

app = App("inference")

models = NetworkVolume("models", size=100)
llama = Model("meta-llama/Llama-3.1-8B-Instruct")

# Autoscaling job queue on H100s, scales to zero when idle
@app.queue(
    gpu="H100",
    workers=(0, 3),
    dependencies=["vllm"],
    mounts={"/runpod-volume": models},
    model=llama,
    env={"HF_TOKEN": Secret("hf-token")},
)
def chat(prompt: str):
    import vllm

    llm = vllm.LLM(model=str(llama.path))
    return llm.generate(prompt)

# One ephemeral Pod per call
@app.task(gpu="H100", gpu_count=2, mounts={"/models": models})
def finetune(steps: int = 1000):
    ...
    return {"loss": 0.1}

# Load-balanced HTTP service
@app.api(cpu="cpu3c-1-2")
class Counter:
    @runpod.init
    def setup(self):
        self.count = 0

    @runpod.post("/bump")
    async def bump(self, body: dict):
        self.count += body.get("by", 1)
        return {"count": self.count}

    @runpod.get("/value")
    async def value(self):
        return {"count": self.count}

In this example:

• chat runs on an autoscaling queue endpoint with between 0 and 3 workers. Runpod stages the model weights on the host before the worker starts (model=llama), the network volume is mounted at /runpod-volume, and the HF_TOKEN environment variable is read from a Runpod secret.
• finetune provisions a two-GPU Pod for each call and terminates it when the function returns.
• Counter is served over HTTP. The method marked with @runpod.init runs once when each worker starts, and methods marked with @runpod.get and @runpod.post become routes. Workers keep state in memory between requests.

Specify hardware with gpu= or cpu=, which are mutually exclusive. For GPUs, use a short name such as "4090", "H100", or "B200", a runpod.GpuGroup pool such as runpod.GpuGroup.ANY, or a runpod.GpuType device. Pass a list to allow several options. If you don’t set workers, resources scale between 0 and 3 workers.

For every option, see Resource parameters, GPU types, and CPU types.

Call remote functions

Decorated functions keep their Python identity. You call them through methods that control where and how they run:

Call	Behavior
fn.remote(...)	Runs the function on Runpod and blocks until the result is ready.
await fn.remote.aio(...)	The async form of .remote().
fn.spawn(...)	Submits the work and returns a Job handle immediately, without waiting for the result.
fn.stream(...)	Streams results back as the function produces them.
fn.local(...)	Runs the function in your local Python process. Remote storage isn’t mounted.

For HTTP services defined with @app.api, call routes through the class, for example Counter.post("/bump", {"by": 3}) or Counter.get("/value").

.remote() and job.wait() don’t impose an execution timeout by default. For streaming examples and how spawned tasks are cleaned up, see Dev sessions.

Local entrypoints

Mark the function that drives your app with @runpod.local_entrypoint. It runs on your machine and is the entry point for rp flash dev:

@runpod.local_entrypoint
def main():
    print(chat.remote("Why is the sky blue?"))  # Blocks for the result
    job = finetune.spawn(steps=500)  # Returns a Job immediately

App discovery imports your modules synchronously. Keep blocking work inside an entrypoint or a remote function, not at the top level of a module.

Storage

Attach storage with the mounts parameter, which maps a path on the worker to a volume:

• NetworkVolume(name_or_id, size=50, datacenter=None, create=True) references a network volume in a single datacenter. The app chooses a datacenter that works for every resource sharing the volume.
• GlobalVolume(name_or_id, create=True) references global storage that isn’t tied to a datacenter.

Both types look up a volume by name or ID and create it if it doesn’t exist. Set create=False to require an existing volume.

To choose between network and global volumes, see Storage overview.

Mount rules

Mount rules depend on the resource type:

• @app.task supports one network volume and one global volume, mounted at different paths that don’t overlap.
• @app.queue and @app.api support one volume, mounted at /runpod-volume.
• Global volumes require GPU compute, for both tasks and endpoints.

Inside a remote function, volume.path returns the path where the volume is mounted. Read .path inside remote functions, not at the top level of a module. See Storage for volume placement and billing.

Develop and deploy

Use the rp flash commands to iterate on an app and ship it:

rp flash dev main.py  # Live dev session: edit, re-run, logs stream back
rp flash deploy       # Deploy production endpoints

A dev session creates temporary endpoints and deletes them when you press Ctrl-C. To learn more, see Build an app, Dev sessions, and Deploy an app. See the CLI reference for all commands.

Endpoint client

To call a Serverless endpoint that already exists, such as one you deployed from the console or with rp flash deploy, use runpod.Endpoint with the endpoint ID.

import runpod

endpoint = runpod.Endpoint("ENDPOINT_ID")

# Wait for the result
result = endpoint.run_sync({"prompt": "Hello"})
print(result)

# Or submit a job and check on it later
run_request = endpoint.run({"prompt": "Hello"})
print(run_request.status())
print(run_request.output())

By default, the client uses the API key from rp login, RUNPOD_API_KEY, or runpod.api_key. To use a different key for one endpoint, pass it to the constructor: runpod.Endpoint("ENDPOINT_ID", api_key="API_KEY"). A key passed to the constructor takes precedence over the global key.

See Send API requests for more examples.

Serverless worker

If you build your own worker container, use runpod.serverless.start to register the handler that processes jobs:

import runpod

def handler(job):
    job_input = job["input"]
    return {"output": job_input}

runpod.serverless.start({"handler": handler})

To learn how to write handlers, including streaming, progress updates, and error handling, see Handler functions.

API wrapper

The SDK includes functions for managing Runpod resources through the REST API:

import runpod

pods = runpod.get_pods()
pod = runpod.create_pod("test", "runpod/stack", "NVIDIA GeForce RTX 4090")
runpod.stop_pod(pod["id"])
runpod.terminate_pod(pod["id"])

Limitations

• 5th-generation CPU instances (cpu5c) are only available in EU-RO-1. 3rd-generation instances are available in more data centers. See CPU types.
• Apps can scale workers quickly across several endpoints, so you may reach your account’s worker limit. Contact Runpod support to raise it.

Next steps

• Run your first function with the SDK quickstart.
• Build an app with several resource types.
• Configure resources such as GPUs, workers, and storage.
• Migrate from Flash if you have an existing Flash project.
• Learn about the runtime that runs your code on workers.
• Browse the examples in the runpod-python repository.