Skip to main content

​
Which models take what

A model’s input_modalities in GET /v1/models lists the input it accepts (text, image, audio, video). The same list is on each model’s page in the dashboard, under Features. A modality that is not listed was either refused when checked or has not been checked on that model (video never is yet); don’t rely on it. The examples below use /v1/chat/completions, which is where image and audio input are checked.

​
Images

Send an image_url content part. The url can be either of two things. A link. A public https URL. The server running the model downloads the image when your request runs:
  • The link must work with no login, cookie or header: the server fetches it as it is.
  • Some hosts refuse requests that don’t come from a browser, and some are slow. If the download fails, the request fails with an error; it is not answered without the image.
  • Downloading from a link is not checked on any model yet. Image input itself is checked with an inline image (below).
Inline, as base64. The image itself, as a data: URL. Nothing is downloaded, so this always works for a model that takes images:
Base64 makes the request about a third larger than the image file. When in doubt, send the image inline.

​
Audio

Send an input_audio content part with the audio as base64 and its format, wav or mp3. There is no link form for audio:

​
Video

Video input is not checked on any model yet. Don’t rely on it.