OpenAI has fully opened its Decisions API model to users for the first time, formally entering the fast, low-cost AI decision-making market.
Recently, TypeSafe, a startup founded by a researcher at a leading AI lab, has achieved success in this market with its Jev model, reaching a valuation of $7.6 billion.
TypeSafe's Jev model pioneered a new class of deep learning models that act as "super classifiers." These models can rapidly make a choice from a given set of options and provide a confidence score for the decision outcome.
Software developers quickly discovered that these technologies, known as "decision models" or "System One" models, can replace large language models (LLMs) in a variety of workflows, running faster and at significantly lower cost.
Nikunj Handa, product lead for the OpenAI API, told TechCrunch that after seeing Jev, his colleagues exclaimed, "Wow, we literally have a million use cases where we could use this." Handa's team then accelerated development of their own version.
However, rather than building a purpose-built decision model calibrated for confidence from scratch, they chose to reconfigure the ChatGPT Luna model to improve inference speed and reduce costs.
Handa said, "The biggest breakthrough with the Decisions API is speed. We reduced tasks that used to take 1 to 3 seconds down to 100, 120, or even 180 milliseconds."
"In the future, we may further improve the model's performance in these areas through specialized training. But for now, the standard Luna model is already capable of getting the job done."
OpenAI announced the Decisions API at its developer conference on September 29 and opened it to all customers on Thursday.
In the public version, OpenAI added support for voice and vision modalities, and is working to reduce access latency for users around the world by deploying inference resources across different geographic regions.
In early tests with customers including Box, Datadog, Ramp, and Speak, engineers used the API to replace OpenAI's original large language models to achieve higher throughput.
Andrew Hsu, chief technology officer of AI language-learning app Speak, said the Decisions API helped his company cut response time between actions from 1.4 seconds to 0.13 seconds.
OpenAI also used this technology to improve the response speed of its Dots voice assistant.
Still, there is no clear standard for what makes a good decision model. Beyond "being able to meet user needs," the optimal balance among intelligence, cost, confidence calibration, and speed appears to vary by workload.
TypeSafe CEO Diogo Almeida is skeptical of benchmarks and encourages users to test models in real-world work. But even TypeSafe published its own model leaderboard last week alongside the latest version of Jev.
The latest Jev version added several features, including image support and smart-if statements.
Public benchmarks such as the AIM System One Index and the Artificial Analysis Decision Index tend to favor Liquid's newly launched decision model D1 Ultra and Base Lab's Elephant-1.
The latest Jev model remains competitive on those two leaderboards, ranking seventh and third, respectively; by comparison, OpenAI's Decision Model ranks eighth and fourteenth.
For now, however, TypeSafe's model remains the most widely queried decision model on the OpenRouter platform.
The most interesting Jev benchmark is JevMan. This test requires decision models to play the classic game Pac-Man, and the latest Jev model ranks first on that leaderboard.
Felix Wunderlich, chief growth officer of Opper AI, which developed the benchmark, said the classic game "is a fun test because players have to make rapid, make-or-break decisions continuously. At every fork in the road, the model has to decide its direction within a few hundred milliseconds."
OpenAI's model can be said to have stronger overall capabilities, and its potential advantage may lie in multimodal processing.
Handa told TechCrunch that in image classification, "Luna is extremely powerful and shows better confidence calibration." He added that OpenAI customers also want the Decisions API to be more flexible in using images as output.
Handa believes further improvements in inference speed will also give rise to new use cases. He said, "Can we reduce latency to around 60 milliseconds? If we can do that, you can imagine an API call pattern like calling the API once for every frame in a video."
As these models become faster and cheaper, we can expect them to confirm the theory of the economist after whom Jev is named: when a resource becomes more efficient to use, its applications become more widespread.
In other words, as Handa sees it, decision models will give rise to more and more "unimaginable use cases." These scenarios have actually been right in front of us all along; it is only after the emergence of decision models that people have truly discovered their possibilities.
Comments