Integrating a third-party LLM lately, the four words “OpenAI-compatible” burned me pretty badly.
I was recently hooking a new model up to our team’s ChatBI, and it took several wrong turns before I realized the problem had nothing to do with API keys or networking — it was hiding in a layer of platform changes behind the word “compatible”. The whole debugging session was worth writing down.
What Happened
ChatBI is our team’s BI analytics product. You ask a question in plain language; it understands the question, picks a dataset, writes SQL, runs the query, and hands the results back as a chart or a table.
In that pipeline, a model that can merely chat is useless. It has to produce, step by step, things the backend can parse — which table it chose, what the SQL looks like, how the fields map. Let it improvise at any point and everything downstream falls apart.
That’s why ChatBI has a “test connection” button when you connect a new model. The name is a bit misleading: it isn’t testing whether the API key works. It’s testing whether the model can do things the ChatBI way. I was connecting through the OpenAI-compatible endpoint of Volcano Engine’s Ark Coding Plan, chose GLM-5.2, hit the button — and face-planted:
The current model does not support the structured output capability ChatBI requires (
response_format=json_object). Please switch to a model that supportsjson_schemaorjson_objectand try again.
What the Test Connection Actually Tests
I dug through the backend code. The button doesn’t just send a “hello” and see if the model answers. It forces the model through a sequence: first strict json_schema, then structured output with tool calls, and only if both fail does it fall back to json_object.
In other words, “can run a normal chat” and “works with ChatBI” are two entirely different bars. What ChatBI needs is “make the model emit JSON in a fixed format” — not “keep the model company in conversation.”
Ruling Out Configuration First
Armed with that theory, I reproduced the issue with curl. First, the plainest possible request with no response_format at all — HTTP 200, GLM-5.2 chatted back normally. Base URL, key, model name: all fine.
Then I added the suspect parameter:
"response_format": { "type": "json_object" }Volcano Engine returned a 400. Here it is verbatim:
{ "error": { "code": "InvalidParameter", "message": "The parameter `response_format.type` specified in the request are not valid: `json_object` is not supported by this model. Request id: 0217823740999218c80d243bafb90575ac6ccce9142197dbcaca6", "param": "response_format.type", "type": "BadRequest" }}Focus on the message and param fields: the error isn’t “I don’t recognize the response_format parameter” — it’s “I recognize it, but the value json_object isn’t supported by this model.” Completely different direction, and that’s exactly the thread I pulled on.
I Almost Bought “the Model Can’t Do It”
At this point a perfectly convenient conclusion was sitting right there: GLM-5.2 doesn’t support structured output. Swap the model, go home.
Honestly, I almost bought it. But something felt off — GLM-5.2 is a mainstream LLM, structured output is table stakes for every major model these days. Would it really not support even json_object? Either Volcano Engine had some special restriction, or… I’d concluded too early.
The most dangerous thing in debugging is the first explanation that sounds reasonable: it happens to explain the symptoms, and it happens to lead you astray.
Bypassing Volcano Engine, Hitting the Official API Directly
Once I suspected the Volcano Engine layer, the most direct verification was: don’t go through it. Connect to the source.
I re-ran the exact same structured request against GLM-5.2’s official endpoint. response_format=json_object went straight through. No problem whatsoever.
Which nailed the conclusion: the model itself is fine. The fault sits in the Volcano Engine layer.
Two More Models, the Same Exact Pattern
To confirm this wasn’t a GLM-5.2 one-off, I pulled in two more models as controls:
| Model | Official endpoint | Volcano Engine (OpenAI-compatible entry) |
|---|---|---|
| GLM-5.2 | Works | json_object not supported |
| DeepSeek v4 | Works | Same error all over again |
| MiniMax M3 | Works | Works |
The contrast is telling: GLM-5.2 and DeepSeek v4 both face-planted the moment they entered Volcano Engine, with identical errors; yet MiniMax M3 ran just fine inside it. So Volcano Engine hadn’t killed structured output across the board — it had only modified a subset of models.
The Actual Root Cause
Put it all together and the conclusion is nailed down:
IMPORTANTIt’s not that the models lack structured-output support, and it’s not a protocol problem. Volcano Engine’s relay layer modified certain models and stripped out
response_formatstructured output on the OpenAI-compatible endpoint. The models’ official APIs all support it — it’s only after passing through Volcano Engine’s gate that part of the capability gets altered.
One more detail points the same way. Volcano Engine’s Coding Plan also exposes an Anthropic-protocol endpoint. Over there, response_format doesn’t exist at all — structured output is done through Tool Use with tools + input_schema. I switched to that endpoint and tested GLM-5.2 again: the model happily emitted structured JSON as instructed. So the structured-output capability wasn’t gutted entirely; only this specific path, the OpenAI entry, had been modified.
What This Means for ChatBI, and What to Do
The annoying part: ChatBI currently implements only the OpenAI-style provider, and its capability detection is hard-wired to json_schema / json_object. The result: even if a model supports structured output and works over its official API, going through the modified Volcano Engine endpoint means it fails the test and can’t be used.
To support models like these, there are roughly three paths:
| Option | Approach | Upside | Cost |
|---|---|---|---|
| ① Stick with OpenAI structured output | Only allow models/endpoints that pass response_format | No architecture changes | Models modified inside Volcano Engine stay locked out |
| ② Add an Anthropic provider | Support Messages API + input_schema + tool_use | Bypasses the restriction, keeps schema enforcement | Requires config restructuring, new test flows, Agent SDK adaptation |
| ③ Weak JSON mode | No response_format; rely on prompting + backend parsing with retries | Smallest change | Unstable — not something you’d dare ship in a BI product |
Direction-wise, ② is the right answer and also the most work; ① is the fallback for now.
Some Parting Thoughts
A few takeaways, for what they’re worth:
“OpenAI-compatible” runs deeper than it sounds. Compatibility usually covers the plain-chat layer; advanced parameters like response_format and tool_choice getting silently stripped is common enough. Don’t take a doc’s “compatible” as “full-featured”.
For agent-style products, “can chat” is just the entry ticket. What actually gates the door is tool calling and structured output — without those two, no amount of model cleverness gets used.
When you suspect a layer is at fault, bypass it and verify against the source once. If I hadn’t connected to the official API this time, I’d probably have written “GLM-5.2 doesn’t support structured output” into our docs as fact.
Read error messages word by word. “Unknown parameter” and “this value isn’t supported by this model” are two different forks in the road — take the wrong one and you’ll spin in circles forever.
Conclusion
The root cause had nothing to do with API keys or configuration, and the model wasn’t broken either. Volcano Engine had modified certain models and cut off structured output on its OpenAI-compatible endpoint. The silver lining: the Anthropic door is still open, and the tool_use path works. To properly support models like this in ChatBI, the way forward is adding an Anthropic provider — not loosening the OpenAI-side test.
Hope this saves you a detour the next time you integrate a third-party model.