1180 words
6 minutes
Being Able to Chat ≠ Being Usable: Pitfalls from Integrating ChatBI with Volcano Engine

Integrating a third-party LLM lately, the four words “OpenAI-compatible” burned me pretty badly.

I was recently hooking a new model up to our team’s ChatBI, and it took several wrong turns before I realized the problem had nothing to do with API keys or networking — it was hiding in a layer of platform changes behind the word “compatible”. The whole debugging session was worth writing down.

What Happened#

ChatBI is our team’s BI analytics product. You ask a question in plain language; it understands the question, picks a dataset, writes SQL, runs the query, and hands the results back as a chart or a table.

In that pipeline, a model that can merely chat is useless. It has to produce, step by step, things the backend can parse — which table it chose, what the SQL looks like, how the fields map. Let it improvise at any point and everything downstream falls apart.

That’s why ChatBI has a “test connection” button when you connect a new model. The name is a bit misleading: it isn’t testing whether the API key works. It’s testing whether the model can do things the ChatBI way. I was connecting through the OpenAI-compatible endpoint of Volcano Engine’s Ark Coding Plan, chose GLM-5.2, hit the button — and face-planted:

The current model does not support the structured output capability ChatBI requires (response_format=json_object). Please switch to a model that supports json_schema or json_object and try again.

What the Test Connection Actually Tests#

I dug through the backend code. The button doesn’t just send a “hello” and see if the model answers. It forces the model through a sequence: first strict json_schema, then structured output with tool calls, and only if both fail does it fall back to json_object.

In other words, “can run a normal chat” and “works with ChatBI” are two entirely different bars. What ChatBI needs is “make the model emit JSON in a fixed format” — not “keep the model company in conversation.”

Ruling Out Configuration First#

Armed with that theory, I reproduced the issue with curl. First, the plainest possible request with no response_format at all — HTTP 200, GLM-5.2 chatted back normally. Base URL, key, model name: all fine.

Then I added the suspect parameter:

"response_format": { "type": "json_object" }

Volcano Engine returned a 400. Here it is verbatim:

{
"error": {
"code": "InvalidParameter",
"message": "The parameter `response_format.type` specified in the request are not valid: `json_object` is not supported by this model. Request id: 0217823740999218c80d243bafb90575ac6ccce9142197dbcaca6",
"param": "response_format.type",
"type": "BadRequest"
}
}

Focus on the message and param fields: the error isn’t “I don’t recognize the response_format parameter” — it’s “I recognize it, but the value json_object isn’t supported by this model.” Completely different direction, and that’s exactly the thread I pulled on.

I Almost Bought “the Model Can’t Do It”#

At this point a perfectly convenient conclusion was sitting right there: GLM-5.2 doesn’t support structured output. Swap the model, go home.

Honestly, I almost bought it. But something felt off — GLM-5.2 is a mainstream LLM, structured output is table stakes for every major model these days. Would it really not support even json_object? Either Volcano Engine had some special restriction, or… I’d concluded too early.

The most dangerous thing in debugging is the first explanation that sounds reasonable: it happens to explain the symptoms, and it happens to lead you astray.

Bypassing Volcano Engine, Hitting the Official API Directly#

Once I suspected the Volcano Engine layer, the most direct verification was: don’t go through it. Connect to the source.

I re-ran the exact same structured request against GLM-5.2’s official endpoint. response_format=json_object went straight through. No problem whatsoever.

Which nailed the conclusion: the model itself is fine. The fault sits in the Volcano Engine layer.

Two More Models, the Same Exact Pattern#

To confirm this wasn’t a GLM-5.2 one-off, I pulled in two more models as controls:

ModelOfficial endpointVolcano Engine (OpenAI-compatible entry)
GLM-5.2Worksjson_object not supported
DeepSeek v4WorksSame error all over again
MiniMax M3WorksWorks

The contrast is telling: GLM-5.2 and DeepSeek v4 both face-planted the moment they entered Volcano Engine, with identical errors; yet MiniMax M3 ran just fine inside it. So Volcano Engine hadn’t killed structured output across the board — it had only modified a subset of models.

The Actual Root Cause#

Put it all together and the conclusion is nailed down:

IMPORTANT

It’s not that the models lack structured-output support, and it’s not a protocol problem. Volcano Engine’s relay layer modified certain models and stripped out response_format structured output on the OpenAI-compatible endpoint. The models’ official APIs all support it — it’s only after passing through Volcano Engine’s gate that part of the capability gets altered.

One more detail points the same way. Volcano Engine’s Coding Plan also exposes an Anthropic-protocol endpoint. Over there, response_format doesn’t exist at all — structured output is done through Tool Use with tools + input_schema. I switched to that endpoint and tested GLM-5.2 again: the model happily emitted structured JSON as instructed. So the structured-output capability wasn’t gutted entirely; only this specific path, the OpenAI entry, had been modified.

What This Means for ChatBI, and What to Do#

The annoying part: ChatBI currently implements only the OpenAI-style provider, and its capability detection is hard-wired to json_schema / json_object. The result: even if a model supports structured output and works over its official API, going through the modified Volcano Engine endpoint means it fails the test and can’t be used.

To support models like these, there are roughly three paths:

OptionApproachUpsideCost
① Stick with OpenAI structured outputOnly allow models/endpoints that pass response_formatNo architecture changesModels modified inside Volcano Engine stay locked out
② Add an Anthropic providerSupport Messages API + input_schema + tool_useBypasses the restriction, keeps schema enforcementRequires config restructuring, new test flows, Agent SDK adaptation
③ Weak JSON modeNo response_format; rely on prompting + backend parsing with retriesSmallest changeUnstable — not something you’d dare ship in a BI product

Direction-wise, ② is the right answer and also the most work; ① is the fallback for now.

Some Parting Thoughts#

A few takeaways, for what they’re worth:

“OpenAI-compatible” runs deeper than it sounds. Compatibility usually covers the plain-chat layer; advanced parameters like response_format and tool_choice getting silently stripped is common enough. Don’t take a doc’s “compatible” as “full-featured”.

For agent-style products, “can chat” is just the entry ticket. What actually gates the door is tool calling and structured output — without those two, no amount of model cleverness gets used.

When you suspect a layer is at fault, bypass it and verify against the source once. If I hadn’t connected to the official API this time, I’d probably have written “GLM-5.2 doesn’t support structured output” into our docs as fact.

Read error messages word by word. “Unknown parameter” and “this value isn’t supported by this model” are two different forks in the road — take the wrong one and you’ll spin in circles forever.

Conclusion#

The root cause had nothing to do with API keys or configuration, and the model wasn’t broken either. Volcano Engine had modified certain models and cut off structured output on its OpenAI-compatible endpoint. The silver lining: the Anthropic door is still open, and the tool_use path works. To properly support models like this in ChatBI, the way forward is adding an Anthropic provider — not loosening the OpenAI-side test.

Hope this saves you a detour the next time you integrate a third-party model.

Being Able to Chat ≠ Being Usable: Pitfalls from Integrating ChatBI with Volcano Engine
https://www.mihouo.com/en/posts/ai/chatbi-接入火山引擎踩坑/
Author
Jia
Published at
2026-06-25
License
CC BY-NC-SA 4.0