Why Did the Same Decision API Produce Different Answers? Question Engineering with Microsoft Decision-1 and JEV

The first test matched the expected labels on just 10 of 17 emails. I initially suspected a difference between the models. Then I opened the adapter code and found that my own implementation was damaging the questions. Fixing it was only the beginning: several business boundaries still needed work.

Why an Azure decision model was welcome news

After building an email classifier with JEV, I could see where a decision model might fit enterprise work. I did not need it to write a customer reply. I needed it to identify the responsible business area and the action being requested.

Adopting that service inside a company raised a separate question. Sending customer emails through a new external model service meant explaining the data path and checking its security and procurement conditions. This was my concern about the adoption process in the environment I was considering, not a finding that JEV was insecure.

I had been waiting for OpenAI’s Decisions API, introduced as a way to focus Luna’s intelligence on predefined questions with finite answers. Then Microsoft Decision-1 appeared in Azure Foundry.

That was welcome news because it added an option to evaluate in companies already using Azure. An Azure listing does not complete a security review: data handling, deployment type, and access controls still need checking. But it changes where that review can begin.

I read Microsoft’s introduction and the Foundry development guide, then decided to port the existing JEV workflow. What began as a model selection exercise quickly became another round of question engineering.

Keeping the policy while changing how questions were sent

The n8n classifier selected a broad business domain first, then evaluated the requested action within it. Level 1 covered Engineering, Sales, Post Sales, Training, and Other. Level 2 handled actions such as parts supply, technical support, commissioning, and repair. A JavaScript resolver assigned the final label.

The original model was jev-1.13.0. For the Microsoft version, I retained the business policy, taxonomy, two-stage flow, and resolver. The model and question adapter changed.

By “the same Decision API” in the title, I mean a shared style of typed, bounded decisions. I am not claiming identical API implementations or model internals. This workflow used choice for the business domain and noul for the detailed conditions. Microsoft also supports score, but that was not the focus of this comparison.

The original JEV instructions used structured fields such as task, decision_rule, and thread_rule. Our Microsoft adapter compiled them into a string. This abbreviated field comparison is not a complete API request:

{
  "jev_excerpt": {
    "instructions": {
      "task": "Select exactly one Level 1 business domain...",
      "decision_rule": "Classify the action the sender expects...",
      "thread_rule": [
        "Prioritize the newest sender-authored message",
        "Use quoted history for unresolved follow-ups"
      ]
    }
  },
  "microsoft_adapter_excerpt": {
    "instructions": "Select exactly one Level 1 business domain...\nClassify the action the sender expects...\nPrioritize the newest sender-authored message\nUse quoted history for unresolved follow-ups"
  }
}

Flattening was our implementation choice. We did not establish that Microsoft cannot interpret nested questions or inherently requires longer prose. Official support for JSON state and compatibility with JEV’s nested question structure are separate questions.

I almost judged the model before checking the request

The first 17-email test matched 10 existing expected labels. My first thought was that the model might be less suited to this workload. Before evaluating that idea, I checked what the workflow actually sent.

Integration had already exposed two unrelated issues. I initially used the catalog model name where the deployment name was required. After correcting that, a Google Sheets locator prevented results from being saved. I verified one email through the call-and-save path before starting the 17-case comparison. API success and classification quality needed separate checks.

Reviewing the question adapter revealed a Level 2 thread_rule bug. The source value was a string, but the adapter spread it as though it were an array.

// Broken: a string is spread into individual characters.
const brokenRules = [...(i.thread_rule || [])];

// Fixed: preserve a string as one rule.
const rules = Array.isArray(i.thread_rule)
  ? i.thread_rule
  : (typeof i.thread_rule === "string" ? [i.thread_rule] : []);

q.instructions = [i.task, i.decision_rule, ...rules]
  .filter(Boolean)
  .join("\n");

In the first run, the parts-handling instructions contained 156 lines, 154 of them single characters. After correction, they returned to three normal lines. I had assumed the model was reading my policy, while my adapter was breaking it before it arrived.

Try 2 reached 11/17. That bug could not explain all the failures. It affected Level 2, while several emails still went to the wrong Level 1 domain.

An unchanged Level 1 request also produced a different judgment on a short follow-up about an earlier parts order and delivery. I could not attribute the entire one-case increase to the serialization fix. Implementation defects and run-to-run variation were both present.

Making quotation and replacement boundaries more explicit

An email asking for a motor or PCB quotation for installed equipment went to general Sales. A request for the latest price on an earlier spare-parts quotation behaved similarly. Both asked about price, but our policy placed installed-equipment parts handling in Post Sales.

The original JEV policy already excluded spare-part quotations from general Sales. In the Microsoft question, I made precedence more explicit: a quotation for components of installed equipment belongs to Post Sales even when the sender uses words such as price, offer, or quotation.

Another email asked whether two product models were compatible or interchangeable. Microsoft routed it toward repair. The requested action, however, was technical confirmation rather than repair performance. I emphasized that distinction in the Engineering boundary.

I was expressing the same policy more explicitly at the points where domains competed. Microsoft Learn recommends adapting state, instructions, and option descriptions to actual workloads and evaluating against labels. That recommendation does not determine which wording will work best for a particular model and dataset.

Three cases improved, and one previously correct case regressed

Try 3 reached 13/17 final matches. The parts-quotation and compatibility cases improved. But a commissioning schedule request that had previously been correct moved into Other. Strengthening some boundaries had disturbed another.

I repeated the same question configuration for Try 4. The result fell to 12/17: the follow-up about an earlier parts order and delivery changed again. A regression after editing questions and variation without editing them are different observations. We did not isolate the cause of the latter, so I did not attribute it to randomness or a service update.

Test	Change or check	Domain matches	Final label matches
Try 1	Initial adapter	13/17	10/17
Try 2	Correct Level 2 string serialization	14/17	11/17
Try 3	Strengthen parts-quotation and compatibility boundaries	16/17	13/17
Try 4	Repeat the same question configuration	15/17	12/17
Try 5	Clarify commissioning and rescheduling	17/17	14/17
Try 6	Distinguish parts supply from repair service	17/17	15/17
Try 7	Repeat the final question configuration	17/17	15/17

These are matches against existing expected labels on the same 17 cases. Try 4 repeated Try 3’s questions, and Try 7 repeated Try 6’s.

For Try 5, I explicitly included initial commissioning and rescheduling in Post Sales. All 17 broad domains then matched the expectations, but final labels were still 14/17. The remaining actionable issue lay within the detailed questions.

Asking for repair parts is different from asking someone to repair

The case I examined most closely asked for a motor and PCB after diagnosis. Its broad domain was correct, but the detailed result was Repair.

The customer’s ultimate goal was to fix the equipment. The immediate action expected from the recipient was to supply parts. We needed to distinguish the person performing the repair from the person receiving this request.

I added the following Level 2 boundary. This is the key excerpt from the instructions recorded in the research notes:

SPARE_PART_HANDLING: If technicians recommended replacing parts
and the sender now asks to RECEIVE THE PARTS, treat the current
requested outcome as Spare Part Handling, not repair service.

REPAIR_RMA: Only mark repair when the recipient is expected
to perform repair/RMA/service replacement. Shipping recommended
components for someone else to install is not the repair service.

The email then moved to Spare Parts. Try 6 reached 15/17, and Try 7 reproduced all 17 final labels from Try 6.

This does not prove that longer instructions caused the improvement. In these cases, results improved after making the actor and requested action more explicit. The effect of length and the effect of boundary wording still need separate experiments.

Why I did not call 15/17 a perfect result

The two mismatches against existing labels concerned on-site support and repair. The earlier JEV and GPT-6 Luna comparison had already identified a need to reconsider those human labels.

Holding those two cases aside leaves 15/15 matches. It does not establish that the human labels are wrong. Microsoft ultimately selected on-site support in one case where JEV selected repair. Microsoft’s final results agreed with JEV on 16/17 cases, not all 17.

The more important limitation is that we used these same emails to tune the questions. Final 15/17 is an in-sample fit, not generalization accuracy on new emails. Two identical final runs are preliminary repeatability evidence, not a guarantee of invariant output.

I read cost and speed within the same limits.

Metric	Historical JEV run	Final Microsoft run
Input tokens across 17 cases	94,996	89,802
Output tokens across 17 cases	4,060	149
Estimated API cost at public list prices	USD 0.003989832	USD 0.003771684
Existing expected-label matches	15/17	15/17

The calculation used USD 0.042 per million input tokens and free output. Microsoft’s price was checked on October 12, 2026; the JEV calculation retains the public-price basis from the earlier research. These are not invoices and exclude discounts, retries, implementation work, and human review. Shorter output is not itself a cost saving when output is free.

The seven Microsoft runs took 2.926–4.353 seconds end to end for the 17-case workflow. The historical JEV run was not measured concurrently with identical instrumentation, so I did not turn those observations into a speed ranking.

The initial parts-quotation errors also carried Sales probabilities of 0.9627 and 0.9521. Routing automatically because a score exceeded 95% would have created a confidently wrong assignment. As in the original JEV implementation, translating that signal into action remains an application responsibility.

One business policy, with validated questions for each model

In an enterprise project, I would manage business policy and model-specific question expression together. That does not require a large router. Versioning the questions and retaining the actual outgoing requests alongside the code is a practical starting point.

Asset	Responsibility
Business policy and taxonomy	Shared classification criteria approved by business owners
Model-specific question contract	Wording and boundaries that express those criteria
Adapter	Serialization, API requests, and typed response validation
Evaluation records	Version-specific outcomes, regressions, and repeat-run variation
Execution policy	Automatic handling, review, abstention, and business mutations

Next, I want to freeze the questions and compare them on an unseen holdout independently labeled by business reviewers. Threads must not overlap between tuning and holdout sets. It should include the parts, repair, compatibility, and commissioning boundaries that caused trouble. Controlled wording experiments are also needed to separate adapter representation from added boundary guidance.

Repeat-run decision flips, high-confidence errors, and calibration remain to be measured. The economic target should include corrections and review time: cost per human-accepted outcome rather than token cost alone. None of those follow-up validations is complete yet.

I am still glad an Azure decision model became available to evaluate. But there was engineering work after the API connected. Were the questions reaching the model intact? Was it applying the intended policy? Did improving one boundary damage another?

That is where Question Engineering proved useful in this experiment: turning business criteria into questions and checking whether those criteria survived into the decisions. We have refined them on 17 familiar emails. The next step is to hold the questions still and ask about emails they have never seen.