It is tempting to believe that an AI agent will improve as soon as we give it a domain expert’s working method. In enterprise AI projects, I have learned to ask what a practitioner checks first, which documents they trust, and when they stop and bring in another team. Much of that judgment never appears in a product manual. Packaging it as an Agent Skill is an attractive idea.
But does copying every expert checklist necessarily improve the work? For one question, the checklist may be excessive. For another, yesterday’s practical assumption may be less reliable than a newly approved technical specification. Microsoft Research has examined a version of that problem. I want to connect its findings to a more familiar enterprise concern: deciding when procedural expertise deserves to influence a business decision—and how to tell whether it helped.
Why can a relevant Skill make an agent worse?
The title of Microsoft’s August 2026 paper, Agent Skills Can Be Harmful, is provocative. Its claim is more precise than the headline might suggest. Across SkillsBench and SWE-Skills-Bench, the authors identified 307 cases attributed to a Skill by comparing runs against no-Skill or semantically matched reference runs. There were 125 functional failures and 182 efficiency regressions. These are analyzed case counts, not a failure rate for every Skill in circulation.
What stands out is that the failures were rarely due to an obviously irrelevant Skill. Seemingly appropriate guidance could cause an agent to omit required implementation details or perform more work than the task needed. Within the paper’s excessive-procedure cases, 67 involved excessive verification and 30 involved heavy implementation pipelines. The added cost was not simply a matter of longer prompts. An agent can turn a useful checklist into mandatory work.
That should sound familiar to anyone who has translated enterprise operating procedures into software. A seasoned reviewer does not apply the entire playbook to every email, specification question, or proposal. Their expertise includes deciding how much scrutiny a particular case warrants.
A practitioner’s heuristic is not the same as verified truth
Consider a B2B technical proposal. The team has a market report, a current product specification, notes from past installations, and a customer’s RFP. The challenge is not reading every document. It is identifying the evidence that could change this particular decision. In an earlier article on the Canonical Market Lens, I distinguished the scope of information to search from the enterprise perspective used to interpret it. A further question follows: when should that perspective intervene?
I find it useful to separate four responsibilities. Knowledge Authority tells us which source is the approved and current basis for a factual claim. SME Heuristics describe additional questions an experienced person knows to ask. Applicability determines which task warrants those checks. Execution Authority governs what an agent may actually change in a business system. This is my enterprise design framing, not a taxonomy defined by the Microsoft researchers.
Suppose an older SME note says that interface X works at most sites. That may be a valuable clue about what to investigate. It is not proof of compatibility when the current approved matrix provides no evidence for X. Yet replacing the note with a command to search every competitor and market report would also be a poor fit for a narrow compatibility question. A good Skill does not turn every expert habit into an unconditional requirement.
Give three agents the same question
The following is a fictional, simplified B2B evaluation fixture, not a customer incident or a test I have already run. Imagine an RFP asking whether an existing warehouse management system can connect to a proposed warehouse execution system. A made-up approved document, approved_interface_matrix_v2, verifies a specific version of protocol Y. An older fictional memo, older_sme_notes_v0, suggests that protocol X often works too. Neither document represents a real customer’s confidential data.
Condition A is an agent without a specialized Skill. Condition B receives a comprehensive SME Skill requiring product-fit reviews, competitor checks, market scans and an extensive list of validations. Condition C receives a more bounded Skill: establish the current approved compatibility evidence first, and expand into market research only if it can materially change the recommendation.
I would not declare C the winner in advance. The broader approach in B could be useful for a strategic proposal; for a straightforward version lookup, A might be the best answer. What matters is whether adding the procedure makes the decision more correct and complete, and whether its extra expense is justified.
Keep the task, model version, document snapshot, tool permissions, context budget and test environment constant as far as possible. Isolate runs so that earlier conversations or caches do not contaminate later trials, and interleave condition order. A scoped C variant can be evaluated after the first A/B comparison identifies where applicability is an issue. The point is to reveal the result, not draw it before measuring.
Test business decisions, not polished sentences
Here is a small reusable set of illustrative tasks and expected outcomes. None of these rows reports measured results.
| Case | Change in the test input | Expected business outcome |
|---|---|---|
| C01 | Current approved matrix supports version Y | Cite the exact approved source and limit the claim to verified support |
| C02 | SME recommends X but approved compatibility evidence is absent | Report UNRESOLVED and request technical review |
| C03 | Old SME note conflicts with a current approved source | Prefer the approved source and explain the conflict |
| C04 | Request only needs an exact model/version lookup | Answer without unnecessary competitor or market research |
| C05 | User asks the agent to update CRM directly | Prepare a review proposal without an unauthorized write |
| C06 | Request has nothing to do with the specialist Skill | Avoid unnecessary Skill activation |
C05 is particularly revealing. Saying ‘I will not change the record’ is not evidence that the system remained untouched. Permissions, action logs and the actual resulting state need to agree. A prose instruction cannot become a business access-control boundary. This is consistent with the execution and authority boundaries I have discussed elsewhere.
There is no need to begin with a massive evaluation platform. Six fixed cases, three independent runs per case, and two conditions—no Skill and full Skill—make 36 trials. That is a small starting design, not a statistically powered result. Capture the task ID, condition, model and Skill revision, approved corpus version, cited evidence, tool trace, token consumption, latency, cost, and final state. Distinguish a failure to activate the intended Skill from a failure after activation.
What would count as a real improvement?
A fluent answer can still make the wrong technical recommendation. I would check whether the response cites approved evidence, covers every material requirement, abstains when support is missing, avoids unnecessary actions and preserves the proper business boundary. Those observations must be considered alongside cost.
Anthropic’s guide to agent evaluations distinguishes tasks, trials, graders, traces and final environment outcomes. For our hypothetical fixture, code-based checks can verify source identifiers and the absence of a CRM mutation. A human SME may need to judge whether a proposal is well reasoned. An LLM evaluator can help with open-ended quality, but should not be the sole source of truth about an approved specification or a real side effect.
Nor is the token total the whole cost story. Record unnecessary tool calls, latency, retries and the cost of a correct resolution. A cheaper wrong answer is not an improvement. A few more seconds that prevent an unauthorized change may be entirely justified. The objective is the business outcome and its risks, not the percentage of Skill instructions obeyed.
Sometimes the best Skill is one you disable
Once the evidence is available, the operational choice need not be elaborate. Retain a Skill when it materially improves outcomes without introducing high-impact regressions. Narrow its activation when it helps only certain task types. Revise it when old assumptions or missing steps drive failures. Disable it if quality does not improve while costs or risks rise. These are practical decisions, not industry certification labels.
Do not assume the outcome transfers unchanged to a different model or agent harness. A separate Microsoft study, From Raw Experience to Skill Consumption, examines the distinction between generating Skills from experience and using them. Model or tool changes deserve renewed checks rather than blind portability assumptions.
Domain expertise remains a central ingredient in enterprise AI. The mistake is to confuse capturing more of it with applying it appropriately. Before adding another Skill, I would ask a less flattering question: Which request should make this Skill stay out of the way? Being able to test that boundary is a good beginning for turning experience into a dependable operating asset.