The AI Specialist Onboarding Tool collapses AI Specialist readiness from weeks of manual scoping into days of guided, validated setup. Point it at your closed-incident history and the tool identifies the specific issue types your L1 AI Specialist can resolve, validates each one against your own KB and catalog content, runs simulated resolutions scored by an LLM judge, and generates the routing criteria that send the right incidents — and only the right incidents — to the AI Specialist in production.
As a by-product, it surfaces where content gaps exist, pointing to the KBs and catalog items that would unlock the next wave of automation.
Deployment guidance: Run this tool on a sub-production instance (a production clone with recent incident, KB, and catalog data). The simulation stage generates evaluation incidents against the AI Specialist, which is not appropriate for production environments. Generated routing criteria can then be promoted to production through your standard update set process.
Here is the documentation on how to use this tool - 2026-Jul-2-Onboarding_tool.docx
1. Ability to identify content coverage and gaps
2. Run the ZTSD evals and followed by LLM judge evaluation of ZTSD response
3. Recommended routing criteria on what types of incidents can be routed to L1 AIS
1. Advanced Filtering — Condition Builder Support
Themes now supports Advanced Filtering in addition to the existing quick filter. You can open the UI16 condition builder, construct an encoded query, and apply it directly to the page. Quick filter remains available — Advanced Filtering is an additional option for more complex filtering needs.
2. Skip Clustering — Go Directly to Evals (Optional, Off by Default)
Enable the property sn_l1s_onboarding.eval.skip_clustering_coverage to bypass the clustering coverage step and move incidents directly into ZTSD evaluation. This is off by default; enable it only if you want to skip clustering analysis for faster evaluation.
3. Improved Cloning Logic — Preserves Original Submission
When evaluations run, incidents are now cloned exactly as the requester originally submitted them, with no modifications. This ensures the evaluation reflects the requester's original input, without any unintended changes.
4. Auto-Generated KB Feedback Tasks (Optional, Off by Default)
Enable the property sn_l1s_onboarding.eval.km_task_creation_enabled to automatically create a KB Feedback task during clustering analysis whenever an incident has no matching KB article. This is off by default; enable it if you want automatic flagging of knowledge base gaps.
5. Apply Routing Criteria from the Review Screen
You can now apply routing criteria directly from the Review screen, without creating a separate update set. Note: routing rules still need to be migrated from sub-production to production through your standard update set process.
NA