HCLTech BLA-Bench evaluation across 782 coding-agent tasks

At a glance Nemotron 3.5 Super VL earned reward on 452 of 782 business logic abuse tasks (57.8%), versus 359 of 782 (45.9%) for OpenAI o4-mini. It also recorded higher detection reward (51.2% versus 40.7%) and remediation reward (64.5% versus 51.2%).

Business logic abuse is among the hardest application security problems to detect. Unlike memory-safety defects or obvious injection paths, these vulnerabilities arise when an application’s intended workflow, authorization rules, or state transitions can be manipulated in ways the developers did not anticipate.

HCLTech Applied Research evaluated the NVIDIA Nemotron 3.5 Super VL model on BLA-Bench, an internal benchmark designed to assess whether coding agents can detect and remediate business-logic vulnerabilities in realistic software projects. In this evaluation, the Nemotron configuration earned higher reward overall, in both detection and remediation, than the o4-mini reference configuration.

The evaluation placed both configured models inside the same Deep Agents Code harness from LangChain with a matched the runtime profile. Each task supplied the CWE class and a scoped investigation region. No benchmark-specific examples, security retrieval layer, or specialized validation tool was added to the stock agent.

The category and language results were broad: Nemotron recorded a higher reward rate in all 10 Business Logic Abuse categories and in 16 of 17 programming-language cohorts.

Why business logic abuse is a challenging test for AI coding agents

Traditional application security tools are highly effective at finding many classes of vulnerabilities with identifiable code or data-flow patterns. Business logic vulnerabilities present a different challenge.

Consider an application that correctly validates every individual API request but allows users to invoke those APIs in an unintended sequence. Or a service that correctly implements authorization checks but exposes an internal state transition that an attacker can manipulate. The individual pieces of code may appear reasonable in isolation. Detecting the vulnerability requires reasoning about what the application is intended to do and how an attacker can violate those assumptions.

To evaluate this problem systematically, HCLTech is developing BLA-Bench, an internal benchmark organized using the 2025 taxonomy from the OWASP Top 10 for Business Logic Abuse incubator project. The version used in this evaluation contains 391 curated challenges derived from real open-source repositories with publicly disclosed and remediated vulnerabilities. The vulnerabilities were selected to be relatively recent to reduce model-saturation risk and cover all ten taxonomy categories.

The benchmark spans 17 programming languages, including C, C++, C#, Java, JavaScript, TypeScript, Python, Go, Rust, Kotlin, Scala, Erlang, Elixir, PHP, Perl, Ruby and Dart.

Each challenge produces two separately evaluated tasks. For detection, the agent receives the vulnerable repository, the CWE class and a search scope, then writes the findings report. The detection verifier requires a finding with the correct file, CWE and accepted line range; it also rejects uncredited findings for the target CWE and limits the number of reported findings. For remediation, the agent receives the vulnerable file and region and edits the repository; all required static checks and behavior tests must pass. This produces 782 binary-reward tasks: 391 detection and 391 remediation.

BLA-Bench is intentionally difficult for a general coding agent. The task prompt identifies the CWE class and a scoped investigation region. Still, the stock agent has no benchmark-specific examples, a Business Logic Abuse category tutorial, a security retrieval layer, or a specialized validation tool. Localization and remediation, therefore, depend on reconstructing the vulnerable invariant from the repository and the behavior checked by the task verifier.

Evaluating Nemotron 3.5 Super VL in a coding agent with BLA-Bench

For the comparison covered in this blog, we use the original Deep Agents Code from LangChain with the OpenAI o4-mini as the reference agent. We use OpenAI o4-mini as a closed-weight, cost-efficient reasoning reference that is believed to be comparable size to the Nemotron 3.5 Super VL model. The objective extends beyond selecting agent models: it is to build a security-specialized coding agent and to continuously improve its detection and remediation quality, as well as its operational efficiency.

• Both configurations ran the same 782 task identities: 391 detection tasks and 391 remediation tasks.
• Both used the stock Deep Agents Code from LangChain, without extra security-specialized tools.
• The configured models were NVIDIA Nemotron-3.5-Super-VL-120B-A12B and OpenAI o4-mini-2025-04-16.
• Both used Harbor, the containerized agent evaluation framework, with the same Deep Agents adapter and runtime profile.
• The results, therefore, describe the complete configured systems on this benchmark cohort, not isolated model weights.
• For more evaluation info, see details below.

Higher overall, detection and remediation reward with Nemotron

Figure 1. Observed overall, detection and remediation reward rates. Higher is better.

Nemotron earned reward on 452 of 782 tasks (57.8%), compared with 359 of 782 (45.9%) for o4-mini, a difference of 93 rewarded tasks and 11.9 percentage points. Because the task identities are paired, the approximate 95% confidence interval for the overall difference is 8.4 to 15.3 percentage points. 

The task-level results favor Nemotron across both parts of the benchmark. Detection reward was 51.2% (200 of 391), versus 40.7% (159 of 391) for o4-mini. Remediation reward was 64.5% (252 of 391), versus 51.2% (200 of 391). The remediation difference was 13.3 percentage points, compared with 10.5 points for detection.

On this benchmark outcome, the difference corresponds to 41 additional detection tasks and 52 additional remediation tasks receiving full verifier reward. For an application-security team, that can mean the Nemotron configuration produced actionable localization and verifiable remediation in more cases. Note that these are benchmark outcomes, not estimates of production vulnerabilities prevented, triage hours saved, or patches accepted without review.

Execution diagnostics show lower error markers and fewer agent exceptions

The serialized tool results show lower explicit error-marker rates with Nemotron on each selected stock tool of the agent in Figure 2. The largest differences appeared for “execute” (6.6% versus 21.6%) and “ls” (0.7% versus 15.5%); “edit_file” remained the highest-friction selected tool for both configurations (40.5% versus 46.0%).

Figure 2. Explicit error-marker rate for selected serialized tool results, ordered by Nemotron rate from lowest to highest. Lower is better.

Across all analyzed serialized tool results, 1,191 of 21,278 Nemotron results (5.6%) contained an explicit error marker, compared with 831 of 8,015 o4-mini results (10.4%). The saved results recorded agent exceptions in 22 of 782 Nemotron trials (2.8%) and 32 of 782 o4-mini trials (4.1%).

Performance varies across business logic abuse categories

Nemotron recorded a higher reward rate than o4-mini in all 10 Business Logic Abuse categories. The largest observed differences were in BLA6, Missing Transition Validation (67.4% versus 45.3%); BLA8, Internal State Disclosure (59.5% versus 39.2%); and BLA10, Shadow Function Abuse (60.7% versus 42.9%).

Nemotron’s highest category reward rates were BLA6 at 67.4% (58 of 86), BLA10 at 60.7% (34 of 56), BLA3 at 60.5% (46 of 76) and BLA5 at 60.3% (41 of 68).

Figure 3. Observed reward by Business Logic Abuse category for Nemotron 3.5 Super VL and o4-mini, sorted by Nemotron reward from highest to lowest. Higher is better.

The lowest Nemotron category reward rates were BLA7, Resource Quota Violation (48.0%, 48 of 100) and BLA1, Action Limit Overrun (50.0%, 20 of 40). Together, they account for 140 of the 782 tasks, making them concrete follow-up targets for better localization, verifier-guided validation and category-specific evaluation.

Higher reward in 16 of 17 languages

Nemotron led in 16 of 17 languages. Kotlin was 80.0% versus 70.0% (16 of 20 versus 14 of 20), a 10.0-point gap, smaller than Scala (25.0), C# (20.8), Rust (20.0) and C++ (16.7). Dart was o4-mini’s only lead (80.0% versus 60.0%; 10 tasks).

Figure 4. Observed reward across all 17 programming languages for Nemotron 3.5 Super VL and o4-mini, sorted by Nemotron reward from highest to lowest. Higher is better.

How to use this result
Use the language view as a coverage and prioritization map. Nemotron led in 16 of 17 cohorts; future regression tests should preserve that breadth while targeting lower-performing slices.

From baseline evaluation to a continuously improving security agent

The evaluation provides a baseline for iterative agent improvement. Its most useful value is diagnostic: BLA7 and BLA1 remain lower-performing category slices; explicit tool-result error markers remain concentrated in edit_file and the trajectories expose opportunities to reduce redundant steps and token consumption. The two configurations also produced materially different tool-result volumes: 21,278 serialized results for Nemotron and 8,015 for o4-mini, about 2.65 times as many. That system-level difference is a concrete target for agent-harness and model optimization and for reporting steps, tokens and end-to-end wall time alongside reward.

The reward was higher for the Nemotron configuration, but efficiency was not. Across the 782 tasks, the Harbor artifacts report 567.8 million input-plus-output tokens for Nemotron and 147.1 million for o4-mini. Median agent execution time was 106.1 seconds versus 62.0 seconds and median end-to-end trial wall time was 615.6 seconds versus 311.3 seconds. Provider token and cache accounting differ, so these totals are descriptive rather than a normalized billing comparison.

Some next steps for the agent:

• Train and evaluate detection before remediation so the patch is grounded in a correctly localized security invariant.
• Use retained successful and failed trajectories to optimize tool choice, edit robustness, validation and termination, not only final answers.
• Improve agent trajectory efficiency by detecting repeated file and tool cycles, requiring measurable validation progress and stopping or escalating when additional iterations produce no new evidence.
• Improve token efficiency through context compaction, selective history retention and limits on repeated repository reads and large tool results.
• Strengthen lower-performing slices. Prioritize BLA7 and BLA1, improve detection localization and false-positive control and re-evaluate the small Dart cohort before drawing language-specific conclusions.
• Conduct controlled adapter and inference ablations before full benchmarks, including model-native reasoning controls, reasoning budgets, output budgets and other supported generation arguments.
• Re-evaluate the same verified cohort after each model or agent harness change, then test generalization on newer held-out vulnerabilities.

This loop is the practical value of BLA-Bench. It turns a difficult application-security problem into a measurable system-development program: evaluate, retain evidence, diagnose model and agent-harness behavior, optimize and verify on unchanged and held-out tasks.

Toward dependable agentic vulnerability remediation

On this 782-task BLA-Bench cohort, the Nemotron configuration produced higher observed reward overall and in both detection and remediation. It led o4-mini in all 10 Business Logic Abuse categories and 16 of 17 language cohorts.

For HCLTech and HCLSoftware, these results establish a baseline for further evaluation. The path to dependable agentic remediation is a security-focused model and agent harness that improve together, with trajectories grounded in executable evidence and each optimization measured against the same security outcomes.

Explore and try NVIDIA Nemotron 3.5 Super VL for your own agents. To discuss security-specialized coding agents and evaluation, contact HCLTech.

Benchmark and evaluation details

• Benchmark: internal HCLTech BLA-Bench evaluation; 391 cases and 782 tasks per model.
• Task scoring: binary reward, 1 only when all checks in a task verifier pass and 0 otherwise.
• Each task had a 4 GiB Docker memory limit.
• Benchmark coverage: all 10 categories in the 2025 OWASP Top 10 for Business Logic Abuse incubator-project taxonomy and 17 programming languages.
• Evaluation design: each task had one scored attempt. The overall comparison is paired across the same 782 task identities.
• Agent: LangChain Deep Agents Code and matched runtime profile.
• Models: NVIDIA Nemotron-3.5-Super-VL-120B-A12B and OpenAI o4-mini-2025-04-16.
• Model configs: Both configurations set a 600-second model/API request timeout per call and max_completion_tokens of 16,384.
• Model configs: Each model used its provider-recommended reasoning configuration. Nemotron: thinking enabled, reasoning_budget 8,192 tokens, temperature 1.0, top_p 0.95. o4-mini: reasoning_effort medium, temperature 1.0.
• Subgroup interpretation: Category and language slices are descriptive and should be read with their cohort sizes.
• Tool-result diagnostics: An explicit error marker is a serialized tool result with a line beginning error, exception, traceback, or failed; this is a text diagnostic, not a measure of semantic correctness or production reliability.
• Evidence: Harbor configuration files, per-trial result.json files, serialized agent result files and trajectories and exact task identities joined to the BLA-Bench task metadata for category and language slices.