AI

Bullshit Benchmark Explorer

/

BullshitBench evaluates model responses to nonsensical questions, assessing their ability to identify and challenge invalid assumptions. A leaderboards ranks models based on their effectiveness, with Claude Sonnet 4.6 (Anthropic) scoring highest at 94.5% for clear pushback, indicating a strong capacity for detecting nonsense. Other models from various organizations follow, showcasing performance differences in reasoning capabilities across responses to absurd inquiries. An example illustrates the stark contrast between a model that correctly identifies no impact of screw type on food flavor versus another that incorrectly attributes culinary changes to a switch in screws.

https://petergpt.github.io/bullshit-benchmark/viewer/index.html

Threat Modeling AI Applications

/ , , , ,

The post explains how to adapt threat modeling for AI systems, which differ from traditional software in that they produce probabilistic outputs, follow instructions, and have expanded attack surfaces. It recommends explicitly defining what assets the system must protect, understanding real usage patterns, and identifying risks such as prompt injection, misuse of tools, data integrity failures, and harmful outputs. It concludes that AI threat modeling requires structured analysis early in design to assess likelihood and impact and inform architectural mitigations. 

https://www.microsoft.com/en-us/security/blog/2026/02/26/threat-modeling-ai-applications/

2026 State of Software Security: Risky Debt Is Rising, But Your Strategy Starts Here

/ , , ,

2026 State of Software Security Report: Rising security debt affects 82% of organizations, with critical vulnerabilities increasing significantly. A three-step strategy—Prioritize, Protect, Prove—addresses these risks: focus on critical flaws, integrate security in development, and provide evidence of compliance. Organizations must shift from reactive to proactive security management. Download the full report for detailed insights.

https://www.veracode.com/blog/2026-state-of-software-security-report-risky-security-debt/

How Not to Measure the ROI From AI in Your Software Organization

/ , ,

Extreme TLDR: Measuring AI ROI in software requires understanding user diversity and context. Avoid assuming uniformity in usage, effects, or focusing solely on individual performance. Emphasize collective outcomes, account for changes over time, and prioritize thoughtful measurements based on evidence and learning culture.

https://www.fightforthehuman.com/how-not-to-measure-the-roi-from-ai-in-your-software-organization/

Measuring AI Agent Autonomy in Practice Anthropic

/ , , , ,

TLDR: This research examines AI agent autonomy, focusing on Claude Code’s interactions and user behavior. It finds that Claude is increasingly autonomous, working longer without interruptions and auto-approving more frequently as users gain experience. However, experienced users also interrupt more, indicating active oversight. Most agent tasks are low-risk, mainly in software engineering, with limited high-risk applications. Recommendations include enhancing post-deployment monitoring, training AI to recognize uncertainty, and designing for effective user oversight. Overall, autonomy levels are rising amid evolving agent applications.

https://www.anthropic.com/research/measuring-agent-autonomy

Detecting and Mitigating Common Agent Misconfigurations

/ , , ,

The article emphasizes the need to detect and mitigate common agent misconfigurations to enhance security. Agents are increasingly integrated into business workflows, but misconfigurations pose risks, including unauthorized access, data leaks, and unmonitored legacy systems. Key mitigation strategies involve using Copilot Studio for authentication, implementing data policies, conducting regular audits on dormant connections, and restricting actions based on user roles. Overall, effective management and monitoring of agents are crucial for maintaining a secure operational environment.

https://www.microsoft.com/en-us/security/blog/2026/02/12/copilot-studio-agent-security-top-10-risks-detect-prevent/

In the AI Era, CISOs Worry About Data Leaks and Doubt Tech Will Solve Skills Gaps

/ , ,

CISOs recognize the need for AI but express concerns about risks, particularly data leaks and skills gaps. Despite AI’s adoption in security, only mixed results are reported, with many affirming the technology won’t resolve workforce shortages. Key worries include AI model hallucinations and regulatory challenges. Splunk’s report recommends CISOs focus on clear AI governance and collaboration to integrate security into business strategy.

https://www.cybersecuritydive.com/news/in-the-ai-era-cisos-worry-about-data-leaks-and-doubt-tech-will-solve-skill/812964/

How to Get AI Democratization Right

/ , ,

The article discusses AI democratization, emphasizing CIOs’ roles in enabling business users to harness AI responsibly while balancing innovation with governance to prevent security risks and operational inefficiencies. Effective strategies involve fostering AI literacy, establishing governance frameworks, and adapting change management practices to maximize AI integration and impact across organizations.

https://www.cio.com/article/4136302/how-to-get-ai-democratization-right.html

AI Won’t Break Microsoft 365. Your Security Backlog Will

/ , , , ,

TLDR: AI attackers exploit existing configuration backlogs in Microsoft 365, targeting long-neglected security settings rather than zero-day vulnerabilities. With rapid deployment of AI technologies and common misconfigurations across tenants, risks escalate while defenders struggle to keep up, emphasizing the need for immediate action on known security gaps.

https://thehackernews.com/expert-insights/2026/02/ai-wont-break-microsoft-365-your.html

AI Isn’t Failing, People Are Failing With AI

/ , , ,

The article emphasizes that AI failures stem from improper application rather than from the technology itself, highlighting the importance of domain expertise and understanding model operations. It distinguishes between the effectiveness of models like BERT and GPT, advocating for a risk-based framework in deploying AI to manage industry-specific challenges and data utilization. Successful AI transformation relies on organizational fluency with technology and strategic planning.

https://www.cio.com/article/4135361/ai-isnt-failing-people-are-failing-with-ai.html

Scroll to Top