ChatGPT 
Summary
Key takeaways
- AI in software development introduces risks that go beyond incorrect code, including security vulnerabilities, maintainability problems, review bottlenecks, data exposure, unclear provenance, and excessive agent permissions.
- AI-generated code can look correct while still failing security requirements, which makes automated scanning and human review essential before production deployment.
- The Uvik guide cites Veracode research showing AI-generated code passing security tests only about 55% of the time and introducing vulnerabilities in roughly 45% of tested tasks.
- Faster code generation does not automatically mean faster software delivery because larger pull requests can increase review, testing, debugging, and release bottlenecks.
- Maintainability can deteriorate when developers accept generated solutions without refactoring, leading to duplicated code, inconsistent patterns, and additional technical debt.
- Autonomous coding agents create an additional risk layer because they can modify repositories, call tools, access credentials, or alter persistent state rather than simply suggest code.
- Prompt instructions should not be treated as security controls; repository permissions, environment separation, scoped credentials, approval gates, and rollback mechanisms must enforce boundaries technically.
- Sensitive source code, credentials, customer information, and proprietary data should not be exposed to unapproved AI tools or model providers without clear retention and privacy policies.
- AI output requires provenance and traceability so teams know which changes were generated or substantially modified by AI and can audit them when problems occur.
- AI development risk is best managed as an engineering process involving automated gates, human review, observability, restricted permissions, evaluation, and measurable quality metrics rather than relying on a single security tool.
When this applies
This applies when software teams use AI coding assistants, coding agents, AI-generated testing, automated code review, documentation tools, or autonomous engineering workflows as part of production software development. It is particularly relevant when AI can write or modify application code, interact with repositories, access internal systems, execute tools, or influence deployment workflows. The guidance becomes increasingly important as teams move from occasional developer assistance to AI-driven or AI-native development, where agents receive more context and authority over engineering tasks.
When this does not apply
This does not apply as strongly to low-risk experimentation where AI is used only to explain code, brainstorm ideas, or generate disposable examples that never enter a production repository. It is also not a complete framework for risks associated with customer-facing AI products such as hallucinations, model bias, RAG quality, or agent behavior inside an end-user application, although some security and governance principles overlap. Teams building AI products need additional controls around model evaluation, retrieval quality, user permissions, prompt injection, and production observability.
Checklist
- Define which AI coding tools, models, and agents developers are permitted to use.
- Document what source code and data may or may not be shared with external AI providers.
- Review model-provider retention, privacy, and training policies before granting repository access.
- Prevent secrets, API keys, credentials, and sensitive customer information from entering AI prompts.
- Require normal human code review for every AI-generated production change.
- Run static analysis and linting against AI-generated diffs.
- Run automated security scanning before AI-generated code can be merged.
- Execute relevant unit, integration, regression, and end-to-end tests before deployment.
- Track provenance for code that was generated or substantially modified by AI.
- Keep autonomous agents restricted to explicitly approved repositories and environments.
- Apply least-privilege permissions to every tool and credential available to an agent.
- Require explicit approval for destructive, irreversible, or production-impacting actions.
- Separate development, staging, and production environments using technical controls rather than prompt instructions.
- Monitor defect rates, review time, rework, duplication, security findings, and delivery metrics after introducing AI.
- Maintain tested rollback and recovery procedures for AI-assisted or agent-executed changes.
Common pitfalls
- Assuming code is safe because it compiles, passes basic tests, or looks professionally written.
- Allowing AI-generated code to bypass the same review and security gates required for human-written code.
- Measuring productivity by the volume of generated code instead of completed, reviewed, production-ready work.
- Generating larger pull requests faster than senior engineers can meaningfully review them.
- Accumulating duplicated or inconsistent code because generating another implementation is easier than understanding and reusing the existing one.
- Giving autonomous coding agents broad repository, infrastructure, database, or production permissions.
- Relying on natural-language instructions such as “do not touch production” instead of enforcing environment boundaries technically.
- Sending proprietary source code, credentials, or customer data to unapproved AI services.
- Failing to record which code and actions originated from AI, making incidents and regressions harder to investigate.
- Expanding AI adoption before proving that it improves delivery without increasing defects, security findings, technical debt, or review workload.
Seven failure modes, what each one looks like in production, and the control that actually stops it.
The main risks of AI in software development are hallucinated APIs, drift from project conventions, plausible but incorrect logic, security defects in generated code, unreviewed dependencies, leaked credentials and context, and shadow AI use outside approved tooling. Google’s DORA programme found in 2025 that AI adoption is associated with increased software delivery instability even as throughput improves. Uvik Software mitigates these with automated gates on every AI-generated change, senior human review before merge, and a written tool and data policy agreed with the client before any engagement starts.
The pattern behind the risks
The 2025 DORA State of AI-assisted Software Development report found something specific. AI adoption is now positively associated with delivery throughput, but it remains associated with increased delivery instability: more change failures, more rework, and longer times to resolve issues.
That combination describes the risk precisely. AI is not introducing new categories of defect. It is producing existing categories faster than existing review capacity can absorb them. The bottleneck moved downstream to testing, code review and quality assurance.
Every control below is therefore a review-capacity control. The goal is not to slow AI down. It is to make verification keep pace.
Seven risks and the control for each
| Risk | What it looks like | Control |
|---|---|---|
| Hallucinated APIs | Generated code calls methods, parameters or libraries that do not exist. Fails at runtime rather than at review. | Type checking and static analysis run before a human sees the change. Dependency allowlists. |
| Convention drift | Code works but does not match your patterns. Over months the codebase splits into two styles. | Rules files built from your existing conventions, committed to your repository and applied to every agent session. |
| Plausible but wrong logic | The most expensive failure. Code reads correctly, passes shallow review, and is wrong. Common in edge cases and error handling. | Senior human review. This is the one control with no automated substitute. |
| Security defects | Generated code reproduces insecure patterns from training data: weak input validation, unsafe deserialisation, missing authorisation checks. | Security scanning gates on every generated change, plus a security review step in the definition of done. |
| Unreviewed dependencies | An agent adds a package to solve a problem. Nobody evaluated its licence, maintenance status or supply chain. | Dependency approval gate. No new dependency merges without explicit human approval. |
| Leaked credentials and context | Secrets, environment files or customer data enter a model context, sometimes into a provider with retention enabled. | Secrets never enter any model context under any configuration. Enterprise API tiers with zero data retention. Pre-commit secret scanning. |
| Shadow AI | Engineers use unapproved tools on production code because the approved path is slower or unavailable. | A clear published AI stance. DORA found that a communicated stance changes AI’s effect on effectiveness and throughput. Make the approved path the easiest path. |
Shadow AI is a policy failure, not a discipline failure
Shadow AI is the use of AI tools that the organisation has not approved, on work the organisation is responsible for. In software development it usually means an engineer pasting proprietary code into a consumer chat tool because the approved workflow is slow, restricted or does not exist.
The instinct is to ban and enforce. The evidence points the other way. DORA identified a clear and communicated AI stance as one of the capabilities that changes AI’s effect on organisational performance, and the emphasis is on clarity rather than restriction: policies that encourage experimentation while stating the boundaries.
The practical fix is to make the approved path faster than the unapproved one. If your engineers have enterprise tooling with zero data retention already configured in their editor, the consumer chat tool loses its appeal on its own.
Building a secure AI software development lifecycle
Six controls, in the order they should be implemented. Each one is cheaper to add before adoption scales than after.
- Write the tool and data policy first. Approved tools, approved model providers, retention terms, and what may never enter a model context. Get it signed before any code is touched.
- Configure enterprise tiers with zero data retention. Use VPC-isolated or self-hosted tooling where the compliance posture requires it.
- Put automated gates on generated changes. Static analysis, type checking, security scanning and coverage thresholds run before human review, not after.
- Require human review on 100 percent of AI-assisted changes. No autonomous merges. This is the control that catches plausible but wrong logic, and it is seniority-dependent.
- Add a dependency approval gate. No new package merges without a human decision on licence, maintenance and supply chain.
- Log and audit. Record which changes were AI-assisted so that post-incident analysis can distinguish generated code from hand-written code.
Uvik Software applies all six by default. No AI tool touches client code without written consent covering the tool list, the model providers and the data retention terms, and secrets never enter a model context under any configuration.
What to put in a vendor contract
If a development partner uses AI on your code, six clauses are worth insisting on.
- An approved tool and model provider list, changeable only with your written consent.
- A data retention term stating that your code is not used for model training and is not retained by the provider.
- A commitment to human review on all AI-assisted changes, with a stated seniority level for the reviewer.
- Full intellectual property assignment covering not only delivered code but also the workflow artefacts: rules files, test generators and generated documentation.
- Agreed delivery KPIs, including change failure rate, reported against rather than benchmarked.
- A requirement that AI-assisted changes are identifiable in version control for later audit.
Uvik Software provides all six as standard, with GDPR-aware delivery by default, EU data residency, and security documentation available under NDA.
Where to go from here
If you need these controls applied to an existing codebase, the relevant page is AI-augmented software development. If the risk you are managing sits in a production AI system rather than in the development process, LLM evaluation and observability covers monitoring, drift detection and guardrails.
FAQ
What are the risks of using AI in software development?
The main risks are hallucinated APIs that fail at runtime, drift from project conventions, plausible but incorrect logic that passes shallow review, security defects reproduced from training data, unreviewed dependencies, leaked credentials or context, and shadow AI use outside approved tooling. Google's DORA programme found in 2025 that AI adoption is associated with increased delivery instability even as throughput improves.
Does AI-generated code introduce security vulnerabilities?
It can. Generated code reproduces patterns from training data, which include insecure ones such as weak input validation, unsafe deserialisation and missing authorisation checks. The control is automated security scanning on every AI-generated change before human review, combined with a security review step in the definition of done. Uvik Software runs both by default.
What is shadow AI in software development?
Shadow AI is the use of AI tools the organisation has not approved, on work the organisation is responsible for. In software development it usually means pasting proprietary code into a consumer chat tool because the approved workflow is slow or does not exist. The fix is a clear published AI stance plus an approved path that is faster than the unapproved one.
How do you build a secure AI software development lifecycle?
Six controls, in order: write the tool and data policy before any code is touched, configure enterprise tiers with zero data retention, put automated gates on generated changes before human review, require human review on all AI-assisted changes with no autonomous merges, add a dependency approval gate, and log which changes were AI-assisted for later audit. Uvik Software applies all six by default.
What is AI governance in the software development lifecycle?
AI governance in the software development lifecycle is the set of written policies and automated controls that determine which AI tools may touch code, what data they may see, which checks run on generated output, and who approves it before merge. It is distinct from model governance, which concerns AI systems in production. Uvik Software agrees the policy in writing with the client before an engagement starts.
How do you stop AI from creating technical debt?
Three controls handle most of it. Rules files built from your existing conventions keep generated output inside your patterns. Automated gates catch regressions before a human reviews the change. Senior human review catches plausible but incorrect logic, which has no automated substitute. Uvik Software staffs only engineers with 7 to 14 years of production experience because verification is a seniority-dependent skill.
