Template draft. Replace everything in [square brackets] with your own decision,
delete what does not describe what you really do, and approve it through your own document control. It is not legal
advice and does not make an organisation compliant or certified.
1. Introduction
A new or changed service can look finished to the team that built it and still be unsafe to run. Evaluation is the independent check, made before release, that the service meets the requirements agreed at design and that the Organisation and its customers can operate it. This statement fixes the criteria and the rules for judging them, so that a decision to release is based on evidence and not on the pressure of a date.
2. Purpose
The purpose is to define how new and changed services are evaluated before release into the live environment: the criteria, the sample tests, the rules that decide pass, conditional pass or fail, and who signs off.
3. Scope
- In scope: every new service, every change to a service that adds, removes or alters functionality, service levels, support arrangements, hosting, suppliers or security controls, and the planned retirement of a service.
- In scope: services built by the Organisation, bought from suppliers and delivered under a customer contract.
- Applies to the evaluation gate in the Service Design and Transition Procedure and the go-live decision in the Release and Deployment Management Procedure.
- Out of scope: standard changes pre-approved in the Change Management Procedure, which carry their own tests; and emergency changes, which are evaluated after the event under the emergency change rules.
4. Terms and Definitions
| Term | Meaning |
|---|
| Evaluation | A structured check of a service against the criteria in this statement, with recorded evidence. |
| Criterion | A stated condition that the service must meet, with a way to test it. |
| Acceptance test | A test performed by or with the customer or service owner to show that the service does what was agreed. |
| Operational acceptance | Confirmation by the support teams that they can monitor, support, recover and change the service. |
| Blocking defect | A defect that prevents a mandatory criterion from being met or leaves a high risk open. |
| Conditional pass | A decision to release with named open items, owners and dates, agreed by the sign-off authority. |
| Evaluation report | The record of criteria, tests, results, open items and decision for one service or change. |
5. Roles and Responsibilities
| Role | Responsibilities |
|---|
| Service owner | Sets the service requirements, takes part in acceptance testing, accepts the evaluation result and signs for the business. |
| Process owner (Head of Service Management) | Owns the criteria, appoints the evaluator, resolves disagreement on a result and reports evaluation measures. |
| Evaluator | Plans and runs the evaluation, collects evidence, writes the report and makes a recommendation. Is independent of the build team. |
| Release manager | Provides the release package and test environment and confirms release readiness. |
| Change authority | Makes the release decision with the evaluation report in front of it. |
| Service desk and resolver groups | Confirm operational acceptance: knowledge articles, routing, monitoring, runbooks and training. |
| Information security representative | Confirms the security criteria and the outcome of security testing. |
| Customer representative | Takes part in acceptance testing where the customer agreement requires it and signs the customer acceptance. |
| Activity | Responsible | Accountable | Consulted | Informed |
|---|
| Set service requirements | Service owner | Service owner | Customer representative, Evaluator | Service desk |
| Plan the evaluation | Evaluator | Head of Service Management | Service owner, Release manager | Change authority |
| Run tests and collect evidence | Evaluator | Head of Service Management | Build team, Resolver groups | Service owner |
| Confirm operational acceptance | Service desk | Head of Service Management | Resolver groups | Service owner |
| Confirm security criteria | Information security representative | Head of Service Management | Evaluator | Service owner |
| Sign off the evaluation | Service owner | Change authority | Evaluator, Customer representative | Release manager |
6. Evaluation Criteria
Each criterion has a class. Mandatory criteria must be met for any release. Important criteria may be released with a conditional pass. Advisory criteria are recorded and fed to improvement. The evaluator may add criteria to suit the service, but may not remove a mandatory one without written approval from the Head of Service Management.
| Area | Criterion | Class | Evidence | Related document |
|---|
| Requirements | Every agreed service requirement is traced to a test with a result | Mandatory | Requirements trace table | Service Design and Transition Procedure |
| Functionality | Agreed functions work as specified for the main user journeys | Mandatory | Test results with defect log | Release and Deployment Management Procedure |
| Service levels | Targets for availability, response and resolution are stated and measurable, and the monitoring needed to measure them is live | Mandatory | Draft service level agreement, monitoring check | Service Level Management Procedure |
| Capacity | Expected volume for the first [12 months] is supported with headroom of at least [30 percent] in load test | Important | Load test report | Capacity and Demand Management Procedure |
| Availability and continuity | Recovery time and recovery point objectives are defined, and a recovery test has been done | Mandatory | Recovery test record | Availability and Service Continuity Management Procedure |
| Security | Security requirements met; no open high or critical vulnerability; access controls and logging tested | Mandatory | Security test report, access review | Information Security Management for Services |
| Data protection | Personal data use is identified, the lawful basis is recorded and the privacy assessment is complete | Mandatory | Privacy assessment | Data Protection and Privacy Procedure |
| Support readiness | Knowledge articles, runbooks, routing, escalation and monitoring alerts are ready; support staff trained | Mandatory | Operational acceptance checklist | Service Operating Model |
| Catalogue and communication | Service is entered in the catalogue; notices to users are ready | Important | Catalogue entry, notice draft | Service Catalogue Management Procedure |
| Suppliers | Supplier contracts, service levels and contacts for the service are in place | Important | Contract record | Supplier Management Procedure |
| Configuration | Configuration items are recorded and linked to the service | Important | Configuration record check | Configuration and Asset Management Procedure |
| Cost and charging | Budget, charges and cost model are approved | Advisory | Approved cost model | Service Reporting and Continual Improvement Procedure |
| Rollback | Backout steps are written and have been tested | Mandatory | Backout test record | Change Management Procedure |
| Usability | Users in the sample complete the main tasks without help in at least [90 percent] of tries | Advisory | User test notes | Service Design and Transition Procedure |
7. Sample Test Cases
The table gives a starting set. The evaluator selects, adapts and extends it for each service and keeps the final set with the evaluation report.
| Ref | Test | Method | Expected result | Criterion |
|---|
| TC-01 | Run each main user journey end to end in the evaluation environment | Scripted test by tester and a user | Journey completes with no blocking defect | Functionality |
| TC-02 | Raise a test incident through each support channel | Test ticket created by the Service desk | Ticket routes to the correct group within [10 minutes] | Support readiness |
| TC-03 | Break a monitored component and watch for an alert | Controlled fault | Alert reaches the on-call person within [5 minutes] | Support readiness |
| TC-04 | Apply load at [130 percent] of the expected peak for [1 hour] | Load test tool | Response time within target, no errors above [0.5 percent] | Capacity |
| TC-05 | Restore the service from backup in the recovery environment | Recovery drill | Restored inside the recovery time objective with data loss within the recovery point objective | Availability and continuity |
| TC-06 | Try to access the service as a user with no rights and as a user with the wrong role | Access tests | Access refused and logged | Security |
| TC-07 | Run the vulnerability scan and review | Scanner and manual review | No open high or critical finding | Security |
| TC-08 | Apply the release and then execute the backout | Rehearsal in the test environment | Service returns to previous state within [30 minutes] | Rollback |
| TC-09 | Check that service levels can be measured from the monitoring data | Report dry run | Report for a test period reproduces the targets | Service levels |
| TC-10 | Walk the Service desk through the knowledge articles with a sample user call | Desk walk-through | Agent resolves or routes correctly in the time stated | Support readiness |
8. Pass and Fail Rules
| Result | Rule | Next step |
|---|
| Pass | All mandatory criteria met; all important criteria met or covered by a recorded risk acceptance; no blocking defect open | Evaluator recommends release; sign-off follows |
| Conditional pass | All mandatory criteria met; one or more important criteria open with an owner and a date no later than [30 days] after release; no blocking defect | Release allowed only with the approval of the Change authority; open items tracked to closure |
| Fail | Any mandatory criterion not met, or any blocking defect, or an untested backout | No release; defects returned to the build team; evaluation repeated on the criteria that failed |
| Deferred | Evidence not complete by the date of the release decision | Release date moves; the evaluator reports the missing evidence to the Service owner |
- A fail may not be overridden by the build team or the sponsor. Only the Change authority may release over a fail, and only with the written risk acceptance of the Service owner and the Information security representative, which is recorded in the risk register.
- A re-evaluation covers every failed criterion and any area touched by the fix. A fix to one defect that changes other functions requires a [regression test] of the affected journeys.
- More than [2] failed evaluations for the same release are reported to the Head of Service Management, who reviews the plan.
- Defects found are classified by the evaluator: blocking, major or minor. Minor defects may pass with an owner and a date.
9. Evaluation Process
9.1 Inputs and outputs
| Inputs | Outputs |
|---|
| Service requirements and design | Evaluation plan |
| Release package and test environment | Evaluation report with results and recommendation |
| Draft service level agreement, catalogue entry, knowledge articles | Defect list and open-item register |
| Security and privacy assessments | Signed acceptance and release recommendation |
9.2 Steps
- PlanThe evaluator reads the requirements and design, selects the criteria and test cases, and agrees the plan and dates with the Service owner. Target: [5 business days] before test start.
- PrepareThe Release manager provides a test environment that matches the live environment and the release package. The Service desk prepares test tickets. Target: ready on test start.
- TestThe evaluator runs the tests and records results and evidence. Defects are raised in the ticketing tool. Target: within the window in the plan, normally [10 business days].
- Operational acceptanceService desk and resolver groups confirm knowledge, routing, monitoring and training with the checklist in Annexure B. Target: [3 business days] after test.
- JudgeThe evaluator applies the pass and fail rules and writes the report in Annexure A with a recommendation. Target: [2 business days].
- Sign offThe Service owner, Information security representative, Head of Service Management and, where required, the Customer representative sign. The Change authority makes the release decision. Target: [2 business days].
- Follow upThe Release manager tracks open items. After [30 days] in live use, the evaluator checks the service against the criteria and reports to the Service owner.
9.3 Process flow
- The Service owner requests evaluation at design review or at the evaluation gate.
- The evaluator plans and agrees the criteria and tests.
- The Release manager provides the environment and the package.
- The evaluator runs tests and the support teams confirm operational acceptance.
- The evaluator applies the rules and recommends pass, conditional pass or fail.
- If fail, defects are fixed and the failed criteria are re-evaluated.
- If pass or conditional pass, sign-off is collected and the Change authority decides on release.
- After release, the post-release check is made and the report is filed.
10. Sign-off
Sign-off records that the named people have seen the evaluation report and accept the result. The required signers depend on the type of release.
| Release type | Signers | Authority to release |
|---|
| New service | Service owner, Head of Service Management, Information security representative, Customer representative if contracted | Change authority; Top management informed |
| Major change to a service | Service owner, Head of Service Management, Information security representative | Change authority |
| Minor change | Service owner, Evaluator | Change authority or delegate |
| Retirement of a service | Service owner, Customer representative, Supplier manager | Change authority |
| Conditional pass | As above, plus owner of each open item | Change authority with a recorded risk acceptance |
Note. A signature without evidence behind it does not count. The evaluator may refuse to present a report until its evidence is attached.
11. Interfaces with Other Processes
| Process | Interface |
|---|
| Service Design and Transition Procedure | Sets requirements and the evaluation gate; receives the evaluation result. |
| Release and Deployment Management Procedure | Supplies the package and environment; acts on the go-live decision. |
| Change Management Procedure | Change authority makes the release decision using the report. |
| Service Catalogue Management Procedure | New and changed services are added to the catalogue only after a pass. |
| Service Level Management Procedure | Service levels are agreed and measurable before release. |
| Information Security Management for Services | Supplies the security criteria and tests. |
12. Key Performance Indicators
| Measure | Target | Source | Frequency |
|---|
| Releases with a signed evaluation report | [100 percent] | Release records | Monthly |
| Releases passing first evaluation | [80 percent] | Evaluation reports | Quarterly |
| Conditional-pass items closed by their date | [95 percent] | Open-item register | Monthly |
| Incidents in the first [30 days] caused by an escaped defect | [no more than 2 per release] | Incident records | Per release |
| Evaluations completed within the plan dates | [90 percent] | Evaluation plans | Quarterly |
13. Records and Evidence
| Record | Kept by | Retention |
|---|
| Evaluation plan and report with evidence | Evaluator | [Life of the service plus 3 years] |
| Test results and defect log | Evaluator | [3 years] |
| Signed acceptance forms | Release manager | [Life of the service plus 3 years] |
| Open-item register | Release manager | [Until closed plus 1 year] |
| Risk acceptance for releases over a fail | Head of Service Management | [5 years] |
Annexure A: Evaluation report
| Field | Entry |
|---|
| Service or change and reference | |
| Version and release | |
| Evaluator and dates | |
| Criteria used (attach table) | |
| Results summary by criterion | |
| Defects: blocking, major, minor | |
| Open items with owner and date | |
| Result (pass, conditional pass, fail, deferred) | |
| Recommendation | |
Annexure B: Operational acceptance checklist
| Item | Confirmed by | Yes or no | Date |
|---|
| Knowledge articles reviewed and published | | | |
| Routing and escalation set in the ticketing tool | | | |
| Monitoring alerts live and tested | | | |
| Runbooks and backup checked | | | |
| Support staff trained | | | |
| Service catalogue entry ready | | | |
Annexure C: Sign-off sheet
| Name | Role | Decision | Signature | Date |
|---|
| Service owner | | | |
| Head of Service Management | | | |
| Information security representative | | | |
| Customer representative | | | |
| Change authority | | | |
Annexure D: Document history
| Version | Date | Author | Change | Approved by |
|---|
| [1.0] | [date] | [author] | Initial issue | [approver] |
| | | | |
14. Exception
Exceptions to this document are allowed only where compliance is not practicable and the risk has been assessed. A request must state what is to be excepted, why, the risk it creates, the compensating controls, and an end date no later than [6 months]. It is approved in writing by the document owner and the risk owner, recorded in the exceptions register, and reviewed at the end date. An exception does not change the document.
15. Enforcement
Compliance is monitored through internal audits, management review and routine checks by the document owner. Non-compliance is reported through the incident or non-conformity process and handled under the organisation's disciplinary procedure, and under the terms of any contract for people who are not employees. Serious or repeated non-compliance may lead to loss of access and, where the law is broken, to referral to the authorities.
16. Review
This document is reviewed at least every [12 months], and when there is a significant change to the organisation, its systems, its risks or the law, or after an incident that shows it was not effective. Each review is recorded in the revision history.
ISO/IEC 20000-1:2018 references
| Reference | Title |
|---|
| 8.5.2 | Service design and transition |
| 8.5.3 | Release and deployment management |
| 8.2.4 | Service catalogue management |
| 8.3.3 | Service level management |