by Brigitte Scott in discussion with Tim Grant PhD
Statistical analysis of clinical trial data is a crucial part of the clinical research process. To ensure the data are a true reflection of the clinical trial, and to maximise the information that can be derived from the data, the statistical analysis should be relevant, precise and comprehensive, to guarantee all data are correctly captured and analysed. The Statistical Analysis Plan (SAP) documents the planned analysis of clinical trials to provide statistically-valid data that are accurate, reliable and reproducible.
According to Tim Grant, an experienced biostatistician and research fellow at StatisticaMedica, the philosophy behind the SAP is simple: To ensure statistically-valid data that are accurate, reliable and reproducible, it is critical to get the SAP right.
Here are five tips for a high-quality, fit-for-purpose SAP:
Regard the SAP as an extension of, or an appendix to, the protocol, not a repetition of it
Clearly define all trial patient populations, variables and statistical test methods in the SAP
Treat the SAP as a ‘living’ rather than a ‘static’ document
Write the SAP early on and create mock tables, figures and listings
Maintain good communication and interaction between the client and trial team throughout every stage of the trial process
To ensure statistically-valid data that are accurate, reliable and reproducible, it is critical to get the SAP right
1. Regard the SAP as an extension of, or an appendix to, the protocol, not a repetition of it
Restating all the information from the protocol in the SAP is an unnecessary duplication of work. It is also potentially problematic and time-consuming because any changes to the protocol necessitate changes in the SAP, and all the extra work this entails (amendments, quality control check of the changes, signing off an updated version of the SAP). The SAP is not a repetition of the protocol.
The biostatistician should be engaged early in the protocol development stage to provide biostatistical input and considerations for the trial protocol, and to ensure transparency and consistency between the protocol and the SAP.
So, what is the SAP?
The best approach is to regard the SAP as an extension of, or an appendix to, the protocol. The SAP is an in-depth technical document that includes detailed procedures for executing statistical analyses on clinical trial data. Variables, statistical test methods and statistical hypotheses are defined in the SAP and do not belong in the protocol. Also, importantly, detailed clinical information does not belong in the SAP. Any trial hypotheses will be stated clinically in the protocol and mathematically in the SAP.
To summarise the difference between the protocol and the SAP:
The protocol includes the principle features of the planned statistical analysis of the data. This document targets the clinical audience.
The SAP contains a more technical and detailed description of the analysis. This document targets the statistical audience.
For example, testing for differences in blood pressure to define the response to a certain intervention:
The protocol provides the clinical definition of the variable, stating that ambulatory blood pressure will be measured at certain timepoints during the trial, and details the timings (day/night, how often) of and clinical methods (how to measure, by whom, patient position) for these measurements.
The SAP provides a precise statistical definition of ambulatory blood pressure and how to calculate it, e.g., average systolic blood pressure across five measurements taken between 7am and 7pm, thereby defining exactly how the data are being used.
Why use a SAP?
There are many reasons for creating and using a SAP. For example, the SAP:
Increases transparency of the analyses
Ensures reproducibility
Avoids data-driven exercises
Decreases the number of decisions during the analyses
Improves the validity of any conclusions
Stimulates communication between the biostatistician and stakeholders
A direct result of the development of large databases and powerful statistical software is that it is now easier to find associations within the trial data and derive conclusions from the results without forming an a-priori hypothesis. This approach could be interpreted as ‘fishing’ for the desired results. Such an approach may produce false-positive or biased findings, or associations within the data that have no clinical relevance, which potentially impacts on the credibility of the analysis. The in-depth technical focus of the SAP, particularly when this document is used in conjunction with the trial protocol, improves the reproducibility, transparency, quality and validity of clinical trial data, thereby ensuring the accuracy and value of data-driven conclusions.
Detailed clinical information is included in the protocol, with some statistics (e.g., primary endpoint)
Detailed statistical information is included in the SAP, with some clinical information (e.g., timings of measurements)
The SAP is not a repetition of the protocol
2. Clearly define all trial patient populations, variables and statistical test methods in the SAP
The SAP needs to be a ‘definition document’ that states the patient populations, variables and statistical test methods for the trial.
The inclusion and exclusion criteria in the protocol do not belong in the SAP unless one or more of the criteria help to define the patient population, in which case the specific criterion/criteria (rather than the complete list) should be added to the SAP. Patients who violate the inclusion/exclusion criteria are not included in the statistical analysis anyway.
The SAP elaborates on the principle features of the statistical analysis that are outlined in the trial protocol:
Converts the clinical endpoint/parameters into mathematical definitions
Clarifies which comparisons are being made in the trial
Shows how to control for multiple comparisons (multiplicity), thereby reducing the chance of errors
Defines the level of randomisation (e.g., country level, regional level, site level)
Details all instrumentation and scoring methods (e.g., for quality of life evaluations)
States the type and version of software used for statistical analysis (including sample codes, where appropriate)
Provides information on how missing data are to be handled
Identifies the medical dictionaries for adverse events and drugs (e.g., MedDRA and WHO-DDE, respectively)
The SAP needs to be a ‘definition document’ that states the patient populations, variables and statistical test methods for the trial
3. Treat the SAP as a ‘living’ rather than a ‘static’ document
For some clinical trials, the definitions in the SAP may need to change as the trial progresses. For example, the number of biological samples taken from the patient per day, as defined in the protocol, may be an unrealistic target for the patient and/or healthcare professional taking the samples. The SAP needs to be a flexible, ‘living’ document that can be adapted to reflect what is clinically or practically feasible, if the schedule, methods, or tasks in the original protocol prove to be unworkable. For example, if the protocol states 15 samples per day yet most patients have only 12 samples taken, the variable in the SAP needs to be reset.
Flexibility and adaptations at the Phase I or II stage can be used to inform the Phase III trial.
Any changes in the SAP should be clearly tracked and described in the SAP and the clinical study report (CSR).
The SAP needs to be a flexible, ‘living’ document that can be adapted to reflect what is clinically or practically feasible, if the schedule, methods, or tasks in the original protocol prove to be unworkable
4.Write the SAP early on and create mock tables, figures and listings
Engaging with other stakeholders early in the clinical trial process is imperative for a successful outcome. Writing the SAP early on in the trial process and creating mock tables, figures and listings (TFLs) are critical for timely and successful data analysis and reporting. Biostatisticians may not necessarily interpret the data in the same way as an experienced clinician or medical writer; therefore, it is imperative to derive mock TFLs and ask the clinician/medical writer whether the tables are going to present the data in the way they need.
The following tasks are recommended to ensure quality and efficiency in the trial process:
Develop the SAP and mock TFLs in tandem and submit them as a package for review by the client, clinician and/or medical writer
Use templates to create the SAP and mock TFLs, thereby promoting standardisation, efficiency and quality
Follow all relevant guidelines to provide regulatory-grade output (e.g., ICH regulatory grade SAP)
Review the database structure and map it against the SAP to guarantee alignment between the source data and the analysis output
The SAP should be documented in such a way that all the data manipulations and analyses performed can be replicated.
The SAP provides transparency around how the analysis will proceed by specifying in advance the methodology that will be applied.
Writing the SAP early in the trial process and creating mock tables, figures and listings are critical for timely and successful data analysis and reporting
5. Maintain good communication and interaction between the client and trial team throughout every stage of the trial process
Productive interaction between the client, clinicians and trial team (including biostatisticians and medical writers) early on in the project to establish the clinical, statistical and marketing goals of the trial smooths the clinical research journey. Involving the biostatistician at the clinical trial design stage and throughout the protocol development process will enhance the quality and accuracy of the clinical trial and ensure all research questions are comprehensively addressed from a statistical perspective.
There is no need to limit the biostatistician’s role to simply calculating the sample size for the trial. The biostatistician can actively:
Engage in the trial design process
Provide biostatistical input and considerations for the trial protocol during the protocol development stage
Contribute to aspects of the trial conduct to protect internal and external validity
Advise on choice of endpoints
Define randomisation schedules
Collaboration between the client, clinician and trial team involved in a research programme should be encouraged from the start of the first Phase I trial and upheld throughout the programme during any Phase II, Phase II/III and Phase III trials. This ensures that information gathering starts early on and any lessons learned can guide development of later stage trials. Similarly, implementing CDISC standards early on optimises the data analysis process from the outset and enables production of high quality CDISC-submission-ready data.
Good communication and interaction between the client and trial team increases efficiency and smooths the clinical research journey
Refer Guidelines for ICH E9 Statistical Principles for Clinical Trials
Refer Guidelines for ICH E3 Clinical Study Report