Sequential drug decision problems in long-term medical conditions: A Case Study of Primary Hypertension

Background: Sequential drug decision problems (SDDP) occur when assigning drugs sequentially in long-term medical conditions. SDDPs are important for both clinical decision-making and resource allocation. They can be large and complex because of the considerable number of drug sequences and disease pathways and the interdependence between them over time. Where classic mathematical programming has a limited capacity for dealing with the complexities of a sequential decision problem, approximate optimisation methods have been widely used to solve the problem more efficiently using simulation.
Objective: This thesis aims to set down the definitions of SDDPs mathematically to understand the nature of SDDPs, to examine the potential methods to identify optimal or near-optimal sequential treatment strategies in a long-term SDDP; and to discuss the performance of the proposed methods using a case study of primary hypertension.
Methods: A mathematical description of SDDPs was developed to gain an understanding of the nature of SDDPs. A systematic review was conducted to examine potential optimisation methods for solving large and complex SDDPs. A hypothetical simple SDDP was used to test the feasibility of incorporating the promising methods into an economic evaluation model. A de novo hypertension cost-effectiveness model estimating blood pressure lowering effects of sequential use of antihypertensive drugs was developed. Enumeration, simulated annealing (SA), genetic algorithm (GA) and reinforcement learning (RL) were used to solve the SDDP in primary hypertension. Their performance was tested in terms of computational time and the quality of solution, which is defined by the closeness of the final objective function values and the real global optimum are obtained from enumeration.
Results: The computational complexity of SDDPs comes from a range of factors, which are: 1) the number of relevant health states, 2) the number of potential drug treatment options, 3) the number of times that a treatment change may occur, 4) whether the transition probability between health states depends on historic health states and drug uses and 5) relevant clinical-based rules that need to be incorporated. Various trade-offs, such as the trade-off between the computational complexity and model validity, the trade-off between the research effort and time required to develop the optimisation model and the underlying evaluation model, and the trade-off between the amount of search time and the quality of the solution, are fundamental features of SDDP modelling. These trade-offs are all interrelated with each other rather than existing separately. In the case study of primary hypertension, the optimal solution identified by enumeration was to start with an angiotensin converting enzyme inhibitor or an angiotensin II receptor blocker (ACEI/ARB), followed by the combination of thiazide-type diuretic (D) and ACEI/ARB, the combination of D, ACEI/ARB and calcium channel blocker (CCB) and the combination of D, ACEI/ARB and beta-blocker (BB) as second, third and fourth-line treatments. The total expected net benefit for this optimal sequential treatment policy was £330,080 (95% CI £330,013-£330,147). SA and GA found the same (or statistically indifferent) solution(s) identified by enumeration with shorter search time and smaller iteration number. The computational time was 4.1-4.6 hours in SA or GA whereas enumeration took 12.20 hours. The performance depended on some key parameters of the methods: cooling rate and the maximum number of iterations within the same temperature for SA and the number of generation, population size, crossover rate and mutation rate for GA. The performance of RL was relatively less favourable. This may be because of the structure of the hypertension SDDP model, whose total net benefit is mostly affected by the add-on Markov model after the drug switching period than the short-term drug switching model.
Conclusion: SA and GA can be used to solve a large and complex SDDP as demonstrated in the primary hypertension case study. They can find the optimal or near optimal solutions efficiently where the key parameters are properly set. The optimal parameter setting is problem specific and requires a tuning procedure considering various scenarios with different sets of parameters. RL needs further investigation to improve the performance possibly by using more complicated RL methods or in a different structure of the underlying evaluation model. This study can be extended to construct the underlying evaluation model using a DES and to technically improve the optimisation methods. Producing the data relevant to SDDPs will also help to make better informed decisions for SDDPs in health technology appraisal.