What drives vaccine quality at the last mile?

WHO EVM assessments across 77 countries | The Quality component

Amirhossein Ghadiri, PhD Student

Data Lab for Social Good, Cardiff Business School, Cardiff University

Bahman Rostami-Tabar, Supervisor

Data Lab for Social Good, Cardiff Business School, Cardiff University

A fridge in a small clinic

Infrastructure, Equipment, IT, HR, Policies & procedures, and Financial resources checks have been conducted in thousands of facilities across 77 countries, scoring each against hundreds of supply chain checks.

Which of those checks actually go with vaccines being in good condition? That is what I set out to answer.

About 60 seconds. Start with the picture and talk through it: the fridge on the left is what an assessor sees; on the right are three opened vials from that fridge. One was dated when it was opened and is still inside its window. One has no date at all, so nobody in that clinic can say whether it is safe. One is past the window and should already have been thrown away. Then the scale line, then read the question and pause.

What I will cover

1. The question
2. The data and the filters
3. Targets and models
4. What drives quality
5. Stress-testing the answer
6. What it means and what comes next

15 seconds. One line each. This slide comes back at every transition with the highlight moved, so you always know where we are.

One assessment, three components, and this is about Quality

The WHO Effective Vaccine Management assessment scores a facility against hundreds of checks, each with a code like R0290 called requirements. It also scores a set of outcomes.

• Outcomes are availability, efficiency, and quality. This analysis is about quality.
• The applied goal is a shorter instrument, an EVM-Lite. If a short checklist could check only a few things, which ones should they be?

About 45 seconds. Define requirement once, out loud, because every later slide uses the word. Then the division of labour in one breath. Land on EVM-Lite as the destination: that is what the whole talk is walking towards.

Quality is six checks: three during a session, three in the fridge

The composite score QU is the combination of them all.

About 70 seconds, and it earns the time because everything later refers back to this grid. Take the timeline first: a vial is opened at the left mark, doses can be given for 28 days, and anything left after the right mark is waste. Then read the grid as a table. Down the left are the three ways a vaccine goes bad: it expires, it gets too hot, which the vial monitor shows by darkening past its discard point, or it sits opened for too long. Across the top is where the assessor catches it, either watching a real immunisation session or inspecting what is in the fridge. Each failure is checked both ways, which gives six requirements, and the composite QU pools all six. Say that the two in blue are the ones I follow, and that the reason the other four fade out is coming in a couple of slides. Worth flagging now that the left column can only be scored when a session is running, because that fact comes back later to explain a strange result.

The aim here is to explain, not to predict or forecast

Following Shmueli (2010), the question here is which requirements go with quality and explain the variations in qualituy, not how to predict a facility’s score. ¹

• Error metrics such as RMSE will not be reported because a small error would not tell me which requirement matters
• Only models that hand back a readable number per requirement are allowed to give the answer

About 40 seconds. Say why the distinction bites: a model can be excellent at guessing scores and still tell you nothing you could act on. That rule disqualifies a model later that fits better than the one I use, and I want the panel to see the rule before they see the casualty.

1. Shmueli, G. (2010). To explain or to predict? Statistical Science, 25(3), 289-310.

How to read every number I show

Three things:

• R squared on unseen facilities. Facilities differ in quality. R squared is the share of that difference the fitted model accounts for, from 0 for none to 1 for all. It will always be measured on facilities the model was never fitted on.
• Conclusive. Each model is fitted on 200 resamples of the facilities 20 times. A requirement will be called conclusive only if 95% of those runs put it on the same side of zero

About 55 seconds. This slide buys back time everywhere else, so do not rush it. The 200 resamples idea is worth a sentence in plain speech: shuffle which facilities you happen to look at, refit, and see whether the answer keeps pointing the same way.

What I will cover

1. The question
2. The data and the filters
3. Targets and models
4. What drives quality
5. Stress-testing the answer
6. What it means and what comes next

5 seconds. Move the highlight and go.

From 4,287 rows down to one clean table

No input requirement shares a questionnaire item with the quality scores, so nothing on the left hand-side restates the answer.

About 75 seconds, the longest single slide, so guide it properly. Top row, left to right: 4,287 rows cover every level of the supply chain; I keep the 2,290 Service Points because that is the last mile, the fridge in the story; 2,258 of those have a quality score. Bottom row: 891 requirements in the workbook; 207 are the input requirements a manager can actually act on; 36 go because they apply to fewer than one facility in ten; one is an exact duplicate of another. The two rows meet on the right: 2,258 facilities, 172 columns, which is the 170 requirements plus assessment year and target population, across 77 countries. Then the line underneath: I checked that no requirement is scored from the same questionnaire item as the outcome, which would have made the whole thing circular.

The year column is really a country label

Every one of the 77 countries was assessed in exactly one year. Not one country appears in two different years.

• So the year tells which group of countries a facility belongs to, and nothing about change over time
• If the year looks important later, it means countries assessed in some years score better than countries assessed in others

• I keep the column, but I never read it as a trend, and this comes back twice

About 35 seconds. Small check, large consequence. Flag now that the year turns out to matter for both components, so when it appears the panel already knows how to read it. Without this slide that result looks like a story about improvement over time, which would be wrong.

What I will cover

1. The question
2. The data and the filters
3. Targets and models
4. What drives quality
5. Stress-testing the answer
6. What it means and what comes next

5 seconds.

Four of the seven targets have not much to explain

Target	Facilities at full marks	Best R² on unseen facilities
R0712 vials, use within 28 days	54.9%	0.185
R0722 vials, none left past 28 days	54.5%	0.174
QU composite quality	58.2%	0.067
R0710 R0711 R0713 R0716	93 to 97%	zero or below

The composite and the two vial components are carried forward. TabPFN, a model that works in a completely different way, ranks the seven targets the same way, which reassures that this is about the data rather than the choice of model.

About 60 seconds. Read the table row by row. The four targets I set aside cover expired stock and heat damage, and almost every facility already passes them, so there is no variation left to explain. That near-universal pass is a finding in itself about the last mile. What still separates facilities is multi-dose vial handling. Then the TabPFN line: same ordering from a model with nothing in common with mine.

The best-fitting model is not useful

Ten model families, each run over twenty repeated splits of the facilities into 75% for fitting and 25% for testing.

• Random forest achieves the best raw test score on the three targets, 0.238, but it scores about 0.7 higher on the facilities it was fitted on. Huge overfit
• XGBoost fits its training facilities perfectly, 1.000. Tre-based models are not useful for this problem
• Among the models that can explain and have a low overfit gap, the elastic net wins all three targets: 0.062 for QU, 0.192 for R0712, 0.183 for R0722

About 65 seconds. This is the rule from the earlier slide biting. Random forest wins on paper and still loses, and I want that said plainly because it is the kind of thing a panel tests you on. The elastic net is worth one sentence of explanation: it keeps a small number of requirements and pushes the rest to exactly zero, so what comes out is already a short list.

Knowing the country helps a lot, which is why it is put aside from the predictors

Between a fifth and two fifths of all variation sits between countries. That is income and infrastructure, and no facility can act on it.

About 55 seconds. Walk the chart: pale bars are the model I report, solid bars add one column per country, and every pair roughly doubles. Then the reasoning, which matters more than the numbers: my question is about requirements, and once country is in the model every requirement number becomes a within-country comparison, which hides how small the requirement signal really is. So I pay the price of the pale bars on purpose, and I measure that price in section 5.

What I will cover

1. The question
2. The data and the filters
3. Targets and models
4. What drives quality
5. Stress-testing the answer
6. What it means and what comes next

5 seconds.

Four requirements drive the composite Quality score, and two have negative effect

Code	What is checked	Effect
R0290	Responsible staff know the discard time of multi-dose vials.	+0.0114
R0322	The facility has SOPs (Standard Operating Procedure) for the multi-dose vial policy.	+0.0105
R0040	The facility has sufficient positive cold storage capacity to accommodate the expected maximum stock levels of vaccines.	−0.0121
R0156	The temperature record is a graph rather than a list	−0.0078

Meeting two WHO requirements have make vaccines quality worse.

About 70 seconds. Read all four rows straight off the slide. The two blue ones are exactly what the assessment intends, and both are about the vials from the opening story. Then slow right down on the orange ones: more fridge space and a tidier temperature chart go with worse quality. Do not explain it yet. Reveal the last line, let it sit, and tell them section 5 explains what is really going on. This is the hook that carries the middle of the talk.

On the two Quality components

R0712, four conclusive

Multi-dose vial vaccines are never administered more than 28 days after opening.

• R0290 Staff know the discard time of multi-dose vials. +0.039
• R0257 Staff are trained how to monitor ISC performance. +0.039
• R0272 Staff know key principles and procedures of vaccine & dry goods storage. +0.029
• R0037 Storage buildings have functional certified fire extinguishers. +0.028

R0722, five conclusive

There are no opened multi-dose vials in stock that were opened more than 28 days ago.

• R0037 Storage buildings have functional certified fire extinguishers. +0.044
• R0293 Staff know the main types of opened vial wastage. +0.039
• R0290 Staff know the discard time of multi-dose vials. +0.035
• R0358 Staff are paid in full and on time. +0.033
• R0129 Waste is stored in plastic or metal bins. +0.030

R0290, knowing when a vial must go, is the only requirement on all three lists.

About 70 seconds. Read both columns aloud, then land the common thread. Point out the odd ones honestly: fire extinguishers, salaries and waste bins cannot touch a vial, so something else is behind them, and the next section deals with it. Add that the assessment year is conclusive for both components too, which is the country label from earlier showing through rather than any change over time.

What I will cover

1. The question
2. The data and the filters
3. Targets and models
4. What drives quality
5. Stress-testing the answer
6. What it means and what comes next

5 seconds.

Results verification

• Shuffling the answers. Values of the target scores were randomly shuffled, then the whole machine was run again. It found zero requirements, against real counts of 4, 4, and 5.
• Removing the filled-in values. Where a requirement does not apply, the model needs a number, the median of the column was put. Refitting only on facilities that were actually scored, all 13 leading effects keep their direction and 9 stay conclusive (expect R0290, R0322, R0257, R0037).
• The one hold loosely is R0322 (The facility has SOPs (Standard Operating Procedure) for the multi-dose vial policy): without the filled rows its effect drops to nearly nothing.

About 60 seconds. The first bullet is the important one, so put it in plain speech: if my method finds four requirements out of pure noise, then finding four in the real data proves nothing. It found none, thirty times running. The second bullet defends the one compromise in the data preparation, and the third is me flagging my own weakest result before Bahman does.

On the composite, every conclusive requirement is named by most families

The six are ridge, fractional logit, boosted trees, bagged trees, randomised-split trees and XGBoost. Boosted trees and XGBoost are never handed a filled-in value, so a requirement they name alongside the other four cannot be an artefact of the filling.

About 30 seconds. Explain the picture once and it carries the next two slides: each row is a requirement I called conclusive earlier, and a filled dot means that family also put it in its own top twenty, ranked by permutation importance with ten shuffles per column. Name the six families off the slide. The point to land is that all four rows sit at four or better, and the graphical form is unanimous. If asked which families fill: ridge, the fractional logit, bagged trees and randomised-split trees are handed a median wherever a requirement did not apply, while boosted trees and XGBoost handle those cells natively and are never given one.

On R0712, the agreement narrows to one requirement

Knowing the discard time is the one requirement every family reaches for.

About 25 seconds. Be straight about the weak rows: fire extinguishers and the training requirement are named by only two families, so the bootstrap intervals are the whole of their evidence. That is a reason to treat them as leads rather than findings, and it is the honest counterweight to the clean shuffle result on the previous slide.

On R0722, most of the list survives the same check

Four of the five sit at four families or better, and knowing the types of opened vial wastage is unanimous.

About 25 seconds. Note the mirror image of the previous slide: R0290 is named by every family on R0712 but by only three here, while R0037 is weak on R0712 and strong here. The two components measure different things, so a requirement can matter for one and not the other. Add that vote counts near the top-twenty boundary can move between runs, so read the pattern rather than the exact number.

Why meeting a requirement can look like it reduces Quality score

Take R0040, having enough cold storage. 90% of facilities meet it. Those that do are asked the multi-dose vial questions 11 percent more often, and they average 0.890 against 0.938 for those that do not.

About 45 seconds, and it is the idea the whole section turns on, so do not rush. Walk the four boxes left to right. Box one points straight back to the grid earlier: the left-hand column there, the three checks that need a session in progress. Box two: those are the questions facilities actually fail, and you can only fail a question you were asked. Box three: a facility with spare fridge capacity is a facility running sessions. Box four: so the better equipped facility sits the harder test and can score lower for a reason that has nothing to do with its vaccines. Then read the cold storage numbers underneath as the worked example.

Average Quality score among facilities meeting and not meeting requirements

These are the composite’s four conclusive requirements.

About 45 seconds. Read the chart before drawing any conclusion. Each row is one of the four requirements that came out conclusive for the composite, the same four from earlier. The hollow circle is the average quality score of facilities that do not meet it, the solid circle is the average for facilities that do, and the number on the right is the difference between them. The top two move right, so meeting them goes with a better score. The bottom two move left. Now the key sentence: the pressure from the previous slide pulls every row leftwards, so the two that still land on the right are managing it against that headwind, while the two on the left are exactly what the artefact predicts. If asked whether this holds beyond these four, the same check was run on the six requirements the model families agree on most, and the two extra ones, knowing the stock system and training in planning, also came out positive at plus 0.032 and plus 0.012. Close on the rule: a statistical signal is not enough, there must also be a route from the requirement to the vial, and cold storage capacity and the shape of a form have no such route.

What I will cover

1. The question
2. The data and the filters
3. Targets and models
4. What drives quality
5. Stress-testing the answer
6. What it means and what comes next

5 seconds.

Conclusive requirements do most of the work of 172 columns

About 55 seconds. Walk it: pale is everything the assessment asks, solid is the handful I named, and the gap is what a shorter instrument would give up. The composite keeps almost everything on four questions. If asked what the 172 covers: the 170 filtered requirements plus the assessment year and the target population, the same specification every earlier figure in the talk uses. Say the caveat in your own words rather than reading it, because Bahman will ask otherwise, and it is better coming from you first.

Limitations and take aways

• This analysis is association-based, not causal
• Out of 170 requirements across 77 countries, quality at the last mile comes down to a handful, and the one that appears everywhere is staff knowing when an opened vial must be thrown away
• The short list passed tests like shuffling answers, six model families verification, and the removal of every filled-in value

Thank you! Please let me know your questions.

About 30 seconds. Three sentences, slowly, then stop. Do not wander into the backup slides unless someone asks.

Backup

Backup: Choosing the model

• Ten families over the same twenty splits: OLS, ridge, LASSO, elastic net, fractional logit, a gamma model on the shortfall from full marks, boosted trees, bagged trees, randomised trees, XGBoost
• Plain OLS is the weakest linear model on unseen facilities for all seven targets, so the penalties earn their place

For questions on model choice. One line on AIC if pushed: a criterion computed inside the sample rewards exactly the hard fitting that repeated splits exist to discount.

Backup: How the values that do not exist are handled

• Some requirements do not apply to a facility, and a linear model still needs a number in that cell
• Two options: fill with the median value, or fill and add a marker column saying the value was filled
• Markers fit better by but it’s negligible, but they double the number of columns
• The simple fill is kept, because a marker coefficient is not something a manager can act on

For questions on missing data. Be straight about it: markers win narrowly on fit and lose on usefulness.

Backup: quality as four ordered bands

• The score is cut at 0.6, 0.8 and 0.95, and 58% of facilities land in the top band
• Simply always naming the biggest band beats on plain accuracy, 0.582 against 0.496
• The top twenty requirements change under banding while no measure of fit improves, so the analysis is continued with the continuous score

For questions about whether the continuous score is the right framing.