What drives vaccine quality at the last mile?

WHO EVM assessments across 77 countries | The Quality component

Amirhossein Ghadiri, PhD Student

Data Lab for Social Good, Cardiff Business School, Cardiff University

Bahman Rostami-Tabar, Supervisor

Data Lab for Social Good, Cardiff Business School, Cardiff University

A fridge in a small clinic

Infrastructure, Equipment, IT, HR, Policies & procedures, and Financial resources checks have been conducted in thousands of facilities across 77 countries, scoring each against hundreds of supply chain checks.

Which of those checks actually go with vaccines being in good condition? That is what I set out to answer.

About 60 seconds. Start with the picture and talk through it: the fridge on the left is what an assessor sees; on the right are three opened vials from that fridge. One was dated when it was opened and is still inside its window. One has no date at all, so nobody in that clinic can say whether it is safe. One is past the window and should already have been thrown away. Then the scale line, then read the question and pause.

What I will cover

1. The question
2. The data and the filters
3. Targets and models
4. What drives quality
5. Stress-testing the answer
6. Causal estimates
7. Subcategory scores
8. What it means and what comes next

5 seconds. Move the highlight and go.

One assessment, three components, and this is about Quality

The WHO Effective Vaccine Management assessment scores a facility against hundreds of checks, each with a code like R0290 called requirements. It also scores a set of outcomes.

• Outcomes are availability, efficiency, and quality. This analysis is about quality.
• The applied goal is a shorter instrument, an EVM-Lite. If a short checklist could check only a few things, which ones should they be?

About 45 seconds. Define requirement once, out loud, because every later slide uses the word. Then the division of labour in one breath. Land on EVM-Lite as the destination: that is what the whole talk is walking towards.

Quality is six checks: three during a session, three in the fridge

The composite score QU is the combination of them all.

About 70 seconds, and it earns the time because everything later refers back to this grid. Take the timeline first: a vial is opened at the left mark, doses can be given for 28 days, and anything left after the right mark is waste. Then read the grid as a table. Down the left are the three ways a vaccine goes bad: it expires, it gets too hot, which the vial monitor shows by darkening past its discard point, or it sits opened for too long. Across the top is where the assessor catches it, either watching a real immunisation session or inspecting what is in the fridge. Each failure is checked both ways, which gives six requirements, and the composite QU pools all six. Say that the two in blue are the ones I follow, and that the reason the other four fade out is coming in a couple of slides. Worth flagging now that the left column can only be scored when a session is running, because that fact comes back later to explain a strange result.

The aim here is to explain, not to predict or forecast

Following Shmueli (2010), the question here is which requirements go with quality and explain the variations in qualituy, not how to predict a facility’s score. ¹

• Error metrics such as RMSE will not be reported because a small error would not tell me which requirement matters
• Only models that hand back a readable number per requirement are allowed to give the answer

About 40 seconds. Say why the distinction bites: a model can be excellent at guessing scores and still tell you nothing you could act on. That rule disqualifies a model later that fits better than the one I use, and I want the panel to see the rule before they see the casualty.

1. Shmueli, G. (2010). To explain or to predict? Statistical Science, 25(3), 289-310.

How to read every number I show

• R squared on unseen facilities. Facilities differ in quality. R squared is the share of that difference a fitted model accounts for, from 0 for none to 1 for all. It is always measured on facilities the model was never fitted on.
• Effect. The change in the quality score, which runs from 0 to 1, for one standard deviation more of a requirement’s score. The causal section later uses a plainer unit: the change from not met to met in full.
• Conclusive. The model is refitted on 200 random resamples of the facilities. A requirement is conclusive when the middle 95% of its effects across those refits lies on one side of zero, and at least half of the refits keep it in the model at all.
• Not applicable. Where a requirement does not apply to a facility, the linear models are given the column’s mean. The checks later remove those filled cells and look again.

About 60 seconds. This slide buys back time everywhere else, so do not rush it. The resamples idea in plain speech: shuffle which facilities you happen to look at, refit, and see whether the answer keeps pointing the same way. Say that the mean filling is new in this version of the analysis, and that a later slide removes it to check.

What I will cover

1. The question
2. The data and the filters
3. Targets and models
4. What drives quality
5. Stress-testing the answer
6. Causal estimates
7. Subcategory scores
8. What it means and what comes next

5 seconds. Move the highlight and go.

From 4,287 rows down to one clean table

No input requirement shares a questionnaire item with the quality scores, so nothing on the left hand-side restates the answer.

About 75 seconds, the longest single slide, so guide it properly. Top row, left to right: 4,287 rows cover every level of the supply chain; I keep the 2,290 Service Points because that is the last mile, the fridge in the story; 2,258 of those have a quality score. Bottom row: 891 requirements in the workbook; 207 are the input requirements a manager can actually act on; 36 go because they apply to fewer than one facility in ten; one is an exact duplicate of another. The two rows meet on the right: 2,258 facilities, 172 columns, which is the 170 requirements plus assessment year and target population, across 77 countries. Then the line underneath: I checked that no requirement is scored from the same questionnaire item as the outcome, which would have made the whole thing circular.

The year column is really a country label

Every one of the 77 countries was assessed in exactly one year. Not one country appears in two different years.

• So the year tells which group of countries a facility belongs to, and nothing about change over time
• If the year looks important later, it means countries assessed in some years score better than countries assessed in others

• I keep the column, but I never read it as a trend, and this comes back twice

About 35 seconds. Small check, large consequence. Flag now that the year turns out to matter for both components, so when it appears the panel already knows how to read it. Without this slide that result looks like a story about improvement over time, which would be wrong.

What I will cover

1. The question
2. The data and the filters
3. Targets and models
4. What drives quality
5. Stress-testing the answer
6. Causal estimates
7. Subcategory scores
8. What it means and what comes next

5 seconds. Move the highlight and go.

Four of the seven targets have almost nothing to explain

Target	Facilities at full marks	Best R² on unseen facilities
R0712 vials used within 28 days of opening	54.9%	0.187
R0722 no opened vials kept beyond 28 days	54.5%	0.171
QU composite quality score	58.2%	0.059
R0710 R0711 R0713 R0716 expired or heat-damaged vaccine	94 to 97%	0.05 or below

The composite and the two vial components are carried forward. Almost every facility already passes the four expired and heat-damaged checks, so there is no variation left to explain.

About 60 seconds. Read the table row by row. The near-universal pass on expired and heat-damaged stock is a finding in itself about the last mile. What still separates facilities is multi-dose vial handling. The R squared figures here come from five repeated splits of the facilities; the next slide uses twenty.

The best-fitting model is not useful

Eleven model families, each run over twenty repeated splits of the facilities into 75% for fitting and 25% for testing.

• Random forest has the best raw test score, 0.239 on R0712, but it scores about 0.65 higher on the facilities it was fitted on. It memorises them.
• XGBoost fits its training facilities perfectly, 1.000, and does worse on new ones. Tree models cannot answer an explanatory question here.
• Among the models that give a readable number per requirement, the elastic net wins all three targets: 0.051 for QU, 0.191 for R0712, 0.180 for R0722.

About 65 seconds. This is the rule from the earlier slide biting. Random forest wins on paper and still loses, and I want that said plainly. The elastic net is worth one sentence: it keeps a small number of requirements and pushes the rest to exactly zero, so what comes out is already a short list.

Knowing the country helps a lot, which is why it is kept out of the predictors

About 55 seconds. Walk the chart: pale bars are the model I report, solid bars add one column per country, and every pair roughly doubles. Between a fifth and two fifths of all variation sits between countries: 21% for the composite, 41% and 41% for the components. That is income and infrastructure, and no facility can act on it. Once country is in the model every requirement number becomes a within-country comparison, which hides how small the requirement signal really is. So I pay the price of the pale bars on purpose.

What I will cover

1. The question
2. The data and the filters
3. Targets and models
4. What drives quality
5. Stress-testing the answer
6. Causal estimates
7. Subcategory scores
8. What it means and what comes next

5 seconds. Move the highlight and go.

Two requirements drive the composite score, and one pulls it down

Code	Subcategory	What is checked	Effect	Status
R0290	C4.3 Knowledge & understanding	Staff know the discard time of multi-dose vials	+0.0102	conclusive
R0040	C2.1.1 Cold storage equipment	Sufficient cold storage capacity for the expected maximum stock	-0.0114	conclusive
R0322	C5.3 SOPs	SOPs for the multi-dose vial policy	+0.0061	borderline
R0156	C3.2 Data management technology	The temperature record form is graphical	-0.0061	borderline

Borderline: the 200-resample interval touches zero but stops there, and the requirement is kept in over 90% of refits.

Meeting a WHO requirement can go with a worse quality score. Section 5 explains why.

About 70 seconds. Two are conclusive, two sit just outside. The first is exactly what the assessment intends, and it is about the vials from the opening story. Then slow right down on cold storage: more fridge space goes with worse quality. Do not explain it yet. Reveal the last line, let it sit.

On R0712, four requirements are conclusive

R0712: multi-dose vial vaccines are never administered more than 28 days after opening.

Code	Subcategory	What is checked	Effect	Interval
R0290	C4.3 Knowledge & understanding	Staff know the discard time of multi-dose vials	+0.0403	0.0137 to 0.0620
R0257	C4.2 Training	Staff are trained to monitor supply chain performance	+0.0387	0.0094 to 0.0691
R0037	C1.2 Buildings & facilities	Storage buildings have functional fire extinguishers	+0.0289	0.0007 to 0.0508
R0272	C4.3 Knowledge & understanding	Staff know the principles of vaccine and dry goods storage	+0.0263	0.0017 to 0.0546

Effect: change in the 0 to 1 score per standard deviation of the requirement. Interval: middle 95% of the effect over 200 resamples.

Knowing the discard time, R0290, has the largest effect.

About 40 seconds. Read the four rows. Two are about what staff know and one is about training, which is what the assessment intends. The fourth, fire extinguishers, cannot touch a vial, so something else is behind it; sections 5 and 6 come back to it. The assessment year is conclusive here too, which is the country label from earlier showing through, not a change over time.

On R0722, five requirements are conclusive

R0722: there are no opened multi-dose vials in stock that were opened more than 28 days ago.

Code	Subcategory	What is checked	Effect	Interval
R0037	C1.2 Buildings & facilities	Storage buildings have functional fire extinguishers	+0.0463	0.0168 to 0.0737
R0293	C4.3 Knowledge & understanding	Staff know the main types of opened-vial wastage	+0.0385	0.0081 to 0.0629
R0290	C4.3 Knowledge & understanding	Staff know the discard time of multi-dose vials	+0.0363	0.0112 to 0.0588
R0358	C6.1 Salaries	Staff are paid in full and on time	+0.0333	0.0088 to 0.0589
R0129	C2.3 Other equipment	Waste is stored in plastic or metal bins	+0.0297	0.0069 to 0.0586

Effect: change in the 0 to 1 score per standard deviation of the requirement. Interval: middle 95% of the effect over 200 resamples.

R0290, knowing when a vial must go, is the only requirement on all three lists.

About 45 seconds. Read the five rows, then land the common thread: the discard-time requirement is on all three lists. Point out the odd ones honestly: fire extinguishers, salaries and waste bins cannot touch a vial, so something else is behind them, and sections 5 and 6 deal with it. The assessment year is conclusive here as well.

Composite score: the 25 largest effects, 1 to 13

#	Code	Subcategory	What is checked	Applies to	At max	Effect	Interval	Kept in	Concl.
1	R0040	C2.1.1 Cold storage equipment	Sufficient cold storage capacity for the expected maximum stock	90%	90%	-0.0114	-0.0167 to -0.0062	100%	✓
2	R0290	C4.3 Knowledge & understanding	Staff know the discard time of multi-dose vials	90%	90%	+0.0102	0.0028 to 0.0171	100%	✓
3	R0322	C5.3 SOPs	SOPs for the multi-dose vial policy	89%	65%	+0.0061	0.0000 to 0.0128	94%
4	R0156	C3.2 Data management technology	The temperature record form is graphical	96%	41%	-0.0061	-0.0119 to 0.0000	96%
5	R0257	C4.2 Training	Staff are trained to monitor supply chain performance	100%	54%	+0.0057	0.0000 to 0.0125	94%
6	R0130	C2.3 Other equipment	Bins are colour coded for different types of waste	70%	56%	+0.0054	0.0000 to 0.0118	93%
7	R0251	C4.1 Staffing	Staff turnover is low	97%	33%	+0.0050	0.0000 to 0.0116	92%
8	R0279	C4.3 Knowledge & understanding	Staff know cool packs are prepared in dedicated compartments	50%	52%	+0.0046	0.0000 to 0.0097	93%
9	R0037	C1.2 Buildings & facilities	Storage buildings have functional fire extinguishers	100%	19%	+0.0045	0.0000 to 0.0110	91%
10	R0136	C3.1 General IT equipment	Supply chain staff have work mobile phones	100%	59%	+0.0044	0.0000 to 0.0129	83%
11	R0328	C5.3 SOPs	There are safe injection SOPs in the immunization room	98%	24%	+0.0042	0.0000 to 0.0123	82%
12	R0292	C4.3 Knowledge & understanding	Staff know how to calculate the unopened vial wastage rate	100%	51%	+0.0035	0.0000 to 0.0096	85%
13	R0275	C4.3 Knowledge & understanding	Staff know how to use the stock management system	100%	73%	+0.0034	0.0000 to 0.0090	78%

Applies to: share of facilities where the requirement was asked. At max: share of those meeting it in full. Effect: change in the 0 to 1 score per standard deviation of the requirement. Interval: middle 95% over 200 resamples. Kept in: share of resamples that kept it. Shaded rows are conclusive.

About 60 seconds. Read the shaded rows and then say what the rest of the table is for: a ranking by size so the reader can see how quickly the evidence runs out, not a list of findings. Point at the two profile columns: a requirement almost everyone meets has little room to show an effect.

Composite score: the 25 largest effects, 14 to 25

#	Code	Subcategory	What is checked	Applies to	At max	Effect	Interval	Kept in	Concl.
14	R0122	C2.3 Other equipment	Waste handlers have boots	16%	60%	+0.0030	0.0000 to 0.0105	74%
15	R0064	C2.1.1 Cold storage equipment	Vaccine refrigerator/freezers have door open alarms	35%	10%	-0.0030	-0.0120 to 0.0000	68%
16	R0269	C4.3 Knowledge & understanding	Staff can correctly read a 30DTR	71%	85%	+0.0027	0.0000 to 0.0100	70%
17	R0186	C3.2 Data management technology	Stock record form has a field for VVM status	95%	60%	-0.0025	-0.0085 to 0.0000	62%
18	R0298	C4.3 Knowledge & understanding	Staff know how to dispose of expired temperature monitoring devices	100%	82%	-0.0022	-0.0075 to 0.0000	72%
19	R0272	C4.3 Knowledge & understanding	Staff know the principles of vaccine and dry goods storage	99%	87%	+0.0019	0.0000 to 0.0083	60%
20	R0212	C3.2 Data management technology	Receipt form has a field for Freeze indicator status of vaccines received	84%	56%	-0.0018	-0.0078 to 0.0000	58%
21	R0154	C3.2 Data management technology	The form has fields for entering AM and PM temperatures	96%	84%	+0.0018	0.0000 to 0.0088	50%
22	R0210	C3.2 Data management technology	Receipt form has a field for Quantity of vaccines received	100%	78%	-0.0017	-0.0067 to 0.0000	57%
23	R0326	C5.3 SOPs	SOPs for disposal of sharps waste	56%	63%	+0.0016	0.0000 to 0.0063	53%
24	R0073	C2.1.1 Cold storage equipment	Sufficient coolant pack storage capacity for daily demand	96%	37%	-0.0016	-0.0071 to 0.0000	60%
25	R0077	C2.2.1 Vehicles	Transport is always available for scheduled vaccine distribution, collect…	37%	95%	+0.0015	0.0000 to 0.0103	40%

Below the conclusive rows the intervals include zero, so the order says where the evidence thins out, not which of these matter.

About 30 seconds. Do not read these rows. Say that from here down the intervals include zero, and move on.

R0712: the 25 largest effects, 1 to 13

#	Code	Subcategory	What is checked	Applies to	At max	Effect	Interval	Kept in	Concl.
1	R0290	C4.3 Knowledge & understanding	Staff know the discard time of multi-dose vials	100%	92%	+0.0403	0.0137 to 0.0620	100%	✓
2	R0257	C4.2 Training	Staff are trained to monitor supply chain performance	100%	56%	+0.0387	0.0094 to 0.0691	100%	✓
3	R0037	C1.2 Buildings & facilities	Storage buildings have functional fire extinguishers	100%	23%	+0.0289	0.0007 to 0.0508	98%	✓
4	R0272	C4.3 Knowledge & understanding	Staff know the principles of vaccine and dry goods storage	99%	88%	+0.0263	0.0017 to 0.0546	98%	✓
5	R0156	C3.2 Data management technology	The temperature record form is graphical	97%	44%	-0.0227	-0.0515 to 0.0000	94%
6	R0270	C4.3 Knowledge & understanding	Staff know the correct storage temperature for each vaccine	99%	41%	+0.0194	0.0000 to 0.0529	83%
7	R0130	C2.3 Other equipment	Bins are colour coded for different types of waste	72%	56%	+0.0187	0.0000 to 0.0431	92%
8	R0328	C5.3 SOPs	There are safe injection SOPs in the immunization room	100%	29%	+0.0185	0.0000 to 0.0519	83%
9	R0002	C1.1 Utilities & services	A functional landline telephone	100%	18%	+0.0178	0.0000 to 0.0454	90%
10	R0279	C4.3 Knowledge & understanding	Staff know cool packs are prepared in dedicated compartments	52%	49%	+0.0175	0.0000 to 0.0426	88%
11	R0129	C2.3 Other equipment	Waste is stored in plastic or metal bins	100%	48%	+0.0156	0.0000 to 0.0442	86%
12	R0251	C4.1 Staffing	Staff turnover is low	98%	33%	+0.0138	0.0000 to 0.0371	84%
13	R0305	C5.3 SOPs	SOPs for vaccine temperature monitoring in storage	99%	30%	+0.0115	0.0000 to 0.0459	60%

Applies to: share of facilities where the requirement was asked. At max: share of those meeting it in full. Effect: change in the 0 to 1 score per standard deviation of the requirement. Interval: middle 95% over 200 resamples. Kept in: share of resamples that kept it. Shaded rows are conclusive.

About 60 seconds. Read the shaded rows and then say what the rest of the table is for: a ranking by size so the reader can see how quickly the evidence runs out, not a list of findings. Point at the two profile columns: a requirement almost everyone meets has little room to show an effect.

R0712: the 25 largest effects, 14 to 25

#	Code	Subcategory	What is checked	Applies to	At max	Effect	Interval	Kept in	Concl.
14	R0358	C6.1 Salaries	Staff are paid in full and on time	100%	79%	+0.0104	0.0000 to 0.0397	70%
15	R0311	C5.3 SOPs	SOPs for managing vaccine stock transactions	99%	69%	+0.0097	0.0000 to 0.0402	52%
16	R0327	C5.3 SOPs	SOPs for management of used vials	100%	67%	+0.0069	0.0000 to 0.0355	39%
17	R0233	C4.1 Staffing	At least one member of staff is responsible for annual work planning	100%	62%	+0.0068	0.0000 to 0.0256	55%
18	R0182	C3.2 Data management technology	Stock record form has a field for Presentation (# doses per vial)	100%	71%	+0.0067	0.0000 to 0.0286	53%
19	R0298	C4.3 Knowledge & understanding	Staff know how to dispose of expired temperature monitoring devices	100%	84%	-0.0066	-0.0312 to 0.0000	60%
20	R0357	C6.1 Salaries	Funds for staff salaries are received on time	19%	88%	+0.0066	0.0000 to 0.0265	60%
21	R0307	C5.3 SOPs	SOPs for vaccine storage	99%	70%	+0.0065	0.0000 to 0.0381	38%
22	R0123	C2.3 Other equipment	Waste handlers have an apron	18%	60%	+0.0063	0.0000 to 0.0287	51%
23	R0236	C4.1 Staffing	At least one member of staff is responsible for monitoring ISC performance	100%	72%	+0.0061	0.0000 to 0.0365	35%
24	R0370	C6.2 Funds for operations	Funds received for electricity are sufficient	16%	88%	-0.0059	-0.0253 to 0.0000	52%
25	R0062	C2.1.1 Cold storage equipment	Vaccine refrigerators/freezers have functioning voltage regulators	97%	41%	+0.0057	0.0000 to 0.0272	54%

Below the conclusive rows the intervals include zero, so the order says where the evidence thins out, not which of these matter.

About 30 seconds. Do not read these rows. Say that from here down the intervals include zero, and move on.

R0722: the 25 largest effects, 1 to 13

#	Code	Subcategory	What is checked	Applies to	At max	Effect	Interval	Kept in	Concl.
1	R0037	C1.2 Buildings & facilities	Storage buildings have functional fire extinguishers	100%	21%	+0.0463	0.0168 to 0.0737	100%	✓
2	R0293	C4.3 Knowledge & understanding	Staff know the main types of opened-vial wastage	100%	88%	+0.0385	0.0081 to 0.0629	100%	✓
3	R0290	C4.3 Knowledge & understanding	Staff know the discard time of multi-dose vials	95%	92%	+0.0363	0.0112 to 0.0588	99%	✓
4	R0358	C6.1 Salaries	Staff are paid in full and on time	100%	81%	+0.0333	0.0088 to 0.0589	100%	✓
5	R0129	C2.3 Other equipment	Waste is stored in plastic or metal bins	100%	48%	+0.0297	0.0069 to 0.0586	99%	✓
6	R0257	C4.2 Training	Staff are trained to monitor supply chain performance	100%	57%	+0.0281	0.0000 to 0.0580	91%
7	R0040	C2.1.1 Cold storage equipment	Sufficient cold storage capacity for the expected maximum stock	88%	92%	-0.0173	-0.0427 to 0.0000	86%
8	R0279	C4.3 Knowledge & understanding	Staff know cool packs are prepared in dedicated compartments	47%	50%	+0.0161	0.0000 to 0.0423	88%
9	R0328	C5.3 SOPs	There are safe injection SOPs in the immunization room	99%	30%	+0.0147	0.0000 to 0.0449	78%
10	R0260	C4.2 Training	Staff are trained in routine refrigeration maintenance	100%	55%	-0.0134	-0.0336 to 0.0000	80%
11	R0155	C3.2 Data management technology	The form has fields for temperature alarms	100%	69%	-0.0113	-0.0359 to 0.0000	72%
12	R0272	C4.3 Knowledge & understanding	Staff know the principles of vaccine and dry goods storage	100%	91%	+0.0110	0.0000 to 0.0336	76%
13	R0251	C4.1 Staffing	Staff turnover is low	98%	30%	+0.0104	0.0000 to 0.0350	76%

Applies to: share of facilities where the requirement was asked. At max: share of those meeting it in full. Effect: change in the 0 to 1 score per standard deviation of the requirement. Interval: middle 95% over 200 resamples. Kept in: share of resamples that kept it. Shaded rows are conclusive.

About 60 seconds. Read the shaded rows and then say what the rest of the table is for: a ranking by size so the reader can see how quickly the evidence runs out, not a list of findings. Point at the two profile columns: a requirement almost everyone meets has little room to show an effect.

R0722: the 25 largest effects, 14 to 25

#	Code	Subcategory	What is checked	Applies to	At max	Effect	Interval	Kept in	Concl.
14	R0326	C5.3 SOPs	SOPs for disposal of sharps waste	58%	69%	+0.0104	0.0000 to 0.0355	71%
15	R0236	C4.1 Staffing	At least one member of staff is responsible for monitoring ISC performance	100%	73%	+0.0104	0.0000 to 0.0386	58%
16	R0062	C2.1.1 Cold storage equipment	Vaccine refrigerators/freezers have functioning voltage regulators	100%	44%	+0.0093	0.0000 to 0.0303	66%
17	R0128	C2.3 Other equipment	The waste storage area is covered	100%	62%	+0.0083	0.0000 to 0.0336	62%
18	R0187	C3.2 Data management technology	Stock record form has a field for Transaction type	100%	61%	+0.0083	0.0000 to 0.0302	64%
19	R0275	C4.3 Knowledge & understanding	Staff know how to use the stock management system	100%	75%	+0.0080	0.0000 to 0.0328	64%
20	R0280	C4.3 Knowledge & understanding	Staff know how to prevent freezing during transportation in cold climates	65%	24%	+0.0077	0.0000 to 0.0311	60%
21	R0323	C5.3 SOPs	SOPs for handling diluents	100%	68%	+0.0071	0.0000 to 0.0327	46%
22	R0308	C5.3 SOPs	A vaccine storage emergency contingency plan	100%	61%	+0.0070	0.0000 to 0.0311	50%
23	R0002	C1.1 Utilities & services	A functional landline telephone	100%	17%	+0.0064	0.0000 to 0.0290	57%
24	R0122	C2.3 Other equipment	Waste handlers have boots	16%	65%	+0.0062	0.0000 to 0.0256	52%
25	R0106	C2.3 Other equipment	The waste burial site has clear warning signs	22%	16%	+0.0058	0.0000 to 0.0270	60%

Below the conclusive rows the intervals include zero, so the order says where the evidence thins out, not which of these matter.

About 30 seconds. Do not read these rows. Say that from here down the intervals include zero, and move on.

How the ten least useful requirements were chosen

The bottom of a list is harder to define than the top, because the elastic net sets many effects to exactly zero and nothing orders them. So the rule has three steps, applied to each target separately.

1. Set aside requirements with no room to vary: those met in full by fewer than 5% or more than 95% of the facilities where they apply.
2. Among the rest, take the ten kept in the fewest of the 200 resamples. A requirement the model drops almost every time is one it has no use for.
3. Break ties by the smallest average effect, then by the weakest raw correlation with the target.

About 45 seconds. The WHO team asked for this as a credibility check: a method that ranks sensibly at the top should rank sensibly at the bottom too. Explain why step 1 is needed: a requirement everyone meets would otherwise land at the bottom for the wrong reason. The correlation is only a tie-breaker; it is not what the model uses.

Composite score: the ten requirements the model has least use for

#	Code	Subcategory	What is checked	Applies to	At max	Kept in	Effect	Correlation
1	R0321	C5.3 SOPs	SOPs for the shake test	86%	59%	0%	+0.0000	0.084
2	R0329	C5.3 SOPs	SOPs are written in a local language	97%	68%	0%	+0.0000	0.095
3	R0327	C5.3 SOPs	SOPs for management of used vials	99%	63%	0%	+0.0000	0.096
4	R0311	C5.3 SOPs	SOPs for managing vaccine stock transactions	99%	66%	0%	+0.0000	0.109
5	R0183	C3.2 Data management technology	Stock record form has a field for Quantity	100%	83%	0%	-0.0000	0.041
6	R0386	C6.2 Funds for operations	Facility funds received for storage are sufficient	14%	89%	0%	-0.0000	0.013
7	R0185	C3.2 Data management technology	Stock record form has a field for Expiry date	100%	78%	0%	-0.0000	0.021
8	R0120	C2.3 Other equipment	Waste handlers have protective eyewear	16%	51%	1%	+0.0000	0.078
9	R0389	C6.2 Funds for operations	Funds for maintenance of buildings are received on time	21%	84%	1%	+0.0000	0.038
10	R0393	C6.2 Funds for operations	Funds for maintenance of cold chain equipment are received on time	16%	84%	1%	+0.0000	0.042

1 is the least used. Kept in: share of the 200 resamples that kept the requirement at all. Correlation: absolute raw correlation with the target, used only to break ties.

About 40 seconds. Read three or four rows, enough to show the pattern. Several are written procedures or record-form fields that most facilities have, and some are funding items that apply to only one facility in six, a theme that returns in the causal section.

R0712: the ten requirements the model has least use for

#	Code	Subcategory	What is checked	Applies to	At max	Kept in	Effect	Correlation
1	R0183	C3.2 Data management technology	Stock record form has a field for Quantity	100%	86%	0%	+0.0000	0.114
2	R0010	C1.1 Utilities & services	Hand washing or hand sanitizing facilities	100%	90%	0%	+0.0000	0.117
3	R0316	C5.3 SOPs	SOPs for packing insulated containers	77%	66%	0%	+0.0000	0.158
4	R0124	C2.3 Other equipment	Waste handlers have trousers	18%	48%	0%	-0.0000	0.078
5	R0329	C5.3 SOPs	SOPs are written in a local language	98%	72%	0%	+0.0000	0.209
6	R0309	C5.3 SOPs	SOPs/guidelines for routine maintenance of refrigeration equipment	97%	56%	0%	-0.0000	0.100
7	R0211	C3.2 Data management technology	Receipt form has a field for VVM status of vaccines received	99%	94%	1%	-0.0000	0.060
8	R0117	C2.3 Other equipment	Waste handlers have protective gloves	35%	74%	1%	-0.0000	0.074
9	R0390	C6.2 Funds for operations	Facility funds received for maintenance of buildings are sufficient	21%	84%	2%	-0.0000	0.043
10	R0260	C4.2 Training	Staff are trained in routine refrigeration maintenance	97%	54%	2%	-0.0000	0.031

1 is the least used. Kept in: share of the 200 resamples that kept the requirement at all. Correlation: absolute raw correlation with the target, used only to break ties.

About 40 seconds. Read three or four rows, enough to show the pattern. Several are written procedures or record-form fields that most facilities have, and some are funding items that apply to only one facility in six, a theme that returns in the causal section.

R0722: the ten requirements the model has least use for

#	Code	Subcategory	What is checked	Applies to	At max	Kept in	Effect	Correlation
1	R0253	C4.2 Training	Staff are trained in annual work planning	100%	47%	0%	+0.0000	0.055
2	R0185	C3.2 Data management technology	Stock record form has a field for Expiry date	100%	82%	0%	+0.0000	0.089
3	R0329	C5.3 SOPs	SOPs are written in a local language	98%	72%	0%	+0.0000	0.171
4	R0307	C5.3 SOPs	SOPs for vaccine storage	100%	70%	0%	+0.0000	0.194
5	R0136	C3.1 General IT equipment	Supply chain staff have work mobile phones	100%	53%	0%	+0.0000	0.116
6	R0250	C4.1 Staffing	Two or more members of staff are assigned to manage immunization waste	100%	61%	0%	+0.0000	0.038
7	R0264	C4.2 Training	Staff are trained in planning vaccine collection/distribution and/or outr…	65%	69%	1%	+0.0000	0.064
8	R0004	C1.1 Utilities & services	A reliable internet connection	100%	45%	1%	+0.0000	0.099
9	R0400	C6.2 Funds for operations	Funds for vaccine transportation or outreach are received in full	24%	87%	1%	-0.0000	0.085
10	R0210	C3.2 Data management technology	Receipt form has a field for Quantity of vaccines received	100%	81%	1%	+0.0000	0.048

1 is the least used. Kept in: share of the 200 resamples that kept the requirement at all. Correlation: absolute raw correlation with the target, used only to break ties.

About 40 seconds. Read three or four rows, enough to show the pattern. Several are written procedures or record-form fields that most facilities have, and some are funding items that apply to only one facility in six, a theme that returns in the causal section.

What I will cover

1. The question
2. The data and the filters
3. Targets and models
4. What drives quality
5. Stress-testing the answer
6. Causal estimates
7. Subcategory scores
8. What it means and what comes next

5 seconds. Move the highlight and go.

Shuffling the answers finds almost nothing, so the real counts mean something

• The target scores were shuffled at random across facilities and the whole method was run again, ten times per target.
• Real counts: 2, 4 and 5 conclusive requirements. On shuffled scores: one false alarm in thirty runs, on the composite, and none on the components.
• The largest effect any shuffled run produced is a noise ceiling: 0.0063, 0.0238 and 0.0192. Every conclusive effect clears its ceiling.

About 60 seconds. If the method found requirements out of pure noise, then finding them in the real data would prove nothing. It found one in thirty tries. The noise ceiling is a second, gentler bar: anything below it could have come from noise. Everything conclusive is above it, and on R0722 so is one more, training to monitor supply chain performance.

Removing the filled-in values changes little

Where a requirement does not apply, the model was given the column’s mean. Refitting only on facilities where the leading requirements were actually scored:

• All 11 leading effects keep their direction, and 8 stay conclusive.
• The three that lose the flag are the discard time on the composite, and training and fire extinguishers on R0712. Their effects shrink but keep their sign.
• The check keeps 1,810 of 2,258 facilities on the composite, 1,066 of 1,077 on R0712 and 1,039 of 1,098 on R0722.

About 45 seconds. This defends the one compromise in the data preparation. Be straight that the discard-time requirement on the composite halves without its filled rows; it stays positive, and it comes back strongly in the causal section.

Six other model families name most of the same requirements

Each family ranks requirements its own way, by how much shuffling a column hurts its predictions. A requirement is named when it lands in that family’s top twenty.

Code	Subcategory	What is checked	Composite	R0712	R0722
R0290	C4.3 Knowledge & understanding	Staff know the discard time of multi-dose vials	5 of 6	6 of 6	5 of 6
R0040	C2.1.1 Cold storage equipment	Sufficient cold storage capacity for the expected maximum stock	5 of 6	3 of 6	4 of 6
R0156	C3.2 Data management technology	The temperature record form is graphical	6 of 6	3 of 6	2 of 6
R0322	C5.3 SOPs	SOPs for the multi-dose vial policy	3 of 6	2 of 6	2 of 6
R0257	C4.2 Training	Staff are trained to monitor supply chain performance	3 of 6	3 of 6	4 of 6
R0037	C1.2 Buildings & facilities	Storage buildings have functional fire extinguishers	2 of 6	3 of 6	5 of 6
R0272	C4.3 Knowledge & understanding	Staff know the principles of vaccine and dry goods storage	–	2 of 6	3 of 6
R0293	C4.3 Knowledge & understanding	Staff know the main types of opened-vial wastage	–	–	6 of 6
R0358	C6.1 Salaries	Staff are paid in full and on time	–	–	5 of 6
R0129	C2.3 Other equipment	Waste is stored in plastic or metal bins	–	3 of 6	4 of 6

– means named by fewer than two families. The six families are ridge, the fractional logit, boosted trees, bagged trees, randomised-split trees and XGBoost.

About 50 seconds. Boosted trees and XGBoost never see a filled-in value, so a requirement they name cannot be an artefact of the filling. Land on the pattern: the discard-time requirement is named by five or six families on every target; the odd ones, fire extinguishers and storage principles on R0712, get the fewest votes.

Why meeting a requirement can look like it reduces Quality score

Take R0040, having enough cold storage. 90% of facilities meet it. Those that do are asked the multi-dose vial questions 11 points more often, and they average 0.890 against 0.938 for those that do not.

About 45 seconds, and it is the idea the whole section turns on. Walk the four boxes left to right. Box one points back to the grid: the three checks that need a session in progress. Box two: those are the questions facilities actually fail, and you can only fail a question you were asked. Box three: a facility with spare fridge capacity is a facility running sessions. Box four: so the better equipped facility sits the harder test and can score lower for a reason that has nothing to do with its vaccines.

A short list keeps most of the signal, and the top 25 beats all 172

Two questions keep 51% of the composite’s fit, four keep 67% on R0712, five keep 81% on R0722. The top 25 beats all 172 because a model with 147 weak columns removed can use a lighter penalty.

About 55 seconds. Pale is everything the assessment asks, solid is the handful I named, and the middle shade is the top 25. Say the caveat yourself: the lists were chosen on the same facilities the refits are split from, so these are upper bounds on what a list fixed in advance would achieve.

What I will cover

1. The question
2. The data and the filters
3. Targets and models
4. What drives quality
5. Stress-testing the answer
6. Causal estimates
7. Subcategory scores
8. What it means and what comes next

5 seconds. Move the highlight and go.

Double machine learning

Take one requirement, say staff know the discard time, as the treatment. The quality score is the outcome. Every other column is a control.

1. Predict the outcome from the controls and keep what is left over, the residual.
2. Predict the treatment from the controls and keep its residual.
3. Fit a straight line through the two residuals. Its slope is the effect.

This association does not come from any control, hence causal effect.

Unit: the change in the score when a facility goes from not meeting the requirement to meeting it in full.

About 90 seconds. This is the one slide that has to land. Use the discard-time example all the way through. Steps 1 and 2 strip out everything the other 171 columns can explain; step 3 asks whether what is left still moves together. Both predictions are made by elastic nets, the same kind of model as before. Then the unit: not a coefficient per standard deviation any more, but the gap between a facility whose staff know the rule and an otherwise similar facility whose staff do not.

How a result counts, and how country is handled

• q-value. A p-value corrected for testing about 170 columns at once. Conclusive means q below 0.05.
• Thin evidence. A column asked of under a quarter of facilities, or answered the same way by over 95% of them, is shown but never flagged.

About 80 seconds. Three definitions and one decision. Leaving country out lets differences between countries, in policies, supply systems and assessment teams, show up as requirement effects. Centring every column within its country throws those away and asks only whether a facility that meets the requirement does better than its neighbours in the same country. That is the fairer test and the part a country programme can act on, which is why the verdict rests on it, and why effects shrink when it is applied.

Seven effects survive adjustment, and three survive every check

Target	Code	Subcategory	What is checked	Effect (points)	q
QU	R0040	C2.1.1 Cold storage equipment	Sufficient cold storage capacity for the expected maximum stock	-6.2	0.005
R0712	R0290	C4.3 Knowledge & understanding	Staff know the discard time of multi-dose vials	+27.4	0.029
R0712	R0272	C4.3 Knowledge & understanding	Staff know the principles of vaccine and dry goods storage	+19.6	0.029
R0722	R0290	C4.3 Knowledge & understanding	Staff know the discard time of multi-dose vials	+28.9	<0.001
R0722	R0293	C4.3 Knowledge & understanding	Staff know the main types of opened-vial wastage	+24.6	0.001
R0722	R0358	C6.1 Salaries	Staff are paid in full and on time	+17.8	0.040
R0722	R0260	C4.2 Training	Staff are trained in routine refrigeration maintenance	-20.4	0.040

Effect: change in points out of 100 from not met to met in full. q below 0.05 means conclusive. Shaded rows also pass every robustness check: three requirements, with the discard time appearing for both targets.

About 90 seconds. Read the three shaded rows first: knowing the discard time on both components, knowing the storage principles, knowing the types of opened-vial wastage. Each keeps about half its size within countries: half of the association came from differences between countries and half from facilities of the same country. Then the three that fail: cold storage passes every statistical check but has no route to a vial and is explained by exposure; salaries keep 30% within countries and are the effect a hidden factor could move most; the negative training effect collapses from 20 points to 7 as soon as the other training requirements leave the controls.

Composite score: the screening’s top 25 under the causal estimate, 1 to 13

#	Code	Subcategory	What is checked	Screening effect	Causal effect (points)	q	Causal rank	Concl.
1	R0040	C2.1.1 Cold storage equipment	Sufficient cold storage capacity for the expected maximum stock	-0.0114	-6.2	0.005	1 of 116	✓
2	R0290	C4.3 Knowledge & understanding	Staff know the discard time of multi-dose vials	+0.0102	+5.6	0.110	3 of 116
3	R0322	C5.3 SOPs	SOPs for the multi-dose vial policy	+0.0061	+2.4	0.689	31 of 116
4	R0156	C3.2 Data management technology	The temperature record form is graphical	-0.0061	-3.6	0.333	15 of 116
5	R0257	C4.2 Training	Staff are trained to monitor supply chain performance	+0.0057	+3.4	0.333	17 of 116
6	R0130	C2.3 Other equipment	Bins are colour coded for different types of waste	+0.0054	+3.9	0.333	8 of 116
7	R0251	C4.1 Staffing	Staff turnover is low	+0.0050	+4.4	0.330	6 of 116
8	R0279	C4.3 Knowledge & understanding	Staff know cool packs are prepared in dedicated compartments	+0.0046	+3.2	0.333	19 of 116
9	R0037	C1.2 Buildings & facilities	Storage buildings have functional fire extinguishers	+0.0045	+3.0	0.382	22 of 116
10	R0136	C3.1 General IT equipment	Supply chain staff have work mobile phones	+0.0044	+5.5	0.330	4 of 116
11	R0328	C5.3 SOPs	There are safe injection SOPs in the immunization room	+0.0042	+3.8	0.333	10 of 116
12	R0292	C4.3 Knowledge & understanding	Staff know how to calculate the unopened vial wastage rate	+0.0035	+2.0	0.494	42 of 116
13	R0275	C4.3 Knowledge & understanding	Staff know how to use the stock management system	+0.0034	+2.3	0.424	36 of 116

The screening’s top 25, in the screening’s order. Causal rank: position among the 116 requirements with enough evidence, by the size of the adjusted effect. Shaded rows are conclusive after adjustment (q below 0.05).

About 45 seconds. The point of the slide is agreement at the top and a reshuffle below it. Say where the screening’s conclusive rows land, and point at one or two that rise or fall a long way. Do not read the table.

Composite score: the screening’s top 25 under the causal estimate, 14 to 25

#	Code	Subcategory	What is checked	Screening effect	Causal effect (points)	q	Causal rank	Concl.
14	R0122	C2.3 Other equipment	Waste handlers have boots	+0.0030	+7.7		thin
15	R0064	C2.1.1 Cold storage equipment	Vaccine refrigerator/freezers have door open alarms	-0.0030	-3.1	0.851	20 of 116
16	R0269	C4.3 Knowledge & understanding	Staff can correctly read a 30DTR	+0.0027	+2.4	0.591	34 of 116
17	R0186	C3.2 Data management technology	Stock record form has a field for VVM status	-0.0025	-1.2	0.876	63 of 116
18	R0298	C4.3 Knowledge & understanding	Staff know how to dispose of expired temperature monitoring devices	-0.0022	-2.4	0.333	32 of 116
19	R0272	C4.3 Knowledge & understanding	Staff know the principles of vaccine and dry goods storage	+0.0019	+2.9	0.591	24 of 116
20	R0212	C3.2 Data management technology	Receipt form has a field for Freeze indicator status of vaccines received	-0.0018	-1.4	0.868	54 of 116
21	R0154	C3.2 Data management technology	The form has fields for entering AM and PM temperatures	+0.0018	+3.8	0.595	9 of 116
22	R0210	C3.2 Data management technology	Receipt form has a field for Quantity of vaccines received	-0.0017	-2.0	0.689	43 of 116
23	R0326	C5.3 SOPs	SOPs for disposal of sharps waste	+0.0016	+0.3	0.987	97 of 116
24	R0073	C2.1.1 Cold storage equipment	Sufficient coolant pack storage capacity for daily demand	-0.0016	-1.2	0.851	61 of 116
25	R0077	C2.2.1 Vehicles	Transport is always available for scheduled vaccine distribution, collect…	+0.0015	+3.7	0.802	13 of 116

Thin: too few facilities on one side of the requirement to rank it.

About 45 seconds. The point of the slide is agreement at the top and a reshuffle below it. Say where the screening’s conclusive rows land, and point at one or two that rise or fall a long way. Do not read the table.

R0712: the screening’s top 25 under the causal estimate, 1 to 13

#	Code	Subcategory	What is checked	Screening effect	Causal effect (points)	q	Causal rank	Concl.
1	R0290	C4.3 Knowledge & understanding	Staff know the discard time of multi-dose vials	+0.0403	+27.4	0.029	3 of 112	✓
2	R0257	C4.2 Training	Staff are trained to monitor supply chain performance	+0.0387	+22.0	0.070	6 of 112
3	R0037	C1.2 Buildings & facilities	Storage buildings have functional fire extinguishers	+0.0289	+13.8	0.413	18 of 112
4	R0272	C4.3 Knowledge & understanding	Staff know the principles of vaccine and dry goods storage	+0.0263	+19.6	0.029	8 of 112	✓
5	R0156	C3.2 Data management technology	The temperature record form is graphical	-0.0227	-14.8	0.298	15 of 112
6	R0270	C4.3 Knowledge & understanding	Staff know the correct storage temperature for each vaccine	+0.0194	+33.3	0.680	1 of 112
7	R0130	C2.3 Other equipment	Bins are colour coded for different types of waste	+0.0187	+20.1	0.119	7 of 112
8	R0328	C5.3 SOPs	There are safe injection SOPs in the immunization room	+0.0185	+13.6	0.599	19 of 112
9	R0002	C1.1 Utilities & services	A functional landline telephone	+0.0178	+15.2	0.183	14 of 112
10	R0279	C4.3 Knowledge & understanding	Staff know cool packs are prepared in dedicated compartments	+0.0175	+11.3	0.298	28 of 112
11	R0129	C2.3 Other equipment	Waste is stored in plastic or metal bins	+0.0156	+12.3	0.213	23 of 112
12	R0251	C4.1 Staffing	Staff turnover is low	+0.0138	+16.6	0.310	9 of 112
13	R0305	C5.3 SOPs	SOPs for vaccine temperature monitoring in storage	+0.0115	+11.0	0.870	30 of 112

The screening’s top 25, in the screening’s order. Causal rank: position among the 112 requirements with enough evidence, by the size of the adjusted effect. Shaded rows are conclusive after adjustment (q below 0.05).

About 45 seconds. The point of the slide is agreement at the top and a reshuffle below it. Say where the screening’s conclusive rows land, and point at one or two that rise or fall a long way. Do not read the table.

R0712: the screening’s top 25 under the causal estimate, 14 to 25

#	Code	Subcategory	What is checked	Screening effect	Causal effect (points)	q	Causal rank	Concl.
14	R0358	C6.1 Salaries	Staff are paid in full and on time	+0.0104	+10.2	0.573	32 of 112
15	R0311	C5.3 SOPs	SOPs for managing vaccine stock transactions	+0.0097	+12.9	0.754	21 of 112
16	R0327	C5.3 SOPs	SOPs for management of used vials	+0.0069	+8.3	0.795	39 of 112
17	R0233	C4.1 Staffing	At least one member of staff is responsible for annual work planning	+0.0068	+9.0	0.763	36 of 112
18	R0182	C3.2 Data management technology	Stock record form has a field for Presentation (# doses per vial)	+0.0067	+6.6	0.718	50 of 112
19	R0298	C4.3 Knowledge & understanding	Staff know how to dispose of expired temperature monitoring devices	-0.0066	-11.7	0.183	24 of 112
20	R0357	C6.1 Salaries	Funds for staff salaries are received on time	+0.0066	+17.3		thin
21	R0307	C5.3 SOPs	SOPs for vaccine storage	+0.0065	+7.3	0.798	43 of 112
22	R0123	C2.3 Other equipment	Waste handlers have an apron	+0.0063	+12.1		thin
23	R0236	C4.1 Staffing	At least one member of staff is responsible for monitoring ISC performance	+0.0061	+1.7	0.942	96 of 112
24	R0370	C6.2 Funds for operations	Funds received for electricity are sufficient	-0.0059	-11.8		thin
25	R0062	C2.1.1 Cold storage equipment	Vaccine refrigerators/freezers have functioning voltage regulators	+0.0057	+7.2	0.395	44 of 112

Thin: too few facilities on one side of the requirement to rank it.

About 45 seconds. The point of the slide is agreement at the top and a reshuffle below it. Say where the screening’s conclusive rows land, and point at one or two that rise or fall a long way. Do not read the table.

R0722: the screening’s top 25 under the causal estimate, 1 to 13

#	Code	Subcategory	What is checked	Screening effect	Causal effect (points)	q	Causal rank	Concl.
1	R0037	C1.2 Buildings & facilities	Storage buildings have functional fire extinguishers	+0.0463	+18.2	0.106	6 of 113
2	R0293	C4.3 Knowledge & understanding	Staff know the main types of opened-vial wastage	+0.0385	+24.6	0.001	2 of 113	✓
3	R0290	C4.3 Knowledge & understanding	Staff know the discard time of multi-dose vials	+0.0363	+28.9	<0.001	1 of 113	✓
4	R0358	C6.1 Salaries	Staff are paid in full and on time	+0.0333	+17.8	0.040	9 of 113	✓
5	R0129	C2.3 Other equipment	Waste is stored in plastic or metal bins	+0.0297	+14.4	0.085	20 of 113
6	R0257	C4.2 Training	Staff are trained to monitor supply chain performance	+0.0281	+17.4	0.179	12 of 113
7	R0040	C2.1.1 Cold storage equipment	Sufficient cold storage capacity for the expected maximum stock	-0.0173	-17.2	0.156	13 of 113
8	R0279	C4.3 Knowledge & understanding	Staff know cool packs are prepared in dedicated compartments	+0.0161	+12.0	0.158	27 of 113
9	R0328	C5.3 SOPs	There are safe injection SOPs in the immunization room	+0.0147	+15.1	0.175	19 of 113
10	R0260	C4.2 Training	Staff are trained in routine refrigeration maintenance	-0.0134	-20.4	0.040	3 of 113	✓
11	R0155	C3.2 Data management technology	The form has fields for temperature alarms	-0.0113	-17.4	0.245	11 of 113
12	R0272	C4.3 Knowledge & understanding	Staff know the principles of vaccine and dry goods storage	+0.0110	+11.6	0.389	30 of 113
13	R0251	C4.1 Staffing	Staff turnover is low	+0.0104	+16.7	0.156	16 of 113

The screening’s top 25, in the screening’s order. Causal rank: position among the 113 requirements with enough evidence, by the size of the adjusted effect. Shaded rows are conclusive after adjustment (q below 0.05).

About 45 seconds. The point of the slide is agreement at the top and a reshuffle below it. Say where the screening’s conclusive rows land, and point at one or two that rise or fall a long way. Do not read the table.

R0722: the screening’s top 25 under the causal estimate, 14 to 25

#	Code	Subcategory	What is checked	Screening effect	Causal effect (points)	q	Causal rank	Concl.
14	R0326	C5.3 SOPs	SOPs for disposal of sharps waste	+0.0104	+11.8	0.753	29 of 113
15	R0236	C4.1 Staffing	At least one member of staff is responsible for monitoring ISC performance	+0.0104	+3.9	0.863	79 of 113
16	R0062	C2.1.1 Cold storage equipment	Vaccine refrigerators/freezers have functioning voltage regulators	+0.0093	+8.3	0.316	46 of 113
17	R0128	C2.3 Other equipment	The waste storage area is covered	+0.0083	+8.2	0.413	47 of 113
18	R0187	C3.2 Data management technology	Stock record form has a field for Transaction type	+0.0083	+11.4	0.320	32 of 113
19	R0275	C4.3 Knowledge & understanding	Staff know how to use the stock management system	+0.0080	+7.6	0.527	55 of 113
20	R0280	C4.3 Knowledge & understanding	Staff know how to prevent freezing during transportation in cold climates	+0.0077	+15.9	0.293	18 of 113
21	R0323	C5.3 SOPs	SOPs for handling diluents	+0.0071	+18.3	0.293	5 of 113
22	R0308	C5.3 SOPs	A vaccine storage emergency contingency plan	+0.0070	+8.2	0.753	48 of 113
23	R0002	C1.1 Utilities & services	A functional landline telephone	+0.0064	+8.1	0.527	49 of 113
24	R0122	C2.3 Other equipment	Waste handlers have boots	+0.0062	+26.4		thin
25	R0106	C2.3 Other equipment	The waste burial site has clear warning signs	+0.0058	+24.4		thin

Thin: too few facilities on one side of the requirement to rank it.

About 45 seconds. The point of the slide is agreement at the top and a reshuffle below it. Say where the screening’s conclusive rows land, and point at one or two that rise or fall a long way. Do not read the table.

The two rankings agree on what matters and differ below it

	Composite	R0712	R0722
Requirements given the same sign	78%	82%	78%
Of the 25 largest effects, in both lists	14	13	13
Conclusive in the screening	2	4	5
Conclusive after adjustment	1	2	4
Conclusive under both	1	2	3

The names that drop out after adjustment are fire extinguishers, training to monitor performance, waste bins and, on the composite, the discard time, which stays positive but misses the threshold.

About 50 seconds. Moderate agreement on the order, strong agreement on the names that matter. Adjustment added one name, a negative effect of refrigeration maintenance training on R0722, which the controls check removes again. The names that drop out are the ones the mechanism reading had already set apart as marks of a well-run facility.

What I will cover

1. The question
2. The data and the filters
3. Targets and models
4. What drives quality
5. Stress-testing the answer
6. Causal estimates
7. Subcategory scores
8. What it means and what comes next

5 seconds. Move the highlight and go.

Fourteen subcategory scores, built by a colleague and still provisional

The EVM framework groups its requirements into subcategories such as C4.3 Knowledge and understanding. Jayeed built one score per subcategory and facility:

• from the requirement-level scores inside each subcategory

Requires confirmation with WHO.

About 50 seconds. Credit the construction and state the caveat in the same breath. The reason the section is here anyway: if the analysis gives the same answer at this coarser level, that is evidence the requirement-level answer is not an artefact of 170 narrow columns.

The fourteen scores that stand in for the 170 requirements

Subcategory	Applies to	Mean score	At the maximum
C1.1 Utilities & services	100%	0.79	10%
C1.2 Buildings & facilities	100%	0.37	16%
C2.1.1 Cold storage equipment	100%	0.69	5%
C2.2.1 Vehicles	37%	0.90	81%
C2.2.2 Insulated containers	90%	0.91	72%
C2.3 Other equipment	100%	0.85	16%
C3.1 General IT equipment	100%	0.73	59%
C3.2 Data management technology	100%	0.79	10%
C4.1 Staffing	100%	0.68	3%
C4.2 Training	100%	0.66	10%
C4.3 Knowledge & understanding	100%	0.80	4%
C5.3 SOPs	100%	0.57	12%
C6.1 Salaries	98%	0.78	76%
C6.2 Funds for operations	33%	0.84	66%

Each score runs from 0 to 1. Vehicles and funds for operations apply to about a third of facilities; the rest apply almost everywhere.

About 35 seconds. Two things to point at: knowledge and understanding and staffing are smooth scores with hundreds of distinct values, while salaries takes only seven, and three quarters of facilities sit at its maximum. The same mean filling as before is used where a score does not apply.

Fourteen scores explain less than 170 requirements, and the same model wins

R² on unseen facilities	Composite	R0712	R0722
Elastic net on the 14 subcategory scores	0.022	0.169	0.144
Elastic net on the 170 requirements	0.051	0.191	0.180

On R0712 the fractional logit edges the elastic net by less than the variation between splits. That counts as a tie, so the elastic net is used for all three targets and the two levels can be compared.

About 55 seconds. Fourteen averages carry less information than the 170 items they summarise, so the fit drops; the ordering of the targets does not change. The tree models still memorise. Country matters even more at this level, and the unseen-country figures are again the honest ones.

Composite score: buildings, IT and written procedures help; cold storage equipment does not

#	Subcategory	Effect	Interval	Kept in	Conclusive
1	C1.2 Buildings & facilities	+0.0119	0.0051 to 0.0183	100%	✓
2	C3.1 General IT equipment	+0.0099	0.0027 to 0.0189	100%	✓
3	C2.1.1 Cold storage equipment	-0.0085	-0.0144 to -0.0025	100%	✓
4	C5.3 SOPs	+0.0084	0.0019 to 0.0161	98%	✓
5	C4.3 Knowledge & understanding	+0.0058	0.0000 to 0.0145	93%
6	C4.2 Training	+0.0036	0.0000 to 0.0108	74%
7	C2.3 Other equipment	+0.0023	0.0000 to 0.0085	64%
8	C2.2.1 Vehicles	-0.0016	-0.0073 to 0.0031	64%
9	C3.2 Data management technology	-0.0016	-0.0077 to 0.0015	58%
10	C6.1 Salaries	+0.0014	-0.0000 to 0.0072	54%
11	C6.2 Funds for operations	+0.0013	0.0000 to 0.0070	50%
12	C1.1 Utilities & services	-0.0003	-0.0044 to 0.0033	35%
13	C2.2.2 Insulated containers	+0.0002	-0.0017 to 0.0041	30%
14	C4.1 Staffing	+0.0001	-0.0016 to 0.0022	18%

All fourteen subcategories, ranked by the size of the average effect. Effect: change in the 0 to 1 score per standard deviation of the subcategory score. Shaded rows are conclusive.

About 45 seconds. Read the shaded rows and the sign of cold storage equipment, which is negative on all three targets, the same artefact as the cold-storage requirement. Knowledge and understanding is conclusive on both components but just misses on the composite.

R0712: written procedures, knowledge and buildings carry it

#	Subcategory	Effect	Interval	Kept in	Conclusive
1	C5.3 SOPs	+0.0626	0.0294 to 0.0974	100%	✓
2	C4.3 Knowledge & understanding	+0.0558	0.0280 to 0.0872	100%	✓
3	C1.2 Buildings & facilities	+0.0469	0.0215 to 0.0700	100%	✓
4	C4.2 Training	+0.0266	0.0000 to 0.0599	92%
5	C6.1 Salaries	+0.0264	0.0000 to 0.0590	96%
6	C2.3 Other equipment	+0.0066	0.0000 to 0.0289	55%
7	C3.1 General IT equipment	-0.0045	-0.0231 to 0.0000	38%
8	C2.1.1 Cold storage equipment	-0.0036	-0.0217 to 0.0000	38%
9	C2.2.2 Insulated containers	+0.0031	-0.0000 to 0.0199	36%
10	C2.2.1 Vehicles	+0.0021	0.0000 to 0.0166	30%
11	C4.1 Staffing	+0.0017	0.0000 to 0.0141	24%
12	C1.1 Utilities & services	+0.0006	-0.0019 to 0.0112	14%
13	C6.2 Funds for operations	+0.0006	-0.0069 to 0.0103	23%
14	C3.2 Data management technology	-0.0002	-0.0072 to 0.0050	13%

All fourteen subcategories, ranked by the size of the average effect. Effect: change in the 0 to 1 score per standard deviation of the subcategory score. Shaded rows are conclusive.

About 45 seconds. Read the shaded rows and the sign of cold storage equipment, which is negative on all three targets, the same artefact as the cold-storage requirement. Knowledge and understanding is conclusive on both components but just misses on the composite.

R0722: knowledge, buildings and salaries carry it

#	Subcategory	Effect	Interval	Kept in	Conclusive
1	C4.3 Knowledge & understanding	+0.0745	0.0473 to 0.1020	100%	✓
2	C1.2 Buildings & facilities	+0.0709	0.0413 to 0.0964	100%	✓
3	C6.1 Salaries	+0.0509	0.0270 to 0.0775	100%	✓
4	C5.3 SOPs	+0.0237	0.0000 to 0.0511	91%
5	C2.3 Other equipment	+0.0150	0.0000 to 0.0426	80%
6	C4.2 Training	+0.0067	0.0000 to 0.0288	60%
7	C4.1 Staffing	+0.0043	0.0000 to 0.0256	43%
8	C1.1 Utilities & services	+0.0039	0.0000 to 0.0229	41%
9	C2.1.1 Cold storage equipment	-0.0026	-0.0168 to 0.0000	32%
10	C2.2.1 Vehicles	-0.0024	-0.0172 to 0.0000	28%
11	C6.2 Funds for operations	+0.0018	0.0000 to 0.0159	27%
12	C3.2 Data management technology	+0.0008	0.0000 to 0.0112	12%
13	C2.2.2 Insulated containers	-0.0008	-0.0091 to 0.0000	14%
14	C3.1 General IT equipment	-0.0000	0.0000 to 0.0000	3%

All fourteen subcategories, ranked by the size of the average effect. Effect: change in the 0 to 1 score per standard deviation of the subcategory score. Shaded rows are conclusive.

About 45 seconds. Read the shaded rows and the sign of cold storage equipment, which is negative on all three targets, the same artefact as the cold-storage requirement. Knowledge and understanding is conclusive on both components but just misses on the composite.

The subcategory results pass the same stress tests

• Shuffled scores. Thirty shuffled runs produced no conclusive subcategory, and every conclusive effect clears its noise ceiling.
• Filled-in values. All ten conclusive effects keep their direction and their flag.

About 70 seconds. The same four checks as at requirement level. The noise ceilings are 0.0063, 0.0194 and 0.0160. The disagreement on the composite is worth saying out loud: with only fourteen columns the elastic net and the tree families spread their choices differently, and the composite is where every method has least to work with. Buildings are named by only two families on R0712. Almost no facility has a filled cell at this level, which is why the filled-value check changes nothing.

The causal estimate at subcategory level tells the same story

Target	Subcategory	Effect (points)	q	Conclusive	Conclusive in the screening
QU	C2.1.1 Cold storage equipment	-6	0.227		✓
QU	C3.1 General IT equipment	+5	0.227		✓
QU	C1.2 Buildings & facilities	+5	0.164		✓
R0712	C5.3 SOPs	+24	0.009	✓	✓
R0712	C4.3 Knowledge & understanding	+51	0.053		✓
R0712	C4.2 Training	+17	0.061
R0712	C1.2 Buildings & facilities	+18	0.103		✓
R0722	C4.3 Knowledge & understanding	+67	0.016	✓	✓
R0722	C1.2 Buildings & facilities	+26	0.005	✓	✓
R0722	C6.1 Salaries	+21	0.005	✓	✓

Effect: change in points out of 100 when the subcategory goes from its lowest possible score to its highest. A whole range is a large move, so a quarter of the knowledge range, 13 points on R0712 and 17 on R0722, is the realistic figure to quote.

About 60 seconds. On R0722 the three conclusive subcategories are the screening’s three. On R0712 only written procedures clears the line; knowledge has the largest effect, 51 points over its full range, but its q-value sits at 0.053, just outside. On the composite nothing is conclusive, and cold storage equipment is again the largest effect, downwards. One level up, the answer is the same: what staff know carries the two vial targets.

What I will cover

1. The question
2. The data and the filters
3. Targets and models
4. What drives quality
5. Stress-testing the answer
6. Causal estimates
7. Subcategory scores
8. What it means and what comes next

5 seconds. Move the highlight and go.

What the evidence supports

• Out of 170 requirements across 77 countries, quality at the last mile comes down to a handful, and what staff know seems significant: the discard time of multi-dose vials, the principles of vaccine storage, and the types of opened-vial wastage, as those three pass every test in the analysis.
• The composite score is a weak target. Its one conclusive effect, cold storage capacity, is a measurement artefact: better equipped facilities sit the harder test.

About 70 seconds. Four sentences, slowly. The first is the headline. The second is why it can be believed: conclusive after adjustment, about half the effect intact within countries, beyond the reach of any measured stand-in for a hidden factor, unmoved when their neighbours leave the controls, and each with a direct route to an opened vial. The third is what the causal section removed. The fourth is the honest reading of the composite.

What the evidence does not support, and what comes next

The causal estimates hold only if nothing unmeasured drives both a requirement and quality.

Article Suggestion:

• 1. Screening and causal effect analysis on the requirements dataset for R0712 and R0722.
• 2. The same analysis on the subcategories dataset.
• 3. Treat the other 4 quality components as anomaly detection problems and identify requirements and categories that lead to NOT HAVING a good quality component score.

About 45 seconds. End on the next step with the new assessments: a list fixed in advance and tested on countries that had no say in it is the one test that would settle the question. If asked about the controls: the follow-up chooses them for each requirement from the structure of the EVM framework, so that consequences of a requirement are never held fixed.

Thank you. Questions welcome.

Backup: choosing the model

• Eleven families over the same twenty splits: ordinary least squares, ridge, LASSO, elastic net, fractional logit, a gamma model on the shortfall from full marks, boosted trees, bagged trees, randomised-split trees, XGBoost, and the mean of the training data as a floor.
• Plain least squares is the weakest linear model on unseen facilities for every target, so the penalties earn their place.
• The elastic net and the LASSO tie on every target; the elastic net is reported because it handles groups of near-identical requirements more gracefully.

For questions on model choice. One line on AIC if pushed: a criterion computed inside the sample rewards exactly the hard fitting that repeated splits exist to discount.

Backup: how the values that do not exist are handled

• Some requirements do not apply to a facility, and a linear model still needs a number in that cell.
• Two options were compared: fill with the column’s mean, or fill and add a marker column saying the value was filled.
• The markers fit slightly better and double the number of columns, and a marker coefficient is not something a manager can act on, so the simple fill is kept.
• Two checks limit the cost: the tree families that need no filling name the same requirements, and refitting on fully scored facilities keeps every leading direction.

For questions on missing data. Be straight: markers win narrowly on fit and lose on usefulness.

Backup: quality as four ordered bands

• The composite score was cut at 0.6, 0.8 and 0.95, and 58% of facilities land in the top band.
• Always naming the biggest band beats the ordinal model on plain accuracy, 0.582 against 0.496, and the model’s agreement with the truth beyond chance is 0.092, weak by any usual standard.
• The top twenty requirements change under banding while no measure of fit improves, so the analysis keeps the continuous score.

For questions about whether the continuous score is the right framing.