Pilot script and observation sheet
Version 1.1.0 · Flagship Weight Management 1.4.0 · AI-assisted source verification; human review none recorded.
Status: not yet run. Not yet run. Human review and the pilot are pending until real participants complete the tasks and a named person records the results. No timing, hesitation or misreading on this page comes from a participant. The tasks below are what to ask; the sheet at the end is where a person records what happened.
Ground rules
- Take part only with consent. Record a participant code, never a name, and no health information.
- Do not coach. If a participant is stuck for two minutes, record where and what they tried, then give the smallest hint and record it.
- Time with a clock, not by estimate. Record the actual minutes for each task.
- Record misreadings in the participant's own words next to what the source says. Do not correct them during the task.
- A named person completes and signs the sheet. Until then human review is pending.
Instructor session (about 20 minutes)
You are deciding whether to use this lesson in a course. Think aloud. The facilitator will not explain anything unless you ask.
I1. Find the Flagship release these materials are pinned to.
Start at: Faculty guide. Done when: Says the version number (the key and the guide both state it).
After the task, ask: Where did you look first? What did you expect to find there?
Start ____ : ____ Finish ____ : ____ Minutes ______ Hesitated at: ______________________
Misread (their words → what the source says): ____________________________________________
I2. Decide which items you would assign in a 50-minute class and which as homework.
Start at: Faculty guide and worksheet. Done when: States a plan and a reason.
After the task, ask: What would you change?
Start ____ : ____ Finish ____ : ____ Minutes ______ Hesitated at: ______________________
Misread (their words → what the source says): ____________________________________________
I3. Grade the three sample answers to item 3 with the rubric.
Start at: Rubric and pilot sample answers. Done when: Gives a rubric level to each sample.
After the task, ask: Which descriptor was hard to apply?
Start ____ : ____ Finish ____ : ____ Minutes ______ Hesitated at: ______________________
Misread (their words → what the source says): ____________________________________________
Student session (about 30 minutes)
You are a student working through the worksheet on your own. Think aloud. The facilitator will only answer questions about the website, not about the answers.
S1. Worksheet item 1: which result compares the two drugs inside one randomized trial?
Start at: Student worksheet. Done when: Names SURMOUNT-5 with a reason.
After the task, ask: What told you it was head to head?
Start ____ : ____ Finish ____ : ____ Minutes ______ Hesitated at: ______________________
Misread (their words → what the source says): ____________________________________________
S2. Worksheet item 2: where does the 6.5 come from?
Start at: SURMOUNT-5 record. Done when: Shows 20.2 minus 13.7 and says why the subtraction is allowed here.
After the task, ask: Why is it allowed here and not for STEP 1 and SURMOUNT-1?
Start ____ : ____ Finish ____ : ____ Minutes ______ Hesitated at: ______________________
Misread (their words → what the source says): ____________________________________________
S3. Worksheet item 7: copy the citation and reopen the record in a private window.
Start at: Comparison page. Done when: Reports the release version and the checksum from the reopened record.
After the task, ask: Where did you hesitate? What did you expect the address to open?
Start ____ : ____ Finish ____ : ____ Minutes ______ Hesitated at: ______________________
Misread (their words → what the source says): ____________________________________________
Researcher session (about 25 minutes)
You are a researcher who reads trial papers. Check the lesson's claims independently. Think aloud and say where you would not trust it.
R1. Pick one comparison row and check one displayed number against its source using Verify this result.
Start at: Comparison page. Done when: Finds the source location and says whether it matches.
After the task, ask: Was the locator enough to find the number? What was missing?
Start ____ : ____ Finish ____ : ____ Minutes ______ Hesitated at: ______________________
Misread (their words → what the source says): ____________________________________________
R2. Find how SURMOUNT-5 handled treatment discontinuation and say what is and is not verified.
Start at: Comparison page or item 8 key. Done when: States the estimand and the registry completion counts, and that stopping-the-drug counts are unverified.
After the task, ask: Do you trust the unverified list? What would you add?
Start ____ : ____ Finish ____ : ____ Minutes ______ Hesitated at: ______________________
Misread (their words → what the source says): ____________________________________________
R3. Open the claim check “when weight-loss results can be compared” and say whether the verdict is justified.
Start at: Peptide Claims & Evidence. Done when: Agrees or disagrees with a reason.
After the task, ask: What would change your mind?
Start ____ : ____ Finish ____ : ____ Minutes ______ Hesitated at: ______________________
Misread (their words → what the source says): ____________________________________________
Sample answers for the instructor to grade (task I3)
Written by WPF for rubric practice in answer to worksheet item 3. They are not real student work.
- Answer A. Yes, you can subtract them. SURMOUNT-1 tirzepatide was minus 20.9 and STEP 1 semaglutide minus 14.9, so tirzepatide wins by 6 points.
- Answer B. You can't really subtract these because they are different trials with different people and different lengths, 68 weeks versus 72. It is an indirect comparison.
- Answer C. No. STEP 1 and SURMOUNT-1 are separate placebo-controlled trials: 1,961 adults for 68 weeks against 2,539 adults for 72 weeks, each compared only with placebo. So any gap between them is an indirect comparison across populations and designs, and only SURMOUNT-5 randomized both drugs together.
Observation sheet (one per participant)
- Participant code (not a name) and role
- Date, device, browser, assistive technology used
- For each task: start time, finish time, minutes taken
- Where the participant hesitated (page, control or sentence) and for how long
- What they misread: what they said, and what the source says
- Help requested and help given (exact words)
- Their own words about what was confusing or helpful
- Facilitator's name and the date the sheet was completed
Completed by (name): ______________________ Date: ____________ Human review status: pending until signed.