Reviewing flags and the quality report

What a flag says

Each quality flag (a finding) names the check that raised it, its severity and the enumerator, and says in plain words which question, what was entered and what was expected. Technical details under it (for owners, managers and reviewers) give the variable names, the number behind the finding and the rule that raised it. A finding about another interview (a duplicate, a copied answer) links to that interview too. Times are shown in the workspace's time zone.

Working through them

Show Open, Resolved, Dismissed or All findings, 50 to a page. Write a note if you like, then:

• Resolve: the problem is dealt with;
• Dismiss: it is not a problem;
• Re-open: put one back.

Who did what, when and with what note is shown under the finding, every change is kept in its History, and the change is recorded in the activity log. The list refreshes itself every 15 seconds, but not while you are writing a note.

When the checks run again

Running the checks again respects what the team decided:

• A dismissed finding stays dismissed unless it gets materially worse: its measure (how far past the limit, how unusual) is at least 50% worse than when it was dismissed, or the answer now crosses a hard limit. A finding with no such measure (a skip rule, a missing GPS reading) stays dismissed.
• A resolved finding about an enumerator or the whole form re-opens if the problem comes back in interviews that arrived after it was resolved (for a daily cap, only when that same day gets more interviews). A resolved finding about one interview re-opens only if it gets materially worse, since the interview's answers do not change unless someone corrects them.
• A finding that is still open is brought up to date each time the checks run, so you always decide on its current numbers.

A re-opened finding says "Re-opened by the checks" and why.

The quality report

The Quality report (Excel; on each form's card in Monitor and under Export) has every finding of the form, with its status, who reviewed it, when, the note and the technical details, then the form's field-work indicators:

• Indicators: completed surveys; the average per active collection day (days with at least one interview) and per calendar day (first to last collection day); the average per enumerator; the median and mean interview duration, and how many durations were left out (a start or end time missing, an end before the start, more than 24 hours). Every average shows its numerator and denominator.
• Enumerator by day: each enumerator's completed surveys on each interview date, with totals.
• Sections: each top-level group and repeat of the deployed form, how many questions it asks and, when the form has a text audit, the median and mean time spent in it. Without a text audit there is no time per section: Joripo does not estimate it, and the sheet says to add a text audit field.
• Text audit summary: how many interviews have a text audit, and the median and mean seconds on each question.

Completed means every submission the server has received: a phone sends a form only once it is finalized, so drafts are never counted. An interview's date is the day it started, in the workspace's time zone (its end when it has no start; the day it was received when it has neither, and the report says how many).

Text audits, processed (one readable workbook) and Text audits, raw (the files as the phones sent them) are offered beside the report for a form with a text audit field. The workspace's Quality flags, every form report has the findings only.

The verdict in the data

Export's Joripo, with quality-flag columns layout adds each submission's number of findings, n_flags, so keep if n_flags == 0 in Stata drops flagged interviews. Findings are never mixed into the SurveyCTO-compatible layout, which stays SurveyCTO's.

External clients

Someone the workspace is shared with sees findings only if their access includes quality, and then sees a finding's status and when it changed, not who reviewed it or the team's notes.