Technical Assessment & Certification Design
Parent: Technical Instruction & Engineering Education · Published reference · snapshot 2026-09-08 · skill technical-instruction/references/assessment-certification-design.md
↓ Facts as markdownall context files
Expert reference for designing, evaluating, and accrediting professional certification programs.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Assessment & Certification Design
- Expert reference for designing, evaluating, and accrediting professional certification programs. Covers the full lifecycle from job task analysis through psychometric validation, cut-score setting, exam security, and digital credential issuance. Grounded in AERA/APA/NCME Standards, NCCA/ICE accreditation requirements, and ANSI/ISO 17024. [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#assessment-certification-design]
Contents
- Core Concepts [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#contents]
- Quick Reference Tables [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#contents]
- Detailed References [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#contents]
1. The Credentialing Lifecycle
- Every defensible certification program follows this sequence: [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#1-the-credentialing-lifecycle]
- Skipping any step creates a defensibility gap under accreditation review. NCCA Standard 14 requires documented JTA linkage to every content domain in the blueprint. [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#1-the-credentialing-lifecycle]
2. Assessment Blueprint / Test Specifications
- The blueprint (also called a table of specifications or content outline) is the governing document for all item development. It must be empirically traceable to the JTA. [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#2-assessment-blueprint-test-specifications]
- Blueprint architecture rules (CEDMA guidance): [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#2-assessment-blueprint-test-specifications]
- No more than 7 major content domains and 20–22 total objectives [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#2-assessment-blueprint-test-specifications]
- No single objective should be assessed by only one item (minimum 3–4 items per objective) [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#2-assessment-blueprint-test-specifications]
- Weight domains proportionally to JTA frequency × importance ratings [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#2-assessment-blueprint-test-specifications]
- Freeze the blueprint before item development begins; mid-cycle revisions invalidate items [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#2-assessment-blueprint-test-specifications]
- Cognitive level distribution by certification tier: [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#2-assessment-blueprint-test-specifications]
- Biggs's constructive alignment principle: intended learning outcomes define assessment tasks; tasks define instructional activities - not the reverse.[^biggs1999] The scenario-removal test: if a candidate can cover the scenario and answer from the stem alone, the item is testing recall, not reasoning, regardless of blueprint labeling.[^cedma] [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#2-assessment-blueprint-test-specifications]
3. Item Writing: Multiple-Choice Questions (MCQs)
- See references/item-writing-and-psychometrics.md for full distractor analysis tables. [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#3-item-writing-multiple-choice-questions-mcqs]
- Well-formed stem rules: [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#3-item-writing-multiple-choice-questions-mcqs]
- Complete problem statement in the stem; candidates should not need to read options to understand the question [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#3-item-writing-multiple-choice-questions-mcqs]
- Use positive phrasing; reserve EXCEPT/NOT stems for cases where the negative is the exact professional skill being tested [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#3-item-writing-multiple-choice-questions-mcqs]
- One clear question per stem (no double-barreled constructions) [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#3-item-writing-multiple-choice-questions-mcqs]
- Avoid window-dressing text that adds length without adding discriminating information [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#3-item-writing-multiple-choice-questions-mcqs]
- Distractor design rules: [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#3-item-writing-multiple-choice-questions-mcqs]
- All distractors must be plausible to a candidate lacking the target knowledge [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#3-item-writing-multiple-choice-questions-mcqs]
- Options must be parallel in grammatical form and similar in length [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#3-item-writing-multiple-choice-questions-mcqs]
- A nonfunctional distractor (selected by <5% of examinees) degrades item discrimination; flag for revision after each administration [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#3-item-writing-multiple-choice-questions-mcqs]
- Never use "all of the above" (rewards partial knowledge) or "none of the above" (unless an exact answer is required, e.g., mathematical calculations) [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#3-item-writing-multiple-choice-questions-mcqs]
- Six flaw categories to eliminate: [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#3-item-writing-multiple-choice-questions-mcqs]
- Structured faculty training + peer review reduces total item flaw rates from ~67% to ~21% within three years (longitudinal medical education data).[^pmc3809311] [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#3-item-writing-multiple-choice-questions-mcqs]
4. Performance-Based Assessment (PBA) Items
- PBAs assess execution, not knowledge recall. Action-verb alignment rule: [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#4-performance-based-assessment-pba-items]
- Blueprint verbs "configure," "demonstrate," "troubleshoot" → hands-on lab / PBA items [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#4-performance-based-assessment-pba-items]
- Blueprint verbs "explain," "identify," "describe" → MCQ or scenario-based items [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#4-performance-based-assessment-pba-items]
- Mixing these creates construct-irrelevant variance [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#4-performance-based-assessment-pba-items]
- Six-step PBA design process: [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#4-performance-based-assessment-pba-items]
- Ground each task in a specific JTA task statement [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#4-performance-based-assessment-pba-items]
- Define scoring criteria (success outcomes) before designing the environment [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#4-performance-based-assessment-pba-items]
- Engage SMEs during design, not as final reviewers only [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#4-performance-based-assessment-pba-items]
- Build scoring rubrics concurrently with task design (not post-hoc) [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#4-performance-based-assessment-pba-items]
- Validate each task against actual job performance data [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#4-performance-based-assessment-pba-items]
- Plan ongoing maintenance (tools and duties evolve) [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#4-performance-based-assessment-pba-items]
- Scoring must accommodate alternative solution paths (multiple valid command sequences achieving the same correct outcome). Partial-credit rubrics for directionally correct but incomplete solutions. [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#4-performance-based-assessment-pba-items]
5. Psychometrics: Classical Test Theory (CTT) Item Analysis
- Run after every exam administration to flag items for revision or retirement. [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#5-psychometrics-classical-test-theory-ctt-item-analysis]
- Key statistics and interpretation thresholds: [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#5-psychometrics-classical-test-theory-ctt-item-analysis]
- CTT limitation: all statistics are sample-dependent. The same item's p-value differs across cohorts with different mean ability.[^ctt-sampledev] This drives the migration to IRT for large-scale programs. [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#5-psychometrics-classical-test-theory-ctt-item-analysis]
6. Psychometrics: Item Response Theory (IRT)
- IRT models the probability of a correct response as a function of latent ability (θ) and item parameters. The key advantage for credentialing: parameter invariance - item difficulty does not depend on who was tested; person ability does not depend on which items were answered. [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#6-psychometrics-item-response-theory-irt]
- Model selection guide: [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#6-psychometrics-item-response-theory-irt]
- Key IRT concepts: [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#6-psychometrics-item-response-theory-irt]
- Item Characteristic Curve (ICC): plots P(correct) vs. θ; inflection point = b, slope ∝ a, lower asymptote = c [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#6-psychometrics-item-response-theory-irt]
- Item Information Function: I(θ) = a²·P(θ)·Q(θ); items contribute maximum information at θ ≈ b [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#6-psychometrics-item-response-theory-irt]
- Test Information Function (TIF): sum of item information functions; enables assembling forms with maximum precision at the cut score [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#6-psychometrics-item-response-theory-irt]
- Conditional SEM: SEM(θ) = 1/√I(θ); unlike CTT's global SEM, CSEM varies and is typically largest at the cut score - must be reported for pass/fail decision accuracy [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#6-psychometrics-item-response-theory-irt]
- IRT assumptions to verify: [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#6-psychometrics-item-response-theory-irt]
- Unidimensionality (one dominant latent trait; use confirmatory factor analysis) [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#6-psychometrics-item-response-theory-irt]
- Local independence (items not correlated after conditioning on θ; case vignette testlets violate this) [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#6-psychometrics-item-response-theory-irt]
- Model fit (fit statistics for each item; misfit → revise or remove) [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#6-psychometrics-item-response-theory-irt]
- Test equating: when multiple exam forms must be compared fairly, IRT true-score equating with a Non-Equivalent groups with Anchor Test (NEAT) design is the standard. Anchor item drift (items that change difficulty between forms due to exposure or coaching) is the primary equating threat. Pre-equating embeds new items as unscored pilots and calibrates them to the existing scale before form assembly. [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#6-psychometrics-item-response-theory-irt]
7. Standard Setting (Cut Score Determination)
- Cut scores must be defensible, documented, and tied to a defined performance standard - "minimally competent candidate." The standard-setting study is a separate formal process. [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#7-standard-setting-cut-score-determination]
- Method comparison: [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#7-standard-setting-cut-score-determination]
- Standard-setting best practices: [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#7-standard-setting-cut-score-determination]
- Train panelists on the definition of "minimally competent" before ratings (not after) [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#7-standard-setting-cut-score-determination]
- Run multiple rounds; show panelists inter-rater disagreement statistics and allow discussion [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#7-standard-setting-cut-score-determination]
- Document all panelist credentials, training procedures, and final decisions for accreditation [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#7-standard-setting-cut-score-determination]
- Apply SEM at the cut score to define a "borderline zone" for decision accuracy analysis [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#7-standard-setting-cut-score-determination]
8. Certification Program Design and Accreditation
- ANSI/ISO 17024:2012 - The international standard for personnel certification bodies. Key requirements: impartiality (governance separation between certification and training arms), documented examination development, reliability and validity evidence, and a competence-based appeal process. Required for programs with international recognition ambitions. [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#8-certification-program-design-and-accreditation]
- NCCA Standards (National Commission for Certifying Agencies) - The US-specific accreditation benchmark administered by ICE (Institute for Credentialing Excellence). 21 Standards organized around: governance, JTA, exam development, psychometric soundness, security, candidate policies, and recertification. NCCA accreditation signals program quality to employers and regulators. [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#8-certification-program-design-and-accreditation]
- Role separation requirement (both standards): The governance body that awards credentials must be structurally independent from any body that provides preparation or training. Conflict-of-interest management policies must be documented and enforced. [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#8-certification-program-design-and-accreditation]
- Recertification / Maintenance of Certification (MOC): [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#8-certification-program-design-and-accreditation]
- CE-based: earn continuing education credits per cycle (most common) [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#8-certification-program-design-and-accreditation]
- Point-based: accumulate points across CE, professional activities, contributions [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#8-certification-program-design-and-accreditation]
- Re-examination: pass the current exam version at renewal [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#8-certification-program-design-and-accreditation]
- Practice requirements: document ongoing professional activity [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#8-certification-program-design-and-accreditation]
- Choice depends on domain velocity - fast-moving technical domains favor re-examination or point-based systems that include currency-of-practice requirements. [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#8-certification-program-design-and-accreditation]
9. Exam Security and Integrity
- Item exposure control: [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#9-exam-security-and-integrity]
- Sympson-Hetter (SH) procedure: assigns probabilistic exposure caps via simulation; prevents overexposure in CAT; two-stage SH (2023) adds minimum exposure floor to prevent underexposure [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#9-exam-security-and-integrity]
- Item bank rotation: partition banks into sub-banks and rotate active pools; most effective for multi-timezone global testing [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#9-exam-security-and-integrity]
- Field test items (beta items): embedded unscored items that collect psychometric data without affecting candidate score; rotate to operational after calibration [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#9-exam-security-and-integrity]
- Online / remote proctoring model comparison: [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#9-exam-security-and-integrity]
- AI-resistant assessment design (post-LLM era): [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#9-exam-security-and-integrity]
- GPT-4-class models score in the 60th–90th percentile on many MCQ credentialing exams. Knowledge recall items are indefensible without layered countermeasures: [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#9-exam-security-and-integrity]
- Key threat vectors (PSI Security Guide): [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#9-exam-security-and-integrity]
- Content harvesting (phone camera while appearing to face forward) - greatest risk [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#9-exam-security-and-integrity]
- Proxy testing / deepfake video substitution [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#9-exam-security-and-integrity]
- Organized collusion rings across time-zone sittings [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#9-exam-security-and-integrity]
- Brain-dump sites and item-memorization services [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#9-exam-security-and-integrity]
10. Micro-Credentials, Digital Badges, and Open Badges 3.0
- Full certification: comprehensive occupational profile; prerequisites; formal exam; renewal/CE [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#10-micro-credentials-digital-badges-and-open-badges-30]
- Micro-credential: discrete skill cluster; short (weeks–months); stackable toward larger qualifications [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#10-micro-credentials-digital-badges-and-open-badges-30]
- Digital badge: the visual + metadata artifact representing any achievement (micro or full credential) [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#10-micro-credentials-digital-badges-and-open-badges-30]
- Each OpenBadgeCredential is issued as a W3C Verifiable Credential (VC Data Model 2.0) [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#10-micro-credentials-digital-badges-and-open-badges-30]
- Cryptographically signed by the issuer's DID using EdDSA (eddsa-rdfc-2022) or ECDSA (ecdsa-sd-2023) [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#10-micro-credentials-digital-badges-and-open-badges-30]
- Badge Connect API: OAuth 2.0-authenticated REST endpoints (getCredentials, getProfile, upsertCredential) for credential portability between any compliant platform and any compliant wallet [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#10-micro-credentials-digital-badges-and-open-badges-30]
- Revocation via BitstringStatusListEntry; expiration status must be displayed by conformant Displayers [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#10-micro-credentials-digital-badges-and-open-badges-30]
- CLR Standard 2.0 (Comprehensive Learner Record) co-evolved with OB 3.0; bundles multiple credentials as a longitudinal transcript [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#10-micro-credentials-digital-badges-and-open-badges-30]
- Issuer signs VC → Holder stores in DID-keyed wallet → Verifier resolves issuer DID, validates signature, checks status - no callback to issuer required [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#10-micro-credentials-digital-badges-and-open-badges-30]
- DID v1.0 became W3C Recommendation July 2022 [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#10-micro-credentials-digital-badges-and-open-badges-30]
- Blockchain anchoring: credential hash written on-chain; hash mismatch = tamper detection; confidential data stays off-chain [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#10-micro-credentials-digital-badges-and-open-badges-30]
- Verification latency: seconds vs. weeks for legacy background check services [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#10-micro-credentials-digital-badges-and-open-badges-30]
References
- [^biggs1999]: Biggs, J.B. (1999). "Aligning Teaching for Constructing Learning." https://www.researchgate.net/publication/255583992 - Constructive alignment framework; ILO-driven assessment design. [^cedma]: CEDMA. "Best Practices for Certification Exam Blueprints." https://www.cedma.org/customeredinsights/best-practices-for-certification-exam-blueprints - Blueprint architecture limits; cognitive weighting by tier; scenario-removal test. [^pmc3809311]: PMC. "Identification of technical item flaws leads to improvement of MCQ quality" (PMC3809311). https://pmc.ncbi.nlm.nih.gov/articles/PMC3809311/ - Two-category flaw taxonomy; longitudinal flaw-rate reduction (67% → 21%). [^ctt-sampledev]: eddata.com. "Item Statistics Overview." https://eddata.com/2019/06/item-statistics-for-classroom-assessments-1/ - CTT sample-dependence limitation; p-value cohort variance. [^ob30]: 1EdTech. "Open Badges 3.0 Standard." https://www.1edtech.org/standards/open-badges - OB 3.0 final approval date; Badge Connect API; W3C VC alignment; conformance roles. [^aera2014]: AERA/APA/NCME. "Standards for Educational and Psychological Testing" (2014). https://www.aera.net/publications/books/standards-for-educational-psychological-testing-2014-edition - Governing framework: validity argument; five sources of validity evidence; fairness as foundational design requirement. [^ncca2021]: ICE/NCCA. "NCCA Standards for the Accreditation of Certification Programs" (2021). https://www.credentialingexcellence.org/Portals/0/NCCA%20Standards%202021%20DRAFT%20REVISIONS_Sept%202021.pdf - 21 Standards; governance, JTA, exam development, security, recertification. [^irt-invariance]: Assessment Systems. "What is Item Response Theory?" https://assess.com/what-is-item-response-theory/ - IRT parameter invariance; CAT; TIF; pre-equating. [^sh-2023]: PubMed. "Controlling the Minimum Item Exposure Rate in CAT: A Two-Stage Sympson-Hetter Procedure" (2023). https://pubmed.ncbi.nlm.nih.gov/37997579/ - Two-stage SH for minimum + maximum exposure control. [^llm-resistant]: arXiv 2304.12203. "Creating Large Language Model Resistant Exams" (2023). https://arxiv.org/pdf/2304.12203 - LLM-resistant item design principles; performance-based countermeasures. [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#references]
Detailed References
- See references/item-writing-and-psychometrics.md for: [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#detailed-references]
- Full distractor analysis worked examples [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#detailed-references]
- IRT parameter estimation procedures [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#detailed-references]
- Equating design decision trees [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#detailed-references]
- DIF analysis (Mantel-Haenszel, logistic regression methods) [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#detailed-references]
- Job Task Analysis survey design template [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#detailed-references]
- Sensitivity review panel guidance [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#detailed-references]
- NCCA Standard-by-standard compliance checklist [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#detailed-references]
- Open Badges 3.0 API reference and conformance guide [archived source: https://llms-explorer.com/sources/mdb-context-hub/assessment-certification-design/#detailed-references]
Where this helps
- Building a professional certification exam from scratch for a technology or role, where the content needs a defensible, documented link back to what practitioners actually do on the job. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Deciding between item types (multiple-choice vs. performance-based) for a specific skill being certified, based on whether the target blueprint verb is "explain" or "configure." [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Setting or defending a pass/fail cut score for an existing exam, especially under scrutiny from an accreditation body like ICE/NCCA or ANSI/ISO 17024. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Designing anti-cheating and exam-security controls for a remote/online certification exam in an era where GPT-4-class models already score in the 60th-90th percentile on many MCQ credentialing exams. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
How to apply this
- Run a Job Task Analysis (JTA) survey with practicing professionals before writing a single exam item, then freeze a blueprint that weights domains by JTA frequency times importance. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Apply the CTT item-analysis statistics (p-value, point-biserial) after each exam administration to flag and retire nonfunctional distractors. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Use the Angoff or bookmark method with a trained panel to set a defensible cut score, documenting panelist credentials and rationale for accreditation review. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Layer AI-resistant countermeasures (proctoring, item-exposure control, performance-based item weighting) into a certification redesign given how well large language models now score on knowledge-recall MCQs. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Antipatterns
- Writing exam items before finishing job task analysis, so the blueprint has no empirically traceable link to the domains it claims to test - an NCCA Standard 14 violation. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Using "all of the above" or "none of the above" distractors, which reward partial knowledge instead of discriminating genuine competence. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Treating a mid-cycle blueprint revision as low-risk - it invalidates already-written items and breaks the JTA linkage the whole exam depends on. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Setting a cut score by gut feel instead of a documented standard-setting study (Angoff, bookmark, or contrasting-groups), leaving the pass/fail line indefensible under review. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Known issues
- CTT item statistics (p-value, discrimination) are sample-dependent - the same item's difficulty reads differently across cohorts with different mean ability, which is why large-scale programs migrate to IRT. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- A nonfunctional distractor (chosen by fewer than 5% of examinees) silently degrades an item's discrimination power and won't be caught without a post-administration item analysis. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Item exposure control (Sympson-Hetter, bank rotation) trades security for psychometric complexity - poorly tuned exposure caps can starve some items of the data needed to calibrate them. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Open Badges 3.0 verification depends on the issuer's DID and signature infrastructure staying live; a defunct issuer breaks the "no callback needed" verification promise the whole model is built around. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Children
- Item Writing (MCQ & performance-based) (frontier)
- Psychometrics (CTT item analysis, IRT, KR-20) (frontier)
- Validity & Reliability (frontier)
- Cut Scores (Angoff, Bookmark) (frontier)
- Job Task Analysis (frontier)
- Test Blueprints (frontier)
- ANSI/ISO 17024 & NCCA Accreditation (frontier)
- Exam Security & AI-Resistant Assessment (frontier)
- Open Badges 3.0 & Micro-Credentials (frontier)
Frontier under this node: ANSI/ISO 17024 & NCCA Accreditation, Cut Scores (Angoff, Bookmark), Exam Security & AI-Resistant Assessment, Item Writing (MCQ & performance-based), Job Task Analysis, Open Badges 3.0 & Micro-Credentials, Psychometrics (CTT item analysis, IRT, KR-20), Test Blueprints, Validity & Reliability