Setting an English Language Proficiency Passing Standard for Entry

www.ncsbn.org 1

The Michigan English Language Assessment Battery

(MELAB) is a complete language proﬁciency assessment

for advanced-level users of English as a second or foreign

language who will need to use English for academic and

professional use. It is a secure test program, administered

worldwide at authorized test centers and is recognized by

a growing list of academic institutions and professional

organizations across North America and beyond. To

provide member boards with another option for evaluating

the English language proﬁciency of internationally-

educated applicants, the National Council of State Boards

of Nursing (NCSBN

) conducted a standard-setting study

in October 2012 to establish the MELAB passing standard,

as related to entry-level nursing practice.

Similar standard setting exercises targeting entry-level

nurses have been performed for other English language

proﬁciency tests. In 2003, NCSBN conducted its ﬁrst

English language proﬁciency workshop with the Test of

English as a Foreign Language Computer-based Test

(TOEFL

CBT). An updated TOEFL passing standard was

set in a standard setting workshop in 2008 as a new version

of the test, TOEFL Internet-based Test (TOEFL iBT

™

became the primary choice for TOEFL candidates. In

2004, a similar standard setting workshop was conducted

with the International English Language Testing System

(IELTS

™

) examination. In 2009, the Pearson Test of English

(PTE) Academic was launched, and a standard setting

workshop was also held in the same year.

Such exercises deﬁne a legally defensible passing standard

for boards of nursing. They also make English language

proﬁciency qualiﬁcations portable across member

jurisdictions. This technical brief describes the MELAB

standard-setting process and the NCSBN recommended

passing standard.

The MELAB

The MELAB was developed to measure the English

language proﬁciency of non-native English speakers

intending to study or work in an English-medium

environment. Owned and developed by Cambridge

Michigan Language Assessments (CaMLA), the test is

developed in accordance with the highest standards in

educational measurement. Rigorous quality procedures

are followed during item and test form development

and the test is administered following standardized

procedures, including strict security measures. All writing

sections and speaking test examiners are trained and

monitored.

The examination comprises four sections covering the

different language skills (listening, reading, writing, and

speaking) and takes approximately 2¾ hours to complete.

The test items reﬂect the wide range of situations in

which test takers will need to use English: public spaces,

workplace settings and educational settings. The topics

are varied, require no specialized content knowledge

or experience and represent a variety of opinions. The

format of the test is shown in Table 1.

Setting an English Language Proﬁciency Passing

Standard for Entry-Level Nursing Practice Using the

Michigan English Language Assessment Battery

Hong Qian, PhD

Psychometrician, NCLEX

Examinations

Ada Woo, PhD

Director of Measurement and Testing, NCLEX

Examinations

Jayanti Banerjee, PhD

Research Director, Cambridge Michigan Language Assessments

October 2014

NCLEX

TECHNICAL BRIEF

Table 1. Structure of PTE Academic

Section Time Activities

Number of

Items

Writing 30 minutes

An essay based upon one of two

topic choices.

1 task

Listening 35-40 minutes

Audio recordings of short

conversations and talks are followed

by multiple choice questions.

Reading and

use of English

75 minutes

Reading passages are followed by

multiple-choice comprehension

questions. Use of English is tested

through a combination of discrete

and integrated tasks.

100

Speaking 15 minutes

Test takers engage in a conversation

with an examiner.

–

www.ncsbn.org 2

Each section of the MELAB is scored separately and

reported on the score report. The report also provides an

overall score that is the average of the writing, listening

and reading sections. The speaking test result is reported

separately on a scale 1 – 4. This may include a plus (+) or

minus (-); for instance, a test taker may be awarded a 3+ or a

4-. The overall MELAB score ranges from 0 to 99.

Selection for the MELAB Standard Setting Panel

The composition of a subject matter expert (SME) panel

of judges is critical to the success of the standard setting

workshop (Cizek & Bunch, 2007). It is important to assemble

“a sufﬁciently large and representative group of judges”

(AERA/APA/NCME, 1999, p. 54). For the MELAB standard-

setting workshop, 11 SMEs were recruited from the four

NCSBN geographic areas to serve on the standard-setting

panel. The panel was composed to include: (1) nurses who

speak a primary language other than English, (2) nursing

regulators, (3) nursing educators, and (4) consumers of

nursing services. In all, the standard setting panel was

quite diverse, representing nine NCSBN member boards

of nursing, one NCSBN associate member board of nursing

and ﬁve languages. Table 2 contains a description of the

panel composition.

Table 2. Characteristics of Panelists

Panelist Characteristic Category % (N)

Gender

Female 81.8 9

Male 18.2 2

Nursing license

Registered nurse (RN) 80.0 8

Licensed practical/vocational nurse

(LPN/VN)

20.0 2

NCSBN geographic region

I 27.3 3

II 9.1 1

III 36.4 4

IV 18.2 2

Associate Member 9.1 1

Years of post licensure experience

1 to 10 years 30.0 3

11 to 20 years 20.0 2

21 to 30 years 20.0 2

31 to 40 years 20.0 2

41 or more years 10.0 1

Primary language

English 45.5 5

Spanish 18.2 2

Tagalog 18.2 2

Filipino 9.1 1

German 9.1 1

Job title

Staff Nurse/Clinical Practice 36.4 4

Board of Nursing Regulator 18.2 2

Nursing Educator 36.4 4

Community Consumer 9.1 1

One panelist, representing consumer of nursing

services, did not hold a nursing license..

Among the 10 panelists who are licensed nurses,

average years of post-licensure experience is 22.3

years (SD = 14.9 years).

Standard-Setting Procedures

The ﬁrst portion of the workshop was spent educating

the panelists on the MELAB assessment and language

acquisition process. The panelists were provided a

description of the format of the test and a demonstration

of the items on the test. This presentation was followed by a

discussion of the language acquisition process and a review

of common English proﬁciency language frameworks. The

panelists were then provided instruction on the standard

setting methods and process. At the beginning of standard

setting pertaining to each section, the panelists participated

in a discussion of how much English proﬁciency was required

in that language skill to safely and effectively perform entry-

level nursing activities. From the discussion, the panelists

generated a list of tasks related to that speciﬁc language

skill that they deemed necessary for safe and effective

entry-level nursing practice (Appendix A). After reaching a

consensus, the panelists were asked to review actual MELAB

items and make a recommendation for a passing standard.

Two standard setting methods were used to set passing

standards for the four sections: the modiﬁed Angoff and the

extended Angoff methods. These methods are commonly

used in certiﬁcation and licensure contexts (Cizek & Bunch,

2007). The central idea of the Angoff method is for judges

to provide ratings on the basis of a “minimally competent

candidate.” Judges are asked to deﬁne what a “minimally

competent candidate” would know and be able to do. In

the modiﬁed Angoff method, judges are asked to consider

how many out of 100 minimally competent candidates

would answer the item correctly. This is appropriate for

www.ncsbn.org 3

the receptive skills (Listening and Reading sections). The

panelists reviewed MELAB items and made a judgment

for each. The panelists estimated the probability (out of

100) that an entry-level nurse with minimally acceptable

language skills would be able to answer the item under

review correctly.

In the extended Angoff method, rather than estimating the

proportion of minimally competent candidates who will

correctly answer an item, judges estimate the number of

points minimally competent candidates will need to obtain

on each item. This is appropriate for the productive skills

(Writing and Speaking sections) where the test-takers have

to produce open-ended responses, which are then assessed

by raters, using a set of predetermined criteria. For the

writing section, the standard setting panelists received a

pack of test-taker responses, one at every score proﬁle

on the rating rubric (ranging from basic English users to

proﬁcient English users). They independently reviewed the

responses and considered whether an entry-level nurse with

minimally acceptable English language skills, as deﬁned

through panel discussions, would be able to provide a

response similar to one of the sample candidates. For the

speaking section, the panelists listened to short speaking

test extracts that represented performances at every score

proﬁle on the rating rubric. As in the writing section, the

panelists considered whether an entry-level nurse with

minimally acceptable English language skills, as deﬁned

through panel discussions, would be able to provide a

response similar to one of the sample candidates.

Within both standard setting methods, the panelists were

asked to provide two rounds of ratings. In the ﬁrst round,

the panel members estimated the ability of candidates

or the difﬁculty of items in relation to the hypothesized

linguistic requirements for entry-level nurses. The ﬁrst round

was followed by a period of discussion and feedback. The

panelists then provided their second round and ﬁnal ratings

on the same set of items.

Standard-Setting Panel Recommendations

For all four sections of the MELAB, panelists were able

to come to a consensus on a recommended standard. In

their post-meeting feedback, the panelists indicated that

they understood the standard setting process and were

conﬁdent in their ratings. In all the skill categories, the round

two recommendations were more consistent as compared

to round 1. This suggests that the panelists converged in

their opinions as the workshop progressed.

As indicated earlier, test takers receive a score for each

section of the MELAB. The writing, listening, and reading

scores are averaged to produce a total score. This score is

used for decision-making. The speaking score is reported

separately for decision making. Table 3 lists the MELAB total

score and speaking score that the panel recommended.

In order to gauge whether the recommended MELAB scores

are in line with the standards set in the TOEFL studies, the

results from the MELAB workshop were compared to pre-

vious results. According to a study conducted by CaMLA,

the MELAB passing standards recommended by the stan-

dard setting panel are comparable to the NCSBN-endorsed

TOEFL passing standards. NCSBN recommended an overall

cut score of 560 on the TOEFL PBT, 220 on the TOEFL CBT

and 84 on the TOEFL iBT. A minimum speaking score of 26

was advised on the TOEFL iBT. The ﬁndings from the study

suggest that the NCSBN recommendations on the PBT, iBT

and CBT all fall within the same score range. As shown in Ta-

ble 4, the score range corresponds to a MELAB score range

of 78-81.

NCSBN Board of Directors (BOD) Deliberation

The NCSBN Board of Directors (BOD) reviewed and

discussed results from the MELAB standard setting

workshop. They reviewed background on the MELAB and

procedures used in the standard setting workshop. The

BOD then considered the passing standards recommended

by the MELAB standard setting panel, as well as the

potential pass/fail impact of these recommendations. After

weighing all available evidence, including comparability to

other NCSBN-endorsed English proﬁciency standards and

BOD’s own knowledge about the nursing profession, they

concurred with the standard setting panel’s recommendation

of an overall passing standard of 81 and a speaking section

score of 3.

Conclusions

NCSBN’s recommendation regarding the English proﬁcien-

cy passing standard of an overall score of 81 with a speaking

score of 3 is consistent with the previous English proﬁciency

passing standards recommended for other English language

tests and the level of English proﬁciency needed to perform

entry-level nursing safely and effectively. Additionally, this

recommendation is consistent with the NCSBN policy po-

sition on international nurse immigration, which states that

“domestic and international nurses need to [be] proﬁcient

in written and spoken English skills” (NCSBN, 2003, as cited

in Woo, Dickison & de Jong, 2010). Boards of nursing can

use this information to make decisions regarding the level of

English proﬁciency needed for entry-level RNs and LPN/VNs

in their jurisdictions.

Table 3. Summary of Panel Recommendations

Total (Reading, Listening, Writing) Speaking

78-81 3

Table 4. Comparison of MELAB and TOEFL Score Ranges

MELAB TOEFL PBT TOEFL iBT TOEFL CBT

78-81 560-577 83-91 220-233

www.ncsbn.org 4

Appendix A: English Language Tasks Performed by Entry-Level Nurses

Reading Listening Writing Speaking

 Advance directives

 Reading reports and charts

 Medication orders

 Equipment instructions

 Medication labels

 References material

 Research/evidenced-based

practice

 Infection control practices

 Patient safety

 Physician consultation

 Narrative notes

 Memos/emails/correspondences

 Continuing education

 Data reports

 Patient feedback

 Clarifying information

 Cultural differences

 Discern

 Phone conversations

 Unscripted conversations

 Communicating with

colleagues

 Receiving feedback and

direction

 Receiving report of lab

values

 Receiving doctor’s orders

 Initial assessment

 Client report

 Nursing care plan

 Client consent to care

 Nurse’s note

 Phone order-clarify

 Client interview

 Client education

 Physical assessment

 Client report

 Collaboration

 Rapid response/Code Blue

 Clarifying language

 Socialization in profession

 Medical terminology

 Client advocacy

References

American Educational Research Association, American Psychological Association, National Council on Measurement in

Education. (1999). Standards for educational and psychological testing. Washington, DC: American Psychological

Association.

Cizek, G. J., & Bunch, M. B. (2007). Standard setting: A guide to establishing and evaluating performance standards on

tests. Thousand Oaks, CA: Sage.

Woo, A., Dickison, P., & de Jong, J. (2010). Setting an English Language Proﬁciency Passing Standard for Entry-Level

Nursing Practice Using the Pearson Test of English Academic. Retrieved October 29, 2014 from

https://www.ncsbn.org/NCLEX_technicalbrief_PTE_2010.pdf

) is a not-for-proﬁt organization whose members include the boards of nursing in the 50

states, the District of Columbia and four U.S. territories — American Samoa, Guam, Northern Mariana Islands and the Virgin Islands. There are 21 associate

members.

Mission: NCSBN provides education, service, and research through collaborative leadership to promote evidence-based regulatory excellence for

patient safety and public protection.

(Mission Statement Adopted by Delegate Assembly 2010)

National Council of State Boards of Nursing, Inc.

111 E. Wacker Drive, Suite 2900, Chicago, IL 60601-4277

312.525.3600 | International Calls: +1.312.525.3688

Testing Services: 312.525.3750 | www.ncsbn.org