P-12

Prekindergarten through Grade 12 Education


Google Translate Disclaimer

RFP #18-013: Questions and Answers

Elementary, Intermediate (Middle) Level, and High School New York State English as a Second Language Achievement Test (NYSESLAT)

QUESTIONS AND ANSWERS

Please note: the “Educator Committees Chart” included on pages 22 and 23 of the RFP has been revised. The updated chart is included as part of this Questions and Answers Summary.

Program

  1. Regarding the Training for Local Scoring for both the 1-day in person and the turnkey training, is the contractor required to provide food for participants? If so, which meals should be provided?
    The contractor should provide a light breakfast and lunch for all attendees.

  2. Is any new NYSITELL Levels II – VIII test development or field testing expected before the end of this contract in December 2022?
    No.

  3. The time allocated for the Operational Final Eyes Review (24 test booklets and DFAs) is 2-3 days. The time allocated for the Field Test Final Eyes Review is only 1 day, yet there are more materials - approximately 36 Test Booklets and DFAs. It does not seem feasible to complete the Field Test review in 1 day with only 14 educators. Should the time for the Field Test Final Eyes Review be adjusted to 3 or 4 days?
    Thank you for bringing this concern to our attention. We are amending the “Educator Committees Chart” (see attached), included on pages 22 and 23 of the RFP, to allow 3-4 days for the Field Test Final Eyes Review.

  4. The RFP states that a total of 14 educators should be invited for both the Field Test Final Eyes and the Operational Final Eyes Committee meetings. On page 25 the RFP states that the Final Eyes Committee meeting must include 3 subcommittees, each reviewing specific grade bands. To evenly distribute committee members, should the number of educators be increased to 15 so that there are 5 educators per subcommittee?
    Thank you for bringing this concern to our attention. We are amending the “Educator Committees Chart” (see attached), included on pages 22 and 23 of the RFP, to allow 15 educators to participate in these educator committee meetings.

  5. Do the number of days outlined in the RFP to complete the rangefinding task (4-5 days) apply to both the Writing (4-5 days) and Speaking (4-5 days) as separate activities or is it NYSED desire to combine these activities to occur concurrently?
    Thank you for bringing this concern to our attention. We are amending the “Educator Committees Chart” (see attached), included on pages 22 and 23 of the RFP, to allow 4-5 days each for the Writing Rangefinding and Speaking Rangefinding for Field Tests.

  6. Are the 54 educators allocated to this meeting to be split between the Writing breakout groups (27) and Speaking breakout groups (27), or are there 54 educators for Writing and 54 for Speaking?
    Thank you for bringing this concern to our attention. We are amending the “Educator Committees Chart” (see attached), included on pages 22 and 23 of the RFP, to allow 4-5 days each for 54 educators to complete the Writing Rangefinding and 54 educators to complete the Speaking Rangefinding for Field Tests.

  7. Does NYSED expect the Rangefinding meetings (Writing and Speaking) to be held simultaneously or sequentially? If sequentially, are 54 educators per Rangefinding required?
    NYSED expects the Rangefinding meetings (Writing and Speaking) to be held sequentially, requiring 54 educators for each meeting.

  8. Only K Entry Initial Identification test design and development are included in the RFP but grades 1-12 are not. Can NYSED please confirm that the current initial Identification tests for grades 1-12 will remain static?
    NYSED confirms that, as stated on page 14, effective February 1, 2018, a newly designed NYSITELL will take the place of the current 2014 edition of the NYSITELL and that the 2018 NYSITELL will remain static for NYSITELL levels II - VIII during the term of the contract resulting from this RFP #18-013. The contractor is expected to design and develop a new NYSITELL only for the Kindergarten Entry Initial Identification Test, Level I.

  9. If the previous answer is no, what changes will need to be made?
    Not applicable.

  10. The K-Entry NYSITELL Level I test requires prototyping in 2020 for a revised assessment. Would NYSED entertain the introduction of stimulus such as flip charts, game boards, or item cards for the new NYSITELL K-Entry test, or must the assessment approach be limited to printed booklets due to NYSED supplying the printed materials for the NYSITELL?
    Yes, NYSED will entertain creative options for the introduction of stimuli for the Kindergarten Entry Initial Identification Test.

  11. The contractor must provide the technology for schools assigned to a Speaking Field Test to securely record and transmit to the contractor student response, and provide digital recorders to these schools, upon request. Does NYSED have an estimated number of digital recorders that would be needed for field testing?
    NYSED cannot provide the exact number of digital recording devices that the contractor will need to provide to schools for Speaking field testing because most schools use their own recording devices. The worst case is that the contractor will need to provide one digital recorder to every school participating in the Speaking Field Tests. This amounted to approximately 320 schools during the most recent round of annual field testing.

  12. Which materials does NYSED expect the contractor to provide for NYSED review a month in advance of the Forms Construction meeting?
    The contractor must provide NYSED with the proposed NYSESLAT operational test forms based on the field test results and research guidelines for item selection, as well as additional field-tested items for possible replacement.

  13. What date should the informational publication that includes test samplers be delivered to schools?
    The initial set of test samplers should be delivered to schools by October 31, 2018. The new test samplers based on new NYS Next Generation ELA standards should be delivered to schools by September 30, 2021.

  14. The RFP indicates that the Rasch and Partial Credit Models are currently used. Partial Credit was used for the previous NYSESLAT, but not the current one. The current NYSESLAT uses a rating scale model. Would NYSED like to continue to use the rating scale model?
    NYSED prefers that, in the future, the Partial Credit Model be used in place of the Rating Scale Model.

  15. If the previous answer is no, would NYSED like to switch to the Partial Credit model?
    NYSED prefers that, in the future, the Partial Credit Model be used in place of the Rating Scale Model.

  16. The contractor must select a representative sample of 10% of the student test recordings for each grade band and rescore the locally scored speaking results for an inter-rater reliability study. In order to achieve a blind rescore of the student population, does NYSED anticipate that all 275,000 students who participate in the NYSESLAT administration each year would have the speaking administration recorded either via secure online recording or be provided digital recording devices by the vendor, and that the representative sample would be pulled from that complete pool?
    As a matter of efficiency, NYSED is not expecting the scoring audit to be a blind rescoring. Schools selected for the Speaking audit should be notified in advance of the administration.

  17. If the previous answer is no, would the state consider pre-selecting schools according to an agreed-upon methodology to participate in recording (either secure online or digital recording) in order to keep overall costs to the state and burden on the schools down, but still produce the representative sample needed for rescore?
    Yes, NYSED expects that the contractor will so notify those schools selected for the Speaking audit in advance of the administration.

  18. The contractor must select a representative sample of 10% of student test recordings for each grade band and rescore the locally scored speaking results for an inter-rater reliability study. In the field test section (page 28) the RFP states the need to supply digital recorders to schools, upon request. If all 275,000 administrations are to be recorded in anticipation of 10% sampling for inter-rater reliability, does NYSED have an estimate of how many digital recorders would be needed for the operational administration?
    Each year, it is expected that only 10% (430) of the approximately 4,300 participating schools will be required to digitally record students’ speaking responses in order to participate in the Scoring audit. The worst case is that the contractor will be required to provide digital recorders at a ratio of 1 per 50 students to all of the schools in the Speaking audit sample.

  19. Does the synthesis group meeting take place on an additional day, after the Standard Setting Meeting?
    The three days for this activity listed in the Educator Committees Chart on pages 22 and 23 of the RFP include the meeting of the synthesis group on the last day.

  20. The RFP describes a synthesis group of table leaders from 4 committees. Should the synthesis group consist of table leaders from 7 committees?
    Yes, the synthesis group should consist of table leaders from 7 committees, rather than 4.

  21. Will a dry run be required for the Standard Setting meeting? This is not listed on the cost proposal.
    Yes. The cost for conducting a dry run with NYSED for the standard setting should be incorporated in the June-July 2022 “Conduct Standard Setting with NYS Educators” deliverable.

  22. For the standard setting studies, the RFP requested that the contractor develop threshold PLDs. Would a methodology that uses range PLDs as opposed to threshold PLDs be acceptable?
    NYSED would consider using range PLDs as opposed to threshold PLDs for the standard setting studies, depending upon the standard setting methodology that is agreed upon.

  23. For the alignment studies, should the contractor apply the same compensation and travel reimbursement policies that would be applied to NY educators for the standard setting studies?
    NYSED is not requiring that NY educators be involved in the alignment studies nor requiring that this activity be carried out in New York State. Consequently, the same compensation and travel reimbursement policies described on pages 21 and 40 of the RFP do not apply. However, as indicated on page 74, “Expenses for travel, lodging, and subsistence shall be reimbursed at the per diem rate in effect at the time for New York State Management/Confidential employees.”

  24. Are Math content specialists needed? On Page 43, Test Development Section, #4, there's a reference to a need for additional content specialists to cover “math, social studies, and science content language.” However, on page 16, Key Principles for the Current NYSESLAT, #6, references “. . . core content (ELA, social studies, and science).” Page 24, in the Item Development Requirements and Guidelines, bottom of the page also references “. . . the following content areas: ELA, social studies, and science.” Attachment B, page 93, states, "The passages represent a mix of the content areas of science, social studies, and English language arts.” In Attachment B, Appendix F, page 113, NYSESLAT Test Design Elements, #6 of the critical elements references “. . . as well as the content areas of ELA, science, and social studies.”
    No, math content specialists are not required.

  25. Is there any expectation that the content area of mathematics will be used as a context for NYSESLAT passages?
    No.

  26. The 2nd paragraph on page 16 states that it may be necessary to attend up to 5 Content Advisory Panel and TAC meetings in New York. At the bottom of page 20, it states that the contractor may be required to attend up to 2 TAC meetings per year. Do the 5 possible meetings mentioned on page 16 include the 2 TAC meetings mentioned on page 20?
    Yes.

  27. NYSED normally requires that Standard Setting be set up with breakout groups for each grade band. Since NYSED is introducing the split of the current grade band 1-2 into separate Grade 1 and a Grade 2 tests, is it NYSED desire to have 7 breakout groups for the Standard Setting?
    Yes, it is desired to have 7 breakout groups for the Standard Setting.

  28. Would NYSED consider 6 breakout groups, with K–1 combined into one grade band?
    NYSED prefers to have 7 breakout groups.

  29. Is the forms construction for the 2023 operational test forms within the scope of this contract? If so, should a line item be added to the cost proposal worksheet in year 5?
    The forms construction for the 2023 operational test forms is not within the scope of this contract.

  30. Does the contractor have any responsibility for retrieval or storage of the scannable answer sheets for the operational tests?
    No.

  31. Please confirm that student score reporting is not within the scope of this contract.
    The student score reporting is not within the scope of this contract.

  32. Please confirm that, other than developing items that may eventually be used on the NYSITELL, the contractor has no additional responsibility for the production of test materials, administration, and scoring for the NYSITELL.
    This is confirmed.

  33. The RFP indicates that development of threshold PLDs is done at the beginning of the standard setting meeting. However, the table on page 23 and the cost proposal worksheet both indicate that there is a separate meeting in Fall 2019 with educators to develop PLDs for the revised NYSESLAT. Please clarify.
    It is expected that, at the standard setting meeting, the educator participants will develop threshold PLDs by reflecting upon the PLDs for each performance level developed by NYS educators at the Fall 2019 meeting.

  34. Page 30, #3: The contractor must provide separate test booklets for each of the sessions for each student. Attachment J, page 128, the print quantity for Operational Speaking test booklets is 1 test booklet per 8-10 per students. What is the print quantity for Speaking test booklets?
    Unlike the L/R/W test booklets, the Speaking test booklets may be reused with successive test takers. The Speaking test booklets must be supplied to each school at a ratio of one booklet per fifteen students with a minimum of two Speaking session booklets per grade band requested by the school. For each grade band except K, there are separate test booklets for each of the three L/R/W sessions. Grade K has separate test booklets for each modality. The quantities of test booklets that were printed for the 2017 administration of each grade band are provided in Attachment J of the RFP.

Fiscal

  1. There is not a line item on the budget for the Content Advisory Panel meetings. Would NYSED be open to revising the cost proposal spreadsheet to include this as a separate line item for budget purposes?
    No. The cost proposal spreadsheet will not be revised. Bidders should distribute the costs of these meetings, and other program administrative activities not attributed to a specific deliverable, throughout the cost proposal.

  2. There is not a line item on the budget for the TAC meetings. Would NYSED be open to revising the cost proposal spreadsheet to include this as a separate line item for budget purposes?
    No. The cost proposal spreadsheet will not be revised. Bidders should distribute the costs of these meetings, and other program administrative activities not attributed to a specific deliverable, throughout the cost proposal.

  3. Would NYSED be open to revising the cost proposal spreadsheet to include the annual start-up meetings and quarterly meetings as a separate line item for budget purposes?
    On page 42, #2 and #3, it mentions that the contractor will need to travel to Albany or New York City to meet with NYSED program staff to participate annual start-up meetings and quarterly status meetings (one full day in length). There is not a line item on the budget for these meetings. No. The cost proposal spreadsheet will not be revised. Bidders should distribute the costs of these meetings, and other program administrative activities not attributed to a specific deliverable, throughout the cost proposal.

  4. Page 51 indicates that all electronic files should be submitted in Word. Please confirm that the Cost Proposal package should be submitted in Excel, per the instructions on page 54.
    This is correct. The Cost Proposal package should be submitted in Excel, not Word.

  5. In year 3 of the cost proposal, it is indicated that the contractor will need to commission passages for the 2021 Field Test, but it does not include Item Writing. Should item writing be included with this task since it has been included in the other contract years?
    Yes. As indicated on page 14 of the RFP, in March 2020, the contractor will "Develop Passages and Items for the 2021 Field Tests Based Upon the New NYS Next Generation ELA Standards." The contractor should include the cost of item writing in the Mar-Apr 2020 “Commission Passages for 2021 Field Tests” deliverable.

M/WBE

  1. Page 10: The overall goal for M/WBE participation is 30% (17% MBE, and 13% WBE). Page 11: Bidders should consider SDVOB participation of 6%. Page 47: Subcontracting is limited to 30%. 17% + 13% + 6% = 36%. Is the subcontracting limit 36%?
    The subcontracting limit is 30%. Proposals that exceed the limit will not be considered for an award.

    All bidders are required to comply with NYSED’s Minority and Women-Owned Business Enterprises (M/WBE) policy and therefore responsible for identifying areas in their budget where an M/WBE can be utilized. M/WBE participation is not limited to subcontracting and includes services, materials, or supplies purchased from minority and women-owned firms certified with the NYS Division of Minority and Women Business Development. The MWBE Coordinator is available to assist bidders throughout the procurement process and can be reached at MWBE@nysed.gov.

    Bidders are also encouraged to utilize SDVOB as subcontractors and/or supplies at a suggested rate of 6%.

RFP 18-013: Educator Committees – Updated 11/17/17

Date

Committee

# of NYS Educators

# of Days

Honorarium
Per Day

Purpose

Product

Summer
2018
2019
2020
2021
2022

Field Test Passage and Item Review

54

4

$200

Review passages and refine prospective field test items

Satisfactory Field Test Passages and Items

Nov-Dec
2018
2019
2020
2021

Field Test Final Eyes Review

15*

3-4*

$200

Final review of field tests

Print-ready Error-free Field Tests

October-November
2019
2020
2021

Operational Tests Forms Construction

36

4-5

$200

Review tests forms proposed by contractor and select alternative passage/
item sets

Educator judged develop-mentally appropriate operational test forms aligned to the test blue prints

December
2019
2020
2021

Operational Test Final Eyes Review

15*

2-3

$200

Final review of operational tests

Print-ready Error-free Operational Tests

Spring/ Summer
2019

Test Specification Creation/Item Prototype Development

54

4

$200

Develop test specifications and item prototypes for the new NYSESLAT aligned to new NYS Next Generation ELA standards

Test Specifications and Prototype Items Aligned to new NYS Next Generation ELA standards


Summer
2019
2020
2021
2022

Rangefinding for Field Tests Speaking and Writing

54

4-5 days for each,
Speaking and Writing*

$200

Choose exemplar papers

Scoring Materials

Fall 2019

Performance Level Descriptor Development

25

2

$200

Develop/ revise performance level descriptions for each of the five performance levels

Performance Level Descriptions

Spring/
Summer
2020

Test Specification Creation/Item Prototype Development for new Kindergarten Entry Initial Identification Test

7

3

$200

Develop test specifications and item prototypes for new Kindergarten Entry Initial Identification Test

Test Specifications and Prototype Items Aligned to new NYS Next Generation ELA standards for the new Kindergarten Entry Initial Identification tests

January
2021

New Kindergarten Entry Initial Identification Field Test Stimuli/Item Review

7

1

$200

Review passages and refine prospective field test items for new Kindergarten Entry Initial Identification field tests

New Kindergarten Entry Initial Identification Field Tests

March
2022

Final Eyes Review of new Kindergarten Entry Initial Identification Field Tests

7

1

$200

Final review of field test forms for new Kindergarten Entry Initial Identification

New Kindergarten Entry Initial Identification Field Tests

Summer 2022

Standard Setting for Operational Tests

106

3

$200

Determine cut scores

Cut Scores

*Updated 11/17/17

 

Last Updated: November 22, 2017 5:15 PM