Today we'll be discussing how to write good multiple choice and free response questions. We'll also discuss ways of improving group dynamics in section.

1. Deeper conceptual knowledge CAN be probed with multiple choice questions, but writing effective questions takes time and practice.
2. Especially when writing free response questions, it can be useful to develop a grading rubric for each question as a way of ensuring that your questions are specific, clear, and not testing the same concept over and over again.
3. Group dynamics are not an easy thing to change. Be willing to try different tactics to get students to collaborate with each other.
4. If you use worksheets, think about how they might impact group dynamics.

First have students discuss in pairs how section went (5 min).

Open the floor up for general questions and sharing about how sections are going. Some questions include:

• What did you do?
• How did you implement your activities?
• What worked?
• What didn't work?
• What would you do differently?
• How did you assess learning?
• Did you receive any unexpected questions/reactions/etc.?
• Did anything unexpected happen?
• What were you thinking about while you were running section? Any moments of panic?

The students should have prepared one multiple choice question and one free response question as last week's homework.

1. (10 min) Share your questions with a partner. Have them attempt to answer the questions or at least determine what concepts were being probed and what would have constituted as an “acceptable answer.” Once you have gone over all the questions, discuss in your small groups the goods and suggested improvements for the questions. Some questions to consider (perhaps project on the board):
   1. For both:
      • Is the wording clear?
      • What concepts are being tested?
   2. For the multiple choice:
      • Is there clearly only one correct answer?
      • Are there any obvious throw-away answers?
      • Are you able to rule out any response because of the wording alone (i.e., are there hints in the structure)?
   3. For the free response:
      • Does part B test the small conceptual/procedural knowledge as part A?
      • What if students cannot solve part A? What does that imply for part B?
      • What sort of responses might students give under the pressures of an exam setting?
2. (5-10 min) Come back as a class and discuss.
   • Did you learn anything surprising?
   • Is this easy? (Unfortunately, NO!)
   • What part of question writing did you find the most difficult?
3. (20-25 min) Go through question example slides as a class exercise.

Some notes on multiple choice questions:

Despite their outward appearance, these questions are actually inherently nonobjective. Grading an essay exam is subjective to the personal feelings of the grader, compared to running a Scantron through a machine. Grading written problems falls somewhere between the two. This is only partially correct, “grading” a Scantron is completely objective, but the subjective aspect of multiple choice questions comes in the creation of the item (the question), the response (the correct answer), and the distractors (the incorrect choices). If everyone in the class was to write a question about the Doppler Effect, we would see a range of different questions and a range of ideas probed. That is subjective.

The ultimate goal of testing is to measure what the students actually understand, and the process of interpreting the meaning of a student's response to a MC test is a subjective one. There are three major issues behind writing these sort of questions:

1. the physical format and layout of the question
2. the conceptual hierarchy of the questions
3. the statistical item analysis

Below are some guidelines for each of these items.

You are testing the students' understanding, not their reading ability. Long passages of text cause slow readers to skim and often miss details. Questions should not include strings of prepositions, parenthetical statements, or extended clarifications. Misinterpretation is impossible to completely predict, but concise, clear questions can do a lot to minimize the chance of students misreading the question or the response. For example,

You forget that the star Betelgeuse is a red giant and apply the method of magnitudes to determine its distance. The true distance to Betelgeuse is actually… (a) shorter than you calculated, (b) the same distance that you previously calculated, © farther than you calculated.

Is short, to the point, and clear. You might be tempted to elaborate on small points that are not the main conceptual item that is being tested, but care must be taken. For example,

You forget that the star Betelgeuse is a red giant (a very luminous star in the top right of the HR diagram with relatively low surface temperature) and apply the method of spectroscopic parallax—a comparison of the star's apparent magnitude, estimated from the HR diagram, and its absolute magnitude—to determine its distance from Earth, which can be considered the sam as its distance to the Sun because the Earth-Sun distance is negligible given the scales involved. The true distance from Earth to Betelgeuse is actually…

In an attempt to be completely clear, the stem has become more difficult for most students to understand!

Over the years, students have learned that when novice faculty includes choices such as “all of the above”, these are frequently the correct answer. It's easier as a test writer to write correct statements than to come up with plausible sounding incorrect statements. Also, students have also learned that the longer answers are usually the correct answers. You can avoid these situations by making sure your choices are all of similar length, contain a similar amount of scientific vocabulary, and ensure that an equal number of choices (A), (B), etc. are correct on the overall test.

Consider

The thermo nuclear reactions in a stellar core are the result of (A) fission, (B) fusion.

This tests what? Unfortunately, it tests word association. A student can get by without knowing what fusion means. Students learn to adopt a strategy of memorizing definitions and words, rather than understanding concepts. As another example,

The Monotillation of Traxoline (attributed to Judy Lanier) It is very important that you learn about traxoline. Traxoline is a new form of zionter. It is monotilled in Ceristanna. The Ceristannians gristerlate large amounts of fevon and then bracter it to quasel traxoline. Traxoline may well be one of our most lukized snezlaus in the future because of our zionter lescelidge.

Directions: Answer the following questions in complete sentences. Be sure to use your best handwriting. (1.) What is traxoline? (2.) Where is traxoline monotilled? (3.) How is traxoline quaselled? (4.) Why is it important to know about traxoline?

Notice how easy it is to get 100% without understanding a single thing about the passage! Pay attention to wording.

Less applicable for GSIs, but if particular questions are frequently missed, it is necessary to probe whether they are missed because of a lack of clarity, or because the question is testing difficult concepts. If the former, revise!

See below for some general bullet points on what to look out for.

Some notes on free response questions:

These notes are written with “Astro C10 quizzes” in mind, but the ideas are generally applicable.

• These points are valid for both quizzes and exams:
  • Test the material emphasized - Exams should reflect the fact that students should know the big concepts really well, as opposed to knowing a bunch of smaller concepts only peripherally.
  • Keep questions short and to the point - Students should spend the majority of their test time thinking and answering/writing, NOT reading.
  • Edit questions for clarity - Clear questions tend to be shorter and if anything is ambiguous it confuses and slows down students and makes it harder for you to grade it fairly. Have someone else take your exam to give some feedback. If you're taking an exam for someone else, be critical and think about possible ambiguities.
  • Don't write a long test - Keep it concise, to the point, and clear! The rule of thumb is your average student will take double or triple the time it takes a GSI to complete the exam. Also, 90% of your students should finish the exam completely in the allotted time.
• Quizzes vs. Homeworks
  • They're quite similar in their construction and type of questions.
  • The main difference is that quizzes should be shorter and have easier questions, since students have much less time to work on quizzes and must work on them alone.
• Quizzes are:
  • short
  • usually given in section
  • questions are of exam difficulty (i.e. easier than homework questions)
  • Not too in-depth or calculation-based. Some light calculation might be OK, but take care. (Not everyone will remember a calculator no matter how many times you remind them, and many people will be petrified of the idea of computing things on their own.)
  • usually only cover recent material (i.e. the past 2-3 weeks)
• Quizzes are used:
  • by the GSI to gauge each students' understanding of recent material individually (as opposed to homeworks which can be done in groups)
  • by the student to gauge their own understanding of recent material and get a feel for what a college level intro science course non-Scantron exam will look like and what level of understanding they are expected to have for the exams
  • by the prof to get a grade early in the semester that's more important than a single homework, but isn't the big production that an exam is (you might not have covered enough material for a full exam)
• What makes good ones?
  • not too long (both in length of individual questions and number of questions)
  • not too hard
  • not too easy (shoot for a variety of difficulties in questions)
  • relevant to recent material
  • varied in the types of questions (multiple choice, fill in the blank, calculation/mathematical, read a graph, free/paragraph response)
  • unambiguous with easy-to-read questions
  • not mathematically demanding - questions should probably not require a calculator and should definitely not include extensive tedious calculations
  • representative of same knowledge required for exams
  • gradable for partial credit (not simply binary right/wrong like Scantron exams)
  • specific about what you're looking for in free response type questions: Don't give students the opportunity to 'core-dump' for a problem, it wastes their time spewing forth useless information and makes your life tougher when you have to grade the mess.
  • quick to grade (this makes your life much easier and helps the grading be more fair for all of your students)
• Quizzes are meant to be relatively low stress (especially compared to full exams)

General notes from previous years in convenient bullet-point form:

• Exams exist to:
  • Evaluate student learning for University-required grading
  • Motivate students to study and understand the material
  • Allow the instructor to evaluate his/her progress educating students about the material
  • Provide feedback to students about their understanding and study habits and illustrate specific gaps in their understanding of the material

• What makes good multiple choice exam questions?
  • Not too long/wordy (neither the question nor the possible answers)
  • Not too much calculation
  • Not too tricky (i.e. there shouldn't be two extremely similar answers)
  • Relevant to important material (as opposed to really obscure/minute details)
  • Very clearly written, precise wording in both question and answers
    • E.g., 'Which best describes…' as opposed to 'What is…' or 'How does…happen'
  • Pedagogical as well as evaluative (e.g., some questions should probe common misconceptions)
  • Questions that are very easy or very difficult are OK as long as the test has questions with a variety of difficulties

• What makes bad ones?
  • Long answers!
  • Excessive use of 'all of the above' (some people say any use of 'all/none of the above' is a bad thing)
  • Questions that can be solved without knowledge of the material (usually because of the use of too many blatantly wrong or “funny” possible answers)
  • Multiple potentially correct answers (usually from vague questions or possible answers)

• Non-multiple choice questions
  • Types of questions:
    • Matching
    • Fill-in-the-table/blank
    • Simple calculations
    • Short answers and paragraph/free responses
    • Diagrams, plots, graphs
  • Most of the same points discussed above apply here:
    • Questions should be clear, easy to read, and unambiguous
    • Questions should be relevant to the material presented and emphasized (do not test on obscure passages of the textbook)
    • For high-value questions, allow for partial credit
    • Make the questions easy to grade! Don't give students the opportunity to 'core-dump' for a problem: be very specific about what you're looking for in these questions.
  • Can be more time-consuming than MC questions, depending on the overall length of the exam.

In previous classes, we talked about the benefits of group work and different ways of organizing group work. Today we want to talk about how to improve group dynamics, both in small groups and class discussion.

In small groups (3-4 people), you'll be given a scenario to think about. What could you do to improve the group dynamics in your scenario? Brainstorm some possibilities in your groups for 10 min and then we'll share with the class as a whole.

Scenarios:

• In one of your small groups, you notice that one of the students is doing most of the work. His partners are mostly copying what he does but with very little interaction between them.
  • Ask one of the other group members to explain their thought process when you walk up to their table. If he/she doesn't know, ask the student what it is that's confusing. Ask other group members if they have an answer. Try to get a conversation going. Point out that being able to explain your reasoning to others is a valuable skill and helps you better understand the material.
  • Talk to the student doing the work after class about the benefits of teaching his partners.
  • Create group activities that require each group to report to the class a whole or to other small groups. Force a different student to present to the class each time.
• In one of your small groups, you notice that one student is very quiet. The others are ignoring him since he's not speaking up.
  • Go over to the table and ask the quiet student his thoughts on the question/project at hand. If he expresses confusion, ask him/her to pose a question to the group and give status to the student, saying “That's a great question.” If he expresses an idea, give him status by saying “That's an interesting idea.” If his idea is wrong, try to involve him in the group conversation by saying something like, “I think there's a lot of students who have that same idea. This may be something your group is asked during large class discussion. What does the rest of the group think about this?” Or you could then ask the same student, “How could you test whether this idea/method is correct?” Try to get students to correct each other and explain their reasoning to each other, instead of stepping in and saying what's wrong or right.
  • Create group activities that require each group to report to the class a whole or to other small groups. Force a different student to present to the class each time.
  • Emphasize to the group or the whole class that everyone has something to contribute to group work. Even wrong ideas can be useful in terms of clarifying the group's thinking and explanation.
• During class discussion, two students almost always raise their hands to answer your questions or volunteer to speak for their group, but the rest of the class seems to be disengaging.
  • Try just waiting more time for other people to raise their hands.
  • Thank the two students for their contributions but say you'd like to hear from other students as well.
  • If the discussion was preceded by group work activity, call on a specific group that the dominant students are not in.
  • Ask students to think about the question in pairs. Then try asking for a volunteer again or walk around and if you hear the right answer, ask that student to share it with the class later.

During class discussion, as groups are sharing, ask if other groups have additional strategies to contribute.

What other group dynamics have you encountered in section? Did you try to “fix” them? If so, how and did it work? If not, how could this type of situation be dealt with?

Sometimes you probably use worksheets as part of group work. How can worksheets help or hinder group work? Consider both the content of the worksheet (e.g. the types of questions) and how you use it (e.g. do you hand one out per group or per person)?

Some takeaways:

• Giving only one worksheet per group can help promote group work. However, giving worksheets to individual students can encourage them to take notes and have something to take away after section.
• If you only give one worksheet per group, think of ways to encourage students to take notes of the important takeaways, perhaps by writing on board during class discussion.
• If you give individual student worksheets, make questions more open-ended, require finding an optimal way of solving a problem, etc. Try to ask questions in a way that encourages them to talk to each other, exchange ideas.

Useful for academic faculty jobs, even at research institutions! We will come back to these in more details at the end of the semester, but we'll give you the material below if you are curious.

• The 5 main points to cover/things to do in a Teaching Philosophy are:
  1. Goals for student learning
  2. Enactment of goals/teaching methods
  3. Assessment of goals (measuring student learning)
  4. Creating an inclusive classroom and acknowledge differences in students' learning abilities and styles
  5. Good structure, rhetoric, language; well-written
• General guidelines:
  1. Brief (1-2 pages)
  2. Narrative, first person approach
  3. Portray yourself as an individual: don't be vague, broad, or too general
  4. Give specific examples of your teaching in practice
  5. Showcase your strengths and accomplishments
  6. Convey reflectiveness
  7. Communicate that teaching is valued
• Michigan's Center for Research on Learning and Teaching has a great site on this stuff here

You might try this to get you started: In at least one paragraph, but not more than one page, answer the following questions:

1. I bring to teaching a belief that …..
2. In the classroom I see myself as …..
3. I believe students are …..
4. I seek to foster in students …..
5. I think the role of discussion section is …..

Or

• What is your role as the instructor? What is the students' role in the classroom?
• What sort of pedagogical tools do you employ and why? Can you SHOW us these tools through an example activity rather than just listing?
• Are there aspects about your philosophy you think are unique?
• How do you assess whether your strategies and activities are working?
• How has your teaching style, philosophy, etc. evolved?

Or you might use some prompts given from the GSI Teaching and Resource Center:

1. What motivates me to learn about astronomy?
2. What are the opportunities and constraints under which I learn and others learn?
3. What do I expect to be the outcomes of my teaching?
4. What is the student-teacher relationship I strive to achieve?
5. How do I know when I have taught successfully?
6. What habits, attitudes, or methods mark my most successful teaching achievements?
7. What values do I impart to my students?
8. What code of ethics guides me?
9. What theme(s) pervade(s) my teaching?
10. Can I give a quick 15-minute window into my section/classroom?

• Draft a one-page teaching philosophy.
• We will be visiting the sections that have not yet been visited next week. For those who received their videos today, watch them and then e-mail us to find a time to meet.
• The second extended teaching log is due next week.