UNDERSTANDING STUDENTS’ FREE-BODY DIAGRAMS USING THE METAREPRESENTATIONS SURVEY FOR PHYSICS

Study purpose. The Metarepresentations Survey for Physics (MSP) was developed to assess students’ metarepresentational knowledge during physics problem solving. Materials and methods. The survey was given to 288 introductory-level college physics students. Psychometric properties of the instrument, including construct validity, were evaluated by confirmatory factor analysis and Rasch analysis. Results. We also examined students’ beliefs about the use of free-body diagrams, as well as thoroughly examined the link between students’ problem solving success and free-body diagrams. Conclusions. We recommend the use of the MSP for physics instructors and science education researchers who want to evaluate students’ free-body diagrams. Additionally, we suggest the subject of physics can be replaced with chemistry, genetics, or another science to assess metarepresentations in other domains.


Introduction
Representations in physics are verbal, mathematical, graphical, or pictorial depictions used by students to interpret concepts and solve problems (Supeno, Subiki, & Rohma, 2018). Metarepresentations, similar to metacognition, are what students know about their representations (Kohl & Finkelstein, 2006). Drawing a pictorial representation of the information presented, typically referred to as a free-body diagram, is an important part of physics problem solving. Physics textbooks, instructors, and researchers emphasize the importance of these free-body diagrams because they allow the problem solver to identify the involved objects and forces and the interaction between them, determine the appropriate approach to the problem, and to reduce the amount of information that must be attended to at one time (diSessa, 2004). Despite the emphasis on free-body diagrams, there is some question as to whether and how students are creating and using diagrams when solving physics problems. Rosengrant, Van Heuvelen, and Etkina (2009) studied students in an introductory-level course for physics majors and found that although most students drew free-body diagrams, only a small portion drew them correctly by including significant components. Students who drew diagrams with those significant components were more likely to correctly solve the problems. Interviews with groups of high achieving and low achieving students indicated stark differences in the creation and use of these diagrams. High achieving students used the diagrams in two important ways: first, to help them solve problems, and second, as a way of evaluating their problem solving equations. In contrast, the low achieving students primarily drew the diagrams using the steps learned in class without having a full understanding of what components they were drawing, and furthermore, they did not check for accuracy between their diagrams and their problem solving approach. One critical finding of Rosengrant et al. 's study was that drawing an incorrect free-body diagram led to more incorrect solutions than having no diagram at all, suggesting that a wrong free-body diagram is worse than no free-body diagram.
However, just drawing a high-quality diagram is not a panacea. Taasoobshirazi and Carr (2009) used structural equation modeling to study the relationships between the use of free-body diagrams, conceptual knowledge, strategy use, and performance among introductory-level college physics students. They found that greater conceptual knowledge was not linked to the use or quality of free-body diagrams; the use and quality of free-body diagrams was also not linked to effective strategy use. Therefore, even when students drew free-body diagrams, and drew them well, this did not result in improved strategy use or performance. Also, more advanced conceptual knowledge was not related to the use of free-body diagrams. The authors theorized that students are drawing diagrams and even including important components of the diagrams, but not actually using them to solve physics problems.
Beyond the two studies described above, there is little research on how students are using free-body diagrams when studying or solving physics problems; there is a complete lack of research on students' understanding of their freebody diagram use. Van Heuvelen and Zou (2001) found that students learn more if they understand the reason behind various pedagogical strategies such as using diagrams to solve problems. Understanding students' free-body diagram use is an essential prerequisite to teaching students this reasoning.
Metarepresentation is parallel to metacognition, but the focus is on what students know about and how they use representations (Sherin, 2000). Metarepresentation is the knowledge and regulation of one's representations rather than the knowledge and regulation of one's cognition. Given the emphasis on free-body diagrams in physics, understanding students' metarepresentational skills is an important goal. Despite its importance, there is a dearth of research on metarepresentations. Because of the lack of research on metarepresentations, the research on metacognition was used as a framework for understanding the components of "thinking about one's representations" and developing the Metarepresentations Survey for Physics (MSP; Taasoobshirazi, Bailey, & Farley, 2015).

Theoretical Framework
Metacognition is a "cognitive activity that takes as its object, or regulates, any aspect of any cognitive enterprise" (Flavell, 1985, p. 104). The research on metacognition distinguishes between two components: 1) knowledge of cognition and 2) regulation of cognition (Dinsmore, Alexander, & Loughlin, 2008). Knowledge of cognition is what individuals know about their thinking and learning and is comprised of three different types of metacognitive knowledge including declarative, procedural, and conditional knowledge (Schraw, 2001;Veenman, 2007). Declarative knowledge is knowledge about oneself as a learner or a problem solver. Procedural knowledge is knowledge about how to complete an activity or problem. Conditional knowledge is knowledge about when and why to use one's declarative and procedural knowledge.
The second component of metacognition, regulation of cognition, comprises behaviors that help learners manage their learning and/or problem solving (Dinsmore et al., 2008;Veenman 2007). Regulation of cognition includes at least three types of metacognitive regulation, including planning, monitoring, and evaluation (Schraw, 2001;Schraw, Crippen, & Hartley, 2006). Planning is described as planning and goal setting prior to completing a task or problem. Monitoring is the continuous assessment of one's goals, work, and performance during a task. Evaluation is the judgement of one's work after completing a task.
We believe there is conceptual overlap between metacognition and metarepresentation as both focus on the meta aspect of cognitive activities. Some metacognition researchers consider an additional component of regulation of cognition called information management (Schraw & Dennison, 1994). Information management includes strategies that help a learner solve problems effectively, such as using free-body diagrams, and has been studied as such (Taasoobshirazi, Bailey, & Farley, 2015). Although metarepresentation could be considered a component of metacognition, we study metarepresentations as the metacognition of representations and as its own construct that focuses specifically on the knowledge and regulation of one's diagrams.
To date, there is no inventory or assessment that evaluates students' knowledge and regulation of their use of representations in science. The Metarepresentations Survey for Physics (MSP) was designed to objectively, validly, and reliably assess students' metarepresentational skills in physics. Assessing students' metarepesentations is a necessary prerequisite for understanding how students are using free-body diagrams and how those diagrams are linked to problem solving success. Free-body diagrams are considered a critical part of the physics problem solving process and are heavily emphasized in physics classrooms and texts. We developed the MSP to evaluate the six components of metacognition discussed above. We also examined students' beliefs about the use of free-body diagrams, as well as the link between students' problem solving success and use of free-body diagrams.

Study participants
Two hundred eighty-eight physics students (194 men, 93 women, and 1 non response) in two sections of an introductory level, calculus-based course at a university in the South Central part of the United States were given the 13 item MSP. There were approximately 400 students in the course. Students completed the anonymous survey for a small amount of extra credit. Regarding ethnicity, approximately 61% of the students self-identified as White, 8% Hispanic/Latino, 7% Black, 4% Middle Eastern, and 8% other. The survey was administered to students during the 10th week of classes. Informed consent was collected from students, participation was voluntary, and the study was conducted in compliance with the university's Institutional Review Board.

Organization of the study
The 13 items were developed using the research on freebody diagrams and on metacognition as well as guidelines for survey development presented by Pett, Lackey, and Sullivan (2003). These guidelines include evaluating research to identify appropriate latent variables and creating well-designed empirical indicators, or items, for those latent variables. Items 1, 2, and 3 were designed to measure procedural knowledge; items 4 and 5 measured declarative knowledge, items 6 and 7 measured conditional knowledge; items 8 and 9 measured planning; items 10 and 11 measured monitoring; and items 12 and 13 measured evaluation. Students were asked to respond to each of the 13 items on a 5-point Likert-type scale ranging from 1 (never true of myself) to 5 (always true of myself) with the instructions: "In order to better understand how you solve problems in physics, please respond to each of the following statements from the perspective of: When solving physics problems". After completing the MSP, students were asked for their gender and race and were asked the question: "Are free-body diagrams important for physics problem solving? Why or why not?" In one section of the course, students completed a set of five physics problems. Fifty of the physics problem sets were randomly selected and analyzed. These problems were used to evaluate students' problem solving success and free-body diagram use.

Confirmatory Factor Analysis
A confirmatory factor analysis using LISREL 10.2 indicated that the MSP had strong construct validity. The items, as proposed, all loaded significantly on their respective factors (using a cutoff of t = 1.96 to assess significance). The measurement model is presented in Figure 1 with items and standardized factor loadings; items, item loadings, and variance explained by the factors are presented in Table 1. All correlations between latent variables were significant. Table 2 presents descriptive statistics for the six factors of the MSP. The data met the assumption of multivariate normality (Mardia's coefficient = 1.25) so maximum likelihood estimation was used to test the model. To evaluate the fit of the model, several fit indices were considered. The normed chi-square was 110.47/50= 2.21. The Steiger-Lind Root Mean Square Error of Approximation (RMSEA) was 0.06. The standardized root-mean-square residual (SRMR) was .04. The Bentler comparative fit index (CFI) was .97. The incremental fit index (IFI) was .97. These fit indices all met recommended cutoff values (e.g., Browne & Cudeck, 1993;Hu & Bentler, 1999;Kline, 2016), signifying that the model fit well.

Rasch Analysis
A Rasch analysis was conducted to evaluate the psychometric properties of the MSP. Rasch analysis is common technique used to evaluate the construct validity of instruments in the health and social sciences (Lim, Rodger, & Brown, 2009). We used guidelines by Boone, Staver, and Yale (2014), analyses interpretation by Taasoobshirazi, Bailey, and Farley (2015), and Winsteps 3.81 to test and evaluate the model. Table 3 presents results from the Rasch analysis: The entry number is the item's order. The total score is the sum of the raw scores of the five point Likert scale items for each item. The Rasch measure is the level of agreeablity of an item presented in logit units; items range from being most difficult to agree with at the top of the table and easiest to agree with at the bottom of the table. The mean Rasch measure is 0 and items that are easier to agree with are below 0 logits whereas items that are more difficult to agree with are above 0 logits. The model standard errors are presented in Table 1 and describe uncertainty around the item measures. The infit and outfit mean square (MNSQ) fit statistics are chi-square statistics that describe how well the data fit the Rasch model (Boone, Townsend & Staver, 2011). Values outside of the suggested ranges (less than .05 or above 1.5) indicate disparity of the data from what is expected from the Rasch model. Our values were within those ranges, suggesting that we had good construct validity (Baghaei, 2008). The point measure correlations are presented in the last column of the table. These correlations assess whether responses to items align with abilities of the persons (Linacre, 2012). Positive correlations are expected, meaning that higher person measures are linked to higher ratings on the items, and this was the case for our data. Figure 2 provides a Wright map that illustrates the relationship between persons and items (Wright & Stone, 1979). Persons and items are on the same units on the Wright map, which offers a comparison of items to items and persons to persons as well as persons to items. On the right side of the Wright map, the 13 items are sorted by level of agreement; items that are the most difficult to agree with are at the top of the map and items easiest to agree with are at the bottom. On the left hand side is the distribution of the 288 participants, and they are sorted so that those with higher metarepresentational skills are at the top and those with lower metarepresentational skills are at the bottom. The vertical line shows the interval scale of the logit values. Each hashtag (#) represents 3 persons and a period (.) represents one or two persons. An Taasoobshirazi, G., Heddy, B.C., Danielson, R.W., Abraham, E.R.I., & Joji, S. (2022). Understanding Students' Free-Body Diagrams Using the Metarepresentations Survey for Physics "M" represents the mean, "S" is one standard deviation from the mean, and "T" is two standard deviations from the mean. 1 The reliability (internal consistency as measured by traditional Cronbach's alpha) of the 13 items of the MSP was 0.90.
Survey items that have the same logit value as a person have a 50% probability of being agreed with by that individual. Items that are below a person's logit value have a greater probability of being agreed with (greater than 50%), whereas items above a person's logit value have a lower probability of being agreed with (less than 50%; Boone, Staver, & Yale, 2014).
Our Wright map shows, in general, that people were more agreeable than the items: the M for the participants was slightly higher than the M for the items. Including items that assess more complex metarepresentational skills (for instance, declarative knowledge items are at the top of the Wright map, suggesting those items are most difficult to agree with) may be useful in aligning the means of persons to items. One important note is that the findings from the Wright map  (Table 2). Average scores on the factors tended to be high, indicating that students responded to the survey in a way that suggested that they were highly aware of their use of representations. The students are highly metarepresentational and creating items that assess more complex facets of metarepresentations would be beneficial.
In the Rasch model, reliability is estimated for items and persons (Bond & Fox, 2007). Item reliability provides reliability information of the survey if the same items were administered to a different, but similar sample of individuals. For the MSP, item reliability was 0.96, which is excellent. The person reliability provides reliability information of the instrument if the same group of students were given a different, but similar set of items measuring metarepresentations. The person reliability for the PMI was 0.87, indicating good reliability 1 . Finally, separation coefficients evaluate whether the items are measuring the underlying concept or just introducing noise (Boone, Staver, & Yale, 2014). Separation coefficients were obtained for both persons (2.59) and items (4.72), and were both considered good levels, suggesting signal to noise ratio is high.
Based on the Rasch analysis, we found good fit statistics, indicating that the items are unidimensional and responses to the survey are consistent with the underlying theoretical construct. The Wright map suggested a solid item to person distribution, but a distribution with higher person scores than item scores.

Students' Beliefs about Free-Body Diagrams
After completing the MSP, students were asked the question "Are free-body diagrams important for physics problem solving? Why or why not?" With the exception of one student, all of the students responded that free-body diagrams are important for physics problem solving. The 288 responses were read and categorized based on themes.
1. The most common explanation for the importance of free body-diagrams was centered around the idea that freebody diagrams are necessary for representing the forces in a problem scenario (n = 157, 55%). Some example responses included: • Yes they help you isolate the forces acting on an object which can help you make sure you are considering all the necessary components and ignoring things that actually don't have an effect. • Yes the signs (directions) of objects or forces can be determined by a free-body diagram, and used as a tool to solve physics problem. • Yes so you can determine what forces act on an object at what angles and in what direction. 2. The second most common explanantion focused on the use of free-body diagrams to organize and/or visualize, in general, information in the problem (n = 96, 33%). Some example responses included: • Yes. it helps organize the information from the problem. • Yes because they allow you to visualize the problem and lay out specific pieces.
• Yes they are because it gives you an easier way to visualize what is going on in the problem and how to go about it. 3. A few students commented that free-body diagrams help with the understanding and simplification of the problem (n = 3). Some example responses included: • Yes. I believe they are important because they allow the reader to truly understand the problem and know what they need to solve. • Yes, it simplifies the problem in one picture. 4. There were a few students who, although they said free body diagrams are important, felt they were not necessarily personally useful (n = 9): • Never used them before. We were always taught to memorize equations or relationships. but in this case we are supposed to find the equation so I think they are useful if you're going to be an engineer. • Free body diagrams help me solve the problem, however, setting them up often confuses me, can mess me up, or can take too long. • I think they are, my peers use them to solve physics problems, I just never learned how to draw them right.

The Link between Students' Free-body Diagrams and Problem Solving Performance
Fifty randomly selected students were asked to solve five mechanics problems for a small amount of extra credit (Table 4). Although we were unable to link students' problem solving to their MSP scores (because of anonymity of the surveys), we wanted to examine the relationship between students free-body diagram use and their problem solving performance. The first of the five problems was designed by Priest and Lindsay (1992) and was used in their physics education research. The other four problems were from a physics final examination for an introductory level physics course that was not a part of this study. The problems were selected to ensure that students had not seen the problems before this exercise.
A physics education researcher scored students' problem solutions (providing partial credit where appropriate). In ad-

Table 4. Physics Problems
A block of mass 7 kg starts sliding down a plane of length 5 m, inclined at an angle of 30 degrees to the horizontal. If the coefficient of friction between the block and the plane is 0.2, find the velocity (vt) of the block when it reaches the bottom of the plane.
An airplane flies horizontally with a speed of 300 m/s at an altitude of 400 m. Assume that the ground is level. What horizontal distance from a target must the pilot release a bomb so as to hit the target?
A 0.15 kg steel ball is dropped onto a steel plate where its speed just before impact and after impact is 4.5 m/s and 4.2 m/s, respectively. If the ball is in contact with the plate for .03 seconds, what is the magnitude of the average force (in N) applied by the plate on the ball?
An escalator is 30.0 meters long and slants 30 degrees relative to the horizontal. If it moves at 1.00 m/s, at what rate does it do work in lifting a 50.0 kg man from the bottom to the top of the escalator? A 1.0 kg block is released from rest at the top of a frictionless incline that makes an angle of 37 degrees with the horizontal. An unknown distance down the incline from the point of release, there is a spring with k = 200 N/m. It is observed that the mass is brought momentarily to rest after compressing the spring 0.20 m. What distance does the mass slide from the point of release until it is brought momentarily to rest? dition, free-body diagrams for each problem were scored. These diagrams were scored for both quantity (a 1 was given if a diagram was drawn and a 0 was given if there was no diagram drawn for the problem) and quality. Quality of diagrams were determined by comparing students' diagrams to a target sketch that was drawn by a physics instructor and included the necessary components (e.g., angles, forces) needed to have a complete and thorough free-body diagram. For the first problem, a complete sketch included the representation of three forces (normal force, frictional force, and force of gravity broken into its horizontal and vertical components) and one angle, resulting in a total of four components.
In the second problem, a complete free-body diagram included: the change in the vertical distance and the horizontal velocity. In the third problem, a complete diagram included: the velocity of the object both before and after collision. In the fourth problem, a complete free-body diagram included four factors: two forces (the normal force and the force of gravity broken into its horizontal and vertical components), length of the slant, and the angle of the slant. In the fifth problem, a complete free-body diagram included the angle, spring, and length of the compression of the spring, for a total of three factors. For each diagram, students received 1 point for each component drawn, with a range of scores being between 0 and 15 (Taasoobshirazi & Carr, 2009). SAS 9.4 was used to examine the correlations between students' scores on the five problems, if a diagram was drawn (yes/no), and diagram quality. Students' scores on the five problems (M = 2.33, SD = 1.50) was negatively, but not significantly correlated with whether a diagram was drawn (M = 4.04, SD = 1.12), r = -.12, p = .37. Scores on the five problems were also negatively, but not significantly correlated with diagram quality (M = 10.08, SD = 3.27), r = -.15, p = .30). Only diagram quality was significantly correlated with drawing a diagram, r = .85, p < .001. There was not a significant correlation between drawing a diagram or quality of diagram and correctly solving the problems.
This finding was not surprising to us. Students are drawing diagrams, but their diagram use is not linked to their performance. After each problem, we asked students "Did you draw a free-body diagram?" If yes, did you use it to solve the problem?" Ninety-four of the 202 times students drew a diagram, students stated that the diagram was not used to solve the problem. This means that students are responding on the MSP that they understand and can monitor their freebody diagram use (please note the averages in Table 2), are in general drawing diagrams 202 diagrams for 250 problems = 81% of the time. However, they are not using the diagrams 47% of the time. The questions is why? If students know that diagrams are important and are drawing them, why are they not using them to solve problems? Are students who say they are using the diagrams more likely to answer questions correctly than those saying they are not using them?
To help answer this last quesion, 50 problems with a freebody diagram drawn were randomly selected and the correlation between score on the problem (correct/incorrect) and whether a diagram was used to solve the problem (yes/ no) was calculated. The correlation phi = .12, p = .94. There was not a significant correlation between self-reported use of a diagram and correctly solving the problems.
There is the perspective that a free-body diagram should only include the body of interest and the external forces acting on it. This is because the purpose of the diagram is to determine the magnitude, direction, and point of application of external forces. For this reason, we randomly selected 50 packets and only scored the two problems (problems 1 and 4) that required forces to be included in the free-body diagrams. We tested the correlations between students' scores on the two problems, if a diagram was drawn for each problem, whether the diagram was used, and diagram quality (five points total for the five forces involved in the two problems). Students' scores on the two problems (M = 0.94, SD = .80) was not significantly correlated with drawing a diagram (M = 1.92, SD = .34), r = .13, p = .36, or diagram quality (M = 3.74, SD = 1.50), r = -.03, p = .83. Students' scores on the problems were also not correlated with their self-reported use of the free-body diagrams (M = 1.32, SD = .68), r = -.13, p = .36. One interesting finding was that drawing a diagram and diagram quality were both significantly correlated with selfreported use of the diagrams, r = .38, p = .007 and r = .50, p <. 001, respectively. Diagram quality was significantly correlated with diagram use, r = .52, p < .001. Therefore, although the relationship between diagrams and successful problem solving was absent, when forces were the focus of the drawings, there was a link between students' self-reported use of diagrams and diagram presence and quality. This may be because forces are a primary focus of the equations that the students set up to solve after drawing the diagram. However, the next important step or connection between diagram use and problem solving accuracy is missing. Tony Wayne, a high school physics instructor, has created a website teaching students why and how to use freebody diagrams (Wayne, 2020). He explains that the diagram is a starting point for developing a mathematical model of the forces acting on an object. If the mathematical model is the set of equations, then students who draw diagrams and draw more thorough diagrams should have increased problem solving accuracy. This was not the case for the students in present study or for students in the research that we reviewed. If free-body diagrams allow for improved understanding of the problem scenario, this should also translate to improved problem solving. This was not the case and, as discussed in the introduction of the paper, Taasoobshirazi and Carr (2009) did not find a link between diagram use and conceptual understanding of physics.
We present the questions: how are students using freebody diagrams? How should free-body diagrams be used and their effectiveness comprehensively measured? Are free-body diagrams changing the way students are writing their equations (such as changing the sign of the forces in the equations)? For instance, if a student represents a force incorrectly in their diagram and, in turn, their equations, this would lead to an incorrect solution. If students are drawing the diagrams with correct forces and are using those diagrams, could the disconnect between diagram use and problem solving accuracy be due to a third variable such as strategy use? Is there something to the effort of drawing the diagram itself? As an example, your instructor may allow you to make a note card to use during a test and a lot of energy is devoted to putting information on the card, but because you created the card, you did not actually need to use it during the test. Does this mean that students who draw the diagrams don't need to use them? Although we reviewed numerous tutorials that emphasize the importance of drawing a thorough free-body, none answered the question about how they should be used after they are drawn.

Discussion
The goal of this study was to develop and validate an instrument designed to evaluate students' metarepresentational skills in physics. A confirmatory factor analysis and Rasch analysis attested to the construct validity of the scale and its alignment to theory. The MSP is a brief, objective, valid, and reliable way to assess and understand students' metarepresentational skills. We recommend its use in physics classrooms and programs and for science education research. Additionally, we suggest that the MSP can be used to evaluate students' metarepresentational skills in other sciences, such as organic chemistry, where molecular diagrams play an essential role in problem solving. In such a case, the word chemistry can be substituted for the word physics.
Given the extensive emphasis on the use of free-body diagrams in physics and the disconnect between students' use of these diagrams and their performance, we felt it was important to be able to assess students' understanding and regulation of their diagram use. The evaluation of metarepresentational skills is a necessary requisite for studies of freebody diagram use and potential intervention studies. The present study adds to the discussion of free-body diagram use in physics and presents a clear need for additional and current work on how students are using free-body diagrams, an assessment of the relationships between free-body diagram use and educational outcomes in physics, and the need to explain the disconnect between free-body diagram use and successful problem solving for introductory level physics. The MSP will be instrumental in this vital research.
As with all research, there were limitations in the current study, one of which is that our findings are restricted to introductory level, calculus-based physics students. At present, there is insufficient data to inform us about free-body diagram use and metarepresentational skills among intermediate and upper level physics students. A second limitation was that this study, as well as the studies that were reviewed in the introduction of the paper, were all cross-sectional evaluations of diagram use. Administering the MSP to varying levels of physics students and at different times during their program of study of physics would help determine growth and change. Conditional, structural equation, and nested growth models can answer questions about mediating and moderating effects of related variables over time. A third limitation was our inability to link the students' diagrams and problems from the packets with their MSP responses. We weren't able to connect student work on the problems to the survey responses due to anonymity reasons. However, we recommend that researchers conduct more mixed methods research on this topic with a focus on, in what ways students solve problems and how they use free-body diagrams support their problem solving and how this relates to metarepresentations. Finally, there is a need for more qualitative research on the topic. Using think aloud protocols to explore how students use freebody diagrams would allow for a more detailed descriptions of this process as it related to metarepresentation in physics education.