The airplane's rudder is a vertical control surface that's part of the tail section. The trailing edge is movable from the cockpit. Turning the rudder to the left makes the plane's nose yaw to the left, and turning it to the right yaws the nose to the right.
The rudder is controlled using two floor pedals in the cockpit. Push on the left pedal to turn the nose left and the right pedal to turn the nose right. This action turns only the nose; it does not make the aircraft bank to the left or right. Ailerons control banking. You'll want to use the rudder pedals to move the plane's nose to compensate for any crosswinds while flying a straight line or landing.
Airplanes have a tendency to turn left, called the p-factor, due to the propeller's pitch or surface angle (asymmetrical propeller loading). This tendency or torque is greater at low speed, high power, and/or high angle of attack. Many planes include design features to minimize p-factor. As the pilot you also can use the rudder to control yaw.
Remember that the brakes (useful only when you're on the ground) are commonly attached to or located adjacent to the rudder pedals. Pushing them accidentally won't do anything while you're in the air, but press the brakes instead of the rudder on landing and you could have an accident report to fill out.