As associate chair of my department, I'm responsible for mentoring new hires during their first semester on campus, which includes writing an observation letter on their teaching. The way we usually handle this in my department is to give our new professors as much choice in the process as possible: among the classes I am free to attend, they are free to choose which course I will observe and when my two visits will take place (usually toward the beginning and toward the end of the semester). No pre-observation or post-observation meetings are required, but they can of course request them, and since we're meeting as needed to discuss any questions and problems they may have over the course of the semester, it's easy to discuss their classes and students along the way. All I ask for before my first visit is a syllabus. Since they go up for renewal for the second year sometime late in the fall semester, thanks to our nutty renewal and promotions schedule, I'll be writing up the letters over Thanksgiving Break so they can get them right after it and decide whether to include them in their files or not.
The point of these visits is to give useful feedback on their course design, pedagogy, classroom management, and so on. What I usually try to do while observing a class is figure out how the lesson is structured and why, how it relates to the overall course goals, and how the students are responding to it and to the professor. Obviously my being there changes the classroom dynamic to some extent, which is why only about a third of my overall focus is on their responses, especially during that first visit. But that doesn't mean I don't do a lot of student-watching. I like to come to a class early and watch/listen as the students come into the classroom, so I can see how they change as the professor enters the room. If there are any group or team discussions or activities, I like to place myself in a location where I can listen in on several teams (while appearing to be focused on taking notes). The great thing about students at my university is that from their words and body language it's very clear what they're thinking and how engaged they are with a lesson plan.
Still, especially for new faculty, that's not the be-all and end-all of an observation, particularly the first one. I've been here long enough (it's the start of my 10th year, if you count the year away) to have realized that it's how the students change over the four (or more) years they're here that matters most, so you have to think in semester-long arcs as well as shorter ones. If a particular class doesn't go as planned, you still have many more chances (and many many more than in Japan, where they meet only once a week for half the overall contact hours as in U.S. universities) to meet your goals for the semester. Of course, in my second visit, I'm looking to see whether/how the class atmosphere has changed, along with the quality of student engagement and discussion, particularly if the professor wasn't happy with how the first class went. But like I said, how the students respond to a new professor before any word-of-mouth has gotten around the student grapevines, which helps students self-select professors who match their own goals and learning styles, is not the biggest of deals to me.
What is is seeing how the professors are adjusting to the students in their class. Coming to a new institution, it's difficult to anticipate what student expectations and habits are with respect to reading load, in-class participation, individual and team assignments, taking responsibility for their own learning, and so on. In your first semester at a new place, you're basically gathering intel for the future--the next class, the next week, the next unit, the next semester--and looking for patterns in student thought and behavior. What can you expect from English majors? from English Adolescence Education majors? from Early Childhood Education English Concentrators? from students from the arts? humanities? social sciences? sciences? What about the mix of first- through fourth-years in your classes? What are their (often different and conflicting) expectations for this course? And so on. The goal is to figure out what the assumptions about them that you might be making that could be getting in the way of teaching them better, as well as what assumptions they seem to be making that they need to be disabused of or lead away from.
By adjusting to the students, then, I don't mean pandering to them, patronizing them, or catering to their every whim. I mean figuring out what are reasonable challenges to be presenting them with at what time in the semester, figuring out what you need to do to prepare them to do as well as possible on the assignments you've laid out for them, figuring out how to get and keep the group of students in front of you (or around you) motivated to push and challenge themselves. I mean thinking realistically and pragmatically about how to achieve your goals in the course. And maybe rethinking your goals and methods for the next time around.
Not all this shows up in my observation letter, of course. A good part of the letter is simply describing what I saw, in as much detail as I can muster from my notes and memories. Along the way or at the end, I usually include a mix of interpretations and assessments and suggestions. In doing this I tend to be more about options and roads not taken (but might be in the future) than about armchair quarterbacking or backseat driving; I consider alternate ways of achieving the apparent goals for the class meeting or adjusting them in light of the course goals. I try to convey to the professor and to anyone else who might happen to read the letter someday the complexities of and subtleties in teaching well.
Which means these things take a long time to write. But I usually learn a lot about teaching from visiting my colleagues' classes and writing up my observations, so I figure I should only return the favor with a letter they may be able to learn from.