Ramblings on Study Methods

So I’m trying to get together a study to study workplace revision practices, and I’m having some problems. My real goal is to study the research methods used to examine workplace revision practices: I’ve read a number of very well-done ethnographic studies, but I keep thinking there must be a faster, cheaper way to do workplace studies than ethnography. I also think there must be a research method that requires less training than ethnography.

But revision itself is a bugbear. Reading through the scholarly attempts (look at Haar, chapter 2 of Horning’s collection, to get an idea) at defining revision is like learning a whole new field. Loads of stuff. But what I noticed in particular, and this is significant, is that these definitions don’t seem to really contradict each other. Unlike other sets of scholarly definitions—like those for technology, or rhetoric—the definitions for revision don’t seem to really be at odds. Well, there’s some disagreement, like whether revision is simply writing or a separate process. But overall, it’s simply a struggle to find meaning: should we talk about revision in terms of the physical process (essentially proofreading) or revision as it falls on the schedule of things to do (it comes after a rough draft but before the final draft)? So really there’s less confusion in learning about revision than in studying other humanities-related subjects.

But to get back to research methods, I’ve been thinking that social media (think Facebook and Twitter) should offer some way of contacting the workplace and gathering data about the revision process within engineering firms (I work with engineers, so contacting engineering firms is the path of least resistance). But I think that Facebook and Twitter offer ways around some of the traditional problems associated with distributing a research instrument, such as a survey: social media allow for few barriers when it comes to distribution, and many connections to boot. For instance, if I want to survey Texas politicians about legal matters related to property ownership, the first few folks I contact may not know anything about it. But they’ll know someone—some real estate-lawyer-turned-politician—who can really comment on the matter. And she’ll, by nature of her job, know someone else, and social media allows for quick distribution: person A knows person B is an expert, and passes the survey along, and person B knows person C, and so on. And there are a number of engineering groups on Facebook, so that would work as a distribution method. I don’t have to spend time tracking down contact info for all those folks; instead, I can simply join a single Facebook group and initiate contact.

But that really only speaks to the distribution side of research, the access to a pool of participants. While that’s certainly interesting, and likely to be a timesaver, it doesn’t have to do with the other side of the study. If research is hunting and gathering, then Facebook and Twitter only help with the hunting. There’s no gathering.

But an article I read the other day—Jones?—pointed out that some folks think applications like Google Docs constitute social networking, or at least social media. And that’s an interesting thought, isn’t it? Wouldn’t it be interesting to set up a number of experiments using Google Docs?

For instance here’s one:
Two groups of five students post their rough drafts to Google Docs. Each group has two professional engineers assigned to it, but one group knows it and the other doesn’t. So Group A knows they’ll be getting some feedback from a professional engineer and Group B only knows they’ll be getting feedback from unidentified parties, other students for all they know.

If everything I read about the valuation of rhetoric as a part of becoming a member of a discourse community proves true, then both groups should pretty quickly identify the professional feedback regardless of whether they know for sure the other participants are professional engineers. Something about the way the pros offer their feedback should let the students know, clue them in, that they’re professionals.

Another experiment would be to simply watch, and record somehow, the revision process of two groups of professional engineers working on technical reports and using Google Docs (or something comparable onsite, such as a local wiki). Simultaneous group also using Google Docs (or some other software) but with students as participants. Then compare the results. Of course, I’d have to be looking for a construct of interest, some textual feature, such that if it appears within one group and not the other I’d have something to compare.

So what would that rhetorical feature be? It’s tempting to say that what’ll happen is that the novice writers—the students—would begin almost immediately to recognize the professional, more experienced discourse of the professional engineers due to word choice or some other feature. How they recognize that professional discourse isn’t the point here. What they do in reaction to it is, at least this time through. So my guess is that the sign they recognize the professional discourse will involve mimicking the style of the discourse, which means in order to judge whether the novice writers have recognized and begun to use a new style I’ll need to have some raters. Having raters opens a whole new can of worms, but it can be done. Perhaps in virtual teams.

Another option would be to interview the students after the exercise and see how they characterize what happened, and whether they recognize the shift in their own style. That shouldn’t be too hard. Make sure the questions are along the lines of “Do you think your writing improved?” or even simply “Do you think it changed?” or “How would you characterize the person providing feedback, and what was your reaction to that feedback?” I think any of those questions would work fairly well, particularly the last one. In fact, they might be delivered via survey.
So that last study would test a number of things, wouldn’t it? In short, they’d be:

Do novice writers recognize the discourse of their chosen profession simply by seeing it?

Do novice writers react differently to the professional engineers’ writing without knowing it’s a professional engineer supplying it?
Is collaborative writing technology a tool by which these changes can be recorded and observed? If so, can we get enough observational data to justify replacing an ethnography with observation of collaborative writing technology?