Technography as Method

Is it possible to use digital methods to replicate the results of ethnographic studies on workplace writing review?

The reason this question is important is because there is loads—and I mean loads—of evidence that novice writers only become advanced writers when they begin to associate themselves with the discourse, and thus the communities formed by that discourse, used by their chosen discipline. Thus, a newly-graduated engineer will start to incorporate characteristics of the professional discourse he or she sees and experiences in the engineering workplace. Technical communication and composition researchers assure me this is true. It’s a fact proven by empirical research. I’ve read that research, and it sure seems solid.

But it begs a couple of questions:
1. Why bother with writing courses in colleges if we’re not able to reproduce the circumstances of a professional writing environment (whether that’s for the law, accounting, marketing, engineering, etc.)?
2. If we can replicate those circumstances, and thus begin graduating students who are already professional writers in terms of their chosen field, how can we keep those circumstances both (a) updated, and (b) specific enough to help each student?
3. Surveys of employers consistently indicate that communication skills are at the top, or near the top, of the list of things those employers look for in a new employee. But if those communication skills are only learned once the novice communicator (writer) begins working within the professional workplace, how can our students hope to get jobs since, as students, they will always be novice writers?

I think working on #2 is our best hope, and, like Princess Leia recording the secret message in R2D2’s innards, we can only beseech the Obi-Kenobi’s of the professional world to help us out. So how do we go about replicating the circumstances of a professional writing environment? As always, that’s my central question.

Working with my genius wife, I’ve sketched out what I think is a preliminary plan for testing whether workplace revision processes, what Katz called “writing review,” can be duplicated within the classroom so long as we have some online contact with professionals. As I’ve lamented before, the way this has been researched in the past is to use a sort of pseudo-ethnography: the researcher gets permission to follow people around and watch what they do. But that’s time consuming. The researcher needs special training, the researcher has to follow folks around for long enough to get a good idea of what they’re doing, and then the researcher winds up with reams of field data that has to somehow be coded.
There has got to be a way around that. I don’t know if it’s intellectual curiosity or an intense case of laziness, but I feel as though we have enough digital tools laying around such that we can learn what we need without leaving the office. But what I need to know first is whether folks in the workplace are using digital tools for writing review, like Word’s track changes and commenting features, or Google Docs or even wikis. If they are, then we’ve got a start, because those are tools we can use for classroom assignments, and they are also the petard through which can hang ethnography.

Here’s the study, based on the premise that I can find three or four engineering firms that use Word’s track changes/commenting feature in their writing review process.

Two groups of students write a document of a professional genre, such as a Statement of Qualifications (SOQ).

That document is reviewed by two groups of professional engineers using MS Word’s track changes/commenting feature. One group of students knows that the reviewers are professional engineers; the other group of students does not.

The students see the reviewed document and are allowed to incorporate or not incorporate the suggested revisions. In interviews, the students asked to justify why and how they used the suggested revisions.

After the students make use (or don’t) of the revisions, the first draft, suggested revisions, and final draft is then reviewed by another group of professional engineers as well as by a group of writing instructors. The two groups (engineers and writing instructors) are asked to evaluate whether the students made effective choices when including or not including the professional feedback.

If the findings are positive: in other words, if the final evaluators indicate that yes, the students did make good use of the professional feedback, then we can argue that gathering feedback in this way, rather than ethnography, is a viable method of beginning the professional rhetorical assimilation of the students. We can also begin to encourage instructors to begin incorporating this method of feedback into their classes. Many professional schools, such as engineering and law, have close ties to the professional community; thus, finding participants to provide occasional feedback shouldn’t be impossible. We could also theorize that digital research methods are a possible alternative to direct data collection via ethnography, thus opening the door to more effective writing instruction within professional disciplines.

If the findings are negative, in other words, if the final evaluators indicate that no, the students not make good use of the professional feedback, then we can argue that perhaps ethnographic data collection is still the way to go when researching workplace writing review practices for inclusion in the classroom. We can also argue that writing revision is more than the written form of feedback: that there is socialization that takes place (as I believe Katz and others have argued) that cannot be recorded in the written word. We could also argue that, contrary to widespread belief, the “digital generation” is still pretty heavily reliant on face-to-face communication, or at least communication that’s not written.

One of the reasons, perhaps THE reason, this method holds so much fascination for me (other than the rampant laziness cited above) is because digital tools offer so many possibilities for recording human activity, and therefore also shaping human activity. It’s clear that we’ve changed some of our habits and work styles because of the Internet, and I think that it’s possible to begin re-shaping how we go about doing writing research, too. While loads of us certainly have done plenty of digital writing research, a great deal of that (think Kairos) deals with exploratory pedagogies and rather academic discussions of new media. I’m more interested in how folks use digital tools to get writing done in the cubicle, not in the classroom.

My wife proposed the word “technography” to mean a technological replacement for ethnography. I’ve just found a book in the A&M library that suggests the same, so that may be my hook: technography rather than ethnography. For me, technography suggests a digital observer rather than an in-person observer, and since so much of our work is digital, testing technography as a way of gathering data on workplace writing review seems like an obvious choice.

Ramblings on Study Methods

So I’m trying to get together a study to study workplace revision practices, and I’m having some problems. My real goal is to study the research methods used to examine workplace revision practices: I’ve read a number of very well-done ethnographic studies, but I keep thinking there must be a faster, cheaper way to do workplace studies than ethnography. I also think there must be a research method that requires less training than ethnography.

But revision itself is a bugbear. Reading through the scholarly attempts (look at Haar, chapter 2 of Horning’s collection, to get an idea) at defining revision is like learning a whole new field. Loads of stuff. But what I noticed in particular, and this is significant, is that these definitions don’t seem to really contradict each other. Unlike other sets of scholarly definitions—like those for technology, or rhetoric—the definitions for revision don’t seem to really be at odds. Well, there’s some disagreement, like whether revision is simply writing or a separate process. But overall, it’s simply a struggle to find meaning: should we talk about revision in terms of the physical process (essentially proofreading) or revision as it falls on the schedule of things to do (it comes after a rough draft but before the final draft)? So really there’s less confusion in learning about revision than in studying other humanities-related subjects.

But to get back to research methods, I’ve been thinking that social media (think Facebook and Twitter) should offer some way of contacting the workplace and gathering data about the revision process within engineering firms (I work with engineers, so contacting engineering firms is the path of least resistance). But I think that Facebook and Twitter offer ways around some of the traditional problems associated with distributing a research instrument, such as a survey: social media allow for few barriers when it comes to distribution, and many connections to boot. For instance, if I want to survey Texas politicians about legal matters related to property ownership, the first few folks I contact may not know anything about it. But they’ll know someone—some real estate-lawyer-turned-politician—who can really comment on the matter. And she’ll, by nature of her job, know someone else, and social media allows for quick distribution: person A knows person B is an expert, and passes the survey along, and person B knows person C, and so on. And there are a number of engineering groups on Facebook, so that would work as a distribution method. I don’t have to spend time tracking down contact info for all those folks; instead, I can simply join a single Facebook group and initiate contact.

But that really only speaks to the distribution side of research, the access to a pool of participants. While that’s certainly interesting, and likely to be a timesaver, it doesn’t have to do with the other side of the study. If research is hunting and gathering, then Facebook and Twitter only help with the hunting. There’s no gathering.

But an article I read the other day—Jones?—pointed out that some folks think applications like Google Docs constitute social networking, or at least social media. And that’s an interesting thought, isn’t it? Wouldn’t it be interesting to set up a number of experiments using Google Docs?

For instance here’s one:
Two groups of five students post their rough drafts to Google Docs. Each group has two professional engineers assigned to it, but one group knows it and the other doesn’t. So Group A knows they’ll be getting some feedback from a professional engineer and Group B only knows they’ll be getting feedback from unidentified parties, other students for all they know.

If everything I read about the valuation of rhetoric as a part of becoming a member of a discourse community proves true, then both groups should pretty quickly identify the professional feedback regardless of whether they know for sure the other participants are professional engineers. Something about the way the pros offer their feedback should let the students know, clue them in, that they’re professionals.

Another experiment would be to simply watch, and record somehow, the revision process of two groups of professional engineers working on technical reports and using Google Docs (or something comparable onsite, such as a local wiki). Simultaneous group also using Google Docs (or some other software) but with students as participants. Then compare the results. Of course, I’d have to be looking for a construct of interest, some textual feature, such that if it appears within one group and not the other I’d have something to compare.

So what would that rhetorical feature be? It’s tempting to say that what’ll happen is that the novice writers—the students—would begin almost immediately to recognize the professional, more experienced discourse of the professional engineers due to word choice or some other feature. How they recognize that professional discourse isn’t the point here. What they do in reaction to it is, at least this time through. So my guess is that the sign they recognize the professional discourse will involve mimicking the style of the discourse, which means in order to judge whether the novice writers have recognized and begun to use a new style I’ll need to have some raters. Having raters opens a whole new can of worms, but it can be done. Perhaps in virtual teams.

Another option would be to interview the students after the exercise and see how they characterize what happened, and whether they recognize the shift in their own style. That shouldn’t be too hard. Make sure the questions are along the lines of “Do you think your writing improved?” or even simply “Do you think it changed?” or “How would you characterize the person providing feedback, and what was your reaction to that feedback?” I think any of those questions would work fairly well, particularly the last one. In fact, they might be delivered via survey.
So that last study would test a number of things, wouldn’t it? In short, they’d be:

Do novice writers recognize the discourse of their chosen profession simply by seeing it?

Do novice writers react differently to the professional engineers’ writing without knowing it’s a professional engineer supplying it?
Is collaborative writing technology a tool by which these changes can be recorded and observed? If so, can we get enough observational data to justify replacing an ethnography with observation of collaborative writing technology?