Coping with issues of sloppiness and carelessness in writing

I appraise and am proud of my students in terms of their skills and capabilities of independently writing the whole drafts of their papers after going through my writing training. In particular, to train the writing of my students, I have iterated with them via marking through their writing and explaining to them why my suggested ways are better. Since my second year of my faculty job, my students have already written the whole drafts for the papers of which they are lead authors (with iterations of my marks and feedback). See some earlier blog posts on writing and my advice portal for relevant advice.

At the same time, I also notice some students still suffer from issues of sloppiness and carelessness in writing. For example, when I go through my pass of writing review on a draft submitted by a student to me, I could still point out many low-level writing issues (which should have been detected and fixed by the student) along with other issues.

I don't expect students to write perfect writing for the first pass; in fact, it has been suggested in the field of technical or general writing that, in the first pass of writing, writers need to focus on the ideas/contents rather than low-level writing issues.

But in later passes of draft reviewing, students need to have eyes equipped with special carefulness to scan through their own writing (including others' writing) to spot out various issues including low-level writing issues (such as those related to typos and grammar), high-level writing style issues (such as those related to bad logic flow, terms used-before-defined, inconsistent usage of the same term throughout the draft), and content issues (such as wrong descriptions of ideas or algorithms).

Below are my initial suggestions to students on how to cope with the issues of sloppiness and carelessness in writing but I welcome your suggestions.

*. You must do spell check on your writing. If you use MS Word, Word has good features of spell check and basic style check. If you use LaTeX, quite some editors support spell check: turn it on! Alternatively or additionally, you could generate a PDF file from your LaTeX files, and then copy and paste the text from the PDF file to an empty MS Word document, and use the spell check and basic style check there (not sure whether basic style check would work since extra line breaks are within the text).

*. For some of the common writing issues of yours that could be turned into rule checking, you should use rule-checking tools such as style-check for checking text in LaTeX.

*. When you go through your writing for reviewing, you should use a pen for marking through a hardcopy of your paper, rather than viewing it on computer screens or revising your draft along the way of reviewing it. Empirical evidence in technical writing or reviewing has shown that marking through hardcopies is more effective for identifying writing issues.

*. When you go through your writing, you should keep those common writing issues of your own in mind (I have an internal guideline for students to write writing-defect recording log, similar to defect recording log in Personal Software Process). For example, if you find that you have subject-verb inconsistency issues commonly in your writing (based on my past marks), you may consider to use a dedicated pass on your writing to simply look for only this issue (without looking for other issues). I know doing so would take time but before you have the capacity of spotting out various kinds of writing issues within a single pass, you would have to do so to identify your common writing issues.

*. You yourself should take time to go through your own writing with high care. Especially for your camera ready versions, please really take extreme care to inspect your own writing. For my pass of both idea and writing review of a camera ready version from a student, I would very likely take 1-3 hours for a single ACM/IEEE-format page. I believe that I may have "special eyes" to catch so many issues in a student's writing in my pass. With my "special eyes", I would still need to take a long time to go through a single page, partly because I spend significant time to ask "why so? is it true?" almost for each single sentence in terms of its content besides those writing issues. For example, if your paper includes a control flow graph drawn for a code snippet, I would redraw it myself on a piece of paper and compare my own graph with the graph included in the paper, without taking for granted that your graph is correct: you should do the same in your own pass of reviewing your draft.

*. You should ask one or more peer students to go through your draft for peer review for writing (of course, before that, you are expected to get peer reviews for your ideas and contents). If you find that, after only one student's peer review, your past drafts still got many marks from me, you should consider to ask more than one peer student to peer-review your future drafts.

*. (Suggestion contributed by Michal Young) Read aloud, preferably to someone who is willing to listen (even if they know nothing of the subject), but alone if necessary. Often our ears catch problems that our eyes silently correct before feeding our brains.

Happy writing!

More suggestions on addressing the issues are welcome and appreciated!

Tips on collecting research papers related to your research topic

*. Tip 1: Type in the keyword of a specific research topic (such as "symbolic execution") or the title of a research paper known to you in
http://scholar.google.com/
Then for relevant papers, you could see a list of papers related to the keyword, typically with those papers with more citations shown earlier.

Then you could browse through the papers citing a specific papers by clicking the "Cited by XXX" for the paper, e.g. "Cited by 922" for an early paper on symbolic execution by King. Then you may browse more recent papers first by clicking "Since 2011", "Since 2010", .... (from a pull down menu shown near the top of the result page) one at a time to browse from newer papers to older papers.

For a specific paper, you could click its "Cited by XXX" to open a new tab or window in your browser to see the list of papers citing the paper so that you don't lose the context of your earlier browsed result page.

*. Tip 2: Similar to Tip 1, but do the search with keywords at
http://academic.research.microsoft.com/
Then click "here" (from "This page shows one keyword best matching your query, you can find other results here.") on the line near the top of the result page to get more matched results. You could navigate to the papers citing a paper by clicking "(citations: XXX)" shown in the end of the paper's title in the result page.

*. Tip 3: Search or browse ACM digital library http://portal.acm.org/ or IEEE Explore library http://ieeexplore.ieee.org/Xplore/ to find papers and their references, as well as the papers citing them. Comparing with Google scholar (http://scholar.google.com/), ACM digital library could more easily allow you to navigate from a paper to papers from the list of references cited by this paper.

*. Tip 4: Type in the keyword of a specific research topic (such as "symbolic execution") or the title of a research paper known to you in
http://www.google.com/
If you find too many irrelevant results, you could add "filetypes:pdf" after your search keyword to return only the PDF files, which are typically the format of research papers. Note that for each research paper collected by Google Scholar as well, "Cited by XXX" is also shown for the entry of the paper in the normal Google result page to allow you to navigate to the papers citing the paper.

*. Tip 5: Browse online proceedings of recent major conference or journal contents in your research topic area from digital libraries (such as ACM digital library http://portal.acm.org/ or IEEE Explore library http://ieeexplore.ieee.org/Xplore/) to find out relevant papers. Or browse online technical programs or accepted papers of major conferences posted on their conference webs before the proceedings are available in digital libraries, and then google specific paper titles since some researchers may post their paper PDF files on their homepages (typically after camera ready deadlines).

If you attend a conference, sometimes the conference organizers may distribute a single PDF (in a USB stick, CD, or online web) including all the research papers in the proceedings. Then you could globally search a keyword such as "symbolic execution" in the single PDF file to find out all locations mentioning this keyword and thus find out relevant papers.

*. Tip 6: Browse the publications webpages of specific researchers or research groups working on your research topic area. You could use a free tool WebMon (http://www.markwell.btinternet.co.uk/webmon/) or watchthatpage (http://www.watchthatpage.com/) to add these webpage URLs there and run the tool frequently (e.g., weekly) to automatically check whether these pages have been updated, and browse only those publications webpages that have been updated since last time you ran the tool.

If you have more tips, please contribute!

Acknowledgment:
Thank Yingfei Xiong for suggesting the watchthatpage online tool.

Time management self checklist

A lot of students have problems in doing work efficiently. One root cause is on bad time management. The following talk video by Randy Pausch could be useful to watch:
http://www.youtube.com/watch?v=oTugjssqOT0

Here are the slides and materials for the talk:
http://www.cs.virginia.edu/~robins/Randy/RandyPauschTimeManagement2007.pdf
http://www.cs.virginia.edu/~robins/Randy/

If you complain about you don't have enough time in getting your work done, do a simple book-keeping on how you spend your time: handling emails, handling text messages, handling IM chats, browsing through or commenting/status-updating at facebook, twittering, watching TV, doing actual real work... Then sum up the portion of time allocated on each type of activities, you could find out some insights and figure out what do to to fix those issues...

Walking through the key points mentioned in Randy's talk, I make the following check list for reflecting what I have done well and what I have done badly. They may be boring to you (if so, stop reading them:) but writing the list down actually helps myself to reflect my time-management habits.

++ I don't have a messy desk (I used to have in my first year of new faculty life)
++ I can find things easily (heavily relying on google but knowing what appropriate search keywords to use, indexed in my mind)
+- I don't miss appointments unless sometimes they are in too early morning (don't arrange your defense in early morning unless you send a reminder to me the night before!).
++ I am normally prepared for my meetings (but the critical factor is my students need to be prepared when meeting with me).
-+ I sometimes am tired/unable to concentrate (but there I often drifted my thinking to some great research ideas)
++ I do planning (more precisely, I am deadline driven)
++ I have a todo list being my "Tasks" in my gmail but there I list only long-term tasks and I treat the emails in my inbox as my todo list)
- I need to do better on "Covey’s four-quadrant TODO": I tend to focus on things due soon (no matter important ones or not)
+- I do touch each piece of email once but I indeed consider my inbox as my TODO list (not seeing too much the negative side of it)
+ I don't call that much so the issues on reducing call duration don't apply to me.
++ I have a comfortable office (not messy and not with a soft comfortable chair)
-- I too much rely on my email inbox for my todo list and I don’t make time enough for important things.
+ I learn to say "No" reasonably well.
- I don't easily find out my creative/thinking time (maybe being late at night). But I don't tend to use such time to do creative thinking. I do creative thinking often when meeting with students; the meeting times may not overlap with my creative/thinking time but I do call students to my office for meetings whenever I like (of course when they are in the lab) rather than arranging fixed time.
- I don't easily find my dead time (maybe morning) so I don't do specific things during it such as scheduling meetings, phone calls, and mundane stuff.
-- I have big problems with interruptions with email "ding" arrivals; I still don't want to turn it off.:( So I rarely have too long blocks of time in devoting to things without email interruptions unless the deadlines for the things are immediately upcoming.
-- I don't tend to cut things short like when chatting with colleagues unplanned at hallway or my office. I do have a desk clock on my desk but the time there is not accurate and I rarely look at it.
-- I don't have a time journal so I cannot analyze it. But I do think about and focus on what things I could do but others couldn't easily (or are not good at). I do often think about how to do things more efficiently.
-- I don't have work-life balance (yet) -- more precisely not much life yet.
-- I am not too much on procrastination but I am a last-minute person (very likely because my todo list is often not short).
+ I am doing fine with delegation such as letting students manage group matters and take group roles, not to say training them how to write papers and carry out research (being maybe a type of delegation of paper writing or research development?). But I am a bit cautious on delegating some tasks to my students unless they get fair recognition/benefits that they deserve.
- I don't have too many meetings with colleagues but I do have frequent meetings with students (where students need to have an agenda beforehand and need to to have a todo list for upcoming period as meeting outcomes).
-- I read frequently my emails over "vacation". Need to stop that!
- I watch TV while working before my laptop for some time at nights. I still don't want to cut off my TV watching time, which is not much each week (I need some time off anyway).
- I don't normally turn money into time (after all, I don't have that much money while still having some spare time). Instead, I may pursue more on turning time into money.:)
+ I normally eat well, sleep well, and exercise ok (but I need to keep it up regularly.. now it time for me to go to gym since I am reaching the end of my blog entry)
+ I normally keep up my promise.

Don't rely on only examples to describe what you have in mind

During my one-on-one meetings with students from time to time, I found that quite some students don't have good skills in describing things clearly (either in spoken or written English). Here I would like to ask students to do a self-check: when you find out that your listener didn't understand what you said, you may tend to say "Let me use an example (to be drawn on the white board) to explain to you."

My advice to students is that don't immediately fall back to use an example to explain what you have in mind (especially during your meetings with your advisor during which you have golden opportunities to improve your skills). If you keep immediately falling back to use an example, you will never be able to explain yourself clearly with your direct description of things. What you need to do is to revisit what you just said and ask your listener on what he or she doesn't understand, and then you will be able to diagnose "bugs" in your initial descriptions, and try to avoid these "bugs" in your future descriptions.

For my weekly one-on-one meetings with students, I demand students to propose new ideas to me and recommend/describe other researchers' papers to me. In this process, students get opportunities to exercise their skills in verbally explaining clearly new things to the listener (i.e., me).

At the same time, students get more opportunities to exercise writing things more clearly by iterating their writing with me over their research development process. See my earlier post on technical writing. In technical writing, one similar pitfall to watch out and avoid is to explain your proposed approach via only examples without direct and clear description of what the approach is. Keep in mind that examples are just sample points in what your approach covers, and often the time readers don't get to know precisely what your approach is by reading only limited sample points covered by your approach.

Don't get me wrong. I in fact encourage students to use examples to help illustrate what they have in mind in their spoken or written English. But students shouldn't rely on ONLY examples to explain things, without being able to explain things directly and clearly up-front.

To help explain things clearly, students should consider to adopt the top-down way. More details can be found in my earlier post on "Advice to Students on Mastering Communication Skills"

In a one-on-one meeting with me, when a student explains an approach in a paper to me, it is better for the student to use the top-down way. The student needs to first explain the problem being addressed by the approach. To help accomplish this goal, besides the direct description of the problem, the student could explain the inputs to the approach and the outputs of the approach, without first getting into details how the approach does it. A common pitfall for students is that students tend to immediately explain what the approach actually does without first giving me any ideas on what problem the approach targets at or what the high level inputs/outputs of the approach are.

Advice to Students on Mastering Communication Skills

I found that some graduate students and undergraduate students (even native English speakers) often have difficulties in their communication with me in describing what they intend to convey. After some recent observations and thinking, I found that these students are lacking in their communication skills especially in terms of the logical organization of their thoughts. In particular, the issues lie in two parts:

(1) bottom-up issue: failing to describe things in a top-down way.

(2) context-less issue: failing to connect the background/context of the listener (me) with what they intend to convey.

Let me illustrate these issues via two concrete scenarios that I extracted from my real interactions with students.

*. A student tried to describe to me a problem encountered in his/her research project.

The student immediately dived into the details of a particular experimental subject in his/her evaluation and immediately got me lost. I had no ideas what the student was talking about for a while when the student tried hard to describe so many low-level details to me. I had to stop the student and asked "can you summarize the problem in one or two sentences that I can understand with the granularity level of my existing understanding of your project so far?" The student couldn't come up with such sentences. I further asked "why do I care about the problem in terms of the success of the research project?" The student still failed to answer this further question.

After some more non-trivial interactions with the student, I finally figured out that the problem that the student intended to convey can cause not-good values for a particular metric (among multiple metrics) for evaluating the proposed approach.

The bottom-up issue here is that the student failed to organize the conversation in a top-down way: I would expect the student to start the conversation with me like below:

"I would like to discuss with you on an encountered problem that can negatively affects the effectiveness of our approach, particularly in terms of the MMM metric. The problem is caused by XXXXXX (high-level description of the problem). For example, YYYYYY. ZZZZZZZ. ...."

Instead, earlier the student started the conversation on "YYYYY.. ZZZZZ, ...." or even "ZZZZZZ, ... YYYYY...". At that time, I had no clue or got lost on why the described phenomenon is a problem and why I would care (if I indeed care about the effectiveness of the approach).

The context-less issue here is that the student failed to connect the background and context that I have on the project (i.e., I know what the high-level idea of the approach is and what the evaluation criteria for evaluating the approach are) with what the students tried to convey (i.e., the problem). To make sure that I have the expected knowledge or context to understand the student's problem, besides the expected top-down way described earlier, the student can better start the conversation like "Let me first remind you that the effectiveness of our approach is measured based on n metrics, among which the MMM metric ..... However, I encountered a problem that can cause bad values for the MMM metric. The problem is due to XXXXXX..."

* A student tried to answer my question in a qualifier exam or thesis defense.

Let's assume that the question is "what metrics do you use in evaluating your approach?" The student would start the answer with "ZZZZZZZ, .... YYYYY....". For quite some seconds and minutes, I had no clue or got lost on why what the student spoke had anything to do with my question. Or even worst, in some situations, after the student finished minutes of talking, I still had no clues on what the answer is (for those questions expecting a "Yes" or "No" answer, I still couldn't figure out whether the student intended to say "Yes" or "No"!).

The bottom-up issue here is that the student failed to organize the conversation in a top-down way: I would expect the student to start the answer to my question like "I used two metrics. The first metric is X/Y where X is ... and Y is ... In particular, I measure X with faults seeded with mutation testing, .... The second metric is... " Instead, earlier the student started the talking w.r.t. my question with "Mutation testing is a commonly used way for measuring fault detection capability, ... YYYYY.. ZZZZZ, ....". For a while since the student's talking started, I had no clue or got lost on why mutation testing has anything to do with the metrics. Even when there are occassions where some background information needs to be laid out before giving out the answer, the student still should give strong signals as starting sentences for making clear how what the students will say first is related to the answer. Some examples can be "Before I describe the metrics that I use, I would like to first describe mutation testing, which is related to three out of four metrics that I use, and then after that I will list these four metrics....."

For those questions expecting a "Yes" or "No" answer, the student can start with "My answer is Yes, with three reasons. First, ...." or "My answer is both Yes and No. The reason for me to say Yes is .... The reason for me to say No is ..."

The context-less issue here is that the student failed to connect the background and context that I have (i.e., my question) with what the students tried to convey (i.e., what the student talked about w.r.t. the answer).

* Final remark

I would suggest students to realize these communication anti-patterns and address them. Students who have these anti-patterns in oral communications often have problems in their writing (especially in terms of logical thinking and writing) such as not laying out enough background or assumptions for readers to understand what the students write next, or not clearly and concisely organize the content to make the writing easy to understand.

The Minto Pyramid Principles (e.g., top-down writing style) are good ones to follow as starting points for students. Below are some slides on some key ideas there:
http://www.dbai.tuwien.ac.at/staff/gatter/work/051104_The_Minto_Pyramid_Principle.pdf
More are described in the Minto text book:
http://www.barbaraminto.com/textbook.html

Update: my colleague Macneil Shonle recommends the following book: The Sense of Structure: Writing from the Reader's Perspective, which may worth checking out. Thank Macneil for the recommendation!

This post more or less follows the top-down writing style, if you realize.:)

Graduate Student Survival/Success Guide

Yesterday, I gave a CSC 600 lecture to new graduate students at NCSU CSC. The slides can be found here.

How not to keep your advisor up all night till last minute before the paper submission deadline?

Posted on Friday Nov 07, 2008

I recently had been through a case where a student submitted the student's draft for my review of writing two days before a submission deadline. Since I had another more urgent task during the period, the required effort for helping improve the draft to a fine shape "forced" me to stay up the whole night till early morning 5am the submission deadline.

When I tried to recall several past submissions of this student, I found that all these submissions kept me up all night in the morning till last minute before the deadline whereas many other students' drafts were often already ready to submit the night preceding the submission deadline.

I asked myself "why would this situation happen for the student?" For this immediately past submission, in fact the student did a very good job in preparing the abstract, introduction, and approach (high level description) sections early on (for my review of ideas). But the problem is that the student submitted the draft for my review of writing late, only two days before the submission deadline. Several possible reasons may contribute to this late submission: the student might think that the student needed to finish all the sections of writing before asking peer review for writing, or the peer reviewing student needed to finish peer review of all sections of the draft before asking for my review of writing, ...

To avoid these issues (or avoid keeping me up all night till last minute before the paper submission deadline), below is a new set of advice to my students:

Please write early, ask peer review early, and ask for my review of writing early!! Here are some specific things you should do:

(1). You don't need to wait till you have a complete draft (with all sections finished) before requesting peer review. You can submit your partial draft (e.g., one or more completed sections at a time) for peer review.

(2). You don't need to wait till your peer-reviewing student finishes reviewing all the sections of your draft before requesting my review of writing. You can submit your peer-reviewed partial draft (e.g., one or more peer-reviewed sections at a time) for my review of writing. Remind you that I have a policy of being able to review your draft for writing ONLY after a peer student has finished reviewing the writing of your draft and you have fixed the issues pointed out by the peer student. But you don't need to get your peer review done before I can review your draft for the ideas described in the draft to avoid the delay on giving you feedback on the ideas in your draft.

(3). Budget your timeline to allow to ask me to review your writing for multiple iterations (instead of just one time or even 0 time iteration on any portion your writing before the submission).

Of course, it is most important for you to write early following my advice posted earlier!