Tao Xie's Research and Advising Blog

Coping with issues of sloppiness and carelessness in writing

2011-06-12T13:38:00.000-07:00

I appraise and am proud of my students in terms of their skills and capabilities of independently writing the whole drafts of their papers after going through my writing training. In particular, to train the writing of my students, I have iterated with them via marking through their writing and explaining to them why my suggested ways are better. Since my second year of my faculty job, my students have already written the whole drafts for the papers of which they are lead authors (with iterations of my marks and feedback). See some earlier blog posts on writing and my advice portal for relevant advice.

At the same time, I also notice some students still suffer from issues of sloppiness and carelessness in writing. For example, when I go through my pass of writing review on a draft submitted by a student to me, I could still point out many low-level writing issues (which should have been detected and fixed by the student) along with other issues.

I don't expect students to write perfect writing for the first pass; in fact, it has been suggested in the field of technical or general writing that, in the first pass of writing, writers need to focus on the ideas/contents rather than low-level writing issues.

But in later passes of draft reviewing, students need to have eyes equipped with special carefulness to scan through their own writing (including others' writing) to spot out various issues including low-level writing issues (such as those related to typos and grammar), high-level writing style issues (such as those related to bad logic flow, terms used-before-defined, inconsistent usage of the same term throughout the draft), and content issues (such as wrong descriptions of ideas or algorithms).

Below are my initial suggestions to students on how to cope with the issues of sloppiness and carelessness in writing but I welcome your suggestions.

*. You must do spell check on your writing. If you use MS Word, Word has good features of spell check and basic style check. If you use LaTeX, quite some editors support spell check: turn it on! Alternatively or additionally, you could generate a PDF file from your LaTeX files, and then copy and paste the text from the PDF file to an empty MS Word document, and use the spell check and basic style check there (not sure whether basic style check would work since extra line breaks are within the text).

*. For some of the common writing issues of yours that could be turned into rule checking, you should use rule-checking tools such as style-check for checking text in LaTeX.

*. When you go through your writing for reviewing, you should use a pen for marking through a hardcopy of your paper, rather than viewing it on computer screens or revising your draft along the way of reviewing it. Empirical evidence in technical writing or reviewing has shown that marking through hardcopies is more effective for identifying writing issues.

*. When you go through your writing, you should keep those common writing issues of your own in mind (I have an internal guideline for students to write writing-defect recording log, similar to defect recording log in Personal Software Process). For example, if you find that you have subject-verb inconsistency issues commonly in your writing (based on my past marks), you may consider to use a dedicated pass on your writing to simply look for only this issue (without looking for other issues). I know doing so would take time but before you have the capacity of spotting out various kinds of writing issues within a single pass, you would have to do so to identify your common writing issues.

*. You yourself should take time to go through your own writing with high care. Especially for your camera ready versions, please really take extreme care to inspect your own writing. For my pass of both idea and writing review of a camera ready version from a student, I would very likely take 1-3 hours for a single ACM/IEEE-format page. I believe that I may have "special eyes" to catch so many issues in a student's writing in my pass. With my "special eyes", I would still need to take a long time to go through a single page, partly because I spend significant time to ask "why so? is it true?" almost for each single sentence in terms of its content besides those writing issues. For example, if your paper includes a control flow graph drawn for a code snippet, I would redraw it myself on a piece of paper and compare my own graph with the graph included in the paper, without taking for granted that your graph is correct: you should do the same in your own pass of reviewing your draft.

*. You should ask one or more peer students to go through your draft for peer review for writing (of course, before that, you are expected to get peer reviews for your ideas and contents). If you find that, after only one student's peer review, your past drafts still got many marks from me, you should consider to ask more than one peer student to peer-review your future drafts.

*. (Suggestion contributed by Michal Young) Read aloud, preferably to someone who is willing to listen (even if they know nothing of the subject), but alone if necessary. Often our ears catch problems that our eyes silently correct before feeding our brains.

Happy writing!

More suggestions on addressing the issues are welcome and appreciated!

Tips on collecting research papers related to your research topic

2011-06-08T16:19:00.000-07:00

*. Tip 1: Type in the keyword of a specific research topic (such as "symbolic execution") or the title of a research paper known to you in
http://scholar.google.com/
Then for relevant papers, you could see a list of papers related to the keyword, typically with those papers with more citations shown earlier.

Then you could browse through the papers citing a specific papers by clicking the "Cited by XXX" for the paper, e.g. "Cited by 922" for an early paper on symbolic execution by King. Then you may browse more recent papers first by clicking "Since 2011", "Since 2010", .... (from a pull down menu shown near the top of the result page) one at a time to browse from newer papers to older papers.

For a specific paper, you could click its "Cited by XXX" to open a new tab or window in your browser to see the list of papers citing the paper so that you don't lose the context of your earlier browsed result page.

*. Tip 2: Similar to Tip 1, but do the search with keywords at
http://academic.research.microsoft.com/
Then click "here" (from "This page shows one keyword best matching your query, you can find other results here.") on the line near the top of the result page to get more matched results. You could navigate to the papers citing a paper by clicking "(citations: XXX)" shown in the end of the paper's title in the result page.

*. Tip 3: Search or browse ACM digital library http://portal.acm.org/ or IEEE Explore library http://ieeexplore.ieee.org/Xplore/ to find papers and their references, as well as the papers citing them. Comparing with Google scholar (http://scholar.google.com/), ACM digital library could more easily allow you to navigate from a paper to papers from the list of references cited by this paper.

*. Tip 4: Type in the keyword of a specific research topic (such as "symbolic execution") or the title of a research paper known to you in
http://www.google.com/
If you find too many irrelevant results, you could add "filetypes:pdf" after your search keyword to return only the PDF files, which are typically the format of research papers. Note that for each research paper collected by Google Scholar as well, "Cited by XXX" is also shown for the entry of the paper in the normal Google result page to allow you to navigate to the papers citing the paper.

*. Tip 5: Browse online proceedings of recent major conference or journal contents in your research topic area from digital libraries (such as ACM digital library http://portal.acm.org/ or IEEE Explore library http://ieeexplore.ieee.org/Xplore/) to find out relevant papers. Or browse online technical programs or accepted papers of major conferences posted on their conference webs before the proceedings are available in digital libraries, and then google specific paper titles since some researchers may post their paper PDF files on their homepages (typically after camera ready deadlines).

If you attend a conference, sometimes the conference organizers may distribute a single PDF (in a USB stick, CD, or online web) including all the research papers in the proceedings. Then you could globally search a keyword such as "symbolic execution" in the single PDF file to find out all locations mentioning this keyword and thus find out relevant papers.

*. Tip 6: Browse the publications webpages of specific researchers or research groups working on your research topic area. You could use a free tool WebMon (http://www.markwell.btinternet.co.uk/webmon/) or watchthatpage (http://www.watchthatpage.com/) to add these webpage URLs there and run the tool frequently (e.g., weekly) to automatically check whether these pages have been updated, and browse only those publications webpages that have been updated since last time you ran the tool.

If you have more tips, please contribute!

Acknowledgment:
Thank Yingfei Xiong for suggesting the watchthatpage online tool.

Time management self checklist

2010-11-03T16:19:00.000-07:00

A lot of students have problems in doing work efficiently. One root cause is on bad time management. The following talk video by Randy Pausch could be useful to watch:
http://www.youtube.com/watch?v=oTugjssqOT0

Here are the slides and materials for the talk:
http://www.cs.virginia.edu/~robins/Randy/RandyPauschTimeManagement2007.pdf
http://www.cs.virginia.edu/~robins/Randy/

If you complain about you don't have enough time in getting your work done, do a simple book-keeping on how you spend your time: handling emails, handling text messages, handling IM chats, browsing through or commenting/status-updating at facebook, twittering, watching TV, doing actual real work... Then sum up the portion of time allocated on each type of activities, you could find out some insights and figure out what do to to fix those issues...

Walking through the key points mentioned in Randy's talk, I make the following check list for reflecting what I have done well and what I have done badly. They may be boring to you (if so, stop reading them:) but writing the list down actually helps myself to reflect my time-management habits.

++ I don't have a messy desk (I used to have in my first year of new faculty life)
++ I can find things easily (heavily relying on google but knowing what appropriate search keywords to use, indexed in my mind)
+- I don't miss appointments unless sometimes they are in too early morning (don't arrange your defense in early morning unless you send a reminder to me the night before!).
++ I am normally prepared for my meetings (but the critical factor is my students need to be prepared when meeting with me).
-+ I sometimes am tired/unable to concentrate (but there I often drifted my thinking to some great research ideas)
++ I do planning (more precisely, I am deadline driven)
++ I have a todo list being my "Tasks" in my gmail but there I list only long-term tasks and I treat the emails in my inbox as my todo list)
- I need to do better on "Covey’s four-quadrant TODO": I tend to focus on things due soon (no matter important ones or not)
+- I do touch each piece of email once but I indeed consider my inbox as my TODO list (not seeing too much the negative side of it)
+ I don't call that much so the issues on reducing call duration don't apply to me.
++ I have a comfortable office (not messy and not with a soft comfortable chair)
-- I too much rely on my email inbox for my todo list and I don’t make time enough for important things.
+ I learn to say "No" reasonably well.
- I don't easily find out my creative/thinking time (maybe being late at night). But I don't tend to use such time to do creative thinking. I do creative thinking often when meeting with students; the meeting times may not overlap with my creative/thinking time but I do call students to my office for meetings whenever I like (of course when they are in the lab) rather than arranging fixed time.
- I don't easily find my dead time (maybe morning) so I don't do specific things during it such as scheduling meetings, phone calls, and mundane stuff.
-- I have big problems with interruptions with email "ding" arrivals; I still don't want to turn it off.:( So I rarely have too long blocks of time in devoting to things without email interruptions unless the deadlines for the things are immediately upcoming.
-- I don't tend to cut things short like when chatting with colleagues unplanned at hallway or my office. I do have a desk clock on my desk but the time there is not accurate and I rarely look at it.
-- I don't have a time journal so I cannot analyze it. But I do think about and focus on what things I could do but others couldn't easily (or are not good at). I do often think about how to do things more efficiently.
-- I don't have work-life balance (yet) -- more precisely not much life yet.
-- I am not too much on procrastination but I am a last-minute person (very likely because my todo list is often not short).
+ I am doing fine with delegation such as letting students manage group matters and take group roles, not to say training them how to write papers and carry out research (being maybe a type of delegation of paper writing or research development?). But I am a bit cautious on delegating some tasks to my students unless they get fair recognition/benefits that they deserve.
- I don't have too many meetings with colleagues but I do have frequent meetings with students (where students need to have an agenda beforehand and need to to have a todo list for upcoming period as meeting outcomes).
-- I read frequently my emails over "vacation". Need to stop that!
- I watch TV while working before my laptop for some time at nights. I still don't want to cut off my TV watching time, which is not much each week (I need some time off anyway).
- I don't normally turn money into time (after all, I don't have that much money while still having some spare time). Instead, I may pursue more on turning time into money.:)
+ I normally eat well, sleep well, and exercise ok (but I need to keep it up regularly.. now it time for me to go to gym since I am reaching the end of my blog entry)
+ I normally keep up my promise.

Don't rely on only examples to describe what you have in mind

2010-11-02T20:50:00.000-07:00

During my one-on-one meetings with students from time to time, I found that quite some students don't have good skills in describing things clearly (either in spoken or written English). Here I would like to ask students to do a self-check: when you find out that your listener didn't understand what you said, you may tend to say "Let me use an example (to be drawn on the white board) to explain to you."

My advice to students is that don't immediately fall back to use an example to explain what you have in mind (especially during your meetings with your advisor during which you have golden opportunities to improve your skills). If you keep immediately falling back to use an example, you will never be able to explain yourself clearly with your direct description of things. What you need to do is to revisit what you just said and ask your listener on what he or she doesn't understand, and then you will be able to diagnose "bugs" in your initial descriptions, and try to avoid these "bugs" in your future descriptions.

For my weekly one-on-one meetings with students, I demand students to propose new ideas to me and recommend/describe other researchers' papers to me. In this process, students get opportunities to exercise their skills in verbally explaining clearly new things to the listener (i.e., me).

At the same time, students get more opportunities to exercise writing things more clearly by iterating their writing with me over their research development process. See my earlier post on technical writing. In technical writing, one similar pitfall to watch out and avoid is to explain your proposed approach via only examples without direct and clear description of what the approach is. Keep in mind that examples are just sample points in what your approach covers, and often the time readers don't get to know precisely what your approach is by reading only limited sample points covered by your approach.

Don't get me wrong. I in fact encourage students to use examples to help illustrate what they have in mind in their spoken or written English. But students shouldn't rely on ONLY examples to explain things, without being able to explain things directly and clearly up-front.

To help explain things clearly, students should consider to adopt the top-down way. More details can be found in my earlier post on "Advice to Students on Mastering Communication Skills"

In a one-on-one meeting with me, when a student explains an approach in a paper to me, it is better for the student to use the top-down way. The student needs to first explain the problem being addressed by the approach. To help accomplish this goal, besides the direct description of the problem, the student could explain the inputs to the approach and the outputs of the approach, without first getting into details how the approach does it. A common pitfall for students is that students tend to immediately explain what the approach actually does without first giving me any ideas on what problem the approach targets at or what the high level inputs/outputs of the approach are.

Advice to Students on Mastering Communication Skills

2009-11-11T15:01:00.000-08:00

I found that some graduate students and undergraduate students (even native English speakers) often have difficulties in their communication with me in describing what they intend to convey. After some recent observations and thinking, I found that these students are lacking in their communication skills especially in terms of the logical organization of their thoughts. In particular, the issues lie in two parts:

(1) bottom-up issue: failing to describe things in a top-down way.

(2) context-less issue: failing to connect the background/context of the listener (me) with what they intend to convey.

Let me illustrate these issues via two concrete scenarios that I extracted from my real interactions with students.

*. A student tried to describe to me a problem encountered in his/her research project.

The student immediately dived into the details of a particular experimental subject in his/her evaluation and immediately got me lost. I had no ideas what the student was talking about for a while when the student tried hard to describe so many low-level details to me. I had to stop the student and asked "can you summarize the problem in one or two sentences that I can understand with the granularity level of my existing understanding of your project so far?" The student couldn't come up with such sentences. I further asked "why do I care about the problem in terms of the success of the research project?" The student still failed to answer this further question.

After some more non-trivial interactions with the student, I finally figured out that the problem that the student intended to convey can cause not-good values for a particular metric (among multiple metrics) for evaluating the proposed approach.

The bottom-up issue here is that the student failed to organize the conversation in a top-down way: I would expect the student to start the conversation with me like below:

"I would like to discuss with you on an encountered problem that can negatively affects the effectiveness of our approach, particularly in terms of the MMM metric. The problem is caused by XXXXXX (high-level description of the problem). For example, YYYYYY. ZZZZZZZ. ...."

Instead, earlier the student started the conversation on "YYYYY.. ZZZZZ, ...." or even "ZZZZZZ, ... YYYYY...". At that time, I had no clue or got lost on why the described phenomenon is a problem and why I would care (if I indeed care about the effectiveness of the approach).

The context-less issue here is that the student failed to connect the background and context that I have on the project (i.e., I know what the high-level idea of the approach is and what the evaluation criteria for evaluating the approach are) with what the students tried to convey (i.e., the problem). To make sure that I have the expected knowledge or context to understand the student's problem, besides the expected top-down way described earlier, the student can better start the conversation like "Let me first remind you that the effectiveness of our approach is measured based on n metrics, among which the MMM metric ..... However, I encountered a problem that can cause bad values for the MMM metric. The problem is due to XXXXXX..."

* A student tried to answer my question in a qualifier exam or thesis defense.

Let's assume that the question is "what metrics do you use in evaluating your approach?" The student would start the answer with "ZZZZZZZ, .... YYYYY....". For quite some seconds and minutes, I had no clue or got lost on why what the student spoke had anything to do with my question. Or even worst, in some situations, after the student finished minutes of talking, I still had no clues on what the answer is (for those questions expecting a "Yes" or "No" answer, I still couldn't figure out whether the student intended to say "Yes" or "No"!).

The bottom-up issue here is that the student failed to organize the conversation in a top-down way: I would expect the student to start the answer to my question like "I used two metrics. The first metric is X/Y where X is ... and Y is ... In particular, I measure X with faults seeded with mutation testing, .... The second metric is... " Instead, earlier the student started the talking w.r.t. my question with "Mutation testing is a commonly used way for measuring fault detection capability, ... YYYYY.. ZZZZZ, ....". For a while since the student's talking started, I had no clue or got lost on why mutation testing has anything to do with the metrics. Even when there are occassions where some background information needs to be laid out before giving out the answer, the student still should give strong signals as starting sentences for making clear how what the students will say first is related to the answer. Some examples can be "Before I describe the metrics that I use, I would like to first describe mutation testing, which is related to three out of four metrics that I use, and then after that I will list these four metrics....."

For those questions expecting a "Yes" or "No" answer, the student can start with "My answer is Yes, with three reasons. First, ...." or "My answer is both Yes and No. The reason for me to say Yes is .... The reason for me to say No is ..."

The context-less issue here is that the student failed to connect the background and context that I have (i.e., my question) with what the students tried to convey (i.e., what the student talked about w.r.t. the answer).

* Final remark

I would suggest students to realize these communication anti-patterns and address them. Students who have these anti-patterns in oral communications often have problems in their writing (especially in terms of logical thinking and writing) such as not laying out enough background or assumptions for readers to understand what the students write next, or not clearly and concisely organize the content to make the writing easy to understand.

The Minto Pyramid Principles (e.g., top-down writing style) are good ones to follow as starting points for students. Below are some slides on some key ideas there:
http://www.dbai.tuwien.ac.at/staff/gatter/work/051104_The_Minto_Pyramid_Principle.pdf
More are described in the Minto text book:
http://www.barbaraminto.com/textbook.html

Update: my colleague Macneil Shonle recommends the following book: The Sense of Structure: Writing from the Reader's Perspective, which may worth checking out. Thank Macneil for the recommendation!

This post more or less follows the top-down writing style, if you realize.:)

Graduate Student Survival/Success Guide

2009-10-31T08:28:00.000-07:00

Yesterday, I gave a CSC 600 lecture to new graduate students at NCSU CSC. The slides can be found here.

How not to keep your advisor up all night till last minute before the paper submission deadline?

2008-12-27T10:06:00.000-08:00

Posted on Friday Nov 07, 2008

I recently had been through a case where a student submitted the student's draft for my review of writing two days before a submission deadline. Since I had another more urgent task during the period, the required effort for helping improve the draft to a fine shape "forced" me to stay up the whole night till early morning 5am the submission deadline.

When I tried to recall several past submissions of this student, I found that all these submissions kept me up all night in the morning till last minute before the deadline whereas many other students' drafts were often already ready to submit the night preceding the submission deadline.

I asked myself "why would this situation happen for the student?" For this immediately past submission, in fact the student did a very good job in preparing the abstract, introduction, and approach (high level description) sections early on (for my review of ideas). But the problem is that the student submitted the draft for my review of writing late, only two days before the submission deadline. Several possible reasons may contribute to this late submission: the student might think that the student needed to finish all the sections of writing before asking peer review for writing, or the peer reviewing student needed to finish peer review of all sections of the draft before asking for my review of writing, ...

To avoid these issues (or avoid keeping me up all night till last minute before the paper submission deadline), below is a new set of advice to my students:

Please write early, ask peer review early, and ask for my review of writing early!! Here are some specific things you should do:

(1). You don't need to wait till you have a complete draft (with all sections finished) before requesting peer review. You can submit your partial draft (e.g., one or more completed sections at a time) for peer review.

(2). You don't need to wait till your peer-reviewing student finishes reviewing all the sections of your draft before requesting my review of writing. You can submit your peer-reviewed partial draft (e.g., one or more peer-reviewed sections at a time) for my review of writing. Remind you that I have a policy of being able to review your draft for writing ONLY after a peer student has finished reviewing the writing of your draft and you have fixed the issues pointed out by the peer student. But you don't need to get your peer review done before I can review your draft for the ideas described in the draft to avoid the delay on giving you feedback on the ideas in your draft.

(3). Budget your timeline to allow to ask me to review your writing for multiple iterations (instead of just one time or even 0 time iteration on any portion your writing before the submission).

Of course, it is most important for you to write early following my advice posted earlier!

On writing weekly lab book entries

2008-12-27T10:04:00.000-08:00

Posted on Sunday Oct 19, 2008

Every week before the one-on-one meeting (if no regular one-on-one meeting arranged, then on a weekly base), a student should submit a lab book entry in our group wiki http://sites.google.com/a/ncsu.edu/ncsu-ase/Home/Labbooks for
*. Planned activities
*. Actual outcomes

For the "Actual outcome" category, you basically copy the "Planned activities" from the preceding week and then annotate them with the actual outcomes.

For each category, you need to organize your items in the following subcategories:

*. Tool development
Description of your task items
Expected artifacts: (here you put only specific tool components, with the details of the location in CVS, e.g., the genAxim method of /toolsrc/jias/jias/axioms/

Axiom.java)

*. Empirical evaluation
Description of your task items
Expected artifacts:(here you put only the writing portions for describing the evaluation or its results, with the details of the location in CVS, e.g., the evaluation section of /papers/icsm08-soa/)

*. Paper writing
Description of your task items
Expected artifacts:(here you put only the writing portions, with the details of the location in CVS, e.g., the approach section of /papers/icsm08-soa/)

*. Misc
Other task items not falling into the three preceding categories

Your task item's description shall be detailed enough so that I can distinguish it from a previous item in previous weeks. For example, you shouldn't put the same item description like "Preparing a Journal Version of XXXX" in multiple weeks. That is, from your description, I can tell the semantic difference of your new task item from any of your previous items in previous weeks.

Note that only recognizable artifacts are tool source code and formal writing in LaTeX being put in CVS. The artifact description shall describe enough details for me to trace down to the artifacts without further asking you. If you cannot put an artifact for a task item in one of the first three categories, you shall move the task item to the "Misc" category. For example, "Explore various tools such as XXX to use in the tool development" shouldn't be put under "Tool development" since there is no artifact (tool source code) being produced by this task item. This task item shall be put under "Misc"; just like reading research papers, you should always explore various tools along the way of your actual tool development.

For the "Actual outcomes", you copy the "Planned activities" over and annotate each item with some description of the completed portion. You also need to list "Actual artifacts" after the "Expected artifacts".

If you don't produce any portion of an expected artifact, you need to put "None produced" and color that item with the red color. If you produce only partial portions of the expected artifact, color that item with the orange color.

On reviewing a student's paper drafts

2008-12-27T10:03:00.000-08:00

Posted on Saturday Oct 18, 2008

We will have three reviewing phases of your paper drafts:

Phase 1: Submit your draft to me for me to review the ideas (in the phase, I won't fix your writing but focus on your ideas, just like the way a PC member reviews your draft). After you receive my feedback, you revise your draft based on my feedback. Then you repeat Phase 1 till I explicitly tell you to do a peer review (moving on to Phase 2). When you send me a new version via iterations, you need to highlight the changed parts that need my attention.

Phase 2: Submit your draft to a peer student for the student to review both the writing and idea. After you receive the feedback from the student, you revise your draft based on the feedback. Move on to Phase 3.

Phase 3: Submit your draft to me for me to review both the writing and ideas. Note that I will review your writing only after you have gone through Phase 2.

This policy is also applicable to the intermediate writing during the research development besides the final writing before the submission deadline.

I want to point out another thing on intermediate writing during research development. Many students sit too long on a writing phase before actual tool development or evaluation. I expect the turnaround time for submitting your writing to me in a cycle of one week or two weeks. But some students take multiple weeks to prepare the writing to submit to me. Doing so loses the point of the original intention: getting in-time and early feedback on your work.

Here are my more explicit guideline on intermediate writing during the life cycle of research development:

(1) Prepare the abstract, introduction, example, and related work sections and submit them to me.

--> Start tool development

(2) Prepare the approach section and submit it to me

--> Continue tool development

(3) Refine the approach section and prepare the evaluation design section and submit them to me

--> Finish tool development
--> Conduct evaluation

(4) Prepare the evaluation results section and submit them to me

--> Now you shall have a relatively complete draft after you add the conclusion section and discussion section if needed.

Additional points:

*. In each later phase, you should refine and improve sections written during early phases.

*. Before you write down your ideas (on your approach or evaluation design, etc.) in your paper as formal writing, you need to discuss with me to get my OK stamp on the things that you are going to write down. I had been through some cases where a student spent a lot of time in writing down what the student thought to be a reasonable idea before discussing with me on the idea. After I read the writing, I rejected the idea with good reasons. Then the student basically had to discard the old writing (with a lot of time previously being wasted). That is, don't rely on only the formal writing as the only media for communicating with me with your ideas or work. You need to fully discuss with me with what you plan to write down in your formal writing.

Again, remember that you shouldn't sit on the writing for one of the above phases for too long.

Why shall we (research advisors) avoid directly writing on students' research papers?

2008-12-27T10:00:00.000-08:00

Posted on Sunday Oct 12, 2008

When students started their research in my research group, I told them that "Don't expect me to do tool implementation for your projects", "Don't expect me to do experiments for your projects", and "Don't expect me to write sections of your papers for you". Indeed, I didn't say that "Don't expect me to give you research ideas to work on" since it is very tough for students to come up with good research ideas to work on when they first start their research. At the same time, I have tried different ways of giving students space and opportunities to think about their own research ideas. Such ways are taking good effect and many students are improving themselves in coming up with good research ideas. That is a separate topic here and deserves another post of discussion.

Why shall I (as a research advisor) avoid directly writing on students' research papers?

Two main reasons. First, if I write for my students, they never get a chance to learn how to write themselves. If they don't know how to write papers, how can they write their own dissertation? The advisor cannot write the dissertation for them! How can they write their research papers after they graduate? The advisor cannot accompany the students for all their professional career.

Second, a bit selfish reason. I don't have much time to write papers for students: I have proposals to write (so far few of my students can write proposals for me but I have been trying to train them to have the capability when they get close to graduation), I have professional services to serve, I have a good number of students to work with, advise, and train, ...

Although I told students that I didn't write for them, I spent much effort in training them how to write and how to improve their writing. Below are several things that I do:

*. I mark my revisions and suggestions on hardcopy of their writing submitted for my review and revision; I don't directly write on the LaTeX source files or Word files (whatever format is used in paper writing). Doing so can "force" students to at least recognize and understand what I marked and commented.

*. I walk through with the students in person on some major issues being marked and tell them the reasons for my revisions. These reasons may not be written down on the hardcopy as comments. Basically, I want them to learn the corrections not only for the particular mistakes that the students made but also for future similar types of mistakes that the students shall avoid making. At the same time, I ask the students whether they have any doubts or questions on my marks and I then address their doubts and questions if any. Smart students will always take opportunities to ask me the reasons for revision places that the students don't understand.

*. Before requesting my review or revision of a student's paper, the student should ask a peer student to review and revise the student's paper. Doing so not only allows other students to be trained with review skills but also reduce my revision load on picking up those low-level, easy-to-spot writing issues. When I finish my revisions on the hardcopy, I expect that the peer student who reviewed the paper also look at my revisions and asked themselves "why didn't I catch these issues?" Another major motivation for reducing my revision load is that when a paper includes too many low-level writing issues, I would be busy in correcting those low-level writing issues and forget about the logical flows and high level technical content issues. That is, the student, the author of the paper, will lose much of the benefit given by me in improving his or her paper. Including too many low-level writing issues in a student's paper before submitting it for my revision would just do harm to the student, not being able to take full advantage of my advice on their high level, more important, writing issues and research issues.

*. I will pay much attention to the abstract and introduction sections, whose writing is very important and students often don't do a good job. These parts often require much more iterations of my revisions than other parts of the paper. Students often don't do a good job in organizing the logical flow and the organization or reasoning of the motivations.

*. I will ask students to turn their common mistakes into some regular expression patterns, and then use a tool called style-check developed by Neil Spring so that the students can guard against some low-level issues themselves (if they cannot guard against these issues manually easily). Frankly, I found that several of my students still make many low-level mistakes repeatedly and they don't use style-check as I suggested. I hope these students can take full advantage of tools to help address the low-level issues of their writing, and exploit and focus my guidance on the high-level issues of their writing.

Next I discuss what I do to help students to write early and write enough so that I can train their writing.

Many of you may wonder: when deadlines are coming and students are lagging behind in their writing, we would have to jump in to make these deadlines by writing the papers for the students since we really don't want to miss these important deadlines. Indeed, I had been through these situations.

One possible way to deal with that is to really impose strict policies to students: if they don't prepare their papers early enough for us (advisors) to review or revise, let's just bypass the deadlines. In principle, bypassing an important deadline will do more harm to students than the advisor: the advisor has multiple students to work with and have multiple papers to submit, but the students individually may just have a limited number of papers to submit yearly or in their whole graduate program. Each year, there exist only several top conference paper deadlines and a small number of good or right conferences for the work. Missing some important deadlines produces very negative impact on students' research record building (indeed as well as the advisor's), given that often the time papers will be rejected by top conferences.

But the above way often doesn't work (either for the advisor or the student). The advisor doesn't want to miss important deadlines either. The students also want to make the deadlines but their deadline-making or engineering skills (c.f. my talk slides on research skills) are often not good. Often the time, the students postpone their paper writing to the last minute; they often don't like paper writing.

Then how can I help students to write early? Here is what I do.

*. After a student (or I, or both) has some promising research idea, and both I and the student feel that the idea could have a high chance of leading to a good research project, before the student goes ahead to carry out the implementation of the approach, I will ask the student to write down the following sections (see my slides on writing papers for a paper's structure):
-- Abstract
-- Introduction
-- Example
-- Approach (the description of the approach could be high level)
(No implementation section or evaluation section is required at this stage)
-- Related work
-- Conclusion

If the student can produce some preliminary results (e.g., by doing some manual walk-through of the approach), the student can also put in a "preliminary results" section. The resulting draft would form a fine workshop submission draft (but note that it is not necessary for us to submit it as a workshop submission unless really needed in some cases to gather early feedback on the work from the community).

Basically, the student presents to me a "design" document (similar to one in software development) before the student starts implementation. Then I can revise and understand what the student is planning to implement, and brainstorm with the student on possible new ideas or improvement of the existing ideas based on the writing. Doing so can avoid some of the past issues on a student's approach or research project being a bit "black-box" to me and I found out issues when reading the student's draft when being very close to the deadline and then the student couldn't fix the issues that I identified.

Note that, at this stage, the student must identify a convincing motivating example and describe it in the example section before moving on to implement the approach. I had one past bad case where a student couldn't find a convincing motivating example for his approach and then moved on to the implementation of the approach, and ended up with an unsuccessful project.

*. After the student implements the approach, the student expands and refines the approach section and adds the implementation section. I will review and revise these sections. Note that in this step when the student does some hands-on stuffs, new ideas may come up and the originally written down approach can be revised and improved. I am very supportive on the student's hands-on experience and generating new ideas from hands-on experience. But that doesn't mean that the student should directly dive into tool implementations without thinking enough or finishing the first step of paper writing described above. In my belief, good new ideas can come from both creative thinking before tool implementation and during/after tool implementation.

*. Before the student carries out the evaluation, the student needs to write the experimental design subsection and other subsections in the evaluation section that can be written before the experimental results are available. Doing so can help me make sure that the student is doing the right things in the evaluation.

*. After the student finishes the evaluation, the student fills in the experimental results subsection. I will review the new subsections as well as the whole draft before submission.

Doing this style of phased writing, students benefit in several major ways. (1). They don't rely on me to write for them before the deadline (since I won't). (2). They allow me to have opportunities to iterate with them on their writing and improve their writing since in the early phases, we don't have the deadline pressure. (3). They can gather feedback from me early on to help them to avoid making wrong decisions or misinterpreting my suggestions in their research development. (4). Our remote research collaborators (if any) can read through the early formal writing to give us feedback and suggestions on the research development.

So far I have quite good experience in training and educating students to write research papers with the above ways. In the end, I indeed enforce what I told my students "Don't expect me to write sections of your papers for you". More importantly, most of the students (after working with me for a short period of time) in my group can independently write whole papers without my writing there (with my helps only in a way of my not-very-significant revising on hardcopies of their papers). I am very glad to see that happen while knowing that many of my fellow junior faculty members are still busy writing papers for their students.

Hope my experiences described above can further help my students to understand why I am doing so and further improve what they are doing while working with me, and help other advisors at least to think how to do better along this line of advising students in writing papers...

Research skills

2008-12-27T09:58:00.000-08:00

Posted on Saturday Oct 04, 2008

Yesterday I gave a talk on research skills, whose slides are here. Feedback is welcome.

Party for NCSU software engineering people

2008-12-27T09:57:00.000-08:00

Posted on Saturday Dec 01, 2007 by HWANG, JEEHYUN

We joined the party for NCSU software engineering people hosted by Dr. Laurie Williams
Some pictures of ASE group as follows!!
pic 1
pic 2
pic 3

More on formal writing before one-on-one meetings

2008-12-27T09:54:00.000-08:00

Posted on Wednesday Oct 24, 2007

Here is the definition on formal writing:

(1). The formal writing includes the text that you can turn into a part of your future paper submission directly or with minor polishing. If you just write in some high-level bulleted points like those in slides, this type of writing is not formal and not acceptable in terms of formal writing.

(2). Because our group uses LaTeX as the format of writing papers, your formal writing needs to be in the LaTeX format. If you don?t know how to use LaTeX in writing papers, take a look at

http://people.engr.ncsu.edu/txie/publications/writingtools.html

Especially on which software packages to use for editing and compiling LaTeX source files.

(3). Because our group uses CVS to keep track of revisions and allow collaborative writing, your formal writing needs to be put in our research server?s CVS repository. Basically after you set up CVS, you can create a subdirectory under /cvs/root/papers/ with the naming convention of ?lastname-conferenceorworkshopname? (e.g., acharya-FSE07). If there is no specific conference or workshop to aim at currently, you can put the name of your project/tool/topic in the place of ?conferencworkshopname?. For info on how to set up CVS and use Eclipse to checkout CVS, take a look at:

http://ase.csc.ncsu.edu/server.html#cvssetup

Then your submission of your formal writing is an email including some words like ?My formal writing so far is included in the CVS directory XXXXXX. You can check it out.?

Basically you can view the formal writing that you submit before our one-on-one meetings as a portion of the paper that you are going to submit eventually. Week after week, you will expand the draft by filling in additional text that describes what you have done in the preceding week(s) and in the upcoming week(s).

Note that initially or early in the phase of your formal writing, you shall write the abstract, introduction, example sections early on. In addition, you may also start writing the related work section when you read other researchers? papers early on. Writing these preceding sections doesn?t require any tool implementation or experiment. Then along the way of week-by-week work, you fill in the approach/implementation sections when you have more implementation details figured out and more development work done, you fill in the experiment setup and design sections when you try to set up your experiment, and you fill in the experimental results section when you finish producing experimental results, ?

This mechanism is to fix several issues being faced nowadays.

(a). students tend not to write serious/formal text along the way but put a lot of efforts in formal writing immediately before a submission deadline. Then the students cannot get helps from me on their writing early on.

(b). students tend not to disclose sufficient technical details or progresses of their projects along the way during one-on-one meetings week by week. Often immediately before the deadline, some students gave me ?surprises?, disclosing to me that they didn?t do some part that they were expected (by me) to do or they did something in an un-optimized or incorrect way; then it is often too late to fix these issues when getting too close to a deadline.

(c). when students don?t write things down in formal writing, they don?t have good feeling in the approach/tool design, experiment design, ? I often come up with good new ideas when I formally write down ideas in my proposals and I expect students to enjoy similar benefits by doing formal writing along the way.

Written materials prepared before one-on-one weekly meetings

2008-12-27T09:51:00.000-08:00

Posted on Saturday Oct 13, 2007

I recently talked to a colleague, Dr. Nagiza Samatova, who kindly shared her experience in training students' writing, and inspired by her way of training, I have tried to install a similar mechanism in my research group. I suspect that it will solve some students' issues in delaying writing in the last minute and turning their research as a black/grey box to me. Below is adapted from my email sent to some students in my group who have already had some concrete research projects ongoing:

Before one-on-one student meetings, the advisor requires the student to bring formal technical writing on the things to be discussed: the written materials later will be turned into a part of a paper submission so it is not wasteful or just specific for being used in one-on-one meetings.

For example,

-- if you plan to discuss a new idea that you may have, write paragraphs describing it, which can be turned into the introduction section, example section, or approach overview section of your future paper.

-- if you plan to discuss about design and implementation of your approach, write paragraphs describing these designs or implementations, which can be turned into the approach and implementation sections of your future paper.

-- If you plan to discuss about your evaluation, write paragraphs describing your experiments (either experiment setup, design, subjects, or results), which can be turned into the experiment section of your future paper.

-- If you plan to discuss other related papers that you read, write paragraphs describing them and the differences of them with your own approach, which can be turned into the related work section of your future paper.

In any case, you shall prepare your writing and present it to me along the way of weekly one-on-one meetings rather than a big bang in the end immediately before the deadline. Doing so can allow me to (1) give you early feedback on your work and writing and to (2) keep track of your work since currently your work?s technical progress is more a black/grey box to me.

In addition, this mechanism would be also very helpful to yourself in keeping yourself in having the habit of writing things down more formally (when you try to write things down more formally, you can have a better idea and generate new good ideas).

I expect you to send me an email telling me the sections/paragraphs in LaTeX in your paper in a specific CVS paper directory ** not later than the same morning ** of an afternoon one-on-one meeting. I don?t accept informal writing being put in the body of an email message or any way other than the preceding specified way.

If you cannot prepare such writing before a one-on-one meeting, I would suggest you to cancel or postpone that week?s one-on-one meeting with me. If you cancel or postpone too many weeks? meetings, the implication of reflecting your work progress/performance is self-evident.

Can we learn from Dr. "House" in doing research?

2008-12-27T09:46:00.000-08:00

Posted on Saturday Oct 13, 2007

I enjoy watching Fox's House TV series.

I find the problem sovling skills and creative thinking there to be inspiring for us in doing research.:)

I enjoy watching how Dr. House advises his "students". I hope to learn the good poritions of his advising styles.:)

"DR. GREGORY HOUSE (Hugh Laurie) is devoid of bedside manner and wouldn?t even talk to his patients if he could get away with it. Dealing with his own constant physical pain, he uses a cane that seems to punctuate his acerbic, brutally honest demeanor. While his behavior can border on antisocial, House is a brilliant diagnostician whose unconventional thinking and flawless instincts afford him a great deal of respect. An infectious disease specialist, he thrives on the challenge of solving medical puzzles in order to save lives.

For the past three seasons, House has shepherded an elite team of young experts who helped him unravel diagnostic mysteries. In addition, he has a good friend and confidant in oncology specialist DR. JAMES WILSON (Robert Sean Leonard). There?s some volatile chemistry between House and DR. LISA CUDDY (Lisa Edelstein), the Dean of Medicine and hospital administrator; the two are in constant conflict over House?s duties and unconventional behavior, but even she would admit that his brilliance is worth the trouble.

In the Season Three finale, the set-in-his-ways House was confronted with a series of major changes to his team. Neurologist DR. ERIC FOREMAN (Omar Epps) left Princeton Plainsboro because he didn?t want to turn into House; House randomly fired old-money intensivist DR. ROBERT CHASE (Jesse Spencer), claiming he learned everything he?s going to learn in the past three years, or nothing at all; and immunologist DR. ALLISON CAMERON (Jennifer Morrison) resigned, knowing House will be completely unaffected by her decision.

As Season Four opens, House is without a team to contribute to the perplexing medical cases he undertakes, and Cuddy and Wilson are adamant that he recruit new fellowship candidates. After 40 applicants applied for the newly vacated spots on his team, a group of five doctors -- played by Olivia Wilde, Kal Penn, Peter Jacobson, Anne Dudek and Edi Gathegi -- have emerged as finalists vying for the coveted and hotly contested openings."

Reading papers - 5 line summaries!

2008-12-27T09:38:00.000-08:00

Posted on Thursday Oct 11, 2007 by ACHARYA, MITHUN

Dr. Xie maintains a very nice bibliography on Mining Software Engineering. We read lot of papers, but with time, tend to forget them. How about having a 5 line summary for each of the paper we read as a part of literature survey? I actually maintain a document which does exactly this and find it very useful. So next time I forget whats in a paper, I go to my document and look for the 5 line summary, and I immediately know what the paper talks about. I dont need to read the paper again. Another useful side-effect of this exercise is when you write related work for any of your papers or thesis. In conferences, when you talk to other researchers, they usually ask - "Have you seen paper X? How is your work different from paper Y?" and its bad not to know some really relevant related work!

Most well written papers, can be read in about 15-30 mins and summarized in about 5 lines. In my field, most papers have a motivating example after introduction. For a well written paper, a reader should get the idea of the whole paper when he completes reading the Example section! So the way I read a paper is - read abstract, look at the conclusion, then read introduction (very fast), and then the example section. This process takes about 15 mins. Then I skim through the framework, implementation, and evaluation details. I spend further time on the paper, only on need basis. Then I summarize the whole paper in about 5 lines! During early years of PhD, it might be beneficial to read the whole paper to learn the art of writing papers... but after getting a hang of writing papers, quick paper reading will be a useful skill!

Promoting Research Group Spirit and Peer Student Support

2008-12-27T09:31:00.000-08:00

Posted on Saturday Oct 06, 2007

Earlier I didn't emphasize much on research group spirit. Recently I realized its importance and tried some measures to promote research group spirit.

I found that UIUC's Prof. Jiawei Han's several measures in his data mining research group could be valuable to borrow. I borrowed them recently in my group.

1. Allow students to volunteer to take on some services in the group. In the past, I (as the advisor) took on most of the services in the group including maintaining the group web pages, coordinating the group meetings, etc. Then students might feel like being managed without feeling to own the research group. In addition, I am too busy in doing these types of things and the students don't learn how to organize things or manage things: an important skill in their future career.

In the research group, early this semester I asked students to volunteer to take on various roles in the group:

*. Group Webmaster (news, group Web page, pictures, etc)
*. Group meeting coordinator
*. Server system administrator
*. Industry/visitor coordinator
*. Conference and journal review coordinator
*. Research proposal coordinator
*. Social activity coordinator

I found this mechanism works pretty well. For example, recently when a visitor from industry gave a guest lecture in my course when I was out of town, I asked the industry/visitor coordinator to organize student meetings with the visitor by introducing our research and doing demo; the whole process was organized by the coordinator with help from other students. The process went well and the students can also improve their independent skills: when the advisor is not around (in the future after they graduate, their advisor won't be around!), they can still successfully carry out things.

But I still need to figure out a way to encourage students to send emails in our group mailing list, whose emails are primarily sent by myself.

2. Acknowledge and honor those students who made great achievements in research so that these students can feel being recognized and other students can learn from these students and try to catch up. Jiawei Han's group honors the best-performing students each semester after students submit their research performance summary for the semester. Recently our research group also held voting among students (each one vote) and myself (with two votes, as suggested by one student, saying that my judgment would be more comprehensive). In the end, we voted one golden award winner and two silver award winner (with the same number of votes).

3. Besides borrowing Jiawei Han's measures, I also tried to promote peer support among the students in the group. Earlier the whole group activities centered around me, including reviewing their paper drafts, giving feedback on their research, etc. I would hope to set up a peer support system so that students can help each other and learn from doing so. Since some time ago I encouraged students to do proof reading each other's papers, and help each other. I will think of more other measures in promoting peer support.

4. As a routine practice in many research groups, asking students to present their own work or other related work by other researchers is quite valuable. Earlier I used the group meeting time slots to go round-table debriefing and I found it not that worthwhile in spending time. Nowadays, instead, in each group meeting, each student makes a presentation and then other students and I give feedback either on the content or presentation skills. Again, in this way, the group meetings shift from being dominated or driven by myself to being managed by students themselves.

I will think of more other measures in promoting peer support and group spirit. If you have any comments, you are welcome to discuss here.

A mechanism to help students in indepenent thinking

2008-12-27T09:27:00.000-08:00

Posted on Monday Sep 17, 2007

As a graduate student, you are supposed to grow to be independent along the way. To help you to do that, I ask you to allocate the last 10 mins of the one-on-one 30 mins meeting time slot to train your independent thinking.

Basically in these 10 mins, you should tell me your thoughts on answering one or more of the following questions:

--- What to do in more details for the current project idea if it is not that detailed or clear enough?

--- What is the next good idea (beyond the current one) that you should work on?

--- What would you do in the next big phase (either in 1, 2, 3, 4, or 5 years)?

What does this mechanism mean? You should think about future research ideas ALL the time!! I have been always thinking about new research ideas all the time. You should do the same rather than relying on me to tell you what to do next. In addition, you should actively read more and think more with the explicit goal of generating ideas for your future research. It is not acceptable that you tell me that you haven?t thought about IT when we reach this 10 min slot, because you are supposed to think along the way (jogging, walking, taking bus, taking shower, sometimes driving but be careful, ?)

Indeed, during this 10 mins, I would help you brainstorm ideas together (I found I myself generate some very good ideas when brainstorming with students together). But you should bring something on the table rather than relying on me to bring something on the table for you.

Levels of research committments

2008-12-27T09:23:00.000-08:00

Posted on Friday Aug 03, 2007

Four levels of research commitments can be classified for students (see below). Each of you shall classify yourself to one of these levels. Each of you shall try you best to reach or keep in Level 1. Not many students are currently reaching or staying in Level 1.

There are positive iterations when you reach Level 1. When a student is efficient and effective in finishing research tasks, the advisor will work with the student on coming out new good ideas for the next research project. If the student stays on an existing old research project for a long time, before the advisor works with the student on a new good research project, the advisor will wait for the student to finish the old one up or wait for the student to tell the advisor that the existing old project has no hope and the student would give it up.

Some students fall into Level 4. It is a very dangerous situation. Staying on Level 4 can cause you to stay in the program for a long time without producing any research results.

Level 1. Self-propose timeline in research tasks and often succeed in accomplishing the research tasks

Level 2. Self-propose timeline in research tasks but often lag or fail in accomplishing the research tasks

Level 3. No self-proposed timeline in research tasks but be willing to discuss with the advisor in research timeline

Level 4. No self-proposed timeline in research tasks and even no responses upon the advisor?s requests on checking research status

Advice on making submission deadlines

2008-12-27T09:20:00.000-08:00

Posted on Saturday Jun 09, 2007

I found that some students who are supposed to drive their research and preparation of a certain submission draft are not active or responsive enough. Below is my advice on dealing with the issue.

1. Students need to early on take full advantage of the advisor and other colleagues (in the co-author list) in helping improve the draft and the work. As I told students in our group, students should write the whole draft (when there are peer colleagues/students in the co-author list of your paper, you may coordinate with them to ask them to write some sections of the paper). Enforcing students to write all sections can help train their capability of independently writing the whole draft. Of course, the advisor will help you by giving you suggestions on how to revise your draft.

That doesn't mean that you need to submit your draft for your advisor to review only after you finish the whole draft. It is great if you can finish your draft very early on and send your whole draft to your advisor. But more commonly many students feel tight in making their drafts ready.

Then the students need to make efforts to gather early feedback from the advisor by giving section by section to the advisor for review comments and feedback if they cannot prepare their full draft early on. Like in bug finding, the earlier that a bug is detected, the better off you will be in fixing the bug.

In all, try to get early feedback from the advisor or co-authors incrementally with available sections early on rather than putting off sending your writing to them very near the deadline. In the latter case, the advisor may not have enough time and you may not be able to incorporate the feedback to improve the draft.

2. Students need to be **responsive** to the advisor or colleagues. Responding your advisor's emails should be on the top priority if your advisor's emails explicitly asked for responses with questions. As you can see, I always give rapid response to students and colleagues. As I discussed in the group meeting, in some other groups either inside or outside NCSU, students may complain that their advisor is slow in responding their emails. In our group, the other way around happens often, believe it or not!

If you are too busy and cannot spend time on some task mentioned in the advisor's email, you can simply respond so and then the advisor or the colleagues can know it and make alternative arrangements or schedule their time line before the deadline.

The advisor's goal is to help you to make these deadlines, produce good work, grow to be independent enough, and then graduate, find your desirable job. Being not responsive or not effective in making deadlines or making progress in your work can hurt yourself much more than anyone else. That is why I told you "Do you want to make the deadline. If not, it is totally fine to me."

That is, it is **you** who want to make the deadlines and it is **you** who need to drive your research, not anyone else.

Good luck on your deadline catching!