Outward Thinking for Our Research Community

In our research community, besides teaching duties, university researchers commonly focus on securing funding resources to support their research group's research activities, recruiting and advising students along with collaborating with external (academic or industrial) collaborators to conduct research, and contributing professional services within the research community such as ACM SIGSOFT and IEEE TCSE along with their sponsored or associated journals or conferences. While sharing many of these focuses, industrial-lab researchers typically do not need to teach, secure external funding resources, or advise students to conduct research (except supervising summer-intern students during summer time). These preceding activities are mostly "inward facing" (e.g., within the research community): competing funding resources within the research community, publishing and presenting inward-facing research to serve the audience within the research community, conducting professional services within the research community, etc.

A couple of years ago, John Regehr (U. Utah) discussed "Inward vs. Outward Facing Research". When discussing why researchers have a tendency to face inward, he stated "This is natural: we know more about our research’s internal workings than anyone else, we find it fascinating (or else we wouldn’t be doing it), we invent some new terminology and notation that we like and want to show off, etc. — in short, we get caught up in the internal issues that we spend most of our time thinking about."

However, the research community (either broadly in computing or specifically in a research area such as software engineering) can be impacted by the broad ecosystem that the research community lives in. For example, a government funding agency needs to allocate total funding across different research areas; a university (college or department) or industrial research lab needs to allocate the total head count for new hires across different research areas; a student applying for graduate schools needs to decide on what research area he or she would like to apply for. Thus, it is important for the research community to sustain and further boost the importance, impact, and reputation of the research area perceived in the eyes of other computing research communities or in the broad society such as the industry and government. Only when we accomplish so, can the community as a whole grow better, e.g., attracting more resources such as funding allocation, faculty positions, industrial-research-lab positions, top student/junior-researcher talents, other communities' researchers to address our important problems.

In particular, besides the typical inward thinking that researchers in the research community already have, more outward thinking is needed for the research community to better grow in this broad ecosystem. In the broad computing community, there are quite some prominent researchers and other research areas with such strong outward thinking. For example, Ed Lazowska (U. Washington) "is widely viewed as the computer science research community's highest impact national leader and spokesperson." For the research area of Artificial Intelligence (AI), Eric Horvitz (Microsoft Research) has addressed "Rise of Concerns about AI: Reflections and Directions: research, leadership, and communication about AI futures". The ACM SIGAI Newsletter (the counterpart of our ACM SIGSOFT Notes) named as "AI Matters" includes its first-listed submission category "AI Impact": "Description of an AI system or method that has had a tangible impact on the world outside of the research community. These submissions are intended for dissemination to a general audience, and any technical content must be accessible to non-AI researchers."

In the research area of software engineering, there have been also various leading researchers with such strong outward thinking, playing important roles in the broad ecosystem. Although the research community has started paying attention to engaging and impacting the software industry or broadly practitioner community, the research community shall also put more eyesight on impact on and perception by other research communities in the computing community along with the broad society such as the industry and government.

Below I list some example activities towards such outward thinking for researchers in the research community to consider:

• Propose and organize events that help establish funding programs and initiatives or help communicate the research community's important problems and great achievements. For example, recently, Bill Scherlis (CMU) co-organized the Workshop on Safety and Control for Artificial Intelligence sponsored by the White House Office of Science and Technology Policy (OSTP) and CMU. Kevin Sullivan (UVa), Annie Anton (Georgia Tech), et al. are organizing the CCC Visioning Activity on "Cyber Social Learning Systems". In FSE 2014, the first CCC Blue Sky Ideas Track "Visions and Challenges" was organized by Jian Lv (Nanjing U.) and myself. In ICSE 2016, another CCC Blue Sky Ideas Track "Visions of 2025 and Beyond" was organized by Gail Murphy (UBC) and David Rosenblum (NUS). A recent example interdisciplinary workshop is the NSF Interdisciplinary Workshop on Statistical NLP and Software Engineering (NL+SE 2015) co-organized by Prem Devanbu (UC Davis).
• Start new outlets and leverage existing outlets to help communicate the research community's important problems and great achievements. For example, CACM's Research Highlights section enables various ACM SIGs to highlight their selected research results to the broad CACM audience. Will Tracz (Lockheed Martin Fellow, Emeritus (retired)), the Most Recent Past Chair of ACM SIGSOFT, has served as the Webinar Coordinator for ACM Professional Development (PD) Committee, and helped start the ACM SIGSOFT and PD sponsored webinars, co-organized also by Robert Dyer (Bowling Green State U.), providing a great outlet for outreaching webinar talks to both within and outside of the research community. I have served in the ACM History Committee as the ACM History SIG Governing Board (SGB) Liaison, outreaching SIGSOFT history and impact activities to other ACM SIGs. 
• Nominate well-deserved colleagues and high-impact work for recognitions at the level of the broad computing community besides recognitions at the level of the research community such as ACM SIGSOFT awards and IEEE TCSE awards. There are various awards from the ACM, IEEE (IEEE Computer Society), CRA, NCWIT, etc. There are various fellow and distinguished member programs from the NAS, NAE, AAAS, ACM, IEEE, etc. A recent prominent example was that Mary Shaw (CMU) received the National Medal of Technology and Innovation in 2014.
• Volunteer yourself or nominate/encourage qualified colleagues to serve the broad computing community besides one's own research community. There are various professional organizations in the broad computing community such as various ACM boards and committees, IEEE Computer Society boards and committees, and CRA boards and committees. In recent years, prominent researchers from our research community already served in various leadership positions in the broad computing community: Alex Wolf (UCSC) as a past ACM President, Barbara Ryder (Virginia Tech) as a past ACM Vice President, Mary Lou Soffa (UVa) as a past member-at-large of the ACM Council, Carlo Ghezzi (Politecnico di Milano) as a Past President of Informatics Europe, etc.
• Apply for relevant temporary/rotator government (funding agency) positions. There are short-term government fellowship programs such as AAAS Science & Technology Policy Fellowships, White House Fellowships, and IEEE-USA Congressional Fellowships. At times, there are various program-manager positions in government funding agencies (such as NSF and DARPA) available for university or industrial-lab researchers to apply. Program managers may have opportunities to start new funding programs or initiatives on various important topics. For example, in past several years, Kathleen Fisher (Tufts U.) served as the original program manager for DARPA's High-Assurance Cyber Military Systems (HACMS) program, within which the supported SMACCM project produced SMACCMCopter as probably "the most secure UAV on the planet". Suresh Jagannathan (Purdue U.) has served as the program manager for DARPA's Mining and Understanding Software Enclaves (MUSE) program and Building Resource Adaptive Software Systems (BRASS) program, etc.

Please let me know if you have additional ideas for such outward thinking by sharing your suggestions here or submitting your writing to be considered for dissemination in the History and Impact column of the ACM SIGSOFT Notes!

From Knowledge Economy to Maker/Innovation Economy: Towards Student Education and Training

I recently watched an online talk on "More Innovation Through Education" given by Richard Miller (President of Olin College). Among many great things mentioned in his talk (e.g., the education-program improvement at the College of Engineering at Illinois), I especially resonated with what he presented (during 6'40''-12'03'') in one of his slides, whose screen snapshot was captured as below:

Although Richard Miller's talk seems to focus on undergrad engineering education, my discussion below centers around educating and training graduate students especially PhD students in engineering based on the points that he made on the above slide.

The traditional education focuses on the Knowledge Economy: teaching students about knowledge, making them "knowledgeable". With technologies such as Internet, Google, and Bing at hand these days, being "knowledgeable" may not be as critical as in the ancient days.

The Maker Economy is about putting What you Know into What you can Do (in 2014, the White House started the movement of "Nation of Makers"). It is related to what I often tell students: "Get Things Done". In these settings, students are given an (often well-defined) problem/requirement (e.g., a programming task or data-analysis task), and need to solve the given problem (desirably in a timely fashion). In graduate-research settings, students are often encouraged to leverage whatever attainable resources to get things done (some of these ways of leveraging resources may not be common in settings of undergrad-course assignment): reusing/adapting open source code, building upon existing infrastructures, searching at Google/Bing or Stack Overflow, getting help from and collaborating with peer students or the advisor, etc. Note that conducting a task in such settings is different from conducting a programming task in a solo homework assignment in a course where students typically need to independently write their code for the task, not allowed to reuse/adapt code from the Internet, etc. 

As a concrete example, in graduate-research settings, graduate students (software engineers in industry too) may be given a programming task such as implementing a new feature upon an existing big code base. Many students may tend to take a relatively long time to understand the big code base, focusing on improving what they Know, without sufficiently focusing on what they need to get Done. Tips to Developers Starting on Large Applications include (1) don't try to understand the whole application, (2) focus on delivering immediate value, (3) important skills required to maintain large applications, (4) tools to find what to change and to find the impact of a change, (5) two caveats to the preceding tips: do not compromise on code quality and do not stop making an effort to understand the architecture. I myself practiced these tips in my past effective and efficient development efforts for tool features, i.e., Get Things Done Efficiently. 

The Innovation Economy is about what you can Conceive, new products, new markets, new ideas, ... Instead of being given a well-defined problem/requirement, students need to come up with (1) what they would like to tackle, either narrowing down to a target problem when given no (even vague) problem to start with, or concretizing a given vague problem to a specific, concrete one; and (2) how they could tackle the target problem with new ideas.

In the past, I summarized five critical research skills for PhD researchers/students in my talk on "PhD-Program Preparation for Successful Post-PhD Career" and below are my mapping of these skills to the Maker Economy and Innovation Economy.

• Assessment skill. (Innovation Economy) 
• Vision skill. (Innovation Economy)
• Design skill. (Maker Economy & Innovation Economy)
• Execution skill. (Maker Economy)
• Communication skill. (Maker Economy & Innovation Economy)

By the way, mentioned in Richard Miller's online talk (at the time of 7'20''), the Nobel-laureate economist James J. Heckman found that success can be better predicted by grit (a combination of perseverance and passion, about attitude, behavior, and motivation), much better than knowledge (how much you know). Angela Lee Duckworth's 6-min TED talk on "Grit: The power of passion and perseverance" provides a nice quick overview of grit. There are also ideas on training grit, such as 16 Learning Strategies To Promote Grit And Delayed Gratification In Students.

Diversity and Inclusion in Research Community: Remembering David Notkin

Three years ago today April 22, my PhD advisor David Notkin passed away after a long battle with cancer. It was a big loss to our research community, especially to those who were close to him. The legacy and impacts that he left behind have been huge in many different ways upon many of us.

David devoted much effort in promoting diversity and inclusion, in both traditional senses and broad senses. To remind you of David's lovely voice, perfect-timing humor, and deep insight, you should (re)watch his flash talk on "increasing flexibility" in the 2012 Summit of the National Center for Women & Information Technology (NCWIT) (David was the founding co-director of the NCWIT Academic Alliance). There were also various blog posts, articles, videos on great memories about David such as a blog post contributed by his sister Debbie Notkin, a Seattle Times article contributed by his friend Jerry Large, and David Notkin ICSE 2013 Tribute.

David was a caring, amazing person with a big heart. I still clearly remember what he said in his remark at the NotkinFest in February 2013, as nicely summarized in Jerry's Seattle Times article (the full quote can be found in the end of Debbie's blog post and in David Notkin ICSE 2013 Tribute in David's own voice):

"David reminded the people gathered that day that we were privileged. He said his parents were the children of poor Russian Jewish immigrants, and taught him and his older sister, Debbie, that every person on Earth has value. “We have to figure out how to give more back,” he said, so that more people can better their lot."

I guess the above quoted words of David served as one of the driving forces for his great passion and devotion to promoting diversity and inclusion. Here, I will share some memories and stories to celebrate his legacy and impacts related to diversity and inclusion in our research community.


Focus on students. David expressed in his 2006 ACM Fellow Profile that students were his greatest influence. He posted the following sentences in the "students" page of his homepage: "My philosophy about working with students is taken from my adviser, Nico Habermann: Focus on the students, since graduating great students means you'll produce great research, while focusing on the research may or may not produce great students." Many professors may have heard of such quote but in reality, very few could faithfully practice such philosophy. Too often many professors immerse themselves in their research, without paying sufficient attention to the main interest or benefit of their students working with them (e.g., putting eyesight on only getting the research done while not sufficiently training/educating the collaborating students). But David had constantly instilled such philosophy in his daily work advising his students. Not only just exercising such philosophy on his own students, David extended this philosophy to people around him, especially those more-junior colleagues and young researchers/students. Jerry's Seattle Times article says it perfectly: "He was a modest-living mensch with a gift for making other people feel special, like they were a big deal. And to him, they were." Videos including first-hand witness statements from his former students, colleagues, and friends can be found here.


Be aware that everyone is different. After David encouraged me to apply for a faculty job back in the end of 2004, I was fortunate to become a professor myself in 2005, being ready to inherit David's philosophy on focusing on students. Since then, I have put much thought into reflecting on how to conduct impactful research, train students' research skills, map out a research agenda, write research papers, improve technical writing, etc. I have documented my reflection and advice publicly under my homepage. I felt that such way of generalizing things into patterns and tips could allow my students (and other students/researchers) to effectively master those skills under such concrete guidance. In my research group, I also experimented and improved various group activities/practices to more effectively and efficiently manage my group (after I communicated with some MBA friends, I later learned there was a term organizational behavior for studying what I experimented with). Some years ago, during a conference's hallway conversation, I was talking to David along with another junior professor on my efforts of trying to generalize best practices and patterns for training my students and managing my group. David patiently listened with slight smile on his face as usual. In the end, he patted my back and said "Tao, what you did was great but also keep in mind every student is different."

His sentence was short but powerful, making me reflect for a while. Indeed, under David's advising and mentoring, every David's former student is different (in a good way) in research directions, research styles, ways of thinking, ways of making impacts, ways of advising students, etc. There is diversity among David's former students! In terms of advising students, it is quite common for a professor (especially a junior professor) to expect his/her students to work or act like himself/herself during his/her old-day grad schools or current faculty job, while often not realizing some of his/her students may have different learning styles or work-life balance, etc. than he/she has. There we should be aware of "every student is different" among themselves and than the advisor.


Encourage and support young people. In the software engineering research community, our flagship conference ICSE has had a great tradition of organizing the New Software Engineering Faculty Symposium (NSEFS), where graduating PhD students or new faculty members attend to get good exposure on tips and lessons learned to excel in the faculty career. Many years ago, after a year's NSEFS, some attending students chatted with David, sharing that the atmosphere in that year's NSEFS was quite intense: quite some speakers expressed the stress and the "toughness" faced during the tenure process, etc. David expressed his concern that such communication might unintentionally discourage promising young people to pursue faculty career, and then they might not want to give it a try during job hunting, given that they might have heard such similar voice in other places for so many times. David thought that our research community should take special care to construct a welcoming and encouraging environment to encourage and support young people to pursue excellence, in either academia or other workplaces.

Sometimes our research community may not pay sufficient attention to encourage or reward young researchers such as students and early-career researchers, who are a group of researchers our research community needs to especially protect and cherish: they are the future! With the help from the ACM SIGSOFT leadership teams of different terms over multiple years, I initiated and gladly witnessed the establishment of the SIGSOFT Early Career Researcher Award along with the earlier established SIGSOFT Outstanding Doctoral Dissertation Award. Following these good moves, our research community should continue to reward and celebrate achievements and impacts made by researchers especially young researchers.


Be open. Many years ago, there was voice from our research community in expressing some concerns. Basically, professors from other non-software-engineering fields along with their students submitted their papers to our flagship conferences, and got their papers accepted; however, these professors typically didn't come to attend our conferences (likely they were too busy to attend both the conferences in their own main field and our conferences) but sent only their students to attend and present the papers in our conferences. Some researchers from our research community concerned that if senior authors of a non-trivial portion of papers accepted in our flagship conferences didn't come to attend our flagship conferences, the nature of our conferences as community building might not be well preserved, and our research community should do something about it. I heard that David was strongly against taking any community policies or activities to make our traditionally open community to be less open; our research community should encourage (instead of discouraging) people from other fields to help our community to tackle some challenging software engineering problems, either working by themselves or collaborating with researchers from our field. David's standpoint reflected his broad and long-term perspective along with his deep belief in diversity and inclusion.


Watch out on dichotomies. In 2009, David wrote an invited article on "Software, Software Engineering and Software Engineering Research: Some Unconventional Thoughts" for the Journal of Computer Science and Technology (JCST). There, David said "There are a number of pressures on researchers, in any discipline, to argue often and loudly about the benefits of "their" approach over those of some of their colleagues. This has benefits, but it also has downsides. In particular, it often leads to a mindset where two alternative approaches are considered to be in competition with one another. At times, of course, one approach to solving a problem may indeed be shown to entirely dominate another. In general, however, there is far more value in viewing alternative approaches as complementary rather than competitive." Such viewpoint might be partly attributed to David's hobby of practicing Aikido, as reflected by his own words in his 2006 ACM Fellow Profile: "Aikido is a purely defensive martial art. It isn't supposed to be attacking. Sometimes, there are too many conflicts in industry or academia over style or approach. We should not be attacking each other. Some people make their issues into dichotomies between industry and research. This is a false dichotomy. We need to work together. I guess I learned this from Aikido."

To remind myself and our research community on striving for diversity and inclusion in broader senses along with producing impacts on the society, and to continue and broaden David Notkin's legacy and impacts, I will end this blog post with two quotes. One quote is from last year's Forbes article on describing one of the five trends driving workplace diversity in 2015:

"Diversity’s Definition Has Changed: In addition to creating a workplace inclusive of race, gender, and sexual orientation (to name a few), many organizations are seeking value in something even simpler, diversity of thought. In some industries that are known for being insular – think law or high-tech companies – seeking out talent with different thinking and problem solving backgrounds in critical. Deloitte research underscores that diverse thinkers help guard against groupthink, a dynamic I observed firsthand last year with a large corporate client. Partnering with the company just after they had experienced a major product failure, the CEO lamented that the failure resulted from too much blind agreement internally – something Deloitte’s study calls “expert overconfidence.” Future-thinking companies see the danger in this lack of diversity and often question their own hiring and retention practices—and even their everyday operating norms."

The other quote is from David's own writing during the last phase of his life, published as his last Editorial in February 2013 of ACM TOSEM, for which David served as the Editor in Chief (I myself highlighted in bold several sentences in the quote as below):

"... let me make just a few quick comments about publishing software engineering research. I care about publication of results, as can be gleaned not only by my work on TOSEM but also as program chair at major software engineering conferences. There is a massive amount of important discussion going on about journals, conferences, who should own the publications, who should pay for the publications, and more, much more. Please pay attention to this as it is highly material to our field. But I want to say something that (I hope) transcends those discussions. Specifically, I’d like very much for each and every reader, contributor, reviewer, and editor to remember that the publications aren’t primarily for promotions, or for citation counts, or such. Rather, the intent is to make the engineering of software more effective so that society can benefit even more from the amazing potential of software. It is sometimes hard to see this on a day-to-day basis given the many external pressures that we all face. But if we never see this, what we do has little value to society. If we care about influence, as I hope we do, then adding value to society is the real measure we should pursue. Of course, this isn’t easy to quantify (as are many important things in life, such as romance), and it’s rarely something a single individual can achieve even in a lifetime. But BHAGs (Big Hairy Audacious Goals) are themselves of value, and we should never let them fade far from our minds and actions."

I recently voiced such emphasis on practice impacts in my History and Impact column in SIGSOFT Notes: "Given that new generation of young researchers may tend to put their eye sights on publishing (many) papers in top venues without paying sufficient attention to research impact, it is time for our research community to incentivize impact, including but not limited to impact to practice".

I deeply believe that while embracing diversity and inclusion, our research community should (and will) find appropriate ways to make great progress towards the BHAGs (Big Hairy Audacious Goals) pointed out by David. But as David said in his 2006 ACM Fellow Profile: "We need to work together"!

We all miss you, David!

Announcement of Notkin Fellowship at Notkinfest in February 2013.

Note and acknowledgment. The (last) point on watching out on dichotomies and the quote from David Notkin's last Editorial in February 2013 of ACM TOSEM near the end of this blog post were newly added on April 24, 2016. Such addition was inspired from a conversation with David Rosenblum, the current ACM TOSEM Editor in Chief, to whom I show my deep appreciation.

Coping with issues of sloppiness and carelessness in writing

I appraise and am proud of my students in terms of their skills and capabilities of independently writing the whole drafts of their papers after going through my writing training. In particular, to train the writing of my students, I have iterated with them via marking through their writing and explaining to them why my suggested ways are better. Since my second year of my faculty job, my students have already written the whole drafts for the papers of which they are lead authors (with iterations of my marks and feedback). See some earlier blog posts on writing and my advice portal for relevant advice.

At the same time, I also notice some students still suffer from issues of sloppiness and carelessness in writing. For example, when I go through my pass of writing review on a draft submitted by a student to me, I could still point out many low-level writing issues (which should have been detected and fixed by the student) along with other issues.

I don't expect students to write perfect writing for the first pass; in fact, it has been suggested in the field of technical or general writing that, in the first pass of writing, writers need to focus on the ideas/contents rather than low-level writing issues.

But in later passes of draft reviewing, students need to have eyes equipped with special carefulness to scan through their own writing (including others' writing) to spot out various issues including low-level writing issues (such as those related to typos and grammar), high-level writing style issues (such as those related to bad logic flow, terms used-before-defined, inconsistent usage of the same term throughout the draft), and content issues (such as wrong descriptions of ideas or algorithms).

Below are my initial suggestions to students on how to cope with the issues of sloppiness and carelessness in writing but I welcome your suggestions.

*. You must do spell check on your writing. If you use MS Word, Word has good features of spell check and basic style check. If you use LaTeX, quite some editors support spell check: turn it on! Alternatively or additionally, you could generate a PDF file from your LaTeX files, and then copy and paste the text from the PDF file to an empty MS Word document, and use the spell check and basic style check there (not sure whether basic style check would work since extra line breaks are within the text).

*. For some of the common writing issues of yours that could be turned into rule checking, you should use rule-checking tools such as style-check for checking text in LaTeX.

*. When you go through your writing for reviewing, you should use a pen for marking through a hardcopy of your paper, rather than viewing it on computer screens or revising your draft along the way of reviewing it. Empirical evidence in technical writing or reviewing has shown that marking through hardcopies is more effective for identifying writing issues.

*. When you go through your writing, you should keep those common writing issues of your own in mind (I have an internal guideline for students to write writing-defect recording log, similar to defect recording log in Personal Software Process). For example, if you find that you have subject-verb inconsistency issues commonly in your writing (based on my past marks), you may consider to use a dedicated pass on your writing to simply look for only this issue (without looking for other issues). I know doing so would take time but before you have the capacity of spotting out various kinds of writing issues within a single pass, you would have to do so to identify your common writing issues.

*. You yourself should take time to go through your own writing with high care. Especially for your camera ready versions, please really take extreme care to inspect your own writing. For my pass of both idea and writing review of a camera ready version from a student, I would very likely take 1-3 hours for a single ACM/IEEE-format page. I believe that I may have "special eyes" to catch so many issues in a student's writing in my pass. With my "special eyes", I would still need to take a long time to go through a single page, partly because I spend significant time to ask "why so? is it true?" almost for each single sentence in terms of its content besides those writing issues. For example, if your paper includes a control flow graph drawn for a code snippet, I would redraw it myself on a piece of paper and compare my own graph with the graph included in the paper, without taking for granted that your graph is correct: you should do the same in your own pass of reviewing your draft.

*. You should ask one or more peer students to go through your draft for peer review for writing (of course, before that, you are expected to get peer reviews for your ideas and contents). If you find that, after only one student's peer review, your past drafts still got many marks from me, you should consider to ask more than one peer student to peer-review your future drafts.

*. (Suggestion contributed by Michal Young) Read aloud, preferably to someone who is willing to listen (even if they know nothing of the subject), but alone if necessary. Often our ears catch problems that our eyes silently correct before feeding our brains.

Happy writing!

More suggestions on addressing the issues are welcome and appreciated!

Tips on collecting research papers related to your research topic

*. Tip 1: Type in the keyword of a specific research topic (such as "symbolic execution") or the title of a research paper known to you in
http://scholar.google.com/
Then for relevant papers, you could see a list of papers related to the keyword, typically with those papers with more citations shown earlier.

Then you could browse through the papers citing a specific papers by clicking the "Cited by XXX" for the paper, e.g. "Cited by 922" for an early paper on symbolic execution by King. Then you may browse more recent papers first by clicking "Since 2011", "Since 2010", .... (from a pull down menu shown near the top of the result page) one at a time to browse from newer papers to older papers.

For a specific paper, you could click its "Cited by XXX" to open a new tab or window in your browser to see the list of papers citing the paper so that you don't lose the context of your earlier browsed result page.

*. Tip 2: Similar to Tip 1, but do the search with keywords at
http://academic.research.microsoft.com/
Then click "here" (from "This page shows one keyword best matching your query, you can find other results here.") on the line near the top of the result page to get more matched results. You could navigate to the papers citing a paper by clicking "(citations: XXX)" shown in the end of the paper's title in the result page.

*. Tip 3: Search or browse ACM digital library http://portal.acm.org/ or IEEE Explore library http://ieeexplore.ieee.org/Xplore/ to find papers and their references, as well as the papers citing them. Comparing with Google scholar (http://scholar.google.com/), ACM digital library could more easily allow you to navigate from a paper to papers from the list of references cited by this paper.

*. Tip 4: Type in the keyword of a specific research topic (such as "symbolic execution") or the title of a research paper known to you in
http://www.google.com/
If you find too many irrelevant results, you could add "filetypes:pdf" after your search keyword to return only the PDF files, which are typically the format of research papers. Note that for each research paper collected by Google Scholar as well, "Cited by XXX" is also shown for the entry of the paper in the normal Google result page to allow you to navigate to the papers citing the paper.

*. Tip 5: Browse online proceedings of recent major conference or journal contents in your research topic area from digital libraries (such as ACM digital library http://portal.acm.org/ or IEEE Explore library http://ieeexplore.ieee.org/Xplore/) to find out relevant papers. Or browse online technical programs or accepted papers of major conferences posted on their conference webs before the proceedings are available in digital libraries, and then google specific paper titles since some researchers may post their paper PDF files on their homepages (typically after camera ready deadlines).

If you attend a conference, sometimes the conference organizers may distribute a single PDF (in a USB stick, CD, or online web) including all the research papers in the proceedings. Then you could globally search a keyword such as "symbolic execution" in the single PDF file to find out all locations mentioning this keyword and thus find out relevant papers.

*. Tip 6: Browse the publications webpages of specific researchers or research groups working on your research topic area. You could use a free tool WebMon (http://www.markwell.btinternet.co.uk/webmon/) or watchthatpage (http://www.watchthatpage.com/) to add these webpage URLs there and run the tool frequently (e.g., weekly) to automatically check whether these pages have been updated, and browse only those publications webpages that have been updated since last time you ran the tool.

If you have more tips, please contribute!

Acknowledgment:
Thank Yingfei Xiong for suggesting the watchthatpage online tool.

Time management self checklist

A lot of students have problems in doing work efficiently. One root cause is on bad time management. The following talk video by Randy Pausch could be useful to watch:
http://www.youtube.com/watch?v=oTugjssqOT0

Here are the slides and materials for the talk:
http://www.cs.virginia.edu/~robins/Randy/RandyPauschTimeManagement2007.pdf
http://www.cs.virginia.edu/~robins/Randy/

If you complain about you don't have enough time in getting your work done, do a simple book-keeping on how you spend your time: handling emails, handling text messages, handling IM chats, browsing through or commenting/status-updating at facebook, twittering, watching TV, doing actual real work... Then sum up the portion of time allocated on each type of activities, you could find out some insights and figure out what do to to fix those issues...

Walking through the key points mentioned in Randy's talk, I make the following check list for reflecting what I have done well and what I have done badly. They may be boring to you (if so, stop reading them:) but writing the list down actually helps myself to reflect my time-management habits.

++ I don't have a messy desk (I used to have in my first year of new faculty life)
++ I can find things easily (heavily relying on google but knowing what appropriate search keywords to use, indexed in my mind)
+- I don't miss appointments unless sometimes they are in too early morning (don't arrange your defense in early morning unless you send a reminder to me the night before!).
++ I am normally prepared for my meetings (but the critical factor is my students need to be prepared when meeting with me).
-+ I sometimes am tired/unable to concentrate (but there I often drifted my thinking to some great research ideas)
++ I do planning (more precisely, I am deadline driven)
++ I have a todo list being my "Tasks" in my gmail but there I list only long-term tasks and I treat the emails in my inbox as my todo list)
- I need to do better on "Covey’s four-quadrant TODO": I tend to focus on things due soon (no matter important ones or not)
+- I do touch each piece of email once but I indeed consider my inbox as my TODO list (not seeing too much the negative side of it)
+ I don't call that much so the issues on reducing call duration don't apply to me.
++ I have a comfortable office (not messy and not with a soft comfortable chair)
-- I too much rely on my email inbox for my todo list and I don’t make time enough for important things.
+ I learn to say "No" reasonably well.
- I don't easily find out my creative/thinking time (maybe being late at night). But I don't tend to use such time to do creative thinking. I do creative thinking often when meeting with students; the meeting times may not overlap with my creative/thinking time but I do call students to my office for meetings whenever I like (of course when they are in the lab) rather than arranging fixed time.
- I don't easily find my dead time (maybe morning) so I don't do specific things during it such as scheduling meetings, phone calls, and mundane stuff.
-- I have big problems with interruptions with email "ding" arrivals; I still don't want to turn it off.:( So I rarely have too long blocks of time in devoting to things without email interruptions unless the deadlines for the things are immediately upcoming.
-- I don't tend to cut things short like when chatting with colleagues unplanned at hallway or my office. I do have a desk clock on my desk but the time there is not accurate and I rarely look at it.
-- I don't have a time journal so I cannot analyze it. But I do think about and focus on what things I could do but others couldn't easily (or are not good at). I do often think about how to do things more efficiently.
-- I don't have work-life balance (yet) -- more precisely not much life yet.
-- I am not too much on procrastination but I am a last-minute person (very likely because my todo list is often not short).
+ I am doing fine with delegation such as letting students manage group matters and take group roles, not to say training them how to write papers and carry out research (being maybe a type of delegation of paper writing or research development?). But I am a bit cautious on delegating some tasks to my students unless they get fair recognition/benefits that they deserve.
- I don't have too many meetings with colleagues but I do have frequent meetings with students (where students need to have an agenda beforehand and need to to have a todo list for upcoming period as meeting outcomes).
-- I read frequently my emails over "vacation". Need to stop that!
- I watch TV while working before my laptop for some time at nights. I still don't want to cut off my TV watching time, which is not much each week (I need some time off anyway).
- I don't normally turn money into time (after all, I don't have that much money while still having some spare time). Instead, I may pursue more on turning time into money.:)
+ I normally eat well, sleep well, and exercise ok (but I need to keep it up regularly.. now it time for me to go to gym since I am reaching the end of my blog entry)
+ I normally keep up my promise.

Don't rely on only examples to describe what you have in mind

During my one-on-one meetings with students from time to time, I found that quite some students don't have good skills in describing things clearly (either in spoken or written English). Here I would like to ask students to do a self-check: when you find out that your listener didn't understand what you said, you may tend to say "Let me use an example (to be drawn on the white board) to explain to you."

My advice to students is that don't immediately fall back to use an example to explain what you have in mind (especially during your meetings with your advisor during which you have golden opportunities to improve your skills). If you keep immediately falling back to use an example, you will never be able to explain yourself clearly with your direct description of things. What you need to do is to revisit what you just said and ask your listener on what he or she doesn't understand, and then you will be able to diagnose "bugs" in your initial descriptions, and try to avoid these "bugs" in your future descriptions.

For my weekly one-on-one meetings with students, I demand students to propose new ideas to me and recommend/describe other researchers' papers to me. In this process, students get opportunities to exercise their skills in verbally explaining clearly new things to the listener (i.e., me).

At the same time, students get more opportunities to exercise writing things more clearly by iterating their writing with me over their research development process. See my earlier post on technical writing. In technical writing, one similar pitfall to watch out and avoid is to explain your proposed approach via only examples without direct and clear description of what the approach is. Keep in mind that examples are just sample points in what your approach covers, and often the time readers don't get to know precisely what your approach is by reading only limited sample points covered by your approach.

Don't get me wrong. I in fact encourage students to use examples to help illustrate what they have in mind in their spoken or written English. But students shouldn't rely on ONLY examples to explain things, without being able to explain things directly and clearly up-front.

To help explain things clearly, students should consider to adopt the top-down way. More details can be found in my earlier post on "Advice to Students on Mastering Communication Skills"

In a one-on-one meeting with me, when a student explains an approach in a paper to me, it is better for the student to use the top-down way. The student needs to first explain the problem being addressed by the approach. To help accomplish this goal, besides the direct description of the problem, the student could explain the inputs to the approach and the outputs of the approach, without first getting into details how the approach does it. A common pitfall for students is that students tend to immediately explain what the approach actually does without first giving me any ideas on what problem the approach targets at or what the high level inputs/outputs of the approach are.