Paper Writing and Research Strategies

[Copied from the net in 2012. Source unknown]

Introduction

Here is a brief reminder of how to write papers and what kinds of papers to write. Please also see the other materials on our web site on paper structure.

What matters most about a paper is what the reader gets out of it, not how much work you put into it. That is, after reading your paper, the reader should feel that he knows something new, has a better insight into something, or has a better idea of how to do something.

Each paper you write should stick to one main idea, and that idea should be clear enough that you can formulate it in a single sentence. If you can't tell someone else what the main contribution of your paper is in a single sentence, you haven't worked it out enough.

Writing papers doesn't need to be all that hard because papers fall into a small number of common categories. Most of them involve some problem that needs to be solved or some observation that needs to be explained; for the paper to be accepted, it's important that this problem or observation is interesting in the first place.

Solving problems that aren’t interesting to reviewers won't get you published. It is therefore important that you keep track of the literature and figure out what topics are likely going to be of interest to reviewers.

In writing your paper, keep in mind that many reviewers are graduate students like yourself. Think about how you approached your last few paper reviews. How well did you yourself know the literature? Did you give the authors the benefit of the doubt when you didn’t understand something? Did you ask for more experiments?

Paper Types

Now let's look at common different paper types. I have given these papers short, catchy names because you should be able to identify clearly what kind of paper you are writing.

"I reviewed the literature about solutions to problem X." (Literature Review)

A literature review should pick a well-defined research area, something that is neither too large nor too small. An area might be something in which there are 100-200 research papers total that you need to read.

A literature review is a little bit like an experimental paper. It usually starts with a question, then looks at data, and then draws its conclusions. The question it asks is usually of the form “what are the unsolved problems” or “where are there areas for improvement”. In the conclusions, you should answer these questions. They form the basis for further research.

In fact, when you start on your thesis, you need to do a good literature review anyway, and you need to incorporate it into your thesis. If you can get this kind of review published, that's really good. However, publishable review papers are a lot of work and usually require upwards of a hundred original references to be organized and analyzed. Nevertheless, you should at least aim for a group seminar on your chosen topic.

"I looked at the best ways of solving problem X, benchmarked, and compared them." (Benchmark Paper)

A benchmarking paper asks the question: “given these standard methods, how do they perform under different conditions?” It isn’t necessarily for finding “the best” of the methods (usually, there is no uniformly best method), but for identifying tradeoffs between different methods.

A significant component of a benchmarking paper is the datasets. If there are standard datasets and standard algorithms, and they simply haven’t been benchmarked against each other, then it’s easy to do that. Otherwise, you may have to create your own benchmark dataset. This can be a lot of work, but it can also be rewarding: if your dataset is novel and interesting, other people will use it and cite your benchmarks.

This is often a good paper to write at the start of your thesis work: you gain experience with the problem you are trying to solve, you implement algorithms that you need to implement as control experiments later, and you collect and/or create databases you can evaluate on. Papers like this have the potential of getting highly cited. They are also conceptually fairly simple and you are almost guaranteed to get a useful result. However, they are a lot of work to do well.

"I came up with a new/better/interesting way for solving problem X." (Method Paper)

This is perhaps the most common paper people write; it's usually the kind of paper that takes an engineering view of the world. You usually need to show experimentally that your method is better than existing methods, at least under some circumstances or on some kinds of data (rarely, a complexity analysis may be sufficient). Here, "better" may mean one of several things:

It yields lower error rates.
It is faster.
It is simpler to implement.

Additional useful distinguishing features can be the following, but they are rarely sufficient by themselves:

It is easier to understand.
It is theoretically better justified.
It is available in open source form.

The required experiments make these kinds of papers similar to benchmarking papers, but there are some differences. While the goal of a benchmarking paper is to gain insights into existing methods (data sets represent a wide range of conditions, authors are supposed to have no preferences), the goal of the experiments of a "new method" paper is to show that the method is actually better. But that also means that the reviewers will take a more adversarial view of your paper, meaning, they will try to poke holes into your arguments. Key questions they will ask are:

Is the problem you're solving interesting at all?
Are the conditions under which your method is better than other methods too restrictive?
Are the advantages of your method sufficient large?
Does your method have so many parameters that it is hard to implement?
Did you tune your method to the specific dataset, i.e. do the results generalize?
Is the comparison fair? I.e., did you unfairly bias the data or preprocessing in favor of your method?

Generally, when you write up your paper, you need to think about the list of objections that reviewers might make, and you need to have answers to them.

"I performed observations/measurements on X" (Research Paper)

This is the more standard scientific research paper; you don’t create something and you don’t invent a new method for doing something, instead you observe and interpret existing systems. In some way, it is like a benchmark paper.

These papers are, on balance, easier to write than method papers, because it is usually easier to get publishable results out of observations. There are a lot of opportunities for these kinds of papers in computer vision, pattern recognition, social network analysis, etc.

"I have a new mathematical model/theory for observation X." (Model Paper)

Mathematical modeling or theory is a very different approach from experimental approaches, but the basic rules still apply: your result needs to be novel, interesting, relevant, and useful somehow. Generally, mathematical models or theories should make predictions. Some of those predictions, you can compare against existing data, others may suggest new experiments.

"I found a new/different interpretation of observation X." (Response Paper)

These papers are generally responses to the literature, where you read a paper and decide that its conclusions, model, or theory aren't quite right. These papers tend to be shorter and fairly easy to write (because someone else has already done all the background research). They may involve experiments illustrating the point you are trying to make. They are often good papers to write, but they are difficult to plan for because, of course, you don't know when you'll come across a paper you can respond to.

"A new framework/'theory' for X." (Framework Paper)

These papers often don't involve any experiments, theory, or serious review of the literature. Instead, they provide a view of a field, observations, or data. If you hit the "Zeitgeist" just right, these kinds of papers can be spectacularly successful and highly cited, in particular if you have already made a name for yourself. However, most of them end up just languishing at a conference.

Preparation for Ph.D.

Although you may be enrolled in a Ph.D. program, you may first need to learn how to work with the literature and how to conduct experimental research. At some other universities, this phase is delimited from the actual start of the Ph.D. program by the submission and presentation of a formal Ph.D. proposal. We don’t have that requirement at our university, but the two phases really still exist.

Seminars and Projects. Before you can write reviews and papers, perform benchmarks, and perform experiments, you need to learn these things. As part of our graduate program, you should learn reviewing as part of your seminar course, and benchmarking and/or experimenting is commonly a project topic. The seminar and project are the main time during which someone will look at your work and give you detailed feedback. They can do this because it’s area and research they understand well. Later, your advisor will be able to give you much less input and help on your Ph.D. project because they will know fewer of the details of your Ph.D. work.
Collaborate. Another way of gaining experience with reviews and research work is to collaborate on research and papers with others. This may include detailed proof-reading of their papers, contributing to their writing, taking over experimental work from them using their methods, or applying special skills you have to their problems. When collaborating, think not only about the obvious benefit (like maybe another publication), but also what you can learn from the collaboration.
Small Projects. If you need more experience or still don’t know what exactly you want to do for your Ph.D., in addition to collaborations, you can also take on smaller projects. These are projects similar to what your advisor might give you in a project course, but they are not formal parts of your course requirements and you don’t get ECTS points for them. You should still aim for getting publications out of them. There’s usually a long list of such small projects to choose from.

The previous steps are preparatory for your Ph.D. You really shouldn’t spend too much time on them because you don’t want to spend too much time on your Ph.D. At some point, when you have a reasonable handle on how to work with the literature, how to run experiments, and what you want to do, you need to pick a Ph.D. topic. That’s when your Ph.D. studies usually start.

Ph.D. Research

Actual Ph.D. research, like all research, really means three steps:

familiarizing yourself with the literature
getting familiar with the research methods and tools
conducting actual research

Note that you don’t usually start this process from zero; presumably, you have already done a little bit of work in the area--maybe as part of collaborations or projects--simply to determine that you actually are interested in it.

Each of these steps should produce some output, in the form of a review, a benchmark, and research contributions.

Review/Proposal. Start by writing a review paper of the field your thesis is in. You should plan on presenting this at a seminar. You can actually do this for seminar credit. Sometimes, you can publish this as a review article, although that usually involves a lot more work. This should also give you a good idea of what the open problems are and how success is measured in your area. A good review with meaningful conclusions is basically equivalent to a Ph.D. proposal.
Benchmark. Next, implement algorithms, collect datasets, and perform benchmarks. You can do this as part of a project course. If your results warrant it, this is fairly easy to publish at least at a conference. This will give you better understanding of the data and get you set up for your own experiments.
Research Contributions. There are actually two major kinds of research contributions: new methods or observational studies. New Methods. Then you start implementing your own ideas and methods, benchmark them against the standard methods, and start writing up the results, resulting in several publications. Observational Studies. You can also make part or all of your thesis observational studies. You should consider this possibility seriously because it can be easier than “new methods” work. However, while for engineering and methods research, the primary deliverable is something that works better, for observational studies, scientific reasoning and understanding is more of a challenge. (The “benchmarks” for observational studies are replication of other people’s observations.)

This division and three step approach are useful, and it’s the way researchers generally approach starting research in a new field. But it is not formally required and many students end up working on a lot of different projects and ideas and then at some point putting everything together into a thesis. Each of project implicitly still requires a literature review, benchmarks, and new contributions, but they happen in different orders and perhaps aren’t made explicit in the form of publications. There is nothing wrong with that if that works for you and if you get a good thesis out of it. However, if you are limited in the amount of time you have for your Ph.D., if you feel lost or are concerned about completing your Ph.D., then you should consider picking a single topic and focusing on the three steps for research.

Scholarships and Ph.D. Proposals

Some scholarship organizations will require you to submit a Ph.D. proposal in order to get the scholarship. That is a useful starting point, but it is rarely as detailed as a true Ph.D. proposal and literature review should be (although the more detailed you make it, the less work and uncertainty you will have later). Furthermore, after getting the scholarship, you may still need to spend some time with seminars, projects, etc. to get more familiar with research methodology.

This means that you shouldn’t assume that your Ph.D. proposal is finished just because you submitted something with the scholarship application. On the other hand, scholarship agencies are generally fairly flexible about proposals and you can change them later as you learn more about a research area.

Problems You Will Encounter

Of course, in real life, things don't necessarily work as smoothly. Common problems are:

The literature seems to confusing and you can't make a coherent review or presentation out of it. This means that you really don't understand the field well enough yet to even start your research work. When you get serious about your Ph.D. thesis (or research in general), you should have a good answer to the question of "what are the open questions in your area", "what are promising ways of attacking them", and "what are the most common ways in which people have tried to solve these problems before and why have they failed." As part of this, you should become more of an expert on your chosen area than your Ph.D. advisor.
The benchmarking doesn't yield meaningful results. One of the most important skills you need to learn as part of your Ph.D. is how to conduct meaningful experiments. This means asking the right questions, choosing the right datasets, performing the right measurements, and presenting the results in a meaningful and compelling way. Your Ph.D. advisor can help you and give you feedback, although it will largely be of the kind of feedback you would get from a reviewer: he will point out problems and limitations with your approach. Fixing those problems is your job. Your advisor may be able to give you a little input on that, but benchmarks and data sets are very hands-on problems, so that kind of input is limited.
Your new method isn't working better than the old one. Perhaps your idea just isn't a good one, or maybe the dataset you are testing it on isn't good for showcasing its advantages. This is the kind of thing your advisor can help most with: he can provide ideas for other, new approaches and suggest other ways of evaluating your method. However, you shouldn't just abandon an approach if it isn't working; you probably invested a lot of time in trying to make it work and you should try to learn from the experience and understand why it isn't working. If you just keep trying one idea after another and never figure out why they are failing, you are going to make very slow progress. On the other hand, if you learn from your failures, even if you don't get a publication out of it, you are still making progress.

Other Strategies

Here are some other strategies you need to keep in mind:

Talk with other people about your work. It doesn't matter whether they are experts in your area or not. One of the most important parts about talking to other people is that you learn to explain your work better. If you are lucky, they will give you useful feedeback.
When you talk to your advisor, have clear and concise explanations of what you have done, and have specific questions you ask. If you don't, your advisor simply can't help you very much.
In addition to your primary work, do some other things: collaborate with others on projects (contributing the expertise you have obtained from your primary research), and also take on some small and easy additional side-projects.
Pick your research area so that there are lots of opportunities for collaboration and additional projects.

Learning and Life

Pages

Sunday, January 26, 2020

Different Types of Research Contributions and Challenges in Defense