The share of a reproduction paper is based on analyses of as well as in insights into current methods and problems—plus the additional certainty that is sold with validating past outcomes.

1. Relevance: Is this paper highly relevant to COLING?
2. Readability/clarity: could be the paper well-structured and well-written?
3. Data/code supply: may be the data/code (as appropriate) open to the study community or perhaps is here a compelling explanation provided why it is not feasible?
4. Analysis: In the event that paper managed to reproduce the link between the previous work, does it obviously set down exactly what must be filled in to do therefore? If it wasn’t in a position to reproduce the outcomes of earlier in the day work, does it plainly recognize just what information ended up being missing/the likely causes?
5. Generalizability: Does the paper exceed replicating the total outcomes from the initial to explore whether or not they may be reproduced an additional environment? Instead, in instances of non-replicability, does the paper talk about the wider implications of the outcome?
6. Informativeness: To exactly exactly exactly what level does the analysis reported when you look at the paper deepen our comprehension of the methodology utilized or perhaps the issue approached? Will the information into the paper assistance professionals along with their selection of technique/resource?
7. Meaningful contrast: along with pinpointing the experimental outcomes being replicated, does the paper motivate why these specific email address details are a essential target for reproduction and just exactly what the long run implications are of the having been reproduced or been discovered to be non-reproducible?
8. General recommendation: there are lots of good submissions contending for slots at COLING 2018; essential will it be to feature this 1? Will people discover great deal by scanning this paper or seeing it presented? Please be decisive—it is much better to vary from other reviewers rather than grade every thing in the centre.

Site paper

Documents in this track provide a language resource that is new. This might be a corpus, but additionally might be an annotation standard, tool, and so forth.

1. Relevance: Is this paper highly relevant to COLING? Will the resource presented likely be of good use to your community?
2. Readability/clarity: Through the real method the paper is written, are you able to inform the way the resource had been produced, how a quality of annotations (if any) ended up being assessed, and exactly why the resource ought to be of great interest?
3. Originality: Does the resource fill a need within the collection that is existing of resources? Keep in mind that originality could be when you look at the range of language/language variety or genre, within the design of this annotation scheme, when you look at the scale of this resource, or nevertheless other parameters.
4. Site quality: what type of quality control ended up being completed? If appropriate, ended up being inter-annotator contract calculated, and when therefore, with appropriate metrics? Otherwise, the other assessment had been conducted, and exactly how acceptable were the outcomes?
5. Site accessibility: could it be simple for researchers to down load or otherwise access the resource to be able to put it to use in their own personal work? As to the level could work according to this resource be provided? answers to add: Yes, We have confirmed
6. Metadata: perform some writers explain whoever language use is captured when you look at the resource and also to exactly exactly what populations results that are experimental regarding the resource could possibly be generalized to? In the event of annotated resources, will be the demographics associated with the annotators also characterized?
7. Meaningful contrast: may be the brand new resource situated with regards to current work with the field, including comparable resources it took motivation from or improves on? Will it be clear what exactly is unique about the resource?
8. General suggestion: there are lots of good submissions contending for slots at COLING 2018; essential will it be to feature that one? Will people discover great deal by looking over this paper or seeing it presented? Please be decisive—it is much better to vary from other reviewers rather than grade every thing at the center.

Position paper

A situation paper presents a challenge to main-stream reasoning or even a futuristic vision that is new. It may start an area that is new unique technology, propose changes in existing research, or offer a fresh group of ground guidelines.

1. Relevance: Is it paper highly relevant to COLING?
2. Readability/clarity: Is it clear just exactly what the career is the fact that paper is arguing for? Will be the arguments because of it laid call at an understandable means?
3. Soundness: Are the arguments presented when you look at the paper appropriate and coherent? Could be the vision well-defined, with success requirements? (Note: it ought to be feasible to give a higher rating right here even though you don’t concur with the position taken by the authors)
4. Imagination: How novel or bold may be the place used the paper? Does it express well-thought through and imaginative ground that is new?
5. Range: How scope that is much new scientific studies are opened by this paper? Exactly just What effect could it have on current areas and concerns?
6. Significant contrast: could be the paper well-situated pertaining to past work, both place documents (taking exactly the same or opposing side for a passing fancy or comparable dilemmas) and appropriate theoretical or work that is experimental?
7. Substance: Does the paper have sufficient substance for a full-length paper? May be the problem adequately crucial? Will be the arguments adequately varied and thoughtful?
8. General suggestion: there are numerous submissions that are good for slots at COLING 2018; how important could it be to feature this 1? Please be decisive—it is much better to vary from other reviewers rather than grade every thing at the center.

9. A study paper provides an organized summary of the literature up to now on a topic that is specific helps the reader understand the kinds of concerns being expected about the subject, the many approaches which have been used, the way they relate genuinely to one another, and exactly exactly what further research areas they open. A conference-length study paper should sufficiently be about a concentrated subject that it could do that successfully with into the page limits.

   1. Relevance: could be the paper highly relevant to COLING?
   2. Readability/clarity: could be the paper generally speaking an easy task to follow and well organized?
   3. Organization: Does the paper arrange the appropriate literary works in a narrative and determine typical strands of inquiry?
   4. Scope: Does the paper recognize a fairly concentrated area to review?
   5. Thoroughness: because of the location identified to survey, does the paper address most of the relevant literary works? May be the literary works evaluated represented accurately?
   6. Outlook: Does the paper recognize areas for future work and/or point out what clearly isn’t yet handled inside the literary works surveyed?
   7. Context: Does the paper situate current research appropriately within its historic context? (We don’t expect papers to begin with Pa?ini resume help , yet during the same time one thing that just cites work from 2017 probably does not capture just exactly how current work pertains to greater photo.)