START: Submission Tracking And Review Toolset

Copyright © 1998-2000 Rich Gerber, Jeff Hollingsworth and Adam Porter. All Rights Reserved

This software is released on a controlled basis. Permission for using START is granted for a single conference or workshop only, unless explicitly obtained for multiple events. The START logo appears on auto-generated forms, reports and HTML pages; this logo may not be removed or altered, unless prior consent is obtained from the authors. The system is released AS IS, without support, and may be modified as desired. Bug reports are solicited, as are suggestions for upgraded functionality. However, we  cannot guarantee that the repairs or upgrades will be made. By installing this software, you hereby agree to these terms, and to all other provisions in the general usage license below.

What is START?

START is a Web-based conference management system. It was created to help streamline some of the tedious duties associated with being a conference chair, namely:


The START software cannot review the papers for you - as usual, your referees have to read the papers, and evaluate them. However, it will substantially reduce the burden involved in handling papers and reviews. You can concentrate more on the technical content in your submitted papers - and leave most of the secretarial drudgery to your computer.

Terminology

The following terms are used throughout this document.

Hardware and Software

The following infrastructure is NECESSARY to run START:

The following is tool is OPTIONAL:

Setting up Software

  1. You have now expanded the conference software tar file. This README file currently resides at an implicit "root" directory, that is, the top-level directory for your conference documents. If you are not happy with the location, move your directory now; its pathname will be used to configure many other files. This directory needs to be accessible from your Unix account, in a directory which you own. Aside from this single constraint, it can go anywhere on your filesystem.

    2. In this document, we refer to your conference's top-level directory as $STARTroot. Unless otherwise specified, we assume you are working in this directory - hence, most other directory references below are considered relative to $STARTroot.

  2. If you do not have already have a binary image of Apache (version 1.1 or higher), you will have to build one. Get it from Apache, and build it in a separate directory. Here's the URL to get your HTTP server:

    3. http://www.apache.org

    Run configure as you normally would, and the Makefile will get built to accommodate START's directories. Then run "make," and you'll get an httpd image to run for your conference. Finally, put the httpd program where you normally store your binaries.

    After you have your httpd binary, the sources used to build it aren't needed for START - including the conf and log files. START generates conf and log files for you, tailored to your conference parameters.

  3. Now go to the $STARTroot directory, and run the program called setup, which customizes files for your conference. As input, you are requested to enter your conference's name, its year, the acronym, your email address, and other information. Then, setup automatically configures the START software, as follows:



  4. At any time, you can change the conference-specific information entered on the setup command line. To do this, you should first shut down your web server (since setup alters its conf files). Then just re-run setup, your changes will be reflected throughout the system. If desired, you can hand-edit most of the fields in parameters.pl. Note that setup will not override these edits.


  5. You will find a template which automatically-generates your submission form in


    6. $STARTroot/templates/submitPaper/submit-template.html.


  6. Note that setup fills in your conference-specific data, including the hostname/port for your web server, and puts the result in www/submit.html. This file will eventually be copied to your public webserver space.


  7. Since you may be running setup multiple times, we suggest you carry out additional editing on the template - and not on submit.html. Then you can just use setup to re-generate the real form.


  8. A keyword check-off list is also initialized in the setup script. Note that the keyword check-off list is not strictly required to run START. Indeed, it has no semantic interpretation inside the database; if desired, you can eliminate it from submit-template.html, and it won't show up on your submission page. But take it from us - this would not be a wise move. To the contrary, we'd advise including any topic tangentially related to your conference area. One reason is obvious: the keywords can help reviewers select the papers they wish to read. Another reason is a bit less obvious: a long list of topics serves as a sort of advertising tool, since researchers in the identified areas will feel "welcome" at your conference, and perhaps submit their papers to it. Finally, the more keywords you include, the more likely it is that your submission page will get listed in search engine queries.


  9. Setup also asks whether you want PS to PDF conversion. To do this, you'll need some additional software (see below). If you check "NO", then all postscript files will be moved "as is" to the program committee directory, and all PDF files will be treated likewise. If you don't wish to hassle with conversion, this is probably the more pain-free way of running things. However, it does have its liabilities, which we discuss below.


  10. To launch your server for the first time, use the script tools/starthttpd. (Note that the script killhttpd does the opposite - it halts your server.) To verify that it works, bring up the conference submission form, and submit it - without filling in any of the fields. You should receive a START error report as a response. If this does not happen, carefully review the steps enumerated above - chances are you made a small error in configuring the software.


  11. Now configure your cron daemon, to ensure that your httpd always stays up and running, even after a system crash. This is trivial on Solaris (via crontab). Most Unix variants have similarly nice interfaces to cron (check your man pages), i.e., via


    12. crontab -e

    This launches your default EDITOR, with a file containing the current cron schedule for your username. Append a line at the end of the file, which will look like this: 

     0,15,30,45 * * * * /usr/jim/bin/httpd -f /usr/jim/myconf/httpd/conf/httpd.conf 2 > /dev/null

    These items have the following meaning:



  12. Most of START's software is written in the perl language. Scripts handling paper submissions and reviews are spawned by the web server; other scripts are invoked by the conference manager; others are used as subroutines. The organization of the START package is as follows:

    www/cgi-bin

    Scripts for author-triggered actions.

    www/PC

    Scripts for reviewer-triggered actions.

    tools

    Subroutines used by the PC chair

    lib

    Shared library files.

    If you need to make radical changes to your process, these are the places you'll need to look for the files to change. Unless you wish to change the submission process in a major way, we would advise against this. The important fields get automatically updated via the variables in parameters.pl.


  13. When papers get submitted, the papers and their log files are placed in the directory papers, and the papers themselves are copied into papers/process for pre-processing. The idea is that no files should ever be deleted from your incoming directory - it holds the record of everything which came in.


    14. Before proceeding, you should test the server's functionality. Fire up your web browser once again, and open the submit.html file - which still resides in the private www directory. Submit a few bogus papers to your conference, and see what happens. Check the contents of the abovementioned directories, to make sure everything works.


  14. As papers come in,some post-processing is often needed. We handle this in steps, which should hopefully minimize your problems in handling poorly typeset papers. When a paper gets submitted, it is copied to the directory papers/process. Then it gets transferred to specific locations depending on file type (ps or PDF). For example, for paper number ID, the following actions are taken based on the file's MIME extent.




    15. At the end of the algorithm, files should be in either papers/in if they're postscript or papers/out if they're PDF - and anything remaining in papers/process should be checked manually.

    NOTE: The following section on document conversion explains how postscript files are also moved to papers/out. If you you're only using postscript submissions, you need not read the next section. However, using this alternative is somewhat risky, since it rests on a faulty assumption - that all fonts are printable by everyone, and that no postprocessing is necessary.

  15. When you are ready, copy submit.html to your external web server, for public access, and link your CFP to it. At that point, the system will be ready to accept paper submissions, and the submission form will now be exposed to the entire world.


  16. As the conference manager, chances are you may be running the START scripts many times - and you may wish to do this from any directory. To get this functionality, you should make two edits to your Unix startup files. Assuming that $STARTroot has the value /usr/jim/myconf, you would insert the following line in .cshrc file, or wherever you keep environmental settings:


    17. setenv STARTroot '/usr/jim/myconf'

    Also, you'd add the following absolute pathname to your $path variable (probably also in .cshrc):

    /usr/jim/myconf/tools

    These settings are optional, but they will save you time.

Conversion to PDF [optional]

In theory, one could handle all electronic submissions in postscript, without using PDF, a method that some conferences use by imposing certain formatting constraints on authors. In practice, this isn't so easy. There are many flavors of postscript around, and some typesetting tools use non-standard subsetting - an enormous problem for generic Type1/Type2 printers. Hence, we recommend converting all postscript papers to PDF; then, you can allow your PC to access papers via either PDF or postscript. There are several ways to generate PDF files:



In the sequel, we'll assume you're using the Distiller as your PDF conversion tool, and that it's running on a Mac or Windows workstation. If you're running the Unix version, much of this information is similar; moreover, many of these instructions can be used for similar tools.

If you are running the Distiller on a NT/Win95/Mac workstation, you'll need read/write accessibility to the files from your Unix machine. This is easy if (i) your Unix system IS also your Win/NT system— e.g., you run a Unix server process on NT or MacOS; or alternatively, (ii) you have NFS or Samba interfaces between your Win95/NT/Mac machine and your Unix system.

  1. Launch the distiller, and click on the "Watched Folders" option. A "watched folder" is, in the Acrobat lexicon, the parent of the directories where the postscript and PDF papers are processed.


  2. Set up the following directory as a "watched folder":

    3. $STARTroot/papers

    Then, the Distiller uses two sub-directories for its work:


    If these directories don't exist, the Distiller creates them for you.

  3. Upon arrival, postscript files get transferred into papers/in. If the Distiller is active, it will automatically plow through each one. The resulting PDF (and the postscript originals) are archived in papers/out.


  4. If the distiller cannot handle a postscript file, it will remain in the papers/in directory - along with an error report. These are the papers that will need personal attention - perhaps conversion via ps2pdf or the equivalent.


  5. Regardless of how clearly you state your submission guidelines, there are always a few clowns who will seek out your street address, and then send their papers via surface mail. You may wish to throw them out. However, note that Acrobat has a scan-in function, which will convert scanned bitmaps into PDF. Hence, you may consider just "submitting" these papers yourself - which will then automatically put them on-line the other papers.

Setting up the PC Directory

  1. The directory www/PC is used for password-protected PC access, and the PC home page is www/PC/index.html, which is the page PC members will see when they login to your server. However, like the submit page, we have also provided a sample PC index page for you, in


    2. templates/webPages/PC-template.html


  2. Chances are this template will need editing for your specific conference; however, again we suggest you edit the template, and not the generated HTML. Then re-run setup, and it'll get copied to the right place.


  3. Generate the lists of papers. If you edited .cshrc as suggested above, you can just type genPaperLists from anywhere - otherwise, go into the tools directory, and run the script:


    4. genPaperLists


    This script carries out the following actions:



  4. Now you can assign usernames/passwords for the PC members. The script addreviewer can be used for this, e.g., if you run


    5. addReviewer joe crazyfish

    It causes a username joe to be added, with password crazyfish. Now your PC member named Joe can logon with his password. The password information is stored in

    httpd/conf/passwords.


  5. Test the password feature. Assume your IP address is fish.idiocy.com, that your webserver is configured for port 8080, and that $STARTroot = /usr/jim/myconf. Try opening the following URL:


    6. fish.idiocy.com:8080/PC


  6. You should get a login/password box, which should allow you to login as joe (or whatever). Also, make sure that you can't login using an unregistered username, and/or password. This is important!


  7. After setting up your PC directory, you can send mail to each PC member with his/her username/password, along with information on how the reviewing protocols will work (see below)

Assigning Papers to Reviewers

  1. The conference software is currently set up to accept PC reviewing preference forms (which also indicate their conflicts of interest). The form - paperForm.html - is generated by the above mentioned scripts, and need only get linked to your index page.  (If preferred, you can do all the assignments yourself - and in this case, just don't include a link to the form on the home page, and set $BidPeriodDone to 1.)


  2. After all preference forms are in, the next job is to actually assign papers to reviewers. The database uses the following file to keep track of this:
    3. reviewData/assignments

    While we do this automatically, it's a good idea to check out the structure of this file, since you may wish to adjust it to balance the strengths of your PC. Each line consists of a PC member's username, and paper id - denoting that the PC member is assigned that paper to review.

    This file is used to by the review module, to restrict access into the database - i.e., to ensure that a paper will only get reviewed by assigned reviewers. So, please head the following warning:

    IMPORTANT: If there are any sample assignments in your file, delete them before proceeding further.

  3. The automatic assignments are generated by the the the script


    4. assignReviewers

    This script takes into account everyone's preferences, excluding their conflicts, and just assigns papers on a round-robin basis. There are a few items to note here:

    You can easily hand-edit the assignments file, to give papers to whomever you wish. Then, in the round-robin scheme, reviewers with pre-assigned papers will wait their turn, until the other PC members have "caught up."

  4. The algorithm produces an assignment report as standard standard output, which you can redirect to a file if desired. (It can be large for a big conference.) The script also produces a "tentative assignments" file, stored in:


    5. reviewData/assignments.out

    Inspect the assingment report; if you're happy with the allocation, just execute:

    mv reviewData/assignments.out reviewData/assignments


  5. Go through a few trial runs until you're satisfied. If your conference has a lot of submissions, you'll probably end up fine tuning the process a bit - either by editing the assignment file, or altering the preference files. The first time this was used - in a conference with about 200 submission, and 25 PC members - tuning the assignments took about 3 hours or so.

The Review Phase

  1. At this point, reviewing can work automatically, without any intervention by you. All you need to do is to include the following line on your (template) PC home page:


    2. <a href="get-assigned.cgi"> Your Assignments</a>: A customized list of papers (with review forms) assigned to you.</a>

    See the sample pages for reference. The script get-assigned.cgi customizes a "to do" list for each PC member, showing the papers allocated to him/her, with links to review forms for those not done yet. The to-do list also allows editing previously written reviews. In fact, the same PC member can review a paper hundreds of times, with  the last submitted review  always considered the "current one."


  2. PC members really like this feature: it allows them to make progress on their assignments, but nobody's committed to their opinions until the start of the vetting period. Even after the paper decisions are made, you'll find people still using this feature - polishing reviews to get sent to authors. For this reason, we've omitted the typical "confidential comments" part of the review form. If a reviewer wants to hold some portion of the comments in confidence, after the decisions are made, he/she can edit that portion out.


  3. Review forms are produced automatically. Whenever a reviewer needs a form, it gets created and uploaded on demand, with most of the bookkeeping info filled in automatically, by the START tools.


  4. Reviews are submitted using the Web form-upload protocol. Each review is given a unique ID, and is placed in the directory reviewData.


  5. Note that these reviews should NOT be hand edited by you; nor should they get moved elsewhere. They are required for online updating/editing, and for generating final reports.


  6. You can keep tabs on PC productivity by running the script:


    7. reviewStatus

    which lists the number of reviews done/outstanding for each PC member. 

  7. At some point in the review process, you may have a "vetting process," during which a reviewer of paper X will be allowed to see all other reviews for paper X. This is essential for on-line PC meetings, and it is supported by START.   To enable this feature, go to parameters.pl, and set $ViewAll to 1. The result: when PC members check their assignments, they will now have this option.

Report Generation

At any point during or after the review process, you can generate a set of reports summarizing the results. There are two programs for this, both of which are in the tools directory.

  1. genReports: This is a multi-purpose report program; it does a lot of work, so don't get scared if it takes a long time to finish, e.g., 20 seconds for about 700 reviews, on a Sparc Ultra. The main input you need for this program is the following file:


    2. reports/blockList

    Due the sensitivity of the review information, genReport will not run at all unless such a file is present. It contains a line-by-line list of papers written by PC members, which genReports uses it to produce a "sanitized" comprehensive report for your committee, which can be handed out at a meeting.

    You'll find a sample blockList in the reports directory - delete the lines, and fill in the lines your wish to block. Using this file (and the review log files), genReport produces the following information:



  2. extractScores: This tool is more basic - it simply puts reviewer scores into colon-separated records; all info is included except comments. It is suitable for importing the raw review records into Excel, Access, or other common statistical tools.

    3. NOTE: None of this report information is published on-line. Don't worry about its security - it is private, accessible only by you. If you want to link it to your PC web page, you must copy it to www/PC yourself, and put the appropriate hyperlinks into your PC home page.

Author Notification

Finally, you've accepted your papers for the program.  Now it's time to wrap things up.

  1. Enter the PC results into 2 files: 




    2.              paperID  ShepherdFirstName  ShepherdLastName ShepherdEmail


    This list is used to have the authors and shepherds contact each other. Make sure you edit this list correctly,  or just leave it blank if you don't have any shepherding.


  2. Edit the letter templates accept.txt, reject.txt, and shepherd.txt, in templates/letters.  (Actually, you may not need to, since the letters are fairly generic.) The following variables can be used in the letters; see the templates for details.


    3. $authorContact

    author name

    $title

    paper title

    $paperID

    paper number

    $shepherdFirstName

    first name of shepherd, if any

    $shepherdLastName

    last name of shepherd, if any

    $shepherdEmail

    contact address for shepherd, if any

    $PC_CHAIR

    PC Chair's name, as signature

    $ACRONYM_YR

    Acronym/yr of conference

  3. Run genLetters, and the letters are produced, with comments and scores appended. For safety sake, the letters are not sent - rather, they are put in reports/Letters as files. Check them, and see if they came out OK. If so, do the following: go to the directory reports, where you'll find a 3-line script to mail them all out. What's done is done.


  4. Finally, run genAccept, which produces an HTML list of accepted papers for external publication. It goes in reports/accepted.html, as well reports/Abstracts, a directory containing all the paper abstracts. Copy both these to your External web directory, and then link them to the conference's public home page - at which point the world can see it.


  5. Run genAuthorList, which produces the formatted author contact information for your conference's publisher (or for you). This information is placed in reports/authorList.txt, and you can then send it to IEEE/ACM, etc.


  6. Get out of your chair, stand up, and scream. You're done.

Copyright © 1998-2000. All Rights Reserved

We provide the Submission Tracking And Review Toolset (below described as "START") on an AS IS basis, and do not warrant its validity or performance. We reserve the right to update, modify, or discontinue this software at any time. We shall have no obligation to supply such updates or modifications or any other form of support to you.

This license is for personal use. For such uses, there is no charge. We define "personal use" to mean you may used the software within your organization, for one conference. You may not re-distribute START or parts of START, in any form source or binary (including derivatives), electronic or otherwise, to any other organization or entity without our permission. All warranties, including without limitation, any warranty of merchantability or fitness for a particular purpose, are hereby excluded.

By your use of START, you understand and agree that we (or any other person or entity with proprietary rights in START) are under no obligation to provide either maintenance services, update services, notices of latent defects, or correction of defects for START. Even if advised of the possibility of such damages, under no circumstances shall we (or any other person or entity with proprietary rights in the software licensed hereunder) be liable to you or any third party for direct, indirect, or consequential damages of any character regardless of type of action, including, without limitation, loss of profits, loss of use, loss of good will, or computer failure or malfunction. You agree to indemnify us (and any other person or entity with proprietary rights in the software licensed hereunder) for any and all liability it may incur to third parties resulting from your use of START.

This software is released on a controlled basis. Permission for using START is granted for a single conference or workshop only, unless explicitly obtained for multiple events. The START logo appears on auto-generated forms, reports and HTML pages; this logo may not be removed or altered, unless prior consent is obtained from the authors. The system is released AS IS, without support, and may be modified as desired. Bug reports are solicited, as are suggestions for upgraded functionality. However, we cannot guarantee that the repairs or upgrades will be made.

Server START Conference Manager
Update Time 14 Jan 2000 at 12:00:00
Maintainer Rich Gerber.
Start Conference Manager
Conference Manager