I hope that this will be the start of a continuing discussion. I have been thinking about the comments and questions raised about improving the reviews. One step behind that are the suggestions that we really need more and better submissions.
I think we all can agree that the reviews could use some improvement. Making sure that everyone knows what we are looking for, how to do a review, how to report it and so forth seem like obvious steps to take.
At the same time, I think improving the reviews is a bit like improving testing and hoping that the quality of code will also improve. As we all know, you need to start earlier in the process to improve the quality of code. In the same way, I would like to suggest that we need to start earlier in the process to improve the quality and number of submissions.
For example, at the very beginning of the research process, we can help by providing a research agenda — what kind of research are we looking for? Along with this, we might provide help with developing research problems or research questions. We might also support the independent development of research through a well-organized and easily accessed literature database.
So we might try to improve:
1. Initiation: research agenda, research questions, literature database
2. Proposal: guidelines/checklists for research methods, proposal reviews, mentor to talk to about conducting research?
3. Writing the article: early feedback, English help, good/bad models, how-to guides
4. Submission: checklist/guidelines, formatting help, time for revision, English help
5. Acceptance: help with presentations/posters
There’s a more detailed discussion below, but the basic idea is to look at the research process and try to improve it. I think there are several areas we could help with!
What do you think? Can we improve the CSEE&T process? Which steps should we start with? Where would you like to contribute?
Personally, I think we could start with three key areas — a CSEE&T research agenda, CSEE&T research mentors, and an expanded program of awards and recognitions. But I look forward to hearing from you.
Possible Areas for Improvements
Let’s consider. If we want to improve or expedite the research process which leads to improved submissions, we can provide help at several points.
1. At the initiation of the research process. Themes, research agenda, suggestions as to what kinds of research are needed. Let people know what we are interested in seeing research on.
2. Along with that, providing guidance to the available literature. What kind of theories, what kinds of research have already been done, where are the holes (which overlaps with the research agenda, of course), what kind of hypotheses need testing? The easiest thing here is simply an annotated guide to the “good” literature, perhaps a wiki with summaries and such available to help people.
3. Feedback on ideas and proposals for research. In particular, helping students to develop a good research proposal and approach before they perform their research. Perhaps allow submission and review of proposals, with feedback to help them?
4. Checklists, guidelines, models of good research approaches? I doubt that we want to get into the business of mentoring people doing their research, but we certainly can provide support. Ah! We might also help with analysis? Sometimes students need a little help sorting out the statistics and such.
5. Guidelines, checklists, and models for writing a report. For models, especially, we need to provide a recommended set of “good” papers, probably based on the “best” papers from various conferences. We might also provide some models of outlines, graphics, and other steps in writing a paper, presentation, etc.
6. Mentoring and feedback for papers. Especially for English as a second language people, some help in organizing and turning ideas into readable English is really needed. This needs to start well before the ordinary review cycle, so that they have something which can be usefully submitted.
7. Submission aids. Checklist and guide to the submission process. Information about what the reviewers are looking for. Formatting help. Providing time for revision nad English help.
8. Improving the review process. Provide checklists or rubrics, training, maybe even some practice sessions to help reviewers. Some models of “good” reviews and comments, and maybe some examples of “not very helpful” reviews and comments also could help. Giving reviewers a model — this is what a good review and comments looks like, this is not helpful — could make a big difference.
9. Presentation and poster help. Guidelines to converting a paper to presentation and/or posters.
On a related note, I think we might want to consider ramping up the award program. As far as I know, we have the Nancy Mead award, and when we remember, we do best papers, but that’s about it. It seems to me that we really should consider what kind of awards we should be handing out — we want to encourage better research and submissions? Hey, give out awards along those lines. How about:
1. Most innovative use of new media for education and training in software engineering education and training
2. Best qualitative research study of education and training in software engineering
3. Best presentation — not just the best paper, but the best presentation also deserves recognition.
Finally, I think we may want to improve the reporting of the conference. I know some individuals have to provide reports or similar summaries when they get back to their organization, but it would be useful for everyone if there were good summaries of the conference available afterwards. Workshops, panels, Q&A discussions — there is often a lot of “good information” which simply vanishes after the conference. I’d like to see some thought given to having volunteer “reporters” who write up the various sessions for a conference report, perhaps available on the website. It might also be good to collect some “how-to” information from the students and others who participated, to help guide the next cycle.
That’s more than enough for a start.