forked from ropensci/dev_guide
-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathsoftwarereview_editor.Rmd
More file actions
159 lines (117 loc) · 12.8 KB
/
softwarereview_editor.Rmd
File metadata and controls
159 lines (117 loc) · 12.8 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
# Guide for Editors {#editorguide}
```{block, type='summaryblock'}
Software Peer Review at rOpenSci is managed by a team of editors. We are piloting
a system of a rotating Editor-in-Chief (EiC).
This chapter presents the responsibilities [of the Editor-in-Chief](#eicchecklist), of [any editor in charge of a submission](#editorchecklist), and [how to respond to an out-of-scope submission](#outofscoperesponse).
If you're a guest editor, thanks for helping! Please contact the editor who invited you to handle a submission for any question you might have.
```
## EiC Responsibilities {#eicchecklist}
The EiC serves for 3 months or a time agreed to by all members of the editorial
board. The EiC plays the following roles
- Watches all issues posted to the software-review repo.
- Assigns package submissions to other editors, including self, to handle. Mostly this just rotates among editors, unless the EiC thinks an editor is particularly suited to a package, or an editor rejects due to being too busy or because of conflicting interests.
- Raises scope/overlap issue with all editors if they see an ambiguous case. This may also be done by handling editors (see below). To initiate discussion, this is posted to the rOpenSci Slack software review channel, tagging all editors.
- Responds to pre-submission inquiries and `meta` issues posted to the software-review and software-review-meta repos, similarly pinging channel if discussion is needed. Any editor should feel free to step in on these. See [this section](#outofscoperesponse) about how to respond to out-of-scope (pre-) submissions.
- Responds to referrals from JOSS or other publication partners.
- Monitors pace of review process and reminds other editors to move packages along as needed.
## Handling Editor's Checklist {#editorchecklist}
### Upon submission:
- Tag issue with `1/editor-checks` tag and assign a main editor. Please strive to finish the checks and start looking for reviewers within 5 working days.
- Use the [editor template](#editortemplate) to guide initial checks and record your response to the submission. You can also streamline your editor checks by using the [`pkgreviewr` package created by associate editor Anna Krystalli](https://ropenscilabs.github.io/pkgreviewr/articles/editors.html)
- Check that template has been properly filled out.
- Check against policies for [fit](#aims-and-scope) and [overlap](#overlap).
Initiate discussion via Slack #software-review channel if needed for edge cases that haven't been caught by previous checks by the EiC.
If reject, see [this section](#outofscoperesponse) about how to respond.
- Check that mandatory parts of template are complete. If not, direct authors toward appropriate instructions.
- Run automated tests: `spelling::spell_check_package()`, `goodpractice::gp()` (most exceptions will need to be justified by the author in the particular context of their package.), `devtools::spell_check()`. Run `covr::package_coverage()` using `NOT_CRAN` if needed, as well. Check that documentation is generated using `roxygen2`, not by hand (this isn't part of automatic tests [yet](https://github.com/MangoTheCat/goodpractice/issues/116)). Report relevant outputs in the issue thread.
- For packages needing continuous integration on multiple platforms (cf [criteria in this section of the CI chapter](#whichci)) make sure the package gets tested on multiple platforms (having the package built on both Travis and AppVeyor for instance).
- Wherever possible when asking for changes, direct authors to automatic tools such as [`usethis`](https://usethis.r-lib.org/) and [`styler`](https://styler.r-lib.org/), and to online resources (sections of this guide, sections of the [R packages book](https://r-pkgs.org/)) to make your feedback easier to use. [Example of editor's checks](https://github.com/ropensci/software-review/issues/207#issuecomment-379909739).
- If initial checks show major gaps, request changes before assigning reviewers.
- If the package raises a new issue for rOpenSci policy, start a conversation in Slack or open a discussion on the [rOpenSci forum](https://discuss.ropensci.org/) to discuss it with other editors ([example of policy discussion](https://discuss.ropensci.org/t/overlap-policy-for-package-onboarding/368)).
### Look for and assign reviewers:
#### Tasks
- Switch numbered tag to `2/seeking-reviewers`.
- Ask author to add a rOpenSci review badge to their README, via [`rodev::use_review_badge()`](https://docs.ropensci.org/rodev/reference/use_review_badge.html), `rodev::use_review_badge(<issue_number>)`. Badge URL is `https://badges.ropensci.org/<issue_id>_status.svg`. Full link should be:
```
[](https://github.com/ropensci/software-review/issues/<issue_id>)
```
- Use the [email template](#reviewrequesttemplate) if needed for inviting reviewers
- When inviting reviewers, include something like "if I don't hear from you in a week, I'll assume you are unable to review," so as to give a clear deadline when you'll move on to looking for someone else.
- Assign a due date 3 weeks after all reviewers have been found.
- Once two or more reviewers are found, assign reviewers by tagging in the issue with the following format:
```
Reviewer: @githubname1
Reviewer: @githubname2
Due date: YYYY-MM-DD
```
- Switch numbered tag `to 3/reviewers-assigned` once reviewers are assigned.
- Invite authors and reviewers to rOpenSci Slack if they aren't on this Slack already.
- Update Airtable database: Add one of the review record numbers associated with the package to the `Reviews` field of each reviewer's record. Insert new `Reviewers` record if required.
#### How to look for reviewers
##### Where to look for reviewers?
As a (guest) editor, use
* the potential suggestions made by the submitter(s), (although submitters may have a narrow view of the types of expertise needed. We suggest not using more than one of suggested reviewers)
* the Airtable database of reviewers and volunteers
* and the authors of [rOpenSci packages](https://ropensci.org/packages/).
When these sources of information are not enough,
* ping other editors in Slack for ideas,
* look for users of the package or of the data source/upstream service the package connects to (via their opening issues in the repository, starring it, citing it in papers, talking about it on Twitter).
* You can also search for authors of related packages on [r-pkg.org](https://r-pkg.org/).
* R-Ladies has a [directory](https://rladies.org/directory/) specifying skills and interests of people listed.
#### Criteria for choosing a reviewer
Here are criteria to keep in mind when choosing a reviewer. You might need to piece this information together by searching CRAN and the potential reviewer’s GitHub page and general online presence (personal website, Twitter).
* Has not reviewed a package for us within the last 6 months.
* Some package development experience.
* Some domain experience in the field of the package or data source
* No [conflicts of interest](#coi).
* Try to balance your sense of the potential reviewer’s experience against the complexity of the package.
* Diversity - with two reviewers both shouldn’t be cis white males.
* Some evidence that they are interested in openness or R community activities, although blind emailing is fine.
Each submission should be reviewed by _two_ package reviewers. Although it is fine for one of them to have less package development experience and more domain knowledge, the review should not be split in two. Both reviewers need to review the package comprehensively, though from their particular perspective. In general, at least one reviewer should have prior reviewing experience, and of course inviting one new reviewer expands our pool of reviewers.
### During review:
- Check in with reviewers and authors occasionally. Offer clarification and help as needed.
- In general aim for 3 weeks for review, 2 weeks for
subsequent changes, and 1 week for reviewer approval of changes.
- Upon all reviews being submitted, change the review status tag to
`4/review-in-awaiting-changes` to update the reminder bot.
- Update Airtable database: Add the link to the review comment to the `review_url` field and the number of review hours to the `review_hours` field of each review record.
- If the author stops responding, refer to [the policies](#package-submission) and/or ping the other editors in the Slack channel for discussion. Importantly, if a reviewer was assigned to a closed issue, contact them when closing the issue to explain the decision, thank them once again for their work, and make a note in our database to assign them to a submission with high chances of smooth software review next time (e.g. a package author who has already submitted packages to us).
- Upon changes being made, change the review status tag to `5/awaiting-reviewer-response`.
### After review:
- Change the status tag to `6/approved`.
- You can use the [comment template](#approvaltemplate).
- Add review/er information to the review database.
- If authors intend to submit to CRAN, direct them to the [section about CRAN gotchas](#crangotchas) and offer to provide support through this process.
- Ask authors to add a mention of the review in `DESCRIPTION` via [`rodev::add_ro_desc()`](https://docs.ropensci.org/rodev/reference/add_ro_desc.html).
- Ask authors to migrate to `ropensci`
- Create a two-person team in rOpenSci's "ropensci" GitHub organization, named for the package, with yourself and the package author as members.
- Have the author transfer the repository to `ropensci`
- Go to the repository settings in rOpenSci's "ropensci" GitHub organization and give the author "Admin" access to the repository.
- Ask author to:
- Add rOpenSci footer to README `[](https://ropensci.org)`
- Add a CodeMeta file by running `codemetar::write_codemeta()` ([`codemetar` GitHub repo](https://github.com/ropensci/codemetar))
- Change any needed links, such those for CI badges
- Re-activate CI services
- For Travis, activating the project in the ropensci account should be sufficient
- For AppVeyor, tell the author to update the GitHub link in their badge, but do not transfer the project: AppVeyor projects should remain under the authors' account. The badge is `[](https://ci.appveyor.com/project/individualaccount/pkgname)`.
- For Codecov, the webhook may need to be reset by the author.
- Nominate a package to be featured in an rOpenSci blog post or tech note if you think it might be of high interest. Please note in the software review issue one or two things the author could highlight, and tag `@stefaniebutland` for follow-up.
- Add a "peer-reviewed" topic to the repo.
- Close the software-review issue.
### For joint JOSS submissions:
- After repo is transferred to ropensci and admin rights assigned, have author generate
a new release with a DOI.
- Ask author to submit package via https://joss.theoj.org/papers/new
- Watch for paper to pop up at https://joss.theoj.org/papers, then
add the following comment to the submission thread:
`This submission has been accepted to rOpenSci. The review thread can be
found at [LINK TO SOFTWARE REVIEW ISSUE]`
### Package promotion:
- Ask authors to write either a blog post or a tech-notes post for the package, as appropriate, and ping [Stefanie Butland](https://github.com/stefaniebutland), rOpenSci community manager.
- Alert maintainers of appropriate [task views](https://github.com/search?utf8=%E2%9C%93&q=user%3Aropensci+%22task+view%22&type=Repositories&ref=searchresults)
- Direct the author to the chapters of the guide about [package releases](#releases), [marketing](#marketing) and [GitHub grooming](#grooming).
## Responding to out-of-scope submissions {#outofscoperesponse}
Thank authors for their submission, explain the reasons for the decision, and direct them to other publication venues if relevant, and to the rOpenSci discussion forum. Use wording from [Aims and scope](#aims-and-scope) in particular regarding the evolution of scope over time, and the overlap and differences between unconf/staff/software-review development.
[Examples of out-of-scope submissions and responses](https://github.com/ropensci/software-review/issues?q=is%3Aissue+is%3Aclosed+label%3Aout-of-scope).
## Managing a dev guide release {#bookrelease}
If you are in charge of managing a release of the very book you are reading, use [the book release guidance](#bookreleaseissue) as an issue template to be posted [in the dev guide issue tracker](https://github.com/ropensci/dev_guide/issues), and do not hesitate to ask questions to other editors.