Peer Review: An Essential Step in OVW's Award-making Process

Minh Ha:

Hi, welcome to Patchwork, the podcast from the Office on Violence Against Women at the U.S. Department of Justice in Washington. I'm Minh Ha from OVW and today, we're joined by OVW's peer review working group. Peer review is an integral part of our review process and today the group joins us to talk about peer review and answer some of the questions that we've heard about OVW's review process. Before we get started, why don't you introduce yourselves to our listeners?

Darlene Johnson:

Hi, my name is Darlene Johnson. I'm from the Office on Violence Against Women. I'm an Associate Director and I oversee three of OWV’s grant programs, the Grants to Improve Criminal Justice Response,  the Campus Program, and the Abuse in Later Life. And I'm also here with a few colleagues and I'll allow them to introduce themselves as well.

Sandi Van Orden:

Hi, my name is Sandi Van Orden. I am a grant program specialist on the the Legal Assistance for Victims unit.

Neelam Patel:

Hi everyone. My name is Neelam Patel. I'm team lead for Training and Technical Assistance.

Myrta Charles:

Hi, and I'm Myrta Charles. I'm with the Transitional Housing unit with the Office on Violence Against Women.

Darlene Johnson:

Together. We are the peer review working group, from the program division within the office. And so today we're going to go over a few important questions that we received over the years about expectations of peer review. So let's begin. I want to ask the group, what qualities does OVW look for in a peer reviewer?

Myrta Charles:

You know, Darlene, I find that work experience is really important - experience working in the violence against women field. Peer reviewers who have that kind of background really make for excellent reviewers.

Sandi Van Orden:

Along with that experience, it's also important, for the unit I work on, to have peer reviewers who are experienced working in the legal, just like in another unit, the type of field that is most directly related to the grant program would be important.

Neelam Patel:

I also would say that it's important for OVW to have current and former grantees and TA providers also be our peer reviewers. It's called peer reviewer for a reason because you want your peers to be able to review your proposal because they're the ones that really know what's capable of being implemented for the programs.

Darlene Johnson:

That's great. What about any specific skill set? What is needed?

Myrta Charles:

I find those that are really detail oriented, um, can look at a proposal well, remember the details, and be able to talk about those. So someone who is detail oriented is key, I think. Also maybe I'd say critical thinking skills, they're analytical in their perspective and in their review, those really come off really well in a group setting, which is what we have when we do our peer reviews.

Sandi Van Orden:

Strong communication skills are also important because there's a need to communicate amongst the other peer reviewers about your thoughts on the application, but there's also a need to be able to develop a well-written statement that would go to the applicant to tell them where they could potentially improve future applications.

Darlene Johnson:

Great. If, I'm interested in becoming a peer review, how much, what is the time commitment to be involved? What is the availability requirement?

Neelam Patel:

That's a good question, Darlene. We get that question a lot from grantees, particularly, and other people in the field because they have full-time jobs, right? They have things they have to do. You need to be able to be available for the period of time that the program’s peer review is going on, which means you need to be able to allocate enough time to thoroughly review and score applications. So you're looking at even reading applications in the evenings, in the daytime, taking some time off and you need to be able to attend the consensus meetings as well and schedule that time out.

Myrta Charles:

I think you're right on point. One of the things I noted also it'll depend on what program you're reading for, but on average it does take a time commitment to be able to review with skill and aptitude if you decide to be committed to this process.

Darlene Johnson:

Thank you. I'm going to get to the real question I know a lot of people want to know: how does OVW select peer reviewers? What is that criteria for selecting?

Neelam Patel:

I'll take that one. Darlene. First, I think first and is OVW is very dedicated to getting reviewers who have expertise in the fields of domestic violence, sexual assault, dating violence, and stalking. It's important that the peers that are reading the applications come from the field, understand the field, understand best practices and promising practice of the work that they are reviewing for the applications. So that's really important for us.

Myrta Charles:

I'd add to that. I think Sandi talked about this a little bit, but I would look at expertise working with communities served on the grant program. And that's another key point. So, we work with 21 different program. We have rural programs. We have tribal programs. We have civil legal programs. People who come from those communities who can talk from a place of experience and knowledge really are key to those that we select to reflect again, their understanding of our grant programs.

Sandi Van Orden:

Also, I would say the not only rural or civil legal communities, but also communities like geographic or specific population of communities, having experience working with the people who will be served by these grant programs.

Myrta Charles:

I would say some other factors in selecting peer reviewers that really are key. Generally a variety of professions are requested on the panels based on the needs of the grant program. So another one would be, we have campus programs, so having people who work or know students who understand the needs of students and what they will encounter and what a university needs to put together would be another way we would look at selecting peer reviewers. Again, based on the campus program. And if you're going to be a program for that, that would be the example of what I would say. We look for professions that, that are part of the community and that our programs are based on. We like to work with a number of new and seasoned pre reviewers. So if you've done this before, you know it's grueling, but it's exciting, it's honorable. And it really helps keep our feet to what the field is doing. And if you're new, it's a great way to get involved.

Neelam Patel:

And I would add to that also Myrta is it is sometimes scary for a new reviewer not to know what to expect and the time commitment. And that's why we also are very careful about putting new reviewers on panels with seasoned reviewers, so that you feel as if you have a peer, so to speak, to be with you on peer review if you have questions and you can learn from them as well. One thing to keep in mind is you may be interested in a specific program because you under specific for like campus or legal assistance, but if your organization is applying for that program, that year, you won't be able to review. So we would probably call on you in another year when your organization is not applying.

Myrta Charles:

Good point, Neelam, good point.

Sandi Van Orden:

I was just going to clarify that you can be a peer reviewer, just not for a program that your organization is applying for a grant under.

Neelam Patel:

Thank you for that clarification, Sandi.

Myrta Charles:

But one thing know that all peer reviewers are subject to a conflict check to make sure we keep the integrity of the peer review high.

Darlene Johnson:

Thank you. Next question. How would one become a peer reviewer? How do you apply to become a peer reviewer? What should I do if I'm interested?

Sandi Van Orden:

There are two ways that someone could potentially apply to be a peer reviewer. The first is if you know someone at OVW you can reach out to them about submitting your resume and what the process would be like, and they can help you walk through that. The other thing that you could do would be to email your resume to OVW and that email address is OVW.peerreview@usdoj.gov, or you can go to OVW's website, and there is a peer review section, and that will give you instructions on how to apply to be a peer reviewer.

Darlene Johnson:

That's great. What are OVWs plan for fiscal year 2022? How will peer reviews be conducted?

Neelam Patel:

We'll take that one, Darlene. So, 2022 will VW all their peer reviews will be virtual, which means we will be using a virtual platform such as Zoom for the consensus meetings. Applications will be sent electronically to review. Usually, before the pandemic, we would do ours in person and reviewers would come to the DC Metro area for a week-long peer review where they would read applications and panel. This year we're doing it virtually which can span across a week and a half to two weeks.

Darlene Johnson:

I know earlier we talked about the availability of a peer reviewer and what that time commit would look like. Can you break that down a little bit as far as like how much time I would have to commit if I'm interested in being a peer reviewer?

Myrta Charles:

Sure. One thing I can say, we'll start from the top, if you're going to be a peer reviewer, we have an orientation that can help you just center your schedule and figure out what you need. Our orientations, especially in a virtual, or we all do Zoom, probably be a Zoom call generally takes about an hour, an hour and a half, depending on the program. We help you figure out how to read a solicitation, what to look for, what questions you may have and get you started on reviewing or reading your applications.

Sandi Van Orden:

After the orientation, our reviewers would begin reading the applications that they'll score and general, depending on the duration of the peer review, there will be about two to four days to read and score the applications.

Myrta Charles:

And can I add that generally, we give a peer review approximately, eight to ten applications to read per panel, so that might help in turn knowing that you're reading for two to four days about that many applications.

Darlene Johnson:

I have a question, a panel consists of whom or how many people?

Neelam Patel:

A panel consists of three individuals. And a consensus, when you meet in the consensus meeting… a consensus on a comment is two or more.

Myrta Charles:

I was going to say, just to elaborate on that consensus meetings are held… we do four hour sessions on specific days. So you come together, you finish reading your applications, and then you meet with your assigned panelists. There will be two others besides yourself as a reviewer and you speak about the applications that you've read and discuss, analyze, bring up the details that you spoke of and that's in a probably four hour session, usually about two days. As a reviewer, you should be available for the full four hour meetings on those meeting days to make sure your committed and that you can square away that time and focus on with your panelists on each of those applications so they get a thorough review.

Neelam Patel:

I was also going to add to what Myrta just said is the scheduling of those consensus meetings is done by OVW and their contractor. So, it is important if there's a day when you're sent to save the date, and you've confirmed that you are available that full day during the work hours.

Darlene Johnson:

What is the peer review schedule like from start to end?

Sandi Van Orden:

So depending on the size of the grant program, most, uh, peer reviews will last one to two weeks and that just varies on the number of applications and the number of panels.

Darlene Johnson:

Thank you. How much time does it take to read, review, and score an application?

Neelam Patel:

So this does vary by expertise in being a peer reviewer and also the type of program. Some programs, the applications are a lot more dense and complicated to read as a new reviewer and even a seasoned reviewer for your first application, it may take you two to two and a half hours to thoroughly review. The applications are in different parts, right? You got the project narrative, MOU, budget, and reviewing it, scoring it and providing clear and concise comments. As you start to get in the flow of reading applications, it may take you low us time between maybe an hour to an hour and a half, but this is why we make that we want you to allocate enough time, because you don't know the applications you're going to read. And there could be one in the batch that may take you a little longer to go through all the applications should have of fair review. You should make sure you allocate enough time for each of the applications so you can really thoroughly review it.

Darlene Johnson:

Thank you. What happens during a consensus meeting? This seems like a very interesting process.

Myrta Charles:

Oh it is. It is. I really enjoy this part of the peer review as do others. I think reviewers come in talk to each other and usually have a perspective that they've taken on for each application. So the reviewers come in, there's three to a panel and they have different either professional backgrounds, different knowledge, different communities that they represent. We put them together and bring that perspective to the table. They talk about the applications and their comments, including the strength and weaknesses of the application that they've new noted each element in the application. And here they go to discuss what they've interpreted about those elements in the application and talk with each other until they can come up with a consensus about what the strength and weaknesses of each of those applicants are. I think Neelam alluded to the fact that census means that two, at least two of the panelists agree that they found these straight and weaknesses to be the same in the application. And so they put down initial score, what they found when they were on their own and working as an individual on the application. And then they discussed those elements that they noted. And from there they adjust the score, each of the scores given the conversation that they had about those elements of the application.

Neelam Patel:

I would also want to add this, Myrta, to what you said is, you know, we read the applications as a peer review. You come in, scoring it based on the criteria that's set forth in the solicitation. It's at also an important time for reviewers to make sure that other reviewers aren't bringing in outside information, right? So you're scoring it based on the criteria, but we're all in the field and we can see where there might be some missing pieces or things you want more clarification. So it is also an opportunity for the fellow reviewers to make sure that everyone is scoring based on what the solicitation states.

Myrta Charles:

That's right on point Neelam. Can I add one other thing? It's really important that all reviewers are respectful of each other during that discussion especially given different communities, the different knowledge, the different disciplines that are present at the table, but they respect of each other during discussion, especially when there's areas of disagreement about what the strengths and weaknesses of the application. It's important they listen to each other, support what their information is given what's in the application or lacking thereof. And that makes for a really great consensus panel, having that respect for each other and what views they bring to the table

Darlene Johnson:

While the consensus meetings are going on is OVW present during that process?

Sandi Van Orden:

Yes. Someone from OVW will be facilitating the meeting. That person can also help if there are any questions regarding if there's clarity needed about an element of the solicitation or a question about the grant program.

Darlene Johnson:

Thank you. Thank you.

Myrta Charles:

And I think another important part to note is that we also have a recorder in each of those sessions so they can capture the exact comments and what the strength and weaknesses are and record that session for each of our panels so that we can accurately reflect that back to the grantee whether they're selected or not. So that work is there and supports the conversations that each of those panels have.

Darlene Johnson:

Great. So let me get to the real nitty gritty question. Our peer reviewers compensated?

Neelam Patel:

I will take that, yes, you are compensated! A reviewer is compensated in fiscal year 2022, $250 per application. So that is for every application you read, score, and panel. We do expect that if you are peer reviewing that you're either taking leave from your job or you're just taking leave without pay. We want make sure you're not working at the same time to give each application a fair review.

Darlene Johnson:

Any other tips or suggestions for potential reviewers?

Myrta Charles:

I would say tips is ask questions. Ask questions of OVW in terms of the process, what they see in application, what they understand in the solicitation and often it makes sure that everybody's on the same page. So it really does help if you say, “Hey, I don't have this. I didn't see this part of the application when I received it, does everybody have that or am I the only one missing that”, questions like that help everyone. So ask questions.

Neelam Patel:

I was also going to add to what Myrta just said is we want questions, right? We want to make sure that you are taking this seriously and that you are not just assuming the answers, but asking us questions. OVW staff is available via phone and email and virtual check all the peer review process, answer questions. So are your fellow peer reviewers. If you are a new reviewer and you have a question, you can also contact your fellow peer reviewers about any general questions you have about peer review.

Sandi Van Orden:

Along with that, if you decide to be a peer review for our program, I would recommend reading the solicitation prior to the orientation for that peer review so that you know the questions you have going into that orientation session.

Darlene Johnson:

That's a good point before we conclude, Sandi, could you tell potential reviewers once again how to apply to be a peer reviewer? You mentioned two options.

Sandi Van Orden:

Sure. If you know someone at OVW, feel free to reach out and we can help you with the process. You can email your resume to OVW.peerreview@usdoj.gov, or you can go to OVWs website and there is a page that gives instructions on applying to be of your reviewer.

Darlene Johnson:

Thank you so much, everyone for participating in the overview of peer review.

Minh Ha:

And thank you for listening to Patchwork. We appreciate any comments you might have to share about this or any of our episodes, feel free to send them to patchwork@usdoj.gov. Thanks for listening.

From OVW, I'm Minh Ha.