Closing Non-Performing Sites

According to a 2009 study by Morrison and Pfizer, roughly 30% of sites that are opened fail to enroll.  Despite careful site selection criteria and thorough feasibility, dud sites are opened in most trials (although unintentionally and we do go to great measures to avoid this). In my career I have only worked on three studies where every single site enrolled at least one patient.  It is more common to have a handful of sites that recruit the bulk of your patients and then between 10-20% of the sites who never randomize a subject (about 75% of these will offer you Screen Fail subjects...oh boy).

Non-performing sites are a drain on resources and will have a negative impact and cost to your study.

So let's talk costs of non-recruiting sites.  There is a huge expense to your timeline when a site fails to perform or under-performs because inevitably you are now spending more time to recruit and pick up the slack at other sites.  It is a large drain on resources to start and carry a non-performing site; you still have to monitor, supply this site, contact this site, train this site, perform budgeting and accruals for this site, and provide additional support and cheer-leading for a nonperforming site.  Almost assuredly, those tied-up resources would have a better return on investment if re-directed to other study activities or to support the sites that are making more significant contributions.  Finally there is an opportunity cost because you likely have selected this particular site in lieu of another site that may have been a better recruiter.

Since non-performing sites cost your study, it may be worthwhile to close them, even before the end of the trial. Cut your losses and limit your disappointment.  Your company may choose to close a site that has enrolled subjects but this is much more rare.  This is more likely to happen because of a change in PI, departure of qualified staff required to perform assessments, or some other irreconcilable issue.

Don't let one bad apple spoil the bunch...

Now, I am beating up on the non-performers so I will pause to say that sometimes sites don't perform and it is not anything that they can control.  Perhaps the inclusion exclusion criteria are too stringent, if they don't have access to the right patients, if a more attractive study is recruiting nearby, and so on.  However, for the purposes of this discussion, let's instead call out on some more avoidable reasons that sites fail to recruit: competitive trials at the study site (your PI is cheating on you!), failure to negotiate an appropriate budget, insufficient staffing and resources, no commitment to an Investigative Site Recruitment Plan, uncoordinated search through clinic charts or database, out-of-sight / out-of-mind, inexperience in "pitching" the trial to potential participants, inattention by the PI/failure to prioritize the trial, etc.

Many site contracts are written now so that a start-up fee is partially or wholly refundable to the sponsor if enrollment does not occur within a pre-defined timeframe.  As a cautionary tale, beware of the risk of losing goodwill at your "nice" sites when you close your "naughty" sites.  The community can be tight (and chatty) so conduct yourself accordingly, and exercise tact and professionalism.  If you tread carefully, you may be able to salvage the relationship and appoach this site for future feasibility (assuming you would even want to...past performance is usually the best indicator of future capabilities...).  As an added bonus, you will avoid the ripple effect of negative morale, concern, or fear at your other sites; these can be very de-motivating and halt/delay your study recruitment.

For more information on the tasks required to close a site please refer to a previous blog post on close-out visits.

Are there other hidden costs to keeping a non-performing site open? I would love to hear your comments or you can send me an email to leadcra-mail@yahoo.com - thanks!



You may also like...from The Lead CRA archives:

• Investigative Site Recruitment Plans September 2010
• Selecting Qualified Investigators April 2010
• Close-Out Visits March 2008

Close-Out Visits this Quarter

I've been performing close-out visits for a variety of studies lately and I am ramping up on a new study. Sorry I haven't been able to post sooner, but in this economy I am happy to have a great job that is keeping me busy with such interesting and varied assignments.

I have been closing other people's sites on studies I didn't even work on so one of my more time-consuming responsibilities has been a full regulatory binder review. Typically for sites I manage over time, I know what is in the binder and can verify that the documentation contents at the study site match the contents at the sponsor's Trial Master File (TMF) exactly (since I work for a large organized CRO we also have a fantastic suite of applications and reports to help us with this reconciliation process). The reports and previous monitoring reports help me identify what documents need to be updated and I email the list to the coordinator several weeks before the visit. Since I need to retrieve a copy of any updated documents for the TMF I just ask the coordinator to slip a second copy for me in a folder or email me an electronic version and this has dramatically reduced the time I would spend taking apart their binder and hovering over the copy machine. Some sponsors want the site to keep originals of all study logs and some sponsors want the site to keep copies so they get the original. Some sponsors consider a signed CV to be good for 2 years whereas some require annual updates. Some sponsors want an updated 1572 and/or Financial Disclosure Forms at the end of the study and some sponsors don't ask for this at all. There are few hard and fast rules for documentation (aside from the ICH/GCP guidelines in section ICH E6, Section 8.4) so I have to understand the study specific requirements in advance of the visit (these are communicated in team meetings and documented in the study monitoring plan) to know which documents to retrieve.

Another area that has required a lot of time and attention is the return of the Investigational Product (IP). To assist me with the drug return I have requested reports from the inventory management system several weeks in advance of the visit and submitted them to the pharmacist or Study Coordinator. I ask them to organize the medication numerically by lot/kit # into used and unused lots and to check that all drug receipt and dispensing forms are accurate and complete and that the quantities match the CRFs. I follow-up over the next several days via phone or email and ensure they have completed the task. Typically they print the inventory reports and highlight them or make comments when there is a discrepancy. I always have them fax me any notes and copies of all the drug logs so I can review them. I can prepare the bulk of all my drug return documentation before I even arrive at this site and that saves a lot of time during the visit.

The last item that has been a real time-synch is reviewing the safety binder to ensure that every safety alert was acknowledged by the Principal Investigator (PI), submitted to the Institutional Review Board (IRB), acknowledged by the IRB, and that all that documentation is filed in the binder and submitted to the TMF. Luckily, I have been able to do work prior to the visit to help this task go as smoothly as possible (one study I closed out had over 5000 safety alerts so that could easily have been several days work whereas another study only had 42 and I knocked out the review task in about an hour). As you know, this review of safety letters should have been occurring all through conduct and regular monitoring visit but unfortunately, when it is not your site, you're really forced to re-review every bit of documentation unless the previous monitoring report(s) can confirm the documentation is current. For each site I closed I sent a safety letter tracker. It was just an excel spreadsheet where each alert was listed by date in the order in which it was sent to the site with a brief description that included the event #, event name, event onset date, patient initials, and a column specifying whether the event was an intial report or follow-up report. Finally, I added three blank columns to the end of the spreadsheet with the titles 'Date Ack. by PI' , 'Sent to IRB Y/N', 'Date of IRB Ack.' In each case, I required the study staff to complete the tracker and email it back to me prior to the visit. By using this technique we easily discovered missing documentation in advance fo the visit. I was able to forward blank copies to the site to prepare before the visit and we had time to obtain sign-offs and do the necessary IRB submissions. It was helpful to be sure pending safety letters didn't delay official notification to the IRB of study closure.

In summation, close-out visits at my own sites go pretty quickly and are fairly routine whereas there is a lot more work involved with closing out someone else's sites. To make the visits easier I worked with the site staff several weeks before the visit to address the documentation gaps in the regulatory binder, review all the IP documentation, and to ensure that all safety letters had been properly acknowledged by both the PI and the IRB. By thoroughly preparing in these 3 time-consuming areas, I had plenty of time while on site to accomplish all of my close-out responsibilities (click here to learn more about COVs in my blog post from last year). One other huge plus is that I knew what to expect once I arrived on the site, had developed a rapport with the staff, had prepared organized checklists, and then I left the sites with minor or no pending action items whatsoever. This all contributed to being able to easily and quickly finalize my close-out visit reports. I hope my experience helps you better prepare when you are asked to close-out someone else's site (or even your own). Do your homework before the visit and it should go like clockwork.

Efficiency Tips: Spend less time on your Timesheet

I am employed by a service company and we bill our clients for our services. Therefore, we must track our time spent on sponsor related work and pass the costs through appropriately. I am asked to track my time in 15 minute increments so I can basically report up to 4 time categories per hour or I can bill a half or whole hour to a specific time code. In order to receive a good performance rating, I must report at least 85% of my 40 hr work week as "billable time".

My non-billable time codes are things like: Sick, Holiday, General and Administrative Duties (G&A), Internal training (not project specific), goofing off (ok, so that is not a category but if ever I am shy of 40 hrs and can't account for the time then G&A is a good catch all), etc.

Billable time is coded by project (since I might work for several sponsors/projects at once) and then broken down into time categories as follows:
Project Specific Training / Sponsor Teleconferences - Reading protocols, watching power points, reviewing the Monitoring Plan, etc.
Preparing and Following Up - This category is used for making travel arrangements or writing confirmation/follow-up letters, or working on reports.
Traveling - Usually your project will have limits to how much travel time you are allowed to pass through. At my company, when long trips are unavoidable, you can lower your travel time by using the time in transit to complete other project related work such as studying or writing reports (billable to either of the 2 categories listed above) - obviously this works when on an airplane or delayed and not necessarily when driving!
Managing Sites - This is phone or email contacts to your study sites or running special project specific reports.
Conducting a Monitoring Visit - Actual time spent on site during SIV, COV, or routine MV.
Query/Database Management - Helping the site solve/close queries raised by data management, reviewing the clinical database, running reports, etc.
Expense Reporting - see my next post!

OK, so now that I explained how I categorize time, let me explain the time tracking system. You can think of it as a glorified version of Excel that lives on a web server. So to enter my time, I have to log into that system and enter all of my hours. We are asked to do this at the end of every day so we are as accurate as possible. Well, I don't have internet access all the time so sometimes I let a few days pass and it really is hard to remember what you did for the past 48-72 hours by 15 minute increments. To solve this conundrum, I made a copy of the fancy web-based timesheet template in Excel and I just leave it on my desktop. Each day before I log out of work, I update my time in my Excel template. Then every 2 weeks (or as I have time) I log into the web-based system and transfer it all. I know this is multiple steps but it seems to work better for me. You could also use a journal or jot notes about how you spent your time in your day planner. Any other tips?

Close-Out Visits

I've done about 10 close-out visits in the last few months so it feels like a good time to write a short article explaining what the objectives of these visits are and how a typical close-out visit (COV) is conducted.

A COV will occur once subjects are no longer being dosed, all the data have been collected (there are no more outstanding AEs/SAEs & all outstanding Queries/data clarification forms have been resolved appropriately), the database is locked and ready for statistical analysis, and the study conduct has ended. At this point, the site's contributions are over so the monitor returns for one final visit to shut down the site. The whole concept behind a close-out visit is to ensure that everything is neat and tidy at the study site and that the documentation is well organized and will remain intact and be accessible in the future as needed for regulatory reasons. A sponsor or the FDA should be able to return to the place of study conduct years later and re-create exactly what occurred at all points during the trial by reviewing the regulatory documentation, subject and source documentation, full medical charts, and any other applicable study records. Documentation is everything in our industry and we are always saying, "if it isn't documented it didn't happen." If thorough and accurate records are not maintained, the PI cannot prove that the study was conducted in accordance with the protocol and all applicable regulations and that subject safety was adequately monitored throughout the conduct of the trial.

Site Supplies and Drug Return

We are using tamper evident tape to
box up investigational product for return.
If the tape is lifted it leaves behind an
artifact to show it has been tampered with.

One of your major objectives at the COV will be to assist the site in dealing with any unneeded study materials or supplies. With permission from the sponsor, some of these activities can even take place before the close-out visit as you remotely supervise. The close-out duties will likely include disposing of or retrieving all unused lab or study supplies such as patient handouts, electronic diaries, etc. In most cases, you will have the site generate a file note or similar documentation so there is a record indicating that study supplies were disposed of or moved off-site. 9 times out of 10, the Investigational Product (unused and the used packaging) will need to be inventoried, accounted for in drug logs, and then shipped to a depot or destruction facility. Sometimes the drug will be destroyed on site by a pharmacist or according to the site's SOP but this is really the decision of the sponsor. If you are lucky, you were able to pack up and ship back drug routinely throughout conduct otherwise you will have to deal with it all at the end.


Essential Documents
You will have the PI sign off on any tracking logs that were used during the study. The original will be placed in the site's Regulatory binder but you will retrieve a copy for the Trial Master File. You will ensure that a Subject Identity List was completed and will be kept that lists the contact information for all treated subjects (you will not take a copy of this document as it has private information and stays at the site only). Documents you will take copies of include: Site Visit Log, Subject Screening AND Enrollment Log, Delegation of Authority Log, Proof of Drug Receipt, Subject Specific Investigational Product (IP) Accountability Logs, Copies of temperature/freezer logs, Site Initiation Statement, Training Documentation, Overall Site IP Log, Protocol/Amendment Signature Pages, Any updated 1572s, medical licenses, or CVs, Site communications to the IRB/IEC (ethic committee), and the IRB Final Status Document. Obviously in a study with many safety reports, a long line of routine monitoring visits, multiple site hand-offs/transitions between several different monitors, or a slew of important correspondence, checking that the essential documents binder(s) is in perfect order can be a time-consuming task.

Subject Records
Although you will have already verified this throughout conduct, the close-out visit is your last opportunity to be absolutely sure that the appropriate version of signed and dated Informed Consent Forms are on file for every subject. You will also check that all source is complete (all lab reports and ECGs have been signed and dated with Clinical Significance assessed by the PI/Sub-I) and that all AEs/SAEs have been signed off by the PI/Sub-I and that they were followed to resolution as specified by the protocol. Finally, check that all significant Protocol Deviations (study procedures not conducted according to protocol, enrollment of inappropriate subjects, dosing errors, consenting errors, unblinding, subjects developing withdrawal criteria yet continuing in study, etc.) have been properly recorded and the sponsor/IRB has been notified as appropriate.

PI Responsibilities
Discuss with the PI his/her responsibilities including: query/data collection following the close-out visit, essential document retention, publication rights, and the necessity to update the Financial Disclosure statement if there are changes in their financial interest for up to one year following completion of the study. Finally, explain to the PI the potential for regulatory agency inspection and the requirement that the site notify the CRO/sponsor immediately if contacted for an audit/inspection.

Assuming you have done a thorough job in monitoring throughout conduct, the COV should be a relatively short-visit. Meeting with the PI to discuss their regulatory responsibilities post trial conduct and obtaining required signatures usually takes less than 20 minutes assuming they are an experienced investigator and are already familiar with the GCP schpeel. Drug return often takes several hours but you can prepare most drug return documents in advance of the visit by using sponsor or IVRS reports and usually save a considerable amount of time on-site. You should have been reviewing the regulatory binder at every visit throughout conduct so it should really be in order at this point and stuffed to the brim - I usually budget no more than an hour to ensuring that the binder is complete.

After you complete the close-out visit, you will write a report to the sponsor to let them know that all of the objectives were completed and a follow-up letter to the site thanking them for their participation and informing them that there are no further pending action items. Any new regulatory documentation you copied while on site will need to be forwarded to the Trial Master File so that the sponsor's documentation is a true mirror of what is on site.

You may also like...from The Lead CRA archives:

• Introduction to Monitoring December 2007
• Meeting with the PI August 2011
• Routine Monitoring Visits March 2008

Introduction to Monitoring

I've been out on the road for the past few weeks assisting some other monitors to review patient's medical records and the data that has been collected for the clinical trial. It occurs to me, that it would be beneficial at this point to further describe to you the roles and expectations of a Clinical Research Associate (CRA).

In a pharmaceutical/biotech company, teams of scientists are working to discover new molecules, compounds, and foundations for new drugs. In addition, these same scientists sometimes work to re-formulate current drugs or apply them for new diseases and indications (all of this work is done in animal models before the human clinical trials can begin). It is at this point that monitors such as myself come into play. Once an experiment that involves humans is designed, monitors help the company choose qualified and interested physicians to carry out the study. These 'investigators' are reimbursed for their efforts to conduct the trial and a CRA may also help negotiate the budget and/or contract between the parties involved.

A CRA job is a traveling job. Sometimes we go to glamorous places but more often than not we are monitoring in hard to reach or otherwise obscure locations. 

Objectives when we visit the investigative site (Doctor office, hospital, research center, etc.) include: 1) perform pre-study tasks during a Pre-Study Visit (PSV), 2) kick-off the study with a Site Initiation Visit (SIV), 3) routinely check back with the site regarding subject enrollment, trial conduct, and data collection during one (or typically many!) Interim Site Monitoring Visits (MV), and finally 4) shut down the site after their participation has ended during a Close Out Visit (COV). Before and after each visit, there are a variety of tasks and communications that need to occur in regards to preparing for, scheduling, confirming, and then following up on items from previous visits. Read some of my other blog posts to learn more about Site Initiation Visits, Routine Monitoring Visits, and Close-Out Visits.

Also, between visits, a CRA needs to regularly contact the sites and perform site management activities such as answering questions regarding the study design, encouraging enrollment, sharing tips from other sites, and distributing study drug (Investigational Product, IP), and managing site supplies inventory levels (often you are assigned up to 10 different sites to manage but this really can vary depending on geography, how many patients/doctor sites are participating, how many monitors are on the study, how complex the trial is, and a variety of other factors).

I'll explain more regarding the key tasks at each visit in later articles. There are many Federal and International regulations that apply to human research to ensure that the studies are valid, reliable, and above all, that the rights of the participating subjects are protected and we'll also get into those soon.