Public Library of Science - Technology

DNS Issue with PLoS.org - Resolved

Richard Cave — Sun, 5 Oct 2008 11:40:09 -0700

We had an outage on our PLoS.org domain resources yesterday due to a DNS issue. As a result, www.plos.org and all of the plos.org subdomains were intermittent as well. The issue was addressed promptly but there was a 2-10 hour delay while DNS servers were updated and ~24 hour delay before a comprehensive international DNS update. Most people were able to access www.plos.org within a few hours.

The DNS issue did not affect any of the PLoS journal websites.

Please be aware that if doi.plos.org is down, you can use CrossRef for DOI resolution.

Trackback URL for this post:

http://www.plos.org/cms/trackback/407

Journal Websites - Topaz 0.9 rc1 Upgrade

Richard Cave — Thu, 17 Jul 2008 17:03:14 -0700

Last night, we upgraded the journal websites to Topaz 0.9 rc1 (rc1 because this is a “beta” 0.9 release). The development for this release focused on performance and stability - specifically to alleviate the sluggish speed of the websites and the pain of ingests.

The development included a major re-architecture of the publishing application and weeks of performance testing. There are a few minor issues but the sites are quite zippy and the new ingest times have, to quote James "exceeded expectations." We’ll probably have a few more quick restarts over the next few days as we shift through the logs/bugs but we won’t have to rebuild the article cache ever again (another pain point).

Thanks to Russ for the heroic migration day, Josh for the speedy drive rebuilds of the Mulgara server last night, and to the Topaz/PLoS developers that got this "beta" release out the door. We still have some cleanup after the dust settles, but I feel confident that the site performance/stability has greatly improved.

Trackback URL for this post:

http://www.plos.org/cms/trackback/381

Journal Websites - Topaz 0.9 rc1 Upgrade

from journal on Fri, 2008-07-18 01:21

Bookmarked your post over at Blog Bookmarker.com!

TOPAZ upgrade

from A Blog Around The Clock on Thu, 2008-07-17 17:09

TOPAZ software, the one that hosts five out of six PLoS journals (ONE, Pathogens, Neglected Tropical Diseases, Genetics and Computational Biology) has just been upgraded. There is not much new in the terms of functionality visible to readers, but the...

Topaz 0.9 (rc1) - Site Maintenance Tonight on the PLoS Journals

Richard Cave — Wed, 16 Jul 2008 18:08:17 -0700

We're going to take down the journal websites for a few hours tonight starting at 7pm PST. Russ is going to upgrade to Topaz 0.9 (rc1) on the production servers while Josh replaces/rebuilds the dead drive on the Mulgara server.

Topaz 0.9 is primarily a performance release which will significantly enhance the speed of the journal websites. This release candidate was motivated by the performance problems uncovered with the CJ migration. There has been a re-architecture of the cache code and many OTM/Mulgara improvements by the Topaz developers. And they've upgraded many of the underlying applications. We've been performance testing this release candidate for the last two weeks and have seen significant improvements across the board from 0.8.

This release candidate isn't available to the public yet but will occur later this month after a bit more cleanup to the configuration files. The 0.9 release will also include the Ambra publishing application and two default "journals" - I'll have more information on Ambra closer to the release date.

Performance enhancements in Topaz 0.9

Pre-ingest script (process_sip) for images transforms.
Re-architected the cache code.
Moved transactions to read-only queries where possible.
Optimized slideshow retrieval/display.
Optimized many actions (e.g. FetchArticleAction, GetAverageRatingsAction, etc.).
Lots of Ambra code improvements.
Upgraded to Dojo 1.1.
Upgraded to Mulgara 1.2 (multi-core!).
Upgraded to Lucene 2.3.
Upgraded to ehCache 1.4.1.
Many Ambra configuration changes for ease of installation.

Features fixed in RC 0.9

Safara bugs fixed; ratings, drop-down menus.
Ingest fixes; ingest doesn’t change .mov to .qt; ingest handles .swf files.
Advanced search by past month & past three months working.
Search results with correct journal context.
XSL fixes.

Trackback URL for this post:

http://www.plos.org/cms/trackback/379

TOPAZ upgrade

from A Blog Around The Clock on Thu, 2008-07-17 17:09

Sunday Outage Sunday

Richard Cave — Mon, 14 Jul 2008 15:38:01 -0700

We experienced a hardware malfunction yesterday that caused the TOPAZ hosted journals to be offline from 4pm - 10pm PST.

James sent an email late afternoon yesterday indicating site errors on the PLoS journal websites. Soon after his email, the IT team started receiving SMS alerts. I assumed that something had occurred with the Topaz framework and started looking at the appropriate log files but couldn't find anything. I spent the requisite amount of time banging my head against the "site error" wall without success and I called up Russ for assistance. After a bit of digging through the server logs, he found the culprit - a drive had failed on the Mulgara server. This drive is part of a RAID 5 configuration, so we didn't lose any data but we also mysteriously lost the connection from the Mulgara server to the DAS array (disk storage for the Mulgara data).

We restarted the server but couldn't confirm that it was rebuilding the RAID correctly. I drove down to the colo, confirmed the drive failure and babysat the server until the platform was healthy. We'll swap out the defective drive on Wednesday during the migration to a pre-release of Topaz 0.9.

In case you missed the reference to U2's Sunday Bloody Sunday....

Trackback URL for this post:

http://www.plos.org/cms/trackback/377

Update on Performance Issues of PLoS Websites

Richard Cave — Tue, 18 Mar 2008 17:56:58 -0700

Performance of the websites hosted on Topaz has increased over the last two weeks with a variety of patches ported to the production servers. We still have an outstanding memory problem that requires a restart of the Topaz applications three times a day (these restarts usually occur around midnight, 8am and 4pm with a duration of less than 10 minutes). I feel that we're close to diagnosing the memory problem which is the last performance hurdle.

We're waiting for the Topaz developers to finish two outstanding tickets before we create a new release candidate (RC 0.8.2.2 for those that are counting). This release candidate will contain all of the patches, even better caching features and XSL/CSS bug fixes. We'll test the release candidate later this week and the production servers will run on this RC no later than Friday, March 28.

We also have a patch for search and will test the patch tomorrow. If testing goes well, then we will be able to place basic and advanced search back on the journal websites by the end of the week.

And now for something completely different.... We had a problem with one of our production servers Friday morning which caused an extended downtime. The Fedora server mysteriously stopped all network traffic and then rebooted. Russ and Josh are bringing the servers down for diagnostic tests and maintenance from 9-9:30pm PST. During this time, the journal websites will display the "undergoing site maintenance" page.

Trackback URL for this post:

http://www.plos.org/cms/trackback/341

TOPAZ update

from A Blog Around The Clock on Tue, 2008-03-18 19:13

The IT/Web has been hard at work to make TOPAZ, the platform for 5 out of 7 PLoS journals work smoothly again....

Performance Issues of PLoS Websites

Richard Cave — Mon, 3 Mar 2008 12:19:25 -0800

We’re continuing to experience slowness and intermittent downtime for the websites hosted on Topaz. At this time, the priorities for the IT and Topaz teams are to improve stability of the Topaz applications and increase the performance of the web sites.

The IT and Topaz teams are digging through many system files, configuration settings and code to improve the situation but it may take one or two weeks before we overcome the performance issues. Please bear with us as during this time as we make additional tweaks/upgrades to improve performance.

Hopefully we’ll address these issues quickly so that our users can enjoy the journal websites.

Trackback URL for this post:

http://www.plos.org/cms/trackback/334

Topaz Release Candidate 0.8.2.1

Richard Cave — Mon, 25 Feb 2008 15:20:10 -0800

Looks like I’ve been a bit remiss about posting release candidates on this blog as we’re on the third release candidate since RC 0.8. We’ve obviously been busy as the release candidate now has an extra numerical identifier. New features include:

minor and formal corrections on articles
advanced search
dynamic table of contents
dynamic journal archives

As with all release candidates, you may need to refresh your browser once or twice to get the new CSS changes.

And while all this development was going on, the I.T. and web teams were busy cleaning and ingesting +1400 articles to make way for PLoS Computational Biology, PLoS Genetics and PLoS Pathogens migration to the Topaz framework. Busy time at PLoS? A bit....

The most exciting features for this release are minor and formal corrections. Users can add a note and select "Correction" from the drop-down list. The PLoS staff will review the note and it will either stay as a note or be changed to a minor or formal correction. A minor correction is represented by a small red triangle "cloud" within the article text. A formal correction is signified the same way but a block appears above the title of the article displaying all formal corrections for the article. Clicking into a formal correction will also show the citation information.

These online corrections are another forward-looking feature for online STM publishing. In the past, PLoS would publish a new "formal correction article" with a link to the original article days/months after the original article published. But this "formal correction article" usually appeared in a different table of contents and could easily be overlooked by the user. Now the formal correction appears directly on the article text and all corrections for a specific article can be retrieved from a single URL. Next steps are to automatically add these corrections to the article PDF as a cover sheet when a user downloads the PDF and create a REST interface to easily retrieve the corrections for an article.

We’ve also re-indexed every published article for advanced search. Now you can search for text within the entire article, the abstract, article title or article references and search by subject category. The search fields also support a variety of search mechanisms include range searches, grouping and Boolean operators. We’ve a bit more work to do on search – look for a "quick article locator" block in the near future.

What else has been going on?!? A small crew at PLoS have been cleaning over 1400 articles from the PLoS Computational Biology, PLoS Genetics and PLoS Pathogens journals for ingest into the Topaz framework. This cleanup, spearheaded by Lynn Murdock, was necessary as we’ve been through a few rounds of XML changes in the last 2.5 years. The ingest of such a large volume of articles has been slowing PLoS ONE, PLoS Neglected Tropical Diseases and PLoS Hub – Clinical Trials down the last few weeks, but we’ve finally finished and the sites should run a bit more smoothly.

We’re still not down yet though – we’re migrating the PLoS community journals to the Topaz platform in short order. And then we’ll get to enjoy the final days of Strong Beer Month!

Here’s a rundown of all the features since RC 0.8 (saving the best for last).

Features implemented in Topaz RC 0.8.1

Display links to related articles on Article and Browse pages.
Administrative interface for collecting Articles into Issue and Issues into Volumes.
Administrative interface for attaching Issue Image article to Issues, and setting the current Issue.
Browse by Issue (Table of Contents) page.

Features implemented in Topaz RC 0.8.2:

Fixes to Browse by Issue (Table of Contents) to add more article types and next/previous links.
A dynamic Browse by Journal (Journal Archive) page.
A single rating field for all front matter articles (e.g. Editorial, Opinion).
Affiliation footnote links are displayed for group authors.
Minor FreeMarker and CSS fixes.
Updates to the volume and issue object models.
Migration from OSCache to ehcache for faster performance.
Administrative interface for viewing and releasing caches.

Features implemented in Topaz RC 0.8.2.1:

Minor corrections on articles.
Formal corrections on articles.
”View All” corrections page.
Admin interface for converting notes to corrections.
Citations for formal corrections.
Advanced search:

Search by author name
Search article text
Search by date published
Search by subject category

Support for note DOIs in the DOI resolver.
Dynamic "Most Viewed" tab for journal home pages.

Features fixed in RC 0.8.2.1

Nomenclature of annotations/discussions changed to notes/comments.
Dynamic Table of Contents with article caching.
Dynamic Journal Archive with article caching.
Re-index of past published articles for advanced search.
Slideshow accommodates small thumbnail issue images.
Overall rating correctly calculates front matter articles.
Order articles in the Table of Contents and RSS feeds ordered by date published and then by article DOI.
Correct ordering of issues in Table of Contents.
Ingest correctly identifies article XML file when multiple XML files are found in the article zip file.
Correctly display front matter ratings when a rating is deleted.
Table of Contents displays "Author Summary" links for research articles.

Trackback URL for this post:

http://www.plos.org/cms/trackback/331

TOPAZ Upgrade and other Big News from PLoS

from A Blog Around The Clock on Wed, 2008-02-27 07:25

It was a heroic (and sometimes nerve-wrecking) couple of months for the IT/Web team at PLoS, but the fruits of their labor are now visible to all. PLoS Computational Biology, PLoS Genetics and PLoS Pathogens have migrated, last night, onto...

Undergoing (Too Much) Site Maintenance

Richard Cave — Fri, 25 Jan 2008 17:11:20 -0800

Many users have seen the "undergoing routine site maintenance" page splashed across their browsers a bit too often recently when browsing to the PLoS ONE, PLoS Neglected Tropical Diseases or PLoS Hub Clinical Trials journals. We've taken some steps to resolve the problems and the TOPAZ developers are digging deep into code. We won't have a full fix in place until TOPAZ RC 0.8.2.1 is released at the end of February, but bandaids are in place that should make the site outages much less frequent. If you're interested in the gorey details, read on....

Russ and I were horrified witnesses to Java processes (that use ehcache) eating away at the 4Gb of memory on the publishing application (pubApp) servers. Eventually, all the memory would fill up and the TOPAZ application would turn into a zombie. To try and remedy the problem, we setup a cron job to automatically restart the pubApp every eight hours. But this caused a secondary problem - if Mulgara was in the middle of a transaction while the pubApp was restarting, the application would hang.

We have alerts setup any time that the pubApp hangs but now Russ and I were waking up in the middle of the night from the unpleasant sounds of SMS alerts inundating our cell phones. And after a few nights of that, we were ready to:

1. Throw the cell phones out the window at 4AM

2. Purchase more memory for the servers.

Yesterday, Josh and I installed 4Gb more RAM (total of 8gb) in the servers. We're still seeing the Java process eat up a bunch of memory, but it's not able to topple the 8Gb of RAM. We see ehcache having some problems when it overflows from memory to disk. And we still have automatic restarts a few times a day. But the sites should be more stable and we'll hopefully have this beast tamed by the next release.

Trackback URL for this post:

http://www.plos.org/cms/trackback/318

PLoS Blog update

Bora Zivkovic — Tue, 22 Jan 2008 10:55:05 -0800

Last week, we made a little upgrade to the PLoS Blog.

If you look at any individual post you will see that we added the "e-mail this page" and "Printer-friendly version" buttons on the bottom of each post.

We have also started allowing trackbacks on our posts. Just like comments, trackbacks will be moderated due to large amounts of spam that are still attacking our system. We check the approval cue for comments and trackbacks on the blog regularly, so yours will show up after a short lag (and if it does not, give me a heads-up by e-mail).

Now look at the right side-bar, where we have made some re-arrangements, some features got new names, and new features have been added.

We have added a new Channel - In The News - where we will highlight the media and blog coverage of our most popular articles.

We have added a Technorati widget so you can see who links to the blog.

Finally, we also added the Blogroll. At the moment, the front page will showcase five from our very long list of favorite blogs and we'll rotate them to eventually give them all their time in the sun. If you want to make suggestions for our list you can let me know.

This is just the first set of small changes on the PLoS Blog. Keep an eye on it as more changes will be coming in the future, making the blog better, prettier and easier to use.

Trackback URL for this post:

http://www.plos.org/cms/trackback/313

Zotero Translator for PLoS Articles

Richard Cave — Tue, 4 Dec 2007 09:27:33 -0800

The Zotero developers announced that Zotero is now compatible with all PLoS journals. The new Zotero translator allows users to capture metadata and full-text PDFs of PLoS articles. I've tried Zotero with PLoS ONE and PLoS NTD articles and it automatically gathers metadata information including the authors of an article, abstract, volume/issue and DOI. You'll need to download the latest Zotero 1.0.1 release to get the new PLoS translator.

There's probably a fair bit of people scratching their head and asking "What is Zotero?" Zotero is a great Firefox extension that allows you to collect, manage and cite your research sources. It works seamlessly in your browser and automatically collects metadata from an article, allows you to create tags about the information and take notes. Best of all - it's free!

The developers have created a huge number of translators for libraries and universities around the world as well as translators for Amazon and Flickr. Zotero also has integration with Microsoft Word and OpenOffice so you can easily cite something from your Zotero collection. On of the Zotero team members is also working on a PLoS bibliographic style export which should be released in the next version.

Zotero is a great application and I highly recommended giving it a try.

Trackback URL for this post:

http://www.plos.org/cms/trackback/295