AO3 News

Post Header

2018-06-17 11:15:45 -0400

Five Things an OTW Volunteer Said

Every month or so the OTW will be doing a Q&A with one of its volunteers about their experiences in the organization. The posts express each volunteer's personal views and do not necessarily reflect the views of the OTW or constitute OTW policy. Today's post is with Adi M., who volunteers as a translator.

How does what you do as a volunteer fit into what the OTW does?

Translation helps make the OTW's content accessible to as many people as possible, which is a fundamental part of the organization's vision. I know a lot of people in my country shy away from any English content, whether because they don't feel comfortable reading in English or because they simply don't understand it enough. Knowing my translations help bring people in my country closer to fandom is one of the best parts of the role.

What is a typical week like for you as a volunteer?

My team is still fairly new, so there's a lot of content to translate and many terms and protocols to discuss, so there's always work to do! I get my task from Translation staff, usually with a deadline of one week, and as soon as I finish a task there's another one waiting.

I also note down any new term that needs to be discussed, or any question I have for the rest of my team, and when the list gets long enough I contact my team to set a meeting to figure out everything. Some terms can be frustrating, but that's all part of the fun!

What made you decide to volunteer?

I have been in fandom for 7 years now, and have always loved to translate everything I could get my hands on. Joining the OTW as a translator brought two of my favorite things in the world together.

You volunteered this year to be a chat room moderator for International Fanworks Day. What was that like?

It was a great experience for me. I was hesitant to volunteer for it in the beginning, but as the day drew closer I got more excited, and in the end, I am very happy I did it. I had the chance to meet new people I don't share a fandom with, and it was amazing how we still managed to bond over fannish joys and ideas even without liking the same things. When we need volunteers for next year, I'll be signing up!

What fannish things do you like to do?

I (try to) write fics, and I can't manage without reading at least one fic a day. I also love translating fics from English to Hebrew, both to make them more accessible for Hebrew-speaking fans and to practice my translation skills.

Now that our volunteer’s said five things about what they do, it’s your turn to ask one more thing! Feel free to ask about their work in comments. Or if you'd like, you can check out earlier Five Things posts.

The Organization for Transformative Works is the non-profit parent organization of multiple projects including Archive of Our Own, Fanlore, Open Doors, Transformative Works and Cultures, and OTW Legal Advocacy. We are a fan run, entirely donor-supported organization staffed by volunteers. Find out more about us on our website.


Post Header

2018-06-09 12:40:25 -0400

Banner by caitie of a newspaper with the name and logos of the OTW and its projects on the pages

I. The GDPR is here!

If you visited the Archive of Our Own any time after May 25th, you ran into a pop-up requiring your consent to our updated Terms of Service (TOS). This notice was probably only one of hundreds of such requests you received from organizations worldwide over the past few weeks since the European Union’s new regulation on data protection, the General Data Protection Regulation (GDPR) went into effect. We’ve been working very hard to bring the OTW into compliance with the GDPR’s strict rules on data privacy and transparency. You can learn more about the GDPR’s impact on the OTW and how we use your data in our AO3, the GDPR and You post. Thanks to Board, Accessibility, Design & Technology, Development & Membership, Fanlore, Open Doors, Systems, Support, Tag Wrangling, Translation, TWC, Volunteers and Recruiting, and Webs for helping with this massive project! Special thanks and kudos to Board, Legal and committee chairs for preparing the updated TOS and a summary of the changes.


The Archive of Our Own reached 1.5 MILLION registered users on May 25th! Thanks so much to everyone who visits the site--you've helped it become what it is today. \o/

During May, Accessibility, Design & Technology was extremely busy dealing with some server issues caused by a combination of the ongoing Elasticsearch upgrade process and some GDPR-related changes, which briefly brought down AO3. Kudos to them for managing a huge month! For real-time updates on AO3, remember to follow @AO3_Status on Twitter.

AO3 Documentation added a new tutorial on importing text-based works, and updated several FAQs including Comments & Kudos, Your Account, Invitations, and Posting & Editing. These updates include information about gift works, our spam filters, what happens when you delete your account, and more.

Open Doors completed the imports of The Spooky Awards, a collection including X-Files fanfiction submitted to the Spooky Awards in 2003, 2004, and 2005, and Firefly's Glow, a collection of fanfiction from the Firefly fandom.

Policy and Abuse received over 800 tickets this month, up slightly from 700 tickets the previous month. By contrast, Support completed a second month with 1300 tickets, down from previous months.

III. Elsewhere at the OTW

Elsewhere at the OTW in May, Systems dealt successfully with a number of hardware failures, including a server failure that briefly took down the TWC site. [log?] Systems are currently putting things back together, as well as working on some improvements to our monitoring setup to reduce the chance of a repeat occurrence.

Legal's advocacy work didn't stop in the face of GDPR-related work. In preparation for submitting Comments to the Canadian Government as part of Canada’s review of its Copyright Act, Legal put out a call for Canadian stories about how fans have benefited from fanworks, fan communities, and fair dealing laws. Canadian fans responded with dozens of powerful, moving, and very helpful stories. Legal sends a huge thank you to everyone who sent in stories!

IV. 2018 Elections

The 2018 election will be held August 10-13. Some key dates coming up in June: The membership deadline to be eligible to vote is June 30, and candidates running for the Board of Directors will be announced on June 18. Check out Elections’ Twitter and Tumblr for updates throughout the election season.


As of 23 May, the OTW has 662 volunteers. \o/ Recent personnel movements are listed below:

New Committee Chairs: briar_pipe (Elections) and Jenny McDevitt (Elections)
New Committee Staff: 1 Development & Membership staffer

Departing Committee Chairs: Araise Solace (Elections)
Departing Committee Staff: Genevieve Marshall (Development & Membership) and 1 other Development & Membership staffer, Connie Pini (Volunteers & Recruiting), Cryllia (Tag Wrangling), Sammie Louise (Translation), 1 Communications staffer, 1 Policy and Abuse staffer, 1 Tag Wrangling staffer
Departing Communications Volunteers: 1 Communications volunteer
Departing Tag Wrangler Volunteers: aethel, CirqueDuCinq, IPutTheSassInAssassin, pearwaldorf, Perydot, plantpun, ShayBriar, Trixxster103 and 27 other Tag Wrangler volunteers
Departing Translation Volunteers: someplacelikebolivia and 3 other Translation volunteers

For more information about the purview of our committees, please access the committee listing on our website.

The Organization for Transformative Works is the non-profit parent organization of multiple projects including Archive of Our Own, Fanlore, Open Doors, Transformative Works and Cultures, and OTW Legal Advocacy. We are a fan run, entirely donor-supported organization staffed by volunteers. Find out more about us on our website.


Post Header

2018-06-08 12:24:53 -0400

Banner by Erin of a spotlight shining the OTW logo behind the text spotlight on legal issues

OTW Legal has been busy working on three advocacy projects around the world--in Canada, Australia, and the European Union. We want to thank you for your help in Canada, and let you know what’s happening and what you can do to help in Australia and the European Union.


Thanks to the many amazing stories we received from Canadian fans, we submitted a comment to the Canadian Parliament’s copyright law review about the power and benefit of laws that allow and promote transformative works. You can find our submission here.


Now we’re working on a submission for Australia! The Australian government is evaluating Australian copyright law and is considering expanding fair dealing provisions or adopting a fair use standard. OTW Legal is writing in support of these changes and would love your help. If you’re Australian and have expressed yourself, gained skills, been part of creative communities, or otherwise experienced the benefits of being able to create transformative works that would benefit from flexibility in Australian copyright law, we’d love to hear your stories. Please send them by June 25 by using our contact form. (Feel free to use a pseudonym if you don't want us to share your personally identifying information.)

European Union

OTW Legal and our allies have been active in fighting on fan-unfriendly legal proposals in the EU. Since these proposals were introduced in 2016, OTW Legal has submitted comments opposing them and has joined in calls for action against them. We’ve managed to hold them off so far and encourage some revisions, but a key vote will be happening in the European Parliament’s JURI committee on 20/21 June that could have a significant impact on the Internet and fan sites. In particular, two provisions of the current proposal would be bad for fans. Article 11 would impose a "link tax" that would make it more expensive for many websites to operate, and Article 13 would impose mandatory content-filtering requirements on websites that host user-generated content. These provisions have been hotly debated and revised a bit since the last time we reported on them. (For more on recent revisions and debates, see these discussions by the Electronic Frontier Foundation and the Hogan Lovells Firm) But despite revisions, they’re still bad deals for fans. Importantly, they don’t preserve the "safe harbors" that websites rely on to operate, and they don’t include user-generated content exceptions.

Without safeguards for user-generated content, Article 13 would require your favourite websites to implement systems that monitor user-generated content and automatically remove any content that could potentially infringe upon copyright, giving publishing giants the power to block your online expression. Sites like YouTube, Tumblr, GitHub, Soundcloud, etc., could be required to block the upload of content based on whether it has been "identified" by big corporations, rather than based on its legality. The law is still being debated, and it is difficult to predict how it would impact the OTW’s projects, including the Archive of Our Own, if it is passed. Regardless of how this vote comes out, the OTW will work as hard as we can to keep the Internet fan-friendly. But we need your help. The most effective thing you can do right now is contact your Member of European Parliament. You can use one of these tools to e-mail your MEP or call your MEP to tell them that having user-generated content on the internet is important to you.

Here’s what you can tell them: Without safe harbors for user-generated content, Article 13 of the Copyright Directive would stifle free expression on the Internet. We don’t want mandatory filtering. Algorithms don’t understand limitations and exceptions to copyright like parody, public interest exceptions, fair use, or fair dealing, and we don’t want our non-infringing videos, website posts and art blocked because of a biased algorithm created by big corporations. We want the law to protect user-generated works, not harm them.

OTW Legal will keep fighting for fan-friendly laws!


Post Header

2018-06-06 13:00:03 -0400

OTW Recruitment banner by Erin

Would you like to help your fellow fans use the AO3? Do you like planning events? Are you passionate about organization or love fandom in general? The Organization for Transformative Works is recruiting!

We're excited to announce the opening of applications for:

  • Communications Staff – Event Coordinator - closing Wednesday 13 June 2018 23:59 UTC
  • Fanlore Graphics Designer Volunteer - closing Wednesday 13 June 2018 23:59 UTC
  • Support Staff - closing Wednesday 13 June 2018 23:59 UTC
  • Tag Wrangling Volunteer - closing Wednesday 13 June 2018 23:59 UTC or at 80 applications

We have included more information on each role below. Open roles and applications will always be available at the volunteering page. If you don't see a role that fits with your skills and interests now, keep an eye on the listings. We plan to put up new applications every few weeks, and we will also publicize new roles as they become available.

All applications generate a confirmation page and an auto-reply to your e-mail address. We encourage you to read the confirmation page and to whitelist our email address in your e-mail client. If you do not receive the auto-reply within 24 hours, please check your spam filters and then contact us.

If you have questions regarding volunteering for the OTW, check out our Volunteering FAQ.

Communications Staff – Event Coordinator

Do you enjoy event planning? Have you ever celebrated International Fanworks Day? OTW Communications is looking for someone to help coordinate events large and small.

The Communications committee is the central information distribution arm of the OTW, responsible for the distribution of information internally to OTW personnel and externally to the general public, the media, fans, and other fannish organizations. Usually, that information is news-related, but sometimes it's celebration-related. The Event Coordinator is responsible for planning and executing organization-wide events throughout the year. Larger events include International Fanworks Day and the OTW's anniversary. Smaller events include single-day celebrations such as SysAdmin Day.

Applications are due Wednesday 13 June 2018 23:59 UTC

Fanlore Graphics Designer Volunteers

Would you like to help Fanlore reach more fans and get new editors? Do you have graphic design skills and enjoy creating social media content? If so, we need your help! The Fanlore team needs designers to create graphics and banners for Tumblr posts, tweets, and other Fanlore announcements, to help us reach more fans and potential editors. We have a lot of amazing fan history content, but we need your help to help others find out about it. If you think you might enjoy that, come and join us!

Applications are due Wednesday 13 June 2018 23:59 UTC

Support Staff

If you've spent time figuring out how to make the Archive dance (or are willing to press the button to see what happens), are patient in the face of strange questions, can self-motivate, and are interested in helping your fellow fans, we would love to hear from you! The Support team is responsible for handling the feedback and requests for assistance we receive from users of the Archive. We answer users’ questions, help to resolve problems they’re experiencing, and pass on information to and from coders, testers, tag wranglers and other teams involved with the Archive.

Applications are due Wednesday 13 June 2018 23:59 UTC

Tag Wrangling Volunteer

The Tag Wranglers are responsible for helping to keep the millions of tags on AO3 in some kind of order! Wranglers follow internal guidelines to choose the tags that appear in the filters and auto-complete, which link related works together. (This makes it easier to browse and search on the archive, whether that’s Steve/Tony with tentacles or g-rated Rose/Kanaya fluff.)

If you’re an experienced AO3 user who likes organizing, working in teams, or excuses to fact-check your favorite fandoms, you might enjoy tag wrangling! To join us, click through to the job description and application form.

Please note: You must be 18+ in order to apply for this role. Additionally, we’re currently looking for wranglers for specific fandoms only, which will change each recruitment round. Please see the application for which fandoms are in need.

Applications will be accepted until Wednesday 13 June 2018 23:59 UTC or until 80 submissions have been received.

Apply at the volunteering page!


Post Header

Our previously announced search engine upgrade will not only provide some much-needed stability improvements, but also bring exciting changes to the way you search, filter, and browse for works and bookmarks on the Archive! These changes range from making current functionality more user-friendly to adding new options and overhauling existing features. We hope you enjoy these improvements to the way you find new things to read or look at or listen to!

Work search and filtering

Over the years, you've sent us a lot of suggestions for enhancing work search and filters, and as fellow Archive users, we're excited to finally implement some!


Preview of the new work filters.

Detail of the new work filters, focusing on tag exclusion, crossover handling, completion status, and word count. The full screenshot (linked) shows the whole list of tag types to include or exclude, and additional options such as a date range and work language.


  • Exclusion filters: Now you can filter things out of work listings the same way you filter them in: just enter their names in the "Other tags to exclude" field or choose them from a list of the most popular tags. This might have been our most requested feature, and we think our solution combines familiarity, flexibility, and accessibility. The new fields are only available in the filters for now, but you can keep using the old tag exclusion method on the Work Search page -- we'll be updating that interface soon!

  • Crossovers: You can now search for crossovers, or choose to exclude them. Crossovers are defined here as works with at least two unrelated fandom tags, as determined by how tags are wrangled.

  • Work Status: Previously, you could opt to show completed works only. Now you can also search specifically for works in progress, if you're into that kind of thing.

  • Date Updated: To limit your results to works posted or updated in a particular time period, the filters now include a date picker to help you choose start and end dates for your query.

  • Word Count: Lastly, we've added a simple way to look for works in a particular word count range. \o/

Everything you select in the filtering sidebar will continue to narrow down your set of results (i.e. AND filtering), as opposed to including more works that might have one or more of your selected tags (i.e. OR filtering).

Bookmark search and filtering

With the old bookmarking code, wrangling changes to large tags could put a strain on the Archive's servers. Thankfully, Elasticsearch 6 provides a much better way to handle our over 74 million (!) bookmarks, so we've given the code an extensive rewrite. However, in addition to giving bookmarks shiny new exclusion filters, we've had to make some changes to searching, browsing, and filtering bookmarks.


Preview of new bookmark listings.

A truncated list of bookmarks, all of the same work, listed under the Veronica Mars tag. The full screenshot (linked) shows the five most recent bookmarks of the work, with bookmarker's notes and tags, and a link to access all bookmarks.


  • Redesigned bookmark listings for tags and collections: No more scrolling past hundreds of bookmarks for the same popular work or series when you're browsing bookmarks in a collection or tag -- each bookmarked item will now be listed only once, with the details of its five most recent bookmarks beneath it. (If an item has more than five bookmarks, there will be an "All Bookmarks" link so you can check out the rest.) User bookmark pages and Bookmark Search results will continue to list bookmarks the way they always have.

  • Bookmarker's tags: With the new code, using the "Angst" tag on a bookmark will no longer add that item to the bookmark page for "Angst," nor will the item be included in the results when filtering a list of bookmarks for the Additional Tag "Angst." Now if you want to see everything bookmarkers have tagged with "Angst," you'll need to use the "Bookmarker's tags" field on the Bookmark Search page.

    If you use the tags for a personal rating system or reminders to comment on a bookmarked work, for example, you can still filter your bookmarks just as before! There will also be "Bookmarker's tags" checkboxes and autocomplete fields on the bookmark pages for collections and tags.

  • Searching bookmarks: While it's still possible to search bookmarks, the performance improvements we've made required us to divide the search into two fields: one for information on bookmarked items (e.g. work tags, titles, and summaries) and another for information on the bookmarks themselves (e.g. bookmarkers' tags and notes).


Preview of new bookmark filters.

Detail of new bookmark filters, showing exclusion options for tags on the work and tags added by the bookmarker. Further options include search fields for work and bookmark information, and checkboxes to only look for recs or bookmarks with notes. The full screenshot (showing all filterable tags) is linked.

People search

You can now search for fellow users who have created works in your favorite fandoms! \o/ The new "Fandom" field in our People Search form suggests canonical tags as you start typing, so you don't have to worry about getting the name of your fandom exactly right. The more fandoms you put in, the more you'll narrow down the set of results.


Preview of the new people search.

The new People Search form, focusing on a search by fandom. The pseud testy has 970 works and 66 bookmarks, including 1 work for the fandom in question. Additionally, one can also search by username or profile content.


To make your pseud stand out in the search results, you can follow these instructions on editing pseuds to add a little blurb and user picture. (You can also add links and information about yourself, your fannish history, transformative works policy, or social media presence to your account profile, as laid out in our Profile FAQ.)


Since even a state-of-the-art search engine like Elasticsearch suggests some gentle limits to the number of search results displayed (by throwing an error after a point), you will only be able to sift through 5,000 pages at a time, which means 100,000 works or bookmarks.

Keep in mind, however, that the Work Search form lets you change how results are sorted. So if you want to access the oldest works in a fandom -- something that wouldn't show within the first 100,000 results in a big fandom -- you can sort by Date Updated in Ascending order to make sure you get the results you’re looking for! (To sort a filtered list of works, use sort:>posted in the "Search within results" field.)

Some things to try

Once the new search is enabled for everyone, here are a few suggestions for getting to know the new options:

  • Rating: For a list of works rated General Audiences or Teen and Up, don't select a rating to include -- instead check "Explicit", "Mature", and "Not Rated" to be excluded, and all your results should be safe for work! (You can reverse this for your bedtime reading.)

  • Word Count: To get all the actual drabbles in your chosen fandom, enter 100 in both the "From" and "To" fields. (#DrabblePurist)

  • Crossovers: Are you a big fan of canon-verse fic? You can eliminate all AUs by excluding the "Alternate Universe" tag, and crossovers or fusions by selecting the "Exclude crossovers" option.

  • Date: Working on your list of favorite podfics of 2017? Go to the Podfic tag and put 2017-01-01 and 2017-12-31 in the "From" and "To" fields under "Date Updated", to refresh your memory!

  • Status: You've consumed all the completed works for your shiny new OTP? You have skipped the works in progress, because you needed a happy ending now? Nothing left to read? Well, gird your loins and check the "Only incomplete works" ticky to take a second look at those!


Post Header

Outlines of a man and woman speaking with word bubbles, one of which has the OTW logo and the other which says 'OTW Announcement

You may have heard that the European Union's new regulation on data protection, the General Data Protection Regulation (GDPR), is coming into effect on May 25. It has effects on sites and organizations all around the world, including Archive of Our Own.

The GDPR's primary focus is on data privacy and transparency, and on giving users more control over what their data is used for. The AO3 team has been working hard to comply with all GDPR requirements, and to make more information available on how we use your data across the site. Here's what you need to know about upcoming changes.

Terms of Service

We've recently updated our Terms of Service. In this update, among other changes:

  • We added more explicit information about how we use the data you provide to offer certain features—such as keeping track of where you've already left kudos—and to display the content you provide in your works, comments, and profile pages.
  • We added a new Age Policy (more on this below).

A summary of all changes and a link to the full document can be found in the Terms of Service update announcement. Everyone who uses AO3 will be required to agree with the new Terms.

Age Policy

We have created a new Age Policy, which is described in our updated Terms of Service. This restricts how old you must be in order to create or own an AO3 account.

While the general minimum age to create an AO3 account remains at 13 years old, there are specific requirements for those who reside in the EU. If you are under 16 years old and live in the European Union, please check your country's current age of consent for data processing.

You must be of age to give us consent to use your data in order to use the Archive, or else we can't legally, for example, store your kudos or comments (even as a guest). Our Policy & Abuse team will contact users if there are reasons to believe that they are under their country's age of consent for data processing, and may delete their accounts as needed. Now is a good time to review your profile and works for any outdated information about your age!

We're sorry for any inconvenience this causes any European users who will now be under the age of consent in their country. We'll be glad to welcome you back when you're of age to use AO3 once again!


Starting soon, a banner will pop up requesting that you accept our new Terms of Service. It will highlight some important aspects of the data processing we do here on AO3 regarding your works, comments and so on. That way, everyone can provide informed consent when they accept the new Terms.

An important note about this: According to the GDPR, information about personal relationships, religious beliefs, political affiliations, and sexual orientation is defined as "Special Categories of Personal Data". We know our users can sometimes mention information related to this in their profiles, works, notes, tags, comments, bookmarks, and so on. Please be mindful of your privacy when entering this information in publicly-accessible pages. Any information you publish in openly accessible spaces will be publicly available to everyone.

Above all, AO3 remains dedicated to being a place to preserve and share fanworks. We're proud to offer this service by fans and for fans, in a format that is free and lets you retain full control of your works. We don't display ads or monetize user data in any shape or form and don't ever intend to.

Learn More

For more information on the GDPR, please check out The European Commission's Rights for Citizens or Wikipedia's information on the GDPR.

If you'd like to know more about how AO3 and the nonprofit behind it, the Organization for Transformative Works, have prepared for GDPR, please contact the OTW Legal team. If you have any specific questions on how these changes affect you or the AO3, or if your native language isn't English and you need some assistance to understand the changes we've made, please feel free to get in touch with our Policy & Abuse committee.

If you have any questions about these upcoming changes, please feel free to leave them in a comment below.


Post Header

2018-05-18 16:19:38 -0400

Following our May 10 deploy, the Archive experienced a number of issues stemming primarily from increased load during the Elasticsearch upgrade process.

As we noted in our March downtime post, the Archive hasn't been running at full strength due to this upgrade. Compounding the issue, it has taken significantly longer than planned to get the new code deployed to production, and we are now entering one of the more active times of the year. (Our daily page views for a Sunday -- our busiest day -- are over 29 million, and the normal load on our database servers is over a million queries per minute.)

You can find more details on the current state of the Archive's servers below, along with a rough timeline of the issues we experienced between Thursday, May 10, and Monday, May 14. However, the main takeaway is these issues are likely to continue until the Elasticsearch upgrade is completed and our network capacity is increased. We're very grateful for the support and patience you've shown, and we look forward to finishing our upgrades so we can provide you with a stable Archive once more.

Background: Server state

We normally have five Elasticsearch servers, but late last year we turned one of our front end machines into an Elasticsearch server, allowing us to divide these six machines into two groups: one three-machine cluster for the production site, and another for testing the upgraded code.

Having only three Elasticsearch servers meant the site experienced significant issues, so on April 11, we reprovisioned one of our old database servers, which had been producing web pages, as an Elasticsearch server in the production cluster.

In addition to the ongoing Elasticsearch upgrade, our Systems team recently completed a major overhaul intended to help with our long term stability and sustainability. Between November 2017 and March 2018, they reinstalled all the application servers, web front ends, and new Elasticsearch systems with a new version of the Debian (Stretch) operating system using FAI and Ansible. This meant rewriting the configuration from the ground up, since we had previously used FAI and CFEngine. They also upgraded various other packages during this process, and now all that's left to upgrade for the Archive are the database servers.


May 10

16:25 UTC: We deploy the code update that will allow us to run the old and new Elasticsearch code simultaneously. (We know the new version still has a few kinks, and we expect to find more, so we're using a Redis-based system called rollout to make sure internal volunteers get the new code while everyone else gets the old version.) Because this is our first deploy since the application servers have been reinstalled, the deploy has to be done by hand.

16:56 UTC: We turn on the new Elasticsearch indexing.

21:03 UTC: We notice -- and fix -- some issues with site skins that resulted from doing a manual deploy.

May 11

05:00 UTC: We see large amounts of traffic on ao3-db06, which is both the Redis server we use for Resque and the MySQL server responsible for writes. We mistakenly believe the traffic is caused by the number of calls to rollout to check if users should see the new filters.

05:36 UTC: We increase the number of Resque workers.

10:06 UTC: The Resque queue is still high, so we increase the number of workers again.

21:00 UTC: We no longer believe the increased traffic is due to rollout, so we turn the new indexing off and schedule 45 minutes of downtime for 06:15 UTC the following morning.

May 12

06:15 UTC: In order to mitigate the extra traffic, we move Redis onto a second network interface on ao3-db01. However, routing means the replies return on the first interface, so it is still overwhelmed.

06:42 UTC: We extend the downtime by 30 minutes so we can change the new interface to a different network, but replies still return on the wrong interface.

07:26 UTC: Since we've used up our downtime window, we roll the change back.

After that, we spend large parts of the day trying to figure out what caused the increase traffic on ao3-db06. With the help of packet dumps and Redis monitoring, we learn that indexing bookmarks on new Elasticsearch is producing a large number of error messages which are stored in Redis and overwhelming the network interface.

May 13

Our coders spend most of Sunday trying to determine the cause of the Elasticsearch errors. We look at logs and try a number of solutions until we conclude that Elasticsearch doesn’t appear to support a particular code shortcut when under load, although it's not clear from the documentation why that would be.

20:45 UTC: We change the code to avoid using this shortcut and confirm that it solves the issue, but we do not resume the indexing process.

23:45 UTC: The Resque Redis instance on ao3-db06 freezes, likely due to load. As a result, some users run into errors when trying to leave comments, post works, or submit other forms.

May 14

06:30 UTC: We restart Redis, resolving the form submission errors. However, we begin to receive reports of two other issues: downloads not working and new works and bookmarks not appearing on tag pages.

16:25 UTC: To help with the download issues, we re-save our admin settings, ensuring the correct settings would be in the cache.

16:34 UTC: Now we look into why works and bookmarks aren't appearing. Investigating the state of the system, we discover a huge InnoDB history length (16 million rather than our more normal 2,000-5,000) on ao3-db06 (our write-related MySQL server). We kill old sleeping connections and the queue returns to normal. The server also returns to normal once the resultant IO has completed.

16:55 UTC: Bookmarks and works are still refusing to appear, so we clear Memcached in case caching is to blame. (It's always -- or at least frequently -- caching!)

17:32 UTC: It is not caching. We conclude Elasticsearch indexing is to blame and start reindexing bookmarks created in the last 21 hours.

17:43 UTC: New bookmarks still aren't being added to tag listings.

17:54 UTC: We notice a large number of Resque workers have died and not been restarted, indicating an issue in this area.

18:03 UTC: We apply the patch that prevents the bookmark indexing errors that previously overwhelmed ao3-db06 and then restart all the unicorns and Resque workers.

18:43 UTC: Once everything is restarted, new bookmarks and old works begin appearing on the tag pages as expected.

19:05 UTC: The site goes down. We investigate and determine the downtime is related to the number of reindexing workers we restarted. Because we believed we had hotfixed the issue with the reindexing code, we started more reindexing workers than usual to help with the indexing process. However, when we started reindexing, we went above 80% of our 1 Gbit/sec of ethernet to our two MySQL read systems (ao3-db01 and ao3-db05).

19:58 UTC: After rebalancing the traffic over the two read MySQL instances and clearing the queues on the front end, the indexers have stopped, the long queues for pages have dissipated, and the site is back.


  • We will either need multiple bonded ethernet or 10 Gbit/sec ethernet in the very near future. While we were already expecting to purchase 10 Gbit networking in September, this purchase may need to happen sooner.
  • Although it has not been budgeted for, we should consider moving Redis on to a separate new dedicated server.

While we are running with reduced capacity in our Elasticsearch cluster and near the capacity of our networking, the reliability of the Archive will be adversely affected.


Post Header

2018-05-17 12:42:48 -0400

Spotlight on Legal Issues

Are you Canadian? OTW Legal wants your stories about being a fan!

Over the years, OTW Legal has spoken for fans and fanwork creators in comments to governments around the world including the U.S., the E.U., Canada, Australia, and South Africa. And we want your help to keep doing that! The Canadian government is currently conducting a review of the Canadian Copyright Act. The Parliamentary Committee responsible for the review has already received some comments complaining about fan-friendly laws like the 2012 expansions to fair dealing and the User-Generated Content exception to copyright infringement. OTW Legal wants to show Parliament the other side of the story: the important value that Canadian fanwork creators get from being able to create transformative works.

Are you Canadian and have you expressed yourself, gained skills, been part of creative communities, or otherwise experienced the benefits of being able to create transformative works--works that are legal to create in Canada because of fair dealing and the UGC exception? If so, OTW Legal would love to hear your stories. We need to submit our comments soon, so please send our Legal Advocacy team your stories about how being able to create fanworks and belong to fan communities has helped you, by the end of May. (Feel free to use a pseudonym if you don't want us to share your personally identifying information.) We’ll use your stories to support our legal advocacy work in Canada and worldwide.



Pages Navigation