AO3 News

Post Header

Published:
2019-02-09 17:01:58 -0500
Tags:

Soon we'll be replacing our current method for generating work downloads with the Calibre command line interface. Our current patchwork system of third-party tools and custom code comes with a number of bugs and limitations, and moving to a well-maintained and documented application like Calibre will enable us to better maintain and even enhance the download feature going forward.

Because this is an entirely new system, there will be some differences between old and new files. We've provided an overview of these changes and some previews below, but we're still testing these changes, so there may be some minor differences between the previews and the final versions. Additionally, exactly how a file looks and behaves may vary depending on your device, software version, and settings.

HTML & PDF

HTML files serve as the basis for all of our download files, so it was necessary to make a few tweaks to help Calibre create tables of contents for other download types. Most of these tweaks are invisible, but we did make one visible change: the work title is now included at the top of the file, above the URL. Aside from these minor changes, HTML downloads will remain the same.

Because generating PDF downloads with Calibre would require us to install additional software on our servers, we will continue to use wkhtmltopdf to create PDFs, although we have upgraded it to a newer version. Like HTML downloads, PDFs will now include the title at the top of the first page but otherwise remain the same.

EPUB & MOBI

The biggest changes will be to MOBI and EPUB files, which are currently very different from each other due to limitations that existed when our system was created nearly ten years ago. The new code will give these files a more consistent look and feel and allow readers to take advantage of some features ebook readers provide.

EPUB

A current EPUB open in the iPhone Books app. The work meta is displayed in two columns inside a box. A new EPUB open in the iPhone Books app. The work meta is displayed as a bold heading, e.g. Rating, above the relevant tags.

The new EPUB files will do away with the two-column display of work meta (i.e. tags and statistics), which doesn't work well for small screens, lots of tags, or large font sizes.

This update will also fix the bug where EPUB files were missing chapter endnotes.

You can download an example of the new EPUB files to preview on your device.

MOBI

First page of a current MOBI file on the iPhone Kindle app, cropped to show the table of contents links provided at the top. A new MOBI file on the iPhone Kindle app, cropped to show the file's table of contents in the app menu.

Our current system provides a list of links on the first page as a makeshift table of contents, but Calibre will create files designed to work with your device or app's built-in table of contents function. (As you navigate with the shiny new table of contents, you may also notice each chapter begins on a fresh page.)

The text of a chapter in a current MOBI file on the iPhone Kindle app. The text of a chapter in a new MOBI file on the iPhone Kindle app.

Calibre will also add margins to prevent the text from running against the edge of the page. Additionally, we've switched from using indented book-style paragraphs to using vertical margins to create blank lines between paragraphs, bringing MOBI files in line with how text is formatted on the site and in other download files.

This update also brings a major improvement for internationalization: MOBI files of works written in non-Latin alphabets will no longer be automatically transliterated! Please note, however, that MOBI offers only limited support of right-to-left languages like Hebrew and Arabic, so Kindle users will need to use AZW3 to read works in those languages.

You can download an example of the new MOBI files to preview on your device.

AZW3

We'll also be adding a new download format: AZW3, which was created by Amazon for use with Kindle devices and apps. It's a newer format than MOBI, and it will allow us to offer Kindle users downloads of works in right-to-left languages, but it will otherwise look the same.

AZW3 downloads will be available on all works, but if you rely on Send to Kindle to transfer files to your device, you'll need to stick with a different format: Amazon doesn't allow you to send this file type. However, you can still transfer AZW3 files to your Kindle by downloading them to your computer and connecting your device.

If you're using the Kindle app on an Android device, you can move AZW3 files you've downloaded to your SD card's Kindle folder in order to access them. Unfortunately, there doesn't appear to be a workaround for the iOS Kindle app, so iPhone and iPad users will need to use another download format or a different, non-Kindle app.

Deploy date

While this upgrade won't immediately resolve all of the existing issues with downloads, it's an important part of the process, and it will ultimately make the Archive more secure and more maintainable. (After the deploy, our release notes will have more detail on specific bugs this change has fixed.)

We're still in the testing stages, but we plan to deploy these changes sometime after February 11. When the time comes, we'll announce it on our @AO3_Status Twitter account.

No downtime is expected, but if you have a work already loaded in your browser at the time of the update, you'll need to refresh the page to download it.

Comment

Post Header

🎁

Thanks to a long-planned database upgrade, the AO3 now lets you use all the latest emoji, as well as extended character sets for several languages (such as some of the rarer characters used for Chinese and Japanese) in your works, comments, or profile. 🎉🍾🎈🎊🎇. In fact, all text fields now support the UTF8MB4 character set, except for a few we've limited to Latin characters and some punctuation marks, like pseuds or collection names.

🔧

UTF8MB4 encoding only became available a few years after the Archive's inception, so we already had a pretty sizeable database using the older encoding when the upgrade became inevitable. After our recent server work, which included an upgrade to MySQL 5.7, people trying to post a work or comment with an unsupported character (e.g. one of the newer emoji) would be served an unhelpfully blank error page, exacerbating the problem.

However, migrating our entire 617GB database would have been a massive undertaking, likely necessitating several days of downtime. Fortunately, we were able to come up with a game plan that allowed us to keep downtime to a minimum by making changes to a new, empty copy of a database table while the "old" table with all the data was still in use. Once the new table was ready, we copied the data over and put the new table to use, then moved on to the next table.

😱

Thanks to the diligent efforts of our volunteer sysadmin james_, we completed the work in a little over a week and with no data lost. However, we ran into an unexpected problem with one of the constraints we put on our database to prevent the creation of multiple tags with identical names.

Due to a change in the way data is compared, some characters such as ss and ß are now treated as identical. That means that when we copied a tag like "Weiss Kreuz" to the new system, and then a few seconds later tried to copy "Weiß Kreuz," the second tag would be considered a duplicate and get quietly dropped. This made works originally posted to the dropped tag hard to find for a few days, since the tag didn't appear on the work anymore.

Once we discovered the issue and tracked down the cause, we were able to use a database backup to reconstruct the changes we'd made up to the point of the final swap, so we could compare the two tables and restore the lost tags.

👍

Together with our tag wranglers, we have put all the affected tags back into the system, so they will now appear on their works again. We are sorry if your works or bookmarks were affected by this issue!

Even though our upgrade is now complete and our database allows all of these new characters, keep in mind that some devices and browsers, as well as apps and e-readers relying on the MOBI format, might not display them properly. However, we hope you'll enjoy all your new emoji options and leave some extra-colorful love for creators in your comments! ❤🧡💛💚💙💜

Comment

Post Header

Our previously announced search engine upgrade will not only provide some much-needed stability improvements, but also bring exciting changes to the way you search, filter, and browse for works and bookmarks on the Archive! These changes range from making current functionality more user-friendly to adding new options and overhauling existing features. We hope you enjoy these improvements to the way you find new things to read or look at or listen to!

Work search and filtering

Over the years, you've sent us a lot of suggestions for enhancing work search and filters, and as fellow Archive users, we're excited to finally implement some!

 

Preview of the new work filters.

Detail of the new work filters, focusing on tag exclusion, crossover handling, completion status, and word count. The full screenshot (linked) shows the whole list of tag types to include or exclude, and additional options such as a date range and work language.

 

  • Exclusion filters: Now you can filter things out of work listings the same way you filter them in: just enter their names in the "Other tags to exclude" field or choose them from a list of the most popular tags. This might have been our most requested feature, and we think our solution combines familiarity, flexibility, and accessibility. The new fields are only available in the filters for now, but you can keep using the old tag exclusion method on the Work Search page -- we'll be updating that interface soon!

  • Crossovers: You can now search for crossovers, or choose to exclude them. Crossovers are defined here as works with at least two unrelated fandom tags, as determined by how tags are wrangled.

  • Work Status: Previously, you could opt to show completed works only. Now you can also search specifically for works in progress, if you're into that kind of thing.

  • Date Updated: To limit your results to works posted or updated in a particular time period, the filters now include a date picker to help you choose start and end dates for your query.

  • Word Count: Lastly, we've added a simple way to look for works in a particular word count range. \o/

Everything you select in the filtering sidebar will continue to narrow down your set of results (i.e. AND filtering), as opposed to including more works that might have one or more of your selected tags (i.e. OR filtering).

Bookmark search and filtering

With the old bookmarking code, wrangling changes to large tags could put a strain on the Archive's servers. Thankfully, Elasticsearch 6 provides a much better way to handle our over 74 million (!) bookmarks, so we've given the code an extensive rewrite. However, in addition to giving bookmarks shiny new exclusion filters, we've had to make some changes to searching, browsing, and filtering bookmarks.

 

Preview of new bookmark listings.

A truncated list of bookmarks, all of the same work, listed under the Veronica Mars tag. The full screenshot (linked) shows the five most recent bookmarks of the work, with bookmarker's notes and tags, and a link to access all bookmarks.

 

  • Redesigned bookmark listings for tags and collections: No more scrolling past hundreds of bookmarks for the same popular work or series when you're browsing bookmarks in a collection or tag -- each bookmarked item will now be listed only once, with the details of its five most recent bookmarks beneath it. (If an item has more than five bookmarks, there will be an "All Bookmarks" link so you can check out the rest.) User bookmark pages and Bookmark Search results will continue to list bookmarks the way they always have.

  • Bookmarker's tags: With the new code, using the "Angst" tag on a bookmark will no longer add that item to the bookmark page for "Angst," nor will the item be included in the results when filtering a list of bookmarks for the Additional Tag "Angst." Now if you want to see everything bookmarkers have tagged with "Angst," you'll need to use the "Bookmarker's tags" field on the Bookmark Search page.

    If you use the tags for a personal rating system or reminders to comment on a bookmarked work, for example, you can still filter your bookmarks just as before! There will also be "Bookmarker's tags" checkboxes and autocomplete fields on the bookmark pages for collections and tags.

  • Searching bookmarks: While it's still possible to search bookmarks, the performance improvements we've made required us to divide the search into two fields: one for information on bookmarked items (e.g. work tags, titles, and summaries) and another for information on the bookmarks themselves (e.g. bookmarkers' tags and notes).

 

Preview of new bookmark filters.

Detail of new bookmark filters, showing exclusion options for tags on the work and tags added by the bookmarker. Further options include search fields for work and bookmark information, and checkboxes to only look for recs or bookmarks with notes. The full screenshot (showing all filterable tags) is linked.

People search

You can now search for fellow users who have created works in your favorite fandoms! \o/ The new "Fandom" field in our People Search form suggests canonical tags as you start typing, so you don't have to worry about getting the name of your fandom exactly right. The more fandoms you put in, the more you'll narrow down the set of results.

 

Preview of the new people search.

The new People Search form, focusing on a search by fandom. The pseud testy has 970 works and 66 bookmarks, including 1 work for the fandom in question. Additionally, one can also search by username or profile content.

 

To make your pseud stand out in the search results, you can follow these instructions on editing pseuds to add a little blurb and user picture. (You can also add links and information about yourself, your fannish history, transformative works policy, or social media presence to your account profile, as laid out in our Profile FAQ.)

Limitations

Since even a state-of-the-art search engine like Elasticsearch suggests some gentle limits to the number of search results displayed (by throwing an error after a point), you will only be able to sift through 5,000 pages at a time, which means 100,000 works or bookmarks.

Keep in mind, however, that the Work Search form lets you change how results are sorted. So if you want to access the oldest works in a fandom -- something that wouldn't show within the first 100,000 results in a big fandom -- you can sort by Date Updated in Ascending order to make sure you get the results you’re looking for! (To sort a filtered list of works, use sort:>posted in the "Search within results" field.)

Some things to try

Once the new search is enabled for everyone, here are a few suggestions for getting to know the new options:

  • Rating: For a list of works rated General Audiences or Teen and Up, don't select a rating to include -- instead check "Explicit", "Mature", and "Not Rated" to be excluded, and all your results should be safe for work! (You can reverse this for your bedtime reading.)

  • Word Count: To get all the actual drabbles in your chosen fandom, enter 100 in both the "From" and "To" fields. (#DrabblePurist)

  • Crossovers: Are you a big fan of canon-verse fic? You can eliminate all AUs by excluding the "Alternate Universe" tag, and crossovers or fusions by selecting the "Exclude crossovers" option.

  • Date: Working on your list of favorite podfics of 2017? Go to the Podfic tag and put 2017-01-01 and 2017-12-31 in the "From" and "To" fields under "Date Updated", to refresh your memory!

  • Status: You've consumed all the completed works for your shiny new OTP? You have skipped the works in progress, because you needed a happy ending now? Nothing left to read? Well, gird your loins and check the "Only incomplete works" ticky to take a second look at those!

Comment

Post Header

At long last, we have upgraded our search engine, Elasticsearch, from version 0.90 aaaaaaall the way to 6.2, which had obviously been long overdue. A lot of other emergencies kept cropping up over the years, and there were never enough volunteers around to handle such a massive code rewrite while also managing the day-to-day upkeep of the Archive. With the help of our contractors (thanks to your generous donations!), we are now getting ready to deploy these changes to the Archive.

For a short period of time, the new search will only be available to a few volunteers for some final testing; everyone else will still see the current search and filtering interface. You might experience some issues in the days before we switch everyone over, as we had to tweak our old code a bit to get both versions of Elasticsearch running. Please bear with us, this will be temporary.

Once we are ready, we will move all accounts to the new system in batches, while monitoring how it holds up under increasing pressure. We currently expect this process to take at least a week, and longer if we run into any problems we have to stop and fix.

As always, we will keep you updated on progress here and through our @AO3_Status account on Twitter. (We'll have a bigger post with more details coming, but you can find a tiny preview below.) Many, many thanks to all the coders and testers who helped carry this over the finish line!

Snapshot of the new work filtering bar, showing the possibility to exclude tags, such as particular warnings

Comment

Post Header

For the past several weeks, the Archive of Our Own has been dealing with an unusually high volume of spam works created to advertise live streams of sporting events. We've taken a number of steps to deal with these spam works, and in the next few days, we'll take one more: automatically hiding works that our spam detection service believes are spam.

We've been using this spam detector behind the scenes for a while now, and it has a 99.3% accuracy rate when it comes to identifying spam works and comments on the Archive. This means there is a small chance some non-spam works will be incorrectly marked as spam.

If your work is marked as spam, we'll send you an email to let you know. Our Abuse team will review your work as soon as possible and unhide it if it isn't a spam advertisement. Works will not be reviewed for other Terms of Service violations while in the spam queue.

We apologize to any users whose works are incorrectly marked as spam and to everyone who is currently waiting for us to review and fulfill their invitation request. We realize this situation is frustrating, but these steps are necessary to help us deliver a spam-free experience to all our users.

Thank you for bearing with us, and if you see any spam that has escaped our net, please report it to Abuse!

Update on November 24, 00:40 UTC: Automatic hiding of spam works is now enabled.

Comment

Post Header

Published:
2016-04-18 16:47:20 -0400
Tags:

We’re adding a new functionality to the Archive! To date, only work creators have been able to add their works to collections. With an upcoming code release, collection owners and maintainers will also be able to invite works to collections!

For Collection Owners & Maintainers

The new code will allow collection owners and maintainers to add works to their collections by visiting the work, selecting the 'Add To Collections' button at the bottom, and entering the name of the collection.

The Add To Collections button at the bottom of a work.

If the work's creator has chosen to automatically allow their works to be included in collections, then the work is added to the collection -- no further action needed!

A work with a blue flash message at the top indicating the work has been successfully added to a collection.

If the creator hasn't chosen to automatically allow their works to be included in collections, a request will be sent asking them to approve or deny the addition of their work to the collection.

After inviting a work to a collection, maintainers will be able to see its status in the collection's 'Manage Items' page. While the request is pending approval, the work will show under 'Invited'; once approved, the item will move to the 'Approved' section. If the item is rejected, it will show under 'Rejected'.

The Manage Items page of a collection. The Invited button is selected, showing that works listed are awaiting creator approval before inclusion.

For Work Creators

Work creators can fully control how to interact with this new feature. Checking the 'Automatically agree to your work being collected by others in the Archive' option in Preferences will allow works to be added to collections directly without approval. A notification will let you know when your work(s) have been added to collections.

The Collections, Challenges and Gifts user preferences.

If that preference is not checked, you will receive an email asking you to approve or deny the invitation.

An invitation email informing a work creator that a collection maintainer wants to include the work in their collection.

Please be aware that these approval request emails fully respect the 'Turn off emails from collections' setting. If collection emails are turned off, but you don't allow your works to be automatically added to collections, you will not receive approval request emails and will need to periodically check your 'Manage Collected Works' page to approve the addition of your works to collections. Similarly, if you have collection emails turned off and do allow your works to be automatically added to collections, you will not receive an email when one of your works is added to a collection.

Creators can review and respond to invitations through the 'Manage Collected Works' page. This page is linked in the approval request email, and is also available in the 'Collections' section of the Dashboard sidebar. From there, you can approve or reject invitations, and review previously approved or rejected invitations.

A user's Manage Collected Works page.

As always, if you have any questions or experience any problems, please contact our Support volunteers, who will be glad to help!

Comment

Post Header

When we redesigned the Archive in 2011, most people were accessing the site on desktop devices, so we designed the layout and skin system with larger screens in mind. A lot has changed in the last four years, and the majority of our traffic now comes from small-screen devices like tablets and phones, so we're making some changes to our layout and skin system to reflect this new reality.

Currently, the Archive determines which layout you see based on how many pixels wide your browser window is. If you're browsing in a window that's 640 pixels or narrower, you see a single-column version of the layout. Otherwise, you see the default layout. However, our default layout was designed for larger screens, so if your browser width is between 640 and 1024 pixels, Dashboard pages with filters might not look their best:

A user's works page with the Dashboard, works, and filters arranged in three columns. The filters partially cut off due to screen size.

Soon we will be adding a new breakpoint to help anyone who has been seeing a slightly squished version of the Archive. The resulting layout will be a halfway point between the default and mobile layouts: the Dashboard will appear at the top of the main content, just below the main navigation, but the filters will still be on the right side of the page.

A user's works page with the Dashboard on the top and the main content and filters arranged in two columns.

We will also stop using pixels to define these breakpoints and instead switch to a relative unit known as ems. Em units are based on font size, and using them to set our breakpoints means the Archive layout will work better for anyone who has changed the default font size in their browser. For example, setting the font size to Very Large in Google Chrome cuts off the text in the filters and makes the work blurbs very narrow and long, even if you are looking at the Archive in a window that is 1024 pixels wide:

A user's works page with the browser font size set to Very Large and the Dashboard, main content, and filters arranged in three columns. The filters are almost completely hidden.

Once we add the new breakpoint, the Dashboard will appear at the top of the page, giving the filters and blurbs more room and making the page easier to browse:

A user's works page with the browser font size set to Very Large. The Dashbaord is now at the top of the page and the main content and filters are in two columns.

The new breakpoints of 42em and 62em will be available for use in skins, and any skins using the existing 640px breakpoint will automatically be converted to the new 42em setting.

These breakpoints don't correspond to the dimensions of any particular device -- there are too many devices for that to be a practical approach -- but are instead based on the point at which the Archive's default layout begins experiencing problems. We realize that's not ideal for skin makers who have chosen to remake the layout entirely, but it should be a positive change for the majority of skin users, and we're looking into adding customizable breakpoints in the future to allow you to target any and all devices you use to browse the Archive.

As always, if you have any questions or experience any problems, please contact our Support volunteers, who will be glad to help!

Comment

Post Header

Published:
2015-07-26 17:40:27 -0400
Translations:
Tags:

Until recently, if you were browsing or posting works in Brazilian Portuguese or European Portuguese, your only option was a generic "Portuguese" language. Answering several user requests, and after consultation with OTW Translation's Brazilian and European Portuguese teams, we have decided to change this setup.

The "Portuguese" language has been split into two languages for posting to the AO3:

  • European Portuguese, labelled "Português europeu"
  • Brazilian Portuguese, labelled "Português brasileiro"

To minimize the tidying-up work, both Translation teams have done their best to identify existing works in the previous "Portuguese" tag that belong to either language. During the migration, Accessibility, Design & Technology coders moved works according to this classification. If you have previously posted a work in Portuguese, please double-check that it's properly classified. You can change your work's language by selecting the "Edit" button at the top of the work page and choosing the correct language in the "Choose a language" dropdown menu.

We hope this will increase the site's usability for our Portuguese-speaking users. Our translation teams are also hard at work on the Brazilian Portuguese AO3 FAQs and the European Portuguese AO3 FAQs. Feel free to submit Support and Abuse tickets in either language, or just drop us a line saying hi!

Comment


Pages Navigation