AO3 News

Post Header

Published:
2010-11-11 23:11:40 -0500
Tags:

Along with the upgrade to Rails 3, there have been significant changes and improvements to our HTML sanitizing and parsing in Release 0.8.2. These changes should make things clearer for authors and much faster for readers!

Here is a quick breakdown for those who just want the highlights, followed by a more detailed explanation of what was changed and how it all works.

Highlights

  • Blank lines and carriage returns will now be converted to paragraph (<p></p>) and line-break (<br />) tags in the text editor.

  • The text will automatically be parsed and "cleaned up" -- any tags that were left open get closed, any mis-nested tags get fixed, etc.

  • The text will be sanitized, to remove any elements that are potentially harmful to our server.

  • This change fixes the known bug where switching from HTML mode to Rich Text mode causes all your paragraphs to disappear. (Yay!)

  • This change will also allow users to embed video from: youtube, vimeo, blip.tv, dailymotion, viddler, metacafe, and 4shared. (Yay!)

What's Behind the Scenes

The new back end for content works in three steps.

  1. There is now a paragraph-adder that converts blank lines and carriage returns into paragraph tags (<p></p>) and break tags (<br />) based on a few simple rules:
  • A blank line left between two pieces of text will be made separate paragraphs:
  • Here is paragraph one.

    Here is paragraph two.

    will become:

    <p>Here is paragraph one.</p>

    <p>Here is paragraph two.</p>

  • A carriage return or newline in the middle of text will add a break tag:
  • Here is a line
    with a carriage return in the middle.

    will become:

    Here is a line <br />
    with a carriage return in the middle.

  • We also will preserve extra blank lines -- if you have TWO blank lines in a row, we will add in an empty paragraph:
  • Here is paragraph one, and I want extra space after it.

    Here is paragraph two.

    will become:

    <p>Here is paragraph one, and I want extra space after it.</p>

    <p> </p>

    <p>Here is paragraph two.</p>

  • Note: The paragraph-adder will put <br /> tags at the end of each line whenever there is a carriage return, even in things like lists. So, if you have a nice chunk of HTML in your story that you coded up by hand like this:
  • <ul>
    <li>Item one.</li>
    <li>Item two.</li>
    </ul>

    You can avoid having <br /> tags added by putting the list into a single line with no carriage returns instead:

    <ul><li>Item one.</li><li>Item two.</li></ul>

  • The next step is a Ruby on Rails gem (basically a kind of plugin) called Nokogiri, which parses the text and gives it back to us as a well-formed chunk of XHTML. What this means among other things is that:

    • any tags that were left open get closed

    • any mis-nested tags get fixed (eg, if you do <strong><em>foo!</strong></em> Nokogiri will turn that into the correct version (<strong><em>Foo!</em></strong>)

    • any attribute values that aren't properly in quotes get fixed

     

  • Finally, we use the gem Sanitize to clean up this XHTML and take out anything that is legal but not necessarily safe. Sanitize uses a whitelist, meaning that only the tags and attributes we specifically tell it are allowed are let through. It's very customizable, and we have been able to write special rules for Sanitize to safely allow embeds of videos from specific sites (currently: youtube, vimeo, blip.tv, dailymotion, viddler, metacafe and 4shared.) Once Sanitize is done, the final version is saved into the database.

  • There is lots of documentation available on Nokogiri and Sanitize on their respective sites.

    What you see when editing

    • If you are working in a field (like content in the Post New Work form) that allows you to use the Rich Text Editor, the tags <p> and <br /> will show, because otherwise if you switch to the Rich Text Editor, it will do that horrible thing where your whitespace disappears and your text all runs together into one giant blob!
    • If you manually put in some <p> tags that had extra attributes on them, like "<p align=center>", the tags will show.
    • The <p> and <br /> tags will not show when you edit fields like notes and summary, however, where there is no option to use the Rich Text Editor.

    Here's an example of how the tags will look on content in the Post New Work form:

    Comment

    Post Header

    Published:
    2010-11-11 06:42:09 -0500
    Tags:

    Barring last minute snags, we'll be deploying new code to the Archive of Our Own today! This is a BIG release - we've upgraded the site to run on Rails 3 (the framework the Archive is built on), a change that involved changing ALL the code. We also have some new code for collections and challenges, some performance enhancements, and a bunch of other nice little tweaks and enhancements.

    Because this release involves lots of big changes, the Archive will have scheduled downtime while we deploy. We expect to take the site down for approximately 12 hours from 13.00 EST on 11 November 2010 (what time is this in my timezone?) . This time may change if we hit unexpected problems, but we'll send out a message on our Archive status Twitter account AO3_Status before we start and when the Archive comes back up.

    We thank you for your patience, and we hope you'll enjoy the improvements in the Archive when the maintenance is done!

    Comment

    Post Header

    Published:
    2010-10-19 23:33:21 -0400
    Tags:

    The Archive has had an exciting year! We bought shiny, shiny servers (of our own!) in September 2009, allowing us to move out of closed beta and share the wonderful fruits of our coders' and testers' labor with the rest of fandom! Since entering Open Beta on 15 November 2009, we've had a dizzying succession of exciting achievements!


    • We've introduced tons of awesome features, both big and small: the ability to backdate a work, new options for bookmarking and reccing, skins, mobile downloads, tools for running collections and challenges, and many, many more!

    • We saw two successful challenges run as test cases for our collections code - Yuletide Treasure in December 2009 and Remix Redux in March 2010 - and saw many more wonderful challenges and collections set up home on the Archive, including the Final Fantasy fanworks exchange, the House M.D. Big Bang Challenge, and the LGBT fest.

    • We celebrated the first birthday of the servers of our own with a party, cake and lots of fannish creativity!

    • We helped set up two new committees in the OTW - Tag Wranglers and Support - who joined Accessibility, Design & Technology in managing the day-to-day work on the Archive!

    • On 20 August 2010 we hit 100,000 works on the Archive!

    • On 10 October 2010 (10/10/10!) we hit 10,000 registered Archive users!

    • We add approximately 150 new users per week (and we have more requests that that - we have to add people slowly so the site can cope).

    We have so many more exciting things planned, including art and vid hosting, subscriptions, improved challenge code, more options for bookmarking and reccing, private messages, and better commenting features.

    All of this cool stuff is made possible by the collective contribution of fandom, which has come together to code, test, run systems, tag wrangle, provide user support, offer legal advice to help protect the fannish mission of the Archive, and support the AO3 financially through donations to the OTW. Our user numbers grow every day, and the servers of our own are already creaking under the weight of all that fannish passion and creativity. Thanks to the support of our members, we're already able to commit to buying new, more powerful servers (our Systems committee are choosing them RIGHT NOW) - but we need your continuing financial support going forward to keep on extending our infrastructure, and to pay for bandwidth and hosting costs. Just $2 from every Archive member would be enough to pay for our new hardware; we're asking for a US$10 donation. That support will make you an OTW member, with voting privileges. You already have a voice in the future of the AO3 - all users do. But your donation will help us make fandom's dreams a reality.

    Make sure fandom can continue to own our own servers! Donate to the OTW now!

    Comment

    Post Header

    Published:
    2010-10-17 17:43:24 -0400
    Tags:

    There have been a few discussions around the last few weeks about commenting on the Archive of Our Own, and while we have some ideas of our own for improving and encouraging more feedback in general, we would love to invite more ideas from all of you!

    Some of our current plans:


    • basic performance improvements to the comment form so it works quicker :)
    • a "kudos" feature that lets a reader indicate (with an optional one-line comment) if they enjoyed a story, complete with happy icon:
      logo-hearts-6


    • adding more nifty stats including # of downloads, and referring pages in the last month or so (so author can see if they were recced somewhere)

    • "badges" for heavy commenters (possibly with bonuses for commenting on older stories)

    • marking stories on the Reading History page as commented-on or not, allowing you to sort by this status

    • an option to reply to multiple comments at once

    • a "private" comment option to send comments just to the author

    • Adding an option to send an invite to people who comment without an AO3 account.

    • This is a more general feature: an optional "tip of the day" that will show the first time you come to the archive on a given day, which might point you to a new feature, suggest creating a new bookmark, or suggest posting a comment on a read story.

    Do you have other ideas? Thoughts on these? Let us know what would make commenting more fun, or what other kinds of feedback you would like to get on your works!

    Comment

    Post Header

    Published:
    2010-09-19 16:47:58 -0400
    Tags:

    All the latest news on the Archive of Our Own! Cake, servers, and Rails 3!

    We all still had slight cake hangovers at this meeting after our wonderful party to celebrate our servers' first birthday!

    Baby servers are one - our birthday party

    Thank you to all the Archive users and volunteers who came along to our birthday party to celebrate the first anniversary of our servers of our own! It was an amazing party full of squee, virtual cake, fannish chat and amazing creativity - check out our round-up post for more details of all our party hijinks! Since the party our users have been busy - we now have over 107500 works hosted on the Archive and over 9496 users!

    We really appreciate all the lovely feedback we receive - thanks to the many people who take a moment to drop us a line via our Support form or leave us a comment to let us know the things they are enjoying. This project is the hard work of lots and lots of volunteers, and it's a huge boost when people let us know our hard work is appreciated!

    Forthcoming changes - Rails 3

    There'll be a longer gap than usual before we release any new features, as the next task our coders will be working on is the upgrade to Rails 3 (the framework the Archive runs on). This is a big job which will affect all the code in the Archive, so we're pausing work on new features to allow this to take place. We think that the shiny new features in the last release make up for a little pause!

    News from our sub-committees and sister committees



    • Coders are working on all the tasks necessary to upgrade to Rails 3 - writing more automated tests, updating our documentation, and figuring out the actual changes that will be needed in our code base.

    • Testers are updating documentation about regression testing in readiness for the Rails 3 upgrade. Regression testing involves working through every aspect of the site to check that everything is working right and looking good (automated tests help to check these things, but for big changes there is no substitute for real people testing!)

    • Tag wranglers have been thinking about ways of making our wrangling system easier for users to understand and keeping up with the many new tags being added to the Archive by our new users!

    • Support have been working hard to keep up with lots of Support requests from new users. They're doing a fantastic job, but there is a higher volume of requests than usual, so please bear with us if it takes them a little while to get back to you.

    If there are things you'd like to do or say, please share them in comments, via the AO3 support and feedback form, by volunteering, or in whatever medium you feel comfortable with. Everyone is welcome to this party!

    Comment

    Post Header

    Published:
    2010-09-13 21:06:44 -0400
    Tags:

    This party had it all: a glittery cake to fuel our server anthropomorphic challenge, a scavenger hunt, reccing and self-reccing fests, teamwork to help each other upload back catalogs, and a special tag just for party fic. Here’s to another year of cheering each other on as we read, create, and transform.

    Midnight to midnight is a long time to party but our wonderful guests have kept the energy high! Here are some details on our marvelous accomplishments:

    We also compared time zones! And there were a LOT of them. During the last hour alone, the partygoers hailed from:

    New Zealand — New York City, New York, USA — Buenos Aires, Argentina — Colorado Springs, Colorado, USA — Indiana, USA — Finland — Romania — Turkey — Quebec City, Quebec, Canada — Maryland, USA — Ann Arbor, Michigan, USA — Los Angeles, California, USA — New Hampshire, USA — upstate New York, USA — North Carolina, USA — Pittsburgh, Pennsylvania, USA — Niedersachsen, Germany — Dublin, Ireland — Seattle, Washington, USA — Missouri, USA — Ontario, Canada — England, United Kingdom

    We even had a fandom roll call in our final hour! Check it out:

    Classical music RPF — Can I call "crack" as a fandom? — Community — Disney RPF — Miami Vice — House — Doctor Who, ElfQuest, Anita Blake — Tokio Hotel — Sophie's World — Highlander, Doctor Who, British Actor RPFSGA — Beatles RPF — Fandom: YES. — baum's oz books! — Sherlock (BBC), The Dresden Files (books), Doctor Who (2005), Temeraire, Hut 33, Traveler, Supernatural, Merlin (BBC), White Collar, Chuck — this one's easy, I'm actually the ONLY ficcer in one of my fandoms! — David Eddings — Black Rock Shooter! — Power Rangers! — (monkees is the only fandom of mine where i have written poetry) — Life on Mars, Ashes to Ashes, DCU, Slings & Arrows, Promethean Age (oh GOD) — robotech is the only fandom that i've painted in! — Lord of the Rings — Vorkosigan — Man from UNCLE — \o/ Vorkosigan! — Star Trek, X-Files, Hilda Tablet and Others, McCarthyism RPF — Covert Affairs — elisabeth, tanz der vampire, mozart! — Jane Austen, Firefly, 28 Days Later, Once and Again, Breakfast Club, Naruto, Hikaru no Go, X-Files, Harry Potter — claymore, black rock shooter, pretty cure, akira (manga), belgariad/malloreon, elenium/tamuli, eureka seven, kino's journey, gundam wing, aaaand dc comics — Hellboy II — oh also JAMES BOND — Sherlock Holmes. Book-verse. — Iron Man — Akira — Harry Potter, Star Trek, X-Men and X-Men: Evolution — Battlestar Galactica! — ooh, robin of sherwood was my very first fic-reading fandom — highlander films, historical fiction, stephen king's rose red, raymond e. feist’s riftwar saga (and everything else in those universes he wrote, jesus christ mr feist), spooks, spice world.

    Gentle reminder: the OTW and AO3 are run through fan energy, commitment, and donations — and eventually we’ll need more servers to slash with the existing servers. Please consider becoming an OTW member and donating to the cause!

    Comment

    Post Header

    Published:
    2010-09-13 07:33:26 -0400
    Tags:

    We're celebrating the birthday of our shiny shiny servers! One of the most important and most exciting things about the Archive is the fact it represents the amazing achievement of fandom pulling together to generate the investment and expertise needed to create a true home of our own. Since September 2009, fandom has owned the servers, and the OTW is committed to making those servers a safe space for the fannish community.

    Owning the servers is awesome, but just as important is what we use those servers for! We aim to make the Archive a welcoming place for fans not just because fandom owns the servers, but because the Archive is built by fans, for fans and has lots of awesome features which make a better experience for readers and creators. During our birthday party, Archive staffers and users have been talking about our favourite features - check out our list below and comment to let us know you what you love the most!

    Here are a few of our favourite things!

    Zooey, AD&T Chair: I LOVE the History feature! It keeps track of everything you view on the Archive, and records the date you viewed it and whether it has changed since. I am incredibly lazy about bookmarking fic, so this is totally a killer feature for me - when I think 'what was that awesome fic I read?' I can look back and find it. We're planning more features for this in future - I can't WAIT till we add the filtering option so it's easy for me to find a specific work without browsing through the whole list.

    Astolat, coder: OMG advanced search! It is a little bit tricksy and we are still working on the GUI for it, but there is nothing not awesome about just being able to do: "tag: merlin/arthur (explicit|mature) words: >1000" and suddenly you have JUST THE STORIES you want.

    Sidra, coder: Download as epub! You can browse the archive inside stanza on your iPhone and when you find something *poof* it's offline and ready to read. And when you get to the end, there's a link which takes you right back to comment. Super cool!

    Enigel, coder: Tags! (Also known as "wuzzles".) Tags and the way they tie into browsing: I'll be reading a John/Sherlock fic and then I can use the tags to hop to the Sherlock (TV) listing, and then follow the Crossover tag and maybe pick up a new fandom... It's a never-ending link-hopping dance. :D

    Yueix, user: Favorite feature so far is being able to read whole works (multi-chapters) as one continuous story.

    bingeling, designer: I love the Archive simply for being multi-fandom. Whenever I'm in the mood to read something that's not my main fandom I simply check what the Archive has to offer and most of the time I'll find something. Pre-Archive it was always quite a bit of work to actually find the fic. If it wasn't your fandom, you didn't know its infrastructure. Thanks to the Archive I now always know where to start looking. ♥

    Helka, kääntäjä: Minusta on hauskaa, että niin moni pieni fandom on edustettuna Arkistossa. Yli 5500 fandomia! Joissakin fandomeissa on tuhansia tarinoita, joissakin vain yksi ainokainen, mutta jokainen tarina on arvokas. Voin mennä fandom-listaukseen ja selata vaikkapa kaikkia kirja-fandomeita, tai sitten voin etsiä jotain tiettyä fandomia etsintä-toiminnolla. Ja joka päivä Arkistoon tulee lisää tarinoita! \o/

    lorax, user: Favorite archive feature by FAR is the challenge sign up features. I did remix through the auto sign up this year, and it's fantastic and keeps everything so easy to find. I'm really looking forward to seeing how yuletide uses the challenge sign up through the site this year, if they do.

    Julia, Übersetzerin: Ich halte das Eigene Archiv jetzt schon für eine wunderschöne, nützliche Sache, aber was es für mich auszeichnet ist dieses Gefühl, dass nach oben hin alles offen ist. Dass durch die Vorschläge der Nutzer gepaart mit der Reaktionsfreudigkeit der Programmiererinnen zusammen Stück für Stück ein Archiv entsteht, das in der Summe seiner Teile dann doch verdammt revolutionär ist -- besonders, weil es im Moment erst einen Bruchteil seines Potentials zeigt.

    mlle, user: My favorite feature is the little symbol box next to each work. It tells me at a glance whether or not a fic has anything in it I want to avoid. Priceless.

    X-parrot, tag wrangler: As a hopelessly pedantic author, I love that whilst replying to a comment, I happen to notice I'm missing yet another "the" in my latest chapter's final paragraph, I can hit "edit" and instantly put that to rights!

    Christie, user: I love the freedom to post anything and the great search :D

    Anne-Li, Support staffer: I love that it is central, you don't have to look at a gazillion places to find (good) fic.

    Derry, user: My favourite feature is the ebook download! I can stuff my PRS full of juicy, glorious fanfic with just one click!

    Tai, tester: My favourite thing is the Reading history - I don't have to muck around bookmarking stuff I want to read and then wondering where I put it, just open the page and come back to it at leisure.

    theliterator, user: Here's what I love the most, which is what I'm using right now. The importer.

    zelempa, coder: I am still discovering new features so what I love in advance is this blog post.

    Alison, tag wrangling committee: My favourite is ALL the features.

    Share some of yours!

    We currently have nearly 9000 users on the Archive, who between them have posted more than 100,000 works, in over 5500 different fandoms. We figure this means we are making lots of people happy! Let us know in the comments to this post what you enjoy about the Archive!

    Comment

    Post Header

    Published:
    2010-09-12 19:56:42 -0400
    Tags:

    Happy birthday to us, happy birthday to us, happy biiiiirthday dear servers, happy birthday to us! The beautiful, beautiful servers of our own are one year old! We're incredibly proud of the fannish energy and investment that enabled us to buy our beautiful servers - thank you fandom!

    Birthday presents

    As our birthday gift to fandom, we've launched an AWESOME code release. Release 0.8.1 includes skins, mobile exports, and a very special guest logo!

    As part of our birthday goody bag, we're also giving every Archive user an invitation so that everyone can invite a friend to the party! If you go to your user home and check out 'Manage invitations' you'll be able to see and use your invitation. You can also request more if you need them!

    If you don't already have an Archive account, you don't have to be left out! Just add your email address to our Invitations Queue and you should receive an invitation within a few days. Everyone who was in the queue after we released our birthday code was issued an invitation right away!

    Birthday party!

    We're having a party to celebrate, starting now! Drop into our Public Discussion chatroom anytime today to join the party - Just head for this URL and type in the name you'd like to represent you, and you'll be in: https://fanarchive.campfirenow.com/28473.

    Singing happy birthday!

    All birthday girls deserve some praise! Throughout the party, we'll be talking about what Archive staffers and users love the most about the Archive -- we'll round up a sample of these, and post them for everyone to enjoy and add their own thoughts to! Everyone's killer features in one place!

    Also posted on the OTW News Blog

    Comment


    Pages Navigation