Isle of Armor: After Action Report

It’s been a week since the launch of Pokemon’s Isle of Armor DLC, and we wanted to talk a little about our experience, what we’ve learned and how we’ll improve further.

As expected, we had a major spike in orders that was 6x of normal load, 3x of the previous peak in March with Pokemon Home’s release, and 4x of the peak last November with Sword and Shield’s original release. Unlike the previous times, we had greatly optimized our strategy – through additional hands, automation, and algorithms – to be scalable. This held up, for the most part.

Learnings from Sword and Shield’s Launch

When Pokemon Sword and Shield first launched, the flood of orders that came in was completely unprecedented. We had originally thought that with the advent of Dexit, the Gen 7 games would still be king – and were very wrong. At the time, only one Switch console was in play (we had 2 more in storage, as backup). Half a day later, it was clear at least two Switch consoles were needed, and we quickly put another in the mix. A few days later, we had a queue timing of 72 hours! Some competitors ended up with timings of over 7 days, though. It clearly took everyone by surprise.

From this, we knew we had to make sure our platform could handle increased loads. The Isle of Armor launch was a testament to that, with orders never exceeding 30 hours before preparation despite a much higher load.

First On Scene

Within one hour of the DLC dropping, we updated the Rawkhet Pokemon Teambuilder to v4.0, allowing users to request the returning and newly added Pokemon from the DLC in Custom Pokemon (among several new features). A day later, we were ready to resume services. The tool that most “genners” use was updated on evening of the 19th. This first-mover advantage was likely instrumental in the massive surge of orders this round. At heart, we are Pokemon fans as well – completing the game and fully understanding the mechanics, making refinements to the Teambuilder and coming up with listings to cover customers’ blind spots.

The Snowball Effect

This was one of the big learning points from the original SWSH launch. The moment we fall behind on schedule, orders will start piling up (for both the Preparation and Trade Queues). One way to mitigate this is to temporarily close Checkout, but our strategy for this round was to maintain as high availability as possible. Because of this, we ended up working 6 consecutive 18-hour days, with little personal rest. Many sacrifices had to be made, including meals, social life (at least, as much as can be found during this period), other work commitments, and sleep.

As orders weren’t closed while we prepared for Isle of Armor, it was extremely difficult to tackle orders in a healthy way when trades reopened. Business may have been good, but was taxing on health. For the Crown Tundra DLC later this year, we will likely change our strategy here and temporarily close Checkout to prevent this from happening again (subject to change).

Communication Conflict

While the previous segment was more on the Preparation Queue, the snowball effect also applies to the Trade Queue. Although we have a very efficient way of handling the Trade Queue, it relies heavily on the customers following instructions. When too many refuse to read the Trade Guide or follow instructions, the queue comes to the standstill.

This is especially for the Pokemon Home listings, because only one customer’s order can be loaded into Bank at a time (for efficient moving). Over the past few days, several customers have neglected to read the instructions and after we move their purchase into Bank, they a) went to sleep, b) thought the trade was in SWSH (like seriously, a 900 Pokemon trade!?), c) sent everything BUT the Moving Key, or d) ghosted us.

It’s extremely frustrating for us when customers do this. While only 3% of customers act this way, it is enough to bring traffic to a standstill. As we wait for these customers to get prepared, other customers continue to pile up in the queue. Some may drop off, resulting in more delays as we try to contact them when it is their turn (we wait 10 mins on reaching a customer’s turn before moving on).

What’s incredible is that users are told to read the Trade Guide in:

The product listing description
Order placed page
Order receipt email
Order ready email

But we still get:

In many of the ghosting cases, our “alright” response to head on was sent a minute or less from their email. The 10 minute wait is enough to cause major setbacks to the rest of the queue, which is even sillier when many of these trades should literally take a minute (1 Pokemon).

Unfortunately, there are plenty who don’t read the Trade Guide. These customers forget to connect to the Internet before attempting to trade, send Pokemon that trade evolve (pet peeve), and ask how to trade just when we are about to. As a major contributing fault to aforementioned communication conflicts, we have placed much further emphasis on the Trade Guide across the service. Hopefully, it’s unmissable this time.

In time to come, we will also be adding more checks to the Checkout page to prevent customers from checking out mismatched products (e.g. Living Dex (Home) for Sword and Shield). While most combinations are already checked for and show a warning text, it is clear this is not enough.

Another contributing fault is our over-reliance on email. It works for most customers, but we have to acknowledge that some customers don’t know how to navigate their emails, such as those who create a new email thread for each message instead of replying to one email thread. Some don’t receive notifications for their emails, and some have really weird pull/fetch settings for their email so it only refreshes every 15 minutes or worse. We have always had multiple social channels for customers to contact us on if they are unable to use emails, but the messaging to move these customers over was missing. The Contact, FAQ, and Trade Guide pages have now been updated with instructions on contacting us through our social channels for trading.

Social Channels

Rawkhet has always had an email-first rhetoric. Today, we aim to scale up our social channels and allow for more diversification of services. We have added a new Discord account (not server) that customers can message, something that has been asked for in user feedback quite frequently.

Through more regular monitoring of our social channels, we aspire to bring an even better service to all our customer, taking messaging to real-time where possible.

Messaging Updates

Some other feedback from users over the past few days has been about Rawkhet’s messaging and tonality. We try to maintain a professional front as much as possible, but the past few days have seen some of the messaging become “pushy” or “harsh”. The customers that mentioned these understood that that was due to the increased number of orders we had to work through, especially with the others who did not read the Trade Guide. Thank you for your understanding.

We’ve made some changes to the messaging around the site and our trade-time messages to try and be warmer and friendlier.

Thank You

But before we say thank you, we’d like to say sorry. The past week has been a tough one, and we feel that the service standard has slipped up a bit – especially if customers are able to see cracks in our messaging style. The Trade Queue at one point was 6 hours long, which is not a good experience for the customers on that day (Jun 21, where we traded 1,612 Pokemon on SWSH alone).

Rawkhet’s goal has always been to “under promise and over deliver”. We’ve seen that we can do even better, and that is what we will strive to be.

With that, we’d like to say thank you to all our patrons who have walked with us this far, evolving alongside Rawkhet. Stay safe, stay healthy, and see you in the Crown Tundra.

The Status Page (Archive)

Introduced when SWSH first released, the Status page was instrumental in keeping our customers up-to-date. Here are all the updates that transpired during the launch.

6/24/2020: We’ll be breaking slightly earlier this morning, with lots of other stuff to catch up on (including sleep). Starting this evening, we should be back to our normal schedule and start cleaning up the announcements 🙂
6/24/2020: We did it! All caught up with order preparation, after 5.5 days of going at it.
6/23/2020: Fifth 18-hour day, but the next batch should see us completely catch up with the queue. Taking a break now, will be back in the evening as always.
6/22/2020: Tough evening. Bad few holding up many of the queues, because they “don’t know” how to trade. Even if reading is hard, the Trade Guide has pictures. It really can’t be clearer than that. In happier news, we expect to finally see an “all done!” for SWSH today or tomorrow.
6/22/2020: Fourth 18-hour day! Pretty good progress today, drastically bringing down the 24-hour ETA on orders. The last preparation batch was only 6 hours after an order was placed.
6/22/2020: When we message to say it’s your turn in the queue and ask if you’re still available, and you reply yes, please really be ready. Making the entire queue wait (especially on Home, where we can only have one user’s order in Bank at a time) is super inconsiderate.
6/21/2020: We’ve changed up our strategy, which should hopefully make queues a lot shorter with load balancing.
6/21/2020: That’s three days in a row of 18-hour trade days. We’re breaking for now and will be back in the evening as always. Some orders may unfortunately cross the 24-hour mark, sorry about that. Numbers seem to be plateauing, so we expect to be returning towards normal Preparation and Trade Queue timings around Monday or Tuesday.
6/21/2020: Happy Father’s Day to all dads! Remember to wish your dad and/or do something nice for him today.
6/21/2020: Ack, we had the Trade Queue peak at 6 hours today, due to the influx of orders over the weekend. Sorry about that. This should return to normalcy over the next few hours.
6/20/2020: First batch of Day 3’s orders are out! Quick reminder – do NOT ping us if we haven’t crossed 24 hours yet, we’re still operating within the 24 hours schedule and it’s really annoying to handle these messages.
6/20/2020: Gotta take a break, been trading for 19 hours straight. Catch y’all this evening.
6/20/2020: Reminder – PLEASE read the Trade Guide, it’s written for a reason. Those that say they are ready and end up making us wait for 10 minutes while they figure out how to trade severely tax the queue.
6/20/2020: We’re about 7x of a normal day’s load, bear with us. Preparation Queue timings are closer to 24 hours than 12, and Trade Queue timings are ~2-4 hours.
6/19/2020: Full speed ahead!
6/19/2020: We’re back, and handling some administrative tasks (like sending an announcement email) before starting the evening trade queue. Emails may be slow (including order receipts), please bear with us for about an hour.
6/19/2020: We’re going to be taking a break until the evening, having been trading for 18 hours straight. We’ve learned so much on streamlining and scaling our order management since November and March, and should be able to keep to < 24 hours on orders despite increased load. Very proud to be the FIRST and ONLY Pokemon provider working post-Isle of Armor.
6/19/2020: Isle of Armor: New and Revised Listings
6/19/2020: We’ve been trading for 14 hours straight. Order load is approximately 4x of a normal day.
6/18/2020: Services are GO! First batch is away. We are the FIRST in the world to offer services post-Isle of Armor.
6/18/2020: YEP, services are a go for tonight. We’re taking a break now from updating our systems, having been at it throughout the night. Catch you all later this evening! (fyi: we don’t rely too much on “public” systems)
6/18/2020: Almost done, we’re on track for a phased restoration of SWSH/LGPE services by tonight.
6/17/2020: Reminder that besides Custom Pokemon, other listings have not been updates for Isle of Armor yet. Existing listings such as the Gigantamax Package and Past-Gen Legendaries Package are subject to change. SWSH orders are likely to continue being throttled till June 19 or 20. Please keep this in mind before placing any orders. We recommend our Home listings in the meantime!
6/17/2020: Please note that 3DS and Home orders are still proceeding as normal. LGPE and SWSH orders are impacted at this time.
6/17/2020: Isle of Armor Updates: Teambuilder v4.0
6/17/2020: Isle of Armor DLC is now LIVE. All services are currently PAUSED while we update our capabilities.
6/17/2020: With the launch of the Isle of Armor DLC, please expect delays in Preparation and Trade Queues while we update our listings and services.
6/16/2020: The calm before the storm…