I’m writing this story for two reasons, firstly, as far as I can see there is no good documentation on this process so hopefully this will save someone from stepping on all of the same rakes as I did. Secondly it serves as a cathartic rant to release all the frustration Apple has bestowed upon me with their pathetic and draconic approach to tech.
The Goal — auto renewing subscriptions
The goal is to implement auto renewing subscriptions using Apples in app payments (IAP) system. Auto renewing subscriptions are debited directly from a users App Store account at regular intervals set by you. You store the information about the user’s subscription and while their subscription is active you provide the user an enhanced service.
Apple will take care of storing payment information and automatically billing the user. As you store the details of the user’s subscription you can then seamlessly offer the same level of service elsewhere, for example on your website. This sounds like a sensible system, benefits everyone. Apple get a great app in their store keeping their users happy. Users get a simple billing process and don’t entrust their payment details directly to us. And you get a regular subscription fee to keep providing your service.
This kind of system of course can be achieved through many other (better) providers, for instance Stripe, but Apple bans any other method of taking payments in Apps. To reinforce this monopoly they stop you from advertising any other payment options in your app, and they purposefully do not have a coupon system, as they believe you will use it to circumvent their system and avoid their charges.
What I want you to keep in mind as I explain the hoops you need to jump through to achieve this goal, is that for the privilege of being forced to use their payment system, Apple charges you 30% of your subscription fee. THIRTY PERCENT.
Step 1 — Creating your subscription levels
The first task is legal based, you have to sign all the agreements and agree to their 30% cut. You can find out about these documents on the Apple help site. After this you can go to App Store Connect and set the plans you wish to offer in your app. There is some documentation on how to do that on the apple help site. What this documentation leaves out is anything about the pricing, I assume because they didn’t want to talk about how ludicrous it is.
After building your great app for Apple’s users, Apple won’t even let you set the price you want. You have to select it from one of their drop downs! Not only does it go up in their set increments, but what’s worse is all of them end in 99p! This is such a ridiculous restriction and makes me wonder if they’re in cahoots with the Monster Raving Looney Party who wanted to introduce a 99p coin in 2005.
Step 2 — The Sandbox
Apple quite sensibly have a sandbox environment for their payment systems. You can set up accounts which when used to pay for things in your app are not charged, thus allowing you to test your apps. Which is great, but, they are very impressed with themselves and laud this as a great feature – it gets a full 5 minute segment in a recent WWDC video (@16mins) . Excuse me while I hold back my giddiness, but isn’t this bare minimum you expect from any payment system…?
You might want to hold back your excitement too because although this sandbox exists it is abhorrently slow, sometimes taking 20 seconds if it chooses to respond at all. The whole system is regularly down, including a solid few weeks in April this year. During these periods you can not test your app and nor can Apple, meaning any new releases may sit in their review process until the problem is resolved. You’ll just have to stomach this performance hit though as there’s no other option for testing.
Step 3— Listen for transactions in the App
Quick disclaimer: I’m a javascript dev and have no clue how to build an app or code in Swift, this snippet comes straight from the above WWDC video (@10mins).
When you receive any transaction you need to unpack it and send the “App Store receipt” to your server along with some identifiable information e.g. a user_id. What is not immediately obvious, to me at least, is that these transactions are not just from your user making a purchase. A transaction is also received here for when the user reaches the end of their subscription and it automatically renews, this seems weird to me, and we’ll revisit it later on.
Of course I’ve not managed to locate any documentation that contains an exhaustive list of transactions which could show up here, because Apple seems to take the same approach to writing stuff down as a teenager, it’s just not cool is it?
Fortunately for App developers you handle every transaction the same, unpack the “App Store receipt” and send it to the server for processing, it’s their job to work out what it means. This is why it is important the observer is created as the app boots, Apple suggest on didFinishLaunchingWithOptions, or transactions will be missed. Once you receive a successful response from your server, make sure to finish the transaction or you’ll receive it again the next time the user boots the App.
Step 4— Verifying Receipts
Background on “receipts”
Now your app is sending random receipts to your server lets talk cover a little about what an “App Store receipt” is. Apple say it’s “just like a receipt that you’d get in a department store”, it’s an encoded string around 6000 characters long. You send this receipt to Apple’s verify receipt endpoint, along with a secret that is set in App Store Connect, and they’ll return to you the receipt information within.
This isn’t entirely accurate, the verify receipt endpoint does not just verify the receipt you send it, and the response is far more than just the receipt you sent. It also sends back all the transaction information about that user, and if there have been any new transactions since the receipt you have was issued, you’ll get back a new encoded receipt too. The department store analogy really breaks down, I don’t think when you try to read a receipt it morphs into a new one. It helps to ignore their misleading terminology and think of the “App store receipt” as a user token, and the verify endpoint as a transactions endpoint.
Verifying a “receipt”
Now you’ve got your head around the concept, lets look at putting it into action. To verify the receipt you’ve received you need to send it the appropriate environment, either ‘production’ to handle your real users, or the ‘Sandbox’ to handle your testing accounts. Whats that? You want to know which one to send it to? Ah see no, there’s no way to tell what kind of receipt you have until you verify it. The solution is to send it one environment, and if it’s rejected because it’s the wrong environment send it to the other one.
You can see here we check the status of the response, 0 means you’re good to go, 21007 and 21008 mean you’ve sent it to the wrong environment and you can use a list in the documentation archives to find out what some of the other errors mean. The list does start at 21000 so what all the codes between that and 0 mean is anyones guess. As most web development teams run at least 2 environments e.g. staging and production, we suggest you switch which Apple endpoint you send your receipt to first based on an environment variable, or if you prefer you can just send it to both at once.
Extracting subscription information
Great so now you’ve managed to verify some kind of apple ‘receipt’, lets look at pulling out the users subscription information and decide what to do with it. The information available is not documented as far as I can tell, the only page I can find is in the Apple documentation archives, which of course is completely out of date. So I’ll pick out the useful pieces for you — note I’m only going through the bare minimum here, there is other useful information available regarding user’s renewal choices and billing retry periods but lets walk before we can run.
What you need to do now is take all that information and store it in your database linked to the user_id the app sent to you alongside the receipt. This might look a little something like the following table. Each time you receive a receipt, update all the fields in this row with the information you extract.
Now when your user logs in, you can look them up in this table and decide based on their subscription_end date, and product_id, decide what level of service to provide them. You store the environment alongside the id because it may not be unique, and you store the receipt for debugging purposes (and without spoiling it, for use in Bonus Step 6).
Receipt gotchas
You didn’t think it would be that easy did you? There are a couple of special cases you should know about. The page in your App which offers subscription plans must also include a button which says “Restore purchases” or similar. This button allows someone who previously purchased a subscription from you, to claim it if for some reason you don’t know about it. What it should do is call Apple to get the user’s transactions and post the latest encoded receipt to your server the same as if purchasing a new subscription.
This situation can arise if the call to your server failed after payment went through, in which case all should be fine. The other scenario where this can occur is if the user logs out of their account for your auth system and logs into a different account. Then they won’t have a subscription as the user_id doesn’t match in your user_subscriptions table, but they will have paid for one with Apple.
The solution to this is when you receive a receipt, you should always look in the user_subscriptions table for a match based on environment and original_transaction_id. Meaning if the user_id is different it will be patched too, thus moving the subscription in your database from one user to the one who restored the purchases. This is seems bonkers to me, my instinct would be to reject the request if the user_id doesn’t match, but this advice comes from experienced App developers, and seems in line with Apple’s priorities.
Step 5— Listening to Status Update Notifications
By this step your user is enjoying the enhanced service in your app and everything is going smoothly. What you want to know now is if anything changes, for example when the subscription renews, or if the user decides to cancel. For this Apple has, like most payment providers, a webhook system. You go to App Store Connect enter a url, and when something happens to a user’s subscription you will receive a “Status Update Notification”.
Accepting Notifications
The first problem you may notice when trying to set this up, is that you can only set one url to receive these notifications and as I mentioned earlier most development set ups have at least 2 environments. Therefore you have to take this notification and forward it on to each of your environments until you find one which accepts it and respond accordingly. If one of your environments accepts the event, then return a 200 status code, if none of them do, return any other code to reject the event. If you reject the event Apple will continue to retry it, which is eminently sensible for them, there is however no documentation for how often or for how long the retrying occurs, which is much more like Apple.
When you receive a notification it will contain a top level property called “password”, this has to match the secret set in App Store Connect. If the password doesn’t match then reject the notification immediately as it is fraudulent. Once this is all hooked up you’ll receive 5 types of notification, they can be distinguished by the top level “notification_type” property on the event.
- RENEWAL — the user has renewed their subscription, you should extend their subscription end date.
- INTERACTIVE_RENEWAL — the user has renewed their subscription, it’s possible one of the other fields changed too, but it’s hard to say what, we’ll just handle this the same as a RENWAL.
- CANCEL — the user has cancelled their subscription.
- INITIAL_BUY — received shortly after a subscription starts.
- DID_CHANGE_RENEWAL_PREF — the user has selected a different plan but nothing changes until next renewal.
You only need to care about the first 3. INITIAL_BUY events are made redundant by the first receipt coming from your app’s transaction observer. DID_CHANGE_RENEWAL_PREF makes no difference to our basic user_subscription system, we will update to the new values if and when the user renews.
Renewal and interactive renewal events
I’m not clear on what the difference is between these two events, but we can handle them both the same. We need to do something similar to what we have done with the receipts, extract the latest subscription information and update the correct row in the user_subscriptions table.
Eagle eyed readers may have noticed here that expires_date is now not postfixed with _ms as it was in the receipts, and yet original_purchase_date_ms is still. This again is not a mistake by me, Apple has no consistency around it’s naming convention which is infuriating when combined with their complete lack of documentation.
Putting that to one side, we can now patch all this information onto the user_subscription row which matches the environment and original_transaction_id. It might look something like the above screenshot, where our user’s subscription is now active for an extra month, Huzzah!
Cancel events
When a cancel event comes in it means the user has cancelled their subscription. Strangely this event is only sent when someone cancels by contacting apple support. I’m not sure there’s any other way to cancel, but that nugget of information might be important to someone debugging cancellations.
Again I’ve found nothing specific about how to handle this, but I believe it’s effective immediately, so we’ll update the end_date of our subscription to be the date on the cancellation. Cancel events are a completely different shape to the renewal events, go figure, so lets dive in and root around for the information we need.
Again take all the values here and update the row in user_subscriptions which matches both the environment and original_transaction_id. Which might look a little something like this.
Great that’s it you’ll now extend users subscriptions and cut them short when necessary. You can listen to the other events and update the latest_receipt property, but I don’t think it’s essential.
The catch.
Great! You’re done right? WRONG. Apologies for drip feeding you this information but I wanted you to share in my frustration when working through this. In any normal payment system, these two processes would be enough to keep up to date with a user’s subscription. But this is no ordinary system.
RENEWAL events are not sent for every renewal, they are only sent when the user renews their subscription after it has expired, perhaps due to a billing issue. I believe this stems from the idea that renewal events will come in via the app’s transaction observer. But this self placed restriction is absolutely insane!! It means you’ll only know that a user has renewed if and when they open the app and that transaction comes through. This was a real desk flip moment for me, I can’t stress how stupid this is.
THIRTY PERCENT?!
Bonus Step 6— Status polling
Ok so the two integrations you’ve already set up still aren’t enough to keep your system up to date. The only option available at this point is to randomly poll Apple to check if anything has changed. This step was not documented anywhere that I could see, it’s now in the 2018 WWDC video (@14mins), where again they seem quite proud of it. It’s a complete joke that this is required and Apple should be deeply ashamed of it.
Lets just get through this so you can move on with your life. What you need to do is skim through your database, find any subscription that is due to expire, take it’s latest_receipt and put it back through the receipt verification process outlined in step 4. This will send the receipt to Apple, extract information from latest transaction and update your database. You can do this a little more cleverly by checking if anything has changed rather than blindly update the row, or you can just patch the row anyway so you can re-use all the same code.
The screenshot above is all of the documentation you will find on when to perform this status polling. Guessing from that I believe they suggest polling once every day from 3 days before an expiry date to 3 days after it. In theory polling after the expires_date is unnecessary as anyone renewing then should be covered by RENEWAL notifications, but trusting any of these systems to work is folly. My suggestion is to poll aggressively, perhaps an increased load on their servers will encourage them to rethink this madness.
Summary
Congratulations! You now have auto-renewing subscriptions in your App and you can begin the required counselling to forget how you got here. I know this walkthrough may have ranted about very petty things, or perhaps I’ve missed something completely obvious to everyone else. But this has been by far the worst and most frustrating development experience in my career, and I once spent a month coding JavaScript in notepad (not plus plus) for IE8 without the development tools.
Putting aside all the small gripes about naming and inconsistency, it’s unbelievable that there are three systems for this. You could achieve the same thing with just a status notification system and a much simpler more transparent receipt handler. For you to then have to poll their servers at random intervals on the off chance there is new information is so outdated and decrepit it really scares me as to what terrible code lies beneath.
There is no way on earth Apple deserves to take a THIRTY PERCENT cut of your revenue for forcing you to use their disgraceful payment system.
As meandering and bitter as this walkthrough has been I hope it contains enough information to guide others through this ridiculous process.
Good luck.
