AWS China: Beyond the Firewall

AWS cloud is the leading solution for delivery around the world. Do you need some computing power in Ireland? Granted. You want big storage on the west coast? No problem. Anytime, 24/7 you can manage your infrastructure in dozens of datacenters, always picking the most appropriate for your use case. What many don’t know is that several regions are special: the U.S. Government cloud and China. They are so-called “AWS partitions” — aws-us-gov and aws-cn, respectively.

While working with U.S. government agencies is not typical for most companies, the huge Chinese market is a very good way to quickly and permanently boost sales. AWS China proposes the same flexibility as AWS global, but with some caveats. Many GlobalLogic customers have recently requested deploying to AWS/Azure in China. There are not too many articles and resources about Chinese specifics, so the only way was to create a new account and test it by ourselves.

It is clear that AWS in China is not exactly AWS. Technically yes, it is controlled by AWS and has many services and APIs that you can see in AWS global, but due to regulations, their data centers are operated by Chinese companies: Sinnet in Beijing, and NWCD in Ningxia. It is also neither connected to any other regions nor shares any global services.

Key differences to note include:

• Separate partition name (in ARNs), separate domain name (so it also affects IAM)
• No direct connection to other regions (traffic goes through internet)
• Smaller amount of services (with some of them never going to be added, like VPN)
• Separate user accounts
• Separate S3 (yes, this also means a separate namespace)
• No access to Route53 global
• To even start working with AWS China, you need to have a license that requires Chinese identification
• Service APIs can be a bit different
• You need to have an ICP license for hosting any public resource
• You probably can’t use VPN solutions at all
• AWS China has separate support

The list is much longer, but you’ve got the point. Since other articles already cover these key differences — as well as other basics —we will focus now on actual deployment issues. We recently developed a production solution using AWS (details below) and discovered some interesting things during testing.

• Everything deployed to Ningxia with Terraform
• EKS cluster with Istio and some basic components (cluster-autoscaler, coredns and so on)
• Istio as a service mesh
• About 25 services deployed with Helm (~100-150 containers)
• Various AWS resources like S3, RDS, SNS, SQS and so on
• Gitlab pipelines, with Gitlab server living in Ireland and Gitlab runner in China
• No direct public access to any resources, the only entrypoint was a separate proprietary gateway

Based on the testing of this product, we found that the biggest impacting aspect is the Great Firewall (GFW) of China. From Wikipedia: “The Great Firewall of China is the combination of legislative actions and technologies enforced by the People’s Republic of China to regulate the Internet domestically. Its role in Internet censorship in China is to block access to selected foreign websites and to slow down cross-border internet traffic”. While most people know that the internet is limited in China, the real implication is not clear, though obviously it makes cross-country connections very slow. As in, dial-up modem level slow.

Why is it a big deal? Well, even if you only expect to work with customers, you should still build and deploy your services, migrate some data, provide access to testers and developers, and so on. Every part of the process might/will be affected. Even opening the AWS Console takes time, so don’t expect to somehow “fix” it in future; just expect that all access from the outside world will not be quite good. For a quick test, you can just check how quickly this page opens: https://www.amazonaws.cn/en/. Based on our tests, dependencies download, pushing artifacts, and even pulling the source code takes a ridiculous amount of time.

To make it even worse, it looks like the GFW has intermittent outages. If your application or some software tries to access blocked services (and believe me, almost every big outside site is blocked), your whole connection might be blocked or reduced in throughput. We’ve seen deployments take multiple hours in China, with the same deployments only taking 30-40 seconds in Europe. While it is not clear what usually leads to this problem, it can also be related to GFW resetting connections due to our Gitlab server being some new IP and using SSL.

And speaking about dependencies: everything is blocked. Don’t even try to pull something from quay.io or any other public repository. You can try to find a Chinese mirror, but there are no guarantees; it can contain malware or be simply outdated. They are also very slow. The only way to go is to mirror every dependency to China by hand — possibly waiting for days, as connection speed can go down to 40-50 kbps. And yes, you should mirror every dependency, as you never know when you will get an issue due to something else being blocked.

So in summary, here is a list of the issues we have faced:

• Console, deployment, interaction (everything is slow in AWS China if you are not accessing it from China)
• Most public repositories are blocked and usually you don’t have any good mirrors
• Due to some random usage of keywords, even your own infrastructure can be blocked
• Due to connection resetting on HTTPS, it is quite impossible to debug some issues (e.g., your Gitlab runner randomly failing to connect back to Gitlab)
• Managing dependencies is going to be very difficult; you will need to keep track of all new includes/requirements and mirror them
• If your application needs to access any external resources, it will be slow

Well, that’s it. Although AWS China is a good place for new companies to start working with the Chinese market, it requires patience, effort, time, and money to comply with all China’s regulations and limits. If you’d like some more information about AWS China, you can check out some of these other articles:

• My experience with AWS China
• Doing Cloud in China
• Tutorial: Setting up AWS in China
• Running Kubernetes And AWS In CHINA