Network shutdown, all around Kazakhstan

#Internet connectivity was shutdown in #Kazakhstan again at ~1300 UTC after 6th brief service restoration since shutdowns started on Jan. 5. @cloudflareradar shows that this one saw peak traffic 2x or more as compared to previous restorations.

That matches the IODA signals as well. The restoration of access of January 10 (starting 00:00 UTC) lasted 13 hours and seemed to include more networks than past ones.

https://ioda.caida.org/ioda/dashboard#lastView=overview&view=inspect&entity=country/KZ&from=1641236120&until=1641840860

We’ve switched all Lantern (https://lantern.io) servers in the region to listen on 3785, 5060, as well as randomized high ports.

I found that port 179 works fine on both ISPs (KazakhTelecom and Beeline).
Thanks @sasha0552 for help!

The Tor community team posted a guide on how to get working bridges. You will not be able to use BridgeDB or Moat; instead, email frontdesk@torproject.org with subject “bridge kz”.

Thank you for the information. I opened port 179 on the the bridge from earlier as a backup in case 3785 gets blocked.

Bridge obfs4 172.105.56.235:179 DD9769A0D6A9F18C24FCE731583597012E66273F cert=AEu2dF5cSjzQwA8kDx4R+38u10TReImk3ERjWFmzBGA0tPGyFxnsJRke5iSBef6+QDejew iat-mode=0
Bridge obfs4 [2400:8904::f03c:92ff:fe93:f42d]:179 DD9769A0D6A9F18C24FCE731583597012E66273F cert=AEu2dF5cSjzQwA8kDx4R+38u10TReImk3ERjWFmzBGA0tPGyFxnsJRke5iSBef6+QDejew iat-mode=0

I did it with port forwarding:

iptables -A PREROUTING -t nat -p tcp --dport 179 -j REDIRECT --to-ports 3785

Репортирую, что вчера при включение интернета было ограничение скорости до 3Мбит /с примерно. На https видимо максимум по 20Кбит/с, не смог даже обновить репозитории.

Thanks! Today we heard from a user that Beeline is blocking 3785:

К сожалению, у Билайн Казахстан заблокирован порт 3785, есть ли другой способ обхода блокировки?

We will try your bridge.

Фиксирую падение интернета до начала шатдауна:
ISP: Beeline KZ (“Интернет Дома”).
Время: 17:10 - 17:24 (GMT+6).
Таймаут до Google DNS.
Внутренние DNS провайдера остались доступны - dnstt работал.
Порты tcp/179 и tcp/3785 были заблокированы.

Tor used to have fteproxy bridge, which claimed to masquerade as unencrypted http. Although, I think it would be easy to block by fingerprint. Binary is still available, but no one is providing this type of bridge right now. However I would like to test it.
Binary is static with python embedded inside.

In networks with low bandwidth it would be useful to use HandyCache caching proxy.
It can also decrypt and cache https traffic, but this functionality in the trial mode only works for the first 30 minutes after each start of application (and then you need to restart HC). There is an English and Russian interface. Works in Wine.

IODA measurements say that access has been restored since about 2022-01-11 00:00 (06:00 Almaty time). Does that match people’s experience? I can access gov.kz now.

https://ioda.caida.org/ioda/dashboard#lastView=overview&view=inspect&entity=country/KZ&from=1641168000&until=1642032000

Yes, there were no more shutdowns.

I have shut down this bridge now.

Here are graphs of its usage over the past few days:
https://metrics.torproject.org/rs.html#details/0E9783A73F029E0910FD72F1EC120CA818868DA0

@anadahz pointed me to a RIPE Labs blog post on the shutdown. It notes that despite being “shut down,” networks in Kazakhstan were still present in the global BGP routing tables, which matches our experience with certain ports being unblocked. It also has some analysis of different levels of access in e.g. data centers versus residential connections.

It is difficult to pinpoint the cause of the outage. However, the affected networks have remained visible in the global routing system (BGP), which means they’ve remained “connected” to the Internet even though they have not been able to send or receive packets. The timing of the outage was synchronised, suggesting it was the result of some centralised action, although we do see small variations per region.

If we try to distinguish between RIPE Atlas vantage points in infrastructure - i.e. RIPE Atlas anchors and other probes with tags that suggest they are in data centres - we see differences in how connectivity developed over the last few days.

The figure below shows infrastructure vantage points in red. While connectivity for most of these vantage points went down in the last few days, it looks like most are able to send and receive packets to/from the Internet again since around midnight UTC on Friday 7 January. The other vantage points, which we think are mostly near end-users show that over the last few days there were periods of multiple hours where some of these vantage points had Internet connectivity.

After a few hours where almost all of our RIPE Atlas vantage points were online again, we see a drop again. If we look at infrastructure (data centres) versus other probes we do see that roughly half of the other probes (homes, offices, etc.) go down again, but many stay connected.

The comments on the post link to an interactive notebook for analyzing outages using RIPE Atlas.

I want to make a post that summarizes the important lessons from the January 2022 shutdown in Kazakhstan. I have written a draft in English (about 1200); is anyone willing to translate it to Kazakh and Russian before I post it?

https://pad.riseup.net/p/RQ1I5QI01qRfWZJBrUBD

You can also edit the document to add something you think is important. I’m planning to make the post next Monday, 2022-01-07.

I made translation to Russian. This is not a direct word-by-word translation, but more like my interpretation of the text according to typical Russian text constructs.

Thank you, I appreciate it. I think that is the right way to translate.

Posted.

An article by Katia Patin gives some details about how working ports were discovered, and efforts to establish proxies.

2022-01-27
Kazakhstan shut down its internet. These programmers opened a backdoor (archive)
Обойти национальный шатдаун: как молодые IT-специалисты вернули интернет тысячам казахстанцев (archive)

A senior software engineer at LinkedIn in Toronto, Maksat Kadyrov jumped into action when he lost touch with his brother in Almaty. He went live on Instagram, looking to crowdsource a way to reach his family. … He live streamed on Instagram for hours as they scanned some of the more than 65,000 existing ports. During the live stream, they found five open ports, tested them and were able to establish a connection. They later learned that it was a bug in outdated Cisco equipment, used widely by Kazakh telecom operators, which had accidentally kept these ports open. Kadyrov, Maksut and the others used these open ports to support their operation, crowdsourcing funds or footing the cloud computing bill themselves from service providers like Digital Ocean and Amazon.

Over the next few days, the loosely organized group set up dozens of proxy servers — first for Telegram and later even for internet browsers like Firefox. Maksut admits their user estimates aren’t exact; not all of them had a chance to collect data. But more recently, on January 19, Zharaskhan Aman, a software engineer at Facebook in London, rounded up some of the numbers he had from Telegram analytics showing that the 9 servers he raised alone had 155,762 users from Kazakhstan between January 4 and 11. … Based on user traffic provided by Telegram, Maksut estimates the group got between 300,000 to 500,000 people online on the message app during the five-day shutdown. … Sharing connection instructions by Telegram, email and text, members of the group said they were overwhelmed with demand. Within 24 hours Kadyrov said he had more than 2,000 requests for access to his servers, which he had been sending out one-by-one.

Когда старший инженер-программист LinkedIn в Торонто Максат Кадыров потерял связь со своим братом в Алматы, он решил, что пора действовать. … В течение нескольких часов он вел прямую трансляцию в Instagram, пока они сканировали несколько из более чем 65 тысяч существующих портов. Во время прямого эфира они обнаружили пять открытых портов, протестировали их и смогли установить соединение. Позже они узнали, что некоторые порты оказались случайно открыты из-за ошибки в устаревшем оборудовании Cisco, широко используемом казахстанскими операторами связи. Кадыров, Максут и другие использовали эти открытые порты для поддержки своей операции и покупали серверы у Digital Ocean, Amazon и других провайдеров на деньги, собранные краудсорсингом или свои собственные средства.

В течение следующих нескольких дней группа энтузиастов установила десятки прокси-серверов — сначала в Telegram, а затем даже в интернет-браузерах, таких как Firefox. Максут признает, что его оценка количества пользователей не точна — не у всех была возможность собрать данные. Но 19 января Жарасхан Аман, инженер-программист, работающий в Facebook в Лондоне, изучил аналитику Telegram и посчитал, что только 9 поднятыми им серверами с 4 по 11 января воспользовались 155 762 пользователя из Казахстана. … По оценкам Максута, основанным на данных о посещаемости Telegram, за время пятидневного отключения группа дала доступ к приложению от 300 до 500 тысячам человек. … Обмениваясь инструкциями по подключению через Telegram, электронную почту и СМС, члены группы говорили, что они не справлялись с потоком запросов. Кадыров говорит, что всего за 24 часа ему поступило более 2 тысяч запросов на доступ к его серверу, которые он рассылал по одному.

I found Katia’s article as a reference in the paper Government Internet Shutdowns Are Changing. How Should Citizens and Democracies Respond? (2022-03-31).

Интересно, как при такой тотальной блокировке проще всего проверить все порты? Может быть есть готовые сервисы которые отвечают на любом порте?

There are some notes about port scanning higher in the thread and in another post.

About hosts that respond on any port, scanme.nmap.org is one such, but there are many. The host does not have to have every port open (SYN/ACK response); it is enough if it exposes its closed ports (RST response).