Network shutdown, all around Kazakhstan

Провайдер Казахтелеком.
Интернет отключили 17:00 05.01.2022
Дальше отключили полностью мобильную связь, не ловило в любых режимах(2G, 3G, 4G)
Через несколько дней включили мобильную связь, но звонки до сих пор отвратно работают.

Вывод traceroute:
traceroute to dns.google (8.8.4.4), 30 hops max, 60 byte packets
1 _gateway (192.168.100.1) 1.340 ms 2.627 ms 2.562 ms
2 82.200.242.218 (82.200.242.218) 6.005 ms 6.513 ms 7.061 ms
Дальше одни звездочки

С этим выводом я воодушёвленный пошёл проверять связь с другими клиентами сети казахтелекома. И пинг был(3 хопа)! И даже больше, кажется на них нету фильтра.
Мы спокойно прокидывали порты, HTTP, SSH, и прочие протоколы.
До других IP происходит полная фильтрация(даже icmp). Режим белый список.
В белом списке находится:
dns.google(8.8.8.8), akorda.kz, IP банков и государственых новостных агенств, а также мобильных операторов
Ставлю предположение, что фильтрующее обуродование на третьем/четвертом хопе стоит.
С этим уже кажется можно получить доступ в интернет, через dns туннель. Но к сожалению у меня нету сервера за рубежом. Также скорее фильтрация, крайне сильная с урезанием функционала до минимума, так я не смог icmp трафик сделать до всех хостов в whitelist. Кроме altel.kz
09.01.2022 дали доступ ко всем подсетям hoster.kz, neolabs.kz, ps.kz. Мне кажется или у хостингов есть интернет, так как судя по зеркалу репозиториев там они относительно свежие.
Сегодня, 10.01.2022 в 8:45 дали интернет.
В Астане давали интернет уже 3 дня назад. Но временно, с 8:00-13:00

I verified that shadowsocks+v2ray works just fine trough 3785 port.

Almaty, Kazakhtelecom

OpenVPN on port 3785 (udp) works.

Some information:

$ ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
^C
--- 1.1.1.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time ms
$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=100 time= ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=100 time= ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=100 time= ms
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time ms
$ dig google.com @8.8.8.8
; <<>> DiG  <<>> google.com @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;google.com.                    IN      A

;; ANSWER SECTION:
google.com.             272     IN      A       173.194.222.113
google.com.             272     IN      A       173.194.222.138
google.com.             272     IN      A       173.194.222.100
google.com.             272     IN      A       173.194.222.102
google.com.             272     IN      A       173.194.222.101
google.com.             272     IN      A       173.194.222.139

;; Query time:  msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Jan 10 18:58:53 +06 2022
;; MSG SIZE  rcvd: 135
$ curl https://8.8.8.8
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="https://dns.google/">here</A>.
</BODY></HTML>

hoster.kz, neolabs.kz, ps.kz - timeout
altel.kz, akorda.kz - works

(If you want to investigate, you can contact me using Discord (invite: rTjTadmYvt))

TCP, UDP, ICMP трейсы (-T, -U, -I) до 8.8.8.8 нормально выглядят в Казахтелекоме?

Wrote you PM but it seems that Kazakhstan net is getting shut down again.

#Internet connectivity was shutdown in #Kazakhstan again at ~1300 UTC after 6th brief service restoration since shutdowns started on Jan. 5. @cloudflareradar shows that this one saw peak traffic 2x or more as compared to previous restorations.

That matches the IODA signals as well. The restoration of access of January 10 (starting 00:00 UTC) lasted 13 hours and seemed to include more networks than past ones.

https://ioda.caida.org/ioda/dashboard#lastView=overview&view=inspect&entity=country/KZ&from=1641236120&until=1641840860

We’ve switched all Lantern (https://lantern.io) servers in the region to listen on 3785, 5060, as well as randomized high ports.

I found that port 179 works fine on both ISPs (KazakhTelecom and Beeline).
Thanks @sasha0552 for help!

The Tor community team posted a guide on how to get working bridges. You will not be able to use BridgeDB or Moat; instead, email frontdesk@torproject.org with subject “bridge kz”.

Thank you for the information. I opened port 179 on the the bridge from earlier as a backup in case 3785 gets blocked.

Bridge obfs4 172.105.56.235:179 DD9769A0D6A9F18C24FCE731583597012E66273F cert=AEu2dF5cSjzQwA8kDx4R+38u10TReImk3ERjWFmzBGA0tPGyFxnsJRke5iSBef6+QDejew iat-mode=0
Bridge obfs4 [2400:8904::f03c:92ff:fe93:f42d]:179 DD9769A0D6A9F18C24FCE731583597012E66273F cert=AEu2dF5cSjzQwA8kDx4R+38u10TReImk3ERjWFmzBGA0tPGyFxnsJRke5iSBef6+QDejew iat-mode=0

I did it with port forwarding:

iptables -A PREROUTING -t nat -p tcp --dport 179 -j REDIRECT --to-ports 3785

Репортирую, что вчера при включение интернета было ограничение скорости до 3Мбит /с примерно. На https видимо максимум по 20Кбит/с, не смог даже обновить репозитории.

Thanks! Today we heard from a user that Beeline is blocking 3785:

К сожалению, у Билайн Казахстан заблокирован порт 3785, есть ли другой способ обхода блокировки?

We will try your bridge.

Фиксирую падение интернета до начала шатдауна:
ISP: Beeline KZ (“Интернет Дома”).
Время: 17:10 - 17:24 (GMT+6).
Таймаут до Google DNS.
Внутренние DNS провайдера остались доступны - dnstt работал.
Порты tcp/179 и tcp/3785 были заблокированы.

Tor used to have fteproxy bridge, which claimed to masquerade as unencrypted http. Although, I think it would be easy to block by fingerprint. Binary is still available, but no one is providing this type of bridge right now. However I would like to test it.
Binary is static with python embedded inside.

In networks with low bandwidth it would be useful to use HandyCache caching proxy.
It can also decrypt and cache https traffic, but this functionality in the trial mode only works for the first 30 minutes after each start of application (and then you need to restart HC). There is an English and Russian interface. Works in Wine.

IODA measurements say that access has been restored since about 2022-01-11 00:00 (06:00 Almaty time). Does that match people’s experience? I can access gov.kz now.

https://ioda.caida.org/ioda/dashboard#lastView=overview&view=inspect&entity=country/KZ&from=1641168000&until=1642032000

Yes, there were no more shutdowns.

I have shut down this bridge now.

Here are graphs of its usage over the past few days:
https://metrics.torproject.org/rs.html#details/0E9783A73F029E0910FD72F1EC120CA818868DA0

@anadahz pointed me to a RIPE Labs blog post on the shutdown. It notes that despite being “shut down,” networks in Kazakhstan were still present in the global BGP routing tables, which matches our experience with certain ports being unblocked. It also has some analysis of different levels of access in e.g. data centers versus residential connections.

It is difficult to pinpoint the cause of the outage. However, the affected networks have remained visible in the global routing system (BGP), which means they’ve remained “connected” to the Internet even though they have not been able to send or receive packets. The timing of the outage was synchronised, suggesting it was the result of some centralised action, although we do see small variations per region.

If we try to distinguish between RIPE Atlas vantage points in infrastructure - i.e. RIPE Atlas anchors and other probes with tags that suggest they are in data centres - we see differences in how connectivity developed over the last few days.

The figure below shows infrastructure vantage points in red. While connectivity for most of these vantage points went down in the last few days, it looks like most are able to send and receive packets to/from the Internet again since around midnight UTC on Friday 7 January. The other vantage points, which we think are mostly near end-users show that over the last few days there were periods of multiple hours where some of these vantage points had Internet connectivity.

After a few hours where almost all of our RIPE Atlas vantage points were online again, we see a drop again. If we look at infrastructure (data centres) versus other probes we do see that roughly half of the other probes (homes, offices, etc.) go down again, but many stay connected.

The comments on the post link to an interactive notebook for analyzing outages using RIPE Atlas.

I want to make a post that summarizes the important lessons from the January 2022 shutdown in Kazakhstan. I have written a draft in English (about 1200); is anyone willing to translate it to Kazakh and Russian before I post it?

https://pad.riseup.net/p/RQ1I5QI01qRfWZJBrUBD

You can also edit the document to add something you think is important. I’m planning to make the post next Monday, 2022-01-07.

I made translation to Russian. This is not a direct word-by-word translation, but more like my interpretation of the text according to typical Russian text constructs.