Фильтр TLS на домены youtube

ValdikSS · 2021-09-16T12:17:18.361Z

В проблемах к альтернативному youtube-плееру NewPipe для Android пишут, что плеер перестал работать в России.

NewPipe can't load anything

opened 08:54PM - 15 Sep 21 UTC

### Checklist - [x] I am using the latest version - 0.21.9 - [x] I checked, but didn't find any duplicates (open OR closed) of this issue in the repo. - [x] I have read the contribution guidelines given at https://github.com/TeamNewPipe/NewPipe/blob/HEAD/.github/CONTRIBUTING.md. - [x] This issue contains only one bug. I will open one issue for every bug report I want to file. ### Steps to reproduce the bug  1. Open app 2. Try to search or open anything from history, and it's loading endlessly ### Actual behavior Nothing loads. Tried reinstalling, nothing changes. ### Expected behavior Everytrhing should load. ### Screenshots/Screen recordings https://ibb.co/d29FXVt ### Logs ## Exception * __User Action:__ searched * __Request:__ lemmino * __Content Country:__ US * __Content Language:__ en-US * __App Language:__ en_US * __Service:__ YouTube * __Version:__ 0.21.9 * __OS:__ Linux Android 6.0.1 - 23 <details><summary>Crash log </summary> ``` java.net.ConnectException: Failed to connect to www.youtube.com/2a00:1450:4010:c05::c6:443 at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:249) at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:167) at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:258) at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135) at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114) at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:127) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257) at okhttp3.RealCall.execute(RealCall.java:93) at org.schabi.newpipe.DownloaderImpl.execute(DownloaderImpl.java:264) at org.schabi.newpipe.extractor.downloader.Downloader.post(Downloader.java:131) at org.schabi.newpipe.extractor.downloader.Downloader.post(Downloader.java:114) at org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.areHardcodedClientVersionAndKeyValid(YoutubeParsingHelper.java:342) at org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.getClientVersion(YoutubeParsingHelper.java:429) at org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.prepareDesktopJsonBuilder(YoutubeParsingHelper.java:814) at org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeSearchExtractor.onFetchPage(YoutubeSearchExtractor.java:73) at org.schabi.newpipe.extractor.Extractor.fetchPage(Extractor.java:54) at org.schabi.newpipe.extractor.search.SearchInfo.getInfo(SearchInfo.java:29) at org.schabi.newpipe.util.ExtractorHelper.lambda$searchFor$0(ExtractorHelper.java:81) at org.schabi.newpipe.util.-$$Lambda$ExtractorHelper$qyxpuXgomWa-cbONQns-pd7zxm0.call(lambda) at io.reactivex.rxjava3.internal.operators.single.SingleFromCallable.subscribeActual(SingleFromCallable.java:43) at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4813) at io.reactivex.rxjava3.internal.operators.single.SingleSubscribeOn$SubscribeOnObserver.run(SingleSubscribeOn.java:89) at io.reactivex.rxjava3.core.Scheduler$DisposeTask.run(Scheduler.java:614) at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:65) at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:56) at java.util.concurrent.FutureTask.run(FutureTask.java:237) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:269) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1113) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:588) at java.lang.Thread.run(Thread.java:818) Caused by: java.net.ConnectException: failed to connect to www.youtube.com/2a00:1450:4010:c05::c6 (port 443) after 10000ms: isConnected failed: ENETUNREACH (Network is unreachable) at libcore.io.IoBridge.isConnected(IoBridge.java:234) at libcore.io.IoBridge.connectErrno(IoBridge.java:171) at libcore.io.IoBridge.connect(IoBridge.java:122) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:183) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:452) at java.net.Socket.connect(Socket.java:884) at okhttp3.internal.platform.AndroidPlatform.connectSocket(AndroidPlatform.java:73) at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:247) ... 39 more Caused by: android.system.ErrnoException: isConnected failed: ENETUNREACH (Network is unreachable) at libcore.io.IoBridge.isConnected(IoBridge.java:223) ... 46 more ``` </details> <hr> ### Device info - Android version/Custom ROM version: CyanogenMod 13.1 - Device model: OnePlus One

Последний комментарий с исправлением кода намекает, что на ТСПУ внедрен какой-то фильтр TLS, не полностью совместимый со всем ПО.
Быть может, проблема в отсутствии SNI, а может и в чём-то другом. Жду PCAP’ов от пользователей.

ValdikSS · 2021-09-16T12:33:25.199Z

Это действительно какой-то фильтр TLS.

newpipe_youtube_notworking.zip (21.6 KB)
termux_curl_youtube_working.zip (1.7 MB)

Домен www.youtube.com резолвится в 173.194.220.198, но в случае NewPipe с сервера не приходит ответ на ClientHello.
Запрос к https://www.youtube.com на тот же IP-адрес через cURL работает без проблем.

ValdikSS · 2021-09-16T12:36:25.945Z

Еще один PCAP-файл, присланный пользователем программы.

По сообщениям пользователей, смена DNS возвращает работоспособность программе, предположительно, из-за другого IP-адреса домена www.youtube.com — фильтр включён с ограничением по адресу.

NewPipe_dns_issue.zip (1.2 KB)

ValdikSS · 2021-09-16T12:47:07.272Z

Comment

Phone A
× not working using mobile data (Beeline)
× not working using wifi wired connection (Dom.ru)

Phone B
✓ working using mobile data (Yota)
× not working using wifi wired connection (Dom.ru)

When I switch to mobile data (Yota), request channel list or video, it works like it should.
After that if I switch back to wi-fi (Dom.ru) it keeps working for some time and then stops loading anything again. In one case it worked for about 1 hour, in other for about 30 minutes

ValdikSS · 2021-09-16T13:02:50.436Z

Команда для проверки:

echo "1603010200010001fc0303b8fcb545209f995728937006732b2317cac1e7d265623198acdde74d267d9c68205bab4ebc80ebd23b030380212470f549430f8892e37be79ac1f682b52ac25626001e130113021303c02bc02ccca9c02fc030cca8c013c014009c009d002f00350100019500000014001200000f7777772e796f75747562652e636f6d00170000ff01000100000a00080006001d00170018000b00020100002300000010000e000c02683208687474702f312e31000500050100000000000d00140012040308040401050308050501080606010201003300260024001d0020a2ddf4a423d52380e4dbdbd817c8e4ebb714d39961dd144dfcaa9553c2ae0367002d00020101002b0009080304030303020301001500ea000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" | xxd -ps -r | nc -v 173.194.220.198 443 > /dev/null

ValdikSS · 2021-09-16T13:08:38.969Z

This is filter works specifically for www.youtube.com domain. Changing SNI to anything other (even wxw.youtube.com) opens the website again.

ValdikSS · 2021-09-16T13:37:07.043Z

TLS filtering is done regardless of IP address.
Here’s how it looks from a server perspective:
youtube-tspu-server.zip (4.2 KB)

As you can see, the clients’ TLS ClientHello packet is received, but the ServerHello reply can’t be delivered. The client does not receive ServerHello. www.youtube.com does not work, www.nottube.com works.

NAT session teardown?

ValdikSS · 2021-09-16T14:01:00.424Z

Фильтр настроен на блокировку комбинации TLS Cipher Suites + Supported Groups + Signature Algorithms + SNI. Удаление какого-либо из шифра в любой части, изменение порядка cipher suites или любая другая модификация приводит к доставке пакета и работоспособности сетевой связности.

vanyaindigo · 2021-09-16T14:01:44.783Z

А для чего именно может применяться такая настройка фильтра?

zhenyolka · 2021-09-16T14:01:52.488Z

@ValdikSS и снова я)
На операторе Tele2 только что заметил отвал newpipe, притом через браузер все нормально( даже www.youtube.com) открывается, твой тест парой постов выше тоже нормально проходит, хотя newpipe не пашет. Странно… Попробую запустить tcpdump

zhenyolka · 2021-09-16T14:02:59.496Z

Раз уже понятна схема блокировки, остался один вопрос: зачем так геморно и с таким паттерно? Почему нельзя просто заблочить по SNI? Или это специально для андроид клиента?

ValdikSS · 2021-09-16T14:03:56.303Z

Right now it seems like a TLS Fingerprinting to block an Android application which uses www.youtube.com as a domain fronting (wild guess: Navalny Android/iOS app?).

darkk · 2021-09-16T14:06:52.589Z

Seems, NewPipe’s fingerprint is quite unique. https://tlsfingerprint.io/ does not parse Client Hello from aforementioned pcaps as known fingerprint that was already observed in the campus traffic.

zhenyolka · 2021-09-16T14:11:02.826Z

Its really useless to block NewPipe. Best candidate to block is Google’s Youtube app. Can it be something like tests or preparations for whole youtube block?

darkk · 2021-09-16T14:36:39.279Z

ValdikSS:

wild guess: Navalny Android/iOS app

I have no iOS devices, but Navalny’s app (ver. 2.0, one from Google Play Market) gives a different fingerprint on my Android:

image866×635 103 KB

Fingerprints https://tlsfingerprint.io/id/f0b2b996867b6380 and https://tlsfingerprint.io/id/f278257cdf4a43aa seem to be related, cetpmhjmdf-vizskfsved.global.ssl.fastly.net is Navalny-related domain (see https://archive.md/xgT57 for archived content).

ValdikSS · 2021-09-16T14:43:27.030Z

darkk:

Seems, NewPipe’s fingerprint is quite unique.

They use okhttp3 library with ConnectionSpec.MODERN_TLS.cipherSuites() and 2 additional ones click

// This will try to enable all modern CipherSuites(+2 more)
// that are supported on the device.
// Necessary because some servers (e.g. Framatube.org)
// don't support the old cipher suites.
// https://github.com/square/okhttp/issues/4053#issuecomment-402579554
final List<CipherSuite> cipherSuites =
        new ArrayList<>(ConnectionSpec.MODERN_TLS.cipherSuites());
cipherSuites.add(CipherSuite.TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA);
cipherSuites.add(CipherSuite.TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA);
final ConnectionSpec legacyTLS = new ConnectionSpec.Builder(ConnectionSpec.MODERN_TLS)
        .cipherSuites(cipherSuites.toArray(new CipherSuite[0]))
        .build();

builder.connectionSpecs(Arrays.asList(legacyTLS, ConnectionSpec.CLEARTEXT));

UPD

UPD: no, they use this cipher set only in this exact case, but in general they have a common okhttp3 cipher list (without 3DES, it is probably filtered by the tls library): https://github.com/square/okhttp/blob/06644bb0507873e9a3b89d9107da537f1b140e91/okhttp/src/main/java/okhttp3/ConnectionSpec.java#L68

ValdikSS · 2021-09-16T15:28:55.574Z

I’ve made a small modification in NewPipe by replacing TLS_RSA_WITH_AES_256_CBC_SHA with TLS_RSA_WITH_AES_128_CBC_SHA in the library cipher list, and the application is now working again.

github.com/TeamNewPipe/NewPipe

NewPipe can't load anything

opened 08:54PM - 15 Sep 21 UTC

azat11

### Checklist - [x] I am using the latest version - 0.21.9 - [x] I checked, but didn't find any duplicates (open OR closed) of this issue in the repo. - [x] I have read the contribution guidelines given at https://github.com/TeamNewPipe/NewPipe/blob/HEAD/.github/CONTRIBUTING.md. - [x] This issue contains only one bug. I will open one issue for every bug report I want to file. ### Steps to reproduce the bug  1. Open app 2. Try to search or open anything from history, and it's loading endlessly ### Actual behavior Nothing loads. Tried reinstalling, nothing changes. ### Expected behavior Everytrhing should load. ### Screenshots/Screen recordings https://ibb.co/d29FXVt ### Logs ## Exception * __User Action:__ searched * __Request:__ lemmino * __Content Country:__ US * __Content Language:__ en-US * __App Language:__ en_US * __Service:__ YouTube * __Version:__ 0.21.9 * __OS:__ Linux Android 6.0.1 - 23 <details><summary>Crash log </summary> ``` java.net.ConnectException: Failed to connect to www.youtube.com/2a00:1450:4010:c05::c6:443 at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:249) at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:167) at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:258) at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135) at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114) at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:127) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257) at okhttp3.RealCall.execute(RealCall.java:93) at org.schabi.newpipe.DownloaderImpl.execute(DownloaderImpl.java:264) at org.schabi.newpipe.extractor.downloader.Downloader.post(Downloader.java:131) at org.schabi.newpipe.extractor.downloader.Downloader.post(Downloader.java:114) at org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.areHardcodedClientVersionAndKeyValid(YoutubeParsingHelper.java:342) at org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.getClientVersion(YoutubeParsingHelper.java:429) at org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.prepareDesktopJsonBuilder(YoutubeParsingHelper.java:814) at org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeSearchExtractor.onFetchPage(YoutubeSearchExtractor.java:73) at org.schabi.newpipe.extractor.Extractor.fetchPage(Extractor.java:54) at org.schabi.newpipe.extractor.search.SearchInfo.getInfo(SearchInfo.java:29) at org.schabi.newpipe.util.ExtractorHelper.lambda$searchFor$0(ExtractorHelper.java:81) at org.schabi.newpipe.util.-$$Lambda$ExtractorHelper$qyxpuXgomWa-cbONQns-pd7zxm0.call(lambda) at io.reactivex.rxjava3.internal.operators.single.SingleFromCallable.subscribeActual(SingleFromCallable.java:43) at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4813) at io.reactivex.rxjava3.internal.operators.single.SingleSubscribeOn$SubscribeOnObserver.run(SingleSubscribeOn.java:89) at io.reactivex.rxjava3.core.Scheduler$DisposeTask.run(Scheduler.java:614) at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:65) at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:56) at java.util.concurrent.FutureTask.run(FutureTask.java:237) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:269) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1113) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:588) at java.lang.Thread.run(Thread.java:818) Caused by: java.net.ConnectException: failed to connect to www.youtube.com/2a00:1450:4010:c05::c6 (port 443) after 10000ms: isConnected failed: ENETUNREACH (Network is unreachable) at libcore.io.IoBridge.isConnected(IoBridge.java:234) at libcore.io.IoBridge.connectErrno(IoBridge.java:171) at libcore.io.IoBridge.connect(IoBridge.java:122) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:183) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:452) at java.net.Socket.connect(Socket.java:884) at okhttp3.internal.platform.AndroidPlatform.connectSocket(AndroidPlatform.java:73) at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:247) ... 39 more Caused by: android.system.ErrnoException: isConnected failed: ENETUNREACH (Network is unreachable) at libcore.io.IoBridge.isConnected(IoBridge.java:223) ... 46 more ``` </details> <hr> ### Device info - Android version/Custom ROM version: CyanogenMod 13.1 - Device model: OnePlus One

Alex01d · 2021-09-16T15:32:17.010Z

SmartTubeNext тоже перестал работать, по крайней мере, с теле2.

ValdikSS · 2021-09-16T15:47:23.618Z

darkk:

Seems, NewPipe’s fingerprint is quite unique.

UPD: no, they use this cipher set only in this exact case, but in general they have a common okhttp3 cipher list (without 3DES, it is probably filtered by the tls library): https://github.com/square/okhttp/blob/06644bb0507873e9a3b89d9107da537f1b140e91/okhttp/src/main/java/okhttp3/ConnectionSpec.java#L68

So they block HTTPS requests to www.youtube.com via okhttp library.

ValdikSS · 2021-09-16T15:54:32.836Z

Hint: this block could be circumvented by changing SNI case. wWw.youtube.com works.
Hint: other domains, such as docs.google.com, are not blocked with this TLS fingerprint.