Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
nginx pull youtube rtmp
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
Vieri
Guru
Guru


Joined: 18 Dec 2005
Posts: 495

PostPosted: Wed Apr 04, 2018 9:52 am    Post subject: nginx pull youtube rtmp Reply with quote

Hi,

This is my first try setting up nginx to retransmit a video stream or playlist from YouTube.

I have no idea if this is legal or not, but it would be technically useful in my case. The idea is to have an nginx server pull the youtube stream/s down by just using one connection, thus saving bandwidth.
This way, clients in my LAN would only connect to my nginx server to play the stream/s.

I understand I could configure nginx with something like:

Code:

rtmp {
    server {
        listen 1935;
        application youtube {
            live on;
            hls on;
            hls_nested on;
            hls_path /tmp/youtube;
            hls_fragment 15s;
            pull rtmp://a.rtmp.youtube.com/live2/streamkey;
            allow play all;
        }
    }
}


Client could then connect via RTMP to the nginx service to view the pulled stream (in this case, they could also connect via HTTP HLS if I set up the server clause properly).

The question I'd like to ask though is how do I get the rtmp "pull" address from a publically available youtube video.
Let's just take an example: https://www.youtube.com/watch?v=RtU_mdL2vBM

Here's what I do:

Code:
youtube-dl --list-formats https://www.youtube.com/watch?v=RtU_mdL2vBM
[youtube] RtU_mdL2vBM: Downloading webpage
[youtube] RtU_mdL2vBM: Downloading video info webpage
[youtube] RtU_mdL2vBM: Extracting video information
[youtube] RtU_mdL2vBM: Downloading m3u8 information
[youtube] RtU_mdL2vBM: Downloading MPD manifest
[info] Available formats for RtU_mdL2vBM:
format code  extension  resolution note
91           mp4        256x144    HLS  197k , avc1.42c00b, 30.0fps, mp4a.40.5@ 48k
92           mp4        426x240    HLS  338k , avc1.4d4015, 30.0fps, mp4a.40.5@ 48k
93           mp4        640x360    HLS  829k , avc1.4d401e, 30.0fps, mp4a.40.2@128k
94           mp4        854x480    HLS 1380k , avc1.4d401f, 30.0fps, mp4a.40.2@128k
300          mp4        1280x720   3806k , avc1.4d4020, 60.0fps, mp4a.40.2
301          mp4        1920x1080  6231k , avc1.64002a, 60.0fps, mp4a.40.2 (best)


Let's say I want to stream format 300.
I run these:

Code:
youtube-dl -f 300 -g https://www.youtube.com/watch?v=RtU_mdL2vBM
https://manifest.googlevideo.com/api/manifest/hls_playlist/id/RtU_mdL2vBM.2/itag/300/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/cmbypass/yes/goi/160/sgoap/gir%3Dyes%3Bitag%3D140/sgovp/gir%3Dyes%3Bitag%3D298/hls_chunk_host/r1---sn-25glen7l.googlevideo.com/ei/oZHEWunOJcvCVrXyqeAO/playlist_type/DVR/gcr/es/mm/32/mn/sn-25glen7l/ms/lv/mv/u/pl/21/dover/10/keepalive/yes/playlist_duration/30/manifest_duration/30/mt/1522831429/ip/79.148.120.225/ipbits/0/expire/1522853377/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,goi,sgoap,sgovp,hls_chunk_host,ei,playlist_type,gcr,mm,mn,ms,mv,pl/signature/470A5AE7785554ABFE0F9B2EE9DB4697D96A3036.2BC0BF6D2C7E4C59BA80C817ECE1E371E4DE2308/key/dg_yt0/playlist/index.m3u8


Code:
youtube-dl -f 300 --get-id https://www.youtube.com/watch?v=RtU_mdL2vBM
RtU_mdL2vBM


Code:
youtube-dl -f 300 --get-filename https://www.youtube.com/watch?v=RtU_mdL2vBM
NASA Live - Earth From Space (HDVR) ♥ ISS LIVE FEED #AstronomyDay2018 _ Subscribe now!-RtU_mdL2vBM.mp4


The problem is that I'm not getting an RTMP address, so I don't really know how to configure my nginx efficiently (I cannot pull a non-RTMP URL).
This is a live stream on youtube so I don't think I can just "play" the mp4 listed above, as nginx will try to fully download it first.

I also used another youtube link analyzer (Firefox add-on) and tried the following in nginx:

Code:
       application yt0 {
           live on;
           play https://manifest.googlevideo.com/api/manifest/hls_playlist/id/RtU_mdL2vBM.2/itag/300/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/cmbypass/yes/goi/160/sgoap/gir%3Dyes%3Bitag%3D140/sgovp/gir%3Dyes%3Bitag%3D298/hls_chunk_host/r1---sn-25glen7l.googlevideo.com/ei/oZHEWunOJcvCVrXyqeAO/playlist_type/DVR/gcr/es/mm/32/mn/sn-25glen7l/ms/lv/mv/u/pl/21/dover/10/keepalive/yes/playlist_duration/30/manifest_duration/30/mt/1522831429/ip/79.148.120.225/ipbits/0/expire/1522853377/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,goi,sgoap,sgovp,hls_chunk_host,ei,playlist_type,gcr,mm,mn,ms,mv,pl/signature/470A5AE7785554ABFE0F9B2EE9DB4697D96A3036.2BC0BF6D2C7E4C59BA80C817ECE1E371E4DE2308/key/dg_yt0/playlist/index.m3u8
           hls on;
           hls_nested on;
           hls_path /var/www/localhost/nginx/hls;
           hls_fragment 15s;
           hls_playlist_length 60;
           allow play all;
       }

        application yt1 {
            live on;
            play https://r3---sn-25glenes.googlevideo.com/videoplayback?source=youtube&mt=1522822113&mv=m&mime=video%2Fmp4&signature=0CF07AE485631DAE9564218AF34545317ABB2788.BB80999806966E5B5155CE810B8A147380133588&ms=au%2Conr&fvip=3&dur=124.180&id=o-AGJOTp70N_3VGXxQIMmSesaMUzRY5ydVBTuqVTzJc6Ag&initcwndbps=755000&pl=21&ipbits=0&sparams=dur%2Cei%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Cratebypass%2Crequiressl%2Csource%2Cexpire&itag=22&ip=79.148.120.225&mm=31%2C26&mn=sn-25glenes%2Csn-4g5e6nl6&requiressl=yes&ratebypass=yes&expire=1522843817&key=yt6&c=WEB&ei=SWzEWqmLI9XIVMC_t4gM&lmt=1411089788247710
            hls on;
            hls_nested on;
            hls_path /var/www/localhost/nginx/hls;
            hls_fragment 15s;
            hls_playlist_length 60;
            allow play all;
        }


None of these work.

On the other hand, I can open and play the index.m3u8 URL with VLC just fine.

How can I get the Youtube RTMP address, or alternatively, how can I retransmit the youtube stream on the nginx server, so that my LAN clients can connect to the local server instead of the youtube servers?

Thanks,

Vieri
Back to top
View user's profile Send private message
Tatsh
Apprentice
Apprentice


Joined: 22 Jul 2007
Posts: 175

PostPosted: Sat Apr 07, 2018 12:48 pm    Post subject: Reply with quote

If you switch your user-agent to an old device you can get the RTSP URL. By default this is the low-quality URL. There's an HQ button on the page.

I used Chromium with the user agent switcher extension and I used this user agent:

Code:
BlackBerry9800/6.6.0.207 Profile/MIDP-2.0 Configuration/CLDC-1.1/UCWEB8.1.0.104/161/999


Once you load a video on YouTube you'll be redirected to the mobile page. Click the HQ button. The 'Watch video' link will now point to the HQ RTSP link. Copy that and use it.

You can try the URL with ffplay:

Code:
ffplay -i 'rtsp://r2---sn-vgqskned.googlevideo.com/Cj0LENy73wIaNAlq0fI...../video.3gp'


With cURL:

Code:
curl -q --user-agent 'BlackBerry9800/6.6.0.207 Profile/MIDP-2.0 Configuration/CLDC-1.1/UCWEB8.1.0.104/161/999' -L 'https://www.youtube.com/watch?v=xtzRxeaf5NQ' | fgrep rtsp | sed -r -e 's/<a[^"]+"([^"]+)".*/\1/' | sort -u


This will only give the 'normal' quality stream. Probably need to send a cookie to get the HQ one.
Back to top
View user's profile Send private message
Vieri
Guru
Guru


Joined: 18 Dec 2005
Posts: 495

PostPosted: Sat Apr 07, 2018 9:57 pm    Post subject: Reply with quote

Thanks for the feedback.

Although your method works for the youtube url you mention,it doesn't for the one I posted. There's no rtsp url in the reply.

Anyway, the nginx rtmp module cannot handle rtsp directly ASAIK unless I put ffmpeg in the middle.

I guess I'd be better off using the m3u8 files directly with something like:
exec_pull ffmpeg <youtube_http_m3u8_url_as_input> <local_rtmp_as_output>
Then I'd use this rtmp stream to create the HLS and DASH HTTP streams locally with nginx.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum