Wednesday 7 September 2011

Getting the AFL Videos from 2008 and 2009

This post continues a series starting at http://sandbelt-sandpit.blogspot.com/2011/09/how-to-download-afl-videos.html. The first post made the following remark:

"[AFL games] from 2008 and 2009 are available only as Windows Media rtsp streams. The quality is more-or-less the same: 672x544 resolution at a video rate of 1.99 mbps, though it turns out to be harder to determine the link address and harder again to download the streams to disk files."


So there are two problems: first we need the urls of the media streams we want, and then we need some means of recording them. But first some background knowledge.


rtsp?
This is Real Time Streaming Protocol, and all you need to know is that it is is a network control protocol designed for use in entertainment and communications systems, that some media players can handle it directly and some can't, and if they can't they may negotiate an alternative distribution over the very common http protocol. 


record media streams?
There are many providers that supply video on a watch-now basis, and they do it by packaging in stream form. These streams are not intended to be recorded, and this means that there are several commercial packages that do just that, in exchange for money. I have tried a few without much liking any of them, and nowadays I try to use VLC Media Player which is very powerful (if sometimes hard to drive) and also free. And also works on ten versions of Linux as well as OSX, iOS, Android and of course Windows. Everybody ought to have it.
VLC is supposed to handle rtsp streams but it doesn't do it very well, at least in version 1.1 11, and is much happier being fed over http. Fortunately this is possible.


How to discover the urls
and who doesn't need to know.
Second part first: you only need to read the following if (a) you want to, or (b) you want a game that Essendon wasn't in. Bombers fans can afford drop right down to the How to record a stream section because all the links they will need can be found at the end of this post.


But if you are still here, to continue, the DownloadHelper/FireFox solution used in my Post number 2  will not work with stream videos and you will have to resort to eavesdropping on your browser's communications with the outside world. The application I use for this is URL Snooper, a program written to help users locate the urls of audio and video files so that they can be recorded. Get this program from http://www.donationcoder.com/Software/Mouser/urlsnooper/index.htmlIt is donationware, which means that it is available for free but you can pay if your own sense of justice tells you to. It runs only on Windows, and (sorry) I don't know about anything else.


So, assuming you have a Windows machine, install URL Snooper, turn it on and click the Sniff Network button. The Keyword Filter box is to reduce clutter and it would be good at this point to type a team name in there.


 As previously, go to BigPond AFL Videos and use the fancy navigator to select ON DEMAND, Essendon TV (or some other club if that is what you really want) and Matches. Set the quality as high as possible in the web player. Pick a match and play it.



URL Snooper will have picked up all the media links that the browser made requests for:


If you can read this image you will observe a whole list of rstp stream requests, including the one we are playing (rtsp://bptvvod.ngcdn.telstra.com/wh_ivideo/AFL/ONDEMAND/2009/ON/iVideo/Premiership/RD13/NV_Rd13_EssendonVCarlton.wmv?WMBitrate=2621440). Further down you see the same game appearing as a WindowsMedia http file, and its URL is otherwise identical except for a changed server name (now fli-cdn220-is-1.se.bptvvod.ngcdn.telstra.com). 


At this point it helps to know that BigPond has a lot of servers and uses a sort of alias system to direct requests to them. The bptvvod.ngcdn.telstra.com server name seems to be an umbrella that resolves to all the other alternatives, or at least it has so far. We might then guess that we could ignore the server name and just change the rtsp:// part of the stream URL to http://, and sure enough the following procedure has proven to be reliable:
  • Play the game, snoop its rstp URL
  • Edit this url so it starts with http:// instead of rstp://.

How to record the stream.

Here is where we use VLC. Get it from http://www.videolan.org/vlc/ (if you don't have it already) and fire it up.

Click the Media menu and select Open Network Stream...
Paste in your edited (http://) game URL and click the little down arrow to the right of the Play button:



Write in a destination file name ending in .mp4. Optionally check the Display the output box.


Click Start. You are up and running.

The recording runs in real time, so a 3-hour video will take 3 hours to record. On the other hand if you have a good internet connection you can start several instances of VLC and record several games at once.


Update: A problem with VLC 2.0?

There have been changes to VLC since all I first wrote all of the above and it seems to have developed a bug that causes it to crash. But you can always just use an older version, such as 1.1.11 which you can download from HERE. The goodish news is that VLC is a portable app that does not need to be actually installed, so you can download the old version as a zip file, unpack it somewhere (even to a USB stick) and then just run its main program (vlc.exe) from your file manager.


For Bombers Fans

Friendly sandbelt has constructed lists of URLs of essendon games for the two years that this video format was current for. Some browsers (for me, only Internet Explorer 9) allow you to check them out directly by clicking the links. But you can always right-click the link, choose Copy, and then paste into VLC to make a recording.


2009


R01 Port Adelaide 15.17 (107) Essendon 9.12 (66) video stream
R02 Essendon 16.13 (109) Fremantle 10.11 (71) video stream
R03 Carlton 16.16 (112) Essendon 17.14 (116) video stream
R04 North Melbourne 10.9 (69) Essendon 7.15 (57) video stream
R05 Essendon 13.15 (93) Collingwood 12.16 (88) video stream
R06 Brisbane Lions 17.9 (111) Essendon 9.14 (68) video stream
R07 Essendon 17.14 (116) Hawthorn 10.12 (72) video stream
R08 St Kilda 13.12 (90) Essendon 10.11 (71) video stream
R09 Richmond (*) 12.13 (85) Essendon 19.11 (125) video stream
R10 Essendon 11.4 (70) Geelong 20.14 (134) video stream
R11 Essendon 18.6 (114) Adelaide 21.4 (130) video stream
R12 Essendon 19.17 (131) Melbourne 13.5 (83) video stream
R13 Essendon 21.10 (136) Carlton 9.13 (67) video stream
R14 Collingwood 15.12 (102) Essendon 9.13 (67) video stream
R15 Sydney 10.12 (72) Essendon 15.17 (107) video stream
R16 Essendon 11.4 (70) Western Bulldogs 15.13 (103) video stream
R17 Essendon 14.12 (96) Richmond 15.11 (101) video stream
R18 West Coast 14.11 (95) Essendon 10.8 (68) unavailable
R19 Essendon 13.9 (87) Brisbane Lions 12.15 (87) unavailable
R20 Essendon 16.14 (110) St Kilda 16.12 (108) video stream
R21 Fremantle 21.15 (141) Essendon 13.9 (87) video stream
R22 Hawthorn 14.15 (99) Essendon 16.20 (116) video stream
EF1 Adelaide 26.10 (166) Essendon 10.10 (70) video stream
(*) Low quality


2008


R01 North Melbourne 9.13 (67) Essendon 19.8 (122) video stream
R02 Geelong 22.18 (150) Essendon 6.15 (51) video stream
R03 Essendon 23.12 (150) Carlton 21.8 (134) video stream
R04 Essendon 14.14 (98) Western Bulldogs 19.14 (128) video stream
R05 St Kilda 18.15 (123) Essendon 14.3 (87) video stream
R06 Collingwood 23.16 (154) Essendon 12.9 (81) video stream
R07 Essendon 15.4 (94) Port Adelaide 24.14 (158) video stream
R08 Sydney 21.17 (143) Essendon 7.10 (52) video stream
R09 Essendon 10.12 (72) Richmond 16.14 (110) video stream
R10 Adelaide 9.20 (74) Essendon 10.9 (69) video stream
R11 Essendon 12.7 (79) Hawthorn 19.16 (130) video stream
R12 Essendon 16.17 (113) West Coast 13.13 (91) video stream
R13 Carlton 15.11 (101) Essendon 20.16 (136) video stream
R14 Fremantle 13.13 (91) Essendon 14.11 (95) video stream
R15 Essendon 24.11 (155) Brisbane Lions 18.10 (118) video stream
R16 Richmond 16.12 (108) Essendon 15.14 (104) video stream
R17 Essendon 19.14 (128) Collingwood 11.14 (80) video stream
R18 Essendon 19.10 (124) Melbourne 17.6 (108) video stream
R19 West Coast 17.11 (113) Essendon 16.7 (103) video stream
R20 Essendon 10.13 (73) Adelaide 19.15 (129) video stream
R21 Western Bulldogs 23.13 (151) Essendon 15.15 (105) video stream
R22 Essendon 5.9 (39) St Kilda 21.21 (147) video stream


The final post in this series, when I write it,  will be about how to record the videos from 2007. This will be getting very much into the land of diminishing returns since their quality is so low, but never mind, completeness is a virtue all in itself.









Getting the AFL Videos from 2010 and 2011

All games from 2010 and 2011 are stored on the BigPond servers as simple mp4 files and would be easy to download if anybody had a link to them. The AFL website used to make this a simple problem, but the world is full of spoilsports and nowadays all the file links are hidden behind a wall of javascript. The BigPond AFL videos website is no friendlier. 


But it is not hard to discover a video link if you use FireFox.


Using FireFox and Download Helper


You can install DownloadHelper from the FireFox Add-ons manager or else by going directly to http://www.downloadhelper.net/. Both FireFox and DownloadHelper are (of course) free.
Then go to BigPond AFL Videos and use the fancy navigator to select ON DEMANDEssendon TV (OK, some other club if that is what you really want) and Matches. Pick a match and it will start to play in an embedded Flash machine. Do not omit to set the quality as high as possible (there is a control for that under the video, on the right).


Now you will notice that the the DownloadHelper icon at the top of your browser has started to animate, and if you click the little down arrow next to the icon you will see the name of the file you are playing.


Wave your mouse over this name and one of your options is to download the file. Another option is to copy the file's URL, and you can do this if you prefer to download later or if (like me) you plan to share the link with somebody else. 


That's about it. As far as I know, any download from a BigPond server is unmetered for anybody who uses BigPond for internet access. I have never bothered to confirm this because I am signed up with iinet and it comes of my quota in any case. Perhaps somebody might comment. 


Patterns.


After you have discovered a few of these link addresses you start to see that they follow a predictable pattern. This is because the Telstra engineers use scripts to create the video files in the first place, and it has the benefit of making it easy for people like myself to use scripts to download them. It also means that you can predict the urls for any game from a few simple rules.


For most of 2010 and all of 2011 the pattern has been like this:
Q1 http://bptvpd.ngcdn.telstra.com/pd_afl0/OnDemand/2011/ON/iVideo/Premiership/RDnn/AFL11_rdnn_hhh_vs_aaa_1st_qr_full_2M.mp4
Q2 http://bptvpd.ngcdn.telstra.com/pd_afl0/OnDemand/2011/ON/iVideo/Premiership/RDnn/AFL11_rdnn_hhh_vs_aaa_2nd_qr_full_2M.mp4
Q3 http://bptvpd.ngcdn.telstra.com/pd_afl0/OnDemand/2011/ON/iVideo/Premiership/RDnn/AFL11_rdnn_hhh_vs_aaa_3rd_qr_full_2M.mp4
Q4 http://bptvpd.ngcdn.telstra.com/pd_afl0/OnDemand/2011/ON/iVideo/Premiership/RDnn/AFL11_rdnn_hhh_vs_aaa_4th_qr_full_2M.mp4


To make real links, replace nn (in both places) with the round number, hhh with the home team name and aaa with the away team name. Round numbers are 01, 02, 03 and so on. Team names are adel, bl, carl, coll, ess, fre, geel, gcfc, haw, melb, nmfc, port, rich, stk, syd, wce or wb.
The last half of 2010 used the same pattern except that 2011 used to be 2010 and AFL11 was  AFL10. You can see the pattern working in the tables underneath.


For Bombers Fans


Friendly sandbelt has a list of all Essendon matches from 2010 and 2011 (corrected 9/9/2011). If you want anything, just right-click a link:


2011



R01 ess v wb Q1 Q2 Q3 Q4
R02 syd v ess Q1 Q2 Q3 Q4
R03 stk v ess Q1 Q2 Q3 Q4
R04 carl v ess Q1 Q2 Q3 Q4
R05 ess v coll Q1 Q2 Q3 Q4
R06 ess v gcfc Q1 Q2 Q3 Q4
R07 ess v wce Q1 Q2 Q3 Q4
R08 bl v ess Q1 Q2 Q3 Q4
R09 rich v ess Q1 Q2 Q3 Q4
R11 ess v melb Q1 Q2 Q3 Q4
R12 fre v ess Q1 Q2 Q3 Q4
R13 ess v nmfc Q1 Q2 Q3 Q4


R14 haw v ess Q1 Q2 Q3 Q4
R15 ess v geel Q1 Q2 Q3 Q4
R16 ess v rich Q1 Q2 Q3 Q4
R17 adel v ess Q1 Q2 Q3 Q4
R18 ess v carl Q1 Q2 Q3 Q4
R19 coll v ess Q1 Q2 Q3 Q4
R20 ess v syd Q1 Q2 Q3 Q4
R21 wb v ess Q1 Q2 Q3 Q4
R22 wce v ess Q1 Q2 Q3 Q4
R23 ess v port Q1 Q2 Q3 Q4
EF1 carl v ess Q1 Q2 Q3 Q4
Mind you. I'm guessing about the final, it hasn't happened yet as I write this.


2010
R01 geel v ess All
R02 ess v fre All
R03 carl v ess All
R04 wce v ess All
R05 coll v ess All
R06 ess v haw All
R07 ess v port All
R08 stk v ess All
R09 ess v rich Q1 Q2 Q3 Q4
R10 ess v wb Q1 Q2 Q3 Q4
R11 syd v ess Q1 Q2 Q3 Q4
R12 ess v geel Q1 Q2 Q3 Q4
R13 haw v ess Q1 Q2 Q3 Q4
R14 adel v ess Q1 Q2 Q3 Q4
R15 melb v ess Q1 Q2 Q3 Q4
R16 ess v wce Q1 Q2 Q3 Q4
R17 nmfc v ess Q1 Q2 Q3 Q4
R18 ess v stk Q1 Q2 Q3 Q4
R19 ess v carl Q1 Q2 Q3 Q4
R20 ess v coll Q1 Q2 Q3 Q4
R21 ess v bl Q1 Q2 Q3 Q4
R22 wb v ess Q1 Q2 Q3 Q4

The next post will go into what you have to do to download the media streams from 2009 and 2008. I will eventually even cover 2007, because even though the quality is lousy there is some good historical stuff there, and I admit I got a kick out of seeing Mark Johnson with long white hair.





Sunday 4 September 2011

How to download AFL videos

Telstra (bless this corporation) have been assembling a very useful archive of AFL videos for the last several years. You can find and watch them on http://bigpondvideo.com/AFL/?ref=Net-Head-TV-Sport-AFL, in the ON DEMAND panel. For Essendon matches, the only ones that have any interest, follow the fancy selector through Essendon TV to Matches and start browsing through.

The subject of this blog is: can I download these videos to my computer so I can save them in my home entertainment system and watch them on my TV?


What's on offer.


All games from last year (2010) up to the present are stored on the BigPond servers as mp4 files with resolutions of  720x400 and video data rates of 2051 kbps, and they look quite respectable on a big screen. Furthermore they are easy to download - all it takes is a right-click in a browser, provided you have the links, which I can provide. They amount to a little under a gigabyte per hour in terms of file size.

Games from 2008 and 2009 are available only as Windows Media rtsp streams. The quality is more-or-less the same: 672x544 resolution at a video rate of 1.99 mbps, though it turns out to be harder to determine the link address and harder again to download the streams to disk files.

The games from 2007 are barely watchable except on a smartphone or iPod: resolution is poor (352 x 288) and the data rate is down to 1.19 mbps. They are also Windows Media rtsp streams with all the problems that come with that format.

There is nothing much before 2007.

Who might want to read more.

  • Essendon fans who want to build up a video archive of past games. I have done most of the work for you already.
  • Followers of any other team who want to build up a video archive of past games. There may be more for you to do, but I can show you how to go about it.

The way ahead.

The next several posts in this series will have to do with
  • listing the video links from 2010 up to the present
  • downloading these files
  • discovering the video links from earlier games
  • downloading these earlier games
Some of the instructions will be Microsoft-Windows-only, but most of them will work equally well in Linux or on an Apple.

This blog is being written in 2011 and there is no knowing what BigPond might do in 2012