Tuesday 5 December 2017

It's Dead

Regrettable, but inevitable.  The AFL's replay infrastructure has been completely changed as of late 2017.



The downloader is broken, and there is no prospect of it ever working again.


Saturday 6 June 2015

Video File Downloader, 2015-2017


At the beginning of 2013 the friendly Telstra engineers threw away the simple filing pattern that was once in place, with the result that the video links no longer showed any sensible pattern at all. It was possible to discover them after the replays went live but it was apparently not possible to predict anything ahead of time. So for two years the community had to be served by altruistic contributors who posted the game links, one by one, as they went up.

The situation became worse in 2015 when Telstra decided to put  all their replays past and present behind a paywall. This reduced the number of people who could discover the links in the first place (at least by any of the methods covered earlier in this series) to those who had paid for an AFL Live Pass.

But a solution to the problem soon came to light, and Sandbelt is indebted to Zappy_32 who in April posted a very useful algorithm on BigFooty's Match Replay Thread.  The algorithm (which I won't go so far as to repeat here) is so regular that it invites someone to write a program to automate it.

And so, Sandbelt Software presents a replay downloader for Windows that gives access to everything from Round 1 2010 onwards. It looks like this:



The latest edition (as of July 2017) is Version 1.0.6. This version includes a function to copy the video links of all four quarters of a game to the Windows clipboard, in case you want to paste them somewhere. It also provides an option to create an NFO file for a Kodi library, which I imagine will interest only a small minority but which is useful to me. As last tested, the app was still functional through the 2017 season.

There is no installer: just download the zip file, extract AFLReplays.exe to somewhere convenient and run it. Be aware though that the AFL servers are geoblocked to Australia and NZ, so if you live elsewhere you will need a VPN.

Other OSs

AFLReplays is a Windows program, based on the .NET framework. As such it has been run successfully in Linux under mono, installed like this. By extension it will also run in MacOS, though Sandbelt Software offers no guides or technical assistance in this respect.

Caveats
This program, as goes without saying, come without warranty, implied support or express permissions other than that you can do what you like in private.





Monday 14 May 2012

Update for the 2012 season

This post continues a series starting at http://sandbelt-sandpit.blogspot.com/2011/09/how-to-download-afl-videos.html. The series was mostly composed in 2011 when there was no guarantee of what might happen in the next year. Now it is a third of the way into the 2012 season and it is clear that the engineers at Telstra are happy to continue with their previous practices, at least as far as locating the game videos is concerned, and this is a good thing.

To recap:

All games from 2010 onwards are stored on the Telstra servers as simple mp4 files that are easy to download once anybody has a link to them. The AFL website used to make this simple to do, but in recent years all the file links have been hidden behind a wall of javascript (the BigPond AFL videos website is just as bad). 


The 7 September 2011 post went into some detail about how you can discover the file links and ended up with a pattern that could predict the url for any quarter of any game in 2011. I am now fairly confident about how to make predictions for the current year.


The Pattern:

The links for 2012 look like this, so far:
Q1 
http://bptvpd.ngcdn.telstra.com/pd_afl0/OnDemand/2012/ON/iVideo/Premiership/RDnn/AFL12_rdnn_hhh_vs_aaa_1st_qr_full_2M.mp4
Q2 
http://bptvpd.ngcdn.telstra.com/pd_afl0/OnDemand/2012/ON/iVideo/Premiership/RDnn/AFL12_rdnn_hhh_vs_aaa_2nd_qr_full_2M.mp4
Q3  
http://bptvpd.ngcdn.telstra.com/pd_afl0/OnDemand/2012/ON/iVideo/Premiership/RDnn/AFL12_rdnn_hhh_vs_aaa_3rd_qr_full_2M.mp4
Q4  
http://bptvpd.ngcdn.telstra.com/pd_afl0/OnDemand/2012/ON/iVideo/Premiership/RDnn/AFL12_rdnn_hhh_vs_aaa_4th_qr_full_2M.mp4


To make real links, replace nn (in both places) with the round number, hhh with the home team name and aaa with the away team name. Round numbers are like 01, 02, 03 and so on. Team names are adel, bl, carl, coll, ess, fre, geel, gcfc, gws, haw, melb, nmfc, port, rich, stk, syd, wce or wb.

One minor glitch:
Gold Coast has to be written gdfc in (and only in) rounds 02 and 03. I assume that this is somebody's carelessness.

For Bombers Fans:
Sandbelt has used this pattern to predict where the Essendon videos will be for the whole of all the current home-and-away season. Left-click a table cell to view, right-click (or, I hear, Apple+click) to download. 

R01
nmfc v ess Q1 Q2 Q3 Q4
R02
ess v port Q1 Q2 Q3 Q4
R03
gcfc v ess Q1 Q2 Q3 Q4
R04
carl v ess Q1 Q2 Q3 Q4
R05
coll v ess Q1 Q2 Q3 Q4
R06
ess v bl Q1 Q2 Q3 Q4
R07
ess v wce Q1 Q2 Q3 Q4
R08
ess v rich Q1 Q2 Q3 Q4
R09
gws v ess Q1 Q2 Q3 Q4
R10
ess v melb Q1 Q2 Q3 Q4
R11
ess v syd Q1 Q2 Q3 Q4
R12
ess: bye



R13
fre v ess Q1 Q2 Q3 Q4
R14
ess v wb Q1 Q2 Q3 Q4
R15
stk v ess Q1 Q2 Q3 Q4
R16
port v ess Q1 Q2 Q3 Q4
R17
geel v ess Q1 Q2 Q3 Q4
R18
ess v haw Q1 Q2 Q3 Q4
R19
adel v ess Q1 Q2 Q3 Q4
R20
ess v nmfc Q1 Q2 Q3 Q4
R21
ess v carl Q1 Q2 Q3 Q4
R22
rich v ess Q1 Q2 Q3 Q4
R23
ess v coll Q1 Q2 Q3 Q4
 







Downloads do not count towards your quota if you are a BigPond customer.
Videos typically appear a few hours after the game ends.

Wednesday 7 September 2011

Getting the AFL Videos from 2008 and 2009

This post continues a series starting at http://sandbelt-sandpit.blogspot.com/2011/09/how-to-download-afl-videos.html. The first post made the following remark:

"[AFL games] from 2008 and 2009 are available only as Windows Media rtsp streams. The quality is more-or-less the same: 672x544 resolution at a video rate of 1.99 mbps, though it turns out to be harder to determine the link address and harder again to download the streams to disk files."


So there are two problems: first we need the urls of the media streams we want, and then we need some means of recording them. But first some background knowledge.


rtsp?
This is Real Time Streaming Protocol, and all you need to know is that it is is a network control protocol designed for use in entertainment and communications systems, that some media players can handle it directly and some can't, and if they can't they may negotiate an alternative distribution over the very common http protocol. 


record media streams?
There are many providers that supply video on a watch-now basis, and they do it by packaging in stream form. These streams are not intended to be recorded, and this means that there are several commercial packages that do just that, in exchange for money. I have tried a few without much liking any of them, and nowadays I try to use VLC Media Player which is very powerful (if sometimes hard to drive) and also free. And also works on ten versions of Linux as well as OSX, iOS, Android and of course Windows. Everybody ought to have it.
VLC is supposed to handle rtsp streams but it doesn't do it very well, at least in version 1.1 11, and is much happier being fed over http. Fortunately this is possible.


How to discover the urls
and who doesn't need to know.
Second part first: you only need to read the following if (a) you want to, or (b) you want a game that Essendon wasn't in. Bombers fans can afford drop right down to the How to record a stream section because all the links they will need can be found at the end of this post.


But if you are still here, to continue, the DownloadHelper/FireFox solution used in my Post number 2  will not work with stream videos and you will have to resort to eavesdropping on your browser's communications with the outside world. The application I use for this is URL Snooper, a program written to help users locate the urls of audio and video files so that they can be recorded. Get this program from http://www.donationcoder.com/Software/Mouser/urlsnooper/index.htmlIt is donationware, which means that it is available for free but you can pay if your own sense of justice tells you to. It runs only on Windows, and (sorry) I don't know about anything else.


So, assuming you have a Windows machine, install URL Snooper, turn it on and click the Sniff Network button. The Keyword Filter box is to reduce clutter and it would be good at this point to type a team name in there.


 As previously, go to BigPond AFL Videos and use the fancy navigator to select ON DEMAND, Essendon TV (or some other club if that is what you really want) and Matches. Set the quality as high as possible in the web player. Pick a match and play it.



URL Snooper will have picked up all the media links that the browser made requests for:


If you can read this image you will observe a whole list of rstp stream requests, including the one we are playing (rtsp://bptvvod.ngcdn.telstra.com/wh_ivideo/AFL/ONDEMAND/2009/ON/iVideo/Premiership/RD13/NV_Rd13_EssendonVCarlton.wmv?WMBitrate=2621440). Further down you see the same game appearing as a WindowsMedia http file, and its URL is otherwise identical except for a changed server name (now fli-cdn220-is-1.se.bptvvod.ngcdn.telstra.com). 


At this point it helps to know that BigPond has a lot of servers and uses a sort of alias system to direct requests to them. The bptvvod.ngcdn.telstra.com server name seems to be an umbrella that resolves to all the other alternatives, or at least it has so far. We might then guess that we could ignore the server name and just change the rtsp:// part of the stream URL to http://, and sure enough the following procedure has proven to be reliable:
  • Play the game, snoop its rstp URL
  • Edit this url so it starts with http:// instead of rstp://.

How to record the stream.

Here is where we use VLC. Get it from http://www.videolan.org/vlc/ (if you don't have it already) and fire it up.

Click the Media menu and select Open Network Stream...
Paste in your edited (http://) game URL and click the little down arrow to the right of the Play button:



Write in a destination file name ending in .mp4. Optionally check the Display the output box.


Click Start. You are up and running.

The recording runs in real time, so a 3-hour video will take 3 hours to record. On the other hand if you have a good internet connection you can start several instances of VLC and record several games at once.


Update: A problem with VLC 2.0?

There have been changes to VLC since all I first wrote all of the above and it seems to have developed a bug that causes it to crash. But you can always just use an older version, such as 1.1.11 which you can download from HERE. The goodish news is that VLC is a portable app that does not need to be actually installed, so you can download the old version as a zip file, unpack it somewhere (even to a USB stick) and then just run its main program (vlc.exe) from your file manager.


For Bombers Fans

Friendly sandbelt has constructed lists of URLs of essendon games for the two years that this video format was current for. Some browsers (for me, only Internet Explorer 9) allow you to check them out directly by clicking the links. But you can always right-click the link, choose Copy, and then paste into VLC to make a recording.


2009


R01 Port Adelaide 15.17 (107) Essendon 9.12 (66) video stream
R02 Essendon 16.13 (109) Fremantle 10.11 (71) video stream
R03 Carlton 16.16 (112) Essendon 17.14 (116) video stream
R04 North Melbourne 10.9 (69) Essendon 7.15 (57) video stream
R05 Essendon 13.15 (93) Collingwood 12.16 (88) video stream
R06 Brisbane Lions 17.9 (111) Essendon 9.14 (68) video stream
R07 Essendon 17.14 (116) Hawthorn 10.12 (72) video stream
R08 St Kilda 13.12 (90) Essendon 10.11 (71) video stream
R09 Richmond (*) 12.13 (85) Essendon 19.11 (125) video stream
R10 Essendon 11.4 (70) Geelong 20.14 (134) video stream
R11 Essendon 18.6 (114) Adelaide 21.4 (130) video stream
R12 Essendon 19.17 (131) Melbourne 13.5 (83) video stream
R13 Essendon 21.10 (136) Carlton 9.13 (67) video stream
R14 Collingwood 15.12 (102) Essendon 9.13 (67) video stream
R15 Sydney 10.12 (72) Essendon 15.17 (107) video stream
R16 Essendon 11.4 (70) Western Bulldogs 15.13 (103) video stream
R17 Essendon 14.12 (96) Richmond 15.11 (101) video stream
R18 West Coast 14.11 (95) Essendon 10.8 (68) unavailable
R19 Essendon 13.9 (87) Brisbane Lions 12.15 (87) unavailable
R20 Essendon 16.14 (110) St Kilda 16.12 (108) video stream
R21 Fremantle 21.15 (141) Essendon 13.9 (87) video stream
R22 Hawthorn 14.15 (99) Essendon 16.20 (116) video stream
EF1 Adelaide 26.10 (166) Essendon 10.10 (70) video stream
(*) Low quality


2008


R01 North Melbourne 9.13 (67) Essendon 19.8 (122) video stream
R02 Geelong 22.18 (150) Essendon 6.15 (51) video stream
R03 Essendon 23.12 (150) Carlton 21.8 (134) video stream
R04 Essendon 14.14 (98) Western Bulldogs 19.14 (128) video stream
R05 St Kilda 18.15 (123) Essendon 14.3 (87) video stream
R06 Collingwood 23.16 (154) Essendon 12.9 (81) video stream
R07 Essendon 15.4 (94) Port Adelaide 24.14 (158) video stream
R08 Sydney 21.17 (143) Essendon 7.10 (52) video stream
R09 Essendon 10.12 (72) Richmond 16.14 (110) video stream
R10 Adelaide 9.20 (74) Essendon 10.9 (69) video stream
R11 Essendon 12.7 (79) Hawthorn 19.16 (130) video stream
R12 Essendon 16.17 (113) West Coast 13.13 (91) video stream
R13 Carlton 15.11 (101) Essendon 20.16 (136) video stream
R14 Fremantle 13.13 (91) Essendon 14.11 (95) video stream
R15 Essendon 24.11 (155) Brisbane Lions 18.10 (118) video stream
R16 Richmond 16.12 (108) Essendon 15.14 (104) video stream
R17 Essendon 19.14 (128) Collingwood 11.14 (80) video stream
R18 Essendon 19.10 (124) Melbourne 17.6 (108) video stream
R19 West Coast 17.11 (113) Essendon 16.7 (103) video stream
R20 Essendon 10.13 (73) Adelaide 19.15 (129) video stream
R21 Western Bulldogs 23.13 (151) Essendon 15.15 (105) video stream
R22 Essendon 5.9 (39) St Kilda 21.21 (147) video stream


The final post in this series, when I write it,  will be about how to record the videos from 2007. This will be getting very much into the land of diminishing returns since their quality is so low, but never mind, completeness is a virtue all in itself.









Getting the AFL Videos from 2010 and 2011

All games from 2010 and 2011 are stored on the BigPond servers as simple mp4 files and would be easy to download if anybody had a link to them. The AFL website used to make this a simple problem, but the world is full of spoilsports and nowadays all the file links are hidden behind a wall of javascript. The BigPond AFL videos website is no friendlier. 


But it is not hard to discover a video link if you use FireFox.


Using FireFox and Download Helper


You can install DownloadHelper from the FireFox Add-ons manager or else by going directly to http://www.downloadhelper.net/. Both FireFox and DownloadHelper are (of course) free.
Then go to BigPond AFL Videos and use the fancy navigator to select ON DEMANDEssendon TV (OK, some other club if that is what you really want) and Matches. Pick a match and it will start to play in an embedded Flash machine. Do not omit to set the quality as high as possible (there is a control for that under the video, on the right).


Now you will notice that the the DownloadHelper icon at the top of your browser has started to animate, and if you click the little down arrow next to the icon you will see the name of the file you are playing.


Wave your mouse over this name and one of your options is to download the file. Another option is to copy the file's URL, and you can do this if you prefer to download later or if (like me) you plan to share the link with somebody else. 


That's about it. As far as I know, any download from a BigPond server is unmetered for anybody who uses BigPond for internet access. I have never bothered to confirm this because I am signed up with iinet and it comes of my quota in any case. Perhaps somebody might comment. 


Patterns.


After you have discovered a few of these link addresses you start to see that they follow a predictable pattern. This is because the Telstra engineers use scripts to create the video files in the first place, and it has the benefit of making it easy for people like myself to use scripts to download them. It also means that you can predict the urls for any game from a few simple rules.


For most of 2010 and all of 2011 the pattern has been like this:
Q1 http://bptvpd.ngcdn.telstra.com/pd_afl0/OnDemand/2011/ON/iVideo/Premiership/RDnn/AFL11_rdnn_hhh_vs_aaa_1st_qr_full_2M.mp4
Q2 http://bptvpd.ngcdn.telstra.com/pd_afl0/OnDemand/2011/ON/iVideo/Premiership/RDnn/AFL11_rdnn_hhh_vs_aaa_2nd_qr_full_2M.mp4
Q3 http://bptvpd.ngcdn.telstra.com/pd_afl0/OnDemand/2011/ON/iVideo/Premiership/RDnn/AFL11_rdnn_hhh_vs_aaa_3rd_qr_full_2M.mp4
Q4 http://bptvpd.ngcdn.telstra.com/pd_afl0/OnDemand/2011/ON/iVideo/Premiership/RDnn/AFL11_rdnn_hhh_vs_aaa_4th_qr_full_2M.mp4


To make real links, replace nn (in both places) with the round number, hhh with the home team name and aaa with the away team name. Round numbers are 01, 02, 03 and so on. Team names are adel, bl, carl, coll, ess, fre, geel, gcfc, haw, melb, nmfc, port, rich, stk, syd, wce or wb.
The last half of 2010 used the same pattern except that 2011 used to be 2010 and AFL11 was  AFL10. You can see the pattern working in the tables underneath.


For Bombers Fans


Friendly sandbelt has a list of all Essendon matches from 2010 and 2011 (corrected 9/9/2011). If you want anything, just right-click a link:


2011



R01 ess v wb Q1 Q2 Q3 Q4
R02 syd v ess Q1 Q2 Q3 Q4
R03 stk v ess Q1 Q2 Q3 Q4
R04 carl v ess Q1 Q2 Q3 Q4
R05 ess v coll Q1 Q2 Q3 Q4
R06 ess v gcfc Q1 Q2 Q3 Q4
R07 ess v wce Q1 Q2 Q3 Q4
R08 bl v ess Q1 Q2 Q3 Q4
R09 rich v ess Q1 Q2 Q3 Q4
R11 ess v melb Q1 Q2 Q3 Q4
R12 fre v ess Q1 Q2 Q3 Q4
R13 ess v nmfc Q1 Q2 Q3 Q4


R14 haw v ess Q1 Q2 Q3 Q4
R15 ess v geel Q1 Q2 Q3 Q4
R16 ess v rich Q1 Q2 Q3 Q4
R17 adel v ess Q1 Q2 Q3 Q4
R18 ess v carl Q1 Q2 Q3 Q4
R19 coll v ess Q1 Q2 Q3 Q4
R20 ess v syd Q1 Q2 Q3 Q4
R21 wb v ess Q1 Q2 Q3 Q4
R22 wce v ess Q1 Q2 Q3 Q4
R23 ess v port Q1 Q2 Q3 Q4
EF1 carl v ess Q1 Q2 Q3 Q4
Mind you. I'm guessing about the final, it hasn't happened yet as I write this.


2010
R01 geel v ess All
R02 ess v fre All
R03 carl v ess All
R04 wce v ess All
R05 coll v ess All
R06 ess v haw All
R07 ess v port All
R08 stk v ess All
R09 ess v rich Q1 Q2 Q3 Q4
R10 ess v wb Q1 Q2 Q3 Q4
R11 syd v ess Q1 Q2 Q3 Q4
R12 ess v geel Q1 Q2 Q3 Q4
R13 haw v ess Q1 Q2 Q3 Q4
R14 adel v ess Q1 Q2 Q3 Q4
R15 melb v ess Q1 Q2 Q3 Q4
R16 ess v wce Q1 Q2 Q3 Q4
R17 nmfc v ess Q1 Q2 Q3 Q4
R18 ess v stk Q1 Q2 Q3 Q4
R19 ess v carl Q1 Q2 Q3 Q4
R20 ess v coll Q1 Q2 Q3 Q4
R21 ess v bl Q1 Q2 Q3 Q4
R22 wb v ess Q1 Q2 Q3 Q4

The next post will go into what you have to do to download the media streams from 2009 and 2008. I will eventually even cover 2007, because even though the quality is lousy there is some good historical stuff there, and I admit I got a kick out of seeing Mark Johnson with long white hair.