Candid’s brain

On our super-fast 446 kbit/s internet connection, which seems above all to be configured wrong by our ISP, downloading a file with full speed results in ping times around 5000 ms. This is of course unacceptable, as every single web page takes several minutes to load then. The problem can be fixed by manually limiting the download rate to 20 KB/s or so when you plan to download a large file, but this is not always easily possible, for example when watching a YouTube video. An additional problem is that two clients that both limit their download rate to 20 KB/s can hang up the internet again when they download simultaneously. Furthermore it would be great if normal surfing, small downloads, would not be limited in bandwidth for keeping the speed at the possible maximum.

The best solution I could imagine was a transparent proxy that limited the download rate of large downloads automatically, letting small downloads pass through at full speed. Those large downloads should share a specified bandwidth, so multiple downloads would not make the internet slower than a single one.

So I wrote a simple TCP proxy that does exactly this. It listens on a port for incoming TCP connections. For each connection, it opens another TCP connection to a specified port on a specified server (so it is best used in combination with a HTTP proxy or whatever procotol you want to use it for) and redirects the traffic between these two connections. As soon as a configured download traffic limit is reached, the transfer is switched to “low-priority mode”, where it has to share a specified bandwidth with all other downloads running on low priority.

On my local server there already is a HTTP proxy running as cache. If I just used my bandwidth-limiter TCP proxy now and connected that to my HTTP proxy, the bandwidth limit would also be applied for transfers that are already cached by the HTTP proxy and thus make the cache useless. So I have to start an additional transparent HTTP proxy server that does not do anything except for redirecting the incoming connections from my bandwidth limiter to the internet. Sounds a bit like too much work and too many redirects, but it is the easiest solution I have found, as Squid does not seem to support the kind of bandwidth limit I need. So my configuration looks like this: My HTTP proxy cache (in my case Squid) is configured to use another proxy, which I set to localhost and the port my bandwidth limiter listens on. The bandwidth limiter connects to another HTTP proxy running on localhost, which is configured to do nothing but forward the request to the Web.

My bandwidth limiter (called “bwproxy”) is written in Java, you can download it from https://github.com/cdauth/bwproxy. It is available under the terms of GPL-3, feel free to send modifications or bug reports to me.

I’d been looking for a proper way to rip last.fm streams for a very long time. All rippers I could find (like TheLastRipper) are only able to rip lastfm:// URLs, which might be useful to get a lot of songs of a specified tag and then pick out the good ones, but if you are looking for a specific song that is playable on Last.fm in full length, a lastfm:// URL does not exist, the song is only playable via the Flash plug-in.

My solution of the problem is a combination of the Firefox plug-ins BetterCache (unstable version 1.24 works on Linux) and CacheViewer. BetterCache allows you to overwrite the HTTP caching instruction headers, thus to force Firefox to cache files that it should not following the HTTP standard (such as Last.fm MP3 files or YouTube Flash videos). CacheViewer provides you with a GUI for the about:cache list. By searching the cached files for the MIME type audio/mpeg and the host name ^s\d+\.last\.fm$ (s*.last.fm, * standing for a number), you can easily copy the cached MP3 files to your hard drive.

Be careful with the BetterCache add-on. In the default configuration, it modifies Firefox to use the cache for all files not transfered over SSL, which is in most cases not the behaviour you desire. If you want to enable BetterCache only for specified MIME types, remove the wildcard entry from the “Always-cache list” and add it to the “Never-cache list”. Also make sure that “Never-cache list works as ignore list” is enabled.

At the moment, BetterCache does not seem to support wildcards in URLs. Personally, I added audio/mpeg for all URLs to the “Always-cache list”. As soon as wildcards are supported, you may enable it only for *.last.fm.