Archive for the ‘Code’ Category

Compiling zlib.lib on Windows

Thursday, August 26th, 2010

zlib is the standard for lossless data compression. The DEFLATE compression algorithm is the basis for just about every lossless compression format out there, including “zip” and “gzip”, which is itself part of zlib.

There are two ways that it can be used from C/C++ projects in Windows.

Firstly, it can be used by dynamic linking (dll). This means using zdll.lib and shipping the appropriate version of zlib1.dll with your project. This is not a problem, as Windows versions of both of these files are provided.

The second way is to use static linking. That is, having all of the code in one .lib file and compiling it into your exe so that you do not have to distribute zlib1.dll. This means compiling zlib.lib.

In version 1.2.4 of zlib, a “projects” directory was provided, with a Microsoft Visual C++ 6 project. However, it seems that version 1.2.5 has not included this project. This means that the best solution is to go and get the 1.2.4 source and compile it yourself. However, the zlib project seems to be kept inside the libpng project on sourceforge.net, so it is not immediately obvious where to find older versions of the zlib source code.

zlib 1.2.4 source (zip)

Extract the zip, open projects\visualc6\zlib.dsp in Visual Studio (I used 2005) and compile “LIB Release” (and optionally “LIB Debug”)

Copy zlib.h and zconf.h from “include” to your Visual Studio “include” directory, and zlib.lib (and zlibd.lib if you made it) to your Visual Studio “lib” directory.

On 64 bit Windows, with Visual Studio 2005, this is “C:\Program Files (x86)\Microsoft Visual Studio 8\VC\” so adjust for your version of Visual Studio.

You now just need to add “zlib.lib” to your “Linker -> Input -> Additional Dependencies” line in your C++ project configuration to use it (and optionally zlibd.lib for the debug version).

Remote recursive sha1sum with php

Wednesday, June 2nd, 2010

To calculate the SHA-1 sums, display them and make them available for download in a sums.gz file:

<?php echo(`find ./some_directory/ -type f | grep -v sums.gz | xargs sha1sum | gzip -c | tee sums.gz | zcat`); ?>

To check the sums:

<?php echo(`zcat sums.gz | sha1sum -c -`); ?>

Get external IP address with Python

Wednesday, May 26th, 2010

Here’s a quick snippet of Python code (tested in 3.0) to quickly look up your external IP address over HTTP and display it:

import urllib.request
print(str(urllib.request.urlopen("http://www.whatismyip.com/automation/n09230945.asp").read(), "utf8"))

wp-syntax to look like Visual Studio

Tuesday, March 23rd, 2010

wp-syntax is a nice plugin for WordPress using GeSHi to produce syntax highlighted blocks of code.

The default colours aren’t very nice though, and there is no way to easily change them. wp-syntax-colorizer (horrible name) makes it easier to set the colours, but defaults to even worse colours.

Most of us want readable colours that we are used to from IDEs, and most of us will be using either Visual Studio or Eclipse. I mainly use the former. To get it to use the visual studio colours I edit wp-syntax-colorizer as follows:

function my_custom_geshi_styles(&$geshi)
{
  $overall = "black";
  $keyword = "blue";
  $literal = "maroon";
  $comment = "green";
 
  $geshi->set_overall_style("color: $overall;", true);
 
  $geshi->set_keyword_group_style(1, "color: $keyword;", true);
  $geshi->set_keyword_group_style(2, "color: $keyword;", true);
  $geshi->set_keyword_group_style(3, "color: $keyword;", true);
  $geshi->set_keyword_group_style(4, "color: $keyword;", true);
 
  $geshi->set_symbols_style("color: $overall;", true);
  $geshi->set_methods_style(1, "color: $overall;", true);
  $geshi->set_regexps_style(1, "color: $overall;", true);
 
  $geshi->set_strings_style("color: $literal;", true);
  $geshi->set_numbers_style("color: $literal;", true);
 
  $geshi->set_comments_style(1, "color: $comment;", true);
  $geshi->set_comments_style('MULTI',"color: $comment;", true);
}

Remove Size bbcode from phpbb

Thursday, March 4th, 2010

One of the most annoying things about phpBB is the ability for people to randomly use [size="200"]Huge[/size] BB code, with no easy way to remove it and every time you update between versions it comes back.

To disable it:
in /include/bbcode.php remove the whole

case 5:
...
break;

section.

To remove it from the posting page:
in /styles/prosilver/template/posting_buttons.html remove the whole

<select>
...
</select>

section.

After that, delete your “cache” directory so that it uses the new copy.

Remember that you have to repeat this process after every update to phpBB3.

Moving comments in WordPress

Sunday, February 7th, 2010

In MySQL, first move the comment:

UPDATE wp_comments SET comment_post_ID = P_TO_ID WHERE comment_ID = C_ID;

P_TO_ID is the ID of the post you are moving it to. C_ID is the ID of the comment you are moving. You can find both of these IDs easily from the dashboard.

Then correct the comment counts for the posts:

UPDATE wp_posts SET comment_count = (SELECT count(*) FROM wp_comments WHERE comment_post_id = id AND comment_approved = 1)
WHERE comment_count != (SELECT count(*) FROM wp_comments WHERE comment_post_id = id AND comment_approved = 1);

The second line in this query is not strictly necessary but it lets you know how many posts were affected.

Convert any audio/video files to mp3 for free

Tuesday, November 24th, 2009

It seems that there is some confusion about audio conversion tools and some people even pay for them.

The truth is that most encoders, even ones you pay for, use ffmpeg internally to do all of the actual work and ffmpeg is free and open source.

You can even get it pre-compiled for Windows, though the version isn’t great.

The only problem with using ffmpeg directly is that it has a command line interface and most novices find it hard to use.

Here’s a batch file for use on Windows which will convert anything you drop onto the file into a 320kbps mp3:

ffmpeg -i %1 -ab 320k -y %1.mp3
@if errorlevel 1 @pause

To create this file, open notepad. Copy and paste the above 2 lines in. Save it as “convert.bat” or something similar (the .bat is important) and select “All files” from the drop down list.

Put the bat file in the same directory as ffmpeg.exe and simply drag anything you want converted on top of the bat file.

You can use almost anything. wav, m4a, m4v, mov, mpg, avi, etc. If you use a file with both audio and video, such as a movie, it will just extract the audio and save that as an mp3 file.

ffmpeg target file sizes (again)

Friday, July 17th, 2009

This script is now available as an ongoing project that can be downloaded. Please see the Video Encoding Project.


As a bit of a revision to a previous entry, I have converted the combination bash/python scripts I gave earlier into a single large python script.

This can now be used without editing any files to set sizes. An example of use is:
$ ./enc.py -s 700 -a 192 -o '-deinterlace' da_*.VOB
All of these flags are optional except the file list, where at least 1 file must be specified.

enc.tar.gz:

#!/usr/bin/python
 
from getopt import getopt, GetoptError
from sys import argv, exit
from subprocess import Popen, PIPE
from re import search
 
def secs(h, m, s):
  return (((h*60)+m)*60)+s
 
def calc_video_bitrate(target_bytes, ab, time):
  audio_bytes = (ab * 1000 / 8) * time
  return (target_bytes - audio_bytes) / time * 8
 
def get_duration(file):
  ffmpeg = Popen(("ffmpeg", "-i", file), stderr=PIPE)
  match = search("(\\d+):(\\d+):(\\d+)\\.\\d+", ffmpeg.communicate()[1])
  ffmpeg.stderr.close()
 
  if match == None: return 0
  dur = match.groups()
  return secs(int(dur[0]), int(dur[1]), int(dur[2]))
 
def encode(source, dest, opts, vbr, abr=128):
  command1 = "ffmpeg -i %s -pass 1 -an -vcodec libx264 -vpre fastfirstpass -b %d -bt %d -threads 0 %s -y pass1.mp4" % (source, vbr, vbr, opts)
  command2 = "ffmpeg -i %s -pass 2 -acodec libfaac -ac 2 -ab %dk -vcodec libx264 -vpre hq -b %d -bt %d -threads 0 %s -y %s" % (source, abr, vbr, vbr, opts, dest)
 
  pass1 = Popen(command1.split())
  pass1.wait()
 
  pass2 = Popen(command2.split())
  pass2.wait()
 
 
target_size = 350 #MiB
ab = 128 #kbps
extra_options = ""
 
# Parse args
try:
  opts, args = getopt(argv[1:], "s:a:o:", ["size=", "ab=", "opts="])
except GetoptError, err:
  print str(err)
  exit(2)
 
for o, a in opts:
  if o in ("-s", "--size"):
    target_size = int(a) * 1024 * 1024
  elif o in ("-a", "--ab"):
    ab = int(a)
  elif o in ("-o", "--opts"):
    extra_options = a
 
if len(args) < 1:
  print "Usage: " + argv[0] + \
    " [-s <size (MiB)>] [-a <audio bit rate (kbps)>] [-o <additional options>] <file list>"
  exit(2)
 
# Loop through files
for file in args:
  duration = get_duration(file)
  if duration <= 0:
    print "Unable to get length of", file
    continue
 
  bitrate = calc_video_bitrate(target_size, ab, duration)
 
  encode(file, file + ".mp4", extra_options, bitrate, ab)

Network speed test

Tuesday, July 14th, 2009

I currently have the need to test the connection speed between two remote machines, so that I know when it is safe to transfer a large file from one to the other without significantly affecting service by causing long periods of downtime. I couldn’t find any software that did this easily, ran as a single executable (no installation) and was free, so I made one.

This currently only tests speed in one direction (upload from the client to the server) because I have symmetric connections on both machines, so either way should be exactly the same, and so that I can just write a client program and use netcat as a server to accept the packets and dump them to /dev/null. It is simple enough to convert this client into a server so that netcat becomes a client or so that they can be used together. Using netcat as a client probably requires quite a large pre-generated random file and may introduce a bottleneck of the hard drive read speed.

This could also be used in the home for something like testing the effect of wifi strength.

The client takes the command line arguments, connects to the server, generates some random data and sends that data repeatedly until the timer expires. It then prints out the results.

The command to use netcat to listen and ignore any data it receives is nc -lp 9000 > /dev/null though the -p isn’t always needed (it was needed on my Debian box but not my ESX 4 box).

An example of use of this program as a client is:

C:\>SpeedTestClient.exe 192.168.1.157 9000
Uploaded 1181614080 bytes in 10.015 seconds.
943875.45 kbps (943.88 mbps)
115219.17 kB/s (112.52 MB/s)

The results are given in kilobits per second and megabits per second (standard measurements for network speed), kilobytes per second and megabytes per second (more useful for knowing how long a file will take to transfer). In this example, a gigabit ethernet connection is getting roughly 944 mbps.

A few seconds later, VLC 1.0 is opened and paused, so nothing is playing and the test is run again. The results are quite interesting:

Uploaded 225083392 bytes in 10.015 seconds.
179797.02 kbps (179.80 mbps)
 21947.88 kB/s ( 21.43 MB/s)

It says it is using next to no system resources when paused but it slows data transfer dramatically and this is with 8GiB of RAM, an otherwise idle CPU and no hard drives being used. When hard drives enter the equation this drops down to about 8MB/s (while VLC is still paused – I have particularly fast hard drives so I still get about 110MB/s when hard drives are used and VLC is turned off).

The code listing is as follows:

using System;
using System.IO;
using System.Net;
using System.Net.Sockets;
using System.Security.Cryptography;
 
namespace SpeedTest
{
    class SpeedTestClient
    {
        static int Main(string[] args)
        {
            if (args.Length < 2)
            {
                Console.WriteLine("Usage: SpeedTestClient <ip> <port> [<seconds>]");
                return 1;
            }
 
            try
            {
                ulong bytes = 0;
                int startTime = Environment.TickCount;
                bool started = false;
                int timeout = 10;
 
                // Get the optional number of seconds argument
                if (args.Length > 2) timeout = int.Parse(args[2]);
 
                // Connect to the server via TCP
                TcpClient client = new TcpClient();
                client.Connect(IPAddress.Parse(args[0]), int.Parse(args[1]));
                NetworkStream stream = client.GetStream();
 
                // Generate some random data. Buffers too small will bottleneck.
                RandomNumberGenerator rng = RandomNumberGenerator.Create();
                byte[] buffer = new byte[32768];
                rng.GetBytes(buffer);
 
                while (true)
                {
                    // Write the data
                    stream.Write(buffer, 0, buffer.Length);
 
                    // Start the timer only after some data has been sent
                    if (!started)
                    {
                        startTime = Environment.TickCount;
                        started = true;
                    }
 
                    // Add to the bytes counter
                    bytes += (ulong)buffer.LongLength;
 
                    if (Environment.TickCount - startTime >= timeout * 1000) break;
                }
 
                PrintFinalStats(startTime, Environment.TickCount, bytes);
 
                try { client.Close(); }
                catch (IOException) { /* Do nothing */ }
            }
            catch (Exception ex)
            {
                Console.Error.WriteLine(ex);
                return 1;
            }
 
            return 0;
        }
 
        static void PrintFinalStats(int start, int end, ulong bytes)
        {
            double seconds = (end - start) / 1000.0;
            ulong bits = bytes * 8;
 
            double bitsPerSecond = bits / seconds;
            double bytesPerSecond = bytes / seconds;
 
            string kbps = string.Format("{0:0.00}", bitsPerSecond / 1000);
            string mbps = string.Format("{0:0.00}", bitsPerSecond / 1000 / 1000);
            string kibs = string.Format("{0:0.00}", bytesPerSecond / 1024);
            string mibs = string.Format("{0:0.00}", bytesPerSecond / 1024 / 1024);
 
            int kPad = Math.Max(kbps.Length, kibs.Length);
            int mPad = Math.Max(mbps.Length, mibs.Length);
 
            Console.WriteLine("Uploaded {0} bytes in {1} seconds.", bytes, seconds);
            Console.WriteLine("{0} kbps ({1} mbps)", kbps.PadLeft(kPad), mbps.PadLeft(mPad));
            Console.WriteLine("{0} kB/s ({1} MB/s)", kibs.PadLeft(kPad), mibs.PadLeft(mPad));
        }
    }
}

A compiled .net executable is available here.

P.S. always remember to turn off VLC before transferring files over the network.

How to set target file sizes in ffmpeg

Tuesday, June 16th, 2009

This script is now available as an ongoing project that can be downloaded. Please see the Video Encoding Project. This post describes the methods and mathematics behind the script.


After much searching, I found almost nothing regarding how to target a specific file size with ffmpeg, such as 700MiB to fit on a CD. Almost everything I found suggested to use a negative bitrate for mencoder, such as -700000. However, this did not work for me at all (even by copying and pasting verbatim). Because of this, I set about doing it manually with some mathematics.

What I am going to do here assumes a single audio stream, a single video stream and nothing else in the output file at all (i.e. no subtitles, no alternate languages, no director’s commentaries, etc).

When encoding media, audio uses a constant bit rate. This means that you can calculate how much space it will take if you know which bit rate you are using and how long the stream is in seconds. This also means that to know how big the video stream should be, we can calculate how big the audio stream will be and subtract it from the target file size. For example, if the target file size is 350MiB and the audio will take up 30MiB, we know the video should aim to be 320MiB (ignoring overheads in the multiplexing, container headers, etc). We can use the same calculations in reverse to work out an average or constant bit rate from a file size and a length in seconds.

In this example, I will be using Dad’s Army Series 6 Episode 1 – The Deadly Attachment. I have already ripped this from the DVD with SlySoft AnyDVD and PgcDemux on Windows. Automating PgcDemux rips is worth a whole entry in itself. Note that I have bought this legally and I do not intend to sell or share it. I just want a copy that I can store on my network and play from any computer. Storing 14 DVD images on hard drives takes quite a lot of space, which is where encoding comes in handy. In fact, some of my box sets contain over 60 DVDs (roughly 270GiB).

———-
To install ffmpeg on Debian, it is best to get it from the Debian Multimedia repository. Add

deb http://www.debian-multimedia.org lenny main
deb-src http://www.debian-multimedia.org lenny main

to your /etc/apt/sources.list and then run

# apt-get update
# apt-get install debian-multimedia-keyring
# apt-get update
# apt-get install ffmpeg

———-

The first thing we need to do to put this into practice is to get the length of the stream. If you supply the file to ffmpeg without telling it to do anything, it will give you an error. However, this error already contains all of the information we need:

$ ffmpeg -i da_6_01.VOB
FFmpeg version SVN-r13582, Copyright (c) 2000-2008 Fabrice Bellard, et al.
  configuration: --prefix=/usr ...
  built on May  3 2009 12:07:18, gcc: 4.3.2
Input #0, mpeg, from 'da_6_01.VOB':
  Duration: 00:29:23.90, start: 0.287267, bitrate: 5987 kb/s
    Stream #0.0[0x1e0]: Video: mpeg2video, yuv420p, 720x576 [PAR 16:15 DAR 4:3], 9800 kb/s, 25.00 tb(r)
    Stream #0.1[0x80]: Audio: ac3, 48000 Hz, stereo, 192 kb/s
Must supply at least one output file

We can see that the duration is 00:29:23.90.

As this is an error, we need to redirect stderr to stdout. This can be done with 2>&1. We then need the line that contains the “Duration”, which we can get with grep. We can cut this line up to take the time out.

$ ffmpeg -i da_6_01.VOB 2>&1 | grep Duration | cut -d ' ' -f 4 | cut -d '.' -f 1
00:29:23

This can be converted into seconds with the formula s' = (h * 60 + m) * 60 + s.

I used a python script to do this and to perform the necessary calculations mentioned earlier. Note that most of this python script is just for the convenience of getopts and that, if needed, it can be cut down significantly at the cost of flexibility. If you always want the same output size and audio bitrate, this script can be simplified to just a couple of lines.

calc_bitrate.py:

#!/usr/bin/python
 
from getopt import getopt, GetoptError
from sys import argv, exit
 
def secs(h, m, s):
  return (h * 60 + m) * 60 + s
 
def calc_video_bitrate(target_size, abr, time):
  target_bytes = target_size * 1024 * 1024
  audio_bytes = ((abr * 1000) / 8) * time
  return (target_bytes - audio_bytes) / time * 8
 
try:
  opts, args = getopt(argv[1:], "s:a:t:", ["size=", "abr=", "time="])
except GetoptError, err:
  print str(err)
  exit(2)
 
target_size = 350 #MiB
abr = 128 #kbps
time = 0
 
for o, a in opts:
  if o in ("-s", "--size"):
    target_size = int(a)
  elif o in ("-a", "--abr"):
    abr = int(a)
 
if len(args) < 1:
  print "Usage: " + argv[0] + " [-s <size (M)>] [-a <audio bit rate (k)>] [[hh:]mm:]ss"
  exit(2)
 
time_part = args[0].split(":")
if len(time_part) < 2: time_part = [0] + time_part
if len(time_part) < 3: time_part = [0] + time_part
time = secs(int(time_part[0]), int(time_part[1]), int(time_part[2]))
 
print "%d" % calc_video_bitrate(target_size, abr, time)

A typical usage for this would be ./calc_python.py -s 350 -a 128 29:23.

We can then use a combination of these inside a shell script easily. I’m using bash syntax just to nest the commands into a single line but for compatibility with other shells, backticks (`) can be used and this is what I normally prefer. This script loops through all of the .VOB files in the directory and gives their target video bitrate for a 350MiB output and 128kbps audio.

#!/bin/bash
 
for vob in *.VOB
do
  BR=$(./calc_bitrate.py -s 350 -a 128 $(ffmpeg -i $vob 2>&1 | grep Duration | cut -d ' ' -f 4 | cut -d '.' -f 1))
  echo $vob $BR
done

I use this in 2-pass VBR mode with the libx264 video encoder and libfaac audio encoder in an mp4 container just by replacing the “echo” line of the above script with the following 2 lines:

ffmpeg -i $vob -an -pass 1 -vcodec libx264 -vpre fastfirstpass -b $BR -bt $BR -deinterlace -threads 0 -y pass1.mp4
ffmpeg -i $vob -acodec libfaac -ab 128k -ac 2 -pass 2 -vcodec libx264 -vpre hq -b $BR -bt $BR -deinterlace -threads 0 -y `echo $vob | sed 's/VOB/mp4/'`

I have 2 versions of this script, one with “-deinterlace” and one without. PAL video (576i – used in Europe), such as Dad’s Army needs deinterlacing. NTSC video (480 – used in North America) usually does not.

For ripping DVDs to H.264 and maintaining the quality, I wouldn’t use a bitrate lower than 1000k or an audio bitrate lower than 128k stereo. This means that for getting media onto a CD, you probably want no smaller than 350MiB for 30 to 40 minutes or 700MiB for 1 hour or more, depending on your resolution. Also note that I only intend to play these in computers and fitting them onto CDs is just a convenience – standalone DVD players will not be able to play H.264 or AAC (Blu-ray and HDDVD players may be able to – mp4 containers and reading CDs would be more of an issue).

Update:

I have successfully tested this in a LG BD370 Blu-ray player.

I have not tried actually burning these onto CDs yet so if 2 won’t fit on a 700MiB CD, even with overburn (because of overheads that were ignored, as stated earlier) then the simplest solution is to just target a smaller file size with something like “-s 348″.

The result

Dad’s Army episodes are not all of the same length. Some are several minutes longer than others. The following shows how no matter the size of the input file, they all produce the same size output file:

$ ll da_5_*.VOB
-r--r--r-- 1 mythtv mythtv 1364660224 2009-06-04 00:01 da_5_01.VOB
-r--r--r-- 1 mythtv mythtv 1360699392 2009-06-04 00:01 da_5_02.VOB
-r--r--r-- 1 mythtv mythtv  956084224 2009-06-04 00:02 da_5_03.VOB
-r--r--r-- 1 mythtv mythtv 1355888640 2009-06-04 00:02 da_5_04.VOB
-r--r--r-- 1 mythtv mythtv 1114351616 2009-06-03 23:59 da_5_05.VOB
-r--r--r-- 1 mythtv mythtv 1034289152 2009-06-04 00:00 da_5_06.VOB
-r--r--r-- 1 mythtv mythtv 1045202944 2009-06-04 00:00 da_5_07.VOB
-r--r--r-- 1 mythtv mythtv 1345318912 2009-06-04 00:00 da_5_08.VOB
-r--r--r-- 1 mythtv mythtv 1391529984 2009-06-04 00:02 da_5_09.VOB
-r--r--r-- 1 mythtv mythtv 1342748672 2009-06-04 00:03 da_5_10.VOB
-r--r--r-- 1 mythtv mythtv 1158017024 2009-06-04 00:03 da_5_11.VOB
-r--r--r-- 1 mythtv mythtv 1157097472 2009-06-04 00:03 da_5_12.VOB
-r--r--r-- 1 mythtv mythtv 1180098560 2009-06-04 00:04 da_5_13.VOB
$ ll -h da_5_*.mp4
-rw-r--r-- 1 mythtv mythtv 352M 2009-06-17 06:18 da_5_01.mp4
-rw-r--r-- 1 mythtv mythtv 352M 2009-06-17 06:55 da_5_02.mp4
-rw-r--r-- 1 mythtv mythtv 352M 2009-06-17 07:30 da_5_03.mp4
-rw-r--r-- 1 mythtv mythtv 352M 2009-06-17 08:08 da_5_04.mp4
-rw-r--r-- 1 mythtv mythtv 352M 2009-06-17 08:44 da_5_05.mp4
-rw-r--r-- 1 mythtv mythtv 352M 2009-06-17 09:21 da_5_06.mp4
-rw-r--r-- 1 mythtv mythtv 352M 2009-06-17 09:58 da_5_07.mp4
-rw-r--r-- 1 mythtv mythtv 352M 2009-06-17 10:36 da_5_08.mp4
-rw-r--r-- 1 mythtv mythtv 352M 2009-06-17 11:15 da_5_09.mp4
-rw-r--r-- 1 mythtv mythtv 352M 2009-06-17 11:54 da_5_10.mp4
-rw-r--r-- 1 mythtv mythtv 352M 2009-06-17 12:29 da_5_11.mp4
-rw-r--r-- 1 mythtv mythtv 352M 2009-06-17 13:05 da_5_12.mp4
-rw-r--r-- 1 mythtv mythtv 352M 2009-06-17 13:41 da_5_13.mp4

Update:

A pure python replacement script is available here that simplifies use.