How slow are (Ruby) Exceptions?

— 0 Comments

If you are used to benchmark your Ruby scripts or if you ever had to improve the performance of some strategic tasks, then this post won't tell you nothing new because you should already know that Exceptions are slow. And this is not really a Ruby problem: .NET Exceptions are slow, JAVA Exceptions are slow just because the begin/raise/rescue (or try/throw/catch) architecture is slow by nature.

But how slow are Ruby Exceptions?

The answer to this question really depends on how complex is your code. Here I just want to show you a very simple example, extracted from a really strategic RoboDomain DNS sorting algorithm.

require 'ipaddr'
require 'benchmark'
class IPAddr
def self.valid_ipv4?(addr)
if /\A(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})\Z/ =~ addr
return $~.captures.all? {|i| i.to_i < 256}
end
false
end
def self.valid_ipv6?(addr)
# IPv6 (normal)
return true if /\A[\dA-Fa-f]{1,4}(:[\dA-Fa-f]{1,4})*\Z/ =~ addr
return true if /\A[\dA-Fa-f]{1,4}(:[\dA-Fa-f]{1,4})*::([\dA-Fa-f]{1,4}(:[\dA-Fa-f]{1,4})*)?\Z/ =~ addr
return true if /\A::([\dA-Fa-f]{1,4}(:[\dA-Fa-f]{1,4})*)?\Z/ =~ addr
# IPv6 (IPv4 compat)
return true if /\A[\dA-Fa-f]{1,4}(:[\dA-Fa-f]{1,4})*:/ =~ addr && valid_ipv4?($')
return true if /\A[\dA-Fa-f]{1,4}(:[\dA-Fa-f]{1,4})*::([\dA-Fa-f]{1,4}(:[\dA-Fa-f]{1,4})*:)?/ =~ addr && valid_ipv4?($')
return true if /\A::([\dA-Fa-f]{1,4}(:[\dA-Fa-f]{1,4})*:)?/ =~ addr && valid_ipv4?($')
false
end
end
class Bim
attr_accessor :cls, :data
def initialize(cls, data)
self.cls = cls
self.data = data
end
def value
IPAddr.new(data).to_i
rescue ArgumentError
data
end
end
class Bum
attr_accessor :cls, :data
def initialize(cls, data)
self.cls = cls
self.data = data
end
def value
if cls == "A"
IPAddr.new(data).to_i
else
data
end
end
end
class Bam
attr_accessor :cls, :data
def initialize(cls, data)
self.cls = cls
self.data = data
end
def value
if IPAddr.valid_ipv4?(data) || IPAddr.valid_ipv6?(data)
IPAddr.new(data).to_i
else
data
end
end
end
TIMES = 50_000
Benchmark.bmbm do |x|
x.report("NS exception") do
TIMES.times { b = Bim.new("NS", "ns2.foo.com"); b.value }
end
x.report("NS if") do
TIMES.times { b = Bum.new("NS", "ns2.foo.com"); b.value }
end
x.report("NS regexp") do
TIMES.times { b = Bam.new("NS", "ns2.foo.com"); b.value }
end
x.report("A exception") do
TIMES.times { b = Bim.new("A", "192.168.1.1"); b.value }
end
x.report("A if") do
TIMES.times { b = Bum.new("A", "192.168.1.1"); b.value }
end
x.report("A regepx") do
TIMES.times { b = Bum.new("A", "192.168.1.1"); b.value }
end
end
__END__
# ruby 1.8.7 (2009-12-24 patchlevel 248) [i686-darwin10.2.0]
$ ruby if_vs_exception.rb
Rehearsal ------------------------------------------------
NS exception 4.330000 5.750000 10.080000 ( 37.599575)
NS if 0.140000 0.010000 0.150000 ( 0.136280)
NS regexp 0.450000 0.000000 0.450000 ( 0.454040)
A exception 2.010000 0.020000 2.030000 ( 2.047711)
A if 2.060000 0.020000 2.080000 ( 2.074642)
A regepx 2.030000 0.020000 2.050000 ( 2.054930)
-------------------------------------- total: 16.840000sec
user system total real
NS exception 4.420000 5.810000 10.230000 ( 38.350608)
NS if 0.130000 0.000000 0.130000 ( 0.130863)
NS regexp 0.450000 0.000000 0.450000 ( 0.447975)
A exception 2.000000 0.020000 2.020000 ( 2.016231)
A if 2.020000 0.020000 2.040000 ( 2.043085)
A regepx 2.030000 0.020000 2.050000 ( 2.048402)
# ruby 1.9.1p376 (2009-12-07 revision 26041) [i386-darwin10.2.0]
Rehearsal ------------------------------------------------
NS exception 4.650000 5.800000 10.450000 ( 39.134858)
NS if 0.080000 0.000000 0.080000 ( 0.079427)
NS regexp 0.320000 0.000000 0.320000 ( 0.320389)
A exception 1.180000 0.010000 1.190000 ( 1.182892)
A if 1.200000 0.000000 1.200000 ( 1.209433)
A regepx 1.220000 0.000000 1.220000 ( 1.219345)
-------------------------------------- total: 14.460000sec
user system total real
NS exception 4.520000 5.720000 10.240000 ( 37.931455)
NS if 0.080000 0.000000 0.080000 ( 0.078286)
NS regexp 0.320000 0.000000 0.320000 ( 0.320988)
A exception 1.190000 0.000000 1.190000 ( 1.186198)
A if 1.210000 0.010000 1.220000 ( 1.220254)
A regepx 1.220000 0.000000 1.220000 ( 1.215634)
view raw if_vs_exception_vs_regexp.rb hosted with ❤ by GitHub

The code is fairly simple: the value for an A record is expected to be an IP Address while the value for a NS record is expected to be represented by a FQDN as string. The code should parse the data and return the normalized value depending on some conditions.

The first version of the algorithm is completely based on Exceptions. IPAddr raises an ArgumentError when the argument is not a valid IP Address. In this case, the script gracefully returns the value as string.

The second version checks against the record value (I agree with you that is quite empiric, but representative for the sake of this benchmark).

The third version is a less empiric alternative that uses some regular expressions to check whether the data looks like an IP before actually feeding the IPAddr class. Under the hood, the IPAddr class performs a really similar task, the only difference here is that in the latter case I'm not using Exceptions at all.

Results speak for themselves.

As pointed out by Curtis Summers in the comments,a huge amount of time is actually spent by IPAddr trying to resolve the hostname. For more benchmarks, also look at this test.

With Ruby 1.8.7, the algorithm (note, this is a super-simplified version of the original algorithm) is about 37 times slower when Exceptions are involved.

$ ruby if_vs_exception.rb
Rehearsal ------------------------------------------------
NS exception   4.330000   5.750000  10.080000 ( 37.599575)
NS if          0.140000   0.010000   0.150000 (  0.136280)
NS regexp      0.450000   0.000000   0.450000 (  0.454040)
A exception    2.010000   0.020000   2.030000 (  2.047711)
A if           2.060000   0.020000   2.080000 (  2.074642)
A regepx       2.030000   0.020000   2.050000 (  2.054930)
-------------------------------------- total: 16.840000sec

                   user     system      total        real
NS exception   4.420000   5.810000  10.230000 ( 38.350608)
NS if          0.130000   0.000000   0.130000 (  0.130863)
NS regexp      0.450000   0.000000   0.450000 (  0.447975)
A exception    2.000000   0.020000   2.020000 (  2.016231)
A if           2.020000   0.020000   2.040000 (  2.043085)
A regepx       2.030000   0.020000   2.050000 (  2.048402)

The same story with Ruby 1.9.1.

Rehearsal ------------------------------------------------
NS exception   4.650000   5.800000  10.450000 ( 39.134858)
NS if          0.080000   0.000000   0.080000 (  0.079427)
NS regexp      0.320000   0.000000   0.320000 (  0.320389)
A exception    1.180000   0.010000   1.190000 (  1.182892)
A if           1.200000   0.000000   1.200000 (  1.209433)
A regepx       1.220000   0.000000   1.220000 (  1.219345)
-------------------------------------- total: 14.460000sec

                   user     system      total        real
NS exception   4.520000   5.720000  10.240000 ( 37.931455)
NS if          0.080000   0.000000   0.080000 (  0.078286)
NS regexp      0.320000   0.000000   0.320000 (  0.320988)
A exception    1.190000   0.000000   1.190000 (  1.186198)
A if           1.210000   0.010000   1.220000 (  1.220254)
A regepx       1.220000   0.000000   1.220000 (  1.215634)

Does this mean I should forget about Exceptions?

Absolutely no! I love Exceptions and you should love them too. However, Exceptions should not be expected and, in some circumstances, an if can be much more convenient, or at least, efficient.

One important lesson to learn from these benchmarks is that exceptions should not be part of the regular application flow. This shouldn't prevent you to use them to handle unexpected situations and exception performance shouldn't be a big deal in this case.

A lesson learned from this specific case, is to avoid using exceptions in place of conditional statements to handle not exceptional situations.

UPDATE: for more interesting comments, check out the Reddit page.

Related Posts

Filed under: — About: Ruby benchmark exceptions performance