2 Feb 01:34
Re: to_json performance
Juan Manuel Cuello <juanmacuello <at> gmail.com>
2012-02-02 00:34:08 GMT
2012-02-02 00:34:08 GMT
"Valium helps a bit here (it should be in AR core in my opinion), it
can selectively pick the columns you need and it will return back
values in an Array per every object, so we can have something like
this: `[['a', 'b', 'c'], ['a', 'c', 'd']]` "
ActiveRecord now has 'pluck' which I think does the same.
On Fri, Jan 27, 2012 at 3:26 AM, Claudio Poli <masterkain <at> gmail.com> wrote:
Originally posted on github, reported to the right place.
I would like to open a discussion about how `to_json` and `as_json`
operates in Rails from a performance standpoint.
I'm using Rails 3.2 but this issue applies to almost all versions of
Rails.
Our use case presents the challenge in sending out potentially large
JSON (or XML, but we'll focus on JSON rendering here) bodies as a
response of a API calls that runs through a normal Rails controller.
Someone states that if you are sending out more than 1MB of data
"you're doing it wrong".
I respectfully disagree, as there are different cases, based also on
what an application should do and is designed for.
There can be few major bottlenecks in rendering JSON as fast as
possibile from start to finish, three of them are:
* database query
* AR objects instantiation
* collection transformation in JSON
I'll exclude the AR timings, so let's tackle the first problem, AR
objects.
Having a result set of 50k entries is obviously painful, and doing it
with AR it will kill your instance, no questions asked.
Worse, in my use case this result set can be asked frequently from the
client.
This however can be easily solved by using a sapient mixture of
`caches_action`and `stale?`, although there may be few key issues when
using memcached and friends, because especially in managed
implementations, they may not support data keys larger than 1MB.
Since instantiating AR objects it's out of the question let's focus
and getting what we really want for our tests, an 'attribute' Hash to
convert to JSON.
Valium helps a bit here (it should be in AR core in my opinion), it
can selectively pick the columns you need and it will return back
values in an Array per every object, so we can have something like
this: `[['a', 'b', 'c'], ['a', 'c', 'd']]`
With little sorcery we can transform each Array representing the
requested record values in a Hash with keys mapped, like so:
class Hash
def self.transpose(keys, values)
self[*keys.zip(values).flatten]
end
end
keys = %W{token artist album genre release_year title duration
duration_in_seconds position play_count favorited disc_number}
my_hash = <at> playlist.media_files.ordered.values_of(*keys).collect{ |
v| Hash.transpose(keys, v) }
Nice, now we have a Array of Hash without instantiating AR objects but
with all the info we need about our objects.
This operation proved to be _extremely_ fast even if we are
transforming and inserting 50k records.
Now, on to the rendering:
render json: my_hash
This should try to call `to_json`on the Hash.
Since few Rails versions JSON encoding has been delegated to
`MultiJson`, which is ok.
Needless to say I'm using and requiring `yajl-ruby`, so I'm pretty
confident that MultiJson will pick it up as my engine, and in fact it
is:
1.9.3p0 :001 > MultiJson.engine
=> MultiJson::Engines::Yajl
The key issue here is that with a large Hash the render action takes
36786ms to finish ( of which 8253.6ms spent for the database query,
that high because of multiple ORDER BY, that's another matter).
So I tried another way:
render text: Yajl::Encoder.encode(my_hash)
Using `Yajl::Encoder.encode`(or `MultiJson.encode`for that matter)
yields another kind of result: 12614ms (8253.6ms always spent on the
database query)
Is there something we can do about this?
Note:
some may point out to try few gems like `rabl`or `acts_as_api`, but
don't forget they are mainly for presentation purpose and they are
comparable to the `to_json` performance in this regard.
Thanks for reading.
--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core <at> googlegroups.com.
To unsubscribe from this group, send email to rubyonrails-core+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en.
--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core <at> googlegroups.com.
To unsubscribe from this group, send email to rubyonrails-core+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en.
> I believe this is correct *as long as* the user regexps have no
> backreferences, because if you have a = /(..)\1/ and b = /(.)\1/ then "xx"
> matches b, but does not match the union a|b because the backreference \1 in
> b now refers to the group in a.
>
> OK, so this is the question: do you guys use backreferences in custom
> inflections? If you didn't we could consider ruling them out for 4.0 to be
> able to implement this.
If you can rule out backreferences, it seems possible you could build a
state machine that would perform in linear time to the string length.
It seems though, with your algorithm, as the string gets longer, the
original implementation is faster:
RSS Feed