Some thoughts on serialization
Karl Wettin <karl.wettin <at> gmail.com>
2012-02-23 06:09:54 GMT
This is me thinking out loud.
It must be the fifth time I bring this up here, and I still haven't done anything about but to complain about
it. I didn't even get in touch with the people previously recommended here that might have good ideas
regarding to how to solve it.
Serialization is terrible slow.
This one application of mine contains some 12GB heap worth of domain data. Due to the complexity of my domain
model and the amount of data it contains now I'm starting to hit stack overflows both while marshalling and
unmarshalling unless I increase my -Xss. Given the amout of threads I usually run this is not a good
solution for me. Thus in deployed environment I don't use snapshots at all, and since I have loads of
changes to the persistent data every second the journals grows huge. It can take an hour to restart.
Theoretically this means that in a year from now it can take me days to load the journals.
Generally speaking this is not a problem, there are many ways around this such as starting the new VM before
bringing down the previous. But it's really annoying.
I have a few ideas. For starters all serialization I've seen use a single thread. Theoretically my computer
could do it 32x faster if all CPU resources was used. (This is of course only applicable for loading
snapshots.) So the other day tried to implement a proof of concept where the marhsaller would associate
all instances with an identity, decouple all instances in my graph and then have the unmarshaller first
instantiate them in multiple threads and then in a second iteration recouple all instances, but due to how
ObjectIn/OutputStream works (in conjunction with my crappy code) I didn't get it working yet. In the end
I'll probably have to instantiate some 3-4 helper object for each object to be unmarshalled. But even
without trying it out for real I'd say that with my 32 threads it sho
uld still be a lot faster than running a single thread. It will also cost quite a bit of RAM to keep track of what
to be coupled with what in the second iteration, but I don't think anyone here mind spending a bit of RAM for
greater speed...
(Continue reading)
RSS Feed