Freetalk 0.1RC3

Freetalk:
- Missed update to Version.java at beginning; update at end. Do not reset namespace.
- Fix extractIdFromFreetalkAddress, missing Freetalk.WOT_CONTEXT; apparently this breaks NNTP login.
- CSS fix.
- Separate deleting a message rating and reverting its effect because the user changed their mind versus
deleting it without removing its effect because we are deleting an identity. In the latter case we don't
want to undo the effect.
- Disallow  <at>  in nicknames.
- Database version increment and auto-upgrade to fix nicknames.

Freetalk:
- Missed update to Version.java at beginning; update at end. Do not reset namespace.
- Fix extractIdFromFreetalkAddress, missing Freetalk.WOT_CONTEXT; apparently this breaks NNTP login.
- CSS fix.
- Separate deleting a message rating and reverting its effect because the user changed their mind versus
deleting it without removing its effect because we are deleting an identity. In the latter case we don't
want to undo the effect.
- Disallow  <at>  in nicknames.
- Database version increment and auto-upgrade to fix nicknames.

Web of Trust 0.4 RC3

WoT RC3:
- Update Version (missing RC2 at beginning of review period, then RC3 at end). Don't change the namespace.
- Add a TODO.
- Make IntroductionPuzzle implement Cloneable. ** Doubtful, as our clone() doesn't use super.clone() - Toad.
- Implement (Own)IntroductionPuzzle.clone(). (Supporting changes: slight changes to constructors
and add getSolver()).
- Implement IntroductionPuzzle.equals(). ** Throws if passed a non-IntroductionPuzzle, should return false.
- Comments etc.
- Indenting.
- Improvements to code used by unit tests e.g. alternate constructors.
- Implement lots of unit tests for IntroductionPuzzleStore.
- Allow changing trust level for identity on the IdentityPage. This includes changing it from any
OwnIdentity (shows a list). * Security implications? I guess it can be changed if we eventually decide to
have logins?
- Add IntroductionPuzzle.hashCode() so can put in a HashSet.
- Make IntroductionPuzzleStore.getFreeIndex work for Identity's not just OwnIdentity's.
- Better toString().
- Fix not updating mLastChangedDate initially in Trust constructor when parameters are at their default values.
- Database version increment and auto-upgrade to fix the previous issue, and the next one.
- Disallow  <at>  in nicknames.
- Fix collisions during puzzle inserts by aborting the previous lot, which would have collided.

p0s
ShadowW4lk3r

WoT RC3:
- Update Version (missing RC2 at beginning of review period, then RC3 at end). Don't change the namespace.
- Add a TODO.
- Make IntroductionPuzzle implement Cloneable. ** Doubtful, as our clone() doesn't use super.clone() - Toad.

WoT and Freetalk RC2 reviews

These are all fine, this is just a third party changelog. The jars are deployed.

One query: Freetalk:

dd9a1927895fac1d592a05d21c367fe4be907a48
- what's the point of this? neither variable is used after...

WoT: up to version-0.4-RC2
- Pass the right string into generateKeyPair() (no effect in practice).
- Correct the startup order to prevent NPEs.
- Fix NPE in score recomputation.
- Set mLastChangedDate correctly (to creation time not epoch) to fix test failure in Identity.
- Optimise recomputation: Correct wrong score to already calculated value rather than recomputing.
- Logging and toString()'s.
- Rollback in database test flushCaches() so missing commits are caught.
- Mark collided inserts as inserted to prevent continual reinserts.
- Use the helper functions in some more places.
- Paranoid checking to track down puzzle reinserts bug.
- Javadocs.
- Make stuff final.
- Rearrange code in update scores to make it more readable.
- Add compute all scores inside asserts (so only happen when debugging) at end of updating scores; if trust
list import is happening, don't do the global recompute.
- Factor removeTrust(OwnIdentity, Identity) out.
- When importing own identities from XML, always allow importing OwnIdentity's and always assume they
have capacity. This is some sort of corner case, it should have worked anyway...
- Avoid unnecessary commit in XML import if nothing changed.
- Add throwIfNotStored() to Persistent, from Freetalk. Use it.
- Add default toString() for Persistent, includes the database oid (which persists across restarts
unless there is a defrag) as well as the Java object ID.

Changes up to RC1 on WoT and Freetalk

WoT: up to cd8d5429772a8aae986877e3371984edc01c2463
- Reset 0.4-final-development into master!
- Add and use TimeUtil.setTimeToZero(Date)
- Bug 4715: Static values should be transient as we persist static values for enum's.
- Logging.
- Increase WoT version to 4004, 0.4.0 RC1.
- Do not abbreviate nicks on known identities page.
- Add two seed identities, xor and nextgens. Improve text for the initial trust for seeds.
- Refetch when capacity changes from 0 to >0 after updating scores. (Related to newly bootstrapped
identities not being initially able to introduce others)
- Fix NPE on startup integrity test.

Freetalk: up to 996d91c335eaddf8eb7af5b444ed048f77e606d1
- Reset 0.1-final-development into master!
- TODO.
- Bug 4715: Static values should be transient as we persist static values for enum's.
- Use the right Date constructor in unit tests.
- Left-align board descriptions.
- Logging.
- Change on-reply trust increment/decrement to 1, since this happens by default. Should be configurable
in future.
- Change version string to 0.1 RC1.
- L10n fix re NNTP.

WoT: up to cd8d5429772a8aae986877e3371984edc01c2463
- Reset 0.4-final-development into master!
- Add and use TimeUtil.setTimeToZero(Date)
- Bug 4715: Static values should be transient as we persist static values for enum's.
- Logging.

Freetalk review and issues

Hashtables and Freetalk version.

f1464e96ea9c0985d066d9f6bb0e36d11b2de639
- I have already explained that it's bad to use Hashtables. Firstly, they synchronize unnecessarily;
secondly, they'll go away eventually and are a somewhat deprecated API. Put manual null checks in if you
need to. Finally, my experience suggests that storing maps in db4o is a good idea.

b2b52cd8a849f27a19b813b939a31fd9d555f018
- This is an issue here too, as evidenced by your comment about losing values! See what I said on WoT.

fbb624c5ba110b9a92f695bfb3e46c626ac774b5
- Adding full Freetalk version may not be sensible/safe long-term - "I'm running a broken version, attack
me!". It's fine for now though.

Reviewing master first, then the 0.1 branch.

master ( up to cf6a17d28ea4555211115907d1cabf7d4e7fb502 ):
- Logging.
- Remove unused WoTMessageInserter.mRequestClient.
- Rename mMessageIDs -> mPutterMessageIDs.
- Add a separate set mMessageIDs for the message lists currently being inserted.
- Improve reliability (and efficiency) of message inserts: Do not abort existing inserts. We won't start
new ones if we're already fetching them, because getNotInsertedOwnMessageLists() now ignores lists
already being inserted.
- Fix database integrity test to validate whether a message is actually visible i.e. there is a reference to it.
- Commit after loading Config.
- Add message *after* creating a reply link in SubscribedBoard.addMessage.
- More sanity checks.
- Work around bug 4620, onMessageAdded is called more than once causing messages not to be marked as read;
recompute it.

Freetalk WoT auto-linking

422ef05d630c5f692046246b105c77016f27f81b
- you should support other protocols, they are not all gone yet:
+                if (url.substring(0,4) != "http") {
+                    url = "http://" + url;
+                }
+                HTMLNode linkNode = new HTMLNode("a", "href", "/?_CHECKED_HTTP_="+url, url);

df68cb9a34511b963ebef07fb55fe0e71d942faf
- Same here. There is a list of protocols in the filter code.
- You should probably do new URI() as a basic sanity check. Or maybe use the GenericReadFilterCallback
filter itself to process the URIs?

422ef05d630c5f692046246b105c77016f27f81b
- you should support other protocols, they are not all gone yet:
+                if (url.substring(0,4) != "http") {
+                    url = "http://" + url;
+                }
+                HTMLNode linkNode = new HTMLNode("a", "href", "/?_CHECKED_HTTP_="+url, url);

df68cb9a34511b963ebef07fb55fe0e71d942faf
- Same here. There is a list of protocols in the filter code.
- You should probably do new URI() as a basic sanity check. Or maybe use the GenericReadFilterCallback
filter itself to process the URIs?

Code review and various issues in new branch WoT up to b5e3ba4ee425919f3ce37372e79aa2b1dde368a3

[20:40:29] <toad_> p0s: Hashtable is sort of obsolete and locks where it doesn't need to; is this the best
fix for the fixme?
(in Config, changed from HashMap to Hashtable)
cc8f2d130a6c56af5b73adbdb8af6edfa22c8277

ca886f6c63a564ba6aacf1fe7262701e8b6d396d
- we use a Hashtable here too.
- this is silly, it adds unnecessary locking; the comments describing it say "we do not need
synchronization" !

ca886f6c63a564ba6aacf1fe7262701e8b6d396d
- in setProperties(), shouldn't you delete the old mProperties before replacing it to prevent a database
leak? Although given the difficulties with maps, you might be better off using queries than a map.

235eca62111d8accecf9036c835fc4cd6363cc31
- Config.storeAndCommit() no longer commits. Why?

c9272be55c7b9a56758701b34938b852c3e4f7d0
- Why not just create a TrivialTicker?

77fddbb78ed5d82306e0ea34289c42e394913d19
- You need to be absolutely sure that you use a fixed timezone!

25ee9e3f3088ce1dd9f8f703949baa990f007ae4
- This is yet another demonstration of why you should not use maps in databases, or always activate to depth 1
and be very very very careful not to indirectly activate maps! Maps are also used in other places, sadly ...

4d424402da71536ddd05997ba47b9c3ab1339e37
- Does it get stored after setDatabaseFormatVersion?

Latest build of Freetalk

1b85e73dec2b5fd1664b5f5cbe44662a7a3051cf
- Javadocs don't entirely make sense on these two functions:
-- getUnavailableNewMessageListIndex() gets the lowest index such that there are no fetched indexes
above it.
-- getUnavailableOldMessageListIndex() gets the highest unfetched index below the last contiguous
group of fetched indexes.

General point: You use an awful lot of exceptions. If you don't need the stack traces you should override
(like we do in e.g. LowLevelPutException) so they are not created.

Review:
Freetalk: 2010-10-17
- Show total downloaded messages on welcome page.
- Make sure we delete ghosts which are only attached to other ghosts.
- Clarify/simplify logic / rename variables.
- Improve a thread name.
- Check for a message being its own parent or thread in constructor.
- A message used to be able to be its own thread. It can't any more. Update the unit tests.
- Extra synchronization to fix deadlock (bug #4351).
- Clobber the database - set version -83.
- Major optimisation: IdentityStatistics: Caches the upper and lower indexes of a contiguous range
(close to the end) of message list indexes that have not been fetched.
- Wire in IdentityStatistics to MessageManager.
- Add test in self-check logic for IdentityStatistics with identity null. (=> we deleted the identity
without deleting the stats).
- Improve a unit test by checking for FetchFailedMarker.
- Synchronization changes in MessageManager.
- Handle ALL_DATA_NOT_FOUND like DATA_NOT_FOUND.

xor

Re: Freetalk message XML format

On Saturday 13 December 2008 04:05:39 am Daniel Cheng wrote:
> On Fri, Dec 12, 2008 at 11:09 PM, Daniel Cheng
> 
> <j16sdiz+freenet@...> wrote:
> > On Fri, Dec 12, 2008 at 10:57 PM, xor <xor@...> wrote:
> >> Hi people,
> >>
> >> you requested a public discussion of the XML format.
> >> It is almost identitical to the format of FMS so message exchange
> >> between the two could be implemented if anybody feels like doing that.
> >> There are some things to know though:
> >> - Freetalk generates the message ID by SHA256-hashing the URI of the
> >> message. The <MessageID> tags in the XML are ignored, it's just there
> >> for FMS.
> >
> > When a database get corrupted ...
> > Can we restore it from backup, reinserting the message, without
> > generating duplicated post?
> >
> > fms allow this, it use message id as unique identifier...
> > i think your solution don't --- your message id depends on the USK
> > edition
> 
> btw, i see you are getting rid of the message list.
> 
> SomeDube on FMS say: "You're shooting yourself in the foot here."
> And I agree with him.
> 
> This break reinsert of message, download message only if the group is
> subscribed .. etc..

Re: Freetalk message XML format

On Friday 12 December 2008 04:09:01 pm Daniel Cheng wrote:
> On Fri, Dec 12, 2008 at 10:57 PM, xor <xor@...> wrote:
> > Hi people,
> >
> > you requested a public discussion of the XML format.
> > It is almost identitical to the format of FMS so message exchange between
> > the two could be implemented if anybody feels like doing that.
> > There are some things to know though:
> > - Freetalk generates the message ID by SHA256-hashing the URI of the
> > message. The <MessageID> tags in the XML are ignored, it's just there for
> > FMS.
> 
> When a database get corrupted ...
> Can we restore it from backup, reinserting the message, without
> generating duplicated post?
> 
> fms allow this, it use message id as unique identifier...
> i think your solution don't --- your message id depends on the USK edition

This is resolved. Message ID is random UUID + SSK public key hash now. 
Reinserts are possible therefore (not implemented yet though).

On Friday 12 December 2008 04:09:01 pm Daniel Cheng wrote:
> On Fri, Dec 12, 2008 at 10:57 PM, xor <xor@...> wrote:
> > Hi people,
> >
> > you requested a public discussion of the XML format.
> > It is almost identitical to the format of FMS so message exchange between
> > the two could be implemented if anybody feels like doing that.

(Important) Freetalk status update

Hi folks,

this update is personal, not technical.

I wanted to tell you that the continuous patience I am demanding from you 
makes me feel very guilty to finish it ASAP.
In fact every time someone asks "when?", which happens quite often nowadays, I 
feel ashamed that I'm unable to give you a precise date.

I feel as obliged to finish Freetalk ASAP as I feel obliged to anything which 
is relevant in real life. Such as helping my friends out when they feel bad, 
which has happened quite often recently, or as helping my mother with 
repairing various stuff in her house related to basic equipment (electricity 
etc.).
Freetalk is equal to those in my personal priorities and sometimes I even 
screwed up real life stuff because I couldn't quit writing code early enough. 
It is not just a project for me, it is MY project and I feel obliged to finish 
to for the community as soon as I can.

I did not commit code this week yet because I have been feeling like shit and 
could not get much more than 4 hours of sleep even though I would have had the 
time to sleep as long as I need to.

This might make you consider to have someone else finish it and release it, but 
I would like to admit that I have been feeling like shit very often for the 
past months due to many personal issues, and yet I managed to be sort of 
productive and write very much code.
And therefore I feel confident that I can get my feet back on the ground during 
the next week and cut down lets say 5 of the remaining 10 bugtracker entries.