Robert Harris | 1 May 2011 17:24

Custom builds of JtR v1.7.7 and v1.7.7 with jumbo 1 for Windows

John Users,

I've compiled the following versions of John the Ripper and uploaded them to
the custom builds wiki (http://openwall.info/wiki/john/custom-builds)

Windows:

Version 1.7.7 

Version 1.7.7 with the jumbo 1 patch

More to come.

-Robert B. Harris from VA

magnum | 2 May 2011 20:42

Re: JtR 1.7.7 & 1.7.7-jumbo-1 + MPI error compilation OS X

On 2011-04-30 03:53, magnum wrote:
> On 2011-04-29 06:55, Solar Designer wrote:
>> All - I don't mind merging the MPI patch into jumbo, which will
>> apparently make life easier for W.A. Thus, please feel free to make and
>> submit a revision of the MPI patch suitable for integration into the
>> jumbo patch. It must not strictly depend on MPI; instead, it should
>> have -DHAVE_MPI, which could be uncommented in the Makefile to activate
>> the functionality (similarly to HAVE_SKEY and -lskey, which we already
>> have - these came from another patch I merged).
>
> Actually I almost finished an #ifdef'd fullmpi patch for 1.7.6 but I
> felt it was too intrusive to include in Jumbo so I never posted it. But
> I do use mpi regularly for non-OMP formats. Maybe I should reconsider.

I found a solution I can live with: It's now so adamantly ifdef'ed  that 
if you do not enable it, you'll get a binary with the same sha1sum as a 
pristine jumbo build. This way any bugs (of mine) are guaranteed to only 
exist in an MPI-enabled build. And after some clean-up, it's not that 
horrific, Now I think it may be valid to include it in Jumbo.

I'll release it soon, I hope. I outsourced some beta testing to W.A :)

magnum

Solar Designer | 2 May 2011 20:51
Favicon

Re: JtR 1.7.7 & 1.7.7-jumbo-1 + MPI error compilation OS X

On Mon, May 02, 2011 at 08:42:39PM +0200, magnum wrote:
> I found a solution I can live with: It's now so adamantly ifdef'ed  that 
> if you do not enable it, you'll get a binary with the same sha1sum as a 
> pristine jumbo build. This way any bugs (of mine) are guaranteed to only 
> exist in an MPI-enabled build. And after some clean-up, it's not that 
> horrific, Now I think it may be valid to include it in Jumbo.
> 
> I'll release it soon, I hope. I outsourced some beta testing to W.A :)

Sounds great.

In general, now is a good time to submit patches for inclusion in jumbo.
I kept 1.7.7-jumbo-1 conservative (compared to 1.7.6-jumbo-12), but I am
OK with adding a lot more stuff to 1.7.7-jumbo-2 and on.  A requirement
is that patches to merge should not introduce non-optional dependencies
on extra libraries.

Thanks,

Alexander

Solar Designer | 2 May 2011 21:03
Favicon

Re: JtR 1.7.7 & 1.7.7-jumbo-1 + MPI error compilation OS X

On Sat, Apr 30, 2011 at 03:53:40AM +0200, magnum wrote:
> W.A - 1.7.7 as of today does not add any new formats, performance or 
> major functionality.

Actually, it does add the dummy format (for testing/tuning of rulesets,
etc.), AVX/XOP support for DES, better performance with large numbers of
hashes loaded for some hash types (most notably, NTLM), and
warnings/hints on alternate hash types.  Perhaps this is mostly
irrelevant to W.A. specifically and it does not count as "major".

> On the contrary, you lose intrinsics

Yes, the intrinsics patch by Bartavelle hasn't yet been updated, nor
merged.  Perhaps now is the time to consider it for merging again, if
it's implemented on top of 1.7.7-jumbo-1 such that it doesn't break
compiles on any systems.

> and enhanced Markov.

What's that?  I might have missed it.

> I do intend to post an update of fullmpi, but it's low prio. I would 
> prefer to see a Jumbo with stuff like intrinsics, new Markov and utf-8 
> added (not to mention Jim's proposed core changes) before posting an mpi 
> patch because I'm not very keen on "maintaining" public versions of mpi. 
> Not that I have any private things in my tree, it's just that it's a 
> fairly long (and boring) procedure to rebase it to plain jumbo and 
> regression test that.

I do intend to try merging those other patches you mention, except that
(Continue reading)

Solar Designer | 2 May 2011 21:35
Favicon

Re: Custom builds of JtR v1.7.7 and v1.7.7 with jumbo 1 for Windows

On Sun, May 01, 2011 at 11:24:27AM -0400, Robert Harris wrote:
> I've compiled the following versions of John the Ripper and uploaded them to
> the custom builds wiki (http://openwall.info/wiki/john/custom-builds)
> 
> Windows:
> 
> Version 1.7.7 
> Version 1.7.7 with the jumbo 1 patch

I've added these to:

http://www.openwall.com/john/#contrib
http://download.openwall.net/pub/projects/john/contrib/win32/

I've also copied Simon John's contributed builds of 1.7.7-jumbo-1,
1.7.7-omp-des-7, and 1.7.7-omp-des-4 for Fedora to:

http://download.openwall.net/pub/projects/john/contrib/linux/

Thanks to both of you!

Alexander

magnum | 3 May 2011 00:06

Re: JtR 1.7.7 & 1.7.7-jumbo-1 + MPI error compilation OS X

On 2011-05-02 21:03, Solar Designer wrote:
> On Sat, Apr 30, 2011 at 03:53:40AM +0200, magnum wrote:
>> W.A - 1.7.7 as of today does not add any new formats, performance or
>> major functionality.
>
> Actually, it does add the dummy format (for testing/tuning of rulesets,
> etc.), AVX/XOP support for DES, better performance with large numbers of
> hashes loaded for some hash types (most notably, NTLM), and
> warnings/hints on alternate hash types.

True, I just meant to relax W.A a little :)

>> On the contrary, you lose intrinsics
>
...
>> and enhanced Markov.
>
> What's that?  I might have missed it.

It comes with Simon's "intrinsics-fixes", unfortunately missing on the 
wiki. It was announced at 
http://www.openwall.com/lists/john-users/2011/02/21/3 and latest version 
I know of was in http://www.openwall.com/lists/john-dev/2011/04/04/1

magnum

bartavelle | 3 May 2011 09:41
Picon

Re: JtR 1.7.7 & 1.7.7-jumbo-1 + MPI error compilation OS X

On 03/05/2011 00:06, magnum wrote:
> It comes with Simon's "intrinsics-fixes", unfortunately missing on the
> wiki. It was announced at
> http://www.openwall.com/lists/john-users/2011/02/21/3 and latest version
> I know of was in http://www.openwall.com/lists/john-dev/2011/04/04/1

Right now I'm waiting for the official git repository. Once it appears,
I will clone, apply my stuff and publish my "branch".

The "new" in "new Markov" is just the ability to define low limits.
There also was a bug with memory allocation that was too low, and this
patch uses "char" instead of larger integer types to store its
probability tables.

RB | 3 May 2011 16:56
Picon

Re: JtR 1.7.7 & 1.7.7-jumbo-1 + MPI error compilation OS X

On Tue, May 3, 2011 at 01:41, bartavelle <bartavelle@...> wrote:
> Right now I'm waiting for the official git repository. Once it appears,
> I will clone, apply my stuff and publish my "branch".

IIRC that's waiting on Solar's action, but I'll just throw it out
there that I've been maintaining a git-cvs clone of JtR privately for
quite some time, change history and all.

git cvsimport -m -v -d :pserver:anoncvs@...:/cvs
Owl/packages/john/john

http://github.com/aozsyn/john-git

magnum | 4 May 2011 20:47

FullMPI for 1.7.7

john-1.7.7-jumbo-1-fullmpi-9.diff.gz is now uploaded to the wiki. It has 
a couple of improvements compared to 1.7.6-fullmpi8.

* -DHAVE_MPI, passing "the sha1sum test" when disabled.
* Do not show any ETA until progress is above 0.00%. NOTE that this 
should really be in non-MPI too but it's currently ifdef'ed.
* Incremental mode now has progress status and ETA (note the previous 
point about muting though), but not after resuming a job. It's useful 
for --inc=alnum or smaller keyspaces and/or very fast formats.
* Incremental mode: the empty string candidate is now only tested on one 
node.
* If one node runs out of hashes, it sends an abort to the other nodes 
(they would otherwise keep attacking already cracked hashes).
* If a node stops with error (eg. file locking problems because of 
flakey NFS) it will send an abort to all other nodes.
* Jobs could not be resumed if some nodes had finished. Now the .rec 
file does not get deleted until all nodes are done.
* Mitigated the interleaving of output lines when showing status for a 
running job (it doesn't happen if you print everything in one call).
* Higher preference of distributing rules instead of words.
* --mem=0 will force loading of wordfile to memory, regardless of size. 
This too could be in non-MPI as well IMO.
* Added some compilation hints to README.mpi.

I also did some tweaking so it applies together with the utf8 patch 
without rejecting.

The code has been stable for a long time now. If you get problems, they 
are most likely due to NFS problems (for multi-host setup) or using a 
buggy MPI version. I recommend using latest stable OpenMPI but mpich2 
(Continue reading)

RB | 4 May 2011 21:03
Picon

Re: FullMPI for 1.7.7

On Wed, May 4, 2011 at 12:47, magnum <rawsmooth@...> wrote:
> john-1.7.7-jumbo-1-fullmpi-9.diff.gz is now uploaded to the wiki. It has a
> couple of improvements compared to 1.7.6-fullmpi8.

Thanks, magnum, for picking up maintenance of the MPI series.  Job
changes made me have to choose personal time to maintain it, of which
I had precious little.


Gmane