Simon Marlow | 1 Oct 12:02 2003
Picon

RE: swap leak problem


> Oh yes, one more datum.  If I run hs_perform_gc before each and every
> mallocForeignPtrBytes and reallocBytes, the leak goes much 
> more slowly, and
> the code gets much further in the job before getting killed.  
> But it still
> leaks, and still gets killed.

It could potentially be a memory leak in the RTS.  There are two ways to
proceed: if you have a GHC build around, you can compile up a debugging
RTS and run with sanity checking enabled (+RTS -DS), which will check
for memory leaks.  If there are any memory leaks in the RTS, this will
throw an assertion failure immediately.

Alternatively, you can package up your code and send it to us with
instructions on how to reproduce.

Cheers,
	Simon
David Roundy | 1 Oct 14:05 2003
Picon

Re: swap leak problem

On Wed, Oct 01, 2003 at 11:02:52AM +0100, Simon Marlow wrote:
>  
> > Oh yes, one more datum.  If I run hs_perform_gc before each and every
> > mallocForeignPtrBytes and reallocBytes, the leak goes much more slowly,
> > and the code gets much further in the job before getting killed.  But
> > it still leaks, and still gets killed.
> 
> It could potentially be a memory leak in the RTS.  There are two ways to
> proceed: if you have a GHC build around, you can compile up a debugging
> RTS and run with sanity checking enabled (+RTS -DS), which will check
> for memory leaks.  If there are any memory leaks in the RTS, this will
> throw an assertion failure immediately.

This sounds like a good plan (I do have a ghc build around).  How do I go
about compiling a debugging RTS?
--

-- 
David Roundy
http://www.abridgegame.org
Petter Egesund | 1 Oct 15:18 2003
Picon

Haskell-beginners problem with memory consuption

Hi, I have just started using Haskell, and it has been really fun so far.
But I have problems with memory usage when handling large files. What I do
is the following;

I load my file in one chunk, and does a lot of substitutes on the string -
this is quick eating all my memory and the computers start to get really
slow.

The problem is of course that the string is copied each time I do a
substitute, and I wonder if a more experienced haskeller has a better
solution to my problem.

I have myself considered these solutions, but they all seems non-elegant;

1. Use some IO extensions to fake a mutable variable - but solving a problem
like this with a solution which is meant for IO does not seem right?

2. Split the file in smaller parts could be a solution, but then I would
have to do a lot of preparsing to make sure I not spilt my file in the
middle of a pattern. Sounds to me like developing a lex-like system for such
a easy task is overkill?

3. I could maybe use some form of mutable arrays, but doing string regexps
on an array...?

4. ?

Any clues or opinions?

Cheers,
(Continue reading)

Wolfgang Jeltsch | 1 Oct 15:36 2003
Picon

Re: Haskell-beginners problem with memory consuption

Am Mittwoch, 1. Oktober 2003, 15:18 schrieb Petter Egesund:
> [...]

> The problem is of course that the string is copied each time I do a
> substitute, and I wonder if a more experienced haskeller has a better
> solution to my problem.

It doesn't have to be a problem that the string is copied each time. If you 
have, e.g., functions
    f1, f2, ..., fn :: String -> String
and do something like
    f1 (f2 (... (fn string)...))
then string and the intermediate data can be removed by the garbage collector 
as soon as they are not needed.

Unfortunately, from your message it's not very clear to me what exactly you 
mean, and, unfortunately again, I'm not an expert in Haskell memory 
management.

> I have myself considered these solutions, but they all seems non-elegant;
> [...]

Indeed, they all look very non-elegant, and I think, there is a better 
solution.

> Cheers,
>
> Petter Egesund

Wolfgang
(Continue reading)

Petter Egesund | 1 Oct 15:41 2003
Picon

SV: Haskell-beginners problem with memory consuption

Hi & thanks for answering;

I should have been more precise, my function works like this:

fun :: String -> String
	look for pat1 in string - if found subst with sub1
	look for pat2 in string - if found subst with sub2
	look for pat3 in string - if found subst with sub3
  recurse until no pattern is found

Cheers

PE

-----Opprinnelig melding-----
Fra: Wolfgang Jeltsch [mailto:wolfgang <at> jeltsch.net]
Sendt: 1. oktober 2003 15:36
Til: The Haskell Cafe
Emne: Re: Haskell-beginners problem with memory consuption

Am Mittwoch, 1. Oktober 2003, 15:18 schrieb Petter Egesund:
> [...]

> The problem is of course that the string is copied each time I do a
> substitute, and I wonder if a more experienced haskeller has a better
> solution to my problem.

It doesn't have to be a problem that the string is copied each time. If you 
have, e.g., functions
    f1, f2, ..., fn :: String -> String
(Continue reading)

Simon Marlow | 1 Oct 16:06 2003
Picon

RE: swap leak problem


> This sounds like a good plan (I do have a ghc build around).  
> How do I go
> about compiling a debugging RTS?

Add the following to mk/build.mk:

  GhcRtsHcOpts = -optc-DDEBUG
  GhcRtsCcOpts = -g

then go into ghc/rts and 'make clean && make'.  Now rebuild your program
using this GHC (or alternatively add a -L flag pointing to this ghc/rts
directory).  Then run it with +RTS -DS -Sstderr, observe it going very
slowly and see if it throws any assertion failures.

Cheers,
	Simon
Simon Marlow | 1 Oct 17:42 2003
Picon

RE: "interact" behaves oddly if used interactively

[ taking this one to haskell-café... ]

> I still do not quite agree with Simon that 'interact' exposes anything
> but non-strictness. Non-strictness means that
> 
>   map toUpper _|_ = _|_
>   map toUpper ('a':_|_) = ('A':_|_)
>   map toUpper ('a':'b':_|_) = ('A':'B':_|_)
> 
> and 'interact (map toUpper)' is a great way to experience 
> this property.
> 
> However, you can also experience the property without 'interact',
> evaluating expressions like
> 
>   take 2 (map toUpper ('a':'b':undefined))

Certainly you can observe non-strictness, that's not the point.  The point is that you can also observe more
than just non-strictness using interact, and I don't think that is desirable.  For example:

  interact (\xs -> let z = length xs in "Hello World\n")

Now, Haskell is a pure language, so it shouldn't matter whether the implementation evaluates z or not, as
long as it is careful not to violate the non-strict semantics by turning a terminating program into a
non-terminating one.  A parallel Haskell implementation might happily spawn off another thread to
evaluate z for example.  An optimistic implementation might evaluate z for a fixed amount of time before
continuing with the main thread of evaluation.

BUT in the presence of lazy I/O this simply isn't true any more.  Why?  Because z is not pure; evaluating it has a
side-effect.  And yet it has type Int.  Am I the only one who thinks this is wrong?
(Continue reading)

Bayley, Alistair | 1 Oct 17:09 2003

Another beginner's memory consumption problem...

I'm trying the add-a-gram challenge from here:
http://www.itasoftware.com/careers/programmers-archive.php

... and I'm also experiencing runaway memory consumption. If I load the
supplied list of words (a 1.6M file) and search for shorter strings, things
are OK. Memory consumption increases dramatically as I search for longer and
longer trails. The function searchWord starts with a particular map and
searches the maps below it (i.e. those containing shorter words), so when I
search for longer trails, a larger number of maps must be examined. When I
search for short trails, only a few (2-3 say) of the maps are examined. I
think this implies some kind of lazy evaluation problem, but I'm not sure
where to go from here.

Was it a bad idea to use a Map of Maps as a data structure? Initially I had
a list of Maps, but this wouldn't even load the full file (so I suppose lazy
evaluation has allowed me to get a bit further...). Maps seemed like the
right data structures to use for this problem. Is there some way I could
force evaluation of the maps as the file is loaded? (Or is blaming the maps
a red herring?)

Program below:

> module Main where

> import Data.FiniteMap
> import Data.List
> import System.IO

Data structure is a Map of Maps. The top-level Map holds Maps indexed by Int
(for words of size n).
(Continue reading)

Robert Ennals | 1 Oct 17:55 2003
Picon
Picon

Re: "interact" behaves oddly if used interactively


[snip]

> Certainly you can observe non-strictness, that's not the point.  The point is
> that you can also observe more than just non-strictness using interact, and I
> don't think that is desirable.  For example:
> 
>   interact (\xs -> let z = length xs in "Hello World\n")
> 
> Now, Haskell is a pure language, so it shouldn't matter whether the
> implementation evaluates z or not, as long as it is careful not to violate
> the non-strict semantics by turning a terminating program into a non-terminating
> one.  A parallel Haskell implementation might happily spawn off another thread
> to evaluate z for example.  An optimistic implementation might evaluate z for
> a fixed amount of time before continuing with the main thread of evaluation.
> 
> BUT in the presence of lazy I/O this simply isn't true any more.  Why?  
> Because z is not pure; evaluating it has a side-effect.  And yet it has type
> Int.  Am I the only one who thinks this is wrong?

I agree with you completely.

It is wrong for all the same reasons that unsafePerformIO is wrong, except 
that it is worse than that because unsafePerformIO has "unsafe" in the title, 
and people are discouraged from using it without due care. By contrast, 
"interact" and "getContents" are presented as being nice friendly functions 
that everyone should use.

If I had my way, getContents would be called unsafeGetContents and interact 
would be called unsafeInteract. Even better would be for them to both be 
(Continue reading)

Olaf Chitil | 1 Oct 18:26 2003
Picon

Re: "interact" behaves oddly if used interactively

Simon Marlow wrote:
> Certainly you can observe non-strictness, that's not the point.  The point is that you can also observe
more than just non-strictness using interact, and I don't think that is desirable.  For example:
> 
>   interact (\xs -> let z = length xs in "Hello World\n")
> 
> Now, Haskell is a pure language, so it shouldn't matter whether the implementation evaluates z or not, as
long as it is careful not to violate the non-strict semantics by turning a terminating program into a
non-terminating one.  A parallel Haskell implementation might happily spawn off another thread to
evaluate z for example.  An optimistic implementation might evaluate z for a fixed amount of time before
continuing with the main thread of evaluation.
> 
> BUT in the presence of lazy I/O this simply isn't true any more.  Why?  Because z is not pure; evaluating it has
a side-effect.  And yet it has type Int.  Am I the only one who thinks this is wrong?

I don't know what everyone else thinks, but I do not see your problem.
There is nothing impure about z. According to non-strict semantics we
have

(\xs -> let z = length xs in "Hello World\n") _|_ = "Hello World\n"

So every implementation has to make sure that this equation holds.

You know it is not enough for optimistic evaluation to `avoid _|_' by
evaluating eagerly only for a fixed amount of time. You also have to
bail out if evaluation raises an exception. This in particular if eager
evaluation causes a black hole: reevaluation at some later time may
yield a perfectly fine value. Likewise the presence of lazy IO forces
you to test if the value is currently available or needs to be read from
some handle; the presence of lazy IO adds another sort of black hole.
(Continue reading)


Gmane