john skaller | 14 Apr 03:42 2014

too many objects

I have a bit of a theory now on this. Its not a GC bug.
Rather, what we have is a generator or recursion making
a long chain of frames which aren't being collected
because they are, actually reachable. I suspect the
iterator generator is the culprit.

The compiler used to (and I think still does) unwind the stack
by NULLing out the caller to prevent this. 

Something like this must be going on. The loop driving the tests
and build leads to increasing number of objects. Numbers like
80,000 object don't make any sense. I'd have said around 1000

john skaller

Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
john skaller | 13 Apr 04:16 2014

Re: [felix] too many roots bug (#61)

I've rewritten the RTL so Felix dynamic libs and their
instances are now garbage collected. A library instance
is the library together with a thread frame object (global store).
The instance is actually only needed transiently if at all,
since any running procedure using the thread frame
refers to it: hanging on the thread frame via the instance
is probably an issue, however in general a library such
as a plugin can have other entry points, and these are
coupled with the instance.

Anyhow, with the new method, libraries do NOT get unloaded
when there are no references to them because this is entirely
unsafe in C++ and very nasty to manage in C.  Bad languages
lead to nasty problems. Any pointer into the library such as
a vtable pointer or C string requires the code remain resident
and there's no reliable way to track this.

However there's a new problem created by this: the objects holding
the library and instance are not themselves tracked. These are the
very top level objects, the felix "world" objects. This is fine if they're
on the stack, since the stack is scanned conservatively, provided the
threads base stack pointer is obtained before creating the world

however if these objects get heaped .. they will have to be garbage 
collected too. Since the whole idea is to get rid of the need to
root and unroot the instances, since unrooting them requires
action under program control which is hard to do.

I may have to actually "destroy" thread frames in instances artificially
(Continue reading)

john skaller | 5 Apr 11:39 2014

Re: [felix] coroutine test failure

On 05/04/2014, at 12:23 PM, srean wrote:

> Slightly tangential question: spawn_fthread takes a proc that do I/O on schannels. Can they take a
generator and can such generators also yield ?

No of course not, spawn_fthread is a library function:

	proc spawn_fthread : (1->0)->0;

so it only accepts a procedure with unit argument.

john skaller

john skaller | 4 Apr 13:51 2014

coroutine test failure

I am getting a failure

Processing [14/127]: coroutines-01.flx
sh: line 1:  1842 Segmentation fault      "env"
"test" > "/Users/johnskaller/.felix/cache/text/Users/johnskaller/felix/test/regress/rt/coroutines-01.stdout"

on my Mac using my clang 3.3 svn. This is with no optimisations.
With -O1 it works. So this is probably a bug in clang. The code is doing
some nasty hacks internally: it uses computed gotos in a way which should
be perfectly safe if the jump address is valid, however the address in function
A is taken in function B by using "assembler labels". This is a gcc trick which
should work in clang too (and does with -O1). The -O flag should only impact
C++ compilation, however I'm using a --prototype for testing which impacts
the amount of inlining by flxg as well. So it could be that Felix inlining is making
the hackery work, without which it doesn't. I think this works with gcc 4.x though.
So again: seems like a clang bug.

Assembler labels and computed gotos are not required in Felix,
however they improve performance by replacing a switch
on an integer with a computed goto. The choice is made by
a MACRO based on configuration data.

Coroutines are a very low level features based on an exchange
jump function which in turn is based on an  extreme low level
feature: Felix level label addressing and computed gotos.
Fthread based lightweight threads do not use these Felix features
but ALL procedures use C++ computed gotos for resumptions
(Continue reading)

john skaller | 6 Mar 08:53 2014

Re: binding eager vs lazy [ was [felix] spawn_fthread new semantics ]

On 06/03/2014, at 3:38 PM, srean wrote:

> On Wed, Mar 5, 2014 at 2:25 PM, srean <srean.list@...> wrote:
> I dont technically need it,
> I take it back. Closures are pervasive in Felix and rightfully so. If one is to use Felix iterators or Felix
objects, they are just closures. So understanding them is pretty essential, I think. And this
conversation helped.

It helps most to practice not talk. If you want I can give you some
exercises. It takes less time to run a Felix program than write
an email.

Copy and paste tutorial examples is a good start :)

Documenting *how* Felix generates C++ would be good but
it is a non-trivial job.

john skaller

Subversion Kills Productivity. Get off Subversion & Make the Move to Perforce.
With Perforce, you get hassle-free workflows. Merge that actually works. 
Faster operations. Version large binaries.  Built-in WAN optimization and the
(Continue reading)

john skaller | 5 Mar 19:59 2014

Re: binding eager vs lazy [ was [felix] spawn_fthread new semantics ]

On 06/03/2014, at 5:36 AM, srean wrote:

> I dont mind it so much, it gets the work done. My problem was that I couldnt figure out how to express the
semantics I wanted. Is this the best way, I dont know, but it can express what I had in mind.

It's unclear why you would need this for your work.
You could presumably do it in C++ which doesn't have closures.
In general coding it's usually the other way: you want lazy evaluation
and don't know how to get it.

john skaller

Subversion Kills Productivity. Get off Subversion & Make the Move to Perforce.
With Perforce, you get hassle-free workflows. Merge that actually works. 
Faster operations. Version large binaries.  Built-in WAN optimization and the
freedom to use Git, Perforce or both. Make the move to Perforce.
john skaller | 5 Mar 08:15 2014

model spec

One possibility for further control of Felix entities would be a model spec:

	fun f() { .. } model "C++HeapFunctor", "Felix-C-Function";

The model clause lets you specify the allowed set of models.
The default is "any of them".

If you say "C++HeapFunctor" then you always get a C++ class object
on the heap.

If you say Felix-C-Function, you get exactly a C function which also
accepts the thread_frame as the first parameter. 

If the compiler gets a context that requires some models it finds
the intersection of the workable models with the specified ones
and fails if a suitable model doesn't exist.

For example, C functions can be passed as arguments, but
not where a Felix closure is required.

The idea is you get fine control of the semantics or a compiler error.
An more real example:

	fun f: int -> int = "$1" model "C++HeapFunctor";

If you say that the normal "macro" behaviour will be suppressed:
no lazy evaluation. Instead a closure wrapper will be generated
and used.

Felix already has some modelling control:
(Continue reading)

john skaller | 5 Mar 02:54 2014

Re: binding eager vs lazy [ was [felix] spawn_fthread new semantics ]

On 05/03/2014, at 11:59 AM, srean wrote:

> Write code
> like C++ and you won't have an issue in Felix either.
> Well for the longest time you did not have closures in C++ in the first place.

And still don't, not really.

> So to copy that style would be to not use closures in Felix, which would be rather sad . One does have, what
they call functors, but then you get to choose the binding explicitly, so you know what you are getting for
the most part.

Right. No lexical scoping so you have to pass all the variables
you want to use explicitly.

Guess what? Felix uses this technique. That's how it works.
It just automates what you would do in C++ with "functors".
It makes all the local variable go into one struct, and passes
the pointer to that struct to your child closure.

in fact, this technology is archaic. The list of scopes, from
root up through to the deepest child, has a name: its
called a "display". Pascal (in the late 70s) used displays.

In fact the x86 has special instructions for managing displays.

Felix procedures are heap allocated structs of variables,
(Continue reading)

john skaller | 5 Mar 02:45 2014

Re: binding eager vs lazy [ was [felix] spawn_fthread new semantics ]

On 05/03/2014, at 11:56 AM, srean wrote:

> I know, the problem is given my use case its high risk because I cant debug the output.

Of course you can. You can use debug prints. You can edit the C++:
the generated code is actually documented. There's a special flag
to generate heaps of documentation.

And Felix has debugging features in the language.

And most importantly .. the developer -- me -- is highly sensitive
to user requirements. If you need extra debugging support I'll
provide it.

Read carefully. *user* requirements. If you're not using it,
I'll have fun arguing with you but it's unlikely anything will be done.

> Right, that was exactly my question, recall, I have never asked for lexical closure or dynamic closures.
Just explanations about how it works, and examples of how you use it.

You have to write code. There are tons of examples.
There are hundreds of pages of documented examples in the
tutorial and regression tests, plus several varying size tools
from a file copy utility to a complete webserver, written
in Felix.

There is evidence it is easy to learn. I had an electrical engineer
with C++ knowledge write a complete prepaid phone service
in Felix in 3 days. Several other developers wrote stuff just
(Continue reading)

john skaller | 5 Mar 02:36 2014

Re: binding eager vs lazy [ was [felix] spawn_fthread new semantics ]

On 05/03/2014, at 11:48 AM, srean wrote:

> The issue here has nothing to do with the "kind" of closure.
> It has to do with the scope over which the closure closes.
> That is indeed a goodway to put it. The Felix compiler may choose to close over a scope that you did not have in
mind, so ou are better of specifying it carefully if your algorithm depends on it.

It doesn't "choose" a scope in this case, the scope of a loop is function scope.
It's a flat scope, the whole function is a single C++ object, the variables
are non-static member variables.

That's if the function is a closure, i.e. not inlined. When you inline
a function its variables become variables of the calling function.
Again, a flat scope. On the heap. An object. Felix doesn't use
automatic (stack) variables (except occasionally, transiently,
for temporaries it generates).

Even if you write a block:

	var x = 1
	begin var x = 2; .. blah x; end
	{ var x = 2; blah x; };

so that the x's are different, the block will be inlined.
The inlining of anonymous blocks like this is forced.
The inlining just renames the inner x variables
(Continue reading)

john skaller | 4 Mar 22:51 2014

Re: binding eager vs lazy [ was [felix] spawn_fthread new semantics ]

On 05/03/2014, at 5:40 AM, srean wrote:

> Are both necessary ? Can I not use 'noinline' and get away with it (I would rather not get in the way of
optimizations). Of course this question is only relevant when I am changing the captured variable outside.

At the moment yes, noinline and inline are not hints, they're
enforced because they have real semantics. They control
whether a "function" is really a macro or not.

However, the "semantics" of closure don't guarantee this, it just works
that way at the moment. There's no reason a closure should not
be specialised over an argument, it just isn't, usually, at the moment.
[There are some cases where Felix eliminates closures :]

Clearly that's not satisfactory. But it will have to do for now,
because I don't know a better way. When a closure is formed,
parameters are copied up, variable references in the body are not.
Its logical, the parameter is a variable in the local scope, an assignment

	p = a

copies a into p.

I would suggest writing functional code. Inside that functional code,
you will have some mutable data structures like varrays which are
just pointers so they're copied by reference (i.e. the pointer is copied).

It is the same in Ocaml. Code isn't referentially transparent
in Ocaml, unless you don't use mutable data structures.
(Continue reading)