Re: A question about table's length

Re: aes encryption

----- Original Message -----
From: Phoenix Sol
Date: 12/30/2009 5:20 PM
> What I would prefer is a small C library that does only what is 
> needed. I'd just use a straight Rijndael cipher implemented in C, but 
> I'm doubtful of my ability to do it correctly and safely.
>
> Also, I see the pure Lua AES implementation, but I want a C binding.
I use Brian Gladman's C code in my own applications: 
http://gladman.plushost.co.uk/oldsite/cryptography_technology/index.php

Josh

Re: A question about table's length

This sounds amazing!

We've been mucking around with bindings to LLVM/clang to achieve  
runtime codegen of structs, functions and FFI into existing libraries  
from within Lua scripts, but of course the JITted code lives in a  
different universe to LuaJIT, preventing optimizations through them  
(just like any other C functions). I can imagine that being able to do  
it all directly in LuaJIT would be a huge advantage (including  
readability).

I guess a next step would be an automatic C to Lua (or C to LuaJIT  
bytecode) translator...

On Dec 31, 2009, at 11:01 AM, Mike Pall wrote:

> Mark Hamburg wrote:
>> Side note to Mike Pall on that subject: You mentioned the
>> possibility of teaching LuaJIT about a matrix userdata type so
>> that it could optimize it. How hard is it to teach LuaJIT about
>> specific userdata types?
>
> This is dependent on the last work item on the status page: access
> to structured binary data plus the FFI. Once this is in place, it
> should be quite easy. You can just cut'n'paste your C struct
> definitions and allocate userdata objects holding such a struct
> (or a union, an array or a nested mix of them).
>
> I haven't given much thought to the API yet, but it might look
> something like this:
>
>  ffi.def[[
>    struct foo {
>      uint16_t bar, baz;
>      int i;
>      double d;
>    };
>    struct foo *foo_in(void);
>    void foo_out(struct foo *);
>  ]]
>  local lib = ffi.bindlib("myfoolib")
>
>  local a = lib.foo_in()
>  local b = ffi.new("struct foo")
>  b.i = a.i + 10
>  b.bar = bit.bxor(a.bar, a.baz)
>  b.baz = 0x1234
>  b.d = a.d - 1.5
>  lib.foo_out(b)
>
>  -- Nice side-effect: efficient, mutable buffers for Lua.
>  local buf = ffi.new("uint8_t[?]", n)
>  for i=0,n-1 do
>    buf[i] = i
>  end
>  io.write(buf) -- Extended to understand binary data.
>
> As you can see, this allows one to access C structs just like Lua
> tables. Obviously these structs can be exchanged with the C side
> via the FFI. All operations on these structures can be reduced by
> the JIT compiler to essentially the same machine code a C compiler
> would produce.
>
> Objects allocated on the Lua side are special userdatas and
> garbage collected as usual. Data passed in from the C side gets a
> tiny userdata wrapper (which may be optimized away by the JIT
> compiler). The userdata wrapper is GCed, but the data itself is
> owned by the C side and needs manual lifetime management. You can
> associate a __gc metamethod to signal the C side.
>
> These special userdatas have no metatable by default, but redirect
> index/newindex to internal accessor functions. You can augment
> them with your own index/newindex or other metamethods to get
> OO-like behavior.
>
> This is just the basic infrastructure. Automatic C library wrappers
> or higher-level bindings can be implemented on top of it in pure
> Lua. This could even be extended to create (mostly) automatic and
> highly-efficient C++ bindings, but this is notoriously difficult.
>
> At least that's the plan ...
>
> --Mike

Re: A question about table's length

* Graham Wakefield:

> We've been mucking around with bindings to LLVM/clang to achieve
> runtime codegen of structs, functions and FFI into existing libraries
> from within Lua scripts,

I guess the clang interface is still useful for obtaining C structs
from include files and calculating the offsets of members (which isn't
entirely straightforward).  Perhaps you can make it available in some
form?

Re: A question about table's length

On 2009-12-31, steve donovan <steve.j.donovan <at> gmail.com> wrote:
> On Thu, Dec 31, 2009 at 3:44 PM, Leo Razoumov <slonik.az <at> gmail.com> wrote:
>  > I wish that "map" would convert nil to false before pushing it into
>  > resultant array so that it becomes {10, false, 20, false} avoiding
>  > holes and preserving the length.
>
>
> So 'false' becomes a way to represent holes? But I don't see why it is
>  crucial to preserve length of the result.
>

It is not for the purpose of preserving the length of the result but
rather to preserve the iteration order. Using "pair" on Lua table does
not guarantee the order of iteration. "ipair" while guaranteeing the
order of iteration will choke on the first "nil" value. Replacing
"nil" with "false" allows to iterate with "ipair" over all the results
from "map" and skip the "false" values using "if" conditionals.

--Leo--

Re: A question about table's length

On Fri, Jan 1, 2010 at 3:22 PM, Leo Razoumov <slonik.az <at> gmail.com> wrote:
> It is not for the purpose of preserving the length of the result but
> rather to preserve the iteration order.

Yes, if you're thinking of doing a loop afterwards, it makes sense.
But leaving out the nils still leaves you with an array in the correct
order.  We agree that it's a bad idea for standard functions to
generate tables-with-holes; you want to replace the nils with
something that works like a boolean, and I want to simply leave them
out.

It is true that this would only work properly with imap (the version
that uses ipairs;  there was a suggestion to make it explicit like so
'map(fun,ipairs(t))').

The map(fun,pairs(t)) version operating on a hash-like table can be
free to leave out values

lhf asked a few or so ago why I thought simple functions like map and
filter posed interesting challenges in Lua.

This is part of the (incomplete) answer to that question ;)

steve d.

iterator sequences (was: Standard libraries)

As a pastime on new years day I updated my code for iterator sequences
from a previous post.  It is simpler, more efficient and more
powerful.    As a new example it shows how to easily combine steps
in a sequece to form a new primitive operator.  In this case: map
tonumber and filter out actual numbers.

The main function is "seq" that creates a sequence from an iterator.
The "map" and "filter" operators are included as basic operators (it
is easy to roll your own).   Finally two examples show how to use all
this...

Bye,
Wim

local type = type

function seq(f, state, key)
    local function self(s, k, ...)
        if type(s) == "function" then
            f = s(self, f, k, ...) or f
            return self, state, key
        else
            return f(s, k)
        end
    end
    return self, state, key
end

function filter(seq, f, test)
    local function self(s, k, ...)
        if k ~= nil then
            if test(k, ...) then
                return k, ...
            end
            return self(s, f(s, k))
        end
    end

    return function(s, k)
        return self(s, f(s, k))
    end
end

function map(seq, f, op)
    local function self(k, ...)
        if k ~= nil then
            return op(k, ...)
        end
    end

    return function(s, k)
        return self(f(s, k))
    end
end

-- -- only example code below this line -- --

-- a first simple example showing the use of "seq", "map" and "filter"

local function is_odd(i, x)
    return x%2 == 1
end

local function square(i, x)
    return i, x^2
end

list = {1,2,3,4,5,6,7,8,9,10}

for i, x in seq(ipairs(list)) (map, square) (filter, is_odd) do
    print(i, x)
end

-- a seconde example that shows how to easily combine steps in a sequence

local function is_number(i, x)
    return type(x) == "number"
end

local function to_number(i, x)
    return i, tonumber(x)
end

local function select_numbers(seq)
    seq(map, to_number)(filter, is_number)
end

list = {"1", "2", "aap", "3", false, "4", 5, {}}

for i, x in seq(ipairs(list)) (select_numbers) do
    print(i, x)
end

Re: A question about table's length

Haven't done a public ANN yet because although the LLVM bindings are  
fairly stable, but the Clang bindings are highly unstable right now,  
but if you're interested to take a peek: http://code.google.com/p/luaclang/

We've been mostly interested in JIT compilation, rather than source- 
code analysis, and part of the current work is finding out what is the  
most natural fit for a binding to Clang (its API isn't as amenable to  
binding as LLVM's is), and part of it is dealing with the complexities  
of translating includes, defines, target machine details and other  
typical compiler command-line flags into a form suitable for a JIT  
compiler in a dynamic library...

You could include a header and inspect the functions and struct  
definitions, including offsets and so on. Though you might have to  
instantiate some dummy stub to force the declarations in your module's  
IR; unless at some point we start binding clang's lexer/AST directly.  
Given the recent discussion of parsing large header files to auto- 
generate bindings, perhaps this wouldn't be a bad idea at all...

On Jan 1, 2010, at 2:05 AM, Florian Weimer wrote:

> * Graham Wakefield:
>
>> We've been mucking around with bindings to LLVM/clang to achieve
>> runtime codegen of structs, functions and FFI into existing libraries
>> from within Lua scripts,
>
> I guess the clang interface is still useful for obtaining C structs
> from include files and calculating the offsets of members (which isn't
> entirely straightforward).  Perhaps you can make it available in some
> form?

Re: iterator sequences (was: Standard libraries)

On Fri, Jan 1, 2010 at 5:42 PM, Wim Couwenberg <wim.couwenberg <at> gmail.com> wrote:
> for i, x in seq(ipairs(list)) (map, square) (filter, is_odd) do
>    print(i, x)
> end

That reads well, not noisy at all.

But your map functions do have to follow the (i,val) convention

> local function square(i, x)
>    return i, x^2
> end

Although an adapter can be written for single-valued functions:

function adapt_fun(fun)
  return function(i,x)
    return i,fun(x)
  end
end

and the iterator known in Python as enumerate() can convert
single-valued iterators to the ipairs form.

steve d.

PS Sequence chains like this have a big advantage over repeated
map/filter array operations, in that there are no potentially large
temporary arrays created.  (I tend to prefer single-valued iterator
operation chains)

Re: iterator sequences (was: Standard libraries)

> But your map functions do have to follow the (i,val) convention

True.  Passing everything that the iterator produces seemed the cleanest way.

> Although an adapter can be written for single-valued functions:

Other options are a "valseq" function to produce values or... use a filter:

function value(seq, f)
    local saved_key

    local function self(k, ...)
        saved_key = k
        return ...
    end
    return function(s, k)
        return self(f(s, saved_key or k))
    end
end

t = {
  aap = "noot",
  mies = "vuur",
  wim = "jet",
}

for x in seq(pairs(t)) (value) do
  print(x)
end

Bye,
Wim