Kenneth Heafield | 1 May 01:23 2011

Re: Versions of Moses

Barry is correct.

Also kenlm doesn't care what the third field is.  I just read it from
the ARPA file.  Using a model with lower order that it was trained for
is incorrect under most smoothing methods.

On 04/30/11 16:39, Barry Haddow wrote:
> Hi Alexandre
> 
> The format of the language model specification is described here
> http://www.statmt.org/moses/?n=FactoredTraining.BuildingLanguageModel
> 
> The first number is the language model type, and the third is the  
> order, so to use a 5-gram kenlm model, you need "8 0 5 /path/to/lm"
> 
> best regards - Barry
> 
> Quoting Alexandre Helle <alexandre.helle@...> on Sat, 30 Apr  
> 2011 20:19:47 +0200:
> 
>> Thanks, i think all is working, but i don't know if I'm using well the
>> kenlm.
>>
>> Is It alright to use the kenlm doing only this?:
>>
>> *Create with srilm (compiled independently of moses) the lm:
>> ngram-count -order 5 -interpolate -kndiscount -text data/low/train.cln.$f
>> -lm lm/train.$f.arpa
>>
>> *and chaging in the file moses.ini the "5" to "8".
(Continue reading)

Hieu Hoang | 1 May 03:53 2011
Picon

Re: how to add -alignment-info

this email might be useful to you
   http://www.mail-archive.com/moses-support-3s7WtUTddSA@public.gmane.org/msg03024.html
it's from october 2010, so make sure you have a recent version of moses, preferably from svn, if you want to use alignment info

On 30/04/2011 19:37, Neda NoorMohammadi wrote:
Hello all,

I want to have alignment information of the words of each segment in the decoding step,

How I can get it?

--
NoorMohammadi
_______________________________________________ Moses-support mailing list Moses-support-3s7WtUTddSA@public.gmane.org http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
Moses-support@...
http://mailman.mit.edu/mailman/listinfo/moses-support
Hieu Hoang | 1 May 06:36 2011
Picon

Re: lm question

i'm not sure if anyone else replied to you. I assume you're using the phrase-based decoder

1. the decoder doesn't use <s> & </s>. It's added somewhere in the LM code, so just commnet it out.
2. You can get the longest phrase by just setting the order of the LM to be very high, eg. 100000. However, the LM order or state control the hypothesis recombination - when it considers 2 hypos to be identical and so throws away the worse hypo. If your LM needs the entire sentence constructed so far, then the decoder can't make that decision. So don't be surprised if your LM causes good hypotheses to be pruned.


On 20 April 2011 15:11, Casey Kennington <bakuzen <at> gmail.com> wrote:
Moses support,

For my masters thesis, I am making a custom language model that I want
to use with moses. I have been successful in being able to compile moses
and use my lm with it. However, to utilize my model the best, I want to
remove all padding (<s> and </s>) and I want moses to stop breaking up
the sentences into the order size; I always want moses to submit the
longest possible phrase that it has, especially during evaluation. I'm
looking at the code and see a few places where I could maybe do it, but
I wanted to check first to see if there wasn't some easier way to make
that happen.

Thank you!
-Casey Kennington
_______________________________________________
Moses-support mailing list
Moses-support-3s7WtUTddSA@public.gmane.org
http://mailman.mit.edu/mailman/listinfo/moses-support


_______________________________________________
Moses-support mailing list
Moses-support@...
http://mailman.mit.edu/mailman/listinfo/moses-support
Casey Kennington | 1 May 07:39 2011
Picon

Re: lm question

Thank you for replying, and yes no one else did yet. I was able to figure it out. I set the order to 1 and I found the method where the phrases/sentences are broken apart and "fixed" it. I'm getting some good results. Thanks again!
-Casey

On 05/01/2011 06:36 AM, Hieu Hoang wrote:
i'm not sure if anyone else replied to you. I assume you're using the phrase-based decoder

1. the decoder doesn't use <s> & </s>. It's added somewhere in the LM code, so just commnet it out.
2. You can get the longest phrase by just setting the order of the LM to be very high, eg. 100000. However, the LM order or state control the hypothesis recombination - when it considers 2 hypos to be identical and so throws away the worse hypo. If your LM needs the entire sentence constructed so far, then the decoder can't make that decision. So don't be surprised if your LM causes good hypotheses to be pruned.


On 20 April 2011 15:11, Casey Kennington <bakuzen-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
Moses support,

For my masters thesis, I am making a custom language model that I want
to use with moses. I have been successful in being able to compile moses
and use my lm with it. However, to utilize my model the best, I want to
remove all padding (<s> and </s>) and I want moses to stop breaking up
the sentences into the order size; I always want moses to submit the
longest possible phrase that it has, especially during evaluation. I'm
looking at the code and see a few places where I could maybe do it, but
I wanted to check first to see if there wasn't some easier way to make
that happen.

Thank you!
-Casey Kennington
_______________________________________________
Moses-support mailing list
Moses-support-3s7WtUTddSA@public.gmane.org
http://mailman.mit.edu/mailman/listinfo/moses-support


_______________________________________________
Moses-support mailing list
Moses-support@...
http://mailman.mit.edu/mailman/listinfo/moses-support
Alexandre Helle | 2 May 23:44 2011
Picon

Re: Versions of Moses

Hi all

It seems that all is working well, and giving better results than with srilm.

Thank you very much people

2011/4/30 Barry Haddow <bhaddow-kQl0mEXrPEcmTMui1t/k3rVCufUGDwFn@public.gmane.org>
Hi Alexandre

The format of the language model specification is described here
http://www.statmt.org/moses/?n=FactoredTraining.BuildingLanguageModel

The first number is the language model type, and the third is the order, so to use a 5-gram kenlm model, you need "8 0 5 /path/to/lm"

best regards - Barry


Quoting Alexandre Helle <alexandre.helle-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> on Sat, 30 Apr 2011 20:19:47 +0200:

Thanks, i think all is working, but i don't know if I'm using well the
kenlm.

Is It alright to use the kenlm doing only this?:

*Create with srilm (compiled independently of moses) the lm:
ngram-count -order 5 -interpolate -kndiscount -text data/low/train.cln.$f
-lm lm/train.$f.arpa

*and chaging in the file moses.ini the "5" to "8".


2011/4/30 Barry Haddow <bhaddow-kQl0mEXrPEcmTMui1t/k3rVCufUGDwFn@public.gmane.org>

Hi Alexandre

The mert-moses script requires an additional argument -mertdir to indicate
where the mert binaries are located. This is normally the mert directory of
the Moses distribution.


Best regards - Barry

Sent from my ZX81

----- Reply message -----
From: "Alexandre Helle" <alexandre.helle-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Date: Wed, Apr 27, 2011 23:40
Subject: [Moses-support] Versions of Moses
To: "Barry Haddow" <bhaddow-kQl0mEXrPEcmTMui1t/k3rVCufUGDwFn@public.gmane.org>, "Lane Schwartz" <
dowobeha-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Cc: <moses-support-3s7WtUTddSA@public.gmane.org>


Hi

The first problem encountered was executing the compiled moses-svn with the
giza compiled (1.0.5) and srilm compiled (1.5.12), in the step "Confirm
Setup Success" of the step-by-step guide on the moses site.

The second problem was compiling moses with the giza and srilm from
repositories (with its "-devel" installed), telling me that the libraries
was not compiled inside.

Finally I have used again the moses and srilm from repositories, and I had
compiled only the moses-scripts with the "make release", but I had problems
with the "mert-moses.pl":

> Not executable:
>
/home/alex/moses/moses-decoder/scripts/bin/scripts-20110426-1407/../mert/extractor
> at
>
/home/alex/moses/moses-decoder/scripts/bin/scripts-20110426-1407/training/
> mert-moses.pl line 303.

but, this binary file doesn't exist. I think it is created with the
compiling of the moses.

As you said, I'm going to test compiling moses without the srilm.

I use the the step-by-step guide on the moses site and the paper "AERFAI
Summer School on New Trends in Pattern Recognition for Language
Technologies, Bilbao, June 25, 2008 Practice Session with Moses[1]" of
Germán Sanchis-Trilles.

P.D.: sorry, I didn't save the output of the first two errors.

Thanks,
Alexandre

2011/4/27 Barry Haddow <bhaddow-kQl0mEXrPEcmTMui1t/k3rVCufUGDwFn@public.gmane.org>

> Hi Alexandre
>
> Using the latest versions of these other tools should work. Although as
> Lane points out, using kenlm should ease compatibility issues.
>
> Could you be more specific about the problems you've encountered?
>
> Have you tried the step-by-step guide on the Moses site? It's slightly
> out-of-date in places, but mostly valid.
>
> Best regards - Barry
>
> Sent from my ZX81
>
>
> ----- Reply message -----
> From: "Alexandre Helle" <alexandre.helle-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
> Date: Wed, Apr 27, 2011 15:51
> Subject: [Moses-support] Versions of Moses
> To: <moses-support-3s7WtUTddSA@public.gmane.org>
>
> Hi all,
>
> finally I have installed moses from svn, but it doesn't worked well. I
> thing
> the problem is related to the versions of giza, srilm and irstlm.
>
> Can you tell me the versions that you use?
>
> Does someone use Moses-svn in OpenSuse without problems?
>
> Thanks all.
>
> --
> Alexandre Helle
>
>
>
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
>


--
Saludos,
Alexandre Helle



The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.




--
Saludos,
Alexandre Helle




--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.





--
Saludos,
Alexandre Helle
_______________________________________________
Moses-support mailing list
Moses-support@...
http://mailman.mit.edu/mailman/listinfo/moses-support
Souhir Gahbiche | 4 May 10:45 2011
Picon

reodering tables

Hi all,


In the case that we use two phrase tables, does we need to use two reodering tables?

Regards
SG
_______________________________________________
Moses-support mailing list
Moses-support@...
http://mailman.mit.edu/mailman/listinfo/moses-support
Barry Haddow | 4 May 11:59 2011
Picon
Picon

Re: reodering tables

> In the case that we use two phrase tables, does we need to use two
> reodering tables?

No

--

-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
hungnv54 | 6 May 22:11 2011

Moses-chart server

Hi all,
I know that Moses server doesn't support for hierarchical model, and I don't know why that ?. Besides, Did anyone customize Moses Server (now support for phrase-based) for supporting hierarchical model ?

thanks,




Thư được gửi từ http://mail.zing.vn

_______________________________________________
Moses-support mailing list
Moses-support@...
http://mailman.mit.edu/mailman/listinfo/moses-support
Philipp Koehn | 7 May 18:31 2011
Picon
Picon

Re: Moses-chart server

Hi,

there is no fundamental problem for doing this.

The two decoder branches (phrase-based and hierarchical)
are 90% merged, but they still have different "font-ends"
(in moses(-chart)-cmd) where the input / output is handled.

If you are interested in fixing this, let me know, I can give
you pointers.

-phi

On Fri, May 6, 2011 at 9:11 PM, hungnv54 <hungnv54 <at> zing.vn> wrote:
> Hi all,
> I know that Moses server doesn't support for hierarchical model, and I don't
> know why that ?. Besides, Did anyone customize Moses Server (now support for
> phrase-based) for supporting hierarchical model ?
>
> thanks,
>
>
> 
> ________________________________
>
> Thư được gửi từ http://mail.zing.vn
>
> _______________________________________________
> Moses-support mailing list
> Moses-support <at> mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>

_______________________________________________
Moses-support mailing list
Moses-support <at> mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
Philipp Koehn | 7 May 19:08 2011
Picon
Picon

Re: reodering tables

Hi,

no, but the reordering table should have entries for all the phrase pairs
in both of the phrase translation ables.

-phi

On Wed, May 4, 2011 at 9:45 AM, Souhir Gahbiche <s.gahbiche@...> wrote:
> Hi all,
> In the case that we use two phrase tables, does we need to use two reodering
> tables?
> Regards
> SG
> _______________________________________________
> Moses-support mailing list
> Moses-support@...
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>

Gmane