jira | 20 Dec 08:00 2014
Picon

Subscription: PIG patch available

Issue Subscription
Filter: PIG patch available (23 issues)

Subscriber: pigdaily

Key         Summary
PIG-4352    Port local mode tests to Tez - TestUnionOnSchema
            https://issues.apache.org/jira/browse/PIG-4352
PIG-4340    PigStorage fails parsing empty map.
            https://issues.apache.org/jira/browse/PIG-4340
PIG-4323    PackageConverter hanging in Spark
            https://issues.apache.org/jira/browse/PIG-4323
PIG-4313    StackOverflowError in LIMIT operation on Spark
            https://issues.apache.org/jira/browse/PIG-4313
PIG-4264    Port TestAvroStorage to tez local mode
            https://issues.apache.org/jira/browse/PIG-4264
PIG-4251    Pig on Storm
            https://issues.apache.org/jira/browse/PIG-4251
PIG-4213    CSVExcelStorage not quoting texts containing \r (CR) when storing
            https://issues.apache.org/jira/browse/PIG-4213
PIG-4193    Make collected group work with Spark
            https://issues.apache.org/jira/browse/PIG-4193
PIG-4111    Make Pig compiles with avro-1.7.7
            https://issues.apache.org/jira/browse/PIG-4111
PIG-4103    Fix TestRegisteredJarVisibility(after PIG-4083)
            https://issues.apache.org/jira/browse/PIG-4103
PIG-4004    Upgrade the Pigmix queries from the (old) mapred API to mapreduce
            https://issues.apache.org/jira/browse/PIG-4004
PIG-4002    Disable combiner when map-side aggregation is used
            https://issues.apache.org/jira/browse/PIG-4002
(Continue reading)

liyunzhang_intel (JIRA | 19 Dec 09:43 2014
Picon

[Updated] (PIG-4282) Enable unit test "TestForEachNestedPlan" for spark


     [
https://issues.apache.org/jira/browse/PIG-4282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

liyunzhang_intel updated PIG-4282:
----------------------------------
    Attachment: PIG-4282_1.patch

submit PIG-4282_1.patch. add a function "tupleListToStringList" which changes tupleList to 
stringList, compare the expectedResults(string list format) equals the actual results(string list format).

> Enable unit test "TestForEachNestedPlan" for spark
> --------------------------------------------------
>
>                 Key: PIG-4282
>                 URL: https://issues.apache.org/jira/browse/PIG-4282
>             Project: Pig
>          Issue Type: Bug
>          Components: spark
>            Reporter: liyunzhang_intel
>         Attachments: PIG-4282.patch, PIG-4282_1.patch, TEST-org.apache.pig.test.TestForEachNestedPlan.txt
>
>
> error log is attached

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

(Continue reading)

jira | 19 Dec 08:00 2014
Picon

Subscription: PIG patch available

Issue Subscription
Filter: PIG patch available (23 issues)

Subscriber: pigdaily

Key         Summary
PIG-4352    Port local mode tests to Tez - TestUnionOnSchema
            https://issues.apache.org/jira/browse/PIG-4352
PIG-4340    PigStorage fails parsing empty map.
            https://issues.apache.org/jira/browse/PIG-4340
PIG-4323    PackageConverter hanging in Spark
            https://issues.apache.org/jira/browse/PIG-4323
PIG-4313    StackOverflowError in LIMIT operation on Spark
            https://issues.apache.org/jira/browse/PIG-4313
PIG-4264    Port TestAvroStorage to tez local mode
            https://issues.apache.org/jira/browse/PIG-4264
PIG-4251    Pig on Storm
            https://issues.apache.org/jira/browse/PIG-4251
PIG-4213    CSVExcelStorage not quoting texts containing \r (CR) when storing
            https://issues.apache.org/jira/browse/PIG-4213
PIG-4193    Make collected group work with Spark
            https://issues.apache.org/jira/browse/PIG-4193
PIG-4111    Make Pig compiles with avro-1.7.7
            https://issues.apache.org/jira/browse/PIG-4111
PIG-4103    Fix TestRegisteredJarVisibility(after PIG-4083)
            https://issues.apache.org/jira/browse/PIG-4103
PIG-4004    Upgrade the Pigmix queries from the (old) mapred API to mapreduce
            https://issues.apache.org/jira/browse/PIG-4004
PIG-4002    Disable combiner when map-side aggregation is used
            https://issues.apache.org/jira/browse/PIG-4002
(Continue reading)

李运田 | 19 Dec 03:05 2014

use pig in eclipse

hi all.
I want to use pig in eclipse.my hadoop(yarn) cluster and eclipse are in the same linux cluster .my pig
configuration  in eclipse::

 Properties props = new Properties();     
     props.setProperty("fs.defaultFS", "hdfs://10.210.90.*:8020");
     props.setProperty("hadoop.job.user", "hadoop");
     props.setProperty("mapreduce.framework.name", "yarn");
     props.setProperty("yarn.resourcemanager.hostname", "10.210.90.*");
     props.setProperty("yarn.resourcemanager.admin.address", "10.210.90.*:8141");
        props.setProperty("yarn.resourcemanager.address", "10.210.90.*:8050");
     props.setProperty("yarn.resourcemanager.resource-tracker.address", "10.210.90.*:8025");
     props.setProperty("yarn.resourcemanager.scheduler.address", "10.210.90.*:8030");

 
but,it  is not connected. I dont know how I can configure the pig in eclipse?
Praveen Rachabattuni (JIRA | 18 Dec 09:02 2014
Picon

[Commented] (PIG-4268) Enable unit test "TestStreamingUDF" in spark


    [
https://issues.apache.org/jira/browse/PIG-4268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14251353#comment-14251353
] 

Praveen Rachabattuni commented on PIG-4268:
-------------------------------------------

Unit test now passes on jenkins too. Thanks [~kellyzly]

> Enable unit test "TestStreamingUDF" in spark
> --------------------------------------------
>
>                 Key: PIG-4268
>                 URL: https://issues.apache.org/jira/browse/PIG-4268
>             Project: Pig
>          Issue Type: Bug
>          Components: spark
>            Reporter: liyunzhang_intel
>            Assignee: liyunzhang_intel
>         Attachments: PIG-4268.patch, TEST-org.apache.pig.impl.builtin.TestStreamingUDF.txt
>
>
> the error log is attached

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

(Continue reading)

Praveen Rachabattuni (JIRA | 18 Dec 09:02 2014
Picon

[Resolved] (PIG-4268) Enable unit test "TestStreamingUDF" in spark


     [
https://issues.apache.org/jira/browse/PIG-4268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Praveen Rachabattuni resolved PIG-4268.
---------------------------------------
    Resolution: Fixed

> Enable unit test "TestStreamingUDF" in spark
> --------------------------------------------
>
>                 Key: PIG-4268
>                 URL: https://issues.apache.org/jira/browse/PIG-4268
>             Project: Pig
>          Issue Type: Bug
>          Components: spark
>            Reporter: liyunzhang_intel
>            Assignee: liyunzhang_intel
>         Attachments: PIG-4268.patch, TEST-org.apache.pig.impl.builtin.TestStreamingUDF.txt
>
>
> the error log is attached

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

jira | 18 Dec 08:00 2014
Picon

Subscription: PIG patch available

Issue Subscription
Filter: PIG patch available (23 issues)

Subscriber: pigdaily

Key         Summary
PIG-4352    Port local mode tests to Tez - TestUnionOnSchema
            https://issues.apache.org/jira/browse/PIG-4352
PIG-4340    PigStorage fails parsing empty map.
            https://issues.apache.org/jira/browse/PIG-4340
PIG-4323    PackageConverter hanging in Spark
            https://issues.apache.org/jira/browse/PIG-4323
PIG-4313    StackOverflowError in LIMIT operation on Spark
            https://issues.apache.org/jira/browse/PIG-4313
PIG-4264    Port TestAvroStorage to tez local mode
            https://issues.apache.org/jira/browse/PIG-4264
PIG-4251    Pig on Storm
            https://issues.apache.org/jira/browse/PIG-4251
PIG-4213    CSVExcelStorage not quoting texts containing \r (CR) when storing
            https://issues.apache.org/jira/browse/PIG-4213
PIG-4193    Make collected group work with Spark
            https://issues.apache.org/jira/browse/PIG-4193
PIG-4111    Make Pig compiles with avro-1.7.7
            https://issues.apache.org/jira/browse/PIG-4111
PIG-4103    Fix TestRegisteredJarVisibility(after PIG-4083)
            https://issues.apache.org/jira/browse/PIG-4103
PIG-4004    Upgrade the Pigmix queries from the (old) mapred API to mapreduce
            https://issues.apache.org/jira/browse/PIG-4004
PIG-4002    Disable combiner when map-side aggregation is used
            https://issues.apache.org/jira/browse/PIG-4002
(Continue reading)

Zhang, Liyun | 18 Dec 07:38 2014
Picon

Is there any way to guarantee the sequence of “group” field as the input when using “group” operator in pig

Hi all,
   I met a problem that “group operator has different results in different engines like "spark" and "mapreduce"(PIG-4282<https://issues.apache.org/jira/browse/PIG-4282>).

groupdistinct.pig
A = load 'input1.txt' as (age:int,gpa:int);
B = group A by age;
C = foreach B {
 D = A.gpa;
 E = distinct D;
generate group, MIN(E);
};
dump C;
input1.txt is:
10 89
20 78
10 68
10 89
20 92
the mapreduce output is:
(10,68),(20,78)
the spark output is
(20,78),(10,68)
These two results are different, because the sequence of field ‘group’ is not same.

Is there any way to guarantee the sequence of “group” field as the input when using “group”
operator in pig?

Best regards
Zhang,Liyun

(Continue reading)

liyunzhang_intel (JIRA | 18 Dec 07:23 2014
Picon

[Updated] (PIG-4282) Enable unit test "TestForEachNestedPlan" for spark


     [
https://issues.apache.org/jira/browse/PIG-4282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

liyunzhang_intel updated PIG-4282:
----------------------------------
    Attachment: PIG-4282.patch

group operator has different results in different engines like "spark" and "mapreduce".
for example:
groupdistinct.pig
{code}
A = load 'input1.txt' as (age:int,gpa:int); 
B = group A by age;  
C = foreach B { 
 D = A.gpa; 
 E = distinct D;
 generate group, MIN(E);
};
dump C;
{code}

input1.txt is:
10	89
20	78
10	68
10	89
20	92

the mapreduce output is:
(Continue reading)

jira | 17 Dec 08:00 2014
Picon

Subscription: PIG patch available

Issue Subscription
Filter: PIG patch available (23 issues)

Subscriber: pigdaily

Key         Summary
PIG-4352    Port local mode tests to Tez - TestUnionOnSchema
            https://issues.apache.org/jira/browse/PIG-4352
PIG-4340    PigStorage fails parsing empty map.
            https://issues.apache.org/jira/browse/PIG-4340
PIG-4323    PackageConverter hanging in Spark
            https://issues.apache.org/jira/browse/PIG-4323
PIG-4313    StackOverflowError in LIMIT operation on Spark
            https://issues.apache.org/jira/browse/PIG-4313
PIG-4264    Port TestAvroStorage to tez local mode
            https://issues.apache.org/jira/browse/PIG-4264
PIG-4251    Pig on Storm
            https://issues.apache.org/jira/browse/PIG-4251
PIG-4213    CSVExcelStorage not quoting texts containing \r (CR) when storing
            https://issues.apache.org/jira/browse/PIG-4213
PIG-4193    Make collected group work with Spark
            https://issues.apache.org/jira/browse/PIG-4193
PIG-4111    Make Pig compiles with avro-1.7.7
            https://issues.apache.org/jira/browse/PIG-4111
PIG-4103    Fix TestRegisteredJarVisibility(after PIG-4083)
            https://issues.apache.org/jira/browse/PIG-4103
PIG-4004    Upgrade the Pigmix queries from the (old) mapred API to mapreduce
            https://issues.apache.org/jira/browse/PIG-4004
PIG-4002    Disable combiner when map-side aggregation is used
            https://issues.apache.org/jira/browse/PIG-4002
(Continue reading)

Praveen Rachabattuni (JIRA | 16 Dec 10:43 2014
Picon

[Commented] (PIG-4268) Enable unit test "TestStreamingUDF" in spark


    [
https://issues.apache.org/jira/browse/PIG-4268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14247984#comment-14247984
] 

Praveen Rachabattuni commented on PIG-4268:
-------------------------------------------

Committed the patch to Spark branch. Thanks [~kellyzly]
Shall verify with jenkins report before it is marked resolved.

> Enable unit test "TestStreamingUDF" in spark
> --------------------------------------------
>
>                 Key: PIG-4268
>                 URL: https://issues.apache.org/jira/browse/PIG-4268
>             Project: Pig
>          Issue Type: Bug
>          Components: spark
>            Reporter: liyunzhang_intel
>            Assignee: liyunzhang_intel
>         Attachments: PIG-4268.patch, TEST-org.apache.pig.impl.builtin.TestStreamingUDF.txt
>
>
> the error log is attached

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

(Continue reading)


Gmane