Praveen Rachabattuni (JIRA | 18 Nov 13:39 2014
Picon

[Updated] (PIG-4336) Add Spark support in generateURI used for unit testing


     [
https://issues.apache.org/jira/browse/PIG-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Praveen Rachabattuni updated PIG-4336:
--------------------------------------
    Attachment: PIG-4366.patch

> Add Spark support in generateURI used for unit testing
> ------------------------------------------------------
>
>                 Key: PIG-4336
>                 URL: https://issues.apache.org/jira/browse/PIG-4336
>             Project: Pig
>          Issue Type: Sub-task
>          Components: spark
>            Reporter: Praveen Rachabattuni
>            Assignee: Praveen Rachabattuni
>         Attachments: PIG-4366.patch
>
>

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Praveen Rachabattuni (JIRA | 18 Nov 13:38 2014
Picon

[Updated] (PIG-4336) Add Spark support in generateURI used for unit testing


     [
https://issues.apache.org/jira/browse/PIG-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Praveen Rachabattuni updated PIG-4336:
--------------------------------------
    Status: Patch Available  (was: Open)

> Add Spark support in generateURI used for unit testing
> ------------------------------------------------------
>
>                 Key: PIG-4336
>                 URL: https://issues.apache.org/jira/browse/PIG-4336
>             Project: Pig
>          Issue Type: Sub-task
>          Components: spark
>            Reporter: Praveen Rachabattuni
>            Assignee: Praveen Rachabattuni
>

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Praveen Rachabattuni (JIRA | 18 Nov 13:34 2014
Picon

[Created] (PIG-4336) Add Spark support in generateURI used for unit testing

Praveen Rachabattuni created PIG-4336:
-----------------------------------------

             Summary: Add Spark support in generateURI used for unit testing
                 Key: PIG-4336
                 URL: https://issues.apache.org/jira/browse/PIG-4336
             Project: Pig
          Issue Type: Sub-task
          Components: spark
            Reporter: Praveen Rachabattuni
            Assignee: Praveen Rachabattuni

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

liyunzhang_intel (JIRA | 18 Nov 13:34 2014
Picon

[Commented] (PIG-4266) Umbrella jira for unit tests for Spark


    [
https://issues.apache.org/jira/browse/PIG-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14216140#comment-14216140
] 

liyunzhang_intel commented on PIG-4266:
---------------------------------------

Hi all,
  My marry leave is from 2014/11/09~2014/11/19(UTC+8). No mail access.

Best Regards
Zhang,Liyun

> Umbrella jira for unit tests for Spark
> --------------------------------------
>
>                 Key: PIG-4266
>                 URL: https://issues.apache.org/jira/browse/PIG-4266
>             Project: Pig
>          Issue Type: Task
>          Components: spark
>            Reporter: Praveen Rachabattuni
>            Assignee: liyunzhang_intel
>         Attachments: spark-tests
>
>
> Currently we have 1 unit test in spark-tests. We should have all unit tests from tez-tests to be enabled for
spark so we can have overview of failure and successful tests.

(Continue reading)

Praveen Rachabattuni (JIRA | 18 Nov 13:31 2014
Picon

[Updated] (PIG-4266) Umbrella jira for unit tests for Spark


     [
https://issues.apache.org/jira/browse/PIG-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Praveen Rachabattuni updated PIG-4266:
--------------------------------------
    Issue Type: Task  (was: Sub-task)
        Parent:     (was: PIG-4059)

> Umbrella jira for unit tests for Spark
> --------------------------------------
>
>                 Key: PIG-4266
>                 URL: https://issues.apache.org/jira/browse/PIG-4266
>             Project: Pig
>          Issue Type: Task
>          Components: spark
>            Reporter: Praveen Rachabattuni
>            Assignee: liyunzhang_intel
>         Attachments: spark-tests
>
>
> Currently we have 1 unit test in spark-tests. We should have all unit tests from tez-tests to be enabled for
spark so we can have overview of failure and successful tests.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

(Continue reading)

Rohini Palaniswamy (JIRA | 18 Nov 11:45 2014
Picon

[Updated] (PIG-4333) Split BigData tests into multiple groups


     [
https://issues.apache.org/jira/browse/PIG-4333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rohini Palaniswamy updated PIG-4333:
------------------------------------
    Attachment: PIG-4333-2.patch

Fix Glob test failures due to studenttab20m removed from local mode. 

> Split BigData tests into multiple groups
> ----------------------------------------
>
>                 Key: PIG-4333
>                 URL: https://issues.apache.org/jira/browse/PIG-4333
>             Project: Pig
>          Issue Type: Improvement
>            Reporter: Rohini Palaniswamy
>            Assignee: Rohini Palaniswamy
>             Fix For: 0.14.1
>
>         Attachments: PIG-4333-1.patch, PIG-4333-2.patch
>
>
>    With parallel test execution, it stalls at the end as BigData tests take more time to execute. In mapred
mode it takes more than 1 hour
> Marking for 0.14.1 even though non-critical as speedier verification of build is very helpful.

--
This message was sent by Atlassian JIRA
(Continue reading)

Davliatov Alexander (JIRA | 18 Nov 08:38 2014
Picon

[Commented] (PIG-2932) Setting high default_parallel causes IOException in local mode


    [
https://issues.apache.org/jira/browse/PIG-2932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14215833#comment-14215833
] 

Davliatov Alexander commented on PIG-2932:
------------------------------------------

Sorry, didn't read the comments properly: I use hadoop 1.2.1, and fix is hadoop 2.+.

> Setting high default_parallel causes IOException in local mode
> --------------------------------------------------------------
>
>                 Key: PIG-2932
>                 URL: https://issues.apache.org/jira/browse/PIG-2932
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Gianmarco De Francisci Morales
>            Assignee: Cheolsoo Park
>            Priority: Critical
>             Fix For: 0.11
>
>         Attachments: PIG-2932.patch
>
>
> This bug has been confirmed only in local mode.
> When setting a high default_parallel, Pig fails on some operations.
> The following data and script reproduce the bug.
> Data:
> {code}
(Continue reading)

jira | 18 Nov 08:00 2014
Picon

Subscription: PIG patch available

Issue Subscription
Filter: PIG patch available (24 issues)

Subscriber: pigdaily

Key         Summary
PIG-4334    PigProcessor does not set pig.datetime.default.tz
            https://issues.apache.org/jira/browse/PIG-4334
PIG-4333    Split BigData tests into multiple groups
            https://issues.apache.org/jira/browse/PIG-4333
PIG-4332    Remove redundant jars packaged into pig-withouthadoop.jar for hadoop 2
            https://issues.apache.org/jira/browse/PIG-4332
PIG-4326    AvroStorageSchemaConversionUtilities does not properly convert schema for maps of arrays of records
            https://issues.apache.org/jira/browse/PIG-4326
PIG-4313    StackOverflowError in LIMIT operation on Spark
            https://issues.apache.org/jira/browse/PIG-4313
PIG-4264    Port TestAvroStorage to tez local mode
            https://issues.apache.org/jira/browse/PIG-4264
PIG-4251    Pig on Storm
            https://issues.apache.org/jira/browse/PIG-4251
PIG-4239    "pig.output.lazy" not works in spark mode
            https://issues.apache.org/jira/browse/PIG-4239
PIG-4207    Make python udfs work with Spark
            https://issues.apache.org/jira/browse/PIG-4207
PIG-4111    Make Pig compiles with avro-1.7.7
            https://issues.apache.org/jira/browse/PIG-4111
PIG-4103    Fix TestRegisteredJarVisibility(after PIG-4083)
            https://issues.apache.org/jira/browse/PIG-4103
PIG-4066    An optimization for ROLLUP operation in Pig
            https://issues.apache.org/jira/browse/PIG-4066
(Continue reading)

Richard So (JIRA | 18 Nov 02:58 2014
Picon

[Updated] (PIG-2692) Make the Pig unit faciliities more generalizable and update javadocs


     [
https://issues.apache.org/jira/browse/PIG-2692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Richard So updated PIG-2692:
----------------------------
    Release Note: 
Added the ability to mock multiple aliases when running a job.
Added assertOutputAnyOrder to be order independent.
Added user docs for mocking.

  was:
Added the ability to mock multiple aliases when running a job.
Modified assertOutput to be order independent.

          Status: Patch Available  (was: Open)

I updated my patch.  As per my previous comment, I didn't want to break existing functionality.  I added a
couple assertOutputAnyOrder methods that can be used instead of assertOutput.  If you have an ordered
output where it matters use assertOutput.  I managed to fix my issue with documentation and added a section
pertaining to mocking in the user docs.

> Make the Pig unit faciliities more generalizable and update javadocs
> --------------------------------------------------------------------
>
>                 Key: PIG-2692
>                 URL: https://issues.apache.org/jira/browse/PIG-2692
>             Project: Pig
>          Issue Type: Improvement
>            Reporter: Jeremy Hanna
(Continue reading)

Richard So (JIRA | 18 Nov 02:56 2014
Picon

[Updated] (PIG-2692) Make the Pig unit faciliities more generalizable and update javadocs


     [
https://issues.apache.org/jira/browse/PIG-2692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Richard So updated PIG-2692:
----------------------------
    Attachment: pig2692.patch

> Make the Pig unit faciliities more generalizable and update javadocs
> --------------------------------------------------------------------
>
>                 Key: PIG-2692
>                 URL: https://issues.apache.org/jira/browse/PIG-2692
>             Project: Pig
>          Issue Type: Improvement
>            Reporter: Jeremy Hanna
>            Assignee: Richard So
>            Priority: Minor
>             Fix For: 0.15.0
>
>         Attachments: pig2692.patch
>
>
> This ticket has two goals for Pig unit:
> 1) Pig unit has a really nice method assertOutput(String inputAlias, String[] inputValues, String
outputAlias, String[] expectedOutputValues).  That method lets you override an input alias variable
with a hardcoded list of values. That way, the script doesn't actually have to read that input variable
from hdfs or cassandra. Then, it runs the script and checks the specified output alias variable against
the expected set of values.  It's a really nice way to test your entire pig script with a single method call,
but only IF your script has exactly 1 input and 1 output.  If you want to test more complicated scripts, you
(Continue reading)

Richard So (JIRA | 18 Nov 02:56 2014
Picon

[Updated] (PIG-2692) Make the Pig unit faciliities more generalizable and update javadocs


     [
https://issues.apache.org/jira/browse/PIG-2692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Richard So updated PIG-2692:
----------------------------
    Attachment:     (was: pig2692.patch)

> Make the Pig unit faciliities more generalizable and update javadocs
> --------------------------------------------------------------------
>
>                 Key: PIG-2692
>                 URL: https://issues.apache.org/jira/browse/PIG-2692
>             Project: Pig
>          Issue Type: Improvement
>            Reporter: Jeremy Hanna
>            Assignee: Richard So
>            Priority: Minor
>             Fix For: 0.15.0
>
>         Attachments: pig2692.patch
>
>
> This ticket has two goals for Pig unit:
> 1) Pig unit has a really nice method assertOutput(String inputAlias, String[] inputValues, String
outputAlias, String[] expectedOutputValues).  That method lets you override an input alias variable
with a hardcoded list of values. That way, the script doesn't actually have to read that input variable
from hdfs or cassandra. Then, it runs the script and checks the specified output alias variable against
the expected set of values.  It's a really nice way to test your entire pig script with a single method call,
but only IF your script has exactly 1 input and 1 output.  If you want to test more complicated scripts, you
(Continue reading)


Gmane