Josh Elser (JIRA | 1 Aug 04:28 2014
Picon

[Updated] (PIG-4083) TestAccumuloPigCluster always failed with timeout error


     [
https://issues.apache.org/jira/browse/PIG-4083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Josh Elser updated PIG-4083:
----------------------------

    Attachment: PIG-4083-debug.patch

Ok, [~fang fang chen]. You can apply this using {{patch -p1 PIG-4083-debug.patch}}.

Then, run just the testcase {{ant test -Dtestcase=TestAccumuloPigCluster}}.

After, please attach {{build/test/logs/TEST-org.apache.pig.backend.hadoop.accumulo.TestAccumuloPigCluster.txt}}.

Also, in that same log file, you will also see a line that matches {{INFO 
org.apache.pig.backend.hadoop.accumulo.TestAccumuloPigCluster  - Starting MiniAccumuloCluster
in ...}}, where {{...}} is some directory on your local filesystem. That directory is where the
MiniAccumuloCluster was started from. Please attach the contents of the {{logs}} directory beneath the
temporary directory path, as well.

Those two logs should help me better understand why this test was failing for you. Thanks.

> TestAccumuloPigCluster always failed with timeout error
> -------------------------------------------------------
>
>                 Key: PIG-4083
>                 URL: https://issues.apache.org/jira/browse/PIG-4083
>             Project: Pig
>          Issue Type: Bug
(Continue reading)

Josh Elser (JIRA | 1 Aug 04:10 2014
Picon

[Commented] (PIG-4083) TestAccumuloPigCluster always failed with timeout error


    [
https://issues.apache.org/jira/browse/PIG-4083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14081823#comment-14081823
] 

Josh Elser commented on PIG-4083:
---------------------------------

Sounds good, I'll get a patch with some extra debugging here for you. Out of curiosity, does it fail quickly?

> TestAccumuloPigCluster always failed with timeout error
> -------------------------------------------------------
>
>                 Key: PIG-4083
>                 URL: https://issues.apache.org/jira/browse/PIG-4083
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.13.0
>            Reporter: fang fang chen
>            Assignee: Josh Elser
>            Priority: Critical
>
> TestAccumuloPigCluster always failed with timeout error.
> Tried with sun jdk 6 and sun jdk 7.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

(Continue reading)

fang fang chen (JIRA | 1 Aug 04:06 2014
Picon

[Commented] (PIG-4083) TestAccumuloPigCluster always failed with timeout error


    [
https://issues.apache.org/jira/browse/PIG-4083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14081817#comment-14081817
] 

fang fang chen commented on PIG-4083:
-------------------------------------

BTW, I was uring sun jdk 1.7.0_60/1.6.0_45 and ibm jdk 1.6.0/1.7.0. All failed. If this is caused by
environment, I want to know what caused this issue and how to resolve. This would be helpful if pig can
provide this information. Thanks.

> TestAccumuloPigCluster always failed with timeout error
> -------------------------------------------------------
>
>                 Key: PIG-4083
>                 URL: https://issues.apache.org/jira/browse/PIG-4083
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.13.0
>            Reporter: fang fang chen
>            Assignee: Josh Elser
>            Priority: Critical
>
> TestAccumuloPigCluster always failed with timeout error.
> Tried with sun jdk 6 and sun jdk 7.

--
This message was sent by Atlassian JIRA
(v6.2#6252)
(Continue reading)

fang fang chen (JIRA | 1 Aug 04:02 2014
Picon

[Commented] (PIG-4083) TestAccumuloPigCluster always failed with timeout error


    [
https://issues.apache.org/jira/browse/PIG-4083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14081814#comment-14081814
] 

fang fang chen commented on PIG-4083:
-------------------------------------

Hi Josh, 
For your 1# comment:
Here is all the output from log file. I did not find any useful information for debug.
Testcase: test took 0.001 sec
	Caused an ERROR
Timeout occurred. Please note the time in the report does not reflect the time until the timeout.
junit.framework.AssertionFailedError: Timeout occurred. Please note the time in the report does not
reflect the time until the timeout.

For your 2# comment:
Yes, please provide the quick patch for debugging. Thanks

> TestAccumuloPigCluster always failed with timeout error
> -------------------------------------------------------
>
>                 Key: PIG-4083
>                 URL: https://issues.apache.org/jira/browse/PIG-4083
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.13.0
>            Reporter: fang fang chen
>            Assignee: Josh Elser
(Continue reading)

jira | 1 Aug 03:04 2014
Picon

Subscription: PIG patch available

Issue Subscription
Filter: PIG patch available (14 issues)

Subscriber: pigdaily

Key         Summary
PIG-4066    An optimization for ROLLUP operation in Pig
            https://issues.apache.org/jira/browse/PIG-4066
PIG-4008    Pig code change to enable Tez Local mode 
            https://issues.apache.org/jira/browse/PIG-4008
PIG-4004    Upgrade the Pigmix queries from the (old) mapred API to mapreduce
            https://issues.apache.org/jira/browse/PIG-4004
PIG-4002    Disable combiner when map-side aggregation is used
            https://issues.apache.org/jira/browse/PIG-4002
PIG-3952    PigStorage accepts '-tagSplit' to return full split information
            https://issues.apache.org/jira/browse/PIG-3952
PIG-3911    Define unique fields with  <at> OutputSchema
            https://issues.apache.org/jira/browse/PIG-3911
PIG-3877    Getting Geo Latitude/Longitude from Address Lines
            https://issues.apache.org/jira/browse/PIG-3877
PIG-3873    Geo distance calculation using Haversine
            https://issues.apache.org/jira/browse/PIG-3873
PIG-3866    Create ThreadLocal classloader per PigContext
            https://issues.apache.org/jira/browse/PIG-3866
PIG-3861    duplicate jars get added to distributed cache
            https://issues.apache.org/jira/browse/PIG-3861
PIG-3668    COR built-in function when atleast one of the coefficient values is NaN
            https://issues.apache.org/jira/browse/PIG-3668
PIG-3635    Fix e2e tests for Hadoop 2.X on Windows
            https://issues.apache.org/jira/browse/PIG-3635
(Continue reading)

Rohini Palaniswamy (JIRA | 1 Aug 02:07 2014
Picon

[Commented] (PIG-3760) Predicate pushdown for columnar file formats


    [
https://issues.apache.org/jira/browse/PIG-3760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14081715#comment-14081715
] 

Rohini Palaniswamy commented on PIG-3760:
-----------------------------------------

Attached initial patch with PIG-4091 with basic functionality required of Predicate Pushdown
interface. The interface needs some more enhancements. Filed PIG-4093 and PIG-4094 for that. 

[~julienledem]/ [~dvryaboy],
     Is there someone in Twitter that we can work with for the Parquet implementation? It would help us flush out
and finalize the APIs. 

> Predicate pushdown for columnar file formats
> --------------------------------------------
>
>                 Key: PIG-3760
>                 URL: https://issues.apache.org/jira/browse/PIG-3760
>             Project: Pig
>          Issue Type: New Feature
>            Reporter: Andrew Musselman
>             Fix For: 0.14.0
>
>
> From the conversation on dev <at> pig:
> "Partition pruning for ORC is not addressed in PIG-3558. We will need
> to do partition pruning for both ORC and Parquet in a new ticket.
> Curently there is no interface to deal with this kind of pushdown
(Continue reading)

Rohini Palaniswamy (JIRA | 1 Aug 02:05 2014
Picon

[Commented] (PIG-4091) Predicate pushdown for ORC


    [
https://issues.apache.org/jira/browse/PIG-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14081711#comment-14081711
] 

Rohini Palaniswamy commented on PIG-4091:
-----------------------------------------

Attached initial patch. Still has some pending TODOs
   - Add e2e tests
   - Add tests for datatypes - boolean, byte, short, biginteger, bigdecimal, datetime

LoadPredicatePushdown interface needs some more enhancements. Filed PIG-4093 and PIG-4094 for that. 

> Predicate pushdown for ORC
> --------------------------
>
>                 Key: PIG-4091
>                 URL: https://issues.apache.org/jira/browse/PIG-4091
>             Project: Pig
>          Issue Type: Sub-task
>            Reporter: Rohini Palaniswamy
>             Fix For: 0.14.0
>
>         Attachments: PIG-3760-initial.patch
>
>

--
This message was sent by Atlassian JIRA
(Continue reading)

Rohini Palaniswamy (JIRA | 1 Aug 02:01 2014
Picon

[Created] (PIG-4095) Collapse multiple OR conditions to IN and BETWEEN

Rohini Palaniswamy created PIG-4095:
---------------------------------------

             Summary: Collapse multiple OR conditions to IN and BETWEEN
                 Key: PIG-4095
                 URL: https://issues.apache.org/jira/browse/PIG-4095
             Project: Pig
          Issue Type: Sub-task
            Reporter: Rohini Palaniswamy

  ORC predicate pushdown supports IN and BETWEEN operators. Need equivalent expressions in Pig.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

Rohini Palaniswamy (JIRA | 1 Aug 02:01 2014
Picon

[Updated] (PIG-4091) Predicate pushdown for ORC


     [
https://issues.apache.org/jira/browse/PIG-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rohini Palaniswamy updated PIG-4091:
------------------------------------

    Attachment: PIG-3760-initial.patch

> Predicate pushdown for ORC
> --------------------------
>
>                 Key: PIG-4091
>                 URL: https://issues.apache.org/jira/browse/PIG-4091
>             Project: Pig
>          Issue Type: Sub-task
>            Reporter: Rohini Palaniswamy
>             Fix For: 0.14.0
>
>         Attachments: PIG-3760-initial.patch
>
>

--
This message was sent by Atlassian JIRA
(v6.2#6252)

Rohini Palaniswamy (JIRA | 1 Aug 01:59 2014
Picon

[Created] (PIG-4094) Predicate pushdown to support complex data types

Rohini Palaniswamy created PIG-4094:
---------------------------------------

             Summary: Predicate pushdown to support complex data types
                 Key: PIG-4094
                 URL: https://issues.apache.org/jira/browse/PIG-4094
             Project: Pig
          Issue Type: Sub-task
            Reporter: Rohini Palaniswamy
             Fix For: 0.14.0

  Parquet has support for pushing predicates on tuples, maps and bags according to [~aniket486]. ORC
currently only supports primitives, but will add support for structs(tuples) in the future.  The API
needs to be there even if not implemented as it will hard to change the interface once released.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

Rohini Palaniswamy (JIRA | 1 Aug 01:57 2014
Picon

[Created] (PIG-4093) Predicate pushdown to support removing filters from pig plan

Rohini Palaniswamy created PIG-4093:
---------------------------------------

             Summary: Predicate pushdown to support removing filters from pig plan
                 Key: PIG-4093
                 URL: https://issues.apache.org/jira/browse/PIG-4093
             Project: Pig
          Issue Type: Sub-task
            Reporter: Rohini Palaniswamy

   It is possible for the loaders to evaluate the pushed filter conditions. In that case it is not necessary to
retain the filter conditions in the pig plan. So need to support two modes :
    1) filter conditions are pushed into loader but also retained in pig plan as loader might do only best effort
filtering based on block metadata
    2) filter conditions are pushed into loader and removed from pig plan when the loader can evaluate the
expression itself and filter out records. In this case, loader can do lazy deserialization adn avoid
deserialization of the full record.

--
This message was sent by Atlassian JIRA
(v6.2#6252)


Gmane