jira | 24 Jul 08:00 2016
Picon

Subscription: PIG patch available

Issue Subscription
Filter: PIG patch available (29 issues)

Subscriber: pigdaily

Key         Summary
PIG-4957    See "Received kill signal" message for a normal run after PIG-4921
            https://issues.apache.org/jira/browse/PIG-4957
PIG-4935    TEZ_USE_CLUSTER_HADOOP_LIBS is always set to true
            https://issues.apache.org/jira/browse/PIG-4935
PIG-4926    Modify the content of start.xml for spark mode
            https://issues.apache.org/jira/browse/PIG-4926
PIG-4922    Deadlock between SpillableMemoryManager and InternalSortedBag$SortedDataBagIterator
            https://issues.apache.org/jira/browse/PIG-4922
PIG-4918    Pig on Tez cannot switch pig.temp.dir to another fs
            https://issues.apache.org/jira/browse/PIG-4918
PIG-4897    Scope of param substitution for run/exec commands
            https://issues.apache.org/jira/browse/PIG-4897
PIG-4886    Add PigSplit#getLocationInfo to fix the NPE found in log in spark mode
            https://issues.apache.org/jira/browse/PIG-4886
PIG-4854    Merge spark branch to trunk
            https://issues.apache.org/jira/browse/PIG-4854
PIG-4849    pig on tez will cause tez-ui to crash,because the content from timeline server is too long. 
            https://issues.apache.org/jira/browse/PIG-4849
PIG-4788    the value BytesRead metric info always returns 0 even the length of input file is not 0 in spark engine
            https://issues.apache.org/jira/browse/PIG-4788
PIG-4745    DataBag should protect content of passed list of tuples
            https://issues.apache.org/jira/browse/PIG-4745
PIG-4684    Exception should be changed to warning when job diagnostics cannot be fetched
            https://issues.apache.org/jira/browse/PIG-4684
(Continue reading)

Rohini Palaniswamy (JIRA | 24 Jul 05:37 2016
Picon

[Updated] (PIG-4957) See "Received kill signal" message for a normal run after PIG-4921


     [
https://issues.apache.org/jira/browse/PIG-4957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rohini Palaniswamy updated PIG-4957:
------------------------------------
    Assignee: Rohini Palaniswamy
      Status: Patch Available  (was: Open)

> See "Received kill signal" message for a normal run after PIG-4921
> ------------------------------------------------------------------
>
>                 Key: PIG-4957
>                 URL: https://issues.apache.org/jira/browse/PIG-4957
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Daniel Dai
>            Assignee: Rohini Palaniswamy
>             Fix For: 0.17.0
>
>         Attachments: PIG-4957-1.patch
>
>
> See message in almost every job:
> {code}
> INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher -
Received kill signal
> {code}
> This needs to be fixed.

(Continue reading)

Rohini Palaniswamy (JIRA | 24 Jul 05:36 2016
Picon

[Updated] (PIG-4957) See "Received kill signal" message for a normal run after PIG-4921


     [
https://issues.apache.org/jira/browse/PIG-4957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rohini Palaniswamy updated PIG-4957:
------------------------------------
    Attachment: PIG-4957-1.patch

> See "Received kill signal" message for a normal run after PIG-4921
> ------------------------------------------------------------------
>
>                 Key: PIG-4957
>                 URL: https://issues.apache.org/jira/browse/PIG-4957
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Daniel Dai
>             Fix For: 0.17.0
>
>         Attachments: PIG-4957-1.patch
>
>
> See message in almost every job:
> {code}
> INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher -
Received kill signal
> {code}
> This needs to be fixed.

--
This message was sent by Atlassian JIRA
(Continue reading)

Vlad Korolev (JIRA | 24 Jul 00:42 2016
Picon

[Commented] (PIG-2599) Mavenize Pig


    [
https://issues.apache.org/jira/browse/PIG-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15390861#comment-15390861
] 

Vlad Korolev commented on PIG-2599:
-----------------------------------

Ok,

    I started to work on this.  Here is what I got so far  https://github.com/vladistan/pig/tree/gradle-conv  
It is only primary steps.  My approach is to import the Ant build script into Gradle and slowly convert tasks
one-by-one.  Also,  I am planning to keep build.xml which will simply forward all the tasks into Gradle. 
This is the least intrusive way and all the old CI servers and other infrastructure will not break.  

   At this point I am looking for initial critique of the approach and suggestions.

   Reason to choose Gradle over maven is that the existing Ant script is quite complex for Maven's simple
declarative approach.  Gradle is as flexible as Ant so there will not be a problem in this regard.

> Mavenize Pig
> ------------
>
>                 Key: PIG-2599
>                 URL: https://issues.apache.org/jira/browse/PIG-2599
>             Project: Pig
>          Issue Type: New Feature
>          Components: build
>            Reporter: Daniel Dai
>            Assignee: Vimuth Fernando
(Continue reading)

jira | 23 Jul 08:00 2016
Picon

Subscription: PIG patch available

Issue Subscription
Filter: PIG patch available (28 issues)

Subscriber: pigdaily

Key         Summary
PIG-4935    TEZ_USE_CLUSTER_HADOOP_LIBS is always set to true
            https://issues.apache.org/jira/browse/PIG-4935
PIG-4926    Modify the content of start.xml for spark mode
            https://issues.apache.org/jira/browse/PIG-4926
PIG-4922    Deadlock between SpillableMemoryManager and InternalSortedBag$SortedDataBagIterator
            https://issues.apache.org/jira/browse/PIG-4922
PIG-4918    Pig on Tez cannot switch pig.temp.dir to another fs
            https://issues.apache.org/jira/browse/PIG-4918
PIG-4897    Scope of param substitution for run/exec commands
            https://issues.apache.org/jira/browse/PIG-4897
PIG-4886    Add PigSplit#getLocationInfo to fix the NPE found in log in spark mode
            https://issues.apache.org/jira/browse/PIG-4886
PIG-4854    Merge spark branch to trunk
            https://issues.apache.org/jira/browse/PIG-4854
PIG-4849    pig on tez will cause tez-ui to crash,because the content from timeline server is too long. 
            https://issues.apache.org/jira/browse/PIG-4849
PIG-4788    the value BytesRead metric info always returns 0 even the length of input file is not 0 in spark engine
            https://issues.apache.org/jira/browse/PIG-4788
PIG-4745    DataBag should protect content of passed list of tuples
            https://issues.apache.org/jira/browse/PIG-4745
PIG-4684    Exception should be changed to warning when job diagnostics cannot be fetched
            https://issues.apache.org/jira/browse/PIG-4684
PIG-4656    Improve String serialization and comparator performance in BinInterSedes
            https://issues.apache.org/jira/browse/PIG-4656
(Continue reading)

Daniel Dai (JIRA | 23 Jul 02:22 2016
Picon

[Commented] (PIG-4957) See "Received kill signal" message for a normal run after PIG-4921


    [
https://issues.apache.org/jira/browse/PIG-4957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15390402#comment-15390402
] 

Daniel Dai commented on PIG-4957:
---------------------------------

It happens in MR job, does not sound like PIG-4928.

> See "Received kill signal" message for a normal run after PIG-4921
> ------------------------------------------------------------------
>
>                 Key: PIG-4957
>                 URL: https://issues.apache.org/jira/browse/PIG-4957
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Daniel Dai
>             Fix For: 0.17.0
>
>
> See message in almost every job:
> {code}
> INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher -
Received kill signal
> {code}
> This needs to be fixed.

--
This message was sent by Atlassian JIRA
(Continue reading)

Rohini Palaniswamy (JIRA | 23 Jul 02:22 2016
Picon

[Commented] (PIG-4957) See "Received kill signal" message for a normal run after PIG-4921


    [
https://issues.apache.org/jira/browse/PIG-4957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15390403#comment-15390403
] 

Rohini Palaniswamy commented on PIG-4957:
-----------------------------------------

Realized that in mapreduce mode, it is always going to log even if there are no running jobs. We can use this
jira to log the message if jc!=null && !jc.getRunningJobs().isEmpty() . PIG-4928 can be used to fix the
MRExecutionEngine instantiation problem in Tez.

> See "Received kill signal" message for a normal run after PIG-4921
> ------------------------------------------------------------------
>
>                 Key: PIG-4957
>                 URL: https://issues.apache.org/jira/browse/PIG-4957
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Daniel Dai
>             Fix For: 0.17.0
>
>
> See message in almost every job:
> {code}
> INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher -
Received kill signal
> {code}
> This needs to be fixed.

(Continue reading)

Rohini Palaniswamy (JIRA | 23 Jul 02:17 2016
Picon

[Commented] (PIG-4957) See "Received kill signal" message for a normal run after PIG-4921


    [
https://issues.apache.org/jira/browse/PIG-4957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15390394#comment-15390394
] 

Rohini Palaniswamy commented on PIG-4957:
-----------------------------------------

Dupe of PIG-4928 ?

> See "Received kill signal" message for a normal run after PIG-4921
> ------------------------------------------------------------------
>
>                 Key: PIG-4957
>                 URL: https://issues.apache.org/jira/browse/PIG-4957
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Daniel Dai
>             Fix For: 0.17.0
>
>
> See message in almost every job:
> {code}
> INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher -
Received kill signal
> {code}
> This needs to be fixed.

--
This message was sent by Atlassian JIRA
(Continue reading)

Daniel Dai (JIRA | 22 Jul 23:44 2016
Picon

[Commented] (PIG-4921) Kill running jobs on InterruptedException


    [
https://issues.apache.org/jira/browse/PIG-4921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15390244#comment-15390244
] 

Daniel Dai commented on PIG-4921:
---------------------------------

See lots of "Received kill signal" message after the patch. Created PIG-4957 for that.

> Kill running jobs on InterruptedException
> -----------------------------------------
>
>                 Key: PIG-4921
>                 URL: https://issues.apache.org/jira/browse/PIG-4921
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Rohini Palaniswamy
>            Assignee: Rohini Palaniswamy
>             Fix For: 0.17.0
>
>         Attachments: PIG-4921-1.patch, PIG-4921-2.patch, PIG-4921-3.patch
>
>
> Shutdown hook kills running jobs, but when running in Oozie launcher sometimes NodeManager can issue a
SIGKILL after AM unregisters and before shutdown hook gets to execute causing orphaned jobs that
continue to run. So it is better to kill when we see the InterruptedException.
> Added additional System.err logging to code in shutdown hooks as LogManager.shutdown() is called by
mapreduce AM (Oozie launcher job is mapreduce) and log4j logging does not appear anymore.  

(Continue reading)

Daniel Dai (JIRA | 22 Jul 23:43 2016
Picon

[Updated] (PIG-4957) See "Received kill signal" message for a normal run after PIG-4921


     [
https://issues.apache.org/jira/browse/PIG-4957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-4957:
----------------------------
    Summary: See "Received kill signal" message for a normal run after PIG-4921  (was: See "Received kill
signal" message for a normal run after pig-4921)

> See "Received kill signal" message for a normal run after PIG-4921
> ------------------------------------------------------------------
>
>                 Key: PIG-4957
>                 URL: https://issues.apache.org/jira/browse/PIG-4957
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Daniel Dai
>             Fix For: 0.17.0
>
>
> See message in almost every job:
> {code}
> INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher -
Received kill signal
> {code}
> This needs to be fixed.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
(Continue reading)

Daniel Dai (JIRA | 22 Jul 23:43 2016
Picon

[Created] (PIG-4957) See "Received kill signal" message for a normal run after pig-4921

Daniel Dai created PIG-4957:
-------------------------------

             Summary: See "Received kill signal" message for a normal run after pig-4921
                 Key: PIG-4957
                 URL: https://issues.apache.org/jira/browse/PIG-4957
             Project: Pig
          Issue Type: Bug
            Reporter: Daniel Dai
             Fix For: 0.17.0

See message in almost every job:
{code}
INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher -
Received kill signal
{code}

This needs to be fixed.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Gmane