Tim Otten | 6 May 08:47 2016
Gravatar

Invitation: CiviCRM Release for June 1st (4.7.8)

As with the previous release-cycle (May 5; v4.7.7), we’re aiming to do more publicly organized peer-review. A full invitation has been sent via Github to developers with open PRs ( https://github.com/civicrm/release-management/issues/2 ), but participation is open to any developer who wants to get issues closed, make Civi a better product, and earn karma.

To participate, please add your name and preferences to the spreadsheet:

https://docs.google.com/spreadsheets/d/10EyNqm3-CbAwUjYzckrwSE7VjpZCfatzh-bES59XqA8/edit?usp=sharing

Thanks.


P.S. If you paid attention to the previous release-cycle, we had some consideration about whether reviewers should list the specific PRs or just indicate a general topic/working-group. Topical working groups are still ideal, but a couple people had trouble with the concept, and it’s good to experiment a little as we tune our ladder of engagement. So for this cycle, I adjusted the instructions try to accommodate both forms of engagement.

P.P.S. The release date for this cycle will come just before CiviCon, so you may want to factor that into your planning. On the upside, the Colorado sprint aligns with the start of the next cycle, so we should have a lot of freedom at the sprint. :)
____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev
Alan Dixon | 5 May 16:55 2016
Picon
Gravatar

A full stack problem to solve ...

A few weeks ago, we started noticing that Firefox on a Mac would present a 'Content Encoding Error' on apparently random-ish CiviCRM administration pages. They would usually be solved by reloading the page.

After running after a few white rabbits, we discovered that it was unrelated to most of the issues that the symptom might suggest [like, something related to content encoding ...], and that in fact what was happening in each case (okay, at least as far as we could tell) was a 404 on one of the page's php-generated elements, corresponding to an apparent php segfault showing up in the server logs.

If that doesn't make sense, then glad to see you're paying attention.

1. What does that generate a content encoding error?

I think that's just Firefox being stupid. It might be related to how apache hands back the response from the segfault, esp. via php-fpm. But ultimately, not my biggest concern.

2. Why are we getting segfaults that are only happening with specific client platforms?

The 404s look like they might all be angular urls, which may be behaving differently on different platforms (i.e. angular is clever enough to be doing different ajax calls for different browsers). Or maybe we're only noticing them on the platforms that handle them badly? Pretty sure no - grepping the error logs suggests they're only happening on the specific client platform (mac/firefox).

3. Why is it random-ish and non-reproducible?

Actually, there are a few specific cases where we've been able to reproduce it reliably, but it seems like there might be a caching thing that prevents it from being reliable.

Other contextual bits:

1. All Drupal 7 (well, that's because that pretty much all we're working on)
2. Mostly happening on a civi 4.7 site, but occasionally on a 4.6 site as well
3. CentOS 6 with apache 2.2 + php-fpm
4. Recently updated from php53 to ius php56u.

Another clue:

I increased the number of php-fpm children allowed and set a timeout and that seemed to improve, but not completely fix the issue. That suggests that a php/system issue related to memory handling.

I haven't found anything helpful on google (yeah, don't bother googling firefox's content encoding error).

Anybody got any ideas or similar experiences?

 - Alan



--
Alan Dixon, Web Developer

Drupal and CiviCRM for Canadian nonprofits.
http://blackflysolutions.ca/
____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev
Frank J. Gómez | 3 May 17:16 2016
Gravatar

are there security concerns around granting "access AJAX API?"

Hello,

What is the purpose of the "access AJAX API" permission? As I understand it, this permission (or "access CiviCRM") is required for accessing the API via the client side. Accordingly, one of these two permissions is going to be needed for just about any Angular UI.

In the case of public Angular UIs -- UIs which we desire to be accessible to users who don't have "access CiviCRM" -- there is no choice but to assign the "access AJAX API" permission to the relevant role (perhaps even to anonymous users). Is there a reason not to do this?

According to the API Security wiki article (which seems to have as many questions as answers, FWIW):
  • each API (or underlying BAO) is supposed to control access with its own permission checks, and
  • client-side calls are made with the check_permissions flag set, so the above-mentioned checks cannot be bypassed like they can be on the server side.
Therefore, granting the "access AJAX API" permission to anonymous users seems to be mitigated by the more granular entity-specific permissions.

So, playing devil's advocate, why do we need this permission at all? The facts above seem to suggest it is redundant, unless there's some reason I'm not thinking of to make the AJAX API completely inaccessible to certain classes of users. If removing the permission is too radical a proposition, why shouldn't it be granted to anonymous users by default (like "Access all custom data" and a handful of others are)?

Thanks in advance for sharing your thoughts.

PS: Please forgive the cross-posting; I wasn't sure whether this topic was appropriate for the dev list, the API list, or both.

--
Frank J. Gómez
Developer & Principal
Ginkgo Street Labs 
2121 Decatur Pl NW, Washington, DC 20008
toll free: 888-223-6609

Please send requests for technical support to support <at> ginkgostreet.com for the most expeditious response.
____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev
Nicolas Ganivet | 30 Apr 22:06 2016
Gravatar

Settings changes in 4.7 ???

ALl,
 
GinkgoFJG tells me: 'As of CiviCRM 4.7, settings groups are toast. Each setting must have a unique name.'
 
I am not quite understanding this change:
  - https://github.com/civicrm/civicrm-core/blob/master/xml/schema/Core/Setting.xml still indicates that the column Group exists in table civicrm_setting
  - https://github.com/civicrm/civicrm-core/blob/master/CRM/Core/BAO/Setting.php still has all group name constant, and same signature functions
 
What was the reason for this change? Where is it documented?
 
This is a very significant change and will impact a lot of code outside of core - including many extensions and most probably a lot of code that interacted through the API.

Can anyone share how they managed this transition for extensions for example? Did you rename all your settings in order to avoid potential collisions? How did you do so without requiring users to go through all setup screens again?
 
Is there a way this could be done transparently through the CRM_Core_BAO_Setting functions so we do not have to go through painful changes?
 
Thanks,
--Nicolas.
____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev
Tim Otten | 28 Apr 03:04 2016
Gravatar

PSA: Updating RC

We’ve had a couple PR’s to fix regression/integrity issues in the RC. A newer RC is posted at:

http://dist.civicrm.org/by-date/2016-04-27/4.7.7-rc/

If you’ve already tested, thank you. :) 

If you plan to do new or additional testing, please use the newer tarball.

If you’re in the middle of testing, please use your own judgment about whether to update. A list of changes
can be found below. (Purple items are merged. Red items are cancelled.)

https://github.com/civicrm/civicrm-core/pulls?q=is%3Apr+label%3A4.7.7-rc+is%3Aclosed
https://github.com/civicrm/civicrm-drupal/pulls?utf8=%E2%9C%93&q=is%3Apr+label%3A7.x-4.7.7-rc+is%3Aclosed+
Jamie McClelland | 27 Apr 15:04 2016

Re: Forced dedupes

Hi all - I just pushed out an extension we developed that creates a
series of custom rules to manage batch merging here:

https://github.com/progressivetech/net.ourpowerbase.merge

It's not really fit for public release, but might provide some ideas to
others.

jamie

On Wed Apr 27, Eileen McNaughton wrote:
> I had a go at a sprint board ...
> 
> 
> https://issues.civicrm.org/jira/secure/RapidBoard.jspa?rapidView=20&view=detail
> 
> 
> Note that the working group idea is around QAing prs into core and it might
> take a couple of cycles to work through stuff. We seem to have a strong
> overlap with people who are interested in addressing group cache clearing
> issues....
> 
> Eileen
> 
> On 25/04/2016 11:00 p.m., Parvez Saleh wrote:
> >Hey John
> >
> >We've done a fair amount of work on automated de-dupe processes for some
> >of our larger clients, would be happy to partake in the group.
> >
> >
> >
> >Veda Consulting
> >t : +44 (0) 20 3239 1156
> >m : +44 (0) 77 4216 3491
> >e : parvez <at> vedaconsulting.co.uk <mailto:parvez <at> vedaconsulting.co.uk>
> >w : www.vedaconsulting.co.uk <http://www.vedaconsulting.co.uk>
> >
> >
> >On 25 April 2016 at 11:27, John Kingsnorth <john <at> johnkingsnorth.co.uk
> ><mailto:john <at> johnkingsnorth.co.uk>> wrote:
> >
> >    Hello all,
> >
> >    Whilst working on fixing the dedupe tests I came across the
> >    'force' merge logic again. I thought it would be good to check
> >    that it works 'as expected' and perhaps spark a conversation about
> >    how it could be improved.
> >
> >    Currently forcing a merge will add any new information from the
> >    'other' contact on to the 'main' contact. Any emails, phones etc.
> >    are also 'added' - no overwrites. If there are any conflicts on
> >    the records (differing data on the same field) then the
> >    information on the 'main' contact is preserved.
> >
> >    There's an example here:
> >    https://issues.civicrm.org/jira/browse/CRM-18338
> >
> >    There's also a ticket about getting more control in the dedupe API:
> >    https://issues.civicrm.org/jira/browse/CRM-12421
> >
> >    However, I think the 'force' option could be cleverer about how it
> >    handles merges - for example taking into account which contact was
> >    'last updated' most recently; and somehow reducing the number of
> >    duplicate phones/emails with the same type when merging -
> >    potentially by overwriting rather than adding them.
> >
> >    Eileen suggested a working group on this for the next release
> >    cycle - so if anyone else is heavily into the 'deduping'
> >    functionality of Civi then shout and maybe we can take a look at
> >    this in more detail. Obviously changes in this area are
> >    potentially quite dangerous/impactful so that's another reason for
> >    reaching out.
> >
> >    All the best,
> >
> >    John
> >    (JKingsnorth)
> >    ____________________________________________________________
> >    You received this message as a subscriber on the list:
> >    civicrm-dev <at> lists.civicrm.org <mailto:civicrm-dev <at> lists.civicrm.org>
> >    To be removed from the list, send any message to:
> >    civicrm-dev-unsubscribe <at> lists.civicrm.org
> >    <mailto:civicrm-dev-unsubscribe <at> lists.civicrm.org>
> >
> >    For all list information and functions, see:
> >    http://lists.civicrm.org/lists/info/civicrm-dev
> >
> >
> >____________________________________________________________
> >You received this message as a subscriber on the list:
> >civicrm-dev <at> lists.civicrm.org <mailto:civicrm-dev <at> lists.civicrm.org>
> >To be removed from the list, send any message to:
> >civicrm-dev-unsubscribe <at> lists.civicrm.org
> ><mailto:civicrm-dev-unsubscribe <at> lists.civicrm.org>
> >
> >For all list information and functions, see:
> >http://lists.civicrm.org/lists/info/civicrm-dev
> 
> 
> 
> ---
> This email has been checked for viruses by Avast antivirus software.
> https://www.avast.com/antivirus
> ____________________________________________________________
> You received this message as a subscriber on the list:
>     civicrm-dev <at> lists.civicrm.org
> To be removed from the list, send any message to:
>     civicrm-dev-unsubscribe <at> lists.civicrm.org
> 
> For all list information and functions, see:
>     http://lists.civicrm.org/lists/info/civicrm-dev

--

-- 
Jamie McClelland
612.724.2600 ext 700
Progressive Technology Project
http://progressivetech.org/

For Powerbase Support Questions, write to mailto:support <at> progressivetech.org
or call 512-782-8478

OpenPGP Key: http://current.workingdirectory.net/pages/identity/ 
Eileen McNaughton | 26 Apr 04:55 2016

Mysql 5.7

Hi,

I'm aware Andrew Perry & Brian have both been using mysql 5.7. I'm 
considering advocating it and would like to know whether you have any 
comments about it.

I am aware that I need a version with this fix in it 
https://github.com/civicrm/civicrm-core/commit/8bf889ef6b0e3c1d64deb6e222ef56ed8f51c2e8 
- which the current RCs do and that there are some tickets open to try 
to take advantage of new features.

Are there any outstanding problems with it?

Eileen

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Joe Murray | 25 Apr 18:45 2016
Gravatar

develStage options?

Tim, I believe you had discussed possible changes to extension statuses. I'd like to deprecate support for the Mandrill extension, or indicate that we are seeking co-maintainer. However, the documentation at https://wiki.civicrm.org/confluence/display/CRMDOC/Extension+Reference doesn't indicate a way to do either. Suggestions, other than putting text into the extension description?

Joe Murray, PhD
President, JMA Consulting
joe.murray <at> jmaconsulting.biz
skype JosephPMurray twitter JoeMurray
416.466.1281
____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev
John Kingsnorth | 25 Apr 12:27 2016
Picon
Gravatar

Forced dedupes

Hello all,

Whilst working on fixing the dedupe tests I came across the 'force' merge logic again. I thought it would be good to check that it works 'as expected' and perhaps spark a conversation about how it could be improved.

Currently forcing a merge will add any new information from the 'other' contact on to the 'main' contact. Any emails, phones etc. are also 'added' - no overwrites. If there are any conflicts on the records (differing data on the same field) then the information on the 'main' contact is preserved.

There's an example here:

There's also a ticket about getting more control in the dedupe API:

However, I think the 'force' option could be cleverer about how it handles merges - for example taking into account which contact was 'last updated' most recently; and somehow reducing the number of duplicate phones/emails with the same type when merging - potentially by overwriting rather than adding them.

Eileen suggested a working group on this for the next release cycle - so if anyone else is heavily into the 'deduping' functionality of Civi then shout and maybe we can take a look at this in more detail. Obviously changes in this area are potentially quite dangerous/impactful so that's another reason for reaching out.

All the best,

John
(JKingsnorth)
____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev
Tim Otten | 23 Apr 13:32 2016
Gravatar

v4.7.7 Release Candidate

Thank you to the folks who participated in April’s Review Week. The release candidate is now available at:


From now until the end of next weekend (Sunday, May 1), we’ll be in Test Week. This is a good opportunity to do a trial-run of the new release on your staging systems. Regressions from v4.7.6 will be given particularly high priority.

The final GA release is set to go out on Wednesday, May 4. To ensure that there is sufficient time to fix any major new issues, please try to finish testing several days before that.

== Changes ==

There have been 93 pull-requests merged into v4.7.7, which have includes a few security-oriented changes. Among the changes, it may be particularly interesting to spot-check some of your typical use-cases in these areas:

1. Fine-grained revision logging and database triggers, especially if you have customizations or external SQL processes which interact with these.

2. Custom data, especially if you target custom-data by subtype.

It’s difficult for me to do justice to all of the issues (especially within the space of an email). For a list of actual changes addressed in the RC, see the “Changelog” at:


If you think that some topics are particularly notable and merit greater awareness, please feel free to put a flag on the relevant rows in the changelog or post to this thread.

== Git Administrativa ==

In all the major CiviCRM repos (civicrm-core.git, civicrm-drupal.get, et al), there is a new branch named “4.7.7-rc”. If there is a notable regression from 4.7.6, please file the PR against 4.7.7-rc and mention " <at> totten”. Otherwise, existing PRs and new PRs should continue to go to “master”.

Thanks and good night,
Tim
____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev
Guy Iaccarino | 22 Apr 16:23 2016

Re: [money] What sort of leap is this?

Does every contribution record create a line item, even if it’s not in a price set and there is only one item?


Guy Iaccarino
Greenleaf Advancement
Phone: 203-624-4636, ext. 111 | Email: guyiac <at> greenleafadvancement.com | Post: PO Box 617, Guilford CT 06437
Website: http://greenleafadvancement.com | LinkedIn:http://www.linkedin.com/in/guyiac | Twitter: http://twitter.com/greenleaf_adv




On Apr 21, 2016, at 10:16 AM, JoAnne Chester <j_chester <at> optusnet.com.au> wrote:

I usually read these and say nothing as I am not qualified to comment as a dev.  However, in this case I feel that an admin/users perspective may be useful. 

 

I would be very happy to see the financial type on the contribution page changed back to contribution type using a different option list.

But then I would expect to be able to search by both contribution type and financial type in the “Find contribution” and advanced searches.

 

Also, it would be fine with me because all my contribution pages use price sets. Not everyone does. That ‘default’ financial type is used for contributions that don’t use a price set, so you would have to force everyone to use price sets or provide for setting the financial types separately for the “other amounts” (if enabled) and the Fixed Contribution Options.

 

Then you have a similar situation with Event registrations and also with Memberships where the financial type is set where the membership is defined (Some of us have different accounts for new memberships and for renewals, so again you can have a contribution with financial type ‘New member’ yet the only line item for it have financial type ‘Renewing member’.

 

So I am not sure that changing to Contribution type would be an easy fix. 

 

JoAnne

 

From: civicrm-dev <at> lists.civicrm.org [mailto:civicrm-dev <at> lists.civicrm.org] On Behalf Of Andrew Perry
Sent: 21 April 2016 21:27
To: Eileen McNaughton <eileen <at> mcnaughty.com>
Cc: civicrm-dev <at> lists.civicrm.org
Subject: Re: [civicrm-dev] [money] What sort of leap is this?

 

I always liked Contribution Type. If this was a step towards changing the field to refer to a new option list that could be things like "Financial", "Time", "In-kind" then I think it can make sense to reintroduce an old term that can have a broader meaning than Financial Type and perhaps help realise the potential of the original broader Contribution meaning. 

 

Andrew

Sent from my iPhone


On 21 Apr 2016, at 6:27 PM, Eileen McNaughton <eileen <at> mcnaughty.com> wrote:

"If we are talking about providing backwards compatibility lets just change the label of the Financial type field on the contribution to something like "Contribution group" (or back to contribution type!)"

Actually I like this! 

There is no problem with the field per se - just the expectation it correlates to line items. I would be happy to see a UI label of
'Contribution Type'
'Contribution Category'
'Contribution Taxonomy'
'Contribution Classification'

and probably prefer Contribution Type.

This is probably something we could tackle incrementally without any obvious breakage.

Eileen


On 21/04/2016 8:05 p.m., Jamie Novick wrote:

Guys,

Im -(minus) 1 on the idea of computed field.

It sounds complicated and doesn't really give any good information. We would be better off spending the time giving better search tools and prominence to line items (i.e. the contribution search to allow to list line items and to show only line items of a financial type etc + export these) and explaining better how these interact with the contribution entity. 

If we are talking about providing backwards compatibility lets just change the label of the Financial type field on the contribution to something like "Contribution group" (or back to contribution type!) and have its own option value list. Then people can use the field like they always did to group and sort contributions, but knowing this has no impact on accounting. (I imagine this would be better for Andrew too!) The technical solution for this is hopefully simpler and we end up with no feature loss.

I'm really not keen on anything that has duplication of data and a strong possibility of a "mixed/multiple" value which I'm not sure will be that useful.

Best

Jamie

On behalf of Compucorp Ltd

 

Jamie Novick / Director

Tel: +44 (0)207 096 3336 / Mob: +44 (0)775 123 8875

Skpe ID: jamie.novick
www.compucorp.co.uk

 

This email is confidential and is intended for the addressee only. If you are not the addressee, please delete the email and do not use it in any way. Compucorp Ltd does not accept or assume responsibility for any use of or reliance on this email by anyone, other than the intended addressee to the extent agreed in the relevant contract for the matter to which this email relates (if any).

 

Unless expressly stated to the contrary, the content of this e-mail does not represent the professional advice, views or opinion of Compucorp Ltd. Compucorp Ltd. reserve the right to monitor all business and personal e-mails to and from its associated e-mail addresses. By responding to this e-mail you confirm that you consent to this action.

 

All reasonable efforts have been made to confirm that this e-mail and any attachments are virus free but you are advised to check. Compucorp Ltd. does not accept any responsibility for any damage caused by any transmission to the recipient's computer systems.

 

 

On 21/04/2016 08:03, Andrew Perry wrote:

I think Tim's "3. Convert the field to a computed field" would be the appropriate change for the purposes of the report as people will expect to see the financial type or reserved text like:

 

"Mismatched" - where the values for each item aren't the same as the contribution

"Various" - where the values for each item differ from the contribution

The actual Financial Type - where the FT is the same on the contribution and the line items or there is no FT on the contribution and the line items are all the same FT.

 

Regards

 

 

Andrew

From: "Nicolas Ganivet" <nicolas <at> cividesk.com>
To: "Eileen McNaughton" <eileen <at> mcnaughty.com>
Cc: "Joshua Gowans" <josh <at> civicrm.org>, "civicrm-dev" <civicrm-dev <at> lists.civicrm.org>
Sent: Thursday, 21 April, 2016 4:29:28 PM
Subject: Re: [civicrm-dev] [money] What sort of leap is this?

 

Thanks Eileen. Maybe Joe can weight in on deleting the column in the report.

Sent from my iPhone


On Apr 21, 2016, at 12:25 AM, Eileen McNaughton <eileen <at> mcnaughty.com> wrote:

The changes were made in 4.3 not 4.5, and no plan has been made at any point for this, in fact other than with Joe a year ago this is the first discussion I've had about it with anyone since around 4-5 years ago.

 Yes, we could easily remove that column from the report if people agreed to it. 

Eileen

On 21/04/2016 4:44 p.m., Nicolas Ganivet wrote:

Eileen - Can't we do a quick fix, such as either:

- deleting the 'Financial Type' column (at the contribution level) from the report,

- or adding the join through the line items to get to the 'Financial Type' for each line items?

 

Josh - It seems like the switch from financial type at the contribution level to line item level is not complete. Yet this has been released in version 4.5. Is that '100 hour+ project' in someone's work plan? Which version is it going to be released in? 

 

------ Original Message ------

From: "Eileen McNaughton" <eileen <at> mcnaughty.com>

Sent: 4/20/2016 4:59:00 PM

Subject: Re: [civicrm-dev] [money] What sort of leap is this?

 

Yeah - it's tricky because it is a training issue - ie. financial_type on the contribution record is not a true reflection of the financial_type of the component line items - and in this case they DO have items within a contribution with different types so it should be obvious really.

Retaining the financial type field on the contribution is confusing. But we are looking at a 100 hour+ project to fully get rid of it and unless we do that as 'leap-by-extension' or 'opt-in-setting' it will include a lot of breaking changes.

We need to break that work down into meaningful chunks.

Eileen




On 21/04/2016 10:19 a.m., Nicolas Ganivet wrote:

As an aside to your aside, I do consider such things "bugs", probably with a rating of "high" if not "critical".

 

You cannot 'just tell a customer to ignore it', or 'yeah, we know it does not work, but this is old stuff and nobody cares about fixing it'. If it's in the software it has to work, period. Can you imagine SalesForce or any of our other competitors pulling the same thing? IMO this should be flagged in JIRA and the contribution report corrected asap.

 

We seriously have to change our culture if we ever want to become "the CRM of choice for nonprofits".

 

------ Original Message ------

From: "Eileen McNaughton" <eileen <at> mcnaughty.com>

Sent: 4/20/2016 3:59:25 PM

Subject: Re: [civicrm-dev] [money] What sort of leap is this?

 

As an aside I have a customer who won't stop logging me tickets saying the bookkeeping report doesn't match the contribution report - I keep saying ' don't use the contribution report' but she just comes back a few months later with the same issue. 

Eileen

On 21/04/2016 4:30 a.m., Alan Dixon wrote:

Great discussion and conclusion. 

 

My only $0.02 worth is that with the proposed resolution of the original issue (which is excellent), perhaps there's no dependence necessary on the proposed extension that deals with some of the ramifications, if we just consider these financial line items as separate entities from contributions. I think both are sensible things to have CRUDly functions on, even though it might be confusing for first time administrators. Having a line-item equivalent to all the contribution search/list/report stuff in a core extension makes complete sense to me and could be backportable to earlier versions.

 

So my alternative is:

 

Re what a 'leap by extension' (CiviFinancial) might look like


1) if you enable it you get new search screens & report screens would directly access the line items.



2) you'd get some helpful advice on the contribution search/report screens about the existence and utility of the line item versions.



My description of 2) would be along the lines of: if you're interested in contributions mainly for their aggregate size and contributor/origin, use the contributions, but if you're interested in the accounting/bookkeeping, use the line items.



I think we've been afraid of confusing administrators, but the lack of exposed interface to the line item data (to the point of obfuscation at times?) has been going on too long now, imho ...



 - Alan



 

 

On Wed, Apr 20, 2016 at 5:18 AM, Eileen McNaughton <eileen <at> mcnaughty.com> wrote:

I feel like this has been good discussion and looking at some concrete examples is helpful when it comes to discussing the path forwards.

On the original minor issue - I feel like there is appetite for a patch like this in core 

https://github.com/civicrm/civicrm-core/commit/3ca9556322597196a7da209f4d3c2134357f43d0

Possibly wrapped in an if $is_fintype_computed setting check. Presumably the same setting would be used for incremental steps towards deprecating that field on an opt-in basis. One schema change which would be required at some point would probably be making the financial_type_id field optional at the DB level.

If there is general support for that then I will write the obligatory unit tests & aim to have a PR ready for possibly inclusion in this month or next month's QA cycle


Eileen



On 20/04/2016 2:27 p.m., Tim Otten wrote:

This thread has shown a few impulses about what to do with the field civicrm_contribution.financial_type_id, e.g.

 

1. Drop the field. This produces cleaner end-result, but breaks compat and requires broad changes (eg changing APIs and search UIs).

 

  => Personally, I don’t like this because it requires forking development (4.7 vs master), which then cuts-off QA-cycle, delays releases, and encourages downstream development that veers off-course from mainline. And multiplies the scope across various UIs/use-cases/integrations.

 

2. Externalize the field (e.g. move the field to an extension). This is basically the same as dropping it, except that there’s a backup-option for folks who really like the old field.

 

 => Personally, I like having a toggle for backward-compat, but I don’t like this because it still has the same costs as dropping the field, with the added complexity of some oddball extension.

 

3. Convert the field to a computed field (e.g. pick the FT type from among line-items; or a pick a synthetic value). When there are multiple values, pick placeholder (like “Varied” or “Mixed” or “Multiple”). Existing APIs/UIs can still read/write this field.

 

 => Personally, I like this most because there’s less need for broad changes or forking development paths, but there may be some odd edge-cases that rely on the old behavior (e.g. a third-party integration that writes its own funny values to the field). For this reason, I’d propose a tweak to Joe’s original suggestion: add a setting like “bool $is_fintype_computed”. In 4.7, this setting is disabled by default so that existing UIs/APIs read/write the field as before. In 5.0, it’s enabled by default. Unit tests can be added for checking both modes.

 

I don’t think the original scope Eileen described was really intended to be a ‘leap’. The improvements that Tobias described — updating the UI so that you can fundamentally search at a slightly better grain -- sound more like a leap that merits an extension to override the current search UI.

 

 

On Apr 19, 2016, at 3:41 PM, Eileen McNaughton <eileen <at> mcnaughty.com> wrote:

 

So, I'm not interested in maintaining new financial functionality outside of core - but the leap-by-extension theory offers the possibility of core-extensions - which has potential. It seems that Tobias's extension (providing different searches) would be the start of the extension potentially. I think we could also consider building some deprecation of financial_type into apiv4 - ie. NOT returning it by default for the entities we want to deprecate it from & excluding it from the metadata. Note this deprecation should NOT be considered trivial & pulling it into scope of the api v4 project would kill the apiv4 so it would have to be a separate but timely project.

The biggest differences IMEO between a core-extension & a community-extension is that it would ship with core, would be considered part of Core in terms of managing PRs & QA and the per-PR unit tests would include the extensions unit tests. Unlike a regular extension it would result in at least as much maintenance burden on the core team as a bug fix approach would (for better and worse).

It's important to also note that dealing with any blockers in core to doing this in an extension in a non-breaky way would be part of the scope of any core-shipping extension. Dealing with any conflicts with other extensions that are not covered by core unit tests is out of scope.

It's interesting how hard it has been to contain this issue to the original 'bug'. This is probably a good place for the larger discussion https://issues.civicrm.org/jira/browse/CRM-17140

Re what a 'leap by extension' (CiviFinancial) might look like
1) if you enable it the search screens & report screens would not show the financial type from the contribution & would directly access the line item
2) the field that exists would be updated to reflect the line items - either a new 'Mixed' type or the type that all the line items belong to.

A couple of notes
1) the intention of the extension would be to be the crown prince - ie to eventually become THE code
2) It would HAVE to be incremental - ie start with a vision but very limited functionality - e.g just the hook to handle 'Type' and build up to forms etc
3) I'm not quite sure we are yet at the place infrastructure-wise to start this process  - we probably need to wait until the core team have shipped their first core-extension to provide a blue print
4) Obviously the ideal is that the search screens when added would be based on apiv4 & new angular goodness - although that relies on the core team having done a bunch of work first which may not be on a different time frame.
5) You all know what I'm going to say here. Fill in the blank :-)

Eileen



On 20/04/2016 9:24 a.m., Nicolas Ganivet wrote:

We might need to consult with Tim on the topic, but the more comments I see the more this all seems like a 5.0 project rather than a leap in 4.7: there are too many things that needs to be changed in core for it to be practical in an extension!

 

Remember that you can only override a core file once, and that the last extension added to CiviCRM will get this override to the expense of other extensions that implement this same override.

 

------ Original Message ------

From: "Toby Lounsbury" <toby <at> ginkgostreet.com>

Sent: 4/19/2016 11:41:56 AM

Subject: Re: [civicrm-dev] [money] What sort of leap is this?

 

Just to chime in: +1 for removing financial type from the contribution. However, something to think about, when doing that, we need to allow searching by financial type, and this would involve adding support for searching by line-item financial type. I started work on that project a while back, but didn't have the time or funding to complete it. I have had several people contact me asking if it would be possible to complete that functionality. In keeping with the theme of using extensions to leap forward, I built that functionality as a discrete extension. I do plan to finish it at some point, though I have no idea when I will have the spare time or funding. I think if financial types are going to be removed from contributions, there will need to be some updates to my original spec. For instance find contributions that contain a line item: of this type, line-items are only of this type, are not all of this type, or do not contain line items of this type. There is also the reporting to consider.  

I don't think all of this needs to be complete out of the gate if we use a sort of pseudo-financial type tied to the contribution, but they are all pieces that I see being connected to the issue being discussed.

 

On Tue, Apr 19, 2016 at 9:02 AM, Joe Murray <joe.murray <at> jmaconsulting.biz> wrote:

That's reasonable. We could make it a reserved pseudo-financial type.


Joe Murray, PhD
President, JMA Consulting
joe.murray <at> jmaconsulting.biz
skype JosephPMurray twitter JoeMurray
416.466.1281

 

On Tue, Apr 19, 2016 at 8:40 AM, Andrew Perry <andrew.perry <at> communitybuilders.com.au> wrote:

I'd suggest displaying 'Various' or something other than blank/NULL if it is not set in the Contribution or not all items are the same FT.  Could also display 'Conflicting' if what is set in the Contribution doesn't match the FT of all the items.

 

From: "Joe Murray" <joe.murray <at> jmaconsulting.biz>
To: "Andrew Perry" <andrew.perry <at> communitybuilders.com.au>
Cc: "Eileen McNaughton" <eileen <at> mcnaughty.com>, "money" <money <at> lists.civicrm.org>, "civicrm-dev" <civicrm-dev <at> lists.civicrm.org>
Sent: Tuesday, 19 April, 2016 10:38:03 PM
Subject: Re: [civicrm-dev] [money] What sort of leap is this?

 

That makes some sense. I really like deprecating it now and not including it in APIv4 or schema for 5.0+. An alternative 'fix' for 4.7 and possibly 5.0+ is to start treating the UI value as a calculated field, displaying blank if the associated item financial types are not all the same. 

On Apr 19, 2016 7:46 AM, "Andrew Perry" <andrew.perry <at> communitybuilders.com.au> wrote:

A first step could be leaving the field on a Contribution in the database and the API (deprecated) but removing it from the UI (except reports for legacy/API data reporting reasons) and slating it for removal in the new API so people have time to update their integrated systems?

 

From: "Joe Murray" <joe.murray <at> jmaconsulting.biz>
To: "Eileen McNaughton" <eileen <at> mcnaughty.com>
Cc: "money" <money <at> lists.civicrm.org>, "civicrm-dev" <civicrm-dev <at> lists.civicrm.org>
Sent: Tuesday, 19 April, 2016 9:05:45 PM
Subject: Re: [money] [civicrm-dev] What sort of leap is this?

 

My own sense is that this could best be treated as a bug. Dave wanted to keep the financialType on the contribution for backward compatibility, especially since a single line item was the predominant use case. But the financialType on the financial item is authoritative for bookkeeping, and the one on the contribution is at most an indication of what the default value is for the financial items. So when they are different with only one line item it is confusing to the point of being a bug to let them be different. My own preference would be to refactor and remove the financialType from contributions like I wanted to do in v. 3.4. It violates a schema version of the DRY directive: don't repeat yourself, aka imperfect normalisation.

On Apr 19, 2016 6:00 AM, "Eileen McNaughton" <eileen <at> mcnaughty.com> wrote:

We've been discussing the idea of 'leap by extension' recently and I'm now encountering an issue which feels worth discussing in that context.

When you create a contribution by price set you set the financial type per option. I love this.

However, you still must specify the Financial Type for the contribution page and that type is used for the contribution. This is leaving me with a contribution with a single line item which has a different financial type to the financial type of the contribution.

It feels like a bug and it certainly doesn't feel like the sort of thing where I should have to install an un-maintained extension to get what it feels like core should do - ie "if there is only one financial type used by the line items then the contribution should have that financial type". And yet I can see fixing it might hurt people who depend on what I see as a bug in ways I can't predict.

So the options are
1) treat it as a bug & fix in core + add unit tests
2) add some custom code for this specific customer & leave others to fend for themselves
3) write a generic extension & post on my github and put in the extensions directory
4) figure out a way to turn it into a core-maintained extension

I should note at this point that option 3 probably seems like the obvious option to everyone except me. If you were proposing it I would probably endorse it - however, it does not confer any benefits of unit testing and  I don't have capacity to support extensions like this, or even to read all the incoming PRs.  So when we get to the end of the conversation & agree that #3 is the best option I will do #2.

In the bigger picture this is small part of the 'we shouldn't really store financialType on the contribution record since it's not actually accurate conversation'. And perhaps it's part of a bigger leap to that situation.

Eileen

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

____________________________________________________________
You received this message as a subscriber on the list:
    civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
    civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
    http://lists.civicrm.org/lists/info/civicrm-dev

____________________________________________________________
You received this message as a subscriber on the list:
money <at> lists.civicrm.org
To be removed from the list, send any message to:
money-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/money

 

____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev

 

 

____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev



 

-- 

Tobias Lounsbury 

Sr. Web Developer

 

Ginkgo Street Labs 
888-223-6609 | 
twitter | linkedin

____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev

____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev




This email has been checked for viruses by Avast antivirus software. 
www.avast.com

 

____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev

 

____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev



This email has been checked for viruses by Avast antivirus software. 
www.avast.com

 

____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev



 

-- 

Alan Dixon, Web Developer

Drupal and CiviCRM for Canadian nonprofits.
http://blackflysolutions.ca/

____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev



This email has been checked for viruses by Avast antivirus software. 
www.avast.com

 

____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev

____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev



This email has been checked for viruses by Avast antivirus software. 
www.avast.com

 

____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev



This email has been checked for viruses by Avast antivirus software. 
www.avast.com

 

____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev

 

____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev

 

____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev

 

____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev

____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev

____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev

____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev

Gmane