Chris Burgess | 25 May 00:57 2016
Picon
Gravatar

Straw poll: reply-all on list?

I see both Eileen and I had list replies go only to the poster this morning, which I think doesn't serve list discussions well. Do we actually want this list set to reply-sender or reply-all? IMO reply-all makes for better list discussion.

Please reply with "reply-sender"  or "reply-all" to state your preference?
____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev
Frank J. Gómez | 25 May 00:20 2016
Gravatar

Re: Defining machine names as constants

Eileen, your ever-growing list of email addresses is staggering.

--
Frank J. Gómez
Developer & Principal
Ginkgo Street Labs 
2121 Decatur Pl NW, Washington, DC 20008
toll free: 888-223-6609

Please send requests for technical support to support <at> ginkgostreet.com for the most expeditious response.

On Tue, May 24, 2016 at 6:16 PM, Eileen McNaughton <emcnaughton <at> wikimedia.org> wrote:
I have at the back of my mind that I would like to see core option groups defined in a meta-data-ey way - mostly because it’s such a pig to alter them in the install sql & you wind up with this huge generated file that is awful to review commits on

Eileen

On May 25, 2016, at 10:12 AM, Frank J. Gómez <frank <at> ginkgostreet.com> wrote:

Class constants are like static methods, in a sense, in that you needn't instantiate the class to use them. You could do: My_Bao::MY_CONST to get the value.

Replacing magic numbers is precisely the reason I three-quarter-heartedly tried to pass the constants to the client side.

--
Frank J. Gómez
Developer & Principal
Ginkgo Street Labs 
2121 Decatur Pl NW, Washington, DC 20008
toll free: 888-223-6609

Please send requests for technical support to support <at> ginkgostreet.com for the most expeditious response.

On Tue, May 24, 2016 at 3:02 PM, Joe Murray <joe.murray <at> jmaconsulting.biz> wrote:
Good question about how to maintain DRY (don't repeat yourself) principle when using PHP constants in js. Here's one approach, but it seems from comments that I'd want to really verify myself that it works: http://stackoverflow.com/questions/440494/share-constants-between-php-and-javascript

My main interest is replacing magic values in code with CONSTANTS, and I don't have a strong view on benefits of constants versus class constants. 

I have a slight concern about putting all option group and option values in a single include given possible performance implications. My sense is that it is likely better to break out by option group into different include files, since it is rare that they are all needed in one file like during initial creation of database. 

Perhaps some of the option groups should be class constants based on a component class or something, if that class would always be instantiated when the constants are needed, or perhaps more specifically, if it would not be too expensive to reference them when needed. Evaluating these tradeoffs is a bit beyond my level of PHP expertise.


Joe Murray, PhD
President, JMA Consulting
joe.murray <at> jmaconsulting.biz
skype JosephPMurray twitter JoeMurray
416.466.1281

On Tue, May 24, 2016 at 1:37 PM, Frank J. Gómez <frank <at> ginkgostreet.com> wrote:
We were pretty good about this in CiviVolunteer, though we could have been more consistent with the naming pattern.

In our case it seemed appropriate to make them class constants, so we did -- e.g.: https://github.com/civicrm/org.civicrm.volunteer/blob/master/CRM/Volunteer/BAO/Assignment.php#L38-L40

It was less clear what to do on the client side (many Vol interfaces are written in JS) -- in some cases we got lazy and used the strings, in other cases we passed constants from PHP to JS via CRM_Core_Resources (e.g., https://github.com/civicrm/org.civicrm.volunteer/blob/master/CRM/Volunteer/Form/Manage.php#L71), about which I have mixed feelings.

--
Frank J. Gómez
Developer & Principal
Ginkgo Street Labs 
2121 Decatur Pl NW, Washington, DC 20008
toll free: 888-223-6609

Please send requests for technical support to support <at> ginkgostreet.com for the most expeditious response.

On Tue, May 24, 2016 at 11:58 AM, Joe Murray <joe.murray <at> jmaconsulting.biz> wrote:
In a separate discussion, Nicolas didn't catch my poorly explained suggestion about defining constants to refer to machine names for option groups and option values.

As a general coding standard, I think we should be using const (which became available in PHP 5.3 and is preferred for D8 (https://www.drupal.org/coding-standards#naming)) to define the machine names for our option groups and option values, i.e. whenever we insert values for civicrm_option_group.name or civicrm_option_value.name. These inserts are mainly found in https://github.com/civicrm/civicrm-core/blob/master/xml/templates/civicrm_data.tpl but also in upgrade scripts. The constants should be defined in an .inc file so that people don't mistype 'Production' or 'production', and get an error if they type RUN_ENVIRONMENT_PORDUCTION. While we should likely do this for all machine names, I'd like to start with a proposal that we do this for civicrm_option_group.name and civicrm_option_value.name. We could define a pattern for the CONSTANT that refers to a particular 'constant' value. Perhaps the option_group name should be a prefix for all of the option values in that group.

So rather than the following in CRM/Upgrade/Incremental/sql/4.6.alpha1.mysql.tpl:
-- add batch type for pledge payments
SELECT <at> option_group_id := id FROM civicrm_option_group WHERE name = 'batch_type';
SELECT <at> max_option_value:= max(ROUND(value)) FROM civicrm_option_value WHERE option_group_id = <at> option_group_id;
INSERT INTO civicrm_option_value(option_group_id, {localize field='label'}`label`{/localize}, value, name,weight)
VALUES ( <at> option_group_id, {localize}'{ts escape="sql"}Pledge Payment{/ts}'{/localize}, <at> max_option_value+1, 'Pledge Payment','3');
We'd see:
 require_once 'sql/option_group_batch_type.inc'; // or some similar name
....
-- add batch type for pledge payments
SELECT <at> option_group_id := id FROM civicrm_option_group WHERE name = OPTION_GROUP_BATCH_TYPE;
SELECT <at> max_option_value:= max(ROUND(value)) FROM civicrm_option_value WHERE option_group_id = <at> option_group_id;
INSERT INTO civicrm_option_value(option_group_id, {localize field='label'}`label`{/localize}, value, name,weight)
VALUES ( <at> option_group_id, {localize}'{ts escape="sql"}Pledge Payment{/ts}'{/localize}, <at> max_option_value+1, OPTION_VALUE_BATCH_TYPE_PLEDGE_PAYMENT,'3');

Joe Murray, PhD
President, JMA Consulting
joe.murray <at> jmaconsulting.biz
skype JosephPMurray twitter JoeMurray
416.466.1281
____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev



____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev


____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev
Karin - Semper IT | 24 May 20:01 2016

Re: Has anyone run into scheduled jobs drifting on Centos7?

Thanks Allan! This one project also happens to be a 4.4.x (VLTS = Very 
Long Term Stable...).

-- Karin

On 16-05-24 11:58 AM, Allen Shaw wrote:
> Just to add fuel to your search, here are these links, which I came
> across a few days ago:
>
> https://forum.civicrm.org/index.php?topic=36031.0
> https://issues.civicrm.org/jira/browse/CRM-16276
>
> Not the same symptoms, and fixed in 4.6.3, but could yield some clues.
>
> - A.
>
>
> On Tue, 24 May 2016 11:46:19 -0600
> Karin - Semper IT <karin <at> semper-it.com> wrote:
>
>> Hi,
>>
>> I asked this on:
>> http://civicrm.stackexchange.com/questions/12003/has-anyone-run-into-scheduled-jobs-drifting-on-centos7
>>
>> But perhaps better asked here. Alan and I are wondering if anyone
>> else has run into this?
>>
>> -- Karin
>>
>> -- 
>> Karin G.M. Gerritsen, Ph.D.
>> Semper Information Technology Inc.
>> Interactive Drupal/CiviCRM websites
>> Phone: 403 613 9531
>> Web: http://www.semper-it.com
>> Twitter:  <at> KarinSemperIT
>>
>>
>> ____________________________________________________________
>> You received this message as a subscriber on the list:
>>       civicrm-dev <at> lists.civicrm.org
>> To be removed from the list, send any message to:
>>       civicrm-dev-unsubscribe <at> lists.civicrm.org
>>
>> For all list information and functions, see:
>>       http://lists.civicrm.org/lists/info/civicrm-dev
>

--

-- 
Karin G.M. Gerritsen, Ph.D.
Semper Information Technology Inc.
Interactive Drupal/CiviCRM websites
Phone: 403 613 9531
Web: http://www.semper-it.com
Twitter:  <at> KarinSemperIT

Joe Murray | 24 May 17:58 2016
Gravatar

Defining machine names as constants

In a separate discussion, Nicolas didn't catch my poorly explained suggestion about defining constants to refer to machine names for option groups and option values.

As a general coding standard, I think we should be using const (which became available in PHP 5.3 and is preferred for D8 (https://www.drupal.org/coding-standards#naming)) to define the machine names for our option groups and option values, i.e. whenever we insert values for civicrm_option_group.name or civicrm_option_value.name. These inserts are mainly found in https://github.com/civicrm/civicrm-core/blob/master/xml/templates/civicrm_data.tpl but also in upgrade scripts. The constants should be defined in an .inc file so that people don't mistype 'Production' or 'production', and get an error if they type RUN_ENVIRONMENT_PORDUCTION. While we should likely do this for all machine names, I'd like to start with a proposal that we do this for civicrm_option_group.name and civicrm_option_value.name. We could define a pattern for the CONSTANT that refers to a particular 'constant' value. Perhaps the option_group name should be a prefix for all of the option values in that group.

So rather than the following in CRM/Upgrade/Incremental/sql/4.6.alpha1.mysql.tpl:
-- add batch type for pledge payments
SELECT <at> option_group_id := id FROM civicrm_option_group WHERE name = 'batch_type';
SELECT <at> max_option_value:= max(ROUND(value)) FROM civicrm_option_value WHERE option_group_id = <at> option_group_id;
INSERT INTO civicrm_option_value(option_group_id, {localize field='label'}`label`{/localize}, value, name,weight)
VALUES ( <at> option_group_id, {localize}'{ts escape="sql"}Pledge Payment{/ts}'{/localize}, <at> max_option_value+1, 'Pledge Payment','3');
We'd see:
 require_once 'sql/option_group_batch_type.inc'; // or some similar name
....
-- add batch type for pledge payments
SELECT <at> option_group_id := id FROM civicrm_option_group WHERE name = OPTION_GROUP_BATCH_TYPE;
SELECT <at> max_option_value:= max(ROUND(value)) FROM civicrm_option_value WHERE option_group_id = <at> option_group_id;
INSERT INTO civicrm_option_value(option_group_id, {localize field='label'}`label`{/localize}, value, name,weight)
VALUES ( <at> option_group_id, {localize}'{ts escape="sql"}Pledge Payment{/ts}'{/localize}, <at> max_option_value+1, OPTION_VALUE_BATCH_TYPE_PLEDGE_PAYMENT,'3');

Joe Murray, PhD
President, JMA Consulting
joe.murray <at> jmaconsulting.biz
skype JosephPMurray twitter JoeMurray
416.466.1281
____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev
Nicolas Ganivet | 23 May 07:56 2016
Gravatar

State/Province in payment profile

Eileen and all,
 
We recently have setup a customer in Europe in a Drupal/CiviCRM setup with Paypal Pro in a membership registration webform. We encountered the issue that the state/province field is required in the billing profile, but state/province is not used Europe and not required by Paypal Pro for European countries.
 
We installed an ugly patch to fix this, but this seems like using a sledgehammer to crack a nut and it should really be fixed in core.
Issues related to this: https://issues.civicrm.org/jira/browse/CRM-5427 and more recently https://issues.civicrm.org/jira/browse/CRM-15555 (hence Eileen in the To:)
 
How do you deal with the state/province field being required in the billing profile for European customers? Is there a better way than 1-off patching? A function override? A hook?
 
If not, then what would be the best way to resolve this issue? It seems like a user-configurable setting would not solve the use case of a global organization accepting payments from different countries, so a 'dynamic' setting that only shows/requires the state/province if country is USA/Canada (and maybe a few more) might make sense.
 
Thoughts? Suggestions?
 
--Nicolas.
 
____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev
Tim Otten | 21 May 08:11 2016
Gravatar

v4.7.8 Release Candidate

Thank you to the folks who participated in Mayl’s Review Week! The release candidate is now available at:

http://dist.civicrm.org/by-date/2016-05-20/4.7.8-rc/

From now until the end of next weekend (Sunday, May 29), we’ll be in Test Week. This is a good opportunity to do a trial-run of the new release on your staging systems. In accepting follow-up PRs, regressions from v4.7.7 will be given particularly high priority.

The final GA release is set to go out on Wednesday, June 1. To ensure that there is sufficient time to fix any major new issues, please try to finish testing several days before that.

== Changes ==

There have been ~120 pull-requests merged into 4.7.8-rc. Many of the patches dealing with usability and framework issues are spread around different, but three topics received the most intense changes. On your staging sites, it may be fruitful to spot-check some of your typical use-cases in these areas:

1. CiviMail
2. CiviContribute
3. Dedupe & Smart Groups

It’s difficult for me to do justice to all of the issues (especially within the space of an email). Further details are available in:


In particular, these tabs are interesting:

 - “Release Notes” - Some reviewers felt that there were particular changes which merit awareness among release-testers.

- “Changelog” - The full list of PRs that were merged.

 - “PRs” - The initial planning document used to divide-and-conquor the review process, organized by general topic.

If you think that some topics are particularly notable and merit greater awareness, please feel free to add to the “Release Notes” or post to this thread.

== Git Administrativa ==

In all the major CiviCRM repos (civicrm-core.git, civicrm-drupal.get, et al), there is a new branch named “4.7.8-rc”. If there is a notable regression from a recent point-release, please file the PR against 4.7.8-rc and mention " <at> totten”. Otherwise, existing PRs and new PRs should continue to go to “master”.

Thanks and good night,
Tim
____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev
Joe Murray | 20 May 23:44 2016
Gravatar

CiviCRM Environment

I'd like to draw people's attention to my proposal/PR to add an environment setting that would indicate whether an instance was the production environment, or something else like staging or development: https://issues.civicrm.org/jira/browse/CRM-18231 

The motivation for this was an incident where a non-CiviCRM Drupal shop moved a production instance to a staging site and an individual developer site according to the then-standard recipe for moving an installation. This resulted in two instances sending out the same mailing, and a few IATS recurring contributions being submitted by the CiviCRM extension twice.

It seems to potentially be useful in combination with some exciting work Alan Dixon is doing to make it easy to copy a production instance of CiviCRM and set it up for testing release candidate code.

Eileen suggested, and I agree, that it would be good to get feedback from this list on the issue.

Cheers,

Joe Murray, PhD
President, JMA Consulting
joe.murray <at> jmaconsulting.biz
skype JosephPMurray twitter JoeMurray
416.466.1281

____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev
Alan Dixon | 20 May 20:54 2016
Picon
Gravatar

core support for deployment environments, CRM-18231

As per our discussion in the partners' meeting yesterday, I want to propose a strategy to deal with this issue




As I understand it, the issue arose from the experience of a new developer who grabbed a copy of a site and caused havoc due to the site copy sending out mail and/or triggering recurring payments. It's easy as an experienced developer to avoid this, and there is documentation about it, but it's also reasonable for us to make development and testing less error-prone.

This also connects to the goal of trying to provide more automation in generating test sites from live sites.

The idea of the current patch is to make it easier to create safer non-production site copies:

1. it provides a web interface to change an isProduction flag, 
2. it provides a way to lock a site into non-production via the settings file.
3. it intercepts a couple of key actions that shouldn't normally happen on a non-production site (normally), e.g. running scheduled jobs, and tries to switch outgoing mail into the logging-only.

I think the overall idea is good, but it has two big problems:

1. It introduces more complex logic into core, including a form hook that we really want to stay away from, and none of it is applicable to a production site (i.e. yet more code debt).

2. It limits the deployment environment to a simple yes/no, whereas most development shops have a variety of environments (staging, development, testing, pre-launch ..?), each of which may have different needs depending on how those environments are implemented (e.g. it's possible to build a site that gets sandboxed in via the network, in which it's safe and desirable for testing purposes for the jobs to run).

While these two issues are pulling in opposite directions, I think there's a solution to both at the same time - put minimal support for deployment environments in core, but put all the configuration and potential complexity into an extension.

Here's how I'd propose to do it:

1. In core, include a simple string system configuration variable "deployment_environment", left as NULL by default.

2. Include a new hook in core, something like:

hook_civicrm_deploymentCheck($action, $deployment_environment)

If it returns FALSE, then it means that the corresponding action should not be taken in the current deployment environment.

Include the hook test before any action that is sensitive to the deployment environment. 

  • This feels like there's some overlap with how rules works - anybody have any clever ideas to combine this?
  • Also, it would be confusing if more than one extension implemented this, so that would need to be thought through.

3. Provide a default behaviour in core that would run if the hook wasn't implemented by any extensions, as:

core_civicrm_deploymentCheck($action, $environment) {
  return empty($environment) ? FALSE : TRUE;
}

Which basically means, if the deployment environment has been set, don't do this action.

4. Provide an official extension (like this one, but more official): 


which would provide support for simple deployment options, and as an example for building your own more complicated ones. What this (prototype one) provides is a big link to an angular form (because that's what I'm trying to learn) that:
a. Tells you about your current environment
b. Allows you to switch environments
c. Allows you to run an anonymizing script on your data.
It also warns you all the time that you're on a non-production environment.

Conclusion:

1. For the original use case, you'd just add this in the civi settings file in your development environment:

$civicrm_setting['CiviCRM Preferences']['deploymentEnvironment'] = 'development';

[or any string you like].


2. If you wanted to be able to switch from/to other environment labels, and have those states do different things, then you'd install/use/modify the base DevelopmentEnvironment extension.



--
Alan Dixon, Web Developer

Drupal and CiviCRM for Canadian nonprofits.
http://blackflysolutions.ca/
____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev
Jamie McClelland | 13 May 18:00 2016

Re: RC Testing notes (or how I learned to love Docker)

I've also created a docker setup that also uses civicrm-buildkit. 

And it's also for dev purposes not testing, but could probably be
re-purposed:

https://github.com/progressivetech/docker-civicrm-buildkit

jamie

On Fri May 13, Alan Dixon wrote:
> So, yes, there's a lot of overlap in functionality between Vagrant and
> Docker, and a lot of ways they can work together, depending on your goal.
> 
> Certainly, if Nicolas' civicrm-buildkit-vagrant is working for you to build
> your development environment, then Docker doesn't add much (though it might
> save you time/machine resources if you use Docker in place of VirtualBox).
> 
> What Docker can help us do is to make things easier, safer and
> less-resource-intensive to auto-build test environments from live sites,
> especially for people for whom a full development environment is not
> desirable.
> 
> Yeah, chasing that plug-and-play test-environment dream ...
> 
>  - Alan
> 
> 
> 
> 
> 
> 
> On Thu, May 12, 2016 at 4:57 PM, Eileen McNaughton <
> emcnaughton <at> wikimedia.org> wrote:
> 
> > When my vagrant box blew up I used Nicolas’s to get going again & it
> > worked very well! That doesn’t really tie into the conversation but my
> > practice now is to plug that vagrant build whenever I have the chance.
> >
> > On May 13, 2016, at 4:36 AM, Nicolas Ganivet <nicolas <at> cividesk.com> wrote:
> >
> > All,
> >
> > On the last Sprint in Vail I have created a vagrant setup that builds-up a
> > complete CiviCRM dev environment with buildkit using just a few
> > commands. It has since been used by a number of folks with good results.
> >     https://github.com/civicrm/civicrm-buildkit-vagrant
> > <https://github.com/civicrm/civicrm-buildkit-vagrant>
> >
> > I guess this could easily be re-purposed for creating testing
> > environments. I am not versed enough to understand the exact differences
> > between Docker and Vagrant but both seem to play in the same space.
> >
> > Thanks,
> > --Nicolas.
> >
> > ------ Original Message ------
> > From: "Alan Dixon" <alan.g.dixon <at> gmail.com>
> > To: civicrm-dev <at> lists.civicrm.org
> > Sent: 5/12/2016 8:44:00 AM
> > Subject: [civicrm-dev] RC Testing notes (or how I learned to love Docker)
> >
> >
> > My bloggish post about testing RCs.
> >
> > 1. I haven't been testing any RCs.
> >
> > I hit a wall a few months backs when 4.7 started requiring php 5.4 - my
> > servers were all on 5.3 and I needed some time to safely update to higher
> > (and yes, that was the weird bug last week).
> >
> > I try to keep all my servers the same, <rant>and my desktop was so old it
> > couldn't accept any more memory and was thrashing with swap just due to the
> > browser</rant>.
> >
> > My desktop is now up to 32Gb and my servers are up to php5.6 so I'm ready
> > to dive in.
> >
> > 2. I decided I needed a bit more of a sophisticated testing environment.
> >
> > I have production environments, staging environments and development
> > environments, all on different machines, but sometimes used erratically
> > (i.e. doing development on my staging environments). None of these are
> > optimal for RC testing, because that would interfere with what they are
> > being used for, and testing has different requirements.
> >
> > I'd been reading about Docker and that seemed like the right choice, so
> > with my new machines and (imaginary) time I've dived in this week and
> > wanted to share a high-level summary of what I learned.
> >
> > 3. My week with Docker.
> >
> > I only use Linux, so YMMV with Mac or Windows. There's a nice simple
> > tutorial on the Docker site that I recommend to understand the basics. The
> > multiple types and layers of virtualization and the messy use of language
> > to describe it all makes it hard to understand sometimes, and I get annoyed
> > with the sales pitch stuff that seems to think everyone should use Docker
> > all the time.
> >
> > For the purposes of the next bits, just three pieces of vocabulary:
> > 1. Image - the file on disk that Docker uses to create the running virtual
> > environment (typically up to about 500MB).
> > 2. Container - the running virtual environment.
> > 3. Docker file - a text file, a human-readable/editable script is used by
> > Docker to build the image.
> >
> > What I didn't like:
> > a. my first tutorial example test took a long time to download the image.
> > b. stop with the sales pitches!
> >
> > What I like:
> > a. I was super impressed with the layers that are exposed that allow you
> > to use Docker in different ways.
> > b. Specifically, the ability to share, understand and re-use/recycle
> > images and Dockerfiles.
> > c. The image format seems to be built with something that smells a lot
> > like git - i.e. extremely efficient storage of changesets. So if you've got
> > someone's image and Dockerfile and you want to make a small change to it,
> > or you want to get someone else's image based on that image, you're only
> > downloading or building the difference. So once you've got a few base
> > images downloaded/built, downloading/building updates is fast.
> > d. Once you've got the image built, launching a container from it is
> > breathlessly fast.
> > e. Building your own image isn't hard. The script language in a Docker
> > file is like any other kind of scripting file, I was able to built a
> > Dockerfile from my notes in a straightforward way.
> > f. Not being a fully virtualized environment means you can access things
> > outside easily (e.g. you can mount host filesystems into the container).
> >
> > So what I've managed to do in my week is to create a docker image that
> > duplicates most of my production environment, allowing me to generate
> > testing environments that are very close exact copies, easily.
> >
> > You're right, that's a sales pitch, and also not true, I was getting
> > carried away.
> >
> > The tricky bit is of course going from the generic to the specific, and
> > that's my next step. What makes a testing site different from a generic
> > environment is of course the docroot filesystem and the database. To do
> > this there are two approaches:
> > 1. Build an image off of the base image by importing the docroot
> > filesystem and the database. And of course, then make the changes we're
> > testing (i.e. updating the civi code). This sounds doable, but I'm not sure
> > how difficult or time-consuming it would be in practice.
> > 2. Make your base image configurable with a sites directory path and db
> > names. In this case, the container does not include the site-specific
> > stuff, but just mounts it at run-time as a configuration. This might be
> > faster and lighter, but would only work if your sites are setup
> > consistently, and also requires you to separately setup the db server with
> > database.
> >
> > A separate issue is how/if to do error/logging. Chris' logging service
> > sounds like a simple solution, I'd like to hear if that sound like it would
> > work for this purpose. Another option would be to mount the logs outside
> > the container, or to have some other way of sending the logging information
> > out.
> >
> > In terms of routing for a test site available to clients - I use varnish
> > as a front end proxy so pulling in a test environment (that is typically
> > running on a non-standard port) to a test domain is super-easy.
> >
> > In theory, you might be able to launch a test site inside docker on your
> > production environment, which may or may not be a good idea.
> >
> > Your thoughts and feedback appreciated ...
> >
> >   - Alan
> >
> > --
> > Alan Dixon, Web Developer
> >
> > Drupal and CiviCRM for Canadian nonprofits.
> > http://blackflysolutions.ca/
> > ____________________________________________________________
> > You received this message as a subscriber on the list:
> > civicrm-dev <at> lists.civicrm.org
> > To be removed from the list, send any message to:
> > civicrm-dev-unsubscribe <at> lists.civicrm.org
> >
> > For all list information and functions, see:
> > http://lists.civicrm.org/lists/info/civicrm-dev
> >
> > ____________________________________________________________
> > You received this message as a subscriber on the list:
> > civicrm-dev <at> lists.civicrm.org
> > To be removed from the list, send any message to:
> > civicrm-dev-unsubscribe <at> lists.civicrm.org
> >
> > For all list information and functions, see:
> > http://lists.civicrm.org/lists/info/civicrm-dev
> >
> >
> >
> 
> 
> -- 
> Alan Dixon, Web Developer
> 
> Drupal and CiviCRM for Canadian nonprofits.
> http://blackflysolutions.ca/
> ____________________________________________________________
> You received this message as a subscriber on the list:
>      civicrm-dev <at> lists.civicrm.org
> To be removed from the list, send any message to:
>      civicrm-dev-unsubscribe <at> lists.civicrm.org
> 
> For all list information and functions, see:
>      http://lists.civicrm.org/lists/info/civicrm-dev

--

-- 
Jamie McClelland
612.724.2600 ext 700
Progressive Technology Project
http://progressivetech.org/

For Powerbase Support Questions, write to mailto:support <at> progressivetech.org
or call 512-782-8478

OpenPGP Key: http://current.workingdirectory.net/pages/identity/ 
Alan Dixon | 13 May 16:02 2016
Picon
Gravatar

Re: RC Testing notes (or how I learned to love Docker)

So, yes, there's a lot of overlap in functionality between Vagrant and Docker, and a lot of ways they can work together, depending on your goal.

Certainly, if Nicolas' civicrm-buildkit-vagrant is working for you to build your development environment, then Docker doesn't add much (though it might save you time/machine resources if you use Docker in place of VirtualBox).

What Docker can help us do is to make things easier, safer and less-resource-intensive to auto-build test environments from live sites, especially for people for whom a full development environment is not desirable.

Yeah, chasing that plug-and-play test-environment dream ...

 - Alan






On Thu, May 12, 2016 at 4:57 PM, Eileen McNaughton <emcnaughton <at> wikimedia.org> wrote:
When my vagrant box blew up I used Nicolas’s to get going again & it worked very well! That doesn’t really tie into the conversation but my practice now is to plug that vagrant build whenever I have the chance.

On May 13, 2016, at 4:36 AM, Nicolas Ganivet <nicolas <at> cividesk.com> wrote:

All,
 
On the last Sprint in Vail I have created a vagrant setup that builds-up a complete CiviCRM dev environment with buildkit using just a few commands. It has since been used by a number of folks with good results.
 
I guess this could easily be re-purposed for creating testing environments. I am not versed enough to understand the exact differences between Docker and Vagrant but both seem to play in the same space.
 
Thanks,
--Nicolas.
 
------ Original Message ------
From: "Alan Dixon" <alan.g.dixon <at> gmail.com>
Sent: 5/12/2016 8:44:00 AM
Subject: [civicrm-dev] RC Testing notes (or how I learned to love Docker)
 
My bloggish post about testing RCs. 

1. I haven't been testing any RCs.

I hit a wall a few months backs when 4.7 started requiring php 5.4 - my servers were all on 5.3 and I needed some time to safely update to higher (and yes, that was the weird bug last week).

I try to keep all my servers the same, <rant>and my desktop was so old it couldn't accept any more memory and was thrashing with swap just due to the browser</rant>. 

My desktop is now up to 32Gb and my servers are up to php5.6 so I'm ready to dive in.

2. I decided I needed a bit more of a sophisticated testing environment.

I have production environments, staging environments and development environments, all on different machines, but sometimes used erratically (i.e. doing development on my staging environments). None of these are optimal for RC testing, because that would interfere with what they are being used for, and testing has different requirements.

I'd been reading about Docker and that seemed like the right choice, so with my new machines and (imaginary) time I've dived in this week and wanted to share a high-level summary of what I learned.

3. My week with Docker.

I only use Linux, so YMMV with Mac or Windows. There's a nice simple tutorial on the Docker site that I recommend to understand the basics. The multiple types and layers of virtualization and the messy use of language to describe it all makes it hard to understand sometimes, and I get annoyed with the sales pitch stuff that seems to think everyone should use Docker all the time.

For the purposes of the next bits, just three pieces of vocabulary:
1. Image - the file on disk that Docker uses to create the running virtual environment (typically up to about 500MB).
2. Container - the running virtual environment.
3. Docker file - a text file, a human-readable/editable script is used by Docker to build the image. 

What I didn't like:
a. my first tutorial example test took a long time to download the image.
b. stop with the sales pitches!

What I like:
a. I was super impressed with the layers that are exposed that allow you to use Docker in different ways. 
b. Specifically, the ability to share, understand and re-use/recycle images and Dockerfiles.
c. The image format seems to be built with something that smells a lot like git - i.e. extremely efficient storage of changesets. So if you've got someone's image and Dockerfile and you want to make a small change to it, or you want to get someone else's image based on that image, you're only downloading or building the difference. So once you've got a few base images downloaded/built, downloading/building updates is fast.
d. Once you've got the image built, launching a container from it is breathlessly fast.
e. Building your own image isn't hard. The script language in a Docker file is like any other kind of scripting file, I was able to built a Dockerfile from my notes in a straightforward way.
f. Not being a fully virtualized environment means you can access things outside easily (e.g. you can mount host filesystems into the container).

So what I've managed to do in my week is to create a docker image that duplicates most of my production environment, allowing me to generate testing environments that are very close exact copies, easily.

You're right, that's a sales pitch, and also not true, I was getting carried away.

The tricky bit is of course going from the generic to the specific, and that's my next step. What makes a testing site different from a generic environment is of course the docroot filesystem and the database. To do this there are two approaches:
1. Build an image off of the base image by importing the docroot filesystem and the database. And of course, then make the changes we're testing (i.e. updating the civi code). This sounds doable, but I'm not sure how difficult or time-consuming it would be in practice.
2. Make your base image configurable with a sites directory path and db names. In this case, the container does not include the site-specific stuff, but just mounts it at run-time as a configuration. This might be faster and lighter, but would only work if your sites are setup consistently, and also requires you to separately setup the db server with database.

A separate issue is how/if to do error/logging. Chris' logging service sounds like a simple solution, I'd like to hear if that sound like it would work for this purpose. Another option would be to mount the logs outside the container, or to have some other way of sending the logging information out.

In terms of routing for a test site available to clients - I use varnish as a front end proxy so pulling in a test environment (that is typically running on a non-standard port) to a test domain is super-easy.

In theory, you might be able to launch a test site inside docker on your production environment, which may or may not be a good idea.

Your thoughts and feedback appreciated ...

  - Alan

-- 
Alan Dixon, Web Developer

Drupal and CiviCRM for Canadian nonprofits.
http://blackflysolutions.ca/
____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev
____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev




--
Alan Dixon, Web Developer

Drupal and CiviCRM for Canadian nonprofits.
http://blackflysolutions.ca/
____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev
Jaap Jansma | CiviCooP | 13 May 11:36 2016
Gravatar

Backport of CIVI-SA-2016-07 and CIVI-SA-2016-08 for 4.4

Hey All,

On request of one of our clients I have backported the security advisories of lastweeek. Those backports are suiteable for civicrm version 4.4.

I (and the client) want to share those patches with the community. So feel free to use them to update your current 4.4 installations.

CIVI-SA-2016-07: SQL Injections in AJAX callbacks

Use the patches 001-backport-of-PR8106.patch, 0002-backport-of-PR8205.patch, 0003-backport-of-PR8216.patch, and 0004-backport-of-PR8275

CIVI-SA-2016-08: Persistent XSS in CiviCRM backend

Use the patch 0001-CRM-17592-backport-for-4.4.patch

How to apply them?

You need some basic understanding of git to apply batches. So you need to be a little technicall ;-) The command to apply is git apply <patch_file>
Execute that command in your civicrm directory.

Thanks,

Jaap Jansma


--

Met vriendelijke groet / Kind regards,

Jaap Jansma
Consultant / Developer
Meester van Hasseltlaan 1
7316 DG Apeldoorn

  

E-mail algemeen
E-mail adres
Telefoon algemeen
Telefoon rechtstreeks
Website
helpdesk <at> civicoop.org
jaap.jansma <at> civicoop.org
055-5762855
+31 (0) 318 743 800
www.civicoop.org

____________________________________________________________
You received this message as a subscriber on the list:
civicrm-dev <at> lists.civicrm.org
To be removed from the list, send any message to:
civicrm-dev-unsubscribe <at> lists.civicrm.org

For all list information and functions, see:
http://lists.civicrm.org/lists/info/civicrm-dev

Gmane