From kemlath at googlemail.com  Thu Dec  6 07:29:53 2018
From: kemlath at googlemail.com (Kemlath Orcslayer)
Date: Thu, 6 Dec 2018 07:29:53 +0100
Subject: [GiNaC-devel] Faster unarchiving of large equation systems
Message-ID: <DA1A0409-6669-47EF-A163-7FF92C155A2F@gmail.com>

Dear All,

I?m using GiNaC for automatic derivation of the Jacobian for very large non-linear equation systems for use in the Sundials suite of solvers.

The need for parallelisation arose and I used MPI for the job since GiNaC is not well suited for multi threading due to its ref counting scheme. So I setup appropriate MPI broadcasting code for GiNaC ex() objects using the available archive classes in GiNaC and boost based memory stream IO.

All went well except for the unarchiving performance?.
The problem lies in symbol::read_archive where for each unarchived symbol the list of ALL symbols is searched for a symbol of the same name:

void symbol::read_archive(const archive_node &n, lst &sym_lst)
{
	?...

	// If symbol is in sym_lst, return the existing symbol
	for (auto & s : sym_lst) {
		if (is_a<symbol>(s) && (ex_to<symbol>(s).name == tmp_name)) {
			*this = ex_to<symbol>(s); 

	??

For cases with 500K symbols this becomes unbearably slow (and this has to happen on each MPI-node)

My solution in my little local GiNaC branch was to introduce a second read_archive interface called read_archive_MPI which instead of a  GiNaC::lst &sym_lst gets a std::map<std::string, ginac::ex> map that allows for quickly finding a previously stored symbol of the same name:

void symbol::read_archive_MPI(const archive_node &n, SymbolMap &sym_lst)
{
	inherited::read_archive_MPI(n, sym_lst);
	serial = next_serial++;
	std::string tmp_name;
	n.find_string("name", tmp_name);

	// If symbol is in sym_lst, return the existing symbol
	SymbolMap::iterator it = sym_lst.find(tmp_name);
	if(it != sym_lst.end())
	{
		*this = ex_to<symbol>((it->second));

This approach is ABI compatible to older code but requires read_archive_MPI members in all ginac::basic derived objects. I?d like to stress that this does not introduce any MPI dependencies in GiNaC, the MPI serialisation code is separate and merely utilises the new archiving system.

None the less I figured I might share this with you since I deem it a reasonable improvement for large equation systems and makes GiNaC available for MPI computing.

Let me know what you think

Klaus





From weinzierl at uni-mainz.de  Wed Dec 12 11:09:51 2018
From: weinzierl at uni-mainz.de (Stefan Weinzierl)
Date: Wed, 12 Dec 2018 11:09:51 +0100
Subject: [GiNaC-devel] bug fix for inifcns_nstdsums.cpp
Message-ID: <alpine.DEB.2.20.1812121058350.26362@iphths.Physik.Uni-Mainz.DE>


Dear all,

attached is a small patch, which fixes a bug in G_do_hoelder related to 
the fact that in CLN a complex number of type cl_N with vanishing 
imaginary part is not necessarily a real number of type cl_R.
The bug is a variant of the issues discussed in
  https://www.ginac.de/pipermail/ginac-devel/2014-January/002077.html

The bug has been reported by T. Huber.

  Best wishes,

     Stefan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: polylog10.patch
Type: text/x-diff
Size: 579 bytes
Desc: diff file
URL: <http://www.ginac.de/pipermail/ginac-devel/attachments/20181212/f123fc45/attachment.bin>

From kreckel at in.terlu.de  Wed Dec 12 22:46:17 2018
From: kreckel at in.terlu.de (Richard B. Kreckel)
Date: Wed, 12 Dec 2018 22:46:17 +0100
Subject: [GiNaC-devel] Faster unarchiving of large equation systems
In-Reply-To: <DA1A0409-6669-47EF-A163-7FF92C155A2F@gmail.com>
References: <DA1A0409-6669-47EF-A163-7FF92C155A2F@gmail.com>
Message-ID: <7e89ae6c-eebe-a807-9f78-fbbefdc23535@in.terlu.de>

Hi Klaus,

On 06.12.18 07:29, Kemlath Orcslayer via GiNaC-devel wrote:
> This approach is ABI compatible to older code but requires read_archive_MPI members in all ginac::basic derived objects. I?d like to stress that this does not introduce any MPI dependencies in GiNaC, the MPI serialisation code is separate and merely utilises the new archiving system.

Thanks for sharing your experience with us!

If unarchiving is relevant for such large numbers of symbols, then the
current design with a sym_lst is quite bad. You are the first to report
this here, though.

Note that adding a virtual member function does generally break the ABI
of derived classes! They have to be recompiled.

Why not fix this by adding a second signature to read_archive, i.e. by
just dropping your _MPI prefix? We also have to consider that
user-written subclasses might have implemented their own read_archive
function. Maybe, we could have a virtual base class of read_archive_MPI
(or the overload without _MPI) that interfaces to the existing
read_archive function so we could keep API compatibility for some time?

Anyhow, it sounds interesting. Can you post your code, please?

Cheers
  -richy.
-- 
Richard B. Kreckel
<https://in.terlu.de/~kreckel/>


From git at ginac.de  Wed Dec 12 23:03:11 2018
From: git at ginac.de (Richard B. Kreckel)
Date: Wed, 12 Dec 2018 23:03:11 +0100 (CET)
Subject: [GiNaC-devel] [SCM] GiNaC -- a C++ library for symbolic
 computations branch, master, updated. release_1-4-0-616-g39eceef4
Message-ID: <20181212220311.C2A20D80289@cebix.net>

This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "GiNaC -- a C++ library for symbolic computations".

The branch, master has been updated
       via  39eceef41403ae77569110626f1fc957243228b7 (commit)
      from  96be7a17790700998e2d071e00ad5c1ffb4a2d3e (commit)

Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.

- Log -----------------------------------------------------------------
commit 39eceef41403ae77569110626f1fc957243228b7
Author: Stefan Weinzierl <weinzierl at uni-mainz.de>
Date:   Wed Dec 12 22:53:12 2018 +0100

    G_do_hoelder: fix case with real x values which are not of type cl_R.
    
    In CLN, a complex number of type cl_N with vanishing imaginary part is not
    necessarily a real number of type cl_R.
    
    This bug has been reported by T. Huber.

-----------------------------------------------------------------------

Summary of changes:
 ginac/inifcns_nstdsums.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


hooks/post-receive
--
GiNaC -- a C++ library for symbolic computations


From kreckel at in.terlu.de  Wed Dec 12 23:04:13 2018
From: kreckel at in.terlu.de (Richard B. Kreckel)
Date: Wed, 12 Dec 2018 23:04:13 +0100
Subject: [GiNaC-devel] bug fix for inifcns_nstdsums.cpp
In-Reply-To: <alpine.DEB.2.20.1812121058350.26362@iphths.Physik.Uni-Mainz.DE>
References: <alpine.DEB.2.20.1812121058350.26362@iphths.Physik.Uni-Mainz.DE>
Message-ID: <bb1f8cf3-fcde-af9a-9f19-d26890714e2b@in.terlu.de>

On 12.12.18 11:09, Stefan Weinzierl wrote:
> attached is a small patch, which fixes a bug in G_do_hoelder related to
> the fact that in CLN a complex number of type cl_N with vanishing
> imaginary part is not necessarily a real number of type cl_R.
> The bug is a variant of the issues discussed in
> ?https://www.ginac.de/pipermail/ginac-devel/2014-January/002077.html
> 
> The bug has been reported by T. Huber.
Thanks Stefan, I've applied it.

   -richard.
-- 
Richard B. Kreckel
<https://in.terlu.de/~kreckel/>