I am building firedrake using the system provided "mpicc" command. On the NERSC systems, the mpicc command is "cc", so I:

1) built my own firedrake fork of PETSc using the appropriate options 

2) passed in "--mpicc cc" to the firedrake-install command.

Everything works until I get to the PyOP2 stage. Upon inspection of firedrake-install.log, it seems the pip install for pyop2 still defaults to the "epic" command (which does not exist on the NERSC system) and not "cc" which I told the firedrake script to use.

2017-06-16 12:31:27,681 INFO   Installing PyOP2/

2017-06-16 12:31:27,681 DEBUG  /global/u1/j/jychang/Software/firedrake/bin/pip install --no-binary mpi4py --no-deps -vvv -e PyOP2/

2017-06-16 12:31:31,251 DEBUG  Obtaining file:///global/u1/j/jychang/Software/firedrake/src/PyOP2

  Running setup.py (path:/global/u1/j/jychang/Software/firedrake/src/PyOP2/setup.py) egg_info for package from file:///global/u1/j/jychang/Software/firedrake/src/PyOP2

    Running command python setup.py egg_info

    /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'test_requires'

      warnings.warn(msg)

    running egg_info

    creating PyOP2.egg-info

    writing requirements to PyOP2.egg-info/requires.txt

    writing PyOP2.egg-info/PKG-INFO

    writing top-level names to PyOP2.egg-info/top_level.txt

    writing dependency_links to PyOP2.egg-info/dependency_links.txt

    writing manifest file 'PyOP2.egg-info/SOURCES.txt'

    reading manifest file 'PyOP2.egg-info/SOURCES.txt'

    reading manifest template 'MANIFEST.in'

    warning: no files found matching '*.c' under directory 'pyop2'

    writing manifest file 'PyOP2.egg-info/SOURCES.txt'

  Source in ./PyOP2 has version 0.13.0+299.g200780c, which satisfies requirement PyOP2==0.13.0+299.g200780c from file:///global/u1/j/jychang/Software/firedrake/src/PyOP2

Installing collected packages: PyOP2

  Running setup.py develop for PyOP2

    Running command /global/u1/j/jychang/Software/firedrake/bin/python -c "import setuptools, tokenize;__file__='/global/u1/j/jychang/Software/firedrake/src/PyOP2/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps

    /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'test_requires'

      warnings.warn(msg)

    running develop

    running egg_info

    writing requirements to PyOP2.egg-info/requires.txt

    writing PyOP2.egg-info/PKG-INFO

    writing top-level names to PyOP2.egg-info/top_level.txt

    writing dependency_links to PyOP2.egg-info/dependency_links.txt

    reading manifest file 'PyOP2.egg-info/SOURCES.txt'

    reading manifest template 'MANIFEST.in'

    warning: no files found matching '*.c' under directory 'pyop2'

    writing manifest file 'PyOP2.egg-info/SOURCES.txt'

    running build_ext

    cythoning pyop2/sparsity.pyx to pyop2/sparsity.cpp

    building 'pyop2.sparsity' extension

    creating build

    creating build/temp.linux-x86_64-2.7

    creating build/temp.linux-x86_64-2.7/pyop2

    mpicc -fno-strict-aliasing -fmessage-length=0 -grecord-gcc-switches -fstack-protector -O2 -Wall -D_FORTIFY_SOURCE=2 -funwind-tables -fasynchronous-unwind-tables -g -DNDEBUG -fmessage-length=0 -grecord-gcc-switches -fstack-protector -O2 -Wall -D_FORTIFY_SOURCE=2 -funwind-tables -fasynchronous-unwind-tables -g -DOPENSSL_LOAD_CONF -fPIC -Ipyop2 -I/global/u1/j/jychang/Software/firedrake/lib/python2.7/site-packages/numpy/core/include -I/global/u1/j/jychang/Software/firedrake/lib/python2.7/site-packages/petsc4py/include -I/global/homes/j/jychang/Software/petsc-fd/include -I/global/homes/j/jychang/Software/petsc-fd/arch-cori-c-opt/include -I/usr/include/python2.7 -c pyop2/sparsity.cpp -o build/temp.linux-x86_64-2.7/pyop2/sparsity.o

    unable to execute 'mpicc': No such file or directory

    error: command 'mpicc' failed with exit status 1

Cleaning up...

Command "/global/u1/j/jychang/Software/firedrake/bin/python -c "import setuptools, tokenize;__file__='/global/u1/j/jychang/Software/firedrake/src/PyOP2/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps" failed with error code 1 in /global/u1/j/jychang/Software/firedrake/src/PyOP2/

Exception information:

Traceback (most recent call last):

  File "/global/u1/j/jychang/Software/firedrake/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main

    status = self.run(options, args)

  File "/global/u1/j/jychang/Software/firedrake/lib/python2.7/site-packages/pip/commands/install.py", line 342, in run

    prefix=options.prefix_path,

  File "/global/u1/j/jychang/Software/firedrake/lib/python2.7/site-packages/pip/req/req_set.py", line 784, in install

    **kwargs

  File "/global/u1/j/jychang/Software/firedrake/lib/python2.7/site-packages/pip/req/req_install.py", line 845, in install

    install_options, global_options, prefix=prefix)

  File "/global/u1/j/jychang/Software/firedrake/lib/python2.7/site-packages/pip/req/req_install.py", line 1007, in install_editable

    show_stdout=False)

  File "/global/u1/j/jychang/Software/firedrake/lib/python2.7/site-packages/pip/utils/__init__.py", line 707, in call_subprocess

    % (command_desc, proc.returncode, cwd))

InstallationError: Command "/global/u1/j/jychang/Software/firedrake/bin/python -c "import setuptools, tokenize;__file__='/global/u1/j/jychang/Software/firedrake/src/PyOP2/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps" failed with error code 1 in /global/u1/j/jychang/Software/firedrake/src/PyOP2/

2017-06-16 12:31:31,253 INFO

Install log saved in firedrake-install.log

I feel like there's a way to circumvent (or fix) this issue. What is the best way to do this?

Thanks,

Justin