.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/plot_benchmarks.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_plot_benchmarks.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_plot_benchmarks.py:


Benchmarking
=================================

We compare the performance of the Graphical Lasso solvers implemented in ``GGLasso`` to two commonly used packages, i.e.

* `regain <https://github.com/fdtomasi/regain>`_ : contains an ADMM solver which is doing almost the same operations as ``ADMM_SGL``. For details, see the original paper. [ref3]_

* `sklearn <https://scikit-learn.org/stable/modules/generated/sklearn.covariance.GraphicalLasso.html#sklearn.covariance.GraphicalLasso>`_: by default uses the coordinate descent algorithm which was originally proposed by Friedman et al. [ref1]_ 

The results can be generated using the notebook in ``benchmarks/benchmarks.ipynb`` in the Github repository.
From ``GGLasso`` we use the standard solver ``ADMM_SGL`` labeled by **gglasso** and the block-wise solver labeled by **gglasso-block**. For details, we refer to :ref:`SGL solver`.

.. GENERATED FROM PYTHON SOURCE LINES 15-22

.. code-block:: Python


    import pandas as pd
    import numpy as np

    from gallery_helper import plot_bm









.. GENERATED FROM PYTHON SOURCE LINES 23-31

Synthetic power-law networks
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
We compare the solvers for a SGL problem using synthetic sparse powerlaw networks, which are generated as described in [ref2]_.
The solvers are tested for different values of :math:`\lambda_1` and different dimensionalities. These values are printed below.
We solve a SGL problem using each of the solvers independently, but using the same CPUs with 64GB of RAM in total.

The results were generated on a machine equipped with `AMD Opteron(tm) 6378 @ 1.40GHz (max 2.40 GHz) (8 Cores per socket, hyper-threading)`.


.. GENERATED FROM PYTHON SOURCE LINES 31-35

.. code-block:: Python


    df = pd.read_csv("../data/synthetic/bm5000.csv", index_col = 0)
    df.reset_index(drop=True)






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>time</th>
          <th>accuracy</th>
          <th>hamming</th>
          <th>iter</th>
          <th>method</th>
          <th>tol</th>
          <th>rtol</th>
          <th>p</th>
          <th>N</th>
          <th>l1</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>0.519696</td>
          <td>9.849324e-03</td>
          <td>30728</td>
          <td>12</td>
          <td>gglasso</td>
          <td>1.000000e-06</td>
          <td>1.000000e-06</td>
          <td>500</td>
          <td>550</td>
          <td>0.05</td>
        </tr>
        <tr>
          <th>1</th>
          <td>0.792978</td>
          <td>1.131881e-03</td>
          <td>30632</td>
          <td>20</td>
          <td>gglasso</td>
          <td>1.000000e-07</td>
          <td>1.000000e-07</td>
          <td>500</td>
          <td>550</td>
          <td>0.05</td>
        </tr>
        <tr>
          <th>2</th>
          <td>0.994830</td>
          <td>5.210053e-04</td>
          <td>30626</td>
          <td>23</td>
          <td>gglasso</td>
          <td>5.000000e-08</td>
          <td>5.000000e-08</td>
          <td>500</td>
          <td>550</td>
          <td>0.05</td>
        </tr>
        <tr>
          <th>3</th>
          <td>1.110264</td>
          <td>1.208771e-04</td>
          <td>30618</td>
          <td>29</td>
          <td>gglasso</td>
          <td>1.000000e-08</td>
          <td>1.000000e-08</td>
          <td>500</td>
          <td>550</td>
          <td>0.05</td>
        </tr>
        <tr>
          <th>4</th>
          <td>0.538926</td>
          <td>9.849324e-03</td>
          <td>30728</td>
          <td>0</td>
          <td>gglasso-block</td>
          <td>1.000000e-06</td>
          <td>1.000000e-06</td>
          <td>500</td>
          <td>550</td>
          <td>0.05</td>
        </tr>
        <tr>
          <th>...</th>
          <td>...</td>
          <td>...</td>
          <td>...</td>
          <td>...</td>
          <td>...</td>
          <td>...</td>
          <td>...</td>
          <td>...</td>
          <td>...</td>
          <td>...</td>
        </tr>
        <tr>
          <th>205</th>
          <td>433.196051</td>
          <td>1.229067e-03</td>
          <td>9838</td>
          <td>17</td>
          <td>regain</td>
          <td>1.000000e-05</td>
          <td>1.000000e-05</td>
          <td>5000</td>
          <td>5500</td>
          <td>0.50</td>
        </tr>
        <tr>
          <th>206</th>
          <td>530.698426</td>
          <td>2.653190e-04</td>
          <td>9838</td>
          <td>21</td>
          <td>regain</td>
          <td>5.000000e-06</td>
          <td>5.000000e-06</td>
          <td>5000</td>
          <td>5500</td>
          <td>0.50</td>
        </tr>
        <tr>
          <th>207</th>
          <td>747.611447</td>
          <td>1.769017e-05</td>
          <td>9838</td>
          <td>30</td>
          <td>regain</td>
          <td>1.000000e-06</td>
          <td>1.000000e-06</td>
          <td>5000</td>
          <td>5500</td>
          <td>0.50</td>
        </tr>
        <tr>
          <th>208</th>
          <td>197.463880</td>
          <td>1.191635e-08</td>
          <td>9838</td>
          <td>2</td>
          <td>sklearn</td>
          <td>1.000000e-02</td>
          <td>1.000000e-04</td>
          <td>5000</td>
          <td>5500</td>
          <td>0.50</td>
        </tr>
        <tr>
          <th>209</th>
          <td>199.192011</td>
          <td>1.191635e-08</td>
          <td>9838</td>
          <td>2</td>
          <td>sklearn</td>
          <td>1.000000e-01</td>
          <td>1.000000e-04</td>
          <td>5000</td>
          <td>5500</td>
          <td>0.50</td>
        </tr>
      </tbody>
    </table>
    <p>210 rows × 10 columns</p>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 36-43

.. code-block:: Python


    all_p_N= list(pd.unique(list(zip(df.p, df.N))))
    print("Dimensionality and sample size: (p,N) =", all_p_N )

    all_l1 = pd.unique(df.l1)
    print("Values of lambda_1:  =", all_l1 )





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    /home/docs/checkouts/readthedocs.org/user_builds/gglasso/checkouts/stable/examples/plot_benchmarks.py:37: FutureWarning: unique with argument that is not not a Series, Index, ExtensionArray, or np.ndarray is deprecated and will raise in a future version.
      all_p_N= list(pd.unique(list(zip(df.p, df.N))))
    Dimensionality and sample size: (p,N) = [(500, 550), (1000, 1100), (2000, 2200), (4000, 4400), (5000, 5500)]
    Values of lambda_1:  = [0.05 0.1  0.5 ]




.. GENERATED FROM PYTHON SOURCE LINES 44-48

Setup
^^^^^^^^^^^^^
Each solver terminates after a given number of maximum iterations or when some optimality condition is met. Hence, the performance is difficult to compare as these optimality criteria may differ.
Thus, we select a range of values for relative (rtol) and absolute (tol) tolerance (used in ADMM) and similarly tolerance values for ``sklearn``.

.. GENERATED FROM PYTHON SOURCE LINES 51-61

Calculating the accuracy 
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

After solving each of the problems which each solver for different tolerance values, we compare the obtained solutions to a reference solution, denoted by :math:`Z^\ast`. 

:math:`Z^\ast` is obtained by solving a SGL problem by one of the solvers for very small tolerance values (we used ``regain`` and set ``tol=rtol=1e-10``).
Finally, for a solution :math:`Z`, we calculate its accuracy using the normalized Euclidean distance:

.. math::
  \text{accuracy}(Z) =  \frac{\|Z^\ast - Z \|}{ \| Z^\ast\| }.

.. GENERATED FROM PYTHON SOURCE LINES 63-69

Runtime and accuracy with respect to :math:`\lambda_1`.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Now, determine a maximal accuracy :math:`\epsilon`. For each solver, we now select the run with minimal runtime where :math:`\text{accuracy}(Z) \leq \epsilon` is fulfilled. 
We plot the results for two values of :math:`\epsilon`:


.. GENERATED FROM PYTHON SOURCE LINES 71-74

Accuracy of :math:`\epsilon=5\cdot10^{-3}`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^


.. GENERATED FROM PYTHON SOURCE LINES 74-78

.. code-block:: Python



    plot_bm(df, min_acc= 5e-3, lambda_list=all_l1)




.. image-sg:: /auto_examples/images/sphx_glr_plot_benchmarks_001.png
   :alt: $\lambda_1$ = 0.05, $\lambda_1$ = 0.1, $\lambda_1$ = 0.5
   :srcset: /auto_examples/images/sphx_glr_plot_benchmarks_001.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 79-82

Accuracy of :math:`\epsilon=5\cdot10^{-2}`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^


.. GENERATED FROM PYTHON SOURCE LINES 82-84

.. code-block:: Python


    plot_bm(df, min_acc = 5e-2, lambda_list=all_l1)



.. image-sg:: /auto_examples/images/sphx_glr_plot_benchmarks_002.png
   :alt: $\lambda_1$ = 0.05, $\lambda_1$ = 0.1, $\lambda_1$ = 0.5
   :srcset: /auto_examples/images/sphx_glr_plot_benchmarks_002.png
   :class: sphx-glr-single-img






.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 1.750 seconds)


.. _sphx_glr_download_auto_examples_plot_benchmarks.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_benchmarks.ipynb <plot_benchmarks.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_benchmarks.py <plot_benchmarks.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: plot_benchmarks.zip <plot_benchmarks.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_