{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Parallel processing with Pastastore\n",
    "\n",
    "This notebook shows parallel processing capabilities of `PastaStore`.\n",
    "\n",
    "\n",
    "<div class=\"alert alert-warning\">\n",
    "\n",
    "<strong>Note</strong> \n",
    "\n",
    "Parallel processing is platform dependent and may not\n",
    "always work. The current implementation works well for Linux users, though this\n",
    "will likely change with Python 3.14 and higher. For Windows users, parallel\n",
    "solving does not work when called directly from Jupyter Notebooks or IPython.\n",
    "To use parallel solving on Windows, the following code should be used in a\n",
    "Python file. \n",
    "\n",
    "</div>\n",
    "\n",
    "```python\n",
    "from multiprocessing import freeze_support\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    freeze_support()\n",
    "    pstore.apply(\"models\", some_func, parallel=True)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pastastore version : 1.12.0\n",
      "\n",
      "Python version     : 3.13.11\n",
      "Pandas version     : 2.3.3\n",
      "Matplotlib version : 3.10.8\n",
      "Pastas version     : 1.12.0\n",
      "PyYAML version     : 6.0.3\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import pastas as ps\n",
    "\n",
    "import pastastore as pst\n",
    "from pastastore.datasets import example_pastastore\n",
    "\n",
    "ps.set_log_level(\"ERROR\")  # silence Pastas logger for this notebook\n",
    "pst.get_color_logger(\"INFO\", \"pastastore\")\n",
    "pst.show_versions()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example pastastore\n",
    "\n",
    "Load some example data, create models and solve them to showcase parallel processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mPasConnector: library 'oseries' created in '/home/david/github/pastastore/examples/notebooks/temp/my_connector/oseries'\u001b[0m\n",
      "\u001b[32mPasConnector: library 'stresses' created in '/home/david/github/pastastore/examples/notebooks/temp/my_connector/stresses'\u001b[0m\n",
      "\u001b[32mPasConnector: library 'models' created in '/home/david/github/pastastore/examples/notebooks/temp/my_connector/models'\u001b[0m\n",
      "\u001b[32mPasConnector: library 'oseries_models' created in '/home/david/github/pastastore/examples/notebooks/temp/my_connector/oseries_models'\u001b[0m\n",
      "\u001b[32mPasConnector: library 'stresses_models' created in '/home/david/github/pastastore/examples/notebooks/temp/my_connector/stresses_models'\u001b[0m\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bed50aeb459b4fec8f1762a7ece2a9fd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Bulk creation models:   0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# get the example pastastore\n",
    "conn = pst.PasConnector(\"my_connector\", \"./temp\")\n",
    "# conn = pst.ArcticDBConnector(\"my_connector\", \"lmdb://./temp\")\n",
    "pstore = example_pastastore(conn)\n",
    "pstore.create_models_bulk();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Solving models\n",
    "\n",
    "The `PastaStore.solve_models()` method supports parallel processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b4090ea609da49038d5ccc2ebd6eece6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Solving models (parallel):   0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pstore.solve_models(parallel=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parallel processing using `.apply()`\n",
    "\n",
    "Define some function that takes a name as input and returns some result. In this case,\n",
    "return the $R^2$ value for each model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rsq(model_name: str) -> float:\n",
    "    \"\"\"Compute the R-squared value of a Pastas model.\"\"\"\n",
    "    ml = pstore.get_models(model_name)\n",
    "    return ml.stats.rsq()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can apply this function to all models in the pastastore using `pstore.apply()`. \n",
    "By default this function is run sequentially. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "04e56cf5b3114507b23ea192b623b61b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Computing rsq:   0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "head_nb5    0.438129\n",
       "oseries2    0.931883\n",
       "head_mw     0.159318\n",
       "oseries1    0.904487\n",
       "oseries3    0.030468\n",
       "dtype: float64"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pstore.apply(\"models\", rsq, progressbar=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to run this function in parallel, set `parallel=True` in the keyword arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6fdc2f44c3174d9c9b46457aa4ee21a7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Computing rsq (parallel):   0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "head_nb5    0.438129\n",
       "oseries2    0.931883\n",
       "head_mw     0.159318\n",
       "oseries1    0.904487\n",
       "oseries3    0.030468\n",
       "dtype: float64"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pstore.apply(\"models\", rsq, progressbar=True, parallel=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get model statistics\n",
    "\n",
    "The function `pstore.get_statistics` also supports parallel processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>rsq</th>\n",
       "      <th>mae</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>head_nb5</th>\n",
       "      <td>0.438129</td>\n",
       "      <td>0.318361</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>oseries2</th>\n",
       "      <td>0.931883</td>\n",
       "      <td>0.087067</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>head_mw</th>\n",
       "      <td>0.159318</td>\n",
       "      <td>0.631517</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>oseries1</th>\n",
       "      <td>0.904487</td>\n",
       "      <td>0.091329</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>oseries3</th>\n",
       "      <td>0.030468</td>\n",
       "      <td>0.106254</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               rsq       mae\n",
       "head_nb5  0.438129  0.318361\n",
       "oseries2  0.931883  0.087067\n",
       "head_mw   0.159318  0.631517\n",
       "oseries1  0.904487  0.091329\n",
       "oseries3  0.030468  0.106254"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pstore.get_statistics([\"rsq\", \"mae\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>rsq</th>\n",
       "      <th>mae</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>_get_statistics</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>head_nb5</th>\n",
       "      <td>0.438129</td>\n",
       "      <td>0.318361</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>oseries2</th>\n",
       "      <td>0.931883</td>\n",
       "      <td>0.087067</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>head_mw</th>\n",
       "      <td>0.159318</td>\n",
       "      <td>0.631517</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>oseries1</th>\n",
       "      <td>0.904487</td>\n",
       "      <td>0.091329</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>oseries3</th>\n",
       "      <td>0.030468</td>\n",
       "      <td>0.106254</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      rsq       mae\n",
       "_get_statistics                    \n",
       "head_nb5         0.438129  0.318361\n",
       "oseries2         0.931883  0.087067\n",
       "head_mw          0.159318  0.631517\n",
       "oseries1         0.904487  0.091329\n",
       "oseries3         0.030468  0.106254"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pstore.get_statistics([\"rsq\", \"mae\"], parallel=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compute prediction intervals\n",
    "\n",
    "Let's try using a more complex function and passing that to apply to use\n",
    "parallel processing. In this case we want to compute the prediction interval,\n",
    "and pass along the $\\alpha$ value via the keyword arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "def prediction_interval(model_name, **kwargs):\n",
    "    \"\"\"Compute the prediction interval for a Pastas model.\"\"\"\n",
    "    ml = pstore.get_models(model_name)\n",
    "    return ml.solver.prediction_interval(**kwargs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8cb46f7fdcf04aa9b2b541fdea020318",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Computing prediction_interval:   0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"2\" halign=\"left\">head_nb5</th>\n",
       "      <th colspan=\"2\" halign=\"left\">oseries2</th>\n",
       "      <th colspan=\"2\" halign=\"left\">head_mw</th>\n",
       "      <th colspan=\"2\" halign=\"left\">oseries1</th>\n",
       "      <th colspan=\"2\" halign=\"left\">oseries3</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>0.025</th>\n",
       "      <th>0.975</th>\n",
       "      <th>0.025</th>\n",
       "      <th>0.975</th>\n",
       "      <th>0.025</th>\n",
       "      <th>0.975</th>\n",
       "      <th>0.025</th>\n",
       "      <th>0.975</th>\n",
       "      <th>0.025</th>\n",
       "      <th>0.975</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1960-04-29</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.300204</td>\n",
       "      <td>9.406311</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1960-04-30</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.340113</td>\n",
       "      <td>9.291056</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1960-05-01</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.197963</td>\n",
       "      <td>9.459711</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1960-05-02</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.256496</td>\n",
       "      <td>9.436107</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1960-05-03</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.161201</td>\n",
       "      <td>9.336737</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-17</th>\n",
       "      <td>7.961349</td>\n",
       "      <td>9.682071</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-18</th>\n",
       "      <td>7.975718</td>\n",
       "      <td>9.576222</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-19</th>\n",
       "      <td>7.914376</td>\n",
       "      <td>9.647849</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-20</th>\n",
       "      <td>7.914663</td>\n",
       "      <td>9.710748</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-21</th>\n",
       "      <td>7.947020</td>\n",
       "      <td>9.636971</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>21817 rows × 10 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            head_nb5           oseries2         head_mw           oseries1  \\\n",
       "               0.025     0.975    0.025 0.975     0.025     0.975    0.025   \n",
       "1960-04-29       NaN       NaN      NaN   NaN  6.300204  9.406311      NaN   \n",
       "1960-04-30       NaN       NaN      NaN   NaN  6.340113  9.291056      NaN   \n",
       "1960-05-01       NaN       NaN      NaN   NaN  6.197963  9.459711      NaN   \n",
       "1960-05-02       NaN       NaN      NaN   NaN  6.256496  9.436107      NaN   \n",
       "1960-05-03       NaN       NaN      NaN   NaN  6.161201  9.336737      NaN   \n",
       "...              ...       ...      ...   ...       ...       ...      ...   \n",
       "2020-01-17  7.961349  9.682071      NaN   NaN       NaN       NaN      NaN   \n",
       "2020-01-18  7.975718  9.576222      NaN   NaN       NaN       NaN      NaN   \n",
       "2020-01-19  7.914376  9.647849      NaN   NaN       NaN       NaN      NaN   \n",
       "2020-01-20  7.914663  9.710748      NaN   NaN       NaN       NaN      NaN   \n",
       "2020-01-21  7.947020  9.636971      NaN   NaN       NaN       NaN      NaN   \n",
       "\n",
       "                 oseries3        \n",
       "           0.975    0.025 0.975  \n",
       "1960-04-29   NaN      NaN   NaN  \n",
       "1960-04-30   NaN      NaN   NaN  \n",
       "1960-05-01   NaN      NaN   NaN  \n",
       "1960-05-02   NaN      NaN   NaN  \n",
       "1960-05-03   NaN      NaN   NaN  \n",
       "...          ...      ...   ...  \n",
       "2020-01-17   NaN      NaN   NaN  \n",
       "2020-01-18   NaN      NaN   NaN  \n",
       "2020-01-19   NaN      NaN   NaN  \n",
       "2020-01-20   NaN      NaN   NaN  \n",
       "2020-01-21   NaN      NaN   NaN  \n",
       "\n",
       "[21817 rows x 10 columns]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pstore.apply(\"models\", prediction_interval, kwargs={\"alpha\": 0.05})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "af6ae1e2cc6e4e1198087958115b31ea",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Computing prediction_interval (parallel):   0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"2\" halign=\"left\">head_nb5</th>\n",
       "      <th colspan=\"2\" halign=\"left\">oseries2</th>\n",
       "      <th colspan=\"2\" halign=\"left\">head_mw</th>\n",
       "      <th colspan=\"2\" halign=\"left\">oseries1</th>\n",
       "      <th colspan=\"2\" halign=\"left\">oseries3</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>0.025</th>\n",
       "      <th>0.975</th>\n",
       "      <th>0.025</th>\n",
       "      <th>0.975</th>\n",
       "      <th>0.025</th>\n",
       "      <th>0.975</th>\n",
       "      <th>0.025</th>\n",
       "      <th>0.975</th>\n",
       "      <th>0.025</th>\n",
       "      <th>0.975</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1960-04-29</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.318763</td>\n",
       "      <td>9.625481</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1960-04-30</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.157354</td>\n",
       "      <td>9.441837</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1960-05-01</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.178571</td>\n",
       "      <td>9.421986</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1960-05-02</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.243335</td>\n",
       "      <td>9.478925</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1960-05-03</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.237774</td>\n",
       "      <td>9.313179</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-17</th>\n",
       "      <td>7.860393</td>\n",
       "      <td>9.630678</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-18</th>\n",
       "      <td>7.874968</td>\n",
       "      <td>9.671061</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-19</th>\n",
       "      <td>7.913059</td>\n",
       "      <td>9.650184</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-20</th>\n",
       "      <td>7.956231</td>\n",
       "      <td>9.667919</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-21</th>\n",
       "      <td>7.927083</td>\n",
       "      <td>9.712015</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>21817 rows × 10 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            head_nb5           oseries2         head_mw           oseries1  \\\n",
       "               0.025     0.975    0.025 0.975     0.025     0.975    0.025   \n",
       "1960-04-29       NaN       NaN      NaN   NaN  6.318763  9.625481      NaN   \n",
       "1960-04-30       NaN       NaN      NaN   NaN  6.157354  9.441837      NaN   \n",
       "1960-05-01       NaN       NaN      NaN   NaN  6.178571  9.421986      NaN   \n",
       "1960-05-02       NaN       NaN      NaN   NaN  6.243335  9.478925      NaN   \n",
       "1960-05-03       NaN       NaN      NaN   NaN  6.237774  9.313179      NaN   \n",
       "...              ...       ...      ...   ...       ...       ...      ...   \n",
       "2020-01-17  7.860393  9.630678      NaN   NaN       NaN       NaN      NaN   \n",
       "2020-01-18  7.874968  9.671061      NaN   NaN       NaN       NaN      NaN   \n",
       "2020-01-19  7.913059  9.650184      NaN   NaN       NaN       NaN      NaN   \n",
       "2020-01-20  7.956231  9.667919      NaN   NaN       NaN       NaN      NaN   \n",
       "2020-01-21  7.927083  9.712015      NaN   NaN       NaN       NaN      NaN   \n",
       "\n",
       "                 oseries3        \n",
       "           0.975    0.025 0.975  \n",
       "1960-04-29   NaN      NaN   NaN  \n",
       "1960-04-30   NaN      NaN   NaN  \n",
       "1960-05-01   NaN      NaN   NaN  \n",
       "1960-05-02   NaN      NaN   NaN  \n",
       "1960-05-03   NaN      NaN   NaN  \n",
       "...          ...      ...   ...  \n",
       "2020-01-17   NaN      NaN   NaN  \n",
       "2020-01-18   NaN      NaN   NaN  \n",
       "2020-01-19   NaN      NaN   NaN  \n",
       "2020-01-20   NaN      NaN   NaN  \n",
       "2020-01-21   NaN      NaN   NaN  \n",
       "\n",
       "[21817 rows x 10 columns]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pstore.apply(\"models\", prediction_interval, kwargs={\"alpha\": 0.05}, parallel=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get signatures\n",
    "\n",
    "The function `pstore.get_signatures` does not explicitly support parallel processing but can be used in combination with `pstore.apply`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "signatures = [\n",
    "    \"cv_period_mean\",\n",
    "    \"cv_date_min\",\n",
    "    \"cv_date_max\",\n",
    "    \"cv_fall_rate\",\n",
    "    \"cv_rise_rate\",\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>head_nb5</th>\n",
       "      <th>oseries2</th>\n",
       "      <th>head_mw</th>\n",
       "      <th>oseries1</th>\n",
       "      <th>oseries3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>cv_period_mean</th>\n",
       "      <td>0.061879</td>\n",
       "      <td>0.015199</td>\n",
       "      <td>0.145062</td>\n",
       "      <td>0.013066</td>\n",
       "      <td>0.029168</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cv_date_min</th>\n",
       "      <td>0.246021</td>\n",
       "      <td>0.128636</td>\n",
       "      <td>0.254627</td>\n",
       "      <td>0.145884</td>\n",
       "      <td>1.394852</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cv_date_max</th>\n",
       "      <td>1.262425</td>\n",
       "      <td>0.722945</td>\n",
       "      <td>1.083929</td>\n",
       "      <td>0.300328</td>\n",
       "      <td>0.444442</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cv_fall_rate</th>\n",
       "      <td>-1.136450</td>\n",
       "      <td>-0.722718</td>\n",
       "      <td>-1.430200</td>\n",
       "      <td>-0.744797</td>\n",
       "      <td>-1.032837</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cv_rise_rate</th>\n",
       "      <td>1.259450</td>\n",
       "      <td>0.836678</td>\n",
       "      <td>1.097257</td>\n",
       "      <td>0.862981</td>\n",
       "      <td>0.931181</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                head_nb5  oseries2   head_mw  oseries1  oseries3\n",
       "cv_period_mean  0.061879  0.015199  0.145062  0.013066  0.029168\n",
       "cv_date_min     0.246021  0.128636  0.254627  0.145884  1.394852\n",
       "cv_date_max     1.262425  0.722945  1.083929  0.300328  0.444442\n",
       "cv_fall_rate   -1.136450 -0.722718 -1.430200 -0.744797 -1.032837\n",
       "cv_rise_rate    1.259450  0.836678  1.097257  0.862981  0.931181"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pstore.get_signatures(signatures=signatures)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "674d9be95e7e41d4b698cdbb8266b2c5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Computing get_signatures (parallel):   0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>get_signatures</th>\n",
       "      <th>head_nb5</th>\n",
       "      <th>oseries2</th>\n",
       "      <th>head_mw</th>\n",
       "      <th>oseries1</th>\n",
       "      <th>oseries3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>cv_period_mean</th>\n",
       "      <td>0.061879</td>\n",
       "      <td>0.015199</td>\n",
       "      <td>0.145062</td>\n",
       "      <td>0.013066</td>\n",
       "      <td>0.029168</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cv_date_min</th>\n",
       "      <td>0.246021</td>\n",
       "      <td>0.128636</td>\n",
       "      <td>0.254627</td>\n",
       "      <td>0.145884</td>\n",
       "      <td>1.394852</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cv_date_max</th>\n",
       "      <td>1.262425</td>\n",
       "      <td>0.722945</td>\n",
       "      <td>1.083929</td>\n",
       "      <td>0.300328</td>\n",
       "      <td>0.444442</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cv_fall_rate</th>\n",
       "      <td>-1.136450</td>\n",
       "      <td>-0.722718</td>\n",
       "      <td>-1.430200</td>\n",
       "      <td>-0.744797</td>\n",
       "      <td>-1.032837</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cv_rise_rate</th>\n",
       "      <td>1.259450</td>\n",
       "      <td>0.836678</td>\n",
       "      <td>1.097257</td>\n",
       "      <td>0.862981</td>\n",
       "      <td>0.931181</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "get_signatures  head_nb5  oseries2   head_mw  oseries1  oseries3\n",
       "cv_period_mean  0.061879  0.015199  0.145062  0.013066  0.029168\n",
       "cv_date_min     0.246021  0.128636  0.254627  0.145884  1.394852\n",
       "cv_date_max     1.262425  0.722945  1.083929  0.300328  0.444442\n",
       "cv_fall_rate   -1.136450 -0.722718 -1.430200 -0.744797 -1.032837\n",
       "cv_rise_rate    1.259450  0.836678  1.097257  0.862981  0.931181"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pstore.apply(\n",
    "    \"oseries\", pstore.get_signatures, kwargs={\"signatures\": signatures}, parallel=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load models\n",
    "\n",
    "Load models in parallel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "95e57bc33e1645beab06a844183063d3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Computing get_models:   0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "{'head_nb5': Model(oseries=head_nb5, name=head_nb5, constant=True, noisemodel=False),\n",
       " 'oseries2': Model(oseries=oseries2, name=oseries2, constant=True, noisemodel=False),\n",
       " 'head_mw': Model(oseries=head_mw, name=head_mw, constant=True, noisemodel=False),\n",
       " 'oseries1': Model(oseries=oseries1, name=oseries1, constant=True, noisemodel=False),\n",
       " 'oseries3': Model(oseries=oseries3, name=oseries3, constant=True, noisemodel=False)}"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pstore.apply(\"models\", pstore.get_models, fancy_output=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `max_workers` keyword argument sets the number of workers that are spawned. The default value is often fine, but it can be set explicitly.\n",
    "\n",
    "The following works for `PasConnector`. See alternative code below for `ArcticDBConnector`.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "46592d696b184f5ca88b107ed5f83279",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Computing get_models (parallel):   0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "{'head_nb5': Model(oseries=head_nb5, name=head_nb5, constant=True, noisemodel=False),\n",
       " 'oseries2': Model(oseries=oseries2, name=oseries2, constant=True, noisemodel=False),\n",
       " 'head_mw': Model(oseries=head_mw, name=head_mw, constant=True, noisemodel=False),\n",
       " 'oseries1': Model(oseries=oseries1, name=oseries1, constant=True, noisemodel=False),\n",
       " 'oseries3': Model(oseries=oseries3, name=oseries3, constant=True, noisemodel=False)}"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pstore.apply(\n",
    "    \"models\", pstore.get_models, fancy_output=True, parallel=True, max_workers=5\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Storing models in parallel\n",
    "<div class=\"alert alert-info\">\n",
    "<strong>Note</strong>\n",
    "\n",
    "This section is mostly for the developer so he doesn't forget why and how \n",
    "delayed updating of the model links was implemented.\n",
    "</div>\n",
    "\n",
    "We want to build and solve our time series models in 2 steps, first without a noise\n",
    "model and then with a noise model, and then store the result. We empty the models\n",
    "library to start from scratch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mEmptied library models in my_connector: <class 'pastastore.connectors.PasConnector'>\u001b[0m\n",
      "\u001b[32mEmptied library oseries_models in my_connector: <class 'pastastore.connectors.PasConnector'>\u001b[0m\n",
      "\u001b[32mEmptied library stresses_models in my_connector: <class 'pastastore.connectors.PasConnector'>\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "pstore.empty_library(\"models\", prompt=False, progressbar=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "def two_step_solve(name):\n",
    "    \"\"\"Solve a Pastas model in two steps: first without noise model, then with.\"\"\"\n",
    "    ml = pstore.create_model(name)\n",
    "    ml.solve(report=False)\n",
    "    ml.add_noisemodel(ps.ArNoiseModel())\n",
    "    ml.solve(initial=False, report=False)\n",
    "    pstore.add_model(ml, overwrite=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the first example we apply the function in parallel. A separate recomputation is\n",
    "performed after the parallel apply to update the links between the time series names\n",
    "and the models.\n",
    "\n",
    "The `conn._added_models` keeps track of added models so that the model links can be\n",
    "updated after all models have been added. In parallel mode, the child processes do not\n",
    "have access to this variable in the main thread, meaning it is not updated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pstore.conn._added_models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a0c23ad5e26b4d9cb9533da49443e4ce",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Computing two_step_solve (parallel):   0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pstore.apply(\"oseries\", two_step_solve, parallel=True, max_workers=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, the list remains empty:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pstore.conn._added_models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The parallel apply automatically updates the model links libraries, so the update flags\n",
    "should be equal to False."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pstore.conn._oseries_links_need_update.value = False\n",
      "pstore.conn._stresses_links_need_update.value = False\n"
     ]
    }
   ],
   "source": [
    "# check if update flags were reset after adding models links after parallel apply\n",
    "print(f\"{pstore.conn._oseries_links_need_update.value = }\")\n",
    "print(f\"{pstore.conn._stresses_links_need_update.value = }\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the `oseries_models` result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'head_nb5': ['head_nb5'],\n",
       " 'oseries2': ['oseries2'],\n",
       " 'head_mw': ['head_mw'],\n",
       " 'oseries1': ['oseries1'],\n",
       " 'oseries3': ['oseries3']}"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pstore.oseries_models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we repeat the process with `parallel=False`. Now the `_added_models` attribute can\n",
    "be updated properly since there is only the main instance of PastaStore.\n",
    "\n",
    "Once again, we empty the models library to start fresh."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mEmptied library models in my_connector: <class 'pastastore.connectors.PasConnector'>\u001b[0m\n",
      "\u001b[32mEmptied library oseries_models in my_connector: <class 'pastastore.connectors.PasConnector'>\u001b[0m\n",
      "\u001b[32mEmptied library stresses_models in my_connector: <class 'pastastore.connectors.PasConnector'>\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "pstore.empty_library(\"models\", prompt=False, progressbar=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fbaf40a90eb147f7a87bc1f07f4dbb32",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Computing two_step_solve:   0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pstore.apply(\"oseries\", two_step_solve, parallel=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `_added_models` attribute should now contain the names of all 5 models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['head_nb5', 'oseries2', 'head_mw', 'oseries1', 'oseries3']"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pstore.conn._added_models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The update flags should be set to True, which should trigger an update once we try to\n",
    "access the out-of-date data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pstore.conn._oseries_links_need_update.value = True\n",
      "pstore.conn._stresses_links_need_update.value = True\n"
     ]
    }
   ],
   "source": [
    "# check if update flags were reset after adding models links after parallel apply\n",
    "print(f\"{pstore.conn._oseries_links_need_update.value = }\")\n",
    "print(f\"{pstore.conn._stresses_links_need_update.value = }\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's trigger the update by looking at `oseries_models`. This will update the\n",
    "database, empty the `_added_models` attribute and set the update flags to False."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'head_nb5': ['head_nb5'],\n",
       " 'oseries2': ['oseries2'],\n",
       " 'head_mw': ['head_mw'],\n",
       " 'oseries1': ['oseries1'],\n",
       " 'oseries3': ['oseries3']}"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pstore.oseries_models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pstore.conn._added_models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pstore.conn._oseries_links_need_update.value = False\n",
      "pstore.conn._stresses_links_need_update.value = False\n"
     ]
    }
   ],
   "source": [
    "# check if update flags were reset after adding models links after parallel apply\n",
    "print(f\"{pstore.conn._oseries_links_need_update.value = }\")\n",
    "print(f\"{pstore.conn._stresses_links_need_update.value = }\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ArcticDBConnector workaround\n",
    "\n",
    "For `ArcticDBConnector`, the underlying database connection objects cannot be pickled, which is required for Python's multiprocessing. Therefore, passing methods directly from the `PastaStore` or `ArcticDBConnector` classes will not work in parallel mode.\n",
    "\n",
    "**The workaround:** The `_parallel()` method uses an initializer that creates a new connector instance in each worker process and stores it in a global `conn` variable. Your custom functions can then access this connector to retrieve data from the database.\n",
    "\n",
    "This is the standard Python pattern for using unpicklable objects with multiprocessing. See the [Python documentation](https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor) for more details.\n",
    "\n",
    "**Note:** If you need access to methods from the `PastaStore` class, just create a new one by passing it the global connector: `PastaStore(conn)`.\n",
    "\n",
    "**Example:** Write a simple function that uses the global `conn` variable to access the database:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Simple function to get models from database\n",
    "def get_model(model_name):\n",
    "    \"\"\"Get model using global connector (ArcticDBConnector workaround).\n",
    "\n",
    "    The global 'conn' variable is set by the _parallel() initializer\n",
    "    in each worker process, providing access to an ArcticDBConnector instance.\n",
    "    \"\"\"\n",
    "    pstore = pst.PastaStore(conn)\n",
    "    return pstore.get_model(model_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f8b9c7286b1d4d80964a724ea124d18a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Computing get_model (parallel):   0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "{'head_nb5': Model(oseries=head_nb5, name=head_nb5, constant=True, noisemodel=True),\n",
       " 'oseries2': Model(oseries=oseries2, name=oseries2, constant=True, noisemodel=True),\n",
       " 'head_mw': Model(oseries=head_mw, name=head_mw, constant=True, noisemodel=True),\n",
       " 'oseries1': Model(oseries=oseries1, name=oseries1, constant=True, noisemodel=True),\n",
       " 'oseries3': Model(oseries=oseries3, name=oseries3, constant=True, noisemodel=True)}"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pstore.apply(\"models\", get_model, fancy_output=True, parallel=True, max_workers=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clean up\n",
    "\n",
    "Clean up temporary pastastore."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mDeleting PasConnector database: 'my_connector' ... \u001b[0m\n",
      "\u001b[32mDone!\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "pst.util.delete_pastastore(pstore)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pastastore (3.13.11)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}