Create container #20

eduardocorrearaujo · 2022-10-04T13:46:50Z

This PR aims to solve issue #18.

I also make some changes on the dashboard:

Add the Q2 of 2022 in the scatter plot comparing the cases and hospitalizations in Geneva;
I change the map plot in the clustering section to a bokeh version, because the bokeh version allows the user to hover over the map and see the name of the canton and the exact value of the correlation. That's the new plot:

About the container: The docker version was taking too long to run on my local machine, and I wasn't able to test it. But, I had the same problem trying to run the streamit-template, so maybe it's a problem in my personal machine. So, please, try to run it on your machine and let me know if you have any problem.

To create the new files, I used as reference the Dockerfile in the streamlit-template repo and this tutorial from docker: https://www.docker.com/blog/how-to-dockerize-your-python-applications/

…ve map plot instead of the matplotlib map plot in the clustering.py section

added poetry.lock to container

fccoelho · 2022-10-04T16:50:18Z

Please build the container as indicated in the updated README, @eduardocorrearaujo .

The container should build normally it takes a while the first time due to the downloads.

eduardocorrearaujo · 2022-10-07T11:36:02Z

I am receiving this error when trying run the docker compose:

#0 27.99   • Installing xarray-einstats (0.3.0)
#0 29.62 
#0 29.62   CalledProcessError
#0 29.62 
#0 29.62   Command '['/root/.cache/pypoetry/virtualenvs/covid-ch-dashboard-9TtSrW0h-py3.9/bin/python', '-m', 'pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/root/.cache/pypoetry/virtualenvs/covid-ch-dashboard-9TtSrW0h-py3.9', '--no-deps', '/root/.cache/pypoetry/artifacts/9e/56/26/0b583bc07db61e0fc88eb894bb554daa81e2905ddbcda92ff923c5e3fe/Fiona-1.8.21.tar.gz']' returned non-zero exit status 1.
#0 29.62 
#0 29.62   at /usr/local/lib/python3.9/subprocess.py:528 in run
#0 29.68        524│             # We don't call process.wait() as .__exit__ does that for us.
#0 29.68        525│             raise
#0 29.68        526│         retcode = process.poll()
#0 29.68        527│         if check and retcode:
#0 29.68     →  528│             raise CalledProcessError(retcode, process.args,
#0 29.68        529│                                      output=stdout, stderr=stderr)
#0 29.68        530│     return CompletedProcess(process.args, retcode, stdout, stderr)
#0 29.68        531│ 
#0 29.68        532│ 
#0 29.68 
#0 29.68 The following error occurred when trying to handle this error:
#0 29.68 
#0 29.68 
#0 29.68   EnvCommandError
#0 29.68 
#0 29.68   Command ['/root/.cache/pypoetry/virtualenvs/covid-ch-dashboard-9TtSrW0h-py3.9/bin/python', '-m', 'pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/root/.cache/pypoetry/virtualenvs/covid-ch-dashboard-9TtSrW0h-py3.9', '--no-deps', '/root/.cache/pypoetry/artifacts/9e/56/26/0b583bc07db61e0fc88eb894bb554daa81e2905ddbcda92ff923c5e3fe/Fiona-1.8.21.tar.gz'] errored with the following return code 1, and output: 
#0 29.68   Processing /root/.cache/pypoetry/artifacts/9e/56/26/0b583bc07db61e0fc88eb894bb554daa81e2905ddbcda92ff923c5e3fe/Fiona-1.8.21.tar.gz
#0 29.68     Installing build dependencies: started
#0 29.68     Installing build dependencies: finished with status 'done'
#0 29.68     Getting requirements to build wheel: started
#0 29.68     Getting requirements to build wheel: finished with status 'error'
#0 29.68     error: subprocess-exited-with-error
#0 29.68     
#0 29.68     × Getting requirements to build wheel did not run successfully.
#0 29.68     │ exit code: 1
#0 29.68     ╰─> [2 lines of output]
#0 29.68         Failed to get options via gdal-config: [Errno 2] No such file or directory: 'gdal-config'
#0 29.68         A GDAL API version must be specified. Provide a path to gdal-config using a GDAL_CONFIG environment variable or use a GDAL_VERSION environment variable.
#0 29.68         [end of output]
#0 29.68     
#0 29.68     note: This error originates from a subprocess, and is likely not a problem with pip.
#0 29.68   error: subprocess-exited-with-error
#0 29.68   
#0 29.68   × Getting requirements to build wheel did not run successfully.
#0 29.68   │ exit code: 1
#0 29.68   ╰─> See above for output.
#0 29.68   
#0 29.68   note: This error originates from a subprocess, and is likely not a problem with pip.
#0 29.68   
#0 29.68 
#0 29.68   at /usr/local/lib/python3.9/site-packages/poetry/utils/env.py:1476 in _run
#0 29.72       1472│                 output = subprocess.check_output(
#0 29.72       1473│                     command, stderr=subprocess.STDOUT, env=env, **kwargs
#0 29.72       1474│                 )
#0 29.72       1475│         except CalledProcessError as e:
#0 29.72     → 1476│             raise EnvCommandError(e, input=input_)
#0 29.72       1477│ 
#0 29.72       1478│         return decode(output)
#0 29.72       1479│ 
#0 29.72       1480│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:
#0 29.72 
#0 29.72 The following error occurred when trying to handle this error:
#0 29.72 
#0 29.72 
#0 29.72   PoetryException
#0 29.72 
#0 29.72   Failed to install /root/.cache/pypoetry/artifacts/9e/56/26/0b583bc07db61e0fc88eb894bb554daa81e2905ddbcda92ff923c5e3fe/Fiona-1.8.21.tar.gz
#0 29.72 
#0 29.72   at /usr/local/lib/python3.9/site-packages/poetry/utils/pip.py:51 in pip_install
#0 29.73        47│ 
#0 29.73        48│     try:
#0 29.73        49│         return environment.run_pip(*args)
#0 29.73        50│     except EnvCommandError as e:
#0 29.73     →  51│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
#0 29.73        52│ 
#0 29.73 
------
failed to solve: executor failed running [/bin/sh -c poetry install]: exit code: 1

fccoelho · 2022-10-07T13:32:56Z

Installation of the GDAL library must be added to the Dockerfile. @esloch , could help here? I think it is just a matter of installing GDAL either via apt or mamba.

eduardocorrearaujo · 2022-10-10T11:49:24Z

I tried poetry add gdal but it returns an error too.

xmnlab · 2022-10-10T11:52:51Z

Gdal is a very special one. I would install it via mamba if possible. Btw, what is the error when you installed that via poetry?

eduardocorrearaujo · 2022-10-10T11:56:02Z

The error is: Updating idna (3.3 -> 3.4): Failed

To add the gdal in the Dockerfile I should add the line:
RUN mamba install gdal
?

fccoelho · 2022-10-10T12:17:26Z

GDAL is a c++ library and thus can't be pip installed. mamba shoul be able to do it.

esloch · 2022-10-10T12:49:46Z

I suggest you pass requirements.txt to a .yaml environment file and use conda/mamba to resolve GDAL dependencies.
Example

xmnlab · 2022-10-10T13:04:52Z

you also can install libgdal fom conda-forge and install gdal from pypi. but probably the recommended way would be install ing everything from conda-forge as mentioned by @fccoelho

and as this dashboard is not a library, it makes sense to use the approach pointed by @esloch

eduardocorrearaujo · 2022-10-10T14:16:23Z

Thank you. I will work on that today.

…zed the codes in diferent folders

eduardocorrearaujo · 2022-10-12T18:50:49Z

I am facing this error to run the app in the container:


OperationalError: (psycopg2.OperationalError) connection to server at "localhost" (127.0.0.1), port 5432 failed: Connection refused Is the server running on that host and accepting TCP/IP connections? connection to server at "localhost" (::1), port 5432 failed: Cannot assign requested address Is the server running on that host and accepting TCP/IP connections? (Background on this error at: https://sqlalche.me/e/14/e3q8)

I found this issue discussing it, but I would like to ask you what is the common workflow in this case:
https://stackoverflow.com/questions/31249112/allow-docker-container-to-connect-to-a-local-host-postgres-database

esloch · 2022-10-12T21:23:28Z

@eduardocorrearaujo. Are you trying to connect to your local database from the container?
I think you need to set psql_host and psql_port in the environment to be able to access outside the docker network.

eduardocorrearaujo · 2022-10-13T03:00:56Z

@eduardocorrearaujo. Are you trying to connect to your local database from the container? I think you need to set psql_host and psql_port in the environment to be able to access outside the docker network.

No, I am trying to connect to the epigraphhub database.

esloch · 2022-10-13T11:01:33Z

Eduardo, vi que usas repetidas vezes a URI de conexão chumbada nos módulos:
engine = create_engine("postgresql://epigraph:epigraph@localhost:5432/epigraphhub")
Talvez centralizar uma função para a conexão na base de dados em um arquivo settings.py, ou usar um arquivo ".env" com as variáveis de conexão na base de dados, assim evitas chumbar as credenciais dentro dos módulos.
Nesse caso, se conectas a um host externo ao contêiner necessitarás informar para o container a porta e o ip de conexão ao qual queres acessar por meio do environment no docker-compose, que permite você adicionar "env variables" ao seu contêiner.

esloch · 2022-10-24T10:55:19Z

app/SIR_forecast.py

+from dotenv import load_dotenv
+load_dotenv('../.env')
+
+engine = create_engine(f'postgresql://{os.getenv("POSTGRES_USER")}:{os.getenv("POSTGRES_PASSWORD")}@{os.getenv("POSTGRES_HOST")}:{os.getenv("POSTGRES_PORT")}/{os.getenv("POSTGRES_DB")}')


Perhaps you can use the same approach here...
Creating config.py, to avoid calling connection variables more than once.

esloch · 2022-10-24T10:55:25Z

app/bayesian_inference.py

+from dotenv import load_dotenv
+load_dotenv('../.env')
+
+engine = create_engine(f'postgresql://{os.getenv("POSTGRES_USER")}:{os.getenv("POSTGRES_PASSWORD")}@{os.getenv("POSTGRES_HOST")}:{os.getenv("POSTGRES_PORT")}/{os.getenv("POSTGRES_DB")}')


Same config.py here.

esloch · 2022-10-24T10:55:51Z

app/for_other_cantons.py

+from dotenv import load_dotenv
+load_dotenv('../.env')
+
+engine = create_engine(f'postgresql://{os.getenv("POSTGRES_USER")}:{os.getenv("POSTGRES_PASSWORD")}@{os.getenv("POSTGRES_HOST")}:{os.getenv("POSTGRES_PORT")}/{os.getenv("POSTGRES_DB")}')


Same config.py here.

esloch · 2022-10-24T10:55:56Z

app/forecast.py


+engine = create_engine(f'postgresql://{os.getenv("POSTGRES_USER")}:{os.getenv("POSTGRES_PASSWORD")}@{os.getenv("POSTGRES_HOST")}:{os.getenv("POSTGRES_PORT")}/{os.getenv("POSTGRES_DB")}')


Same config.py here.

esloch · 2022-10-24T10:56:03Z

app/get_data.py


-engine_public = create_engine("postgresql://epigraph:epigraph@localhost:5432/epigraphhub")
+engine = create_engine(f'postgresql://{os.getenv("POSTGRES_USER")}:{os.getenv("POSTGRES_PASSWORD")}@{os.getenv("POSTGRES_HOST")}:{os.getenv("POSTGRES_PORT")}/{os.getenv("POSTGRES_DB")}')


Same config.py here.

esloch · 2022-10-27T18:30:24Z

app/config.py

+POSTGRES_PASSWORD = os.getenv("POSTGRES_PASSWORD")
+POSTGRES_HOST = os.getenv("POSTGRES_HOST")
+POSTGRES_PORT = os.getenv("POSTGRES_PORT")
+POSTGRES_DB = os.getenv("POSTGRES_DB")


The idea would be to keep the call of all variables just inside that config file with like this:

DB_URI = ( f"postgresql://{DB_USER}:{DB_PASSWORD}" f"@{DB_HOST}:{DB_PORT}/{DB_NAME}" )

and in the modules use like this:

engine = create_engine(config.DB_URI)

eduardocorrearaujo and others added 5 commits October 4, 2022 07:42

Update the scatter plot to show the Q2-2022 and add a bokeh interacti…

8db9657

…ve map plot instead of the matplotlib map plot in the clustering.py section

addedfiles to allow the creation of the docker container

d92542b

added missing docker-compose file

6840fee

Updated README with docker-compose instructions

79286aa

Updated dependencies in pyproject.toml

6f078c3

added poetry.lock to container

Added missing requirements

fe84745

Updated readme, added download of dependencies by conda, and reorgani…

bff2d6b

…zed the codes in diferent folders

Minor issues

b7fb448

esloch requested changes Oct 24, 2022

View reviewed changes

Add config file and some dependencies to be installed under mamba

d858610

esloch reviewed Oct 27, 2022

View reviewed changes

Create DB_URI

9dd291b

eduardocorrearaujo force-pushed the create_container branch from 510f005 to 9dd291b Compare October 27, 2022 19:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create container #20

Create container #20

eduardocorrearaujo commented Oct 4, 2022

fccoelho commented Oct 4, 2022 •

edited

Loading

eduardocorrearaujo commented Oct 7, 2022 •

edited

Loading

fccoelho commented Oct 7, 2022

eduardocorrearaujo commented Oct 10, 2022

xmnlab commented Oct 10, 2022

eduardocorrearaujo commented Oct 10, 2022 •

edited

Loading

fccoelho commented Oct 10, 2022

esloch commented Oct 10, 2022 •

edited

Loading

xmnlab commented Oct 10, 2022

eduardocorrearaujo commented Oct 10, 2022

eduardocorrearaujo commented Oct 12, 2022 •

edited

Loading

esloch commented Oct 12, 2022 •

edited

Loading

eduardocorrearaujo commented Oct 13, 2022

esloch commented Oct 13, 2022

esloch Oct 24, 2022

esloch Oct 24, 2022

esloch Oct 24, 2022

esloch Oct 24, 2022

esloch Oct 24, 2022

esloch Oct 27, 2022

eduardocorrearaujo Oct 27, 2022


		engine = create_engine(f'postgresql://{os.getenv("POSTGRES_USER")}:{os.getenv("POSTGRES_PASSWORD")}@{os.getenv("POSTGRES_HOST")}:{os.getenv("POSTGRES_PORT")}/{os.getenv("POSTGRES_DB")}')


		engine_public = create_engine("postgresql://epigraph:epigraph@localhost:5432/epigraphhub")
		engine = create_engine(f'postgresql://{os.getenv("POSTGRES_USER")}:{os.getenv("POSTGRES_PASSWORD")}@{os.getenv("POSTGRES_HOST")}:{os.getenv("POSTGRES_PORT")}/{os.getenv("POSTGRES_DB")}')

Create container #20

Are you sure you want to change the base?

Create container #20

Conversation

eduardocorrearaujo commented Oct 4, 2022

fccoelho commented Oct 4, 2022 • edited Loading

eduardocorrearaujo commented Oct 7, 2022 • edited Loading

fccoelho commented Oct 7, 2022

eduardocorrearaujo commented Oct 10, 2022

xmnlab commented Oct 10, 2022

eduardocorrearaujo commented Oct 10, 2022 • edited Loading

fccoelho commented Oct 10, 2022

esloch commented Oct 10, 2022 • edited Loading

xmnlab commented Oct 10, 2022

eduardocorrearaujo commented Oct 10, 2022

eduardocorrearaujo commented Oct 12, 2022 • edited Loading

esloch commented Oct 12, 2022 • edited Loading

eduardocorrearaujo commented Oct 13, 2022

esloch commented Oct 13, 2022

esloch Oct 24, 2022

Choose a reason for hiding this comment

esloch Oct 24, 2022

Choose a reason for hiding this comment

esloch Oct 24, 2022

Choose a reason for hiding this comment

esloch Oct 24, 2022

Choose a reason for hiding this comment

esloch Oct 24, 2022

Choose a reason for hiding this comment

esloch Oct 27, 2022

Choose a reason for hiding this comment

eduardocorrearaujo Oct 27, 2022

Choose a reason for hiding this comment

fccoelho commented Oct 4, 2022 •

edited

Loading

eduardocorrearaujo commented Oct 7, 2022 •

edited

Loading

eduardocorrearaujo commented Oct 10, 2022 •

edited

Loading

esloch commented Oct 10, 2022 •

edited

Loading

eduardocorrearaujo commented Oct 12, 2022 •

edited

Loading

esloch commented Oct 12, 2022 •

edited

Loading