Collectives™ on Stack Overflow
Find centralized, trusted content and collaborate around the technologies you use most.
Learn more about Collectives
Teams
Q&A for work
Connect and share knowledge within a single location that is structured and easy to search.
Learn more about Teams
I have a very simple python script test.py using mlflow (for those unfamiliar with it, it's a library used, among other things, to track machine learning experiments: basically some metadata and metrics are saved in a sqlalchemy compatible db, e.g. sqlite, and eventually visualized in a browser user interface)
import mlflow
import random
mlflow.set_tracking_uri("sqlite:///mlflow.db")
experiment_id = mlflow.set_experiment('test_experiment')
with mlflow.start_run() as run:
for i in range(10):
mlflow.log_metric(key='mse', value=i*random.random(), step=i)
When I execute it I receive the error
(sqlite3.OperationalError) database is locked
Below the full error
2023/03/12 19:34:17 INFO mlflow.store.db.utils: Creating initial MLflow database tables...
Traceback (most recent call last):
File "/jupyter/venvs/myenv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
self.dialect.do_execute(
File "/jupyter/venvs/myenv/lib64/python3.8/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: database is locked
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "test.py", line 10, in <module>
with mlflow.start_run() as run:
File "/jupyter/venvs/myenv/lib64/python3.8/site-packages/mlflow/tracking/fluent.py", line 278, in start_run
client = MlflowClient()
File "/jupyter/venvs/myenv/lib64/python3.8/site-packages/mlflow/tracking/client.py", line 66, in __init__
self._tracking_client = TrackingServiceClient(final_tracking_uri)
File "/jupyter/venvs/myenv/lib64/python3.8/site-packages/mlflow/tracking/_tracking_service/client.py", line 51, in __init__
self.store
File "/jupyter/venvs/myenv/lib64/python3.8/site-packages/mlflow/tracking/_tracking_service/client.py", line 55, in store
return utils._get_store(self.tracking_uri)
File "/jupyter/venvs/myenv/lib64/python3.8/site-packages/mlflow/tracking/_tracking_service/utils.py", line 188, in _get_store
return _tracking_store_registry.get_store(store_uri, artifact_uri)
File "/jupyter/venvs/myenv/lib64/python3.8/site-packages/mlflow/tracking/_tracking_service/registry.py", line 39, in get_store
return self._get_store_with_resolved_uri(resolved_store_uri, artifact_uri)
File "/jupyter/venvs/myenv/lib64/python3.8/site-packages/mlflow/tracking/_tracking_service/registry.py", line 49, in _get_store_with_resolved_uri
return builder(store_uri=resolved_store_uri, artifact_uri=artifact_uri)
File "/jupyter/venvs/myenv/lib64/python3.8/site-packages/mlflow/tracking/_tracking_service/utils.py", line 149, in _get_sqlalchemy_store
return SqlAlchemyStore(store_uri, artifact_uri)
File "/jupyter/venvs/myenv/lib64/python3.8/site-packages/mlflow/store/tracking/sqlalchemy_store.py", line 145, in __init__
mlflow.store.db.utils._initialize_tables(self.engine)
File "/jupyter/venvs/myenv/lib64/python3.8/site-packages/mlflow/store/db/utils.py", line 49, in _initialize_tables
InitialBase.metadata.create_all(engine)
File "/jupyter/venvs/myenv/lib64/python3.8/site-packages/sqlalchemy/sql/schema.py", line 4930, in create_all
bind._run_ddl_visitor(
File "/jupyter/venvs/myenv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 3228, in _run_ddl_visitor
conn._run_ddl_visitor(visitorcallable, element, **kwargs)
File "/jupyter/venvs/myenv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 2211, in _run_ddl_visitor
visitorcallable(self.dialect, self, **kwargs).traverse_single(element)
File "/jupyter/venvs/myenv/lib64/python3.8/site-packages/sqlalchemy/sql/visitors.py", line 524, in traverse_single
return meth(obj, **kw)
File "/jupyter/venvs/myenv/lib64/python3.8/site-packages/sqlalchemy/sql/ddl.py", line 851, in visit_metadata
self.traverse_single(
File "/jupyter/venvs/myenv/lib64/python3.8/site-packages/sqlalchemy/sql/visitors.py", line 524, in traverse_single
return meth(obj, **kw)
File "/jupyter/venvs/myenv/lib64/python3.8/site-packages/sqlalchemy/sql/ddl.py", line 895, in visit_table
self.connection.execute(
File "/jupyter/venvs/myenv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1380, in execute
return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
File "/jupyter/venvs/myenv/lib64/python3.8/site-packages/sqlalchemy/sql/ddl.py", line 80, in _execute_on_connection
return connection._execute_ddl(
File "/jupyter/venvs/myenv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1472, in _execute_ddl
ret = self._execute_context(
File "/jupyter/venvs/myenv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
self._handle_dbapi_exception(
File "/jupyter/venvs/myenv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 2124, in _handle_dbapi_exception
util.raise_(
File "/jupyter/venvs/myenv/lib64/python3.8/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
raise exception
File "/jupyter/venvs/myenv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
self.dialect.do_execute(
File "/jupyter/venvs/myenv/lib64/python3.8/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked
[SQL:
CREATE TABLE experiments (
experiment_id INTEGER NOT NULL,
name VARCHAR(256) NOT NULL,
artifact_location VARCHAR(256),
lifecycle_stage VARCHAR(32),
CONSTRAINT experiment_pk PRIMARY KEY (experiment_id),
CONSTRAINT experiments_lifecycle_stage CHECK (lifecycle_stage IN ('active', 'deleted')),
UNIQUE (name)
(Background on this error at: https://sqlalche.me/e/14/e3q8)
I'm not an expert of sqlite, but I'm sure the database isn't locked for real: the db_file mlflow.db doesn't even exist when I run the python script (it is created by the script itself).
I get this error only on a specific Linux machine (of which I'm not an admin), below some details:
cat /etc/os-release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.9 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.9"
PRETTY_NAME="Red Hat Enterprise Linux"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.9:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.9
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.9"
uname -ai
Linux ##censored## 3.10.0-1160.45.1.el7.x86_64 #1 SMP Fri Sep 24 10:17:16 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
If I run the same script on my personal, local Ubuntu machine or even on windows I don't get such error. I used mlflow with sqlite backend many times and on different machines/OS/python versions/environments, (but never on Red Hat Enterprise Linux 7). This is the first time I get this error and on the internet I couldn't figure out not even a hint of why it could happen. Some help would be appreciated
Below the details to reproduce the script
Python 3.8.11
Requirements.txt (I couldn't test on a smaller venv with only mlflow installed because on that machine I can't even create a virtual env, So there are other packages non essential for our problem. If it can help, sqlite wasn't installed by its own, but only mlflow was installed and, as a dependency , SQlalchemy was installed)
absl-py==1.4.0
alembic==1.9.3
asttokens==2.2.1
backcall==0.2.0
certifi==2022.12.7
charset-normalizer==3.0.1
click==8.1.3
cloudpickle==2.2.1
comm==0.1.2
contourpy==1.0.7
cvxpy==1.3.0
cycler==0.11.0
databricks-cli==0.17.4
dataclasses==0.6
debugpy==1.6.6
decorator==5.1.1
docker==6.0.1
ecos==2.0.12
entrypoints==0.4
et-xmlfile==1.1.0
executing==1.2.0
Flask==2.2.2
fonttools==4.38.0
future==0.18.3
gitdb==4.0.10
GitPython==3.1.30
greenlet==2.0.2
gunicorn==20.1.0
hyperopt==0.2.7
idna==3.4
importlib-metadata==5.2.0
importlib-resources==5.10.2
ipykernel==6.15.2
ipython==7.34.0
itsdangerous==2.1.2
jedi==0.18.2
Jinja2==3.1.2
joblib==1.2.0
jupyter_client==7.4.9
jupyter_core==5.2.0
kiwisolver==1.4.4
Mako==1.2.4
MarkupSafe==2.1.2
matplotlib==3.7.0
matplotlib-inline==0.1.6
mlflow==1.30.0
nest-asyncio==1.5.6
networkx==3.0
numpy==1.24.2
oauthlib==3.2.2
openpyxl==3.1.1
optbinning==0.17.3
ortools==9.5.2237
osqp==0.6.2.post8
packaging==21.3
pandas==1.5.3
parso==0.8.3
patsy==0.5.3
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.4.0
platformdirs==3.0.0
plotly==5.13.0
prometheus-client==0.16.0
prometheus-flask-exporter==0.21.0
prompt-toolkit==3.0.36
protobuf==4.21.12
psutil==5.9.4
ptyprocess==0.7.0
pure-eval==0.2.2
py4j==0.10.9.7
Pygments==2.14.0
PyJWT==2.6.0
pyparsing==3.0.9
python-dateutil==2.8.2
pytz==2022.7.1
PyYAML==5.4.1
pyzmq==25.0.0
qdldl==0.1.5.post3
querystring-parser==1.2.4
requests==2.28.2
ropwr==1.0.0
scikit-learn==1.2.1
scipy==1.10.0
scs==3.2.2
six==1.16.0
smmap==5.0.0
SQLAlchemy==1.4.46
sqlparse==0.4.3
stack-data==0.6.2
statsmodels==0.13.5
tabulate==0.9.0
tenacity==8.2.1
threadpoolctl==3.1.0
tornado==6.2
tqdm==4.64.1
traitlets==5.9.0
typing==3.7.4.3
urllib3==1.26.14
wcwidth==0.2.6
websocket-client==1.5.1
Werkzeug==2.2.3
xgboost==1.7.4
zipp==3.13.0
If the script is run without errors, the user interface can be activated from a terminal (with the virtual environment activated, and positioned at the level of the test.py file) with
mlflow ui --backend-store-uri sqlite:///mlflow.db
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
