GitLab has an advanced log system where everything is logged, so you can analyze your instance using various system log
files. The log system is similar to
audit events
.
System log files are typically plain text in a standard log file format.
This guide talks about how to read and use these system log files.
Each line contains JSON that can be ingested by services like Elasticsearch and Splunk.
Line breaks were added to examples for legibility:
"method":"GET","path":"/gitlab/gitlab-foss/issues/1234","format":"html","controller":"Projects::IssuesController","action":"show","status":200,"time":"2017-08-08T20:15:54.821Z","params":[{"key":"param_key","value":"param_value"}],"remote_ip":"18.245.0.1","user_id":1,"username":"admin","queue_duration_s":0.0,"gitaly_calls":16,"gitaly_duration_s":0.16,"redis_calls":115,"redis_duration_s":0.13,"redis_read_bytes":1507378,"redis_write_bytes":2920,"correlation_id":"O1SdybnnIq7","cpu_s":17.50,"db_duration_s":0.08,"view_duration_s":2.39,"duration_s":20.54,"pid":81836,"worker_id":"puma_0"
This example was a GET request for a specific
issue. Each line also contains performance data, with times in
seconds:
duration_s: Total time to retrieve the request
queue_duration_s: Total time the request was queued inside GitLab Workhorse
view_duration_s: Total time inside the Rails views
db_duration_s: Total time to retrieve data from PostgreSQL
cpu_s: Total time spent on CPU
gitaly_duration_s: Total time by Gitaly calls
gitaly_calls: Total number of calls made to Gitaly
redis_calls: Total number of calls made to Redis
redis_cross_slot_calls: Total number of cross-slot calls made to Redis
redis_allowed_cross_slot_calls: Total number of allowed cross-slot calls made to Redis
redis_duration_s: Total time to retrieve data from Redis
redis_read_bytes: Total bytes read from Redis
redis_write_bytes: Total bytes written to Redis
redis_<instance>_calls: Total number of calls made to a Redis instance
redis_<instance>_cross_slot_calls: Total number of cross-slot calls made to a Redis instance
redis_<instance>_allowed_cross_slot_calls: Total number of allowed cross-slot calls made to a Redis instance
redis_<instance>_duration_s: Total time to retrieve data from a Redis instance
redis_<instance>_read_bytes: Total bytes read from a Redis instance
redis_<instance>_write_bytes: Total bytes written to a Redis instance
pid: The worker’s Linux process ID (changes when workers restart)
worker_id: The worker’s logical ID (does not change when workers restart)
User clone and fetch activity using HTTP transport appears in the log as action: git_upload_pack.
In addition, the log contains the originating IP address,
(remote_ip), the user’s ID (user_id), and username (username).
Some endpoints (such as /search) may make requests to Elasticsearch if using
advanced search. These
additionally log elasticsearch_calls and elasticsearch_call_duration_s,
which correspond to:
elasticsearch_calls: Total number of calls to Elasticsearch
elasticsearch_duration_s: Total time taken by Elasticsearch calls
elasticsearch_timed_out_count: Total number of calls to Elasticsearch that
timed out and therefore returned partial results
ActionCable connection and subscription events are also logged to this file and they follow the
previous format. The method, path, and format fields are not applicable, and are always empty.
The ActionCable connection or channel class is used as the controller.
"method":null,"path":null,"format":null,"controller":"IssuesChannel","action":"subscribe","status":200,"time":"2020-05-14T19:46:22.008Z","params":[{"key":"project_path","value":"gitlab/gitlab-foss"},{"key":"iid","value":"1"}],"remote_ip":"127.0.0.1","user_id":1,"username":"admin","ua":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:76.0) Gecko/20100101 Firefox/76.0","correlation_id":"jSOIEynHCUa","duration_s":0.32566
If an error occurs, an
exception field is included with class, message, and
backtrace. Previous versions included an error field instead of
exception.class and exception.message. For example:
"method":"GET","path":"/admin","format":"html","controller":"Admin::DashboardController","action":"index","status":500,"time":"2019-11-14T13:12:46.156Z","params":[],"remote_ip":"127.0.0.1","user_id":1,"username":"root","ua":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:70.0) Gecko/20100101 Firefox/70.0","queue_duration":274.35,"correlation_id":"KjDVUhNvvV3","queue_duration_s":0.0,"gitaly_calls":16,"gitaly_duration_s":0.16,"redis_calls":115,"redis_duration_s":0.13,"correlation_id":"O1SdybnnIq7","cpu_s":17.50,"db_duration_s":0.08,"view_duration_s":2.39,"duration_s":20.54,"pid":81836,"worker_id":"puma_0","exception.class":"NameError","exception.message":"undefined local variable or method `adsf' for #<Admin::DashboardController:0x00007ff3c9648588>","exception.backtrace":["app/controllers/admin/dashboard_controller.rb:11:in `index'","ee/app/controllers/ee/admin/dashboard_controller.rb:14:in `index'","ee/lib/gitlab/ip_address_state.rb:10:in `with'","ee/app/controllers/ee/application_controller.rb:43:in `set_current_ip_address'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_controller.rb:450:in `set_session_storage'","app/controllers/application_controller.rb:444:in `set_locale'","ee/lib/gitlab/jira/middleware.rb:19:in `call'"production.log
As of Grape Logging v1.8.4,
the view_duration_s is calculated by duration_s - db_duration_s.
Therefore, view_duration_s can be affected by multiple different factors, like read-write
process on Redis or external HTTP, not only the serialization process.
October 06, 2014 11:56: User "Administrator" (admin@example.com) was created
October 06, 2014 11:56: Documentcloud created a new project "Documentcloud / Underscore"
October 06, 2014 11:56: Gitlab Org created a new project "Gitlab Org / Gitlab Ce"
October 07, 2014 11:25: User "Claudie Hodkiewicz" (nasir_stehr@olson.co.uk) was removed
October 07, 2014 11:25: Project "project133" was removed
application_json.log
It contains information about integration
activities, such as Jira, Asana, and irker services. It uses JSON format,
like this example:
"severity":"ERROR","time":"2018-09-06T14:56:20.439Z","service_class":"Integrations::Jira","project_id":8,"project_path":"h5bp/html5-boilerplate","message":"Error sending message","client_url":"http://jira.gitlap.com:8080","error":"execution expired""severity":"INFO","time":"2018-09-06T17:15:16.365Z","service_class":"Integrations::Jira","project_id":3,"project_path":"namespace2/project2","message":"Successfully posted","client_url":"http://jira.example.com"kubernetes.log (deprecated)History/var/log/gitlab/gitlab-rails/kubernetes.log on Linux package installations.
/home/git/gitlab/log/kubernetes.log on self-compiled installations.
This file is located at:
/var/log/gitlab/sidekiq/current
on Linux package installations.
/home/git/gitlab/log/sidekiq.log
on self-compiled installations.
GitLab uses background jobs for processing tasks which can take a long
time. All information about processing these jobs are written to this
file. For example:
"severity"
:
"INFO"
,
"time"
:
"2018-04-03T22:57:22.071Z"
,
"queue"
:
"cronjob:update_all_mirrors"
,
"args"
:[],
"class"
:
"UpdateAllMirrorsWorker"
,
"retry"
:
false
,
"queue_namespace"
:
"cronjob"
,
"jid"
:
"06aeaa3b0aadacf9981f368e"
,
"created_at"
:
"2018-04-03T22:57:21.930Z"
,
"enqueued_at"
:
"2018-04-03T22:57:21.931Z"
,
"pid"
:
10077
,
"worker_id"
:
"sidekiq_0"
,
"message"
:
"UpdateAllMirrorsWorker JID-06aeaa3b0aadacf9981f368e: done: 0.139 sec"
,
"job_status"
:
"done"
,
"duration"
:
0.139
,
"completed_at"
:
"2018-04-03T22:57:22.071Z"
,
"db_duration"
:
0.05
,
"db_duration_s"
:
0.0005
,
"gitaly_duration"
:
0
,
"gitaly_calls"
:
0
Instead of JSON logs, you can opt to generate text logs for Sidekiq. For example:
2023-05-16T16:08:55.272Z pid=82525 tid=23rl INFO: Initializing websocket
2023-05-16T16:08:55.279Z pid=82525 tid=23rl INFO: Booted Rails 6.1.7.2 application in production environment
2023-05-16T16:08:55.279Z pid=82525 tid=23rl INFO: Running in ruby 3.0.5p211 (2022-11-24 revision ba5cf0f7c5) [arm64-darwin22]
2023-05-16T16:08:55.279Z pid=82525 tid=23rl INFO: See LICENSE and the LGPL-3.0 for licensing details.
2023-05-16T16:08:55.279Z pid=82525 tid=23rl INFO: Upgrade to Sidekiq Pro for more features and support: https://sidekiq.org
2023-05-16T16:08:55.286Z pid=82525 tid=7p4t INFO: Cleaning working queues
2023-05-16T16:09:06.043Z pid=82525 tid=7p7d class=ScheduleMergeRequestCleanupRefsWorker jid=efcc73f169c09a514b06da3f INFO: start
2023-05-16T16:09:06.050Z pid=82525 tid=7p7d class=ScheduleMergeRequestCleanupRefsWorker jid=efcc73f169c09a514b06da3f INFO: arguments: []
2023-05-16T16:09:06.065Z pid=82525 tid=7p81 class=UserStatusCleanup::BatchWorker jid=e279aa6409ac33031a314822 INFO: start
2023-05-16T16:09:06.066Z pid=82525 tid=7p81 class=UserStatusCleanup::BatchWorker jid=e279aa6409ac33031a314822 INFO: arguments: []
For Linux package installations, add the configuration option:
sidekiq['log_format']='text'
For self-compiled installations, edit the gitlab.yml and set the Sidekiq
log_format configuration option:
{"severity":"INFO","time":"2020-11-24T02:30:59.860Z","correlation_id":null,"key":"cd_auto_rollback","action":"enable","extra.thing":"true"}{"severity":"INFO","time":"2020-11-24T02:31:29.108Z","correlation_id":null,"key":"cd_auto_rollback","action":"enable","extra.thing":"true"}{"severity":"INFO","time":"2020-11-24T02:31:29.129Z","correlation_id":null,"key":"cd_auto_rollback","action":"disable","extra.thing":"false"}{"severity":"INFO","time":"2020-11-24T02:31:29.177Z","correlation_id":null,"key":"cd_auto_rollback","action":"enable","extra.thing":"Project:1"}{"severity":"INFO","time":"2020-11-24T02:31:29.183Z","correlation_id":null,"key":"cd_auto_rollback","action":"disable","extra.thing":"Project:1"}{"severity":"INFO","time":"2020-11-24T02:31:29.188Z","correlation_id":null,"key":"cd_auto_rollback","action":"enable_percentage_of_time","extra.percentage":"50"}{"severity":"INFO","time":"2020-11-24T02:31:29.193Z","correlation_id":null,"key":"cd_auto_rollback","action":"disable_percentage_of_time"}{"severity":"INFO","time":"2020-11-24T02:31:29.198Z","correlation_id":null,"key":"cd_auto_rollback","action":"enable_percentage_of_actors","extra.percentage":"50"}{"severity":"INFO","time":"2020-11-24T02:31:29.203Z","correlation_id":null,"key":"cd_auto_rollback","action":"disable_percentage_of_actors"}{"severity":"INFO","time":"2020-11-24T02:31:29.329Z","correlation_id":null,"key":"cd_auto_rollback","action":"remove"}ci_resource_groups_json.logHistory/var/log/gitlab/gitlab-rails/ci_resource_group_json.log on Linux package installations.
/home/git/gitlab/log/ci_resource_group_json.log on self-compiled installations.
{"severity":"INFO","time":"2023-02-10T23:02:06.095Z","correlation_id":"01GRYS10C2DZQ9J1G12ZVAD4YD","resource_group_id":1,"processable_id":288,"message":"attempted to assign resource to processable","success":true}{"severity":"INFO","time":"2023-02-10T23:02:08.945Z","correlation_id":"01GRYS138MYEG32C0QEWMC4BDM","resource_group_id":1,"processable_id":288,"message":"attempted to release resource from processable","success":true}
The examples show the resource_group_id, processable_id, message, and success fields for each entry.
auth.log
When troubleshooting issues that aren’t localized to one of the
previously listed components, it’s helpful to simultaneously gather multiple logs and statistics
from a GitLab instance.
GitLab Support often asks for one of these, and maintains the required tools.
Fast-stats is a tool
for creating and comparing performance statistics from GitLab logs.
For more details and instructions to run it, read the
documentation for fast-stats.
View pricing
to see all GitLab tiers and features, or to upgrade. Try GitLab for free
with access to all features for 30 days. search the docs.
If you want help with something specific and could use community support,
post on the GitLab forum.
For problems setting up or using this feature (depending on your GitLab
subscription). Request support
