1. 前言最近我写了一些使用OpenTelemetry监控几种编程语言(比如JavaNode.JS)的文档还有跨编程语言的OBI方式。当然这些文章都只介绍了非侵入式的方式在OpenTelemetry里叫做Automatic instrumentation(自动插桩)现在又叫做Zero-code instrumentation(零代码插桩)。 其实几乎每种编程语言都有使用OpenTelemetry SDK编程的侵入式方式适于平台开发者、商品软件提供者和希望提供精品应用的开发者可以参考OpenTelemetry的文档。这篇文章再介绍一下如何使用OpenTelemetry来监控Python应用当然也是只讲非侵入式的方式做起来也非常容易。2. 启动OpenTelemetry的后端工具OpenTelemetry的后端工具就是支持OpenTelemetry metrics/traces/logs/profiles的数据库和UI的工具集。最常见的做法还是使用OpenTelemetry Collector来连接不同的后端工具。假如您是初学者或者系统很小可以直接使用基于Docker的Grafana LGTM几乎是一键安装完成非常简单易行。前提是需要您有个支持Docker的环境。假设您想把LGTM安装到/opt/lgtm目录 (任何目录均可)下面是命令(假设在Linux系统)docker pull grafana/otel-lgtm mkdir /opt/lgtm cd /opt/lgtm wget https://raw.githubusercontent.com/grafana/docker-otel-lgtm/main/run-lgtm.sh chmod x run-lgtm.sh sed -i s/3000:3000/3100:3000/ run-lgtm.sh注意最后一行命令因为LGTM的Grafana的默认对外端口是3000这个端口经常和一些应用程序冲突我就改成了3100.前面几篇文章我都在启动LGTM时激活了OBI可以增加更多的metrics。但是如果LGTM和被监控的应用不在同一台机器时会多出一些配置OBI的步骤。这次我干脆用默认方式(不激活OBI)。下面是启动LGTM的方法就一个命令cd /opt/lgtm ./run-lgtm.shLGTM要监听以下的端口4317/4318 是OTLP端口用来接收metrics/traces/logs数据3100 是Grafana UI的端口9090 是Prometheus的端口用于调试4040 是Pyroscope接收Profiles的端口将来会整合进入4317/43183. 如何配置和启动Python应用来激活OpenTelemetry监控1) 增加Python包opentelemetry-distro 和 opentelemetry-exporter-otlp一般先要进到应用使用的虚拟环境(venv, Conda之类的)。然后可以简单地使用如下的命令安装Python包pip install opentelemetry-distro opentelemetry-exporter-otlp当然更正规的方式是在应用的requirements.txt里面增加两行opentelemetry-distro opentelemetry-exporter-otlp然后重新执行pip install -r requirements.txt2) 生成自动插桩的包执行如下命令opentelemetry-bootstrap -a install该命令会针对目前安装的Python包配置相应的自动插桩的包。单独执行opentelemetry-bootstrap可以看到这些自动插桩的包。下面是我在某一个虚拟环境执行opentelemetry-bootstrap的结果$ opentelemetry-bootstrap opentelemetry-instrumentation-asyncio0.61b0 opentelemetry-instrumentation-dbapi0.61b0 opentelemetry-instrumentation-logging0.61b0 opentelemetry-instrumentation-sqlite30.61b0 opentelemetry-instrumentation-threading0.61b0 opentelemetry-instrumentation-urllib0.61b0 opentelemetry-instrumentation-wsgi0.61b0 opentelemetry-instrumentation-asgi0.61b0 opentelemetry-instrumentation-click0.61b0 opentelemetry-instrumentation-fastapi0.61b0 opentelemetry-instrumentation-grpc0.61b0 opentelemetry-instrumentation-requests0.61b0 opentelemetry-instrumentation-sqlalchemy0.61b0 opentelemetry-instrumentation-starlette0.61b0 opentelemetry-instrumentation-tortoiseorm0.61b0 opentelemetry-instrumentation-urllib30.61b03) 使用 opentelemetry-instrument 启动应用假如您启动应用的命令是python app.py那可以改成如下命令来启动应用同时激活OpenTelemetryexport OTEL_SERVICE_NAMEpython-demo export OTEL_EXPORTER_OTLP_ENDPOINThttp://localhost:4317 export OTEL_TRACES_EXPORTERotlp export OTEL_METRICS_EXPORTERotlp export OTEL_LOGS_EXPORTERotlp cd /opt/dev/otel/otel-python-demo/ opentelemetry-instrument python app.py当然也可以用uvicorn或者gunicorn。假如您启动应用的命令是uvicorn app:app --host 0.0.0.0 --port 8000那可以改成如下命令来启动应用同时激活OpenTelemetryexport OTEL_SERVICE_NAMEpython-demo export OTEL_EXPORTER_OTLP_ENDPOINThttp://localhost:4317 export OTEL_TRACES_EXPORTERotlp export OTEL_METRICS_EXPORTERotlp export OTEL_LOGS_EXPORTERotlp cd /opt/dev/otel/otel-python-demo/ opentelemetry-instrument uvicorn app:app --host 0.0.0.0 --port 8000注意按照您自己的应用内容来修改OTEL_SERVICE_NAME环境变量如果OpenTelemetry后端(本例采用LGTM)不在本机假如在1.2.3.4则设置相应的OTEL_EXPORTER_OTLP_ENDPOINT为export OTEL_EXPORTER_OTLP_ENDPOINThttp://1.2.3.4:43174. 监控Python应用的Dashboard使用“http://localhost:3100”就可以访问LGTM的Grafana UI界面(如果是远程的话用主机名或者IP替换localhost)使用admin/admin登录。虽然LGTM支持各种类型的数据metrics/traces/logs/profiles但是由于OpenTelemetry对Python的自动监控目前只支持metrics/traces所以我们的Grafana Dashboard只能有这两种数据。以下是我做的Dashboard的截图以下是点击某一个Trace的截图我简单介绍一下这个Dashboard第一行左图是描述吞吐量主要的PromQL是:sum by (http_target, http_status_code) ( rate(http_server_duration_milliseconds_count[2m]) )第一行右图是描述HTTP服务器延迟时间(90%)主要的PromQL是:histogram_quantile(0.90, sum by (le, http_target) (rate(http_server_duration_milliseconds_bucket[2m])))第二行左图是描述返回包的大小主要的PromQL是:histogram_quantile(0.90, sum by (le, http_target) (rate(http_server_response_size_bytes_bucket[2m])))第二行右图是描述HTTP客户机延迟时间(P90)主要的PromQL是:histogram_quantile(0.90, sum without (http_flavor,http_scheme,job) (rate(http_client_duration_milliseconds_bucket[2m])))第三行左图是描述活跃的HTTP请求数主要的PromQL是:sum without (__name__,job,http_flavor,http_scheme) (http_server_active_requests)第三行右图是描述DB连接数主要的PromQL是:avg without (__name__,job) ( db_client_connections_usage )第四行左图是描述服务之间的调用关系。第四行右图是所有Traces。以下是整个Dashboard的代码可以直接导入。{ annotations: { list: [ { builtIn: 1, datasource: { type: grafana, uid: -- Grafana -- }, enable: true, hide: true, iconColor: rgba(0, 211, 255, 1), name: Annotations Alerts, type: dashboard } ] }, editable: true, fiscalYearStartMonth: 0, graphTooltip: 0, links: [], panels: [ { datasource: { type: prometheus, uid: prometheus }, fieldConfig: { defaults: { color: { mode: palette-classic }, custom: { axisBorderShow: false, axisCenteredZero: false, axisColorMode: text, axisLabel: , axisPlacement: auto, barAlignment: 0, barWidthFactor: 0.6, drawStyle: line, fillOpacity: 0, gradientMode: none, hideFrom: { legend: false, tooltip: false, viz: false }, insertNulls: false, lineInterpolation: linear, lineStyle: { fill: solid }, lineWidth: 1, pointSize: 5, scaleDistribution: { type: linear }, showPoints: never, showValues: false, spanNulls: false, stacking: { group: A, mode: none }, thresholdsStyle: { mode: off } }, mappings: [], thresholds: { mode: absolute, steps: [ { color: green, value: 0 }, { color: red, value: 80 } ] }, unit: reqps }, overrides: [] }, gridPos: { h: 8, w: 12, x: 0, y: 0 }, id: 2, options: { legend: { calcs: [], displayMode: list, placement: bottom, showLegend: true }, tooltip: { hideZeros: false, mode: single, sort: none } }, pluginVersion: 12.4.1, targets: [ { datasource: { type: prometheus, uid: prometheus }, editorMode: code, expr: sum by (http_target, http_status_code) (\r\n rate(http_server_duration_milliseconds_count[2m])\r\n), instant: true, interval: , legendFormat: __auto, range: true, refId: A } ], title: 吞吐量, type: timeseries }, { datasource: { type: prometheus, uid: prometheus }, fieldConfig: { defaults: { color: { mode: palette-classic }, custom: { axisBorderShow: false, axisCenteredZero: false, axisColorMode: text, axisLabel: , axisPlacement: auto, barAlignment: 0, barWidthFactor: 0.6, drawStyle: line, fillOpacity: 0, gradientMode: none, hideFrom: { legend: false, tooltip: false, viz: false }, insertNulls: false, lineInterpolation: linear, lineWidth: 1, pointSize: 5, scaleDistribution: { type: linear }, showPoints: never, showValues: false, spanNulls: false, stacking: { group: A, mode: none }, thresholdsStyle: { mode: off } }, mappings: [], thresholds: { mode: absolute, steps: [ { color: green, value: 0 }, { color: red, value: 80 } ] }, unit: ms }, overrides: [] }, gridPos: { h: 8, w: 12, x: 12, y: 0 }, id: 1, options: { legend: { calcs: [], displayMode: list, placement: bottom, showLegend: true }, tooltip: { hideZeros: false, mode: single, sort: none } }, pluginVersion: 12.4.1, targets: [ { datasource: { type: prometheus, uid: prometheus }, editorMode: code, expr: histogram_quantile(0.90, sum by (le, http_target) (rate(http_server_duration_milliseconds_bucket[2m]))), instant: true, legendFormat: __auto, range: true, refId: A } ], title: HTTP服务器延迟时间(P90), type: timeseries }, { datasource: { type: prometheus, uid: prometheus }, fieldConfig: { defaults: { color: { mode: palette-classic }, custom: { axisBorderShow: false, axisCenteredZero: false, axisColorMode: text, axisLabel: , axisPlacement: auto, barAlignment: 0, barWidthFactor: 0.6, drawStyle: line, fillOpacity: 0, gradientMode: none, hideFrom: { legend: false, tooltip: false, viz: false }, insertNulls: false, lineInterpolation: linear, lineWidth: 1, pointSize: 5, scaleDistribution: { type: linear }, showPoints: auto, showValues: false, spanNulls: false, stacking: { group: A, mode: none }, thresholdsStyle: { mode: off } }, mappings: [], thresholds: { mode: absolute, steps: [ { color: green, value: 0 }, { color: red, value: 80 } ] }, unit: bytes }, overrides: [] }, gridPos: { h: 8, w: 12, x: 0, y: 8 }, id: 3, options: { legend: { calcs: [], displayMode: list, placement: bottom, showLegend: true }, tooltip: { hideZeros: false, mode: single, sort: none } }, pluginVersion: 12.4.1, targets: [ { datasource: { type: prometheus, uid: prometheus }, editorMode: code, expr: histogram_quantile(0.90, sum by (le, http_target) (rate(http_server_response_size_bytes_bucket[2m]))), instant: false, legendFormat: __auto, range: true, refId: A } ], title: 返回包的大小, type: timeseries }, { datasource: { type: prometheus, uid: prometheus }, fieldConfig: { defaults: { color: { mode: palette-classic }, custom: { axisBorderShow: false, axisCenteredZero: false, axisColorMode: text, axisLabel: , axisPlacement: auto, barAlignment: 0, barWidthFactor: 0.6, drawStyle: line, fillOpacity: 0, gradientMode: none, hideFrom: { legend: false, tooltip: false, viz: false }, insertNulls: false, lineInterpolation: linear, lineWidth: 1, pointSize: 5, scaleDistribution: { type: linear }, showPoints: auto, showValues: false, spanNulls: false, stacking: { group: A, mode: none }, thresholdsStyle: { mode: off } }, mappings: [], thresholds: { mode: absolute, steps: [ { color: green, value: 0 }, { color: red, value: 80 } ] }, unit: ms }, overrides: [] }, gridPos: { h: 8, w: 12, x: 12, y: 8 }, id: 5, options: { legend: { calcs: [], displayMode: list, placement: bottom, showLegend: true }, tooltip: { hideZeros: false, mode: single, sort: none } }, pluginVersion: 12.4.1, targets: [ { datasource: { type: prometheus, uid: prometheus }, editorMode: code, expr: histogram_quantile(0.90, sum without (http_flavor,http_scheme,job) (rate(http_client_duration_milliseconds_bucket[2m]))), instant: false, legendFormat: __auto, range: true, refId: A } ], title: HTTP客户机延迟时间(P90), type: timeseries }, { datasource: { type: prometheus, uid: prometheus }, fieldConfig: { defaults: { color: { mode: palette-classic }, custom: { axisBorderShow: false, axisCenteredZero: false, axisColorMode: text, axisLabel: , axisPlacement: auto, barAlignment: 0, barWidthFactor: 0.6, drawStyle: line, fillOpacity: 0, gradientMode: none, hideFrom: { legend: false, tooltip: false, viz: false }, insertNulls: false, lineInterpolation: linear, lineWidth: 1, pointSize: 5, scaleDistribution: { type: linear }, showPoints: auto, showValues: false, spanNulls: false, stacking: { group: A, mode: none }, thresholdsStyle: { mode: off } }, mappings: [], thresholds: { mode: absolute, steps: [ { color: green, value: 0 }, { color: red, value: 80 } ] } }, overrides: [] }, gridPos: { h: 8, w: 12, x: 0, y: 16 }, id: 6, options: { legend: { calcs: [], displayMode: list, placement: bottom, showLegend: true }, tooltip: { hideZeros: false, mode: single, sort: none } }, pluginVersion: 12.4.1, targets: [ { datasource: { type: prometheus, uid: prometheus }, editorMode: code, expr: sum without (__name__,job,http_flavor,http_scheme) (http_server_active_requests), instant: false, legendFormat: __auto, range: true, refId: A } ], title: 活跃的HTTP请求数, type: timeseries }, { datasource: { type: prometheus, uid: prometheus }, fieldConfig: { defaults: { color: { mode: palette-classic }, custom: { axisBorderShow: false, axisCenteredZero: false, axisColorMode: text, axisLabel: , axisPlacement: auto, barAlignment: 0, barWidthFactor: 0.6, drawStyle: line, fillOpacity: 0, gradientMode: none, hideFrom: { legend: false, tooltip: false, viz: false }, insertNulls: false, lineInterpolation: linear, lineWidth: 1, pointSize: 5, scaleDistribution: { type: linear }, showPoints: auto, showValues: false, spanNulls: false, stacking: { group: A, mode: none }, thresholdsStyle: { mode: off } }, mappings: [], thresholds: { mode: absolute, steps: [ { color: green, value: 0 }, { color: red, value: 80 } ] } }, overrides: [] }, gridPos: { h: 8, w: 12, x: 12, y: 16 }, id: 4, options: { legend: { calcs: [], displayMode: list, placement: bottom, showLegend: true }, tooltip: { hideZeros: false, mode: single, sort: none } }, pluginVersion: 12.4.1, targets: [ { datasource: { type: prometheus, uid: prometheus }, editorMode: code, expr: avg without (__name__,job) (\r\n db_client_connections_usage\r\n)\r\n, instant: false, legendFormat: __auto, range: true, refId: A } ], title: DB连接数, type: timeseries }, { datasource: { type: tempo, uid: tempo }, fieldConfig: { defaults: {}, overrides: [] }, gridPos: { h: 8, w: 12, x: 0, y: 24 }, id: 7, options: { edges: {}, layoutAlgorithm: layered, nodes: {}, zoomMode: cooperative }, pluginVersion: 12.4.1, targets: [ { datasource: { type: tempo, uid: tempo }, key: Q-00d81243-72c6-4f8c-b817-2f3b91dacad4-0, limit: 20, metricsQueryType: range, queryType: serviceMap, refId: A, serviceMapUseNativeHistograms: false, tableType: traces } ], title: 服务调用图, type: nodeGraph }, { datasource: { type: tempo, uid: tempo }, fieldConfig: { defaults: { custom: { align: auto, cellOptions: { type: auto }, footer: { reducers: [] }, inspect: false }, mappings: [], thresholds: { mode: absolute, steps: [ { color: green, value: 0 }, { color: red, value: 80 } ] } }, overrides: [] }, gridPos: { h: 8, w: 12, x: 12, y: 24 }, id: 8, options: { cellHeight: sm, showHeader: true }, pluginVersion: 12.4.1, targets: [ { datasource: { type: tempo, uid: tempo }, filters: [ { id: 0d6f56a3, operator: , scope: span } ], limit: 20, metricsQueryType: range, queryType: traceqlSearch, refId: A, serviceMapUseNativeHistograms: false, tableType: traces } ], title: Traces, type: table } ], preload: false, schemaVersion: 42, tags: [], templating: { list: [] }, time: { from: now-1h, to: now }, timepicker: {}, timezone: browser, title: Python Dashboard, uid: admnssk, version: 20, weekStart: }5. 总结本文介绍了使用OpenTelemetry零代码插桩方式监控Python应用的方法。主要内容包括1快速部署基于Docker的Grafana LGTM作为监控后端2通过安装opentelemetry-distro包和自动插桩工具实现Python应用的自动监控3配置环境变量并使用opentelemetry-instrument启动应用4展示了包含吞吐量、延迟时间、服务调用关系等关键指标的Grafana仪表板。该方法无需修改代码即可实现Python应用的全面监控适合中小型系统快速搭建监控体系。