日志归档

简要概述

工作流运行默认不依赖数据库。

Argo 将工作流存储为 Kubernetes 资源,因为此类型资源默认必须小于 1MB,每个资源都包含每个节点的状态,这些状态存储在 “/status/nodes” 字段中。当这个字段超过 1MB 时,会尝试压缩节点状态并将其存储在 “/status/compressedNodes” 中。如果状态仍然过大,则会要求将其存储在数据库中。

或者因为审计等问题需要把已经完成的任务做归档以便后续查询,此时就需要配置上数据库(支持:mysql 或 postgres)。

当数据库异常时:

  1. 如果 “workflow-controller” 服务已在运行,则不影响后续任务执行;
  2. 如果 “workflow-controller” 重启则会无法启动,因为无法连接上数据库;

有两类数据需要归档:

  1. 任务运行的元数据,归档至数据库;
  2. 任务运行的日志与附件,归档至对象存储;

归档元数据

配置开启

apiVersion: v1
kind: ConfigMap
metadata:
  name: workflow-controller-configmap
  namespace: argo
data:
  ......

  # enable persistence using postgres
  persistence: |
    connectionPool:
      maxIdleConns: 100
      maxOpenConns: 0
      connMaxLifetime: 0s # 0 means connections don't have a max lifetime
    #  if true node status is only saved to the persistence DB to avoid the 1MB limit in etcd
    nodeStatusOffLoad: false
    # save completed workloads to the workflow archive
    # 在工作流结束后把信息归档到数据库
    archive: true
    # the number of days to keep archived workflows (the default is forever)
    # 多少天之后移除已归档的信息
    archiveTTL: 180d
    # skip database migration if needed.
    # skipMigration: true

    # LabelSelector determines the workflow that matches with the matchlabels or matchrequirements, will be archived.
    # https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
    # 仅符合特定标签任务才做归档
    #archiveLabelSelector:
    #  matchLabels:
    #    workflows.argoproj.io/archive-strategy: "always"

    # Optional name of the cluster I'm running in. This must be unique for your cluster.
    # 改集群唯一标识,用于多 argo 集群下区分
    clusterName: default
    postgresql:
      host: localhost
      port: 5432
      database: argo
      tableName: argo_workflows
      # the database secrets must be in the same namespace of the controller
      userNameSecret:
        name: argo-database-config
        key: username
      passwordSecret:
        name: argo-database-config
        key: password
      ssl: false
      # sslMode must be one of: disable, require, verify-ca, verify-full
      # you can find more information about those ssl options here: https://godoc.org/github.com/lib/pq
      sslMode: disable

    # Optional config for mysql:
    # mysql:
    #   host: localhost
    #   port: 3306
    #   database: argo
    #   tableName: argo_workflows
    #   userNameSecret:
    #     name: argo-mysql-config
    #     key: username
    #   passwordSecret:
    #     name: argo-mysql-config
    #     key: password    

  ......

数据库权限

apiVersion: v1
kind: Secret
metadata:
  name: argo-database-config
  namespace: argo
type: Opaque
data:
  username: dXB0aW1l
  password: dGVzdGtleTI=
create database argo;

数据库表结构

argo=> \dt
                    List of relations
 Schema |              Name              | Type  | Owner
--------+--------------------------------+-------+--------
 public | argo_archived_workflows        | table | uptime
 public | argo_archived_workflows_labels | table | uptime
 public | schema_history                 | table | uptime
 public | workflows                      | table | uptime
(4 rows)

argo=>
argo=> \d argo_archived_workflows
                        Table "public.argo_archived_workflows"
   Column    |            Type             | Collation | Nullable |      Default
-------------+-----------------------------+-----------+----------+-------------------
 uid         | character varying(128)      |           | not null |
 name        | character varying(256)      |           | not null |
 phase       | character varying(25)       |           | not null |
 namespace   | character varying(256)      |           | not null |
 workflow    | json                        |           | not null |
 startedat   | timestamp without time zone |           | not null | CURRENT_TIMESTAMP
 finishedat  | timestamp without time zone |           | not null | CURRENT_TIMESTAMP
 clustername | character varying(64)       |           | not null |
 instanceid  | character varying(64)       |           | not null |
Indexes:
    "argo_archived_workflows_pkey" PRIMARY KEY, btree (clustername, uid)
    "argo_archived_workflows_i1" btree (clustername, instanceid, namespace)
    "argo_archived_workflows_i2" btree (clustername, instanceid, finishedat)
    "argo_archived_workflows_i3" btree (clustername, instanceid, name)
    "argo_archived_workflows_i4" btree (startedat)
Referenced by:
    TABLE "argo_archived_workflows_labels" CONSTRAINT "argo_archived_workflows_labels_clustername_uid_fkey" FOREIGN KEY (clustername, uid) REFERENCES argo_archived_workflows(clustername, uid) ON DELETE CASCADE

argo=>
argo=> \d argo_archived_workflows_labels
             Table "public.argo_archived_workflows_labels"
   Column    |          Type          | Collation | Nullable | Default
-------------+------------------------+-----------+----------+---------
 clustername | character varying(64)  |           | not null |
 uid         | character varying(128) |           | not null |
 name        | character varying(317) |           | not null |
 value       | character varying(63)  |           | not null |
Indexes:
    "argo_archived_workflows_labels_pkey" PRIMARY KEY, btree (clustername, uid, name)
    "argo_archived_workflows_labels_i1" btree (name, value)
Foreign-key constraints:
    "argo_archived_workflows_labels_clustername_uid_fkey" FOREIGN KEY (clustername, uid) REFERENCES argo_archived_workflows(clustername, uid) ON DELETE CASCADE

argo=>
argo=> \d schema_history
               Table "public.schema_history"
     Column     |  Type   | Collation | Nullable | Default
----------------+---------+-----------+----------+---------
 schema_version | integer |           | not null |

argo=>
argo=> \d workflows
                               Table "public.workflows"
   Column    |            Type             | Collation | Nullable |      Default
-------------+-----------------------------+-----------+----------+-------------------
 uid         | character varying(128)      |           | not null |
 namespace   | character varying(256)      |           | not null |
 clustername | character varying(64)       |           | not null |
 version     | character varying(64)       |           | not null |
 nodes       | json                        |           | not null |
 updatedat   | timestamp without time zone |           | not null | CURRENT_TIMESTAMP
Indexes:
    "workflows_pkey" PRIMARY KEY, btree (clustername, uid, version)
    "workflows_i1" btree (clustername, namespace, updatedat)

argo=>

归档日志与附件

配置开启