Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-8976

Add heuristic-based influences to the unbuffered resume position frequency

XMLWordPrintable

    • Icon: Enhancement Enhancement
    • Resolution: Unresolved
    • Icon: Major Major
    • 3.2.0.Beta2
    • None
    • oracle-connector
    • None
    • False
    • None
    • False

      In the unbuffered LogMiner adapter implementation (DBZ-8924), the low watermark position is recomputed based on a configurable interval. While this works in practice, there may be situations where the configured interval may be too long during high activity windows or may be too frequent in low activity windows.

      Based on this conversation:
      https://github.com/debezium/debezium/pull/6341#issuecomment-2831745557

      I propose we consider introducing several heuristic-based influences that would override the user's defined interval. Some ideas may include:

      1. If the number of logs or combined log sizes exceed a specific threshold
      2. If the distance between the low watermark and most recent commit exceed a threshold
      3. If the number of recently seen transactions exceed a threshold
      4. If the number of events of a transaction exceed a threshold

      This makes sure that (1) addresses high burst log windows, (3) addresses recalculating early in high activity windows, (4) addresses recalculation in large transaction windows.

              Unassigned Unassigned
              ccranfor@redhat.com Chris Cranford
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: